Implement a checkpointing scheme

[LivInTheLookingGlass]

Potential idea for a general case, assuming no external I/O.

Provide a context object and wrap the requested job in a function that starts a helper thread. This helper thread should attempt to serialize this context object.

• the job reports the object is busy by locking it
• the job opens files via this object, whose states are copied/reflinked at the same frequency as program state
• the serializer thread does this every 30s if no I/O active, 120s if I/O inactive
• serializer thread stores the last {configurable} snapshots
• serializer thread stores this with name job_{number}.snapshot_{number}.bin

Open questions:

• is the random object snapshottable?

A better world would let me do this using this package, but I'm not sure that you use it this way given their API. I'd like to have this be as non-invasive as possible.