Save resource buildout every few optimizer iterations and allow optimizer warm start

[wrgunther]

GenX runs with longer time domain reduction periods can extend to multiple days, and oftentimes corporate computers force restart on e.g., Patch Tuesday causing a loss of all work. Writing the in-progress resource buildout to a temp file every few iterations could help salvage results and provide early insights. As low hanging fruit, my understanding is that the first cycle of a perfect foresight run is the same as a myopic run.

As a further improvement, it would be helpful to have the ability to warm start an optimization run using an initial condition based on an incomplete run as described above or based on a previously completed scenario with similar inputs. The ability to pause and restart a run could also be helpful when working on portable computers.

On a related note, the use of a scratch file to store in-progress results (e.g., prior/future years not currently being optimized) might also free up computer memory to facilitate larger simulations.

[JesseJenkins]

Can you clarify: are you running multi-stage with foresight (if so, over how many periods)? Or myopic?

It would be relatively easy for us to make myopic multi-stage expansion write outputs for each stage after completing, and you could then pick up where you left off by running only the remaining stages after your computer reboots.

It would be much more difficult to do this with the foresight model.

[wrgunther]

Thanks, and yes, the myopic functionality would be helpful. We typically run myopic multistage with 5 periods, 3 of which are very important (2030, 2035, and 2040) and two of which are because another group likes to see numbers through 2050.

Perfect foresight is the gold standard but including it doesn’t change our results much and the extra run time is not practical in our system. If I knew perfect foresight would come after exporting the myopic results and the computer was otherwise idle, I might consider trying it again.

[JesseJenkins]

Ok thanks. I will tag this as a feature request (to have a settings flag that will cache myopic results after each stage instead of waiting until the end to write) and add it to our to do list.

[GenXProject]

Flagged this to be addressed on v0.4.1

[sambuddhac]

PR #704 and #703 has addressed this issue on develop