[Meta] Packaging Spinup-NEMO benchmark

[ma595]

We are now in a position to package the Spinup-NEMO code:

High level view of the algorithm:

1. [PRE-RUN] Run DINO for 50-100 years (minimum). Slurm script has been provided in NEMO notes. If we need to train on more data we then need to concatentate simulation outputs *grid_T.nc using ncrcat.
2. Evaluate metrics here.
3. Run the resampling notebook (See resample_dino_data) branch.
   This notebook converts DINO 2d monthly SSH output DINO_1m_grid_T.nc to annual DINO_1m_To_1y_grid_T.nc. Temperature and salinity (3D) are sampled annually already and are in DINO_1y_grid_T.nc. We can then read these files in the updated notebook for DINO (but it still works for NEMO).
4. Load output in updated Jumper.ipynb notebook and run to create projected state.
5. Evaluate metrics at this point? This relies on having sufficient output from step 1.6. Prepare restart file:
   Combine mesh_mask[0000].nc files and DINO_[<time>]_restart_[<process>].nc (last files) using REBUILD_NEMO tools
   Create new restart file: Run main_restart.py.
   main_restart.py --restart_path /path/to/nemo_data/ --radical DINO_00576000_restart --mask_file /path/to/mesh_mask.nc --prediction_path /path/to/simus_predicted
main_restart.py has been modified to work on DINO data. This is in the run_with_DINO_data branch.
6. Restart DINO with updated restart file.
7. Evaluate metrics at regular intervals (like every 10 years) to see how close the correction brings us to ground truth. Does it converge or diverge?

Specification for packaging the above:

The aim is to provide a tool that allows people in IPSL to do ML assisted spinup without external help. The packaging should enable researchers to experiment with different parameters / inputs / addition of other metrics:

• jump interval.
• quantity of data at start.
• a jump strategy i.e. 10 intervals, scaling to 30 years at end progressively.
• algorithms like PCA.
• add more metrics.

1. Provide a script that automates these above steps:
   • This needs to be executed end-to-end
     • Docker
     • pip install
2. Provide documentation:
   • A report with output file. What should this output report provide? Some idea of how far we are away from the ground truth at various points?
   • Metrics (provide instructions on how to add more metrics).
3. Take in other inputs, i.e., .npy from Etienne's diffusion process.
4. Run iteratively
   • Nemo + projection multiple times.

Development strategy:

• Convert notebooks into scripts / modules.
  • Resampling.py
  • What do main_forecast.py and main_restart.py do - can probably be modified for purposes of providing an automated tool.