2024‐01‐24

January 24, 2024

Agenda:

1. Follow-Up from ESDS Annual Event
2. Next steps
   • High priority issues for bare-bones deployment
   • Parallelization
3. Collaborative work time
   • Help with open PRs
   • Work on installing / running (if you haven’t done so already)
   • New notebooks

Slides

Notes

• Note that you need to be part of the NCAR Github organization to view the Project Boards.
• ADF notebook is currently pretty bare-bones, Ocean notebook is similar (in terms of number of diagnostics)
• Probably need to figure out the parallelization first before adding diagnostics
  • Multiple pathways for addressing this
  • Notebooks should be run in parallel
• Also need to consider multiple machines, queueing systems, not hardcoding project accounts
  • LocalCluster probably the way to go - need to test out on HPC
  • Notebooks will need to be connected to that cluster
  • CUPiD should handle top-level request (i.e., spin-up the cluster)
  • Alternatively, CUPiD could be configured to be "machine-aware"
• Original implementation was spinning up one cluster and passing scheduler to individual notebooks
  • Mysterious timeout errors trying to troubleshoot - might need more people to help
• How long did CESM_postprocessing take? What's our target time?
  • Sea ice alone: 2-3 hours
  • All 4 component packages: 6-8 hours?
  • Parallelism in that was ~8 processors, only task parallelism
• Workflow with PRs and rapid changes: will be adding documentation for contributing
• Breakout groups
  • Work on open PRs
  • First time install/running CUPiD
  • Parallelization
  • New notebooks