Cookbook ideas

[jukent]

Ideas from today's ESDS forum. Please comment yours!

[jukent]

• How best to use Binder with large data that is stored somewhere else? (intake most likely!)

• Best practices for running Jupyter on Supercomputers (how to make sure you shut down your server, e.g.)

• Environment management best practices and troubleshooting techniques to solve certain problems (downgrade a package etc -- will need more examples of these problems - should we show how GitHub actions does this for us?)

• Making a Python package

[samsrabin]

We have some cookbook-type notebooks for CTSM at the ctsm_python_gallery repo. At least some of these are based on a set of utilities (utils.py) for doing various things with xarray including:

• Convert DataArray units
• Clean up DataArray unit strings
• Get a weighted annual mean from monthly outputs
• Set Dataset time values to midpoint of time_bounds values
• Get time as year + fraction of year
• Add a cyclic coordinate to a DataArray
• Convert a latitude axis that's -180 to 180 around the international date line to one that's 0 to 360 around the prime meridian
• As the above, but vice-versa
• Read multiple netCDF files to in-memory Dataset, optionally removing unneeded variables and/or slices.
• Grid a variable from a netCDF in the collapsed-grid format (with patches1d_ixy etc.)

Would be great to see if y'all have any optimizations or generalizations for those functions as well!

[clyne]

@samsrabin while we're' still trying to figure out what the definition of a 'cookbook' is, some of the useful things that you describe sound more like Xarray 'operators': functions that consume and possibly produce new Xarray objects. If you think some of your utilities might be of general use to the geoscience communities, please consider contributing them to GeoCAT.

CC: @erogluorhan

[samsrabin]

Yes, everything in utils.py is a function, but there are some notebooks in that repo that I think I'd call "cookbooks." Or at least, they're examples that could be translated into cookbooks.

I didn't know about GeoCAT, looks useful. Thanks!

[erogluorhan]

Both GeoCAT and CTSM's utils even have some overlapping functions (e.g. annual mean, cyclic longitudes, etc.) already. Your cookbook notebooks look great, providing examples of how to use those functions, @samsrabin ! I second @clyne here, in addition to having those functions showcased in your notebooks, if you'd like to deliver them to the existing GeoCAT user base, please consider contributing them (we are happy to credit any individuals and groups).

Additionally, I'd like to point you out to our GeoCAT-examples, for example this calendar average example, which demonstrates examples of analysis functions in use, in hopes it might be helpful.

Best,

[clyne]

I've created a shared draft document for planning the hackathon. Please review and edit as you see fit.

[r-ford]

I'm working on a cookbook that starts with some of the Pangeo Gallery material. It will probably include using

• Intake-ESM (Pangeo Gallery notebook)
• CMIP6 Preprocessing
• xarrayutils
• xESMF

The goal is to start with the CMIP6 data on Pangeo's Google Storage and end with a certain plot of the multimodel mean ocean heat uptake (specifically Fig. 1b in this paper).

For another cookbook, I think it would be cool to do some sort of EOF analysis with the eofs package, probably also using that CMIP6 data.

[erogluorhan]

You may find checking the GeoCAT's EOF analysis functions that use the eofs package under the hood also useful.

[jukent]

From Teagan King - scripts for interpolating data from terrain following model levels to specific pressure levels

[ktyle]

@kmpaul is this vertical interpolation capability going to be in the initial release of xwrf? Cookbooks related to WRF would be quite popular ...

[kmpaul]

It doesn't look like it. The capability is getting fixed in xgcm, and xwrf is going to use that, but we are waiting for xgcm.