Download a regional subset instead of the full grid?

[blaylockbk]

Question asked via email...

I am doing an analysis over a long time range. However, my region of interest is small. Given how large the HRRR datasets are, I'm looking for a way to subset a given region from within the dataset. For example, either by latitude and longitude bounds or a defined number of grid points around a point of interest.

How can I download a regional subset of the HRRR data instead of the full grid?

[blaylockbk]

Solving this very problem is an active area of research and development in the earth sciences. Unfortunately, subsetting a GRIB file stored on the cloud is not possible without downloading the full file. This is one of the disadvantages of the GRIB file format--it is not very "cloud friendly."

There is a newish file format called Zarr that solves this problem. Zarr files makes it possible to get regional "chunks" of the data without downloading the full file. In 2021, Taylor Gowan started converting some of the HRRR data to Zarr format and much of the real-time HRRR data and recent archive is available on AWS and is maintained by the MesoWest group at the University of Utah Department of Atmospheric Sciences.

NOAA HRRR Archive in Zarr format on AWS (link to hrrrzarr bucket)

Downloading this format is not built into my Herbie package (but maybe someday if I had time and a reason to do it). I do have links to some notebooks that might help you get started.

There is also some documentation from the MesoWest group that may be helpful.

Hopefully that points you in the right direction. Since I'm not currently involved with this effort, if you have questions about the Zarr data you can email [email protected]

[blaylockbk]

Refer to #8

[williamhobbs]

Would it make sense to add an option to use the wgrib2 -small_grib feature [1] to make regional subsets when saving GRIB files? It wouldn't reduce download times and data use, but it would reduce local disk space needed (and maybe processing time/RAM later when re-accessing GRIB files?).

[1] https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/small_grib.html

[blaylockbk]

Yes, that would be nice to have a shortcut in Herbie to do this if wgrib2 is installed.
#109