grid_from_radars performance for considerable domains

[tealmill]

Hello, I've been trying to use pyart for "simple" task of essentially assembling the composite of NEXRAD radar Level 2 data (but specifically limited to simple reflectivity) between the latitude of 29N and 44N and longitude of 87W 70W, so... essentially the most of USA East Coast and then a good bit inland. I've actually been successful, and I'm at glance fairly happy with the result.

The issue is, this area (with real world boundary of roughly 1650x1650km) with the ~40 NEXRAD radars that seem active at April 2015 (my testing sample) take 30s to load and then staggering 3 minutes to run through grid_from_radars on a fairly powerful machine (Ryzen 5950X). It also requires ~40GB of RAM when processing the grid. The load I can probably get a good bit of performance for by parallelizing it and making it run concurrently with the processing, but the grid_from_radars I am helpless with.

I'm aiming to produce a dataset here that will span years with 10 minute steps; that's a lot of pictures and, since it runs at ~third of the ten minute "realtime"; it'd process at this rate for years again.

my (truncated) code looks like this;

rad = pyart.io.read("KDOX20150401_120002_V06.gz")
...
grid = pyart.map.grid_from_radars(loaded_rads, grid_shape=(1, 1650, 1650), grid_origin=((midlat, midlon)), grid_origin_alt=1500, fields=['reflectivity'], grid_limits=((1500.,1500,), (-1000000., 1000000.), (-1000000, 1000000.)))

Making the grid smaller doesn't really seem to have much effect. The non-default gridding_algo was so slow I haven't even waited to find out how long is it going to take.

What can I do to bring this closer to realm of speed that's realistically acceptable for me? Worst case, I could make do with plain nearest-neighbor zmax aggregate (and worst case, I'll write that from scratch), but I was really hoping to make use of pyart's native support for reading the files, taking care of the earth curvature, etcetera. Is there way to squeeze that speed out?

[scollis]

First, this great! It shows how far Py-ART has come this is doable.
Good news is I have a quick speedup solution for you. Gatefilters.
Create a gatefilter using https://arm-doe.github.io/pyart/API/generated/pyart.correct.GateFilter.exclude_invalid.html
you pass like this (ignore the stuff here about the ring)
https://arm-doe.github.io/pyart/examples/mapping/plot_map_one_radar_to_grid.html

Just getting rid of invalid gates will speed things up
you can get fancy then by despeckling like in here:
https://github.com/openradar/aus-open-radar-science-school/blob/master/notebooks/2.1_Deeper_dive_into_pyart_aussie_example.ipynb

my_gf.exclude_above('differential_reflectivity', 6)
my_gf.exclude_below('cross_correlation_ratio', .8)
my_ds_gf = pyart.correct.despeckle_field(radar, 'reflectivity', gatefilter=my_gf)```

Also, you could use the gate x/y/z locations and gate filter based on that if you are only interested in a mosaic below a certain height.. 

[scollis]

@mgrover1 we should move this or document this in the discourse

[scollis]

here is a better example https://github.com/ARM-Development/PyART-Training/blob/main/2-Cloud_Examples/1-NEXRAD_on_Amazon.ipynb

KILX_f.exclude_masked('reflectivity')
KILX_f.exclude_invalid('reflectivity')

KLOT_f = pyart.filters.GateFilter(KLOT)
KLOT_f.exclude_masked('reflectivity')
KLOT_f.exclude_invalid('reflectivity')

KIWX_f = pyart.filters.GateFilter(KIWX)
KIWX_f.exclude_masked('reflectivity')
KIWX_f.exclude_invalid('reflectivity')
#Lets construct a grid. NOTE: Origin is at the first radar in the tuple
#This takes ~1 minute

grids = pyart.map.grid_from_radars((KILX, KIWX, KLOT),(16,401,401),
                   ((0.,15000.),(-400000.,400000.),(-400000.,400000.)), 
                                   weighting_function='Barnes2',
                                  fields=['reflectivity','cross_correlation_ratio'],
                                  gatefilters=(KILX_f, KIWX_f, KLOT_f))

Also you can save memory by popping unused fields.

for key in ['clutter_filter_power_removed','velocity', 'differential_phase','spectrum_width' ]:
    KILX.fields.pop(key)
    KLOT.fields.pop(key)
    KIWX.fields.pop(key)```

[tealmill]

Thanks for the tips! I'll try all of these and see if it gets me anywhere. Funnily enough I've tried very similar thing to the pops with memory, but instead I tried to use python delete() on the objects coming from the dictionary; I figured it didn't work because of some view/object dualism, I'll see if this works better!

[tealmill]

Even essentially this couple filters alone and trimming the unneeded fields does cut the time grid_from_radars takes from over 3 minutes to 1 minute; that's remarkable and slowly entering realm of feasibility 🙂

[jthielen]

While this isn't in Py-ART itself, the "radar mosaic creation" use-case is what I am building OpenMosaic for on top of Py-ART using Dask and other Pangeo tools. I'm in the midst of a significant API overhaul, so there unfortunately isn't a version immediately ready to use (hopefully by mid-January it'll be ready), but if you wanted to take a look at the prior version (either to use, or take inspiration from), see https://github.com/jthielen/OpenMosaic/tree/old-api.

[tealmill]

Thank you for the suggestion, that's good reference code.