Running outlier_detection with in_memory=False reads all models into memory

[stscijgbot-jp]

Issue JP-3619 was created on JIRA by Brett Graham:

Running outlier detection with the recent main commit: 13e0927

With an association containing 33 members (with no custom grouping or tweakreg_catalog entries for any member).

And with "in_memory=False" results in the following memory usage:

The attached graph shows that even with "in_memory=False" all models are read into memory (the largest plateau in the graph). The main cause is that the implementation of ModelContainer.models_grouped (which is used in resample) opens all models and keeps them in memory.

[jdavies-st]

I will add here that when in_memory=True for outlier_detection, it still writes out the blot files and then reads them back in.

2024-05-15 15:48:21,169 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (2048, 2048) <-- (6321, 9320)
2024-05-15 15:48:21,455 - stpipe.Image3Pipeline.outlier_detection - INFO - Saved model in jw02321001009_02101_00006_nrcblong_c1001_blot.fits

So it is writing to disk when it should be doing everything in memory. This is with release 1.14.0.

[stscijgbot-jp]

Comment by Ned Molter on JIRA:

This should be fixed as part of implementing ModelLibrary (see JP-3690).

Here is a small summary of results from profiling memory for this step on its own in my PR branch, using as input an association containing a 46-cal-file subset of the data in 

███████████████████████████████████████████ 

 

The total size of the input _cal files on disk is roughly 5 GiB.

Setting in_memory=True, the peak memory usage is 31.3 GiB, and I'm attaching a graph of usage over time to the ticket (outlier_in_memory.png, sorry for the similar name to what Brett attached originally).

Setting in=memory=False, the peak memory usage is 7.1 GiB, and a graph is again attached (outlier_on_disk.png).

So this appears to be fixed: although additional memory improvements to this step are certainly possible and probably still necessary, it is clear that the input models are never being loaded into memory all at once.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Fixed by #8683