Maybe make generic._clone and generic._ChunkInfo public

[Hendrik1987]

Hey,
I find above two functions very useful and wonder if they could not be made public. Or, what is the reason that they are private?
Cheers!

[ecomodeller]

They are private because we haven't ourselves needed to access them directly, only via the other functions in the generic module.

Could you put a few words on how you use these and then we can think about how to make them public?

[Hendrik1987]

We have various aggregation routines (vertical averaging, exceedence duration etc.) on large (sometimes larger than memory) 3D dfsu files, for example MT output. I don't think they are generic enough and certainly not coded clean enough to be a part of mikeio.generic. I am taking inspiration in the mikeio.generic functions and thereby rely on above private functions.

Now that you challenged my request, however (thanks for that), I am wondering if the best approach would instead be to try to contribute to some form of generic apply function. Somewhat similar to pandas apply or xarray map, but not on in-memory objects.

This could be something like mikeio.generic.apply(func, filename_in, dim='time', filename_out=None, items=None).

func can be any function on np.arrays returning np.arrays (and thereby applied on da.values), a custom function operating on mikeio.ds or a custom fuction not returning anything (but maybe saving plots to a directory). dim can be time or z, possibly others.

[ecomodeller]

Eager execution is simple, but if it doesn't fit in memory it is 😐
Out of curiosity, how much larger than memory? Have you considered renting a cloud VM, you can get lot's of RAM these days?

[Hendrik1987]

~2-4 times larger on my PC. And yes we have machines with RAM available that could solve this problem. But simply throuwing hardware (especially when in the cloud) at any problem is maybe not always the most sustainable solution in terms of workflow, economy and energy consumption :-)