Best-practice for zarr format data collections on linux filesystems

[Thomas-Moore-Creative]

@matear & @e-peacey

RE: the conversation about how to best use zarr on Gadi I have my own issue here that I've not yet had the chance to make progress on. write code to convert zarr collections to zarr-zipstore

While I don't yet have any direct experience with using zarr-ZipStore my understanding is it doesn't appear to effect performance much ( this needs to be tested ) and it solves the inode problem that cloud optimised formats like zarr have on linux filesystems. Normally zarr looks like "one file per chunk" to a linux FS which is typically a much larger inode footprint than netcdf.

See: https://zarr.readthedocs.io/en/stable/api/storage.html#zarr.storage.ZipStore