packing & compression

[mo-marqh]

Investigate opportunities and costs of lossy packing and lossless compression of data.

• Packing: reducing the precision of the data to quantity-specific levels of precision
  • lossy - by definition
• Compression: algorithmically processing the data to store a compression representation on storage
  • lossless - by expectation

This ticket is targeting the exploration of the capabilities of XIOS+netCDF+HDF to pack and compress data.

• Packing Techniques:
  • type coercion to integer with scale_factor & add_offset
  • quantization
• Compression Algorithms
  • GNU compression with gzip / zlib
  • ZStandard compression with zstd
• Parallelisation
  • client
  • server (needs a fairly recent version of parallel netCDF+HDF)

[mo-marqh]

XIOS & netCDF support lossless gzip compression.

however, this is only available in more recent netCDF + hdf stacks for parallel write, so some older platforms with old versions of netCDF will not be able to provide gzip compression and parallel writing at the same time

platforms using netcdf 4.8.x and above can support parallel compression

[mo-marqh]

chunking of HDF data
https://portal.hdfgroup.org/documentation/hdf5-docs/advanced_topics/chunking_in_hdf5.html
is important to consider alongside compression, as compression algorithms will run on chunks of data, so how data is segregated within a file has significant influence on the computational impact and storage reduction of files

[mo-marqh]

there is a demonstration programme using scale_factor & add_offset to pack data using XIOS, to a defined precision
https://github.com/MetOffice/tcd-XIOS-demonstration/tree/main/xios_examples/packing_scale_offset

deciding on the precision control values is a significant effort, requiring computer science and atmospheric science input in order to obtain a computationally and scientifically valid packing

mistakes can lead to numerical error, which are problematic within libraries

[mo-marqh]

#27

explores an issue with handling missing data, which is unresolved at this time

[mo-marqh]

https://www.researchgate.net/publication/365006139_NetCDF_Compression_Improvements
https://docs.unidata.ucar.edu/netcdf-c/current/md__media_psf_Home_Desktop_netcdf_releases_v4_9_2_release_netcdf_c_docs_quantize.html
https://www.hdfgroup.org/wp-content/uploads/2021/10/HDF5-Users-Workshop-2021-Additional-Compression-Methods-for-NetCDF.pdf

NetCDF Quantization is an interesting approach to packing, with subtle implementation differences compared to the more traditional type changing

work is required to explore how this facility could interact with XIOS

[mo-marqh]

#27 has now been updated and merged

there is functionality within XIOS2 & XIOS3 to enable XIOS to handle the type conversion of data, and to intercept out of range values and set these