Some imported tables contain too much metadata for HDF5 (object header message is too large)

[ljleppan]

NB: This is not strictly an issue with py3-opendata. It is merely documented here to make future debugging easier.

---

Some tables (see partial list below) contain too many columns for HDF5 after being flattened. This manifests as the following stacktrace:

HDF5 error back trace
  File "H5A.c", line 259, in H5Acreate2
    unable to create attribute
  File "H5Aint.c", line 280, in H5A_create
    unable to create attribute in object header
  File "H5Oattribute.c", line 347, in H5O_attr_create
    unable to create new attribute in header
  File "H5Omessage.c", line 224, in H5O_msg_append_real
    unable to create new message
  File "H5Omessage.c", line 1945, in H5O_msg_alloc
    unable to allocate space for message
  File "H5Oalloc.c", line 1142, in H5O_alloc
    object header message is too large

As per Limits in HDF5: Is there an object header limit and how does that affect HDF5? this is a limit in HDF5 versions up-to and including 1.8. At the time of writing, the latest release is 1.10 where this issue -- supposedly -- does not manifest.

Unfortunately, the latest version of HDF5 available on Ubuntu 16.04LTS is libhdf5-10 which, despite its name, merely installs version 1.8.16+docs-4ubuntu1. The likely solution is to install a never HDF5 from sources.

Further testing is required to determine whether the imported DataFrames fit into the 64KB header limit of HDF5 <=1.8 when not flattened.