Per element metadata

[ivirshup]

It would be nice to have metadata per element in the anndata object. That is, X and each layer could have metadata attached.

• All of our on disk formats support this
• R generally supports this
• numpy arrays/ sparse arrays do NOT support this

Use cases

• Storing provenance information on the objects
• Storing colors with a categorical labelling
• Storing parameters of methods
• Storing information on the semantic meaning of an array. E.g. “this is coordinate”, “this is a probabilistic labelling”

Possible solutions

Metadata is stored separately from the elements

The anndata object could carry around the metadata, not the elements themselves. This could be implemented as a kind of “shadow” of the object which only contains metadata at the same paths. This is not so different from storing metadata in uns, but it has “path based” access, which could be easier to keep up to date.

Every element carries around it’s own metadata

This works for some objects. pandas DataFrames and xarray objects have metadata slots on them. However numpy arrays and basic python objects do not.

Two implementation strategies immediately come to mind here:

Use xarray for everything

We could try and use xarray for everything. But there are some incompatibilities there we’d have to work out #744

Custom classes 😰

We could also make custom subclasses for all the element types we want to support (that don’t already cover this). But subclassing pydata objects is scary.

Alternatives

• Metadata could all be stored in uns, basically how we do it right now.
• Metadata could all be stored externally through a logging system
  • Other frameworks would have to adopt this
  • Ideally metadata could be transferred in the same object as the data. We'd need a way of encoding this logs.

[ivirshup]

Some concrete use cases include:

• OME-NGFF metadata
• FOM/ MAMS metadata