v3 core spec: purl urls as identifiers

[jstriebel]

Citing from @jbms's and @manzt's comments in #149 (comment):

Currently zarr v3 uses a mixture of full URLs and short strings as identifiers. I can see that there are some advantages to using full URLs: rely on existing DNS registration mechanism so there is no need for a separate "name reservation" process, automatic linking to a corresponding explanatory document assuming that the URLs actually remain valid.

However, I think the long URLs would be cumbersome for a user to actually type manually, and even just looking at the metadata, it will be harder to read at a glance, since e.g. all of the codecs will start with the same long prefix, and just the part at the end is relevant. There are also surely a lot of cases where the user will want to specify something that is currently identified only by a URL, e.g. the codec to use, the metadata format to use, the grid to use, the storage transformer to use, etc.

In order to save users from having to type out full URLs to identify these things, implementations may be tempted to invent their own short identifiers as well, and it may be confusing if these identifiers conflict, and there will be the added complexity of having to relate the short identifiers to the full URLs.

It depends on the implementation, but my guess is that users often won't type these full URLs directly (moreover the implementation will likely fill in this metadata).

It would be nice not to have to define an alternative more concise representation of the zarr v3 metadata, as that would likely lead to added confusion.

More generally, users will need some way to specify everything that is in the metadata when creating a zarr repo/array, ideally in a concise way. If the zarr core spec or extension specs do not provide a concise representation, then each implementation may need to invent its own concise representation, which would be unfortunate.

The following questions seem to arise:

• Do we want to keep the purl URLs as identifiers in the metadata?
• If yes, should there be a common shorthand format for these?
• Technical question: Who updates and maintains these URLs? cc @joshmoore @MSanKeys963 @alimanfoo

My proposal is to keep the purl URLs, and recommend a shorthand format for implementations (the actual metadata must have the full URL, but implementations might use the common shorthands in their API), e.g. https://purl.org/zarr/spec/codec/gzip/1.0 <-> codec/gzip/1.0, dropping the https://purl.org/zarr/ prefix for URLs that are hosted under this prefix.

[jstriebel]

cc @jbms

[jbms]

Related to this --- I wonder if purl has an advantage over just using zarr.dev since it is shorter. If zarr.dev in the future becomes defunct, someone could still access the specification via internet archive.

[joshmoore]

I wonder if purl has an advantage over just using zarr.dev

In terms of trust of the URL, I'd say it does. Perhaps the other way around, what's the cost from your perspective, @jbms?

recommend a shorthand format for implementations ...

I'll add an additional possible feature which is somewhere a @base or similar is specified which defines how URLs are interpreted elsewhere in a fileset. This of course wouldn't be "invariant" (as @rabernat) over moving arrays between filesets.

[jstriebel]

During the last ZEP meeting it seemed like everybody would be fine with using short identifiers (e.g. simply gzip) for everything that part of the official zarr specs. If any external extensions are added, those should probably still use a URL, which also avoids naming conflicts.