-
Notifications
You must be signed in to change notification settings - Fork 81
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kerchunk and Zarr V3 #235
Comments
Some of this I'll have to think about, but some things I can answer immediately.
I also want to mention that kerchunk should be useful for more than just zarr, so I will tend to favour things being coded in the storage layer rather than zarr-specific extensions. For example, reordering and selecting parquet files without touching the originals is something that kerchunk can do now. If you wanted full tabular iceberg compatibility using kerchunk/referenceFS, one could implement that now without too much trouble. Here is the simplest non-zarr idea for CSVs: #66 (and, more generally, random access of delimited/block compressed data). |
An application for this would be backporting Zarr v3 shards for availability via Zarr v2. |
? I thought one of the main reasons for having a V3 at all was so that we could have new things like sharding ? That is presumably why my working variable-chunking implementation for v2 was not given consideration. |
The Zarr V3 spec is now undergoing public review and testing. This issue raises the question of how Kerchunk should integrate with the new spec.
Key changes in the V3 spec that are relevant particularly to Kerchunk (zarr-developers/zarr-specs#149):
Questions:
The text was updated successfully, but these errors were encountered: