From 03166c6ab1570d76ad518fa1bd5be37d3cf987fa Mon Sep 17 00:00:00 2001 From: mattjala <124107509+mattjala@users.noreply.github.com> Date: Wed, 24 Apr 2024 10:48:35 -0500 Subject: [PATCH 1/6] Fix header in schema docs --- docs/design/obj_store_schema/obj_store_schema_v2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/obj_store_schema/obj_store_schema_v2.md b/docs/design/obj_store_schema/obj_store_schema_v2.md index 87054c0c..55d9b184 100644 --- a/docs/design/obj_store_schema/obj_store_schema_v2.md +++ b/docs/design/obj_store_schema/obj_store_schema_v2.md @@ -92,7 +92,7 @@ By contrast the object storage schema stores each HDF object as an object storag The goal of the object schema is to be of sufficient fidelity that it should be possible to convert a traditional HDF5 file to a set of objects, and then convert the set of objects to a HDF5 file that is equivalent to the original file. -Comparison of managing HDF5 entities in a file vs. an object store +## Comparison of managing HDF5 entities in a file vs. an object store --- From d88de061b5e011517118f86289d92280c7cc5336 Mon Sep 17 00:00:00 2001 From: mattjala <124107509+mattjala@users.noreply.github.com> Date: Wed, 24 Apr 2024 10:57:09 -0500 Subject: [PATCH 2/6] Update obj_store_schema_v2.md --- docs/design/obj_store_schema/obj_store_schema_v2.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design/obj_store_schema/obj_store_schema_v2.md b/docs/design/obj_store_schema/obj_store_schema_v2.md index 55d9b184..0dd584e4 100644 --- a/docs/design/obj_store_schema/obj_store_schema_v2.md +++ b/docs/design/obj_store_schema/obj_store_schema_v2.md @@ -103,7 +103,7 @@ Management of HDF5 entities in an object store brings up a different set of cons 3. The object storage system doesn't provide the equivalent of an append operation, so the entire object must be re-written for each write (though partial reads are supported) 4. Performance is sensitive to the size of objects in the object store (c.f. ) 5. Given that writes to the object store are atomic, there is no possibility that the storage system will be left in an inconsistent state -6. Certain functions that are typically performed by the filesystem (e.g. listing files, file permissions) we need to be managed by the service (e.g. there needs to be the ability to store the access rights for a given object +6. Certain functions that are typically performed by the filesystem (e.g. listing files, file permissions) we need to be managed by the service (e.g. there needs to be the ability to store the access rights for a given object) 7. Unlike HDF5 entities in a file, the "file" an object store object is contained in is not immediately apparent. The connection between objects and the "file" they are contained in needs to be explicitly managed. ## Additions to the HDF5 data model to support the HDF REST API @@ -130,7 +130,7 @@ within the same domain: d-b03b24ef-69f244b6-56e5-25125a-89ba79 The id for the root group uses the same layout, but the second half of the id is based on the first half. A root group id formed by -taking a handom 16 character hex string and rotating each character by 8 to form the next 16 characters (exclusive of the hyphens). For +taking a random 16 character hex string and rotating each character by 8 to form the next 16 characters (exclusive of the hyphens). For example the root group for the two ids above would be: g-b03b24ef-69f244b6-38b3-ac67e1-7acc3e From 9f245c5b37e5030e3f5c2182ac8f41b70b49cd14 Mon Sep 17 00:00:00 2001 From: mattjala <124107509+mattjala@users.noreply.github.com> Date: Wed, 24 Apr 2024 11:09:32 -0500 Subject: [PATCH 3/6] Update obj_store_schema_v2.md --- docs/design/obj_store_schema/obj_store_schema_v2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/obj_store_schema/obj_store_schema_v2.md b/docs/design/obj_store_schema/obj_store_schema_v2.md index 0dd584e4..c20afb1b 100644 --- a/docs/design/obj_store_schema/obj_store_schema_v2.md +++ b/docs/design/obj_store_schema/obj_store_schema_v2.md @@ -269,7 +269,7 @@ Within the username key there are six required sub-keys that each have a value o - "readACL" - If true, the user has permission to read any ACL in the domain - "updateACL" - If true, the has permission to modify the ACL (including adding additional usernames) -Note: optionally, an ACL key can be used in a group, dataset, or committed datatype object. If an ACL is present, it is can be used to enforce permissions for that object. If not present, the domain ACL is used as described above. +Note: optionally, an ACL key can be used in a group, dataset, or committed datatype object. If an ACL is present, it can be used to enforce permissions for that object. If not present, the domain ACL is used as described above. Example: Using the ACLs defined for the "my_domain" object above, user "test_user1" would be authorized to make any change to objects in the domain, or change the ACL itself. User "joebob" (not listed in the ACL keys), would have permission to perform any read operation (assuming a more restrictive ACL is not present in the requested object), but not have authority to modify or delete any object. From 685a98ecd81c3bd78379f290f2537ec50f3d9fc1 Mon Sep 17 00:00:00 2001 From: mattjala <124107509+mattjala@users.noreply.github.com> Date: Wed, 24 Apr 2024 11:35:11 -0500 Subject: [PATCH 4/6] Update obj_store_schema_v2.md --- docs/design/obj_store_schema/obj_store_schema_v2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/obj_store_schema/obj_store_schema_v2.md b/docs/design/obj_store_schema/obj_store_schema_v2.md index c20afb1b..5fa10454 100644 --- a/docs/design/obj_store_schema/obj_store_schema_v2.md +++ b/docs/design/obj_store_schema/obj_store_schema_v2.md @@ -538,7 +538,7 @@ The chunk storage key is of the form: Where: - <uuid1> is the first 16 hex characters of the dataset id the chunk belongs to -- <uuid1> is the second 16 hex characters of the dataset id +- <uuid2> is the second 16 hex characters of the dataset id - Following the <uuid> there is a series of stringified integers separated by underscores. The number of integers is equal to the rank (number of dimensions) of the dataset. - The coordinates <i>, <j>, <k>, etc. identify the coordinate of the chunk (fastest varying dimension last) From be0e8bb1d0edb4af321e6746de438c76a9bc0f3c Mon Sep 17 00:00:00 2001 From: mattjala <124107509+mattjala@users.noreply.github.com> Date: Wed, 24 Apr 2024 11:38:21 -0500 Subject: [PATCH 5/6] Update obj_store_schema_v2.md --- docs/design/obj_store_schema/obj_store_schema_v2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/obj_store_schema/obj_store_schema_v2.md b/docs/design/obj_store_schema/obj_store_schema_v2.md index 5fa10454..99082e41 100644 --- a/docs/design/obj_store_schema/obj_store_schema_v2.md +++ b/docs/design/obj_store_schema/obj_store_schema_v2.md @@ -562,7 +562,7 @@ If the chunk is not compressed, the size of the object would be `10 * 10 * Date: Wed, 24 Apr 2024 11:48:10 -0500 Subject: [PATCH 6/6] Add description of links as sub-objects --- .../obj_store_schema/obj_store_schema_v2.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/design/obj_store_schema/obj_store_schema_v2.md b/docs/design/obj_store_schema/obj_store_schema_v2.md index 99082e41..84846ffd 100644 --- a/docs/design/obj_store_schema/obj_store_schema_v2.md +++ b/docs/design/obj_store_schema/obj_store_schema_v2.md @@ -644,6 +644,7 @@ The specification for these borrows heavily from the HDF5/JSON specification, so - dataspace - attribute - creationProperties +- link ### Type @@ -757,6 +758,24 @@ The following example shows properties for "allocTime", "fillValue", and "layout } ``` +### Links + +Links are stored on group objects, and represent pointers to other HDF5 data model objects. Links may be hard, soft, or external. + +An example of JSON describing links within a group is given here: . + +#### Link example + +The following example describes a hard link named "g1" which uses a UUID to point to a group. +```json +{ + "collection": "groups", + "title": "g1", + "class": "H5L_TYPE_HARD", + "id": "a6c3f58c-7bf7-11e4-a370-3c15c2da029e" +} +``` + ## Related documents The following documents provided related material that may be of use: