From 438598eaaab1c44ad4dcdafba858f65ee6d1bb24 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 12 Apr 2024 16:34:27 +0200 Subject: [PATCH 01/12] OCI cep --- cep-oci.md | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 cep-oci.md diff --git a/cep-oci.md b/cep-oci.md new file mode 100644 index 00000000..bf8be312 --- /dev/null +++ b/cep-oci.md @@ -0,0 +1,62 @@ +# OCI registries as conda channels + +We want to use OCI registries as a storage for conda packages. This CEP specifies how we lay out conda packages on an OCI registry. + +## Specification + +An OCI artifact consists of a manifest and a set of blobs. The manifest is a JSON document that describes the contents of the artifact. The blobs are the actual data that the manifest refers to. The manifest is stored in the registry as a blob, and the blobs are stored in the registry as blobs. + +The manifest consists of some metadata and a number of "layers". Each layer is a reference to a blob. + +Layers can have arbitrary names and mediaTypes. + +An OCI manifest is referenced by a name and a tag. + +### Conda package artifacts on an OCI registry + +The manifest for a conda package on an OCI registry should look like follows. + +It should have a name and a tag. The name is `//`. +The tag is the version and build string of the packages, using a `-` as a separator. + +For example, a package like `xtensor-0.10.4-h431234.conda` would map to a OCI registry `conda-forge/linux-64/xtensor:0.10.4-h431234`. + +### Layers + +A conda package, in an OCI registry, should ship up to 3 layers: + +- The package itself, as a tarball. (mandatory) +- The package `info` folder as a gzipped "tar.gz" file. +- The package `info/index.json` file as a plain JSON file. + +The mediaType for the different layers is as follows: + +- for a .tar.bz2 package, the mediaType is `application/vnd.conda.package.v1` +- for a .conda package, the mediaType is `application/vnd.conda.package.v2` +- for the `info` folder as gzip the mediaType is `application/vnd.conda.info.v1.tar+gzip` +- for the `index.json` file the mediaType is `application/vnd.conda.info.index.v1+json` + +Using the `mediaType` field in the manifest, we can find the layer + SHA256 hash to pull the corresponding blob. +Each `mediaType` should only be present in one layer. + +## Repodata on OCI registries + +The `repodata.json` file is a JSON file that contains metadata about the packages in a channel. +It is used by conda to find packages in a channel. + +On an OCI registry it should be stored under `//repodata.json`. +The repodata file should have one entry that has the `latest` tag. This entry should point to the latest version of the repodata. +All versions of the repodata should also be tagged with a timestamp of the following format: `YYYY.MM.DD.HH.MM`, e.g. `2024.04.12.07.06`. + +The mediaType for the raw `repodata.json` file is `application/vnd.conda.repodata.v1+json`. However, for large repositories it's advised to store the `zstd` encoded repodata file with the mediaType `application/vnd.conda.repodata.v1+json+zst`. + +Other encoding are also accepted: + +- `application/vnd.conda.repodata.v1+json+gzip` +- `application/vnd.conda.repodata.v1+json+bz2` + +For `jlap`, the following mediaType is used: + +- `application/vnd.conda.jlap.v1` + +The `jlap` file should also be stored under the `//repodata.json` path as an additional layer. \ No newline at end of file From 5e3daede097a40164ec3e2fa75131cfd0228022c Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 12 Apr 2024 17:50:55 +0200 Subject: [PATCH 02/12] Update cep-oci.md Co-authored-by: Matthew R. Becker --- cep-oci.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-oci.md b/cep-oci.md index bf8be312..eed8ca00 100644 --- a/cep-oci.md +++ b/cep-oci.md @@ -48,7 +48,7 @@ On an OCI registry it should be stored under `//repodata.json`. The repodata file should have one entry that has the `latest` tag. This entry should point to the latest version of the repodata. All versions of the repodata should also be tagged with a timestamp of the following format: `YYYY.MM.DD.HH.MM`, e.g. `2024.04.12.07.06`. -The mediaType for the raw `repodata.json` file is `application/vnd.conda.repodata.v1+json`. However, for large repositories it's advised to store the `zstd` encoded repodata file with the mediaType `application/vnd.conda.repodata.v1+json+zst`. +The mediaType for the raw `repodata.json` file is `application/vnd.conda.repodata.v1+json`. However, for large repositories it's advised to store the `zstd` encoded repodata file with the mediaType `application/vnd.conda.repodata.v1+json+zst` as an additional layer in `//repodata.json`. Other encoding are also accepted: From ea109b7bde1a33982f9bdc1007fafc64adfd87b2 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 12 Apr 2024 17:51:58 +0200 Subject: [PATCH 03/12] Update cep-oci.md Co-authored-by: jaimergp --- cep-oci.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-oci.md b/cep-oci.md index eed8ca00..52c3adac 100644 --- a/cep-oci.md +++ b/cep-oci.md @@ -50,7 +50,7 @@ All versions of the repodata should also be tagged with a timestamp of the foll The mediaType for the raw `repodata.json` file is `application/vnd.conda.repodata.v1+json`. However, for large repositories it's advised to store the `zstd` encoded repodata file with the mediaType `application/vnd.conda.repodata.v1+json+zst` as an additional layer in `//repodata.json`. -Other encoding are also accepted: +Other encodings are also accepted: - `application/vnd.conda.repodata.v1+json+gzip` - `application/vnd.conda.repodata.v1+json+bz2` From 0416a33244fc84257afb9b058cb657a87ea2f338 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Thu, 25 Apr 2024 16:06:03 +0200 Subject: [PATCH 04/12] Update cep-oci.md Co-authored-by: jaimergp --- cep-oci.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-oci.md b/cep-oci.md index 52c3adac..5af2cd18 100644 --- a/cep-oci.md +++ b/cep-oci.md @@ -46,7 +46,7 @@ It is used by conda to find packages in a channel. On an OCI registry it should be stored under `//repodata.json`. The repodata file should have one entry that has the `latest` tag. This entry should point to the latest version of the repodata. -All versions of the repodata should also be tagged with a timestamp of the following format: `YYYY.MM.DD.HH.MM`, e.g. `2024.04.12.07.06`. +All versions of the repodata should also be tagged with a UTC timestamp of the following format: `YYYY.MM.DD.HH.MM.SS`, e.g. `2024.04.12.07.06.32`. The mediaType for the raw `repodata.json` file is `application/vnd.conda.repodata.v1+json`. However, for large repositories it's advised to store the `zstd` encoded repodata file with the mediaType `application/vnd.conda.repodata.v1+json+zst` as an additional layer in `//repodata.json`. From e8f19c1c4f2fd326257e1b9bfd7cbea9bba5a6fd Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Thu, 25 Apr 2024 16:38:03 +0200 Subject: [PATCH 05/12] add some details for the oci <-> conda mapping --- cep-oci.md | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/cep-oci.md b/cep-oci.md index 5af2cd18..2db97239 100644 --- a/cep-oci.md +++ b/cep-oci.md @@ -12,6 +12,8 @@ Layers can have arbitrary names and mediaTypes. An OCI manifest is referenced by a name and a tag. +For further details, please refer to the official [OCI Distribution spec](https://github.com/opencontainers/distribution-spec/blob/v1.0/spec.md#definitions). + ### Conda package artifacts on an OCI registry The manifest for a conda package on an OCI registry should look like follows. @@ -48,7 +50,7 @@ On an OCI registry it should be stored under `//repodata.json`. The repodata file should have one entry that has the `latest` tag. This entry should point to the latest version of the repodata. All versions of the repodata should also be tagged with a UTC timestamp of the following format: `YYYY.MM.DD.HH.MM.SS`, e.g. `2024.04.12.07.06.32`. -The mediaType for the raw `repodata.json` file is `application/vnd.conda.repodata.v1+json`. However, for large repositories it's advised to store the `zstd` encoded repodata file with the mediaType `application/vnd.conda.repodata.v1+json+zst` as an additional layer in `//repodata.json`. +The mediaType for the raw `repodata.json` file is `application/vnd.conda.repodata.v1+json`. However, for large repositories it's advised to store the `zstd` encoded repodata file with the mediaType `application/vnd.conda.repodata.v1+json+zstd` as an additional layer in `//repodata.json`. ([ref](https://github.com/opencontainers/image-spec/blob/main/layer.md#gzip-media-types)) Other encodings are also accepted: @@ -59,4 +61,26 @@ For `jlap`, the following mediaType is used: - `application/vnd.conda.jlap.v1` -The `jlap` file should also be stored under the `//repodata.json` path as an additional layer. \ No newline at end of file +The `jlap` file should also be stored under the `//repodata.json` path as an additional layer. + +### Mapping a conda-package to the OCI registry + +A given conda-package is identified by a URL like `/--.` where `` is the platform and architecture, `` is the name of the package, `` is the version of the package, `` is the build string of the package, and `` is the extension of the package file. + +To store this package on an OCI registry, we need to map it to a name and tag. The name is `//`. The tag is `-`. There are some special rules for OCI registry names and tags for which we need some mapping. The regex for valid names is as follows: + +`[a-z0-9]+((\.|_|__|-+)[a-z0-9]+)*(\/[a-z0-9]+((\.|_|__|-+)[a-z0-9]+)*)*` ([ref](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pulling-manifests)) + +The regex expresses that names can only start with an alphanumeric letter. + +In `conda`, names can start with an underscore and it is used by conda-forge (e.g. `_libgcc_mutex`). For this reason, we replace a leading underscore with the string `zzz`. + +The tag is the version and build string of the packages, using a `-` as a separator. However, a OCI tag can only contain the following regex: + +`[a-zA-Z0-9_][a-zA-Z0-9._-]{0,127}` + +Some characters that are used in the conda-forge repository as part of the build string are not allowed in the OCI registry. For this reason, we use the following mapping: + +- `+` is replaced by `__p__` +- `!` is replaced by `__e__` +- `=` is replaced by `__eq__` \ No newline at end of file From 9bb27ea2c52b670eb781c7a4f834c7aecd86b43d Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Wed, 1 May 2024 08:38:31 +0200 Subject: [PATCH 06/12] add some more info --- cep-oci.md | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/cep-oci.md b/cep-oci.md index 2db97239..e78e22cd 100644 --- a/cep-oci.md +++ b/cep-oci.md @@ -27,7 +27,8 @@ For example, a package like `xtensor-0.10.4-h431234.conda` would map to a OCI re A conda package, in an OCI registry, should ship up to 3 layers: -- The package itself, as a tarball. (mandatory) +- The package data itself, as a tarball. (mandatory) + - This can be either a `.tar.bz2` (v1) or a `.conda` (v2) file, or both as separate layers. - The package `info` folder as a gzipped "tar.gz" file. - The package `info/index.json` file as a plain JSON file. @@ -48,7 +49,7 @@ It is used by conda to find packages in a channel. On an OCI registry it should be stored under `//repodata.json`. The repodata file should have one entry that has the `latest` tag. This entry should point to the latest version of the repodata. -All versions of the repodata should also be tagged with a UTC timestamp of the following format: `YYYY.MM.DD.HH.MM.SS`, e.g. `2024.04.12.07.06.32`. +All versions of the repodata should also be tagged with a UTC timestamp of the following format: `YYYY.MM.DD.HH.MM.SS`, e.g. `2024.04.12.07.06.32`. The mediaType for the raw `repodata.json` file is `application/vnd.conda.repodata.v1+json`. However, for large repositories it's advised to store the `zstd` encoded repodata file with the mediaType `application/vnd.conda.repodata.v1+json+zstd` as an additional layer in `//repodata.json`. ([ref](https://github.com/opencontainers/image-spec/blob/main/layer.md#gzip-media-types)) @@ -67,15 +68,24 @@ The `jlap` file should also be stored under the `//repodata.jso A given conda-package is identified by a URL like `/--.` where `` is the platform and architecture, `` is the name of the package, `` is the version of the package, `` is the build string of the package, and `` is the extension of the package file. +#### Mapping the package name + +> ![NOTE] +> **Package names in the conda world** +> The following regex is given by `conda/schemas` for a valid package name: `^[a-z0-9_](?!_)[._-]?([a-z0-9]+(\.|-|_|$))*$` +> That means, a package can start with an alphanumeric character or a _single_ underscore (not multiple), and can contain dots, dashes, and underscores. It also has to end with a alphanumeric character (cannot end with a dot, dash, or underscore). + To store this package on an OCI registry, we need to map it to a name and tag. The name is `//`. The tag is `-`. There are some special rules for OCI registry names and tags for which we need some mapping. The regex for valid names is as follows: `[a-z0-9]+((\.|_|__|-+)[a-z0-9]+)*(\/[a-z0-9]+((\.|_|__|-+)[a-z0-9]+)*)*` ([ref](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pulling-manifests)) The regex expresses that names can only start with an alphanumeric letter. -In `conda`, names can start with an underscore and it is used by conda-forge (e.g. `_libgcc_mutex`). For this reason, we replace a leading underscore with the string `zzz`. +In `conda`, names can start with an underscore and it is used by conda-forge (e.g. `_libgcc_mutex`). For this reason, we prepend packages with a leading underscore with the string `zzz`. The name would thus be changed to `zzz_libgcc_mutex`. + +#### Mapping the tag -The tag is the version and build string of the packages, using a `-` as a separator. However, a OCI tag can only contain the following regex: +The tag is the version and build string of the packages, using a `-` as a separator. However, a OCI tag has to conform to the following regex: `[a-zA-Z0-9_][a-zA-Z0-9._-]{0,127}` @@ -83,4 +93,4 @@ Some characters that are used in the conda-forge repository as part of the build - `+` is replaced by `__p__` - `!` is replaced by `__e__` -- `=` is replaced by `__eq__` \ No newline at end of file +- `=` is replaced by `__eq__` From 6ee85fab29413118748bfaa15aa906148a88edc3 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Wed, 1 May 2024 08:41:15 +0200 Subject: [PATCH 07/12] Update note formatting and add link --- cep-oci.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/cep-oci.md b/cep-oci.md index e78e22cd..383feb3a 100644 --- a/cep-oci.md +++ b/cep-oci.md @@ -70,9 +70,11 @@ A given conda-package is identified by a URL like `/- ![NOTE] +> [!NOTE] > **Package names in the conda world** -> The following regex is given by `conda/schemas` for a valid package name: `^[a-z0-9_](?!_)[._-]?([a-z0-9]+(\.|-|_|$))*$` +> +> The following regex is given by [`conda/schemas`](https://github.com/conda/schemas/blob/473708ac97283708d6664cbd89b8049ad1623489/common-1.schema.json#L58-L82) for a valid package name: `^[a-z0-9_](?!_)[._-]?([a-z0-9]+(\.|-|_|$))*$` +> > That means, a package can start with an alphanumeric character or a _single_ underscore (not multiple), and can contain dots, dashes, and underscores. It also has to end with a alphanumeric character (cannot end with a dot, dash, or underscore). To store this package on an OCI registry, we need to map it to a name and tag. The name is `//`. The tag is `-`. There are some special rules for OCI registry names and tags for which we need some mapping. The regex for valid names is as follows: From d339ca0e17a94c86a489180da0768ac31023c909 Mon Sep 17 00:00:00 2001 From: Hind Montassif Date: Mon, 3 Jun 2024 17:11:58 +0200 Subject: [PATCH 08/12] Add more info (mediaTypes, implementation...) --- cep-oci.md | 91 +++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 73 insertions(+), 18 deletions(-) diff --git a/cep-oci.md b/cep-oci.md index 383feb3a..e04a7141 100644 --- a/cep-oci.md +++ b/cep-oci.md @@ -14,30 +14,33 @@ An OCI manifest is referenced by a name and a tag. For further details, please refer to the official [OCI Distribution spec](https://github.com/opencontainers/distribution-spec/blob/v1.0/spec.md#definitions). -### Conda package artifacts on an OCI registry - -The manifest for a conda package on an OCI registry should look like follows. +### Layers -It should have a name and a tag. The name is `//`. -The tag is the version and build string of the packages, using a `-` as a separator. +Each layer must be a [descriptor](https://github.com/opencontainers/image-spec/blob/main/descriptor.md#properties) containing at least the 3 required fields: -For example, a package like `xtensor-0.10.4-h431234.conda` would map to a OCI registry `conda-forge/linux-64/xtensor:0.10.4-h431234`. +- The `mediaType` of the referenced content. +- The `digest` of the targeted content. +- The `size` of the raw content (in bytes). -### Layers +### MediaTypes -A conda package, in an OCI registry, should ship up to 3 layers: +Global and already defined mediaTypes are described [here](https://github.com/opencontainers/image-spec/blob/main/media-types.md#oci-image-media-types). -- The package data itself, as a tarball. (mandatory) - - This can be either a `.tar.bz2` (v1) or a `.conda` (v2) file, or both as separate layers. -- The package `info` folder as a gzipped "tar.gz" file. -- The package `info/index.json` file as a plain JSON file. +Custom mediaTypes defined for the conda channels use case are as follows: -The mediaType for the different layers is as follows: +| Blob type | Content type | mediaType | +|------------------|---------------------------|---------------------------------------------| +| conda package | .tar.bz2 package | application/vnd.conda.package.v1 | +| conda package | .conda package | application/vnd.conda.package.v2 | +| package info | `info` folder as gzip | application/vnd.conda.info.v1.tar+gzip | +| package info | `index.json` file | application/vnd.conda.info.index.v1+json | +| repodata | `repodata.json` file | application/vnd.conda.repodata.v1+json | +| repodata | `repodata.json.zst` file | application/vnd.conda.repodata.v1+json+zst | +| repodata | `repodata.json.gz` file | application/vnd.conda.repodata.v1+json+gzip | +| repodata | `repodata.json.bz2` file | application/vnd.conda.repodata.v1+json+bz2 | +| repodata | `repodata.json.jlap` file | application/vnd.conda.jlap.v1 | -- for a .tar.bz2 package, the mediaType is `application/vnd.conda.package.v1` -- for a .conda package, the mediaType is `application/vnd.conda.package.v2` -- for the `info` folder as gzip the mediaType is `application/vnd.conda.info.v1.tar+gzip` -- for the `index.json` file the mediaType is `application/vnd.conda.info.index.v1+json` +If needed, more mediaTypes could be specified (i.e `application/vnd.conda.info.v1.tar+zst`). Using the `mediaType` field in the manifest, we can find the layer + SHA256 hash to pull the corresponding blob. Each `mediaType` should only be present in one layer. @@ -51,7 +54,7 @@ On an OCI registry it should be stored under `//repodata.json`. The repodata file should have one entry that has the `latest` tag. This entry should point to the latest version of the repodata. All versions of the repodata should also be tagged with a UTC timestamp of the following format: `YYYY.MM.DD.HH.MM.SS`, e.g. `2024.04.12.07.06.32`. -The mediaType for the raw `repodata.json` file is `application/vnd.conda.repodata.v1+json`. However, for large repositories it's advised to store the `zstd` encoded repodata file with the mediaType `application/vnd.conda.repodata.v1+json+zstd` as an additional layer in `//repodata.json`. ([ref](https://github.com/opencontainers/image-spec/blob/main/layer.md#gzip-media-types)) +The mediaType for the raw `repodata.json` file is `application/vnd.conda.repodata.v1+json`. However, for large repositories it's advised to store the `zstd` encoded repodata file with the mediaType `application/vnd.conda.repodata.v1+json+zst` as an additional layer in `//repodata.json`. ([ref](https://github.com/opencontainers/image-spec/blob/main/layer.md#gzip-media-types)) Other encodings are also accepted: @@ -64,6 +67,22 @@ For `jlap`, the following mediaType is used: The `jlap` file should also be stored under the `//repodata.json` path as an additional layer. +### Conda package artifacts on an OCI registry + +The manifest for a conda package on an OCI registry should look like follows. + +It should have a name and a tag. The name is `//`. +The tag is the version and build string of the packages, using a `-` as a separator. + +For example, a package like `xtensor-0.10.4-h431234.conda` would map to a OCI registry `conda-forge/linux-64/xtensor:0.10.4-h431234`. + +A conda package, in an OCI registry, should ship up to 3 layers: + +- The package data itself, as a tarball. (mandatory) + - This can be either a `.tar.bz2` (v1) or a `.conda` (v2) file, or both as separate layers. +- The package `info` folder as a gzipped "tar.gz" file. +- The package `info/index.json` file as a plain JSON file. + ### Mapping a conda-package to the OCI registry A given conda-package is identified by a URL like `/--.` where `` is the platform and architecture, `` is the name of the package, `` is the version of the package, `` is the build string of the package, and `` is the extension of the package file. @@ -96,3 +115,39 @@ Some characters that are used in the conda-forge repository as part of the build - `+` is replaced by `__p__` - `!` is replaced by `__e__` - `=` is replaced by `__eq__` + +#### Authentication + +Pulling a public image from a Container registry can be done anonymously ([ref](https://docs.github.com/en/packages/learn-github-packages/about-permissions-for-github-packages#visibility-and-access-permissions-for-packages)). + +A token can be requested with `pull` scope, using the following URL: +`https://ghcr.io/token?scope=repository:///:pull` + +Note that in the case of pulling repodata, the name `repodata.json` is always used in the URL regardless of the encoding. + +#### Implementation (conda / mamba / rattler) + +##### mamba + +In order to fetch packages from an OCI registry, we need to set a mirror (can be more than one) for the channel to be used (e.g `conda-forge`). +This can be done in the rc file as follows: + +``` +mirrored_channels: + conda-forge: ["oci://ghcr.io/channel-mirrors/conda-forge"] +``` + +When a user requests installing a package (with the configuration set above, and using `conda-forge` channel), a set of requests to fetch `repodata.json` are first performed as follows: + +- A token is requested to anonymously pull `repodata.json` using the following URL:\ +`https://ghcr.io/token?scope=repository:channel-mirrors/conda-forge//repodata.json:pull` +- The manifest is then pulled using `https://ghcr.io/v2/channel-mirrors/conda-forge//repodata.json/manifests/`.\ +`` is always set to `latest` in `mamba`.\ +This is also where the repodata file encoding is handled (checking `mediaType` field in the layers).\ +In `mamba`, `zstd` encoding has priority if present, otherwise, raw `repodata.json` is picked, and the corresponding SHA256 hash is set for the next step. +- Repodata blob is then downloaded using:\ +`https://ghcr.io/v2/channel-mirrors/conda-forge//repodata.json/blobs/sha256:` + +Then, to fetch the package itself, and using the same token, the corresponding blob is downloaded using: +`https://ghcr.io/v2/channel-mirrors/conda-forge///blobs/sha256:` +where is the SHA256 hash of the requested package, retrieved from `repodata.json`. From 8cdc5f29e0a8743c7df0c7755e5bcc0a92099122 Mon Sep 17 00:00:00 2001 From: Hind Montassif Date: Wed, 7 Aug 2024 11:06:57 +0200 Subject: [PATCH 09/12] Comply with ceps template --- cep-oci.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/cep-oci.md b/cep-oci.md index e04a7141..47256b45 100644 --- a/cep-oci.md +++ b/cep-oci.md @@ -1,4 +1,14 @@ -# OCI registries as conda channels + + + + + + + + +
Title OCI registries as conda channels
Status Proposed
Author(s) Wolf Vollprecht <wolf@prefix.dev>
Created April 12, 2024
Updated August 7, 2024
Discussion https://github.com/conda/ceps/pull/70
Implementation
+ +# Abstract We want to use OCI registries as a storage for conda packages. This CEP specifies how we lay out conda packages on an OCI registry. From 5c88967f59cc92b7d1918d37ac59f222c1e1b4ea Mon Sep 17 00:00:00 2001 From: Hind Montassif Date: Wed, 7 Aug 2024 15:01:08 +0200 Subject: [PATCH 10/12] Remove mentioning mirrors --- cep-oci.md | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/cep-oci.md b/cep-oci.md index 47256b45..6e4266a5 100644 --- a/cep-oci.md +++ b/cep-oci.md @@ -139,15 +139,9 @@ Note that in the case of pulling repodata, the name `repodata.json` is always us ##### mamba -In order to fetch packages from an OCI registry, we need to set a mirror (can be more than one) for the channel to be used (e.g `conda-forge`). -This can be done in the rc file as follows: +In order to fetch packages from an OCI registry, the corresponding URL should be used as a channel (i.e `oci://ghcr.io/channel-mirrors/conda-forge`). -``` -mirrored_channels: - conda-forge: ["oci://ghcr.io/channel-mirrors/conda-forge"] -``` - -When a user requests installing a package (with the configuration set above, and using `conda-forge` channel), a set of requests to fetch `repodata.json` are first performed as follows: +When a user requests installing a package, a set of requests to fetch `repodata.json` are first performed as follows: - A token is requested to anonymously pull `repodata.json` using the following URL:\ `https://ghcr.io/token?scope=repository:channel-mirrors/conda-forge//repodata.json:pull` From 193259a7b68d9c2b4834e829501191432fac8a21 Mon Sep 17 00:00:00 2001 From: Hind Montassif Date: Tue, 20 Aug 2024 15:00:46 +0200 Subject: [PATCH 11/12] Review: -Add me as author -Remove jlap as not relevant anymore --- cep-oci.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/cep-oci.md b/cep-oci.md index 6e4266a5..c5b408ac 100644 --- a/cep-oci.md +++ b/cep-oci.md @@ -1,7 +1,7 @@ - + @@ -48,7 +48,6 @@ Custom mediaTypes defined for the conda channels use case are as follows: | repodata | `repodata.json.zst` file | application/vnd.conda.repodata.v1+json+zst | | repodata | `repodata.json.gz` file | application/vnd.conda.repodata.v1+json+gzip | | repodata | `repodata.json.bz2` file | application/vnd.conda.repodata.v1+json+bz2 | -| repodata | `repodata.json.jlap` file | application/vnd.conda.jlap.v1 | If needed, more mediaTypes could be specified (i.e `application/vnd.conda.info.v1.tar+zst`). @@ -71,12 +70,6 @@ Other encodings are also accepted: - `application/vnd.conda.repodata.v1+json+gzip` - `application/vnd.conda.repodata.v1+json+bz2` -For `jlap`, the following mediaType is used: - -- `application/vnd.conda.jlap.v1` - -The `jlap` file should also be stored under the `//repodata.json` path as an additional layer. - ### Conda package artifacts on an OCI registry The manifest for a conda package on an OCI registry should look like follows. From 6e6cd7d8f8c6dd337ce3d2f94f83103061a72387 Mon Sep 17 00:00:00 2001 From: Hind Montassif Date: Tue, 20 Aug 2024 15:49:11 +0200 Subject: [PATCH 12/12] Add important note regarding the mapping of package names and tags --- cep-oci.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/cep-oci.md b/cep-oci.md index c5b408ac..857ab494 100644 --- a/cep-oci.md +++ b/cep-oci.md @@ -119,6 +119,12 @@ Some characters that are used in the conda-forge repository as part of the build - `!` is replaced by `__e__` - `=` is replaced by `__eq__` +> [!IMPORTANT] +> +> The mapping of the package names and tags, as previously outlined, is solely intended for internal storage within an OCI registry. +> +> From the user's perspective, or in contexts outside of the OCI registry—beyond the middleware that interacts with it—, the package names and tags will remain unchanged, preserving the original strings. + #### Authentication Pulling a public image from a Container registry can be done anonymously ([ref](https://docs.github.com/en/packages/learn-github-packages/about-permissions-for-github-packages#visibility-and-access-permissions-for-packages)).
Title OCI registries as conda channels
Status Proposed
Author(s) Wolf Vollprecht <wolf@prefix.dev>
Author(s) Wolf Vollprecht <wolf@prefix.dev>
Hind Montassif <hind.montassif@quantstack.net>
Created April 12, 2024
Updated August 7, 2024
Discussion https://github.com/conda/ceps/pull/70