From 9f45b381d7a8d51a1b7ce1351c0834777315b66a Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Sat, 13 Apr 2024 11:15:40 +0200 Subject: [PATCH 01/11] add recipe extension cep --- cep-recipe-extensions.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 cep-recipe-extensions.md diff --git a/cep-recipe-extensions.md b/cep-recipe-extensions.md new file mode 100644 index 00000000..ff695a1f --- /dev/null +++ b/cep-recipe-extensions.md @@ -0,0 +1,33 @@ +# Extensions to the recipe spec + +This document describes extensions to the recipe specification as merged in cep-14. + +## The `build.post_process` section + +We became aware that the `conda_build_config.yaml` has ways to specify post-processing steps. These are regex replacements that are performed on any new files. + +Instead of adding these instructions to the `conda_build_config.yaml`, we decided to add a new section to the recipe spec: `build.post_process`. This section is a list of dictionaries, each with the following keys: + +- `files`: globs to select files to process +- `regex`: the regex to apply +- `replacement`: the replacement string + +The regex specification follows Rust `regex` syntax. Most notably, the replacement string can refer to capture groups with `$1`, `$2`, etc. +This also means that replacement strings need to escape `$` with `$$`. + +Internally, we use `replace_all` from the `regex` crate. This means that the regex is applied to the entire file, not line by line. + +### Example + +```yaml +build: + post_process: + - files: + - "*.txt" + regex: "foo" + replacement: "bar" + - files: + - "*.cmake" + regex: "/sysroot/" + replacement: "$${PREFIX}/" +``` \ No newline at end of file From 138b910e0f92d6d3f95d5c8edb60a2e7502041ec Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Wed, 17 Apr 2024 13:40:07 +0200 Subject: [PATCH 02/11] add section for new glob include / exclude --- cep-recipe-extensions.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/cep-recipe-extensions.md b/cep-recipe-extensions.md index ff695a1f..74dc7fbf 100644 --- a/cep-recipe-extensions.md +++ b/cep-recipe-extensions.md @@ -30,4 +30,19 @@ build: - "*.cmake" regex: "/sysroot/" replacement: "$${PREFIX}/" -``` \ No newline at end of file +``` + +## Globs, positive and negative + +Following some community discussion we would like to extend the recipe format in such a way that anywhere where a list of globs is accepted, we alternatively accept a dictionary with `include` and `exclude` keys. The values of these keys are lists of globs that are included or excluded respectively. + +For example: + +```yaml +files: + include: + - "*.txt" + exclude: + - "foo.txt" +``` + From e46d8a9ce57c932073a5970d1dbb8fecc41d8645 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Wed, 17 Apr 2024 13:41:09 +0200 Subject: [PATCH 03/11] add section about extending `GlobVec` --- cep-recipe-extensions.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/cep-recipe-extensions.md b/cep-recipe-extensions.md index 74dc7fbf..9b52ad5a 100644 --- a/cep-recipe-extensions.md +++ b/cep-recipe-extensions.md @@ -46,3 +46,9 @@ files: - "foo.txt" ``` +The evaluation would go as follows: + +- first record all matches for the `include` globs +- then remove all matches for the `exclude` globs + +In conda-build, this is discussed here: [Link to PR 5216](https://github.com/conda/conda-build/pull/5216) \ No newline at end of file From 3809c587b23472a2702e49dd26e86d062755da6e Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 10 May 2024 09:27:12 +0200 Subject: [PATCH 04/11] Update cep-recipe-extensions.md Co-authored-by: jaimergp --- cep-recipe-extensions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-recipe-extensions.md b/cep-recipe-extensions.md index 9b52ad5a..c99b4acd 100644 --- a/cep-recipe-extensions.md +++ b/cep-recipe-extensions.md @@ -4,7 +4,7 @@ This document describes extensions to the recipe specification as merged in cep- ## The `build.post_process` section -We became aware that the `conda_build_config.yaml` has ways to specify post-processing steps. These are regex replacements that are performed on any new files. +The `conda_build_config.yaml` configuration file has ways to specify post-processing steps. These are regex replacements that are performed on any new files. Instead of adding these instructions to the `conda_build_config.yaml`, we decided to add a new section to the recipe spec: `build.post_process`. This section is a list of dictionaries, each with the following keys: From ea7aa7d1fc15324db025286bae91b1478085bb8e Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 10 May 2024 22:22:57 +0200 Subject: [PATCH 05/11] add `compression_level` --- cep-recipe-extensions.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/cep-recipe-extensions.md b/cep-recipe-extensions.md index c99b4acd..8594092a 100644 --- a/cep-recipe-extensions.md +++ b/cep-recipe-extensions.md @@ -51,4 +51,20 @@ The evaluation would go as follows: - first record all matches for the `include` globs - then remove all matches for the `exclude` globs -In conda-build, this is discussed here: [Link to PR 5216](https://github.com/conda/conda-build/pull/5216) \ No newline at end of file +In conda-build, this is discussed here: [Link to PR 5216](https://github.com/conda/conda-build/pull/5216) + +## Compression + +There are use cases where it is interesting for a output to set the compression level. For this we would like to implement the following syntax: + +```yaml + +build: + compression_level: 1-10 +``` + +The compression level maps to the following levels in the `.zstd` and `bz2` formats: + +- `zstd`: 1-19 (1 = 1, 10 = 19) +- `bz2`: 1-9 (1 = 1, 10 = 9) + From a1b32a5eacb2dd0217db52c1e9825be48eb574e1 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 10 May 2024 22:25:14 +0200 Subject: [PATCH 06/11] add compression level to `build` section --- cep-recipe-extensions.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cep-recipe-extensions.md b/cep-recipe-extensions.md index 8594092a..72f802db 100644 --- a/cep-recipe-extensions.md +++ b/cep-recipe-extensions.md @@ -55,12 +55,11 @@ In conda-build, this is discussed here: [Link to PR 5216](https://github.com/con ## Compression -There are use cases where it is interesting for a output to set the compression level. For this we would like to implement the following syntax: +There are use cases where it is interesting for a output to set the compression level - for example when packaging pre-compressed data. We would like to extend the recipe format so that each output is able to set a "default" compression level with the following syntax: ```yaml - build: - compression_level: 1-10 + compression_level: 1-10 # (integer) ``` The compression level maps to the following levels in the `.zstd` and `bz2` formats: @@ -68,3 +67,4 @@ The compression level maps to the following levels in the `.zstd` and `bz2` form - `zstd`: 1-19 (1 = 1, 10 = 19) - `bz2`: 1-9 (1 = 1, 10 = 9) +Note that this compression level can still be overriden by the command line. It also does not define wether a `.conda` or `.tar.bz2` file is created. \ No newline at end of file From e2ba466e255f28e2c8bad86824642baa05108656 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 10 May 2024 22:39:25 +0200 Subject: [PATCH 07/11] formatting --- cep-recipe-extensions.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/cep-recipe-extensions.md b/cep-recipe-extensions.md index 72f802db..e33ebf66 100644 --- a/cep-recipe-extensions.md +++ b/cep-recipe-extensions.md @@ -67,4 +67,5 @@ The compression level maps to the following levels in the `.zstd` and `bz2` form - `zstd`: 1-19 (1 = 1, 10 = 19) - `bz2`: 1-9 (1 = 1, 10 = 9) -Note that this compression level can still be overriden by the command line. It also does not define wether a `.conda` or `.tar.bz2` file is created. \ No newline at end of file +Note that this compression level can still be overridden by the command line. +It also does not define wether a `.conda` or `.tar.bz2` file is created. From f501df200cd23db6bfd78f147763260c499bc64d Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 10 May 2024 22:39:39 +0200 Subject: [PATCH 08/11] header --- cep-recipe-extensions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-recipe-extensions.md b/cep-recipe-extensions.md index e33ebf66..b470b58b 100644 --- a/cep-recipe-extensions.md +++ b/cep-recipe-extensions.md @@ -53,7 +53,7 @@ The evaluation would go as follows: In conda-build, this is discussed here: [Link to PR 5216](https://github.com/conda/conda-build/pull/5216) -## Compression +## Compression level There are use cases where it is interesting for a output to set the compression level - for example when packaging pre-compressed data. We would like to extend the recipe format so that each output is able to set a "default" compression level with the following syntax: From 3619ae2b42eb680b8b9a5adef4cf8d49078e138f Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Wed, 29 May 2024 17:10:24 +0200 Subject: [PATCH 09/11] add system-requirements idea to the cep --- cep-recipe-extensions.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/cep-recipe-extensions.md b/cep-recipe-extensions.md index b470b58b..29680326 100644 --- a/cep-recipe-extensions.md +++ b/cep-recipe-extensions.md @@ -69,3 +69,24 @@ The compression level maps to the following levels in the `.zstd` and `bz2` form Note that this compression level can still be overridden by the command line. It also does not define wether a `.conda` or `.tar.bz2` file is created. + + +## System requirements + +This is a new section in the `build` section that defines the lower bound system requirements for a given package. + +In this section you can define the "virtual" packages that will be injected in the `host` sections of the specs: + +```yaml +build: + system_requirements: + # this one is used on `osx-*` platforms + - osx: 11.0 # (or should we call this / alias it to `macos`?) + # the following are used on `linux-*` platforms + - glibc: 2.17 + - linux: 3.10 + # the following is used on `win-*` platforms + - win: 10.1 +``` + +The system requirements are used to inject virtual packages in the `host` section of the recipe. They will also automatically add a dependency in the `run` section of the recipe, e.g. `__glibc >=2.17`, or `__osx >=11.0`. \ No newline at end of file From 575fd781817238d4f97573f58e1e5c8c9caedba4 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Tue, 4 Jun 2024 14:52:32 +0200 Subject: [PATCH 10/11] update recipe extensions --- cep-recipe-extensions.md | 110 ++++++++++++++++++++++++++++++--------- 1 file changed, 86 insertions(+), 24 deletions(-) diff --git a/cep-recipe-extensions.md b/cep-recipe-extensions.md index 29680326..2d69257d 100644 --- a/cep-recipe-extensions.md +++ b/cep-recipe-extensions.md @@ -1,21 +1,28 @@ # Extensions to the recipe spec -This document describes extensions to the recipe specification as merged in cep-14. +This document describes extensions to the recipe specification as merged in +cep-14. ## The `build.post_process` section -The `conda_build_config.yaml` configuration file has ways to specify post-processing steps. These are regex replacements that are performed on any new files. +The `conda_build_config.yaml` configuration file has ways to specify +post-processing steps. These are regex replacements that are performed on any +new files. -Instead of adding these instructions to the `conda_build_config.yaml`, we decided to add a new section to the recipe spec: `build.post_process`. This section is a list of dictionaries, each with the following keys: +Instead of adding these instructions to the `conda_build_config.yaml`, we +decided to add a new section to the recipe spec: `build.post_process`. This +section is a list of dictionaries, each with the following keys: - `files`: globs to select files to process - `regex`: the regex to apply - `replacement`: the replacement string -The regex specification follows Rust `regex` syntax. Most notably, the replacement string can refer to capture groups with `$1`, `$2`, etc. -This also means that replacement strings need to escape `$` with `$$`. +The regex specification follows Rust `regex` syntax. Most notably, the +replacement string can refer to capture groups with `$1`, `$2`, etc. This also +means that replacement strings need to escape `$` with `$$`. -Internally, we use `replace_all` from the `regex` crate. This means that the regex is applied to the entire file, not line by line. +Internally, we use `replace_all` from the `regex` crate. This means that the +regex is applied to the entire file, not line by line. ### Example @@ -32,56 +39,109 @@ build: replacement: "$${PREFIX}/" ``` -## Globs, positive and negative +## Files selector in the `build` section -Following some community discussion we would like to extend the recipe format in such a way that anywhere where a list of globs is accepted, we alternatively accept a dictionary with `include` and `exclude` keys. The values of these keys are lists of globs that are included or excluded respectively. +A `files` key is added to the `build` section to simplify the inclusion / +exclusion of files in the package. + +This key accepts a list of globs that are used to select files to include in the +package. + +```yaml +build: + files: + - "*.txt" + - "*.md" +``` + +This is especially useful when building multiple outputs, as the same build +script can be used for all outputs, but the files to include can be different. +The globs are evaluated against any new files in the prefix, after the build +script has been run. + +Files that are not selected by the globs are excluded from the package. + +## Update to the "globs" definition + +Anywhere the recipe format accepts a list of globs is extended to accept: + +- a single glob (`files: foo/*.txt`) +- a list of globs (`files: [foo/*.txt, bar/*.txt]`) +- a dictionary with `include` and `exclude` keys (`files: {include: [foo/*.txt], + exclude: [foo/bar.txt]}`), where `include` and `exclude` are lists of globs. + +The lists are "conditional" lists and can contain `if` and `else` statements. For example: ```yaml +# single glob +files: foo/*.txt + +# list of globs +files: + - foo/*.txt + - bar/*.txt + - if: win + then: + - win.txt + +# dictionary with include and exclude keys files: include: - - "*.txt" + - *.txt exclude: - - "foo.txt" + - foo.txt ``` -The evaluation would go as follows: +A single glob or a list of globs is equivalent to a dictionary with only an +`include` key. -- first record all matches for the `include` globs -- then remove all matches for the `exclude` globs +The evaluation of the globs is done in two steps: -In conda-build, this is discussed here: [Link to PR 5216](https://github.com/conda/conda-build/pull/5216) +- first record all matches for the `include` globs. If no `include` globs are + given but an `exclude` glob is provided, all files are included. +- Any file that matched an `include` glob is tested against the `exclude` globs. + If it matches an `exclude` glob, it is removed from the list of files to + include. If no exclude globs are given, no files are removed. + +In conda-build, this is discussed here: [Link to PR +5216](https://github.com/conda/conda-build/pull/5216) ## Compression level -There are use cases where it is interesting for a output to set the compression level - for example when packaging pre-compressed data. We would like to extend the recipe format so that each output is able to set a "default" compression level with the following syntax: +There are use cases where it is interesting for a output to set the compression +level - for example when packaging pre-compressed data. We would like to extend +the recipe format so that each output is able to set a "default" compression +level with the following syntax: ```yaml build: - compression_level: 1-10 # (integer) + compression_level: 1-10 # (integer) ``` -The compression level maps to the following levels in the `.zstd` and `bz2` formats: +The compression level maps to the following levels in the `.zstd` and `bz2` +formats: - `zstd`: 1-19 (1 = 1, 10 = 19) - `bz2`: 1-9 (1 = 1, 10 = 9) -Note that this compression level can still be overridden by the command line. -It also does not define wether a `.conda` or `.tar.bz2` file is created. - +Note that this compression level can still be overridden by the command line. It +also does not define wether a `.conda` or `.tar.bz2` file is created. ## System requirements -This is a new section in the `build` section that defines the lower bound system requirements for a given package. +This is a new section in the `build` section that defines the lower bound system +requirements for a given package. -In this section you can define the "virtual" packages that will be injected in the `host` sections of the specs: +In this section you can define the "virtual" packages that will be injected in +the `host` sections of the specs: ```yaml build: system_requirements: # this one is used on `osx-*` platforms - - osx: 11.0 # (or should we call this / alias it to `macos`?) + - osx: 11.0 # (or should we call this / alias it to `macos`?) # the following are used on `linux-*` platforms - glibc: 2.17 - linux: 3.10 @@ -89,4 +149,6 @@ build: - win: 10.1 ``` -The system requirements are used to inject virtual packages in the `host` section of the recipe. They will also automatically add a dependency in the `run` section of the recipe, e.g. `__glibc >=2.17`, or `__osx >=11.0`. \ No newline at end of file +The system requirements are used to inject virtual packages in the `host` +section of the recipe. They will also automatically add a dependency in the +`run` section of the recipe, e.g. `__glibc >=2.17`, or `__osx >=11.0`. From 7a9c87b9eb20ead9cbb6df583f9d559f34e97dcf Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Tue, 4 Jun 2024 14:53:54 +0200 Subject: [PATCH 11/11] format --- cep-recipe-extensions.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/cep-recipe-extensions.md b/cep-recipe-extensions.md index 2d69257d..1d1ee8ad 100644 --- a/cep-recipe-extensions.md +++ b/cep-recipe-extensions.md @@ -24,7 +24,7 @@ means that replacement strings need to escape `$` with `$$`. Internally, we use `replace_all` from the `regex` crate. This means that the regex is applied to the entire file, not line by line. -### Example +### Example for `post_process` ```yaml build: @@ -72,6 +72,19 @@ Anywhere the recipe format accepts a list of globs is extended to accept: The lists are "conditional" lists and can contain `if` and `else` statements. +A single glob or a list of globs is equivalent to a dictionary with only an +`include` key. + +The evaluation of the globs is done in two steps: + +- first record all matches for the `include` globs. If no `include` globs are + given but an `exclude` glob is provided, all files are included. +- Any file that matched an `include` glob is tested against the `exclude` globs. + If it matches an `exclude` glob, it is removed from the list of files to + include. If no exclude globs are given, no files are removed. + +### Example for `glob` definition + For example: ```yaml @@ -94,17 +107,6 @@ files: - foo.txt ``` -A single glob or a list of globs is equivalent to a dictionary with only an -`include` key. - -The evaluation of the globs is done in two steps: - -- first record all matches for the `include` globs. If no `include` globs are - given but an `exclude` glob is provided, all files are included. -- Any file that matched an `include` glob is tested against the `exclude` globs. - If it matches an `exclude` glob, it is removed from the list of files to - include. If no exclude globs are given, no files are removed. - In conda-build, this is discussed here: [Link to PR 5216](https://github.com/conda/conda-build/pull/5216)