Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add more docs for the multi-output cache #1235

Merged
merged 3 commits into from
Dec 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 79 additions & 4 deletions docs/multiple_output_cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,14 @@ recipe:
version: '0.1.0'

cache:
source:
- url: https://example.com/library.tar.gz
sha256: 1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef

requirements:
build:
- ${{ compiler('c') }}

build:
script:
- mkdir -p $PREFIX/lib
Expand Down Expand Up @@ -56,15 +61,85 @@ You can use a list of globs to select only the files that you want.

For something more complicated you can also use `include` and `exclude` fields in the `files` selector. Please refer to the [the build options documentation](build_options.md#include-only-certain-files-in-the-package).

## Run exports from the cache
### Run exports from the cache

Since the cache output also has build- and host requirements we need to additionally take care of eventual "run-exports" from the cache output.
Run exports from the cache-dependencies are handled very similar to the run exports from a given output. We append any run exports to the outputs.

If the cache has an "ignore run exports" section, than we apply those filters at the cache level. If the output ignores any run exports, then we also ignore the run-exports if they would come from the cache.

## Caching in the $SRC_DIR
### Source code in the cache

The cache output has its own `source` section. For every output, the (dirty) source is restored from the cache directory. Outputs can layer additional files on top of the cache source.
However, if you already ran `cmake` in the cache output, you can continue from where the build left off. This is useful when you want to e.g. build additional components (such as Python bindings) on top of the already-built library.


If you used `conda-build` a lot, you might have noticed that a top-level build is also caching the changes in the `$SRC_DIR`. This is not the case for `rattler-build` yet.
## C++ Example that builds Python bindings on top of a library

You can find an example (with source code) here: [Link](https://github.com/wolfv/rattler-build-cache-test/).

```yaml title="variants.yaml"
python:
- "3.12.*"
- "3.11.*"
```

And the corresponding recipe:

```yaml title="recipe.yaml"
recipe:
name: calculator
version: 1.0.0

cache:
source:
path: ../

You could try to work around by e.g. copying files into the `$PREFIX` and restoring them in each output.
requirements:
build:
- ${{ compiler('cxx') }}
- cmake
- ninja
build:
script:
# make sure that `alternative_name.md` is not present
- test ! -f ./alternative_name.md
- mkdir build
- cd build
- cmake $SRC_DIR -GNinja ${CMAKE_ARGS}
- ninja install

outputs:
# this first output will include all files installed during the cache build
- package:
name: libcalculator

requirements:
run_exports:
- ${{ pin_subpackage('libcalculator') }}
# This output will build the Python bindings using CMake and then create new
# packages with the Python bindings
- package:
name: py-calculator
source:
- path: ../README.md
file_name: alternative_name.md

requirements:
build:
- ${{ compiler('cxx') }}
- cmake
- ninja
host:
- pybind11
- python
- libcalculator

build:
script:
# assert that the README.md file is present
- test -f ./alternative_name.md
- cd build
- cmake $SRC_DIR -GNinja ${CMAKE_ARGS} -DBUILD_PYTHON_BINDINGS=ON
- ninja install
```
28 changes: 26 additions & 2 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,12 @@ Build a package from a recipe
Path to an auth-file to read authentication information from


- `--channel-priority <CHANNEL_PRIORITY>`

Channel priority to use when solving

- Default value: `strict`

- `--tui`

Launch the terminal user interface
Expand Down Expand Up @@ -334,6 +340,12 @@ These test files are written at "package creation time" and are part of the pack
Path to an auth-file to read authentication information from


- `--channel-priority <CHANNEL_PRIORITY>`

Channel priority to use when solving

- Default value: `strict`

###### **Modifying result**

- `--output-dir <OUTPUT_DIR>`
Expand Down Expand Up @@ -399,6 +411,12 @@ Rebuild a package from a package file instead of a recipe
Path to an auth-file to read authentication information from


- `--channel-priority <CHANNEL_PRIORITY>`

Channel priority to use when solving

- Default value: `strict`

###### **Modifying result**

- `--test <TEST>`
Expand Down Expand Up @@ -431,7 +449,7 @@ Upload a package

##### **Subcommands:**

* `quetz` — Upload to aQuetz server. Authentication is used from the keychain / auth-file
* `quetz` — Upload to a Quetz server. Authentication is used from the keychain / auth-file
* `artifactory` — Options for uploading to a Artifactory channel. Authentication is used from the keychain / auth-file
* `prefix` — Options for uploading to a prefix.dev server. Authentication is used from the keychain / auth-file
* `anaconda` — Options for uploading to a Anaconda.org server
Expand Down Expand Up @@ -474,6 +492,12 @@ Upload a package
Path to an auth-file to read authentication information from


- `--channel-priority <CHANNEL_PRIORITY>`

Channel priority to use when solving

- Default value: `strict`

###### **Modifying result**

- `--output-dir <OUTPUT_DIR>`
Expand All @@ -487,7 +511,7 @@ Upload a package

#### `quetz`

Upload to aQuetz server. Authentication is used from the keychain / auth-file
Upload to a Quetz server. Authentication is used from the keychain / auth-file

**Usage:** `rattler-build upload quetz [OPTIONS] --url <URL> --channel <CHANNEL>`

Expand Down
Loading