From dab04ff1e60ff0e34e6e5224747206e3f5ebddbd Mon Sep 17 00:00:00 2001 From: jhartman Date: Fri, 17 Feb 2023 19:14:01 -0600 Subject: [PATCH 1/7] create file with draft pasted --- docs/walkthrough.md | 320 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 320 insertions(+) create mode 100644 docs/walkthrough.md diff --git a/docs/walkthrough.md b/docs/walkthrough.md new file mode 100644 index 000000000..d61e15e98 --- /dev/null +++ b/docs/walkthrough.md @@ -0,0 +1,320 @@ + +# Table of Contents +- [How to Use This Guide](#how-to-use-this-guide) + - [Gotchas](#gotchas) +- [Goals for the Environment](#goals-for-the-environment) + - [Q&A](#qa) +- [Your Platform Setup](#your-platform-setup) + - [Global Dependencies](#global-dependencies) +- [Workflow 1: Locking a New Project](#workflow-1-locking-a-new-project) +- [Workflow 2: Locking an Existing Project (TBD)](#workflow-2-locking-an-existing-project) +- [Reproduce](#reproduce) + - [With Docker in Mind](#with-docker-in-mind) +- [Upgrade](#upgrade) + - [Upgrade an existing package](#upgrade-an-existing-package) + - [Adding a New Package via `conda`](#adding-a-new-package-via-conda) + - [Adding a New Package via `pip`](#adding-a-new-package-via-pip) + - [A Quick Note on Downgrading](#a-quick-note-on-downgrading) +- [Debugging](#debugging) + - [`conda-lock` giving vague `AssertionError`](#conda-lock-giving-vague-assertionerror) +- [Terms](#terms) +- [References and Resources](#references-and-resources) +- [Versions](#versions) + +# How to Use This Guide +This is a guide on how to produce a reproducible and upgradeable environment for Python applications. A great deal of inspiration came from a pythonspeed.com article about this very topic.[4](#references-and-resources) While the article is a wonderful resource, I needed more so I developed this guide. + +You can use the table of contents to get a quick look at topics. Or you can skip ahead to [Your Platform Setup](#your-platform-setup) which is the beginning of the walkthrough. + +Ultimately, this guide is just that, a *guide*. The goal was to make the concepts easy to follow so that you can apply it to your use case. You may use a different platform than this guide but hopefully still benefit from the content. + +## Gotchas +* I use `pipx` for parts of this guide, you may choose not to. +* My platform may not be your platform so keep an eye out for incompatible commands. +* For simplicity, the guide assumes no explicit versions of packages. +# Goals for the Environment +**Reproducible**: The environment can be reproduced across platforms. +**Upgradeable**: Packages in the environment can be easily changed (e.g., version, new package, etc.) + +## Q&A + +### Why do you suggest using `conda` AND `pip`? +Sometimes a package installed via `conda` (even using the `conda-forge` channel) breaks my environment. I have found using `pip` as the *exception to the rule* has resolved most of these issues. + +### Why is this process slow? +Solving the dependencies of dependencies for one operating system is time consuming. Out of the box, `conda-lock` will try to solve for 3 different operating systems (i.e., linux-64, osx-64, and win-64). However, `conda` recently announced partnering with a `mamba` solver. You could configure your `conda` instance to use a faster solver like this one. This is beyond the scope of this walkthrough. Regardless, it will still take time. + +### Where should I install `conda-lock`? +Using `pipx` to install `conda-lock` allows you to have peace of mind about running the package outside of your `conda` environment (even your `conda` "base" environment). Alternatively, you could install it in your `conda` "base," but I ran into issues that way and this walkthrough will not cover that angle. + +### How can I find the version of my conda dependency? +A command I like to use is, +```bash +(base) $ conda list | grep PACKAGE_NAME +``` +This should show you the package with the version. + +# Your Platform Setup +"Your *platform*" is the platform being used by you from a global perspective. Your environment is going to be the place we initialize the process of standardizing the environment for the goals listed above. By the end of this guide, you should be able to share the environment to another platform. + +## Global Dependencies +- [pipx](https://pypa.github.io/pipx/) — "...help you install and run end-user applications written in Python" +- [conda](https://docs.conda.io/en/latest/) — "Package, dependency and environment management for any language..." +- [pip](https://pypi.org/project/pip/) — "...the package installer for Python" +- [conda-lock](https://github.com/conda-incubator/conda-lock) — "...a lightweight library that can be used to generate fully reproducible lock files for conda environments." +- [conda-forge](https://conda-forge.org/) — "A community-led collection of recipes, build infrastructure and distributions for the conda package manager." + +### Global Install Instructions +Find more detailed guides to installing these tools on their websites [below](#references-and-resources). +* Install `pipx` +* Install `conda-lock` via `pipx` or install `'conda-lock[pip_support]'` if you expect the need for `pip` +* Install `conda` + +# Workflow 1: Locking a New Project +In an ideal world, you can do environment management workflow before you write one line of code. + +If you have conda installed properly, your terminal should open like this, +```bash +(base) $ +``` +1. Change directories to your project. For the tutorial, the project is called `env-management-demo`. +```bash +(base) $ cd env-management-demo +``` +2. Create a new `conda` environment for your project +```bash +(base) $ conda create -n env-management-demo -c conda-forge python +``` +3. Activate your environment. Note that my environment name is the same as my project name. I do this, but you don't have to. This is indicated by how `(base)` changes to `(env-management-demo)`. +```bash +(base) $ conda activate env-management-demo +(env-management-demo) $ +``` +4. Before any other packages are installed, generate a `conda` `environment.yml` file. This step will generate a file that will need some editing in a moment. +```bash +(env-management-demo) $ conda env export --from-history > environment.yml +``` +5. `environment.yml` will populate with your current environment parameters and some configurations. It may look different from mine but your channels must match your actual `conda` channels. I have noted which lines to remove. + +(Optional) Check your channels +```bash +(env-management-demo) $ conda config --show channels +``` + +The following shows a before and after of edits. You will see a `# REMOVE` next to changes. +Before... +```yml +# environment.yml +name: env-management-demo # REMOVE +channels: + - conda-forge + - defaults +dependencies: + - python +prefix: /Users/uname/miniconda3/envs/env-management-demo # REMOVE +``` +After... +```yml +# environment.yml +channels: + - conda-forge + - defaults +dependencies: + - python +``` +6. Now add the platforms you want to support with this `conda-forge` non-standard format. +```yml +# environment.yml +channels: + - conda-forge + - defaults +dependencies: + - python +platforms: # This is non-standard and recommended by conda-lock. + - linux-64 + - osx-64 + - win-64 + - osx-arm64 # For Apple Silicon, e.g. M1/M2 +``` +7. From now on, you will add packages manually to this document under the `dependencies` sequence key unless you have to, for some reason, download a package using pip. Here is an example of that process: +```bash +(env-management-demo) $ conda install -c conda-forge pandas +... +``` +Then add it to the `environment.yml` (indicated by `# NEW`.) +```yml +# environment.yml +channels: + - conda-forge + - defaults +dependencies: + - python + - pandas # NEW +platforms: + - linux-64 + - osx-64 + - win-64 + - osx-arm64 +``` +8. Once you are at a checkpoint in your code and ready to share it, you should generate a lockfile `conda-lock.yml` using the `conda-lock` command. +```bash +(env-management-demo) $ conda-lock -f environment.yml +Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']... +... +``` +9. Now that your environment is "locked," you can test this by creating a new environment from the `conda-lock.yml` file. +```bash +(env-management-demo) $ conda deactivate +(base) $ +(base) $ conda-lock install -n env-locked +INFO:root:Downloading and Extracting Packages +INFO:root:numpy-1.24.1 +INFO:root:pandas-1.5.3 +... +(base) $ conda activate env-locked +(env-locked) $ +``` +10. After testing this environment on your code, you can share the `conda-lock.yml` file with other developers and have them run the code with the same `conda-lock` command. + +# Workflow 2: Locking an Existing Project (TBD) +While this part of the tutorial is unfinished, you could get a pretty good idea of how to do this without this section. For now, you can continue to [Reproduce](#reproduce). Good luck! + +# Reproduce +Once you have the environment locked into a `conda-lock.yml` multi-platform file, you will be able to reproduce the environment (i.e., create and activate it) with the following command as long as the file is in your current directory: +```bash +(base) $ conda-lock install -n NAME +(base) $ conda activate NAME +(NAME) $ +``` +## With Docker in Mind +To minimize the size of the lockfile, you may want to render a single-platform lockfile for linux. +```bash +(base) $ conda-lock render -p linux-64 +... +``` +The default file result is a file named `conda-linux-64.lock`. This lockfile can then be copied into an image and used from within the image. +``` +conda create -n my-locked-env --file conda-linux-64.lock +``` + +# Upgrade +Most work related to upgrading your environment requires manually updating your `environment.yml` you created in an earlier step. In the following examples, I will use the existing `environment.yml` to demonstrate upgrading. + +## Upgrade an existing package +Say there is a new version of pandas available and you want to update to the latest one. This is the simplest use case. +```bash +(base) $ conda-lock --update pandas +``` + +## Adding a New Package via `conda` +We have `pandas` installed, but of course we want to use `matplotlib`. +1. Install `matplotlib` +```bash +(base) $ conda install -c conda-forge matplotlib +... +``` +2. Add `matplotlib` to the `environment.yml` file. +```yml +# environment.yml +channels: + - conda-forge + - defaults +dependencies: + - python + - pandas + - matplotlib # NEW +platforms: + - linux-64 + - osx-64 + - win-64 + - osx-arm64 +``` +3. Update the file using `conda-lock` +```bash +(base) $ conda-lock --update matplotlib +Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']... +... +``` +Note: If you update to a specific version of a package, you will need to provide the version in the command. For example, `matplotlib=3.5` should be included after the `--update` flag. This update flag is useful for the following: +* Updating to the latest version of a package (no version specified) +* Updating a package to a specific version +* Adding a new package + +## Adding a New Package via `pip` +Sometimes `conda` does not have everything you need so you lean on `pip` for a dependency. `conda-lock` comes with `pip` support if you install it with the extra like so: +``` +(base) $ pipx install 'conda-lock[pip_support]' +... +``` +This allows you to add pip dependencies as needed. +1. Install your package within your `conda` env with `pip`. +```bash +(env-management-demo) $ pip install fastapi +... +``` +2. Update the `environment.yml`. +```yml +# environment.yml +channels: + - conda-forge + - defaults +dependencies: + - python + - pandas + - matplotlib=3.5 + - pip: # NEW + - fastapi # NEW +platforms: + - linux-64 + - osx-64 + - win-64 + - osx-arm64 +``` +3. Update with the `conda-lock` command. +``` +(env-management-demo) $ conda deactivate +(base) $ conda-lock --update fastapi +Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']... +... +``` +## A Quick Note on Downgrading +Downgrading is done by updating your `environment.yml` with the downgraded version of a package and then running the `conda-lock` command, +```bash +(base) $ conda-lock --update conda_package=version +``` +Or, if it is a pip package (note the double `==`), +```bash +(base) $ conda-lock --update pip_package==version +``` + +# Debugging +Last Update: February 6, 2023 +This section will be updated as issues are discovered. If issues are discovered, they should be checked against the `conda-lock` repo linked below so contributors can be made aware. + +## `conda-lock` giving vague `AssertionError` +Sometimes an error from `conda-lock` is vague and hard to debug. I am logging some fixes I have discovered during development. +### Are your channels consistent? +Are the channels you have set in `conda` and your `environment.yml` file consistent? If not, make sure they are by editing your `environment.yml` file to match. Once I have found that deleting the existing `conda-lock.yml` file and trying again will resolve this issue. + +# Terms +platform + : the OS being used for a task (e.g., MacOS, Windows, etc.) + +# References and Resources +1. [pypa.github.io — Global Python package manager `pipx`](https://pypa.github.io/pipx/) +2. [docs.conda.io — Environment manager `conda` via miniconda](https://docs.conda.io/en/latest/miniconda.html) +3. [github.com — Cross-platform environment locker `conda-lock`](https://github.com/conda/conda-lock) +4. [pythonspeed.com — Reproducible and upgradable Conda environments with conda-lock](https://pythonspeed.com/articles/conda-dependency-management/) +5. [github.com — Recommended direct dependency update path](https://github.com/conda-incubator/conda-lock/issues/248) +6. [github.com — Optional groups of dependencies #7502](https://github.com/conda/conda/issues/7502) + +# Versions +My local versions for reference. +```text +System Version: macOS 13.0 (22A380) +Kernel Version: Darwin 22.1.0 +conda 22.11.1 +pipx 1.1.0 +conda-lock 1.4.0, installed using Python 3.11.0 +zsh 5.8.1 (x86_64-apple-darwin22.0) +``` \ No newline at end of file From 6c3f2d8a254e9c6db154073dc2a9808454ad2ab9 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Sat, 18 Feb 2023 09:39:00 +0000 Subject: [PATCH 2/7] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/walkthrough.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/walkthrough.md b/docs/walkthrough.md index d61e15e98..a943f4304 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -135,7 +135,7 @@ platforms: # This is non-standard and recommended by conda-lock. - osx-64 - win-64 - osx-arm64 # For Apple Silicon, e.g. M1/M2 -``` +``` 7. From now on, you will add packages manually to this document under the `dependencies` sequence key unless you have to, for some reason, download a package using pip. Here is an example of that process: ```bash (env-management-demo) $ conda install -c conda-forge pandas @@ -165,7 +165,7 @@ Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']... 9. Now that your environment is "locked," you can test this by creating a new environment from the `conda-lock.yml` file. ```bash (env-management-demo) $ conda deactivate -(base) $ +(base) $ (base) $ conda-lock install -n env-locked INFO:root:Downloading and Extracting Packages INFO:root:numpy-1.24.1 From d1359ae5a5d67a8a891b8a0313d2b01cc3e18144 Mon Sep 17 00:00:00 2001 From: jhartman Date: Tue, 14 Mar 2023 15:21:11 -0500 Subject: [PATCH 3/7] replace conda with mamba --- docs/walkthrough.md | 48 ++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 25 deletions(-) diff --git a/docs/walkthrough.md b/docs/walkthrough.md index d61e15e98..f2e7d3e4f 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -12,7 +12,7 @@ - [With Docker in Mind](#with-docker-in-mind) - [Upgrade](#upgrade) - [Upgrade an existing package](#upgrade-an-existing-package) - - [Adding a New Package via `conda`](#adding-a-new-package-via-conda) + - [Adding a New Package via `mamba`](#adding-a-new-package-via-mamba) - [Adding a New Package via `pip`](#adding-a-new-package-via-pip) - [A Quick Note on Downgrading](#a-quick-note-on-downgrading) - [Debugging](#debugging) @@ -38,19 +38,16 @@ Ultimately, this guide is just that, a *guide*. The goal was to make the concept ## Q&A -### Why do you suggest using `conda` AND `pip`? -Sometimes a package installed via `conda` (even using the `conda-forge` channel) breaks my environment. I have found using `pip` as the *exception to the rule* has resolved most of these issues. - -### Why is this process slow? -Solving the dependencies of dependencies for one operating system is time consuming. Out of the box, `conda-lock` will try to solve for 3 different operating systems (i.e., linux-64, osx-64, and win-64). However, `conda` recently announced partnering with a `mamba` solver. You could configure your `conda` instance to use a faster solver like this one. This is beyond the scope of this walkthrough. Regardless, it will still take time. +### Why do you suggest using `mamba` AND `pip`? +Sometimes a package installed via `mamba` breaks my environment. I have found using `pip` as the *exception to the rule* has resolved most of these issues. ### Where should I install `conda-lock`? -Using `pipx` to install `conda-lock` allows you to have peace of mind about running the package outside of your `conda` environment (even your `conda` "base" environment). Alternatively, you could install it in your `conda` "base," but I ran into issues that way and this walkthrough will not cover that angle. +Using `pipx` to install `conda-lock` allows you to have peace of mind about running the package outside of your `mamba` environment (even your `mamba` "base" environment). Alternatively, you could install it in your `mamba` "base," but I ran into issues that way and this walkthrough will not cover that angle. ### How can I find the version of my conda dependency? A command I like to use is, ```bash -(base) $ conda list | grep PACKAGE_NAME +(base) $ mamba list | grep PACKAGE_NAME ``` This should show you the package with the version. @@ -59,7 +56,7 @@ This should show you the package with the version. ## Global Dependencies - [pipx](https://pypa.github.io/pipx/) — "...help you install and run end-user applications written in Python" -- [conda](https://docs.conda.io/en/latest/) — "Package, dependency and environment management for any language..." +- [mamba (via mambaforge)](https://github.com/conda-forge/miniforge#mambaforge) — "Drop-in, faster conda..." - [pip](https://pypi.org/project/pip/) — "...the package installer for Python" - [conda-lock](https://github.com/conda-incubator/conda-lock) — "...a lightweight library that can be used to generate fully reproducible lock files for conda environments." - [conda-forge](https://conda-forge.org/) — "A community-led collection of recipes, build infrastructure and distributions for the conda package manager." @@ -68,12 +65,12 @@ This should show you the package with the version. Find more detailed guides to installing these tools on their websites [below](#references-and-resources). * Install `pipx` * Install `conda-lock` via `pipx` or install `'conda-lock[pip_support]'` if you expect the need for `pip` -* Install `conda` +* Install `mamba` # Workflow 1: Locking a New Project In an ideal world, you can do environment management workflow before you write one line of code. -If you have conda installed properly, your terminal should open like this, +If you have mamba installed properly, your terminal should open like this, ```bash (base) $ ``` @@ -81,24 +78,24 @@ If you have conda installed properly, your terminal should open like this, ```bash (base) $ cd env-management-demo ``` -2. Create a new `conda` environment for your project +2. Create a new `mamba` environment for your project ```bash -(base) $ conda create -n env-management-demo -c conda-forge python +(base) $ mamba create -n env-management-demo -c conda-forge python ``` 3. Activate your environment. Note that my environment name is the same as my project name. I do this, but you don't have to. This is indicated by how `(base)` changes to `(env-management-demo)`. ```bash -(base) $ conda activate env-management-demo +(base) $ mamba activate env-management-demo (env-management-demo) $ ``` -4. Before any other packages are installed, generate a `conda` `environment.yml` file. This step will generate a file that will need some editing in a moment. +4. Before any other packages are installed, generate a `mamba` `environment.yml` file. This step will generate a file that will need some editing in a moment. ```bash -(env-management-demo) $ conda env export --from-history > environment.yml +(env-management-demo) $ mamba env export --from-history > environment.yml ``` -5. `environment.yml` will populate with your current environment parameters and some configurations. It may look different from mine but your channels must match your actual `conda` channels. I have noted which lines to remove. +5. `environment.yml` will populate with your current environment parameters and some configurations. It may look different from mine but your channels must match your actual `mamba` channels. I have noted which lines to remove. (Optional) Check your channels ```bash -(env-management-demo) $ conda config --show channels +(env-management-demo) $ mamba config --show channels ``` The following shows a before and after of edits. You will see a `# REMOVE` next to changes. @@ -138,7 +135,7 @@ platforms: # This is non-standard and recommended by conda-lock. ``` 7. From now on, you will add packages manually to this document under the `dependencies` sequence key unless you have to, for some reason, download a package using pip. Here is an example of that process: ```bash -(env-management-demo) $ conda install -c conda-forge pandas +(env-management-demo) $ mamba install -c conda-forge pandas ... ``` Then add it to the `environment.yml` (indicated by `# NEW`.) @@ -164,14 +161,14 @@ Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']... ``` 9. Now that your environment is "locked," you can test this by creating a new environment from the `conda-lock.yml` file. ```bash -(env-management-demo) $ conda deactivate +(env-management-demo) $ mamba deactivate (base) $ (base) $ conda-lock install -n env-locked INFO:root:Downloading and Extracting Packages INFO:root:numpy-1.24.1 INFO:root:pandas-1.5.3 ... -(base) $ conda activate env-locked +(base) $ mamba activate env-locked (env-locked) $ ``` 10. After testing this environment on your code, you can share the `conda-lock.yml` file with other developers and have them run the code with the same `conda-lock` command. @@ -183,7 +180,7 @@ While this part of the tutorial is unfinished, you could get a pretty good idea Once you have the environment locked into a `conda-lock.yml` multi-platform file, you will be able to reproduce the environment (i.e., create and activate it) with the following command as long as the file is in your current directory: ```bash (base) $ conda-lock install -n NAME -(base) $ conda activate NAME +(base) $ mamba activate NAME (NAME) $ ``` ## With Docker in Mind @@ -194,7 +191,7 @@ To minimize the size of the lockfile, you may want to render a single-platform l ``` The default file result is a file named `conda-linux-64.lock`. This lockfile can then be copied into an image and used from within the image. ``` -conda create -n my-locked-env --file conda-linux-64.lock +mamba create -n my-locked-env --file conda-linux-64.lock ``` # Upgrade @@ -210,7 +207,7 @@ Say there is a new version of pandas available and you want to update to the lat We have `pandas` installed, but of course we want to use `matplotlib`. 1. Install `matplotlib` ```bash -(base) $ conda install -c conda-forge matplotlib +(base) $ mamba install -c conda-forge matplotlib ... ``` 2. Add `matplotlib` to the `environment.yml` file. @@ -272,7 +269,7 @@ platforms: ``` 3. Update with the `conda-lock` command. ``` -(env-management-demo) $ conda deactivate +(env-management-demo) $ mamba deactivate (base) $ conda-lock --update fastapi Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']... ... @@ -314,6 +311,7 @@ My local versions for reference. System Version: macOS 13.0 (22A380) Kernel Version: Darwin 22.1.0 conda 22.11.1 +mamba 1.1.0 pipx 1.1.0 conda-lock 1.4.0, installed using Python 3.11.0 zsh 5.8.1 (x86_64-apple-darwin22.0) From 36406d7c79f8bd11c3f622aec1babd747b6d1686 Mon Sep 17 00:00:00 2001 From: jhartman Date: Tue, 14 Mar 2023 15:30:49 -0500 Subject: [PATCH 4/7] remove defaults channel from conda approach artifact --- docs/walkthrough.md | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/docs/walkthrough.md b/docs/walkthrough.md index 7ada4ff7b..5b992315d 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -31,9 +31,9 @@ Ultimately, this guide is just that, a *guide*. The goal was to make the concept ## Gotchas * I use `pipx` for parts of this guide, you may choose not to. * My platform may not be your platform so keep an eye out for incompatible commands. -* For simplicity, the guide assumes no explicit versions of packages. + # Goals for the Environment -**Reproducible**: The environment can be reproduced across platforms. +**Reproducible**: The environment can be reproduced across platforms. **Upgradeable**: Packages in the environment can be easily changed (e.g., version, new package, etc.) ## Q&A @@ -105,7 +105,6 @@ Before... name: env-management-demo # REMOVE channels: - conda-forge - - defaults dependencies: - python prefix: /Users/uname/miniconda3/envs/env-management-demo # REMOVE @@ -115,7 +114,6 @@ After... # environment.yml channels: - conda-forge - - defaults dependencies: - python ``` @@ -124,7 +122,6 @@ dependencies: # environment.yml channels: - conda-forge - - defaults dependencies: - python platforms: # This is non-standard and recommended by conda-lock. @@ -143,7 +140,6 @@ Then add it to the `environment.yml` (indicated by `# NEW`.) # environment.yml channels: - conda-forge - - defaults dependencies: - python - pandas # NEW @@ -215,7 +211,6 @@ We have `pandas` installed, but of course we want to use `matplotlib`. # environment.yml channels: - conda-forge - - defaults dependencies: - python - pandas @@ -254,7 +249,6 @@ This allows you to add pip dependencies as needed. # environment.yml channels: - conda-forge - - defaults dependencies: - python - pandas From bb2ced3f2b64b92f8c58fee469bf34b5ad41ea17 Mon Sep 17 00:00:00 2001 From: jhartman Date: Tue, 14 Mar 2023 15:41:31 -0500 Subject: [PATCH 5/7] fix update section and make other minor edits --- docs/walkthrough.md | 43 ++++++++++--------------------------------- 1 file changed, 10 insertions(+), 33 deletions(-) diff --git a/docs/walkthrough.md b/docs/walkthrough.md index 5b992315d..47e870371 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -190,23 +190,13 @@ The default file result is a file named `conda-linux-64.lock`. This lockfile can mamba create -n my-locked-env --file conda-linux-64.lock ``` -# Upgrade -Most work related to upgrading your environment requires manually updating your `environment.yml` you created in an earlier step. In the following examples, I will use the existing `environment.yml` to demonstrate upgrading. - -## Upgrade an existing package -Say there is a new version of pandas available and you want to update to the latest one. This is the simplest use case. +# Updating the Environment +Currently the `--update` flag for the `conda-lock` command does not work as expected. For now, the way to update a package (e.g., upgrade, downgrade, remove, or add a package) is a workflow described like so: +1. Update your environment, for example, add the package `matplotlib` via `mamba`: ```bash -(base) $ conda-lock --update pandas +(env-management-demo) $ mamba install matplotlib ``` - -## Adding a New Package via `conda` -We have `pandas` installed, but of course we want to use `matplotlib`. -1. Install `matplotlib` -```bash -(base) $ mamba install -c conda-forge matplotlib -... -``` -2. Add `matplotlib` to the `environment.yml` file. +2. Manually add this to the `environment.yml` file: ```yml # environment.yml channels: @@ -221,16 +211,12 @@ platforms: - win-64 - osx-arm64 ``` -3. Update the file using `conda-lock` +3. Rerun `conda-lock` which will resolve the entire environent: ```bash -(base) $ conda-lock --update matplotlib -Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']... -... +(env-management-demo) $ conda-lock -f environment.yml ``` -Note: If you update to a specific version of a package, you will need to provide the version in the command. For example, `matplotlib=3.5` should be included after the `--update` flag. This update flag is useful for the following: -* Updating to the latest version of a package (no version specified) -* Updating a package to a specific version -* Adding a new package + +Note that it is not necessary to add the package to your environment as seen in step 1 above, but I would reccomend doing this and testing your code before persisitng this change to `conda-lock.yml`. ## Adding a New Package via `pip` Sometimes `conda` does not have everything you need so you lean on `pip` for a dependency. `conda-lock` comes with `pip` support if you install it with the extra like so: @@ -268,15 +254,6 @@ platforms: Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']... ... ``` -## A Quick Note on Downgrading -Downgrading is done by updating your `environment.yml` with the downgraded version of a package and then running the `conda-lock` command, -```bash -(base) $ conda-lock --update conda_package=version -``` -Or, if it is a pip package (note the double `==`), -```bash -(base) $ conda-lock --update pip_package==version -``` # Debugging Last Update: February 6, 2023 @@ -285,7 +262,7 @@ This section will be updated as issues are discovered. If issues are discovered, ## `conda-lock` giving vague `AssertionError` Sometimes an error from `conda-lock` is vague and hard to debug. I am logging some fixes I have discovered during development. ### Are your channels consistent? -Are the channels you have set in `conda` and your `environment.yml` file consistent? If not, make sure they are by editing your `environment.yml` file to match. Once I have found that deleting the existing `conda-lock.yml` file and trying again will resolve this issue. +Are the channels you have set in `mamba` and your `environment.yml` file consistent? If not, make sure they are by editing your `environment.yml` file to match. Once I have found that deleting the existing `conda-lock.yml` file and trying again will resolve this issue. # Terms platform From 557bd7a84ba5623cd7a20f1e72096d5241b3ad0e Mon Sep 17 00:00:00 2001 From: jhartman Date: Thu, 16 Mar 2023 14:50:37 -0500 Subject: [PATCH 6/7] remove table of contents, improve number reference --- docs/walkthrough.md | 40 +++++++++------------------------------- 1 file changed, 9 insertions(+), 31 deletions(-) diff --git a/docs/walkthrough.md b/docs/walkthrough.md index 47e870371..591b0b79d 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -1,35 +1,13 @@ - -# Table of Contents -- [How to Use This Guide](#how-to-use-this-guide) - - [Gotchas](#gotchas) -- [Goals for the Environment](#goals-for-the-environment) - - [Q&A](#qa) -- [Your Platform Setup](#your-platform-setup) - - [Global Dependencies](#global-dependencies) -- [Workflow 1: Locking a New Project](#workflow-1-locking-a-new-project) -- [Workflow 2: Locking an Existing Project (TBD)](#workflow-2-locking-an-existing-project) -- [Reproduce](#reproduce) - - [With Docker in Mind](#with-docker-in-mind) -- [Upgrade](#upgrade) - - [Upgrade an existing package](#upgrade-an-existing-package) - - [Adding a New Package via `mamba`](#adding-a-new-package-via-mamba) - - [Adding a New Package via `pip`](#adding-a-new-package-via-pip) - - [A Quick Note on Downgrading](#a-quick-note-on-downgrading) -- [Debugging](#debugging) - - [`conda-lock` giving vague `AssertionError`](#conda-lock-giving-vague-assertionerror) -- [Terms](#terms) -- [References and Resources](#references-and-resources) -- [Versions](#versions) - # How to Use This Guide -This is a guide on how to produce a reproducible and upgradeable environment for Python applications. A great deal of inspiration came from a pythonspeed.com article about this very topic.[4](#references-and-resources) While the article is a wonderful resource, I needed more so I developed this guide. +This is a guide on how to produce a reproducible and upgradeable environment for Python applications. A great deal of inspiration came from a pythonspeed.com article about this very topic.[1](#references-and-resources) While the article is a wonderful resource, I needed more so I developed this guide. You can use the table of contents to get a quick look at topics. Or you can skip ahead to [Your Platform Setup](#your-platform-setup) which is the beginning of the walkthrough. Ultimately, this guide is just that, a *guide*. The goal was to make the concepts easy to follow so that you can apply it to your use case. You may use a different platform than this guide but hopefully still benefit from the content. ## Gotchas -* I use `pipx` for parts of this guide, you may choose not to. +* This guide uses `pipx` for parts of this guide, you may choose not to. +* This guide use `mamba` in place of `conda`. * My platform may not be your platform so keep an eye out for incompatible commands. # Goals for the Environment @@ -269,12 +247,12 @@ platform : the OS being used for a task (e.g., MacOS, Windows, etc.) # References and Resources -1. [pypa.github.io — Global Python package manager `pipx`](https://pypa.github.io/pipx/) -2. [docs.conda.io — Environment manager `conda` via miniconda](https://docs.conda.io/en/latest/miniconda.html) -3. [github.com — Cross-platform environment locker `conda-lock`](https://github.com/conda/conda-lock) -4. [pythonspeed.com — Reproducible and upgradable Conda environments with conda-lock](https://pythonspeed.com/articles/conda-dependency-management/) -5. [github.com — Recommended direct dependency update path](https://github.com/conda-incubator/conda-lock/issues/248) -6. [github.com — Optional groups of dependencies #7502](https://github.com/conda/conda/issues/7502) +1. [pythonspeed.com — Reproducible and upgradable Conda environments with conda-lock](https://pythonspeed.com/articles/conda-dependency-management/) +2. [pypa.github.io — Global Python package manager `pipx`](https://pypa.github.io/pipx/) +3. [docs.conda.io — Environment manager `conda` via miniconda](https://docs.conda.io/en/latest/miniconda.html) +4. [github.com — Cross-platform environment locker `conda-lock`](https://github.com/conda/conda-lock) +5. [github.com — Recommended direct dependency update path](https://github.com/conda-incubator/conda-lock/issues/248) +6. [github.com — Optional groups of dependencies #7502](https://github.com/conda/conda/issues/7502) # Versions My local versions for reference. From f4476c5037d35fcea30f77b8964f460725ef8d65 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Thu, 23 Mar 2023 19:03:12 +0000 Subject: [PATCH 7/7] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/walkthrough.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/walkthrough.md b/docs/walkthrough.md index 591b0b79d..7b3453ecf 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -11,7 +11,7 @@ Ultimately, this guide is just that, a *guide*. The goal was to make the concept * My platform may not be your platform so keep an eye out for incompatible commands. # Goals for the Environment -**Reproducible**: The environment can be reproduced across platforms. +**Reproducible**: The environment can be reproduced across platforms. **Upgradeable**: Packages in the environment can be easily changed (e.g., version, new package, etc.) ## Q&A @@ -136,7 +136,7 @@ Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']... 9. Now that your environment is "locked," you can test this by creating a new environment from the `conda-lock.yml` file. ```bash (env-management-demo) $ mamba deactivate -(base) $ +(base) $ (base) $ conda-lock install -n env-locked INFO:root:Downloading and Extracting Packages INFO:root:numpy-1.24.1