Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change workflow to mamba #271

Merged
merged 5 commits into from
Jun 19, 2023
Merged

Change workflow to mamba #271

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c0aadc7
Change workflow to mamba
forsyth2 Jun 15, 2023
8450cb8
Address comments
forsyth2 Jun 15, 2023
da0807c
Revert sphinx-multiversion handling
forsyth2 Jun 15, 2023
4e6c430
Get sphinx-multiversion from conda-forge, not pypi
xylar Jun 16, 2023
ef92f3f
Use conda environment for docs in CI
xylar Jun 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 23 additions & 4 deletions .github/workflows/build_workflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,29 @@ jobs:
steps:
- uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v4
- name: Cache Conda
uses: actions/cache@v3
env:
CACHE_NUMBER: 0
with:
python-version: 3.9
path: ~/conda_pkgs_dir
key: ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{
hashFiles('conda/dev.yml') }}


- name: Build Conda Environment
uses: conda-incubator/setup-miniconda@v2
with:
activate-environment: zstash_dev
miniforge-variant: Mambaforge
miniforge-version: latest
use-mamba: true
mamba-version: "*"
environment-file: conda/dev.yml
channel-priority: strict
auto-update-conda: true
# IMPORTANT: This needs to be set for caching to work properly!
use-only-tar-bz2: true

- name: Install `zstash` Package
run: |
Expand Down Expand Up @@ -81,7 +100,7 @@ jobs:

- name: Install Dependencies
run: |
pip install sphinx==5.2.3 sphinx_rtd_theme==1.0.0 sphinx-multiversion==0.2.4 docutils==0.16 "jinja2<3.1"
mamba install -y sphinx==5.2.3 sphinx_rtd_theme==1.0.0 sphinx-multiversion==0.2.4 docutils==0.16 "jinja2<3.1"

- name: Build Sphinx Docs
run: |
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/release_workflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,9 @@ jobs:
${{ runner.os }}-pip-
${{ runner.os }}-

# Using pip for Sphinx dependencies because it takes too long to reproduce a conda environment (~10 secs vs. 3-4 mins)
- name: Install Dependencies
run: |
pip install sphinx==5.2.3 sphinx_rtd_theme==1.0.0 sphinx-multiversion==0.2.4 docutils==0.16 "jinja2<3.1"
mamba install -y sphinx==5.2.3 sphinx_rtd_theme==1.0.0 sphinx-multiversion==0.2.4 docutils==0.16 "jinja2<3.1"

- name: Build Sphinx Docs
run: |
Expand Down
59 changes: 37 additions & 22 deletions docs/source/getting_started.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,32 +81,38 @@ Others/Local

If the system doesn't come with conda pre-installed, follow these instructions:

1. Download Conda
1. Download Mambaforge

Linux
::

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh

MacOS (note that ``zstash`` is not supported on MacOS, but it may be useful to contribute to the documentation on MacOS)
MacOS x86_64 (note that ``zstash`` is not supported on MacOS, but it may be useful to contribute to the documentation on MacOS)
::

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-MacOSX-x86_64.sh

2. Install Conda
2. Install Mambaforge

Linux
::

bash ./Miniconda3-latest-Linux-x86_64.sh
bash ./Mambaforge-Linux-x86_64.sh


MacOS
MacOS x86_64
::

bash ./Miniconda3-latest-MacOSX-x86_64.sh
bash ./Mambaforge-MacOSX-x86_64.sh

- ``Do you wish the installer to initialize Miniconda3 by running conda init? [yes|no] yes``
When you see: ::

by running conda init? [yes|no]
[no] >>> yes

respond with ``yes`` so ``conda`` and ``mamba`` commands are available on
initializing a new bash terminal.

3. If you are working on a machine/network that intercepts SSL communications (such as
acme1), you will get an SSL error unless you disable the SSL verification:
Expand All @@ -116,31 +122,40 @@ acme1), you will get an SSL error unless you disable the SSL verification:
conda config --set ssl_verify false
binstar config --set ssl_verify False

4. Once conda is properly working, you can install the **(a) Latest Stable Release** or
4. Once conda and mamba are properly working, you can install the **(a) Latest Stable Release** or
create a **(b) Development Environment**.

(a) Latest Stable Release
=========================

Installation using conda
Installation using mamba
------------------------

First, make sure that you're using ``bash``. ::

$ bash
bash

You must have Anaconda installed as well.
You must have a conda base enviornment installed as well.
See :ref:`"Installation in a Conda Environment" <conda_environment>` section above for
installing conda.
Create a new Anaconda environment with zstash installed and activate it: ::

$ conda create -n zstash_env -c e3sm -c conda-forge zstash
$ source activate zstash_env
These steps should not be necessary if you installed Mambaforge as suggested
above but may be needed if you have previously installed Miniconda3 instead: ::

conda install -y -n base mamba
conda config --add channels conda-forge
conda config --set channel_priority strict

Create a new conda environment with ``zstash`` installed and activate it: ::

Or you can install zstash in an existing environment. ::
mamba create -n zstash_env zstash
conda activate zstash_env

$ conda install zstash -c e3sm -c conda-forge
Or (less recommended because of potential conflicts) you can install ``zstash``
in an existing environment. ::

mamba install zppy

Installation on NERSC
---------------------
Expand All @@ -154,10 +169,10 @@ not directly available there, so you will need to manually activate Anaconda bef
Updating
--------

If you **installed via Anaconda** (e.g., not through the unified environment),
you can update ``zstash`` by doing the following: ::
If you **installed into your own conda environment** (e.g., not through the
unified environment), you can update ``zstash`` by doing the following: ::

conda update zstash -c e3sm -c conda-forge
mamba update zstash

.. _dev-env:

Expand Down Expand Up @@ -224,7 +239,7 @@ Furthermore, the dev environment includes quality assurance (QA) tools such as c

::

conda clean --all
mamba clean --all

4. Enter the fork's clone.

Expand All @@ -239,7 +254,7 @@ Furthermore, the dev environment includes quality assurance (QA) tools such as c

::

conda env create -f conda/dev.yml
mamba env create -f conda/dev.yml
conda activate zstash_dev

6. Install ``pre-commit``.
Expand Down
5 changes: 0 additions & 5 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,5 @@
description="Long term HPSS archiving software for E3SM",
packages=find_packages(include=["zstash", "zstash.*"]),
python_requires=">=3.6",
install_requires=[
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is zstash supposed to have zero dependencies? Should we remove these deps from conda-forge as well?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We just don't want dependencies to come from pip. I don't remember the exact details. Hopefully @forsyth2 can point us to the relevant discussion, but these were causing trouble in the recent past.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it was just this comment from @tomvothecoder:
E3SM-Project/zppy#397 (comment)

Copy link
Collaborator Author

@forsyth2 forsyth2 Jun 16, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

point us to the relevant discussion

I'm not sure if this is what you meant, but these installation requirements were introduced in #154 to enable Globus support. A couple other files subsequently had to be modified, which was done in #186.

"fair-research-login>=0.2.6,<0.3.0",
"globus-sdk>=3.0.0,<4.0.0",
"six",
],
Comment on lines -11 to -15
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tomvothecoder It looks like the tests don't like this part being removed. Am I supposed to add these requirements somewhere else?

Copy link
Collaborator

@tomvothecoder tomvothecoder Jun 15, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue is that the build step of the build_workflow.yml is using a pip environment and referencing the install_requires section from setup.py for the dependencies.

You need to replace these lines

zstash/.github/workflows/build_workflow.yml

Lines 38 to 41 in c0aadc7

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3.9

with the step to cache and create the conda environment (zppy example):

      - name: Cache Conda
        uses: actions/cache@v3
        env:
          CACHE_NUMBER: 0
        with:
          path: ~/conda_pkgs_dir
          key: ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{
            hashFiles('conda/dev.yml') }}


      - name: Build Conda Environment
        uses: conda-incubator/setup-miniconda@v2
        with:
          activate-environment: zstash_dev
      miniforge-variant: Mambaforge
          miniforge-version: latest
          use-mamba: true
          mamba-version: "*"
          environment-file: conda/dev.yml
          channel-priority: strict
          auto-update-conda: true
          # IMPORTANT: This needs to be set for caching to work properly!
          use-only-tar-bz2: true

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tomvothecoder Thanks! Now I'm getting sphinx-multiversion 0.2.4 does not exist (perhaps a typo or a missing channel). on the tests though...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And everything seems to match up with what was done for zppy, which did work.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tomvothecoder Ah, actually zppy did not work. I missed a failing GitHub action. Fixing a syntax error in E3SM-Project/zppy#434, I end up with the same error as zstash described here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, my latest commit reverting that change makes the GitHub Actions check pass. I may keep it like that to get this merged and address this issue later in the release testing process.

Copy link
Collaborator Author

@forsyth2 forsyth2 Jun 16, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The checks appeared to pass for E3SM-Project/zppy#434, but after merging, the publish-docs Action failed (https://github.com/E3SM-Project/zppy/actions/runs/5284999456/jobs/9563041218: line 1: mamba: command not found). Because of this, I think it's best to hold off on merging this PR until the sphinx-multiversion error is resolved.

entry_points={"console_scripts": ["zstash=zstash.main:main"]},
)