Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add caching to CI workflow #109

Merged
merged 6 commits into from
Nov 27, 2023
Merged

Add caching to CI workflow #109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
727e531
Add caching to CI workflow
naegelejd Nov 22, 2023
c1a2c99
Fix conda cache dir
naegelejd Nov 22, 2023
2b87898
Cache conda environment instead of packages
naegelejd Nov 22, 2023
7ad47a7
Fix missing shell in github action
naegelejd Nov 22, 2023
d5b64f4
Clean up github action
naegelejd Nov 22, 2023
6cf4150
Upgrade Github cache action to v3
naegelejd Nov 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 29 additions & 2 deletions .github/actions/configure-environment/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,38 @@ runs:
/#.*arch=/ && $0 ~ "arch="arch { print }
' environment.yml > ci-environment.yml

- uses: conda-incubator/setup-miniconda@v2
- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2
with:
miniforge-variant: Mambaforge
miniforge-version: latest
# Do not specify environment file - see Cache step below
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So at this point, we have an empty environment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the env always be updated in the last step, whether the cache exists or not.

activate-environment: yardl
environment-file: ci-environment.yml
use-mamba: true

- name: Get Date
id: get-date
shell: bash
run: echo "DATE=$(date -u +'%Y%m%d')" >> $GITHUB_ENV

- name: Cache Conda Env
id: cache-conda
uses: actions/cache@v3
with:
path: ${{ env.CONDA }}/envs
key:
conda-${{ runner.os }}--${{ runner.arch }}--${{
hashFiles('ci-environment.yml') }}-${{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to throw the conda version in there too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's necessary. The cache is invalidated every 24 hours, so within that time period, if the conda version changes and somehow the resolved set of packages for our ci-environment.yml changes we can easily invalidate the cache on this page: https://github.com/microsoft/yardl/actions/caches.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense.

env.DATE }}-${{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we want the date to be in the cache key?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding the date ensure the cache is updated daily to account for any changes in conda package versions. See https://github.com/marketplace/actions/setup-miniconda#caching-environments, which I essentially copied here.

env.CONDA_CACHE_NUMBER }}

- name: Update Environment
shell: bash
run: mamba env update -n yardl -f ci-environment.yml
if: steps.cache-conda.outputs.cache-hit != 'true'


- uses: actions/setup-go@v4
with:
go-version: '1.21.3'
cache-dependency-path: tooling/go.sum
10 changes: 8 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ defaults:
# See https://github.com/marketplace/actions/setup-miniconda#important
shell: bash -el {0}

env:
# Increase this to manually reset the conda environment cache
CONDA_CACHE_NUMBER: 0

jobs:
validate:
strategy:
Expand All @@ -39,9 +43,10 @@ jobs:
steps:
- uses: actions/checkout@v3

- uses: actions/setup-go@v3
- uses: actions/setup-go@v4
with:
go-version: "1.21.3"
cache-dependency-path: tooling/go.sum

- name: Build yardl binaries for multiple platforms
uses: goreleaser/goreleaser-action@v3
Expand Down Expand Up @@ -136,9 +141,10 @@ jobs:
steps:
- uses: actions/checkout@v3

- uses: actions/setup-go@v3
- uses: actions/setup-go@v4
with:
go-version: "1.21.3"
cache-dependency-path: tooling/go.sum

- name: Set variables
run: |
Expand Down
Loading