Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] if latest contexts have changed, retrieve the previous contexts' test cache and then use ./setup_test_cache u to update #1099

Merged
merged 4 commits into from
Nov 27, 2024
Merged

[CI] if latest contexts have changed, retrieve the previous contexts' test cache and then use ./setup_test_cache u to update #1099

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
393f1cb
use `./setup_test_cache u` to update a previous existing cache
zacharyburnett Nov 25, 2024
9189971
only clone cache test repo if cache not hit
zacharyburnett Nov 25, 2024
c30d031
don't uninstall crds just to install it again
zacharyburnett Nov 25, 2024
58fe658
always run `./setup_test_cache` with `u`
zacharyburnett Nov 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 21 additions & 25 deletions .github/workflows/cache.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: test cache
name: download CRDS test cache

on:
workflow_call:
Expand Down Expand Up @@ -43,7 +43,7 @@ jobs:
repository: spacetelescope/crds
- id: key
run: echo "key=test-cache-${{ needs.contexts.outputs.hst }}-${{ needs.contexts.outputs.jwst }}" >> $GITHUB_OUTPUT
- id: lookup-combined-cache
- id: lookup-cache
name: check for existence of combined CRDS cache (`${{ steps.key.outputs.key }}`)
uses: actions/cache/restore@v4
with:
Expand All @@ -52,26 +52,22 @@ jobs:
${{ env.CRDS_TESTING_CACHE }}
key: ${{ steps.key.outputs.key }}
lookup-only: true
- if: steps.lookup-combined-cache.outputs.cache-hit != 'true'
id: crds-cache-test
- if: steps.lookup-cache.outputs.cache-hit != 'true'
id: retrieve-previous-cache
name: retrieve a previous combined CRDS cache
uses: actions/cache/restore@v4
with:
path: |
${{ env.CRDS_PATH }}
${{ env.CRDS_TESTING_CACHE }}
key: ${{ steps.key.outputs.key }}
restore-keys: |
test-cache-
- if: steps.retrieve-previous-cache.outputs.cache-matched-key == ''
Comment on lines +55 to +65
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if a cache with the latest contexts does not exist, attempt to retrieve a previous cache that does exist (with the intent of running ./setup_test_cache /tmp u on it later)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really sure what is happening there. I know there are differences between the two contexts, but I feel like the test cache had nothing to update.

run: |
git clone https://github.com/spacetelescope/crds-cache-test.git
mv crds-cache-test ${{ env.CRDS_TEST_ROOT }}
- if: steps.lookup-combined-cache.outputs.cache-hit != 'true'
id: retrieve-hst-cache
name: retrieve HST CRDS cache (`test-cache-${{ needs.contexts.outputs.hst }}`)
uses: actions/cache@v4
with:
path: ${{ env.CRDS_PATH }}/**/hst*
key: test-cache-${{ needs.contexts.outputs.hst }}
- if: steps.lookup-combined-cache.outputs.cache-hit != 'true'
id: retrieve-jwst-cache
name: retrieve JWST CRDS cache (`test-cache-${{ needs.contexts.outputs.jwst }}`)
uses: actions/cache@v4
with:
path: ${{ env.CRDS_PATH }}/**/jwst*
key: test-cache-${{ needs.contexts.outputs.jwst }}
- if: steps.lookup-combined-cache.outputs.cache-hit != 'true'
- if: steps.lookup-cache.outputs.cache-hit != 'true'
uses: mamba-org/setup-micromamba@v2
with:
environment-name: crds-testing
Expand All @@ -87,13 +83,13 @@ jobs:
init-shell: bash
cache-environment: true
cache-downloads: true
- if: steps.lookup-combined-cache.outputs.cache-hit != 'true'
- if: steps.lookup-cache.outputs.cache-hit != 'true'
run: ./install
- if: steps.lookup-cache.outputs.cache-hit != 'true'
run: pip install git+https://github.com/spacetelescope/jwst roman-datamodels
- if: steps.lookup-combined-cache.outputs.cache-hit != 'true'
run: pip uninstall --yes crds && ./install && pip install .
- if: steps.lookup-combined-cache.outputs.cache-hit != 'true'
run: ./setup_test_cache ${{ env.CRDS_TEST_ROOT }} ${{ (steps.retrieve-hst-cache.outputs.cache-hit == 'true' || steps.retrieve-jwst-cache.outputs.cache-hit == 'true') && 'u' || 'c' }}
- if: steps.lookup-combined-cache.outputs.cache-hit != 'true'
- if: steps.lookup-cache.outputs.cache-hit != 'true'
run: ./setup_test_cache ${{ env.CRDS_TEST_ROOT }} u
- if: steps.lookup-cache.outputs.cache-hit != 'true'
Comment on lines -94 to +91
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't ever need to run with c here, only u

name: save combined CRDS cache (`${{ steps.key.outputs.key }}`)
uses: actions/cache/save@v4
with:
Expand Down
Loading