Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download GIN Data for Pipeline Tests #795

Merged
merged 101 commits into from
May 30, 2024
Merged
Show file tree
Hide file tree
Changes from 84 commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
1bbc780
Update testing-pipelines.yml
garrettmflynn May 24, 2024
cc563d7
Create example_data_cache.yml
CodyCBakerPhD May 27, 2024
79aec20
Create testing_pipelines.yml
CodyCBakerPhD May 27, 2024
e70ba8e
Rename testing_pipelines.yml to deploy_testing_pipelines.yml
CodyCBakerPhD May 27, 2024
5f6a4fc
Update and rename testing-pipelines.yml to testing_pipelines.yml
CodyCBakerPhD May 27, 2024
5deda1e
Update deploy_testing_pipelines.yml
CodyCBakerPhD May 27, 2024
130e546
Merge branch 'main' into download-gin-data-for-tests
CodyCBakerPhD May 27, 2024
c238e6f
snake case
CodyCBakerPhD May 27, 2024
2e76258
Merge branch 'download-gin-data-for-tests' of https://github.com/neur…
CodyCBakerPhD May 27, 2024
8fffa11
fix yaml
CodyCBakerPhD May 27, 2024
b9895a6
fix yaml
CodyCBakerPhD May 27, 2024
4f4e32c
remove unused action
CodyCBakerPhD May 27, 2024
34a48e7
Rename pyflask_build_and_dist_tests.yml to testing_pyflask_build_and_…
CodyCBakerPhD May 27, 2024
c9f147b
Rename project_action.yml to add_to_nwb_dashboard.yml
CodyCBakerPhD May 27, 2024
5a7e7df
move codespell from action to pre-commit
CodyCBakerPhD May 27, 2024
9bd42e6
Delete .github/workflows/codespell.yml
CodyCBakerPhD May 27, 2024
ca546be
Update testing_pipelines.yml
CodyCBakerPhD May 27, 2024
633f66c
remove concurrency from triggered
CodyCBakerPhD May 27, 2024
886fad2
Update testing_pipelines.yml
CodyCBakerPhD May 27, 2024
1c79fad
streamline CI deployment; adjust all names
CodyCBakerPhD May 27, 2024
4ac8d0b
try again
CodyCBakerPhD May 27, 2024
4197ab9
try again
CodyCBakerPhD May 27, 2024
2cf529c
try again
CodyCBakerPhD May 27, 2024
68e971e
try again
CodyCBakerPhD May 27, 2024
7f5e19d
Fail if no path
garrettmflynn May 27, 2024
e21df99
Update testing_pipelines.yml
garrettmflynn May 27, 2024
0d23d60
Archive screenshots from the test pipelines
garrettmflynn May 27, 2024
6b67a34
Remove pipeline screenshots
garrettmflynn May 27, 2024
4c99f5f
Update example_data_cache.yml
garrettmflynn May 27, 2024
b7e40c3
Update example_data_cache.yml
garrettmflynn May 27, 2024
8245334
Merge branch 'main' into download-gin-data-for-tests
CodyCBakerPhD May 28, 2024
6bdc053
Update example_data_cache.yml
CodyCBakerPhD May 28, 2024
0c8ea73
Update example_data_cache.yml
CodyCBakerPhD May 28, 2024
d361629
try without cd
CodyCBakerPhD May 28, 2024
e224ec1
Update example_data_cache.yml
CodyCBakerPhD May 28, 2024
093eff3
Update example_data_cache.yml
CodyCBakerPhD May 28, 2024
110ecbc
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 28, 2024
7a33def
Update pipelines.test.ts
garrettmflynn May 28, 2024
ca70bad
Merge branch 'download-gin-data-for-tests' of https://github.com/Neur…
garrettmflynn May 28, 2024
fd3d193
Update example_pipelines.yml
garrettmflynn May 28, 2024
3aa7ac9
Update pipelines.test.ts
garrettmflynn May 28, 2024
b2a3627
Merge branch 'main' into download-gin-data-for-tests
garrettmflynn May 28, 2024
06cdfab
Add additional assertion
garrettmflynn May 28, 2024
c5254ff
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 28, 2024
498018c
Update pipelines.test.ts
garrettmflynn May 28, 2024
0b4eec8
Merge branch 'download-gin-data-for-tests' of https://github.com/Neur…
garrettmflynn May 28, 2024
8c07aa5
Update pipelines.test.ts
garrettmflynn May 28, 2024
e726e82
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 28, 2024
c5acde9
Overwrite previous screenshots
garrettmflynn May 28, 2024
f441b60
Merge branch 'download-gin-data-for-tests' of https://github.com/Neur…
garrettmflynn May 28, 2024
0e4cc99
Use an actual assertion to test if test data exists
garrettmflynn May 28, 2024
b65f1e2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 28, 2024
c9889fe
Properly dismiss notifications
garrettmflynn May 28, 2024
7b30236
Merge branch 'main' into download-gin-data-for-tests
CodyCBakerPhD May 28, 2024
0731f0f
Run a test that provides insight into the errors occuring for each te…
garrettmflynn May 29, 2024
98a34ed
Use relative paths for cache
garrettmflynn May 29, 2024
2be2f00
Update testing_pipelines.yml
garrettmflynn May 29, 2024
67134a5
Update testing_pipelines.yml
CodyCBakerPhD May 29, 2024
be020b4
Show env file
garrettmflynn May 29, 2024
a637d0a
Merge branch 'download-gin-data-for-tests' of https://github.com/Neur…
garrettmflynn May 29, 2024
06fb327
Merge branch 'main' into download-gin-data-for-tests
garrettmflynn May 29, 2024
d3bc6c8
Update testing_pipelines.yml
garrettmflynn May 29, 2024
f84d191
Update testing_pipelines.yml
garrettmflynn May 29, 2024
3474c6d
Update testing_pipelines.yml
garrettmflynn May 29, 2024
cfba1fc
Merge branch 'main' into download-gin-data-for-tests
CodyCBakerPhD May 29, 2024
b898150
separate printout for cleaner view
CodyCBakerPhD May 29, 2024
7223dd1
Update pipelines.test.ts
garrettmflynn May 29, 2024
fa0ee12
Update testing_pipelines.yml
garrettmflynn May 29, 2024
a3e070f
try 'more' instead of 'type'
CodyCBakerPhD May 29, 2024
61f74cd
Merge branch 'main' into download-gin-data-for-tests
CodyCBakerPhD May 29, 2024
f86a6d5
try using action
CodyCBakerPhD May 29, 2024
ac47699
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 29, 2024
d94dcdb
Update testing_pipelines.yml
CodyCBakerPhD May 29, 2024
7fe5cb1
revert action
CodyCBakerPhD May 29, 2024
6f3e300
Merge branch 'main' into download-gin-data-for-tests
CodyCBakerPhD May 29, 2024
b54c06d
Create test workflow
garrettmflynn May 29, 2024
41f60f2
Merge branch 'main' into download-gin-data-for-tests
CodyCBakerPhD May 29, 2024
69e82fd
Update test_env_creation.yml
garrettmflynn May 29, 2024
f2083c6
Merge branch 'download-gin-data-for-tests' of https://github.com/Neur…
garrettmflynn May 29, 2024
4965cbe
Update deploy_tests_on_pull_request.yml
garrettmflynn May 29, 2024
43851d7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 29, 2024
7cb4587
Update testing_pipelines.yml
garrettmflynn May 29, 2024
ab3162c
Merge branch 'download-gin-data-for-tests' of https://github.com/Neur…
garrettmflynn May 29, 2024
82e85f2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 29, 2024
ce64563
Update to match test workflow exactly
garrettmflynn May 29, 2024
b2503a7
Update test_env_creation.yml
garrettmflynn May 29, 2024
487676c
Update testing_pipelines.yml
garrettmflynn May 29, 2024
0e6a899
Explicitly specify bash shell
garrettmflynn May 29, 2024
3ec5164
Update testing_dev_with_live_services.yml
garrettmflynn May 29, 2024
aee6993
Fix dev tests
garrettmflynn May 29, 2024
d1b86d6
propagate secret
CodyCBakerPhD May 29, 2024
e5ff2c7
Merge branch 'main' into download-gin-data-for-tests
CodyCBakerPhD May 29, 2024
e7714ba
suppress windows on pipeline CI
CodyCBakerPhD May 30, 2024
a631c25
suppress inclusion as well
CodyCBakerPhD May 30, 2024
98d6af2
suppress on dev as well
CodyCBakerPhD May 30, 2024
58de00d
attempt another rename
CodyCBakerPhD May 30, 2024
761de27
suppress inclusion; comment out os
CodyCBakerPhD May 30, 2024
f6593b1
restore os
CodyCBakerPhD May 30, 2024
4ece0ed
try empty os list in dev
CodyCBakerPhD May 30, 2024
f6154e3
fix yaml; try os removal again
CodyCBakerPhD May 30, 2024
67f5803
complete removal
CodyCBakerPhD May 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 0 additions & 19 deletions .github/workflows/codespell.yml

This file was deleted.

36 changes: 36 additions & 0 deletions .github/workflows/daily_tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Daily Tests

on:
workflow_dispatch:
schedule:
- cron: "0 14 * * *" # Daily at 10am EST

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:

testing_dev:
uses: ./.github/workflows/testing_dev.yml
secrets:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

testing_dev_with_live_services:
uses: ./.github/workflows/testing_dev_with_live_services.yml
secrets:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

testing_flask_build_and_dist:
uses: ./.github/workflows/testing_flask_build_and_dist.yml

example_data_cache:
uses: ./.github/workflows/example_data_cache.yml
secrets:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
S3_GIN_BUCKET: ${{ secrets.S3_GIN_BUCKET }}

testing_pipelines:
needs: example_data_cache
uses: ./.github/workflows/testing_pipelines.yml
37 changes: 37 additions & 0 deletions .github/workflows/deploy_tests_on_pull_request.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: Deploy

on:
pull_request:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:

Dev:
uses: ./.github/workflows/testing_dev.yml
secrets:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

Live:
uses: ./.github/workflows/testing_dev_with_live_services.yml
secrets:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

Build:
uses: ./.github/workflows/testing_flask_build_and_dist.yml

Data:
uses: ./.github/workflows/example_data_cache.yml
secrets:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
S3_GIN_BUCKET: ${{ secrets.S3_GIN_BUCKET }}

Temp:
uses: ./.github/workflows/test_env_creation.yml

Pipelines:
needs: Data
uses: ./.github/workflows/testing_pipelines.yml
74 changes: 74 additions & 0 deletions .github/workflows/example_data_cache.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
name: Example data cache
on:
workflow_call:
secrets:
AWS_ACCESS_KEY_ID:
required: true
AWS_SECRET_ACCESS_KEY:
required: true
S3_GIN_BUCKET:
required: true

jobs:

run:
# Will read on PR dashboard as 'Deploy / Data / {os} cache'
# Action dashboard identified by 'Deploy'
# Requirement settings identified as '{os} cache'
name: ${{ matrix.os }} cache
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.12"]
os: [ubuntu-latest, macos-latest, macos-13, windows-latest]

steps:

- name: Get ephy_testing_data current head hash
id: ephys
run: echo "::set-output name=HASH_EPHY_DATASET::$(git ls-remote https://gin.g-node.org/NeuralEnsemble/ephy_testing_data.git HEAD | cut -f1)"
- name: Get cached ephys example data - ${{ steps.ephys.outputs.HASH_EPHY_DATASET }}
uses: actions/cache@v4
id: cache-ephys-datasets
with:
path: ./ephy_testing_data
key: ephys-datasets-${{ matrix.os }}-${{ steps.ephys.outputs.HASH_EPHY_DATASET }}
- name: Get ophys_testing_data current head hash
id: ophys
run: echo "::set-output name=HASH_OPHYS_DATASET::$(git ls-remote https://gin.g-node.org/CatalystNeuro/ophys_testing_data.git HEAD | cut -f1)"
- name: Get cached ophys example data - ${{ steps.ophys.outputs.HASH_OPHYS_DATASET }}
uses: actions/cache@v4
id: cache-ophys-datasets
with:
path: ./ophys_testing_data
key: ophys-datasets-${{ matrix.os }}-${{ steps.ophys.outputs.HASH_OPHYS_DATASET }}
- name: Get behavior_testing_data current head hash
id: behavior
run: echo "::set-output name=HASH_BEHAVIOR_DATASET::$(git ls-remote https://gin.g-node.org/CatalystNeuro/behavior_testing_data.git HEAD | cut -f1)"
- name: Get cached behavior example data - ${{ steps.behavior.outputs.HASH_BEHAVIOR_DATASET }}
uses: actions/cache@v4
id: cache-behavior-datasets
with:
path: ./behavior_testing_data
key: behavior-datasets-${{ matrix.os }}-${{ steps.behavior.outputs.HASH_behavior_DATASET }}

- if: steps.cache-ephys-datasets.outputs.cache-hit != 'true' || steps.cache-ophys-datasets.outputs.cache-hit != 'true' || steps.cache-behavior-datasets.outputs.cache-hit != 'true'
name: Install and configure AWS CLI
run: |
pip install awscli
aws configure set aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID }}
aws configure set aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY }}

- if: steps.cache-ephys-datasets.outputs.cache-hit != 'true'
name: Download ephys dataset from S3
run: |
aws s3 cp --region=us-east-2 ${{ secrets.S3_GIN_BUCKET }}/ephy_testing_data ./ephy_testing_data --recursive
- if: steps.cache-ophys-datasets.outputs.cache-hit != 'true'
name: Download ophys dataset from S3
run: |
aws s3 cp --region=us-east-2 ${{ secrets.S3_GIN_BUCKET }}/ophys_testing_data ./ophys_testing_data --recursive
- if: steps.cache-behavior-datasets.outputs.cache-hit != 'true'
name: Download behavior dataset from S3
run: |
aws s3 cp --region=us-east-2 ${{ secrets.S3_GIN_BUCKET }}/behavior_testing_data ./behavior_testing_data --recursive
66 changes: 66 additions & 0 deletions .github/workflows/test_env_creation.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
name: Test environment creation
on:
workflow_call:

jobs:

run:

name: ${{ matrix.os }} environment test
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash -l {0}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, macos-13, windows-latest]
include:
- os: ubuntu-latest
label: environments/environment-Linux.yml

- os: macos-latest # Mac arm64 runner
label: environments/environment-MAC-apple-silicon.yml

- os: macos-13 # Mac x64 runner
label: environments/environment-MAC-intel.yml

- os: windows-latest
label: environments/environment-Windows.yml


steps:
- name: Checkout code
uses: actions/checkout@v2

- name: Set working directory as environment variable
run: echo "CURRENT_WORKING_DIR=$(pwd)" >> $GITHUB_ENV
if: runner.os != 'Windows'

- name: Set working directory as environment variable (Windows)
run: echo "CURRENT_WORKING_DIR=%cd%" >> $env:GITHUB_ENV
shell: cmd
if: runner.os == 'Windows'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this style, I would recommend this syntax at the top of the file

env:
  CURRENT_WORKING_DIR: ${{ github.workspace }}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh sure. I'm hoping to remove this ASAP.

It shows that we can, in fact, save a .env file and display the value on Windows. The Set working directory as environment variable option was simply to see if we can access something like this cross-platform. Will try this global env variable out!

Just switched the test pipeline workflow to use the exact same configuration for saving the .env file. Should tell us whether we're configuring GitHub Actions wrong or the test pipeline action is somehow not compatible with saving an .env file.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shows that we can, in fact, save a .env file and display the value on Windows. The Set working directory as environment variable option was simply to see if we can access something like this cross-platform. Will try this global env variable out!

I saw that... CI can be really confusing sometimes

One thing it might be is the conda activation step, which swaps to a different shell type (and maybe the .env thing is only available from the default bash??)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep looks like it. Now that we're digging into this, it looks like the Live service tests are also not behaving as expected—possibly as a consequence of these inconsistencies.

In both Dev and Live tests, No DANDI API key provided is printed to the console—despite the latter doing a similar dump to an environment file.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm although is this only happening in this PR?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nevermind, this was just a consequence of not registering the DANDI API Key secret


- name: Set working directory as environment file
run: echo "CURRENT_WORKING_DIR=$(pwd)" >> .env
if: runner.os != 'Windows'

- name: Set working directory as environment file (Windows)
run: echo "CURRENT_WORKING_DIR=%cd%" >> .env
shell: cmd
if: runner.os == 'Windows'


- name: Print working directory
run: echo "The current working directory is $CURRENT_WORKING_DIR"


- name: Print environment file
run: cat .env
if: runner.os != 'Windows'

- name: Print environment file
run: type .env
shell: cmd
if: runner.os == 'Windows'
83 changes: 0 additions & 83 deletions .github/workflows/testing-pipelines.yml

This file was deleted.

28 changes: 12 additions & 16 deletions .github/workflows/testing.yml → .github/workflows/testing_dev.yml
Original file line number Diff line number Diff line change
@@ -1,24 +1,21 @@
name: Dev Tests
name: Dev tests
on:
schedule:
- cron: "0 16 * * *" # Daily at noon EST
pull_request:

concurrency: # Cancel previous workflows on the same pull request
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

env:
CACHE_NUMBER: 2 # increase to reset cache manually
workflow_call:
secrets:
CODECOV_TOKEN:
required: true

jobs:
testing:
name: ${{ matrix.os }} # Will read on the dashboard as 'Dev Tests / {os}'

run:
# Will read on PR dashboard as 'Deploy / Dev / {os} tests'
# Action dashboard identified by 'Deploy'
# Requirement settings identified as '{os} tests'
name: ${{ matrix.os }} tests # Will read on PR dashboard as 'Deploy / Dev / {os} tests'
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash -l {0}

strategy:
fail-fast: false
matrix:
Expand All @@ -36,7 +33,6 @@ jobs:
- os: windows-latest
label: environments/environment-Windows.yml


steps:
- uses: actions/checkout@v4
- run: git fetch --prune --unshallow --tags
Expand Down Expand Up @@ -89,7 +85,7 @@ jobs:
name: test-screenshots-${{ matrix.os }}
path: docs/assets/tutorials
retention-days: 1

overwrite: true


- name: Upload coverage reports to Codecov
Expand Down
Loading
Loading