Skip to content

Commit

Permalink
Run all tests in the nix-shell; eliminate docker infrastructure (#379)
Browse files Browse the repository at this point in the history 
This PR refactors the build infrastructure in this repo to eliminate the need for the Docker component.  All development and testing is now done in the `nix` shell.  This should be a quality of life improvement for anyone developing the fortran model, as it no longer requires maintaining checksums in two separate build environments.

In so doing it introduces the following changes:
- New `make` rules are provided for compiling the model in different modes:
  - `build` -- build executables in `repro` (our production mode) and `debug` mode.
  - `build_repro` -- build only the `repro` mode executable.
  - `build_debug` -- build only the `debug` mode executable.
- Tests are run with each of the executables available in the local `bin` directory, and are tagged with the associated compile mode.  
- An option, `check_layout_invariance`, is provided to trigger regression tests be run with a 1x2 domain decomposition instead of a 1x1 domain decomposition to check invariance to the domain decomposition layout; this is used for the all the coarse-graining regression tests and replaces the previous `test_run_reproduces_across_layouts` test that would run in the docker image.
- `debug`-mode and `repro`-mode simulations produce different answers, which is something we noticed in #364 when upgrading compiler versions as well, and so require different reference checksums.

In working on this PR, we ran the fortran model in `debug` mode in more contexts than we had previously, some of which turned up errors, which we currently work around by using `pytest.skip` (something we had implicitly already been doing before):
- #365
- #381 

Working on this PR also brought my attention to the fact that `pytest`'s `tmpdir` fixture does not automatically get cleaned up after each test; `pytest` versions older than 7.3.0 keep around directories from the last three runs of `pytest`, which fill up disk space quickly since running these tests requires creating 10's of run directories, each with their own initial conditions and input files (#380).  For the time being I manually clean up these run directories after successful tests.

Resolves #340.
  • Loading branch information
@spencerkclark
spencerkclark authored Sep 7, 2023
1 parent b379330 commit adafc50
Show file tree
Hide file tree
Showing 48 changed files with 1,022 additions and 4,472 deletions.
248 changes: 138 additions & 110 deletions .circleci/config.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,7 @@ version: 2.1
orbs:
gcp-gcr: circleci/[email protected]
commands:
nix:
description: "Build fortran and wrapper in nix environment"
initialize_nix:
steps:
- run: nix-env -i git openssh google-cloud-sdk
- checkout
Expand All @@ -16,106 +15,175 @@ commands:
command: |
git submodule init
git submodule update
- restore_cache:
name: Restore nix build cache
keys:
- nix-fv3gfs-fortran-{{ .Environment.CIRCLE_SHA1 }}
- run: |
nix-env -iA cachix -f https://cachix.org/api/v1/install
cachix use vulcanclimatemodeling
- run: |
nix-build -j 4 -A fms | cachix push vulcanclimatemodeling
nix-shell --run "cd FV3 && bash configure nix && make -j 4"
build_repro:
steps:
- restore_cache:
name: Restore nix build cache
keys:
- nix-fv3gfs-fortran-repro-{{ .Environment.CIRCLE_SHA1 }}
- run: |
nix-shell --run "cd FV3 && bash configure nix && cd .. && make build_repro"
- run:
name: Build wrapper
command: nix-shell --run "make -j 4 -C FV3 wrapper_build"
command: nix-shell --run "make build_wrapper"
- save_cache:
name: Save nix build cache
key: nix-fv3gfs-fortran-repro-{{ .Environment.CIRCLE_SHA1 }}
paths:
- FV3/
build_debug:
steps:
- restore_cache:
name: Restore nix build cache
keys:
- nix-fv3gfs-fortran-debug-{{ .Environment.CIRCLE_SHA1 }}
- run: |
nix-shell --run "cd FV3 && bash configure nix && cd .. && make build_debug"
- save_cache:
name: Save nix build cache
key: nix-fv3gfs-fortran-{{ .Environment.CIRCLE_SHA1 }}
key: nix-fv3gfs-fortran-debug-{{ .Environment.CIRCLE_SHA1 }}
paths:
- FV3/
test_fortran_basic:
steps:
- run:
name: Test fortran
command: nix-shell --run "make test_fortran_basic"
test_wrapper_basic:
steps:
- run:
name: Test wrapper
command: nix-shell --run "make -C FV3/wrapper test_basic"
test_fortran_coarse_graining:
steps:
- run:
name: Test fortran
command: nix-shell --run "make test_fortran_coarse_graining"
test_wrapper_coarse_graining:
steps:
- run:
name: Test wrapper
command: nix-shell --run "make -C FV3/wrapper test_coarse_graining"

jobs:
nix-full:
nix-full-repro:
docker:
- image: nixos/nix
environment:
FV3CONFIG_CACHE_DIR: /tmp/.fv3config
GOOGLE_APPLICATION_CREDENTIALS: /tmp/key.json
FSSPEC_GS_REQUESTER_PAYS: vcm-ml
steps:
- nix
- initialize_nix
- build_repro
- run:
name: Test fortran
command: nix-shell --run "make test_native_fortran"
command: nix-shell --run "make test_fortran"
- run:
name: Test wrapper
command: nix-shell --run "make -C FV3/wrapper test"
nix-basic:
nix-full-debug:
docker:
- image: nixos/nix
environment:
FV3CONFIG_CACHE_DIR: /tmp/.fv3config
GOOGLE_APPLICATION_CREDENTIALS: /tmp/key.json
FSSPEC_GS_REQUESTER_PAYS: vcm-ml
steps:
- nix
- initialize_nix
- build_debug
- run:
name: Test fortran
command: nix-shell --run "make test_native_fortran_basic"
- run:
name: Test wrapper
command: nix-shell --run "make -C FV3/wrapper test_basic"
nix-coarse-graining:
command: nix-shell --run "make test_fortran"
nix-basic-repro:
docker:
- image: nixos/nix
environment:
FV3CONFIG_CACHE_DIR: /tmp/.fv3config
GOOGLE_APPLICATION_CREDENTIALS: /tmp/key.json
FSSPEC_GS_REQUESTER_PAYS: vcm-ml
steps:
- nix
- run:
name: Test fortran
command: nix-shell --run "make test_native_fortran_coarse_graining"
- run:
name: Test wrapper
command: nix-shell --run "make -C FV3/wrapper test_coarse_graining"
nix-emulation:
- initialize_nix
- build_repro
- test_fortran_basic
- test_wrapper_basic
nix-basic-debug:
docker:
- image: nixos/nix
environment:
FV3CONFIG_CACHE_DIR: /tmp/.fv3config
GOOGLE_APPLICATION_CREDENTIALS: /tmp/key.json
FSSPEC_GS_REQUESTER_PAYS: vcm-ml
steps:
- initialize_nix
- build_debug
- test_fortran_basic
nix-coarse-graining-repro:
docker:
- image: nixos/nix
environment:
FV3CONFIG_CACHE_DIR: /tmp/.fv3config
GOOGLE_APPLICATION_CREDENTIALS: /tmp/key.json
FSSPEC_GS_REQUESTER_PAYS: vcm-ml
steps:
- initialize_nix
- build_repro
- test_fortran_coarse_graining
- test_wrapper_coarse_graining
nix-coarse-graining-debug:
docker:
- image: nixos/nix
environment:
FV3CONFIG_CACHE_DIR: /tmp/.fv3config
GOOGLE_APPLICATION_CREDENTIALS: /tmp/key.json
FSSPEC_GS_REQUESTER_PAYS: vcm-ml
steps:
- initialize_nix
- build_debug
- test_fortran_coarse_graining
nix-emulation-repro:
docker:
- image: nixos/nix
environment:
FV3CONFIG_CACHE_DIR: /tmp/.fv3config
GOOGLE_APPLICATION_CREDENTIALS: /tmp/key.json
FSSPEC_GS_REQUESTER_PAYS: vcm-ml
steps:
- nix
- initialize_nix
- build_repro
- run:
name: Test fortran
command: nix-shell --run "make test_native_fortran_emulation"
command: nix-shell --run "make test_fortran_emulation"
- run:
name: Test wrapper
command: nix-shell --run "make -C FV3/wrapper test_emulation"
nix-unmarked:
nix-unmarked-repro:
docker:
- image: nixos/nix
environment:
FV3CONFIG_CACHE_DIR: /tmp/.fv3config
GOOGLE_APPLICATION_CREDENTIALS: /tmp/key.json
FSSPEC_GS_REQUESTER_PAYS: vcm-ml
steps:
- nix
- initialize_nix
- build_repro
- run:
name: Test fortran
command: nix-shell --run "make test_native_fortran_unmarked"
nix-wrapper:
command: nix-shell --run "make test_fortran_unmarked"
nix-wrapper-repro:
docker:
- image: nixos/nix
environment:
FV3CONFIG_CACHE_DIR: /tmp/.fv3config
GOOGLE_APPLICATION_CREDENTIALS: /tmp/key.json
FSSPEC_GS_REQUESTER_PAYS: vcm-ml
steps:
- nix
- initialize_nix
- build_repro
- run:
name: Test wrapper
command: nix-shell --run "make -C FV3/wrapper test"
Expand All @@ -126,71 +194,19 @@ jobs:
- checkout
- run: sudo pip3 install pre-commit==2.15.0
- run: pre-commit
build_default:
machine:
docker_layer_caching: true
image: ubuntu-2004:202111-02
environment:
FV3CONFIG_CACHE_DIR: /tmp/.fv3config
GOOGLE_APPLICATION_CREDENTIALS: /tmp/key.json
FSSPEC_GS_REQUESTER_PAYS: vcm-ml
steps:
- checkout
- run:
name: "gcloud auth"
command: |
echo $ENCODED_GCR_KEY | base64 -d > $GOOGLE_APPLICATION_CREDENTIALS
echo "export GCLOUD_SERVICE_KEY=\$(echo \$ENCODED_GCR_KEY | base64 --decode)" >> $BASH_ENV
- gcp-gcr/gcr-auth
- run:
name: "Pull dependency images"
command: |
echo "$ENCODED_GCR_KEY" | base64 --decode | docker login --username _json_key --password-stdin https://gcr.io
make pull_deps
- run:
name: "Pull Submodules"
command: |
git submodule init
git submodule update
- run: DOCKER_BUILDKIT=1 BUILDKIT_PROGRESS=plain BUILD_FROM_INTERMEDIATE=y make build build_debug build_serialize
- restore_cache:
keys:
- v1.4-{{ checksum "requirements.txt"}}
- run:
name: "Install Python test dependencies"
command: |
pyenv versions
[ -d /opt/circleci/.pyenv/versions/3.9.7 ] || pyenv install 3.9.7
pyenv global 3.9.7
pip install -r requirements.txt
- save_cache:
paths:
- $FV3CONFIG_CACHE_DIR
- /opt/circleci/.pyenv/versions/3.9.7
key: v1.4-{{ checksum "requirements.txt"}}
# tests
- run:
name: "Run pytest tests"
# -s to avoid capturing stdout/stderr of md5sum call
command: |
make test
no_output_timeout: 1200
- run: make push_deps
- run:
name: "Delete data files"
command: |
rm ./tests/pytest/output/*/*/INPUT/*.nc
rm ./tests/pytest/output/*/*/*/*.grb
when: on_fail
- store_artifacts:
path: ./tests/pytest/output

workflows:
version: 2
build:
jobs:
- lint
- nix-basic:
name: Minimal native fortran and wrapper tests
- nix-basic-repro:
name: Minimal fortran and wrapper tests in repro mode
filters:
branches:
ignore: master
- nix-basic-debug:
name: Minimal fortran tests in debug mode
filters:
branches:
ignore: master
Expand All @@ -200,59 +216,71 @@ workflows:
filters:
branches:
ignore: master
- nix-coarse-graining:
name: Coarse graining tests
- nix-coarse-graining-repro:
name: Coarse graining tests in repro mode
requires:
- Minimal fortran and wrapper tests in repro mode
- Launch coarse graining tests
filters:
branches:
ignore: master
- nix-coarse-graining-debug:
name: Coarse graining tests in debug mode
requires:
- Minimal native fortran and wrapper tests
- Minimal fortran tests in debug mode
- Launch coarse graining tests
filters:
branches:
ignore: master
- hold-nix-emulation:
- hold-nix-emulation-repro:
name: Launch emulation tests
type: approval
filters:
branches:
ignore: master
- nix-emulation:
- nix-emulation-repro:
name: Emulation tests
requires:
- Minimal native fortran and wrapper tests
- Minimal fortran and wrapper tests in repro mode
- Launch emulation tests
filters:
branches:
ignore: master
- hold-nix-unmarked:
- hold-nix-unmarked-repro:
name: Launch unmarked fortran tests
type: approval
filters:
branches:
ignore: master
- nix-unmarked:
- nix-unmarked-repro:
name: Unmarked fortran tests
requires:
- Minimal native fortran and wrapper tests
- Minimal fortran and wrapper tests in repro mode
- Launch unmarked fortran tests
filters:
branches:
ignore: master
- hold-nix-wrapper:
- hold-nix-wrapper-repro:
name: Launch wrapper tests
type: approval
filters:
branches:
ignore: master
- nix-wrapper:
- nix-wrapper-repro:
name: Wrapper tests
requires:
- Minimal native fortran and wrapper tests
- Minimal fortran and wrapper tests in repro mode
- Launch wrapper tests
filters:
branches:
ignore: master
- build_default
- nix-full:
name: "Full native tests"
- nix-full-repro:
name: "Full tests in repro mode"
filters:
branches:
only: master
- nix-full-debug:
name: "Full tests in debug mode"
filters:
branches:
only: master
7 changes: 0 additions & 7 deletions .dockerignore
View file

This file was deleted.

Loading

0 comments on commit adafc50

Please sign in to comment.