Update CI driver #1401

DavidNew-NOAA · 2024-12-05T13:59:42Z

Description

This PR consolidates the run_ci.sh and run_gw_ci.sh as well as the driver.sh and gw_driver.sh scripts. Now, a -w option indicates whether testing is taking place in the Global Workflow.

A -E ci_test1|ci_test2|... is added to driver.sh to exclude CI tests which don't need to be run. Those options are passed directly to the ctest command. The PR testing functionality in driver.sh knows which tests to exclude based on the check boxes below. The output from the EMC bot for PR testing will now show the exact CTest command to indicate which tests were indeed excluded, just in case the check boxes are changed after the CI testing has begun.

A PR template is now added for GDASApp that includes the CI testing checkboxes.

You can see below that I've tested the PR testing functionality.

Companion PRs

Issues

Automated CI tests to run in Global Workflow

emcbot · 2024-12-05T15:09:45Z

Automated GDASApp Testing Results:
Machine: hera

Start: Thu Dec  5 14:21:05 UTC 2024 on hfe10
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu Dec  5 15:09:11 UTC 2024
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Thu Dec  5 15:09:44 UTC 2024
Tests: 100% tests passed, 0 tests failed out of 31

emcbot · 2024-12-05T16:16:01Z

Automated GDASApp Testing Results:
Machine: hera

Start: Thu Dec  5 14:24:11 UTC 2024 on hfe10
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu Dec  5 15:13:26 UTC 2024
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E C384mx025_3DVarAOWCDA|C96C48_hybatmDA
Tests:                                 *SUCCESS*
Tests: Completed at Thu Dec  5 16:16:01 UTC 2024
Tests: 83% tests passed, 19 tests failed out of 115

emcbot · 2024-12-12T17:14:09Z

Automated GDASApp Testing Results:
Machine: orion

Start: Thu Dec 12 10:19:34 AM CST 2024 on orion-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu Dec 12 11:13:16 AM CST 2024
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                  *Failed*
Tests: Failed at Thu Dec 12 11:14:08 AM CST 2024
Tests: 94% tests passed, 2 tests failed out of 31
	1952 - test_gdasapp_snow_apply_jediincr (Failed)
	1953 - test_gdasapp_snow_letkfoi_snowda (Failed)
Tests: see output at /work2/noaa/da/role-da/CI/orion/GDASApp/PR/1401/GDASApp/build/log.ctest

emcbot · 2024-12-12T19:00:59Z

Automated GDASApp Testing Results:
Machine: orion

Start: Thu Dec 12 12:05:04 PM CST 2024 on orion-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu Dec 12 12:59:57 PM CST 2024
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                  *Failed*
Tests: Failed at Thu Dec 12 01:00:58 PM CST 2024
Tests: 94% tests passed, 2 tests failed out of 31
	1952 - test_gdasapp_snow_apply_jediincr (Failed)
	1953 - test_gdasapp_snow_letkfoi_snowda (Failed)
Tests: see output at /work2/noaa/da/role-da/CI/orion/GDASApp/PR/1401/GDASApp/build/log.ctest

RussTreadon-NOAA · 2024-12-12T19:03:50Z

@DavidNew-NOAA , I am looking into the Orion failures from the role-da account.

DavidNew-NOAA · 2024-12-12T20:02:50Z

@RussTreadon-NOAA Thanks, yeah I'm a bit perplexed since Cory was able to run these tests successfully on Orion #1406 (comment)

RussTreadon-NOAA · 2024-12-12T20:23:36Z

According to groups role-da, this account does not have access to any project codes

orion-login-4:~$ groups role-da
role-da : noaa-hpc rstprod da
orion-login-4:~$

Cory, you, and I can access da-cpu. Cory and I can also use accounting code fv3-cpu

test/snow/apply_jedi_incr.sh contains

 # (n=6) -> this is fixed, at one task per tile (with minor code change, could run on a single proc).
srun '--export=ALL' -n 6 ${EXECDIR}/apply_incr.exe ${WORKDIR}/apply_incr.log

test/snow/letkfoi_snowda.sh contains

srun '--export=ALL' -n 6 ${EXECDIR}/${JEDI_EXEC} letkf_snow.yaml

I added --account=da-cpu --qos=batch to the srun lines above and the ctests still failed. That's when I realized that role-da can not use da-cpu.

Let me ask Daryl if he can add da-cpu to role-da.

RussTreadon-NOAA · 2024-12-12T21:35:39Z

Line 116 of ci/stable_driver.sh is the caution line below

  git diff-index --quiet HEAD || git commit -m "Update to new stable build on $datestr"
  total=$(($total+$?))
  caution=""
  if [ $total -ne 0 ]; then
    echo "Unable to commit" >> $stableroot/$datestr/output
  fi

Variable caution is not used in stable_driver.sh. This PR did not add caution. I think it's safe for this PR to remove the caution line.

RussTreadon-NOAA · 2024-12-13T03:03:02Z

Configure local copy of GDASApp feature/update-ci at c74dd4f to run stable_driver.sh using g-w develop at 0c7a3c6c. The following JEDI repos were updated in gdas.cd/sorc/

diff --git a/sorc/fv3-jedi b/sorc/fv3-jedi
index 8ab22d5..136dfb9 160000
--- a/sorc/fv3-jedi
+++ b/sorc/fv3-jedi
@@ -1 +1 @@
-Subproject commit 8ab22d520e6145d4d2f993c3eea3f6a37ea3f5bc
+Subproject commit 136dfb9c2f8541584e62fa74c616d686602bcdad
diff --git a/sorc/ioda b/sorc/ioda
index f16b50c..22cd20e 160000
--- a/sorc/ioda
+++ b/sorc/ioda
@@ -1 +1 @@
-Subproject commit f16b50c0552b7a08f8f93b83687d1750e67d5821
+Subproject commit 22cd20eae0685914a5b967e13f95779b57bb448c
diff --git a/sorc/iodaconv b/sorc/iodaconv
index a2c29b8..6f87a0f 160000
--- a/sorc/iodaconv
+++ b/sorc/iodaconv
@@ -1 +1 @@
-Subproject commit a2c29b867443182ce24360d4ca53a6f56af55faa
+Subproject commit 6f87a0f279e836fd604e5b313a25bd1e54bff80e
diff --git a/sorc/oops b/sorc/oops
index 814c994..1ba321f 160000
--- a/sorc/oops
+++ b/sorc/oops
@@ -1 +1 @@
-Subproject commit 814c994f3aa58cacc3eea0826485a4fece7a3bc1
+Subproject commit 1ba321ff912c6338d7362667eff37ddbf569cb18
diff --git a/sorc/saber b/sorc/saber
index 957e7d4..de5015c 160000
--- a/sorc/saber
+++ b/sorc/saber
@@ -1 +1 @@
-Subproject commit 957e7d45c20553704230361f590abd46c3a4e932
+Subproject commit de5015c8328f5b3d64acc99739fbaa64ef571172
diff --git a/sorc/soca b/sorc/soca
index 108d322..44c4d6e 160000
--- a/sorc/soca
+++ b/sorc/soca
@@ -1 +1 @@
-Subproject commit 108d3220fecd6a7025db71b77992ac9be810b29a
+Subproject commit 44c4d6e53efa774bbaa038b0ecc4c34e3a33c9df
diff --git a/sorc/ufo b/sorc/ufo
index af545b5..85ef98c 160000
--- a/sorc/ufo
+++ b/sorc/ufo
@@ -1 +1 @@
-Subproject commit af545b53851254a11c4a9730e8284b316438f16d
+Subproject commit 85ef98cb99f3eae1ed15e39f0cb05046af36fef1
diff --git a/sorc/vader b/sorc/vader
index 0792f69..3049658 160000
--- a/sorc/vader
+++ b/sorc/vader
@@ -1 +1 @@
-Subproject commit 0792f693148c3d09a166021e6b8c2758cb8a1251
+Subproject commit 3049658d185c8095caf0e506795d0e5995fa92cf

To this the following patches were added

        modified:   parm/aero/berror/aero_diagb.yaml.j2
        modified:   parm/io/fv3jedi_fieldmetadata_history.yaml
        modified:   parm/jcb-gdas (modified content)
        modified:   test/testreference/C96C48_ufs_hybatmDA_3dvar.ref
        modified:   test/testreference/C96C48_ufs_hybatmDA_lgetkf_observer.ref
        modified:   test/testreference/C96C48_ufs_hybatmDA_lgetkf_solver.ref
        modified:   test/testreference/atm_jjob_3dvar.ref
        modified:   test/testreference/atm_jjob_lgetkf.ref
        modified:   test/testreference/atm_jjob_lgetkf_observer.ref
        modified:   test/testreference/atm_jjob_lgetkf_solver.ref

With these changes all ctests pass

Test project /scratch1/NCEPDEV/da/Russ.Treadon/CI/GDASApp/test/stable/20241213/global-workflow/sorc/gdas.cd/build
        Start 1988: test_gdasapp_C96C48_ufs_hybatmDA
        Start 1969: test_gdasapp_C96C48_hybatmDA
        Start 2013: test_gdasapp_C96C48_hybatmaerosnowDA
        Start 2038: test_gdasapp_C48mx500_3DVarAOWCDA
        Start 2049: test_gdasapp_setup_atm_jjob_cycled_exp
        Start 1591: test_gdasapp_util_prepdata
        Start 1589: test_gdasapp_util_coding_norms
        Start 1590: test_gdasapp_util_ioda_example
        Start 1594: test_gdasapp_util_rtofstmp
        Start 1595: test_gdasapp_util_rtofssal
        Start 1962: test_gdasapp_check_python_norms
        Start 1963: test_gdasapp_check_yaml_keys
  1/122 Test #1963: test_gdasapp_check_yaml_keys .............................................   Passed    0.32 sec
        Start 1964: test_gdasapp_snow_create_ens
  2/122 Test #1590: test_gdasapp_util_ioda_example ...........................................   Passed    0.80 sec
        Start 1965: test_gdasapp_snow_imsproc
  3/122 Test #2049: test_gdasapp_setup_atm_jjob_cycled_exp ...................................   Passed    1.39 sec
        Start 2054: test_gdasapp_atm_jjob_ens_init
  4/122 Test #1595: test_gdasapp_util_rtofssal ...............................................   Passed    1.49 sec
        Start 2050: test_gdasapp_atm_jjob_var_init
  5/122 Test #1594: test_gdasapp_util_rtofstmp ...............................................   Passed    1.77 sec
        Start 1966: test_gdasapp_snow_apply_jediincr
  6/122 Test #1964: test_gdasapp_snow_create_ens .............................................   Passed    1.99 sec
        Start 1967: test_gdasapp_snow_letkfoi_snowda
  7/122 Test #1589: test_gdasapp_util_coding_norms ...........................................   Passed    3.45 sec
        Start 1968: test_gdasapp_convert_bufr_adpsfc_snow
  8/122 Test #1591: test_gdasapp_util_prepdata ...............................................   Passed    3.55 sec
        Start 1592: test_gdasapp_util_rads2ioda
  9/122 Test #1592: test_gdasapp_util_rads2ioda ..............................................   Passed    0.19 sec
        Start 1593: test_gdasapp_util_ghrsst2ioda
 10/122 Test #1962: test_gdasapp_check_python_norms ..........................................   Passed    3.66 sec
        Start 1596: test_gdasapp_util_smap2ioda
 11/122 Test #1593: test_gdasapp_util_ghrsst2ioda ............................................   Passed    0.19 sec
        Start 1597: test_gdasapp_util_smos2ioda
 12/122 Test #1596: test_gdasapp_util_smap2ioda ..............................................   Passed    0.18 sec
        Start 1598: test_gdasapp_util_viirsaod2ioda
 13/122 Test #1597: test_gdasapp_util_smos2ioda ..............................................   Passed    0.21 sec
        Start 1599: test_gdasapp_util_icecabi2ioda
 14/122 Test #1598: test_gdasapp_util_viirsaod2ioda ..........................................   Passed    0.22 sec
        Start 1600: test_gdasapp_util_icecamsr2ioda
 15/122 Test #1599: test_gdasapp_util_icecabi2ioda ...........................................   Passed    0.22 sec
        Start 1601: test_gdasapp_util_icecmirs2ioda
 16/122 Test #1600: test_gdasapp_util_icecamsr2ioda ..........................................   Passed    0.22 sec
        Start 1602: test_gdasapp_util_icecjpssrr2ioda
 17/122 Test #1601: test_gdasapp_util_icecmirs2ioda ..........................................   Passed    0.24 sec
        Start 2047: test_gdasapp_convert_bufr_adpsfc
 18/122 Test #1602: test_gdasapp_util_icecjpssrr2ioda ........................................   Passed    0.24 sec
        Start 2048: test_gdasapp_convert_gsi_satbias
 19/122 Test #1965: test_gdasapp_snow_imsproc ................................................   Passed    3.97 sec
        Start 2061: test_gdasapp_aero_gen_3dvar_yaml
 20/122 Test #2061: test_gdasapp_aero_gen_3dvar_yaml .........................................   Passed    0.76 sec
        Start 2062: test_gdasapp_bufr2ioda_insitu_profile_argo
 21/122 Test #1966: test_gdasapp_snow_apply_jediincr .........................................   Passed    3.96 sec
        Start 2063: test_gdasapp_bufr2ioda_insitu_profile_bathy
 22/122 Test #2038: test_gdasapp_C48mx500_3DVarAOWCDA ........................................   Passed    6.41 sec
        Start 2039: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_stage_ic_202103241800
 23/122 Test #1968: test_gdasapp_convert_bufr_adpsfc_snow ....................................   Passed    3.53 sec
        Start 2064: test_gdasapp_bufr2ioda_insitu_profile_glider
 24/122 Test #2063: test_gdasapp_bufr2ioda_insitu_profile_bathy ..............................   Passed    2.10 sec
        Start 2065: test_gdasapp_bufr2ioda_insitu_profile_tesac
 25/122 Test #2047: test_gdasapp_convert_bufr_adpsfc .........................................   Passed    4.40 sec
        Start 2066: test_gdasapp_bufr2ioda_insitu_profile_tropical
 26/122 Test #2064: test_gdasapp_bufr2ioda_insitu_profile_glider .............................   Passed    2.86 sec
        Start 2067: test_gdasapp_bufr2ioda_insitu_profile_xbtctd
 27/122 Test #1969: test_gdasapp_C96C48_hybatmDA .............................................   Passed   10.57 sec
        Start 1970: test_gdasapp_C96C48_hybatmDA_gdas_stage_ic_202112201800
 28/122 Test #2048: test_gdasapp_convert_gsi_satbias .........................................   Passed    5.95 sec
        Start 1973: test_gdasapp_C96C48_hybatmDA_enkfgdas_stage_ic_202112201800
 29/122 Test #1988: test_gdasapp_C96C48_ufs_hybatmDA .........................................   Passed   10.67 sec
        Start 1989: test_gdasapp_C96C48_ufs_hybatmDA_gdas_stage_ic_202402231800
 30/122 Test #2066: test_gdasapp_bufr2ioda_insitu_profile_tropical ...........................   Passed    1.75 sec
        Start 1992: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_stage_ic_202402231800
 31/122 Test #2067: test_gdasapp_bufr2ioda_insitu_profile_xbtctd .............................   Passed    1.37 sec
        Start 2068: test_gdasapp_bufr2ioda_insitu_surface_drifter
 32/122 Test #2013: test_gdasapp_C96C48_hybatmaerosnowDA .....................................   Passed   11.44 sec
        Start 2014: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_stage_ic_202112201200
 33/122 Test #2068: test_gdasapp_bufr2ioda_insitu_surface_drifter ............................   Passed    1.41 sec
        Start 2018: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_stage_ic_202112201200
 34/122 Test #2062: test_gdasapp_bufr2ioda_insitu_profile_argo ...............................   Passed    7.56 sec
        Start 2069: test_gdasapp_bufr2ioda_insitu_surface_trkob
 35/122 Test #2065: test_gdasapp_bufr2ioda_insitu_profile_tesac ..............................   Passed    6.02 sec
 36/122 Test #2069: test_gdasapp_bufr2ioda_insitu_surface_trkob ..............................   Passed    1.32 sec
 37/122 Test #1967: test_gdasapp_snow_letkfoi_snowda .........................................   Passed   20.56 sec
 38/122 Test #1973: test_gdasapp_C96C48_hybatmDA_enkfgdas_stage_ic_202112201800 ..............   Passed   38.96 sec
        Start 1974: test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112201800
 39/122 Test #1989: test_gdasapp_C96C48_ufs_hybatmDA_gdas_stage_ic_202402231800 ..............   Passed   39.68 sec
        Start 1990: test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402231800
 40/122 Test #1970: test_gdasapp_C96C48_hybatmDA_gdas_stage_ic_202112201800 ..................   Passed   39.82 sec
        Start 1971: test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112201800
 41/122 Test #2018: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_stage_ic_202112201200 ......   Passed   38.97 sec
        Start 2019: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_fcst_202112201200
 42/122 Test #2039: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_stage_ic_202103241800 .............   Passed   52.70 sec
        Start 2040: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_fcst_202103241800
 43/122 Test #1992: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_stage_ic_202402231800 ..........   Passed   53.37 sec
        Start 1993: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402231800
 44/122 Test #2014: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_stage_ic_202112201200 ..........   Passed   53.79 sec
        Start 2015: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_fcst_202112201200
 45/122 Test #2054: test_gdasapp_atm_jjob_ens_init ...........................................   Passed   83.71 sec
        Start 2055: test_gdasapp_atm_jjob_ens_letkf
 46/122 Test #2050: test_gdasapp_atm_jjob_var_init ...........................................   Passed   84.09 sec
        Start 2051: test_gdasapp_atm_jjob_var_run
 47/122 Test #2051: test_gdasapp_atm_jjob_var_run ............................................   Passed  106.27 sec
        Start 2052: test_gdasapp_atm_jjob_var_inc
 48/122 Test #2052: test_gdasapp_atm_jjob_var_inc ............................................   Passed   74.27 sec
        Start 2053: test_gdasapp_atm_jjob_var_final
 49/122 Test #2053: test_gdasapp_atm_jjob_var_final ..........................................   Passed   74.33 sec
 50/122 Test #2019: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_fcst_202112201200 ..........   Passed  350.01 sec
        Start 2021: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_epmn_202112201200
 51/122 Test #1974: test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112201800 ..................   Passed  354.42 sec
        Start 1976: test_gdasapp_C96C48_hybatmDA_enkfgdas_epmn_202112201800
 52/122 Test #1993: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402231800 ..............   Passed  388.96 sec
        Start 1995: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_epmn_202402231800
 53/122 Test #2040: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_fcst_202103241800 .................   Passed  427.31 sec
        Start 2041: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_prepoceanobs_202103250000
        Start 2042: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marinebmat_202103250000
 54/122 Test #1990: test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402231800 ..................   Passed  450.79 sec
        Start 1991: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmos_prod_202402231800
        Start 1994: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_echgres_202402231800
 55/122 Test #1971: test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112201800 ......................   Passed  452.83 sec
        Start 1972: test_gdasapp_C96C48_hybatmDA_gdas_atmos_prod_202112201800
        Start 1975: test_gdasapp_C96C48_hybatmDA_enkfgdas_echgres_202112201800
 56/122 Test #2055: test_gdasapp_atm_jjob_ens_letkf ..........................................   Passed  458.36 sec
        Start 2056: test_gdasapp_atm_jjob_ens_init_split
 57/122 Test #2056: test_gdasapp_atm_jjob_ens_init_split .....................................   Passed   80.12 sec
        Start 2057: test_gdasapp_atm_jjob_ens_obs
 58/122 Test #2057: test_gdasapp_atm_jjob_ens_obs ............................................   Passed   74.34 sec
        Start 2058: test_gdasapp_atm_jjob_ens_sol
 59/122 Test #2015: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_fcst_202112201200 ..............   Passed  647.65 sec
        Start 2016: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_atmos_prod_202112201200
        Start 2020: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_echgres_202112201200
 60/122 Test #2058: test_gdasapp_atm_jjob_ens_sol ............................................   Passed   42.35 sec
        Start 2059: test_gdasapp_atm_jjob_ens_inc
 61/122 Test #1975: test_gdasapp_C96C48_hybatmDA_enkfgdas_echgres_202112201800 ...............   Passed  246.74 sec
        Start 2017: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlgenb_202112201200
 62/122 Test #1994: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_echgres_202402231800 ...........   Passed  260.73 sec
 63/122 Test #1976: test_gdasapp_C96C48_hybatmDA_enkfgdas_epmn_202112201800 ..................   Passed  373.43 sec
 64/122 Test #2021: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_epmn_202112201200 ..........   Passed  376.18 sec
 65/122 Test #2020: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_echgres_202112201200 .......   Passed   97.22 sec
 66/122 Test #1995: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_epmn_202402231800 ..............   Passed  375.25 sec
 67/122 Test #2042: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marinebmat_202103250000 ...........   Passed  372.90 sec
 68/122 Test #2017: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlgenb_202112201200 .......   Passed  126.70 sec
 69/122 Test #2059: test_gdasapp_atm_jjob_ens_inc ............................................   Passed  138.27 sec
        Start 2060: test_gdasapp_atm_jjob_ens_final
 70/122 Test #1991: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmos_prod_202402231800 ............   Passed  399.17 sec
        Start 1996: test_gdasapp_C96C48_ufs_hybatmDA_gdas_prep_202402240000
 71/122 Test #1972: test_gdasapp_C96C48_hybatmDA_gdas_atmos_prod_202112201800 ................   Passed  399.58 sec
        Start 1977: test_gdasapp_C96C48_hybatmDA_gdas_prep_202112210000
 72/122 Test #2060: test_gdasapp_atm_jjob_ens_final ..........................................   Passed   42.24 sec
 73/122 Test #2016: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_atmos_prod_202112201200 ........   Passed  254.87 sec
        Start 2022: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_prep_202112201800
 74/122 Test #2041: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_prepoceanobs_202103250000 .........   Passed  593.54 sec
        Start 2043: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlinit_202103250000
 75/122 Test #2043: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlinit_202103250000 ........   Passed   63.42 sec
        Start 2044: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000
 76/122 Test #2044: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000 .........   Passed   91.33 sec
        Start 2045: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000
 77/122 Test #2045: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000 .......   Passed   86.99 sec
        Start 2046: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000
 78/122 Test #2046: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 .......   Passed  119.71 sec
 79/122 Test #1977: test_gdasapp_C96C48_hybatmDA_gdas_prep_202112210000 ......................   Passed  607.78 sec
        Start 1978: test_gdasapp_C96C48_hybatmDA_gdas_anal_202112210000
        Start 1982: test_gdasapp_C96C48_hybatmDA_enkfgdas_eobs_202112210000
 80/122 Test #1996: test_gdasapp_C96C48_ufs_hybatmDA_gdas_prep_202402240000 ..................   Passed  627.29 sec
        Start 1997: test_gdasapp_C96C48_ufs_hybatmDA_gdas_prepatmiodaobs_202402240000
 81/122 Test #2022: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_prep_202112201800 ..............   Passed  767.36 sec
        Start 2023: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_anal_202112201800
        Start 2031: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_eobs_202112201800
        Start 2024: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlinit_202112201800
        Start 2027: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_snowanl_202112201800
 82/122 Test #2027: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_snowanl_202112201800 ...........   Passed  106.39 sec
        Start 2035: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_esnowrecen_202112201800
 83/122 Test #2024: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlinit_202112201800 .......   Passed  112.56 sec
        Start 2025: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlvar_202112201800
 84/122 Test #2035: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_esnowrecen_202112201800 ....   Passed   79.24 sec
 85/122 Test #1982: test_gdasapp_C96C48_hybatmDA_enkfgdas_eobs_202112210000 ..................   Passed  472.83 sec
        Start 1983: test_gdasapp_C96C48_hybatmDA_enkfgdas_ediag_202112210000
 86/122 Test #1997: test_gdasapp_C96C48_ufs_hybatmDA_gdas_prepatmiodaobs_202402240000 ........   Passed  485.95 sec
        Start 1998: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlinit_202402240000
        Start 2005: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlinit_202402240000
 87/122 Test #2025: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlvar_202112201800 ........   Passed  270.73 sec
        Start 2026: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlfinal_202112201800
 88/122 Test #1983: test_gdasapp_C96C48_hybatmDA_enkfgdas_ediag_202112210000 .................   Passed  166.06 sec
        Start 1984: test_gdasapp_C96C48_hybatmDA_enkfgdas_eupd_202112210000
 89/122 Test #1998: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlinit_202402240000 ............   Passed  165.88 sec
        Start 1999: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlvar_202402240000
 90/122 Test #2005: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlinit_202402240000 .....   Passed  169.85 sec
        Start 2006: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlobs_202402240000
 91/122 Test #2026: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlfinal_202112201800 ......   Passed   79.83 sec
 92/122 Test #2031: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_eobs_202112201800 ..........   Passed  486.28 sec
        Start 2032: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_ediag_202112201800
 93/122 Test #1978: test_gdasapp_C96C48_hybatmDA_gdas_anal_202112210000 ......................   Passed  863.53 sec
        Start 1979: test_gdasapp_C96C48_hybatmDA_gdas_sfcanl_202112210000
 94/122 Test #2032: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_ediag_202112201800 .........   Passed  205.80 sec
        Start 2033: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_eupd_202112201800
 95/122 Test #1984: test_gdasapp_C96C48_hybatmDA_enkfgdas_eupd_202112210000 ..................   Passed  281.26 sec
 96/122 Test #1979: test_gdasapp_C96C48_hybatmDA_gdas_sfcanl_202112210000 ....................   Passed   79.45 sec
        Start 1980: test_gdasapp_C96C48_hybatmDA_gdas_analcalc_202112210000
        Start 1981: test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112210000
 97/122 Test #2023: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_anal_202112201800 ..............   Passed  750.16 sec
        Start 2028: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_sfcanl_202112201800
 98/122 Test #1980: test_gdasapp_C96C48_hybatmDA_gdas_analcalc_202112210000 ..................   Passed  203.42 sec
        Start 1985: test_gdasapp_C96C48_hybatmDA_enkfgdas_ecmn_202112210000
        Start 1986: test_gdasapp_C96C48_hybatmDA_enkfgdas_esfc_202112210000
 99/122 Test #2033: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_eupd_202112201800 ..........   Passed  233.15 sec
100/122 Test #2028: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_sfcanl_202112201800 ............   Passed  218.66 sec
        Start 2029: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_analcalc_202112201800
        Start 2030: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_fcst_202112201800
101/122 Test #1985: test_gdasapp_C96C48_hybatmDA_enkfgdas_ecmn_202112210000 ..................   Passed  108.59 sec
102/122 Test #1986: test_gdasapp_C96C48_hybatmDA_enkfgdas_esfc_202112210000 ..................   Passed  157.93 sec
        Start 1987: test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112210000
103/122 Test #2029: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_analcalc_202112201800 ..........   Passed  208.65 sec
        Start 2034: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_ecmn_202112201800
        Start 2036: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_esfc_202112201800
104/122 Test #1981: test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112210000 ......................   Passed  591.85 sec
105/122 Test #2036: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_esfc_202112201800 ..........   Passed  153.04 sec
106/122 Test #2034: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_ecmn_202112201800 ..........   Passed  156.76 sec
        Start 2037: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_fcst_202112201800
107/122 Test #1987: test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112210000 ..................   Passed  589.54 sec
108/122 Test #2006: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlobs_202402240000 ......   Passed  1257.18 sec
        Start 2007: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlsol_202402240000
109/122 Test #2037: test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_fcst_202112201800 ..........   Passed  415.81 sec
110/122 Test #2007: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlsol_202402240000 ......   Passed  122.48 sec
        Start 2008: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfv3inc_202402240000
111/122 Test #2008: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfv3inc_202402240000 ...   Passed  107.76 sec
        Start 2009: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfinal_202402240000
112/122 Test #1999: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlvar_202402240000 .............   Passed  1530.89 sec
        Start 2000: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfv3inc_202402240000
113/122 Test #2030: test_gdasapp_C96C48_hybatmaerosnowDA_gdas_fcst_202112201800 ..............   Passed  1091.15 sec
114/122 Test #2000: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfv3inc_202402240000 ..........   Passed  142.58 sec
        Start 2001: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfinal_202402240000
115/122 Test #2009: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfinal_202402240000 ....   Passed  253.58 sec
116/122 Test #2001: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfinal_202402240000 ...........   Passed  338.02 sec
        Start 2002: test_gdasapp_C96C48_ufs_hybatmDA_gdas_sfcanl_202402240000
117/122 Test #2002: test_gdasapp_C96C48_ufs_hybatmDA_gdas_sfcanl_202402240000 ................   Passed  170.89 sec
        Start 2003: test_gdasapp_C96C48_ufs_hybatmDA_gdas_analcalc_202402240000
        Start 2004: test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402240000
118/122 Test #2003: test_gdasapp_C96C48_ufs_hybatmDA_gdas_analcalc_202402240000 ..............   Passed   75.83 sec
        Start 2010: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_ecmn_202402240000
        Start 2011: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_esfc_202402240000
119/122 Test #2010: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_ecmn_202402240000 ..............   Passed   49.96 sec
120/122 Test #2011: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_esfc_202402240000 ..............   Passed  164.54 sec
        Start 2012: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402240000
121/122 Test #2004: test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402240000 ..................   Passed  450.70 sec
122/122 Test #2012: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402240000 ..............   Passed  548.13 sec

100% tests passed, 0 tests failed out of 122

Label Time Summary:
gdas-utils    =  12.96 sec*proc (14 tests)
gdasapp       = 25045.14 sec*proc (90 tests)
script        = 25058.10 sec*proc (104 tests)

Total Test time (real) = 5152.26 sec

Received the following email

From: Darth Vader - NOAA Affiliate <[email protected]>
Date: Thu, Dec 12, 2024 at 9:55 PM
Subject: Success updating feature/stable-nightly branch of GDASApp
To: <[email protected]>

feature/stable-nightly branch of GDASApp updated successfully. See /scratch1/NCEPDEV/da/Russ.Treadon/CI/GDASApp/test/stable/20241213/global-workflow for details.

RussTreadon-NOAA

Approve.

RussTreadon-NOAA · 2024-12-13T03:07:30Z

Orion and Hercules labels fail because role-da does not have access to the da-cpu or other accounting codes. MSU helpdesk ticket RDHPCS#2024121254000069 submitted requesting role-da access to da-cpu.

RussTreadon-NOAA

Approve

RussTreadon-NOAA · 2024-12-13T19:17:50Z

@DavidNew-NOAA , do we expect any more changes to this PR or is it ready for final review, approval, and merger?

DavidNew-NOAA · 2024-12-13T19:21:30Z

@RussTreadon-NOAA I was kind of waiting on resolving the issue with MSU role accounts, but honestly that shouldn't hold up this PR. We can resolve any future problems with another quick bugfix PR.

RussTreadon-NOAA · 2024-12-13T19:24:34Z

OK. We can wait. The request to add MSU role-da to da-cpu has been approved and passed onto MSU admins for action. groups role-da shows that da-cpu has not yet been added.

DavidNew-NOAA added 8 commits December 3, 2024 21:05

Initial commit

15a8f5e

Consolidate GW and standalone GDASApp CI scripts

97fd677

Remove test line

e4aff0a

Update

576526d

Move CI test list to its own file

6ad7811

Clean up

75e95df

Add "jjob" to first jjob ctest name

6523965

Add orion authorized users

13eae53

DavidNew-NOAA self-assigned this Dec 5, 2024

DavidNew-NOAA added hera-GW-RT Queue for automated testing with global-workflow on Hera hera-RT Queue for automated testing on Hera labels Dec 5, 2024

emcbot added hera-RT-Running Automated testing running on Hera hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-RT Queue for automated testing on Hera hera-GW-RT Queue for automated testing with global-workflow on Hera labels Dec 5, 2024

emcbot added hera-RT-Passed Automated testing successful on Hera and removed hera-RT-Running Automated testing running on Hera labels Dec 5, 2024

emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Dec 5, 2024

DavidNew-NOAA and others added 3 commits December 5, 2024 16:37

Add pull request template

6ab9d97

Update stable_driver.sh to use new run_ci.sh -w command

3af926b

Merge branch 'develop' into feature/update-ci

837b8ef

DavidNew-NOAA requested review from RussTreadon-NOAA, CoryMartin-NOAA, danholdaway and guillaumevernieres and removed request for RussTreadon-NOAA December 5, 2024 17:08

DavidNew-NOAA added orion-RT Queue for automated testing on Orion and removed orion-RT-Failed Automated testing failed on Orion labels Dec 12, 2024

emcbot added orion-RT-Running Automated testing running on Orion and removed orion-RT Queue for automated testing on Orion labels Dec 12, 2024

emcbot added orion-RT-Failed Automated testing failed on Orion and removed orion-RT-Running Automated testing running on Orion labels Dec 12, 2024

DavidNew-NOAA added orion-RT Queue for automated testing on Orion and removed orion-RT-Failed Automated testing failed on Orion labels Dec 12, 2024

emcbot added orion-RT-Running Automated testing running on Orion and removed orion-RT Queue for automated testing on Orion labels Dec 12, 2024

emcbot added orion-RT-Failed Automated testing failed on Orion and removed orion-RT-Running Automated testing running on Orion labels Dec 12, 2024

Remove caution

c74dd4f

RussTreadon-NOAA self-requested a review December 13, 2024 03:03

RussTreadon-NOAA previously approved these changes Dec 13, 2024

View reviewed changes

DavidNew-NOAA mentioned this pull request Dec 13, 2024

Only run 2 members for the marine hybrid #1413

Merged

6 tasks

Update stable_driver.sh

e9223df

DavidNew-NOAA dismissed RussTreadon-NOAA’s stale review via e9223df December 13, 2024 14:33

RussTreadon-NOAA approved these changes Dec 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update CI driver #1401

Update CI driver #1401

DavidNew-NOAA commented Dec 5, 2024 •

edited

Loading

emcbot commented Dec 5, 2024

emcbot commented Dec 5, 2024

emcbot commented Dec 12, 2024

emcbot commented Dec 12, 2024

RussTreadon-NOAA commented Dec 12, 2024

DavidNew-NOAA commented Dec 12, 2024

RussTreadon-NOAA commented Dec 12, 2024

RussTreadon-NOAA commented Dec 12, 2024

RussTreadon-NOAA commented Dec 13, 2024 •

edited

Loading

RussTreadon-NOAA left a comment

RussTreadon-NOAA commented Dec 13, 2024

RussTreadon-NOAA left a comment

RussTreadon-NOAA commented Dec 13, 2024

DavidNew-NOAA commented Dec 13, 2024

RussTreadon-NOAA commented Dec 13, 2024

Update CI driver #1401

Are you sure you want to change the base?

Update CI driver #1401

Conversation

DavidNew-NOAA commented Dec 5, 2024 • edited Loading

Description

Companion PRs

Issues

Automated CI tests to run in Global Workflow

emcbot commented Dec 5, 2024

emcbot commented Dec 5, 2024

emcbot commented Dec 12, 2024

emcbot commented Dec 12, 2024

RussTreadon-NOAA commented Dec 12, 2024

DavidNew-NOAA commented Dec 12, 2024

RussTreadon-NOAA commented Dec 12, 2024

RussTreadon-NOAA commented Dec 12, 2024

RussTreadon-NOAA commented Dec 13, 2024 • edited Loading

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

RussTreadon-NOAA commented Dec 13, 2024

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

RussTreadon-NOAA commented Dec 13, 2024

DavidNew-NOAA commented Dec 13, 2024

RussTreadon-NOAA commented Dec 13, 2024

DavidNew-NOAA commented Dec 5, 2024 •

edited

Loading

RussTreadon-NOAA commented Dec 13, 2024 •

edited

Loading