Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separate use of initial increment/perturbation file from REPLAY/+03 ICs #3119

Open
wants to merge 113 commits into
base: develop
Choose a base branch
from
Open

Separate use of initial increment/perturbation file from REPLAY/+03 ICs #3119

wants to merge 113 commits into from

Conversation

NeilBarton-NOAA
Copy link
Contributor

Description

This PR is a continuation of the previous SFS PR. The use of initial perturbations/increments for ensemble long forecasts (e.g., SFS and GEFS) was missing from the previous PR because this option was included with the REPLAY_IC option.

In this PR, the REPLAY_IC option is now strictly for the +03 ICs, and the variables USE_ATM_ENS_PERTURB_FILES and USE_OCN_ENS_PERTURB_FILES are used for initialization with the initial perturbation/increment files used for SFS and GEFS reforecasting. This separation will also make it easier to remove the +03 IC optoin.

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
    • EMC verif-global
    • GDAS
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

I tested long forecast runs on hercules and multiple CI tests.

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

NeilBarton-NOAA and others added 30 commits July 2, 2024 15:04
@NeilBarton-NOAA
initial creation of CI using replay ICs
30cf5df
@NeilBarton-NOAA
change name
4b3ae39
@NeilBarton-NOAA
making an S2SWA case
fe2054e
@NeilBarton-NOAA
Merge remote-tracking branch 'origin/develop' into gefs_replay_ci
7b7ee9b
@NeilBarton-NOAA
Merge remote-tracking branch 'origin/develop' into gefs_replay_ci
225d280
@NeilBarton-NOAA
start of CI for GEFS replays # Please enter the commit message for your
b17765e 
changes. Lines starting
@NeilBarton-NOAA
Merge branch 'develop' into gefs_replay_ci
675cb77
@NeilBarton-NOAA
removing shell check errors
e49dbcf
@NeilBarton-NOAA
Merge branch 'gefs_replay_ci' of https://github.com/NeilBarton-NOAA/g…
ec4becc 
…lobal-workflow into gefs_replay_ci
@NeilBarton-NOAA
fixes for wave post processing
b69d653
@NeilBarton-NOAA
Merge branch 'NOAA-EMC:develop' into gefs_replay_ci
084d744
@NeilBarton-NOAA
Merge remote-tracking branch 'origin/develop' into gefs_replay_ci
a4bb8f3
@NeilBarton-NOAA
add BREAKPOINTS variable
4ff31fd
@NeilBarton-NOAA
fix shell checks
97b92ee
@NeilBarton-NOAA
edit defult config.efc to be close to re-forecasting
713f41d
@EricSinsky-NOAA
Fix bug in variable check
af31d2f 
Fix a bug in a variable check in the extractvars task where parenthesis in the variable name will prompt a false warning message.
@EricSinsky-NOAA
Add more settings to GEFS replay CI test
cec7b81
@EricSinsky-NOAA
Add atmos variables requested for reforeacst
4aaca9a
@EricSinsky-NOAA
Reorganize TSNOWP in paramlist
11e1957
@EricSinsky-NOAA
Extract ice variables from history file instead
a005e5f 
Ice variables are extracted from the history files instead of the product files. This is necessary because there are certain ice variables that are needed that are only available in the history files and not the product files.
@EricSinsky-NOAA
Fix typo
2dd9a7e
@EricSinsky-NOAA
Update TCDC variable to correct name
b132408
@NeilBarton-NOAA
Merge pull request #7 from EricSinsky-NOAA/gefs_replay_ci_pr
3d9455b 
Add suggested patches to gefs_replay_ci
@NeilBarton-NOAA
Merge branch 'NOAA-EMC:develop' into gefs_replay_ci
447224c
@NeilBarton-NOAA
edits to stage_ic for replay perturbations
c6b7eaa
@NeilBarton-NOAA
adding icsdir to pr CI yaml
51aa2dd
@NeilBarton-NOAA
Merge branch 'NOAA-EMC:develop' into gefs_replay_ci
a641591
@EricSinsky-NOAA
Fix ensstat failures
4985361
@NeilBarton-NOAA
updates for products
8eaef27
@EricSinsky-NOAA
Add shellcheck disable and skip f000 for ocn/ice
2e7241f 
Shellcheck disable has been added. Also, f000 is skipped in extractvars for ocn and ice since there are no products for f000 for ocn and ice. This change will remove warning messages.
@emcbot emcbot added CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Jan 2, 2025
@emcbot
Copy link

emcbot commented Jan 2, 2025

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3119/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_0c42898d

@emcbot
Copy link

emcbot commented Jan 2, 2025

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 1 with error logs:

/work2/noaa/global/CI/HERCULES/3119/RUNTESTS/COMROOT/C48_S2SWA_gefs_0c42898d/logs/2021032312/gefs_stage_ic.log

Follow link here to view the contents of the above file(s): (link)

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Jan 2, 2025
@emcbot
Copy link

emcbot commented Jan 2, 2025

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 1 in
/work2/noaa/global/CI/HERCULES/3119/RUNTESTS/EXPDIR/C48_S2SWA_gefs_0c42898d

@emcbot
Copy link

emcbot commented Jan 2, 2025

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 2 with error logs:

/scratch1/NCEPDEV/global/CI/3119/RUNTESTS/COMROOT/C48_S2SWA_gefs_0c42898d/logs/2021032312/gefs_stage_ic.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Jan 2, 2025

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3119/RUNTESTS/EXPDIR/C48_S2SWA_gefs_0c42898d

@aerorahul
Copy link
Contributor

@NeilBarton-NOAA
Does this PR need additional files to be staged for initial conditions?

@emcbot
Copy link

emcbot commented Jan 2, 2025

Experiment C96C48_hybatmDA FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3119/RUNTESTS/EXPDIR/C96C48_hybatmDA_0c42898d

@emcbot
Copy link

emcbot commented Jan 2, 2025

Experiment C48_S2SW FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3119/RUNTESTS/EXPDIR/C48_S2SW_0c42898d

@emcbot
Copy link

emcbot commented Jan 2, 2025

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 2 with error logs:

/scratch1/NCEPDEV/global/CI/3119/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_0c42898d/logs/2021122018/gdas_aeroanlvar.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Jan 2, 2025

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3119/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_0c42898d

@RussTreadon-NOAA
Copy link
Contributor

C96C48_hybatmaerosnowDA failed due to job gdas_aeroanlvar.log reaching its specified 30 minute wall clock limit

 0:  DRPCG Starting Iteration 27
 0: slurmstepd: error: *** STEP 4666462.0 ON h5c31 CANCELLED AT 2025-01-02T19:08:59 DUE TO TIME LIMIT ***
slurmstepd: error: *** JOB 4666462 ON h5c31 CANCELLED AT 2025-01-02T19:08:59 DUE TO TIME LIMIT ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.

Normally this job takes 7 to 8 minutes to run.

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Jan 2, 2025
@emcbot
Copy link

emcbot commented Jan 2, 2025

CI Failed on Hera in Build# 2
Built and ran in directory /scratch1/NCEPDEV/global/CI/3119


Experiment C48_S2SWA_gefs_0c42898d Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Jan  2 17:38:35 UTC 2025
Experiment C48_S2SWA_gefs_0c42898d Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3119/RUNTESTS/COMROOT/C48_S2SWA_gefs_0c42898d/logs/2021032312/gefs_stage_ic.log
Experiment C96C48_hybatmDA_0c42898d Terminated with  tasks failed and  dead at Thu Jan  2 18:50:40 UTC 2025
Experiment C96C48_hybatmDA_0c42898d Terminated: **
Experiment C48_S2SW_0c42898d Terminated with  tasks failed and  dead at Thu Jan  2 18:50:45 UTC 2025
Experiment C48_S2SW_0c42898d Terminated: **
Experiment C48mx500_3DVarAOWCDA_0c42898d Completed 2 Cycles: *SUCCESS* at Thu Jan  2 19:08:17 UTC 2025
Experiment C48_ATM_0c42898d Completed 2 Cycles: *SUCCESS* at Thu Jan  2 19:12:15 UTC 2025
Experiment C96C48_hybatmaerosnowDA_0c42898d Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Jan  2 19:13:25 UTC 2025
Experiment C96C48_hybatmaerosnowDA_0c42898d Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3119/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_0c42898d/logs/2021122018/gdas_aeroanlvar.log
Experiment C48mx500_hybAOWCDA_0c42898d Completed 2 Cycles: *SUCCESS* at Thu Jan  2 19:25:46 UTC 2025
Experiment C96_S2SWA_gefs_replay_ics_0c42898d Completed 1 Cycles: *SUCCESS* at Thu Jan  2 19:25:49 UTC 2025
Experiment C96_atm3DVar_0c42898d Completed 3 Cycles: *SUCCESS* at Thu Jan  2 20:34:47 UTC 2025

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Jan 2, 2025
@emcbot
Copy link

emcbot commented Jan 2, 2025

CI Failed on Hercules in Build# 1
Built and ran in directory /work2/noaa/global/CI/HERCULES/3119


Experiment C48_S2SWA_gefs_0c42898d Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Jan  2 11:25:41 CST 2025
Experiment C48_S2SWA_gefs_0c42898d Terminated: *FAIL*
Error logs:
/work2/noaa/global/CI/HERCULES/3119/RUNTESTS/COMROOT/C48_S2SWA_gefs_0c42898d/logs/2021032312/gefs_stage_ic.log
Experiment C48_ATM_0c42898d Completed 2 Cycles: *SUCCESS* at Thu Jan  2 12:31:42 CST 2025
Experiment C48mx500_3DVarAOWCDA_0c42898d Completed 2 Cycles: *SUCCESS* at Thu Jan  2 12:37:44 CST 2025
Experiment C48mx500_hybAOWCDA_0c42898d Completed 2 Cycles: *SUCCESS* at Thu Jan  2 12:49:55 CST 2025
Experiment C96_S2SWA_gefs_replay_ics_0c42898d Completed 1 Cycles: *SUCCESS* at Thu Jan  2 13:26:57 CST 2025
Experiment C96C48_hybatmDA_0c42898d Completed 3 Cycles: *SUCCESS* at Thu Jan  2 13:38:33 CST 2025
Experiment C96_atm3DVar_0c42898d Completed 3 Cycles: *SUCCESS* at Thu Jan  2 13:44:25 CST 2025
Experiment C48_S2SW_0c42898d Completed 2 Cycles: *SUCCESS* at Thu Jan  2 14:39:31 CST 2025

@NeilBarton-NOAA
Copy link
Contributor Author

fix for C48_S2SW CI case uploaded @WalterKolczynski-NOAA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@XiaqiongZhou-NOAA XiaqiongZhou-NOAA XiaqiongZhou-NOAA left review comments

@KateFriedman-NOAA KateFriedman-NOAA KateFriedman-NOAA left review comments

@RuiyuSun RuiyuSun RuiyuSun approved these changes

@EricSinsky-NOAA EricSinsky-NOAA EricSinsky-NOAA left review comments

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA left review comments

@JessicaMeixner-NOAA JessicaMeixner-NOAA JessicaMeixner-NOAA left review comments

@aerorahul aerorahul Awaiting requested review from aerorahul aerorahul is a code owner

@junwang-noaa junwang-noaa Awaiting requested review from junwang-noaa

@DavidHuber-NOAA DavidHuber-NOAA Awaiting requested review from DavidHuber-NOAA

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Awaiting requested review from TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.