Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Port shoc_assumed_pdf functions individually (like in shoc.F90) #2932

Merged
merged 3 commits into from
Aug 7, 2024
Merged

Port shoc_assumed_pdf functions individually (like in shoc.F90) #2932

merged 3 commits into from
Aug 7, 2024

Conversation

tcclevenger
Copy link
Contributor

Closes #1377. This is needed before all property tests can be switched to C++ from F90.

I didn't make individual BFB tests since I think I would need a work around for randomizing a single pack of data, and the BFBness is tested in shoc_assumed_bfb.

@tcclevenger tcclevenger added BFB Bit for bit shoc regarding shoc macrophysics labels Aug 1, 2024
@tcclevenger tcclevenger requested review from bartgol and bogensch August 1, 2024 21:23
@tcclevenger tcclevenger self-assigned this Aug 1, 2024
@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 1, 2024

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5745
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 5c6cfd9
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5974
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 5c6cfd9
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: tcclevenger/refactor_shoc_function
  • SHA: 5c6cfd9
  • Mode: TEST_REPO

Pull Request Author: tcclevenger

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 1, 2024

Status Flag 'Pull Request AutoTester' - Error: Jenkins Jobs - A user has pushed a change to the PR before testing completed. NEW EVENT 'committed', ID C_kwDOCEfuetoAKDIzMDQyOGY3ZTdlMWYwODExMmNkMGM5NDAyNjQ1N2ZmYzQwNGEzMTA... The Jenkins Jobs will be shutdown; Testing of this PR must occur again.

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 1, 2024

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5745
  • Status: ERROR

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 5c6cfd9
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5974
  • Status: ERROR

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 5c6cfd9
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Mappy # 5745 ERROR (click to see last 100 lines of console output) 

+++ export _ModuleTable_Sz_
++ export PATH=/ascldap/users/jgfouca/packages/Python-3.8.5/bin:/usr/lib64/qt-3.3/bin:/usr/condabin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/opt/dell/srvadmin/sbin:/ascldap/users/e3sm-jenkins/.local/bin:/ascldap/users/e3sm-jenkins/bin
++ PATH=/ascldap/users/jgfouca/packages/Python-3.8.5/bin:/usr/lib64/qt-3.3/bin:/usr/condabin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/opt/dell/srvadmin/sbin:/ascldap/users/e3sm-jenkins/.local/bin:/ascldap/users/e3sm-jenkins/bin
++ module load sems-archive-git/2.10.1
+++ /projects/sems/install/rhel7-x86_64/sems/v2/lmod/lmod/8.3/gcc/10.1.0/zbzzu7k/lmod/lmod/libexec/lmod sh load sems-archive-git/2.10.1
++ eval '__LMOD_REF_COUNT_LD_LIBRARY_PATH=/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/lib:1;' export '__LMOD_REF_COUNT_LD_LIBRARY_PATH;' 'LD_LIBRARY_PATH=/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/lib;' export 'LD_LIBRARY_PATH;' '__LMOD_REF_COUNT_LOADEDMODULES=sems-archive-env:1\;sems-archive-git/2.10.1:1;' export '__LMOD_REF_COUNT_LOADEDMODULES;' 'LOADEDMODULES=sems-archive-env:sems-archive-git/2.10.1;' export 'LOADEDMODULES;' '__LMOD_REF_COUNT_MANPATH=/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/share/man:1\;/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/man:1;' export '__LMOD_REF_COUNT_MANPATH;' 'MANPATH=/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/share/man:/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/man;' export 'MANPATH;' 'MODULEPATH=/projects/sems/modulefiles/rhel7-x86_64/sems-archive/devpack:/projects/sems/modulefiles/rhel7-x86_64/sems-archive/compiler:/projects/sems/modulefiles/rhel7-x86_64/sems-archive/tpl:/projects/sems/modulefiles/rhel7-x86_64/sems-archive/utility:/projects/sems/modulefiles/projects:/projects/sems/cee-sierra-modules:/projects/sems/modulefiles/rhel7-x86_64/sems/linux-rhel7-x86_64/compilers:/projects/sems/modulefiles/rhel7-x86_64/sems/linux-rhel7-x86_64/Core:/projects/sems/modulefiles/rhel7-x86_64/sems/linux-rhel7-x86_64/project-modulefiles:/projects/aue/modules/cee/x86_64/rhel7:/usr/share/Modules/modulefiles:/etc/modulefiles;' export 'MODULEPATH;' '__LMOD_REF_COUNT_PATH=/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/sbin:1\;/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/bin:1\;/ascldap/users/jgfouca/packages/Python-3.8.5/bin:1\;/usr/lib64/qt-3.3/bin:1\;/usr/condabin:1\;/usr/local/sbin:1\;/usr/local/bin:1\;/sbin:1\;/bin:1\;/usr/sbin:1\;/usr/bin:1\;/root/bin:1\;/opt/dell/srvadmin/sbin:1\;/ascldap/users/e3sm-jenkins/.local/bin:1\;/ascldap/users/e3sm-jenkins/bin:1;' export '__LMOD_REF_COUNT_PATH;' 'PATH=/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/sbin:/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/bin:/ascldap/users/jgfouca/packages/Python-3.8.5/bin:/usr/lib64/qt-3.3/bin:/usr/condabin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/opt/dell/srvadmin/sbin:/ascldap/users/e3sm-jenkins/.local/bin:/ascldap/users/e3sm-jenkins/bin;' export 'PATH;' 'SEMS_GIT_LOCAL_COMPILER_VERSION=4.8.5;' export 'SEMS_GIT_LOCAL_COMPILER_VERSION;' 'SEMS_GIT_LOCAL_PYTHON_VERSION=2.7.5;' export 'SEMS_GIT_LOCAL_PYTHON_VERSION;' 'SEMS_GIT_ROOT=/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1;' export 'SEMS_GIT_ROOT;' 'SEMS_GIT_VERSION=2.10.1;' export 'SEMS_GIT_VERSION;' '__LMOD_REF_COUNT__LMFILES_=/projects/sems/modulefiles/projects/sems-archive-env:1\;/projects/sems/modulefiles/rhel7-x86_64/sems-archive/utility/sems-archive-git/2.10.1:1;' export '__LMOD_REF_COUNT__LMFILES_;' '_LMFILES_=/projects/sems/modulefiles/projects/sems-archive-env:/projects/sems/modulefiles/rhel7-x86_64/sems-archive/utility/sems-archive-git/2.10.1;' export '_LMFILES_;' '_ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT1mYWxzZSxkZXB0aFQ9e30sZmFtaWx5PXt9LG1UPXtbInNlbXMtYXJjaGl2ZS1lbnYiXT17WyJmbiJdPSIvcHJvamVjdHMvc2Vtcy9tb2R1bGVmaWxlcy9wcm9qZWN0cy9zZW1zLWFyY2hpdmUtZW52IixbImZ1bGxOYW1lIl09InNlbXMtYXJjaGl2ZS1lbnYiLFsibG9hZE9yZGVyIl09MSxwcm9wVD17fSxbInN0YWNrRGVwdGgiXT0wLFsic3RhdHVzIl09ImFjdGl2ZSIsWyJ1c2VyTmFtZSJdPSJzZW1zLWFyY2hpdmUtZW52Iix9LFsic2Vtcy1hcmNoaXZlLWdpdCJdPXtbImZuIl09Ii9wcm9qZWN0cy9zZW1zL21vZHVsZWZpbGVzL3JoZWw3LXg4Nl82NC9z;' export '_ModuleTable001_;' '_ModuleTable002_=ZW1zLWFyY2hpdmUvdXRpbGl0eS9zZW1zLWFyY2hpdmUtZ2l0LzIuMTAuMSIsWyJmdWxsTmFtZSJdPSJzZW1zLWFyY2hpdmUtZ2l0LzIuMTAuMSIsWyJsb2FkT3JkZXIiXT0yLHByb3BUPXt9LFsic3RhY2tEZXB0aCJdPTAsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09InNlbXMtYXJjaGl2ZS1naXQvMi4xMC4xIix9LH0sbXBhdGhBPXsiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMtYXJjaGl2ZS9kZXZwYWNrIiwiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMtYXJjaGl2ZS9jb21waWxlciIsIi9wcm9qZWN0cy9zZW1zL21vZHVsZWZpbGVzL3JoZWw3LXg4Nl82NC9zZW1zLWFyY2hpdmUvdHBsIiwi;' export '_ModuleTable002_;' '_ModuleTable003_=L3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMtYXJjaGl2ZS91dGlsaXR5IiwiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcHJvamVjdHMiLCIvcHJvamVjdHMvc2Vtcy9jZWUtc2llcnJhLW1vZHVsZXMiLCIvcHJvamVjdHMvc2Vtcy9tb2R1bGVmaWxlcy9yaGVsNy14ODZfNjQvc2Vtcy9saW51eC1yaGVsNy14ODZfNjQvY29tcGlsZXJzIiwiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMvbGludXgtcmhlbDcteDg2XzY0L0NvcmUiLCIvcHJvamVjdHMvc2Vtcy9tb2R1bGVmaWxlcy9yaGVsNy14ODZfNjQvc2Vtcy9saW51eC1yaGVsNy14ODZfNjQvcHJvamVjdC1tb2R1bGVmaWxlcyIsIi9wcm9qZWN0cy9hdWUv;' export '_ModuleTable003_;' '_ModuleTable004_=bW9kdWxlcy9jZWUveDg2XzY0L3JoZWw3IiwiL3Vzci9zaGFyZS9Nb2R1bGVzL21vZHVsZWZpbGVzIiwiL2V0Yy9tb2R1bGVmaWxlcyIsfSxbInN5c3RlbUJhc2VNUEFUSCJdPSIvdXNyL3NoYXJlL01vZHVsZXMvbW9kdWxlZmlsZXM6L2V0Yy9tb2R1bGVmaWxlcyIsfQ==;' export '_ModuleTable004_;' '_ModuleTable_Sz_=4;' export '_ModuleTable_Sz_;'
+++ __LMOD_REF_COUNT_LD_LIBRARY_PATH=/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/lib:1
+++ export __LMOD_REF_COUNT_LD_LIBRARY_PATH
+++ LD_LIBRARY_PATH=/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/lib
+++ export LD_LIBRARY_PATH
+++ __LMOD_REF_COUNT_LOADEDMODULES='sems-archive-env:1;sems-archive-git/2.10.1:1'
+++ export __LMOD_REF_COUNT_LOADEDMODULES
+++ LOADEDMODULES=sems-archive-env:sems-archive-git/2.10.1
+++ export LOADEDMODULES
+++ __LMOD_REF_COUNT_MANPATH='/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/share/man:1;/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/man:1'
+++ export __LMOD_REF_COUNT_MANPATH
+++ MANPATH=/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/share/man:/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/man
+++ export MANPATH
+++ MODULEPATH=/projects/sems/modulefiles/rhel7-x86_64/sems-archive/devpack:/projects/sems/modulefiles/rhel7-x86_64/sems-archive/compiler:/projects/sems/modulefiles/rhel7-x86_64/sems-archive/tpl:/projects/sems/modulefiles/rhel7-x86_64/sems-archive/utility:/projects/sems/modulefiles/projects:/projects/sems/cee-sierra-modules:/projects/sems/modulefiles/rhel7-x86_64/sems/linux-rhel7-x86_64/compilers:/projects/sems/modulefiles/rhel7-x86_64/sems/linux-rhel7-x86_64/Core:/projects/sems/modulefiles/rhel7-x86_64/sems/linux-rhel7-x86_64/project-modulefiles:/projects/aue/modules/cee/x86_64/rhel7:/usr/share/Modules/modulefiles:/etc/modulefiles
+++ export MODULEPATH
+++ __LMOD_REF_COUNT_PATH='/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/sbin:1;/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/bin:1;/ascldap/users/jgfouca/packages/Python-3.8.5/bin:1;/usr/lib64/qt-3.3/bin:1;/usr/condabin:1;/usr/local/sbin:1;/usr/local/bin:1;/sbin:1;/bin:1;/usr/sbin:1;/usr/bin:1;/root/bin:1;/opt/dell/srvadmin/sbin:1;/ascldap/users/e3sm-jenkins/.local/bin:1;/ascldap/users/e3sm-jenkins/bin:1'
+++ export __LMOD_REF_COUNT_PATH
+++ PATH=/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/sbin:/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/bin:/ascldap/users/jgfouca/packages/Python-3.8.5/bin:/usr/lib64/qt-3.3/bin:/usr/condabin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/opt/dell/srvadmin/sbin:/ascldap/users/e3sm-jenkins/.local/bin:/ascldap/users/e3sm-jenkins/bin
+++ export PATH
+++ SEMS_GIT_LOCAL_COMPILER_VERSION=4.8.5
+++ export SEMS_GIT_LOCAL_COMPILER_VERSION
+++ SEMS_GIT_LOCAL_PYTHON_VERSION=2.7.5
+++ export SEMS_GIT_LOCAL_PYTHON_VERSION
+++ SEMS_GIT_ROOT=/projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1
+++ export SEMS_GIT_ROOT
+++ SEMS_GIT_VERSION=2.10.1
+++ export SEMS_GIT_VERSION
+++ __LMOD_REF_COUNT__LMFILES_='/projects/sems/modulefiles/projects/sems-archive-env:1;/projects/sems/modulefiles/rhel7-x86_64/sems-archive/utility/sems-archive-git/2.10.1:1'
+++ export __LMOD_REF_COUNT__LMFILES_
+++ _LMFILES_=/projects/sems/modulefiles/projects/sems-archive-env:/projects/sems/modulefiles/rhel7-x86_64/sems-archive/utility/sems-archive-git/2.10.1
+++ export _LMFILES_
+++ _ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT1mYWxzZSxkZXB0aFQ9e30sZmFtaWx5PXt9LG1UPXtbInNlbXMtYXJjaGl2ZS1lbnYiXT17WyJmbiJdPSIvcHJvamVjdHMvc2Vtcy9tb2R1bGVmaWxlcy9wcm9qZWN0cy9zZW1zLWFyY2hpdmUtZW52IixbImZ1bGxOYW1lIl09InNlbXMtYXJjaGl2ZS1lbnYiLFsibG9hZE9yZGVyIl09MSxwcm9wVD17fSxbInN0YWNrRGVwdGgiXT0wLFsic3RhdHVzIl09ImFjdGl2ZSIsWyJ1c2VyTmFtZSJdPSJzZW1zLWFyY2hpdmUtZW52Iix9LFsic2Vtcy1hcmNoaXZlLWdpdCJdPXtbImZuIl09Ii9wcm9qZWN0cy9zZW1zL21vZHVsZWZpbGVzL3JoZWw3LXg4Nl82NC9z
+++ export _ModuleTable001_
+++ _ModuleTable002_=ZW1zLWFyY2hpdmUvdXRpbGl0eS9zZW1zLWFyY2hpdmUtZ2l0LzIuMTAuMSIsWyJmdWxsTmFtZSJdPSJzZW1zLWFyY2hpdmUtZ2l0LzIuMTAuMSIsWyJsb2FkT3JkZXIiXT0yLHByb3BUPXt9LFsic3RhY2tEZXB0aCJdPTAsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09InNlbXMtYXJjaGl2ZS1naXQvMi4xMC4xIix9LH0sbXBhdGhBPXsiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMtYXJjaGl2ZS9kZXZwYWNrIiwiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMtYXJjaGl2ZS9jb21waWxlciIsIi9wcm9qZWN0cy9zZW1zL21vZHVsZWZpbGVzL3JoZWw3LXg4Nl82NC9zZW1zLWFyY2hpdmUvdHBsIiwi
+++ export _ModuleTable002_
+++ _ModuleTable003_=L3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMtYXJjaGl2ZS91dGlsaXR5IiwiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcHJvamVjdHMiLCIvcHJvamVjdHMvc2Vtcy9jZWUtc2llcnJhLW1vZHVsZXMiLCIvcHJvamVjdHMvc2Vtcy9tb2R1bGVmaWxlcy9yaGVsNy14ODZfNjQvc2Vtcy9saW51eC1yaGVsNy14ODZfNjQvY29tcGlsZXJzIiwiL3Byb2plY3RzL3NlbXMvbW9kdWxlZmlsZXMvcmhlbDcteDg2XzY0L3NlbXMvbGludXgtcmhlbDcteDg2XzY0L0NvcmUiLCIvcHJvamVjdHMvc2Vtcy9tb2R1bGVmaWxlcy9yaGVsNy14ODZfNjQvc2Vtcy9saW51eC1yaGVsNy14ODZfNjQvcHJvamVjdC1tb2R1bGVmaWxlcyIsIi9wcm9qZWN0cy9hdWUv
+++ export _ModuleTable003_
+++ _ModuleTable004_=bW9kdWxlcy9jZWUveDg2XzY0L3JoZWw3IiwiL3Vzci9zaGFyZS9Nb2R1bGVzL21vZHVsZWZpbGVzIiwiL2V0Yy9tb2R1bGVmaWxlcyIsfSxbInN5c3RlbUJhc2VNUEFUSCJdPSIvdXNyL3NoYXJlL01vZHVsZXMvbW9kdWxlZmlsZXM6L2V0Yy9tb2R1bGVmaWxlcyIsfQ==
+++ export _ModuleTable004_
+++ _ModuleTable_Sz_=4
+++ export _ModuleTable_Sz_
++ source /ascldap/users/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5745/scream/components/eamxx/scripts/jenkins/sandia_son_proxy
+++ export http_proxy=http://proxy.sandia.gov:80
+++ http_proxy=http://proxy.sandia.gov:80
+++ export RSYNC_PROXY=proxy.sandia.gov:80
+++ RSYNC_PROXY=proxy.sandia.gov:80
+++ export rsync_proxy=proxy.sandia.gov:80
+++ rsync_proxy=proxy.sandia.gov:80
+++ export HTTPS_PROXY=http://proxy.sandia.gov:80
+++ HTTPS_PROXY=http://proxy.sandia.gov:80
+++ export https_proxy=http://proxy.sandia.gov:80
+++ https_proxy=http://proxy.sandia.gov:80
+++ export HTTP_PROXY=http://proxy.sandia.gov:80
+++ HTTP_PROXY=http://proxy.sandia.gov:80
++ SCREAM_MACHINE=mappy
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
++ whoami
+ [[ e3sm-jenkins == \e\3\s\m\-\j\e\n\k\i\n\s ]]
+ git config --local user.email [email protected]
+ git config --local user.name 'Jenkins Jenkins'
+ declare -i fails=0
+ BASELINES_DIR=AUTO
+ TAS_ARGS='--baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m $machine'
+ [[ mappy == \p\m\-\g\p\u ]]
+ set +e
+ '[' -n 2932 ']'
+ is_at_run=1
+ SA_FAILURES_DETAILS=
+ '[' 1 -eq 1 ']'
++ ./scripts/gather-all-data './scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m $machine' -l -m mappy
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 127195 killed;
[ssh-agent] Stopped.
Build was aborted
Aborted by Luca Bertagna
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins16421848226114659539.sh

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Finished: ABORTED
SCREAM_PullRequest_Autotester_Weaver # 5974 ERROR (click to see last 100 lines of console output) 

PYTHON_BIN=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/bin;
export PYTHON_BIN;
PYTHON_INC=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/include;
export PYTHON_INC;
PYTHON_LIB=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib;
export PYTHON_LIB;
PYTHON_ROOT=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe;
export PYTHON_ROOT;
PYTHON_VERSION=3.10.8;
export PYTHON_VERSION;
_LMFILES_=/projects/ppc64le-pwr9-rhel8/modulefiles/lmod/utilities/linux-rhel8-ppc64le/Core/python/3.10.8.lua;
export _LMFILES_;
_ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0ge30sCm1UID0gewpweXRob24gPSB7CmZuID0gIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC91dGlsaXRpZXMvbGludXgtcmhlbDgtcHBjNjRsZS9Db3JlL3B5dGhvbi8zLjEwLjgubHVhIiwKZnVsbE5hbWUgPSAicHl0aG9uLzMuMTAuOCIsCmxvYWRPcmRlciA9IDEsCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAwLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAicHl0aG9uLzMuMTAuOCIsCndWID0gIjAwMDAwMDAwMy4wMDAwMDAwMTAuMDAwMDAwMDA4Lip6;
export _ModuleTable001_;
_ModuleTable002_=ZmluYWwiLAp9LAp9LAptcGF0aEEgPSB7CiIvcHJvamVjdHMvcHBjNjRsZS1wd3I5LXJoZWw4L21vZHVsZWZpbGVzL2xtb2QvY29tcGlsZXJzIiwgIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC91dGlsaXRpZXMvbGludXgtcmhlbDgtcHBjNjRsZS9Db3JlIiwKfSwKc3lzdGVtQmFzZU1QQVRIID0gIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC9jb21waWxlcnM6L3Byb2plY3RzL3BwYzY0bGUtcHdyOS1yaGVsOC9tb2R1bGVmaWxlcy9sbW9kL3V0aWxpdGllcy9saW51eC1yaGVsOC1wcGM2NGxlL0NvcmUiLAp9Cg==;
export _ModuleTable002_;
_ModuleTable_Sz_=2;
export _ModuleTable_Sz_;'
+++ __LMOD_REF_COUNT_CMAKE_PREFIX_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe:1
+++ export __LMOD_REF_COUNT_CMAKE_PREFIX_PATH
+++ CMAKE_PREFIX_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe
+++ export CMAKE_PREFIX_PATH
+++ __LMOD_REF_COUNT_LD_LIBRARY_PATH='/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib:1;/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/lib:1'
+++ export __LMOD_REF_COUNT_LD_LIBRARY_PATH
+++ LD_LIBRARY_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib:/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/lib
+++ export LD_LIBRARY_PATH
+++ __LMOD_REF_COUNT_LIBRARY_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib:1
+++ export __LMOD_REF_COUNT_LIBRARY_PATH
+++ LIBRARY_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib
+++ export LIBRARY_PATH
+++ LOADEDMODULES=python/3.10.8
+++ export LOADEDMODULES
+++ __LMOD_REF_COUNT_MANPATH='/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/share/man:1;/opt/lsf/10.1/man:1'
+++ export __LMOD_REF_COUNT_MANPATH
+++ MANPATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/share/man:/opt/lsf/10.1/man::
+++ export MANPATH
+++ __LMOD_REF_COUNT_MODULEPATH='/projects/ppc64le-pwr9-rhel8/modulefiles/lmod/compilers:1;/projects/ppc64le-pwr9-rhel8/modulefiles/lmod/utilities/linux-rhel8-ppc64le/Core:1'
+++ export __LMOD_REF_COUNT_MODULEPATH
+++ MODULEPATH=/projects/ppc64le-pwr9-rhel8/modulefiles/lmod/compilers:/projects/ppc64le-pwr9-rhel8/modulefiles/lmod/utilities/linux-rhel8-ppc64le/Core
+++ export MODULEPATH
+++ __LMOD_REF_COUNT_PATH='/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/bin:1;/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/etc:1;/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/bin:1;/usr/local/bin:1;/usr/bin:1;/usr/local/sbin:1;/usr/sbin:1;/home/e3sm-jenkins/.local/bin:1;/home/e3sm-jenkins/bin:1'
+++ export __LMOD_REF_COUNT_PATH
+++ PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/bin:/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/etc:/opt/lsf/10.1/linux3.10-glibc2.17-ppc64le/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/e3sm-jenkins/.local/bin:/home/e3sm-jenkins/bin
+++ export PATH
+++ __LMOD_REF_COUNT_PKG_CONFIG_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib/pkgconfig:1
+++ export __LMOD_REF_COUNT_PKG_CONFIG_PATH
+++ PKG_CONFIG_PATH=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib/pkgconfig
+++ export PKG_CONFIG_PATH
+++ PYTHON_BIN=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/bin
+++ export PYTHON_BIN
+++ PYTHON_INC=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/include
+++ export PYTHON_INC
+++ PYTHON_LIB=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe/lib
+++ export PYTHON_LIB
+++ PYTHON_ROOT=/projects/ppc64le-pwr9-rhel8/utilities/python/3.10.8/gcc/8.3.1/base/qmix2fe
+++ export PYTHON_ROOT
+++ PYTHON_VERSION=3.10.8
+++ export PYTHON_VERSION
+++ _LMFILES_=/projects/ppc64le-pwr9-rhel8/modulefiles/lmod/utilities/linux-rhel8-ppc64le/Core/python/3.10.8.lua
+++ export _LMFILES_
+++ _ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0ge30sCm1UID0gewpweXRob24gPSB7CmZuID0gIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC91dGlsaXRpZXMvbGludXgtcmhlbDgtcHBjNjRsZS9Db3JlL3B5dGhvbi8zLjEwLjgubHVhIiwKZnVsbE5hbWUgPSAicHl0aG9uLzMuMTAuOCIsCmxvYWRPcmRlciA9IDEsCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAwLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAicHl0aG9uLzMuMTAuOCIsCndWID0gIjAwMDAwMDAwMy4wMDAwMDAwMTAuMDAwMDAwMDA4Lip6
+++ export _ModuleTable001_
+++ _ModuleTable002_=ZmluYWwiLAp9LAp9LAptcGF0aEEgPSB7CiIvcHJvamVjdHMvcHBjNjRsZS1wd3I5LXJoZWw4L21vZHVsZWZpbGVzL2xtb2QvY29tcGlsZXJzIiwgIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC91dGlsaXRpZXMvbGludXgtcmhlbDgtcHBjNjRsZS9Db3JlIiwKfSwKc3lzdGVtQmFzZU1QQVRIID0gIi9wcm9qZWN0cy9wcGM2NGxlLXB3cjktcmhlbDgvbW9kdWxlZmlsZXMvbG1vZC9jb21waWxlcnM6L3Byb2plY3RzL3BwYzY0bGUtcHdyOS1yaGVsOC9tb2R1bGVmaWxlcy9sbW9kL3V0aWxpdGllcy9saW51eC1yaGVsOC1wcGM2NGxlL0NvcmUiLAp9Cg==
+++ export _ModuleTable002_
+++ _ModuleTable_Sz_=2
+++ export _ModuleTable_Sz_
++ SCREAM_MACHINE=weaver
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
++ whoami
+ [[ e3sm-jenkins == \e\3\s\m\-\j\e\n\k\i\n\s ]]
+ git config --local user.email [email protected]
+ git config --local user.name 'Jenkins Jenkins'
+ declare -i fails=0
+ BASELINES_DIR=AUTO
+ TAS_ARGS='--baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m $machine'
+ [[ weaver == \p\m\-\g\p\u ]]
+ set +e
+ '[' -n 2932 ']'
+ is_at_run=1
+ SA_FAILURES_DETAILS=
+ '[' 1 -eq 1 ']'
++ ./scripts/gather-all-data './scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m $machine' -l -m weaver
***Forced exclusive execution
<>
<>
Build was aborted
Aborted by Luca Bertagna
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins8659466934634988372.sh

./scream/components/eamxx/scripts/jenkins/jenkins_common.sh: line 7: 1790522 Broken pipe             $JENKINS_SCRIPT_DIR/jenkins_common_impl.sh 2>&1

1790523 Terminated              | tee JENKINS_$DATE_STAMP

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 1, 2024

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5746
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 2130b73
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5975
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 2130b73
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: tcclevenger/refactor_shoc_function
  • SHA: 2130b73
  • Mode: TEST_REPO

Pull Request Author: tcclevenger

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 1, 2024

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5746
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 2130b73
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5975
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 2130b73
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Mappy # 5746 FAILED (click to see last 100 lines of console output) 

Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 96.476754 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 192.175785 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 212.468488 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 206.425152 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 1.349735 seconds (PEND). [COMPLETED 1 of 10]
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav in 210.899800 seconds (PASS)
Starting RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav in 1.424416 seconds (PEND). [COMPLETED 2 of 10]
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci in 213.002474 seconds (PASS)
Starting RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci with 1 proc on interactive node and 64 procs on compute nodes
Finished MODEL_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 214.424867 seconds (PASS)
Starting RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci in 2.581924 seconds (PEND). [COMPLETED 3 of 10]
Finished RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 1.163868 seconds (PEND). [COMPLETED 4 of 10]
Starting MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 16 procs
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 16 procs
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 334.015683 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 8.925473 seconds (PEND). [COMPLETED 5 of 10]
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 424.509966 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 1.900005 seconds (PEND). [COMPLETED 6 of 10]
Finished MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 445.859493 seconds (PASS)
Starting RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 1 proc on interactive node and 16 procs on compute nodes
Finished RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 4.994407 seconds (PEND). [COMPLETED 7 of 10]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 592.768073 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 304.997347 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 1.292995 seconds (PEND). [COMPLETED 8 of 10]
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 0.973531 seconds (PEND). [COMPLETED 9 of 10]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 724.600408 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 1.811024 seconds (PEND). [COMPLETED 10 of 10]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240801_164530_dakmop
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240801_164530_dakmop
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240801_164530_dakmop
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240801_164530_dakmop
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240801_164530_dakmop
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240801_164530_dakmop
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci.C.20240801_164530_dakmop
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240801_164530_dakmop
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20240801_164530_dakmop
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240801_164530_dakmop
test-scheduler took 1038.6914145946503 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
Build type release failed at testing time. Here's a list of failed tests:
311:shoc_standalone_baseline_cmp
479:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5746/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5746/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5746/scream/components/eamxx

mappy failed

######################################################

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 118803 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le


cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins7699004504312317492.sh

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE
SCREAM_PullRequest_Autotester_Weaver # 5975 FAILED (click to see last 100 lines of console output) 

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx/ctest-build/full_sp_debug
===============================================================================
Testing '\''2130b732def65c8f025b3c369e587e2abcb1be23'\'' for test '\''full_debug'\''
===============================================================================
RUN: taskset -c 0-51 sh -c '\''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='\''\'\'''\''ON'\''\'\'''\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '\''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx/ctest-build/full_debug
===============================================================================
Testing '\''2130b732def65c8f025b3c369e587e2abcb1be23'\'' for test '\''release'\''
===============================================================================
RUN: taskset -c 104-155 sh -c '\''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='\''\'\'''\''ON'\''\'\'''\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '\''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx/ctest-build/release
Build type full_debug failed at testing time. Here'\''s a list of failed tests:
110:mam4_wetscav_standalone_baseline_cmp
147:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp

Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

95:mam4_wetscav_standalone_baseline_cmp


Build type release failed at testing time. Here'''s a list of failed tests:

109:mam4_wetscav_standalone_baseline_cmp

146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx

weaver failed'


    

  • errors='Build type full_debug failed at testing time. Here'''s a list of failed tests:
    
110:mam4_wetscav_standalone_baseline_cmp
    
147:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
    • 



Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

95:mam4_wetscav_standalone_baseline_cmp


Build type release failed at testing time. Here'''s a list of failed tests:

109:mam4_wetscav_standalone_baseline_cmp

146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx

weaver failed'


    

  • SA_FAILURES_DETAILS+='Build type full_debug failed at testing time. Here'''s a list of failed tests:
    
110:mam4_wetscav_standalone_baseline_cmp
    
147:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
    • 



Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

95:mam4_wetscav_standalone_baseline_cmp


Build type release failed at testing time. Here'''s a list of failed tests:

109:mam4_wetscav_standalone_baseline_cmp

146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx

weaver failed'


    

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
######################################################
    
FAILS DETECTED:
    
SCREAM STANDALONE TESTING FAILED!
    
Build type full_debug failed at testing time. Here's a list of failed tests:
    
110:mam4_wetscav_standalone_baseline_cmp
    
147:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
    • 



Build type full_sp_debug failed at testing time. Here's a list of failed tests:

95:mam4_wetscav_standalone_baseline_cmp


Build type release failed at testing time. Here's a list of failed tests:

109:mam4_wetscav_standalone_baseline_cmp

146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5975/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le


cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins17783702786906835169.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 5, 2024

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 5, 2024

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5753
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc;AT: RETEST
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 2130b73
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5981
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc;AT: RETEST
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 2130b73
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: tcclevenger/refactor_shoc_function
  • SHA: 2130b73
  • Mode: TEST_REPO

Pull Request Author: tcclevenger

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 5, 2024

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5753
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc;AT: RETEST
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 2130b73
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5981
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc;AT: RETEST
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 2130b73
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Mappy # 5753 FAILED (click to see last 100 lines of console output) 

Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 93.846931 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 191.977404 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 212.225704 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 214.605515 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 with 1 proc on interactive node and 64 procs on compute nodes
Finished MODEL_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 215.844891 seconds (PASS)
Starting RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics with 1 proc on interactive node and 64 procs on compute nodes
Finished MODEL_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci in 214.896623 seconds (PASS)
Starting RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci with 1 proc on interactive node and 64 procs on compute nodes
Finished MODEL_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav in 216.118098 seconds (PASS)
Starting RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 1.156944 seconds (PEND). [COMPLETED 1 of 10]
Finished RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci in 1.110132 seconds (PEND). [COMPLETED 2 of 10]
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 1.215221 seconds (PEND). [COMPLETED 3 of 10]
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 16 procs
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Starting MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 16 procs
Finished RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav in 1.199667 seconds (PEND). [COMPLETED 4 of 10]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 16 procs
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 335.657034 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 9.262872 seconds (PEND). [COMPLETED 5 of 10]
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 424.219368 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 1 proc on interactive node and 64 procs on compute nodes
Finished MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 424.271905 seconds (PASS)
Starting RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 1 proc on interactive node and 16 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 1.385839 seconds (PEND). [COMPLETED 6 of 10]
Finished RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 1.387781 seconds (PEND). [COMPLETED 7 of 10]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 608.860397 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 1.287401 seconds (PEND). [COMPLETED 8 of 10]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 336.644266 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 0.905537 seconds (PEND). [COMPLETED 9 of 10]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 737.452080 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 1.467301 seconds (PEND). [COMPLETED 10 of 10]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240805_101649_bfldbf
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240805_101649_bfldbf
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240805_101649_bfldbf
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240805_101649_bfldbf
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240805_101649_bfldbf
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240805_101649_bfldbf
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci.C.20240805_101649_bfldbf
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240805_101649_bfldbf
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20240805_101649_bfldbf
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240805_101649_bfldbf
test-scheduler took 1050.9391071796417 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
Build type release failed at testing time. Here's a list of failed tests:
311:shoc_standalone_baseline_cmp
479:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5753/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5753/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5753/scream/components/eamxx

mappy failed

######################################################

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 110879 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le


cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins12518771222116407945.sh

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE
SCREAM_PullRequest_Autotester_Weaver # 5981 FAILED (click to see last 100 lines of console output) 

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx/ctest-build/full_sp_debug
===============================================================================
Testing '\''4aced36c878e1aa6ed9984cca4fa873c3f4da82b'\'' for test '\''release'\''
===============================================================================
RUN: taskset -c 104-155 sh -c '\''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='\''\'\'''\''ON'\''\'\'''\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '\''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx/ctest-build/release
===============================================================================
Testing '\''4aced36c878e1aa6ed9984cca4fa873c3f4da82b'\'' for test '\''full_debug'\''
===============================================================================
RUN: taskset -c 0-51 sh -c '\''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='\''\'\'''\''ON'\''\'\'''\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '\''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx/ctest-build/full_debug
Build type full_debug failed at testing time. Here'\''s a list of failed tests:
110:mam4_wetscav_standalone_baseline_cmp
147:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp

Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

95:mam4_wetscav_standalone_baseline_cmp


Build type release failed at testing time. Here'''s a list of failed tests:

109:mam4_wetscav_standalone_baseline_cmp

146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx

weaver failed'


    

  • errors='Build type full_debug failed at testing time. Here'''s a list of failed tests:
    
110:mam4_wetscav_standalone_baseline_cmp
    
147:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
    • 



Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

95:mam4_wetscav_standalone_baseline_cmp


Build type release failed at testing time. Here'''s a list of failed tests:

109:mam4_wetscav_standalone_baseline_cmp

146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx

weaver failed'


    

  • SA_FAILURES_DETAILS+='Build type full_debug failed at testing time. Here'''s a list of failed tests:
    
110:mam4_wetscav_standalone_baseline_cmp
    
147:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
    • 



Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

95:mam4_wetscav_standalone_baseline_cmp


Build type release failed at testing time. Here'''s a list of failed tests:

109:mam4_wetscav_standalone_baseline_cmp

146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx

weaver failed'


    

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
######################################################
    
FAILS DETECTED:
    
SCREAM STANDALONE TESTING FAILED!
    
Build type full_debug failed at testing time. Here's a list of failed tests:
    
110:mam4_wetscav_standalone_baseline_cmp
    
147:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
    • 



Build type full_sp_debug failed at testing time. Here's a list of failed tests:

95:mam4_wetscav_standalone_baseline_cmp


Build type release failed at testing time. Here's a list of failed tests:

109:mam4_wetscav_standalone_baseline_cmp

146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5981/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le


cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins6795691498993623176.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

@tcclevenger
Copy link
Contributor Author

Investigating CPU fail for baseline tests.

@tcclevenger
Copy link
Contributor Author

@bartgol We get compile time warning about inlining functions without definition on CPU which I believe is because we are interfacing through eti/func_name.cpp, so I need to revert.

@bartgol
Copy link
Contributor

bartgol commented Aug 5, 2024

@bartgol We get compile time warning about inlining functions without definition on CPU which I believe is because we are interfacing through eti/func_name.cpp, so I need to revert.

You mean that there's no available definition, and the code is linked? How does that even work on GPU then, workout the relocatable device code flags?

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 5, 2024

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5755
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 400afca
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5983
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 400afca
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: tcclevenger/refactor_shoc_function
  • SHA: 400afca
  • Mode: TEST_REPO

Pull Request Author: tcclevenger

@tcclevenger
Copy link
Contributor Author

You mean that there's no available definition, and the code is linked? How does that even work on GPU then, workout the relocatable device code flags?

In shoc_functions.hpp, after all the function declarations, we have

#ifdef ENABLE_GPU and not RELOCATABLE_DEVICE
  #include func_name_impl.hpp
#endif

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 5, 2024

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5755
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 400afca
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5983
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 400afca
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d50836
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Mappy # 5755 FAILED (click to see last 100 lines of console output) 

324:spa_standalone_baseline_cmp
334:rrtmgp_standalone_not_chunked_np2_vs_np1
335:rrtmgp_standalone_not_chunked_np3_vs_np1
336:rrtmgp_standalone_not_chunked_np4_vs_np1
338:rrtmgp_standalone_chunked_vs_not_chunked
339:rrtmgp_standalone_chunked_baseline_cmp
344:cosp_standalone_baseline_cmp
349:mam4_optics_standalone_np2_vs_np1
350:mam4_optics_standalone_np3_vs_np1
351:mam4_optics_standalone_np4_vs_np1
352:mam4_optics_standalone_baseline_cmp
357:mam4_aci_standalone_np2_vs_np1
358:mam4_aci_standalone_np3_vs_np1
359:mam4_aci_standalone_np4_vs_np1
360:mam4_aci_standalone_baseline_cmp
365:mam4_wetscav_standalone_np2_vs_np1
366:mam4_wetscav_standalone_np3_vs_np1
367:mam4_wetscav_standalone_np4_vs_np1
368:mam4_wetscav_standalone_baseline_cmp
373:shoc_cld_p3_rrtmgp_np2_vs_np1
374:shoc_cld_p3_rrtmgp_np3_vs_np1
375:shoc_cld_p3_rrtmgp_np4_vs_np1
380:shoc_cld_spa_p3_rrtmgp_np2_vs_np1
381:shoc_cld_spa_p3_rrtmgp_np3_vs_np1
382:shoc_cld_spa_p3_rrtmgp_np4_vs_np1
387:mam_optics_rrtmgp_np2_vs_np1
388:mam_optics_rrtmgp_np3_vs_np1
389:mam_optics_rrtmgp_np4_vs_np1
394:shoc_mam4_aci_np2_vs_np1
395:shoc_mam4_aci_np3_vs_np1
396:shoc_mam4_aci_np4_vs_np1
401:shoc_cldfrac_mam4_aci_p3_np2_vs_np1
402:shoc_cldfrac_mam4_aci_p3_np3_vs_np1
403:shoc_cldfrac_mam4_aci_p3_np4_vs_np1
408:shoc_cldfrac_mam4_aci_p3_rrtmgp_np2_vs_np1
409:shoc_cldfrac_mam4_aci_p3_rrtmgp_np3_vs_np1
410:shoc_cldfrac_mam4_aci_p3_rrtmgp_np4_vs_np1
415:shoc_cldfrac_mam4_aci_p3_mam4_optics_rrtmgp_np2_vs_np1
416:shoc_cldfrac_mam4_aci_p3_mam4_optics_rrtmgp_np3_vs_np1
417:shoc_cldfrac_mam4_aci_p3_mam4_optics_rrtmgp_np4_vs_np1
422:p3_mam4_wetscav_np2_vs_np1
423:p3_mam4_wetscav_np3_vs_np1
424:p3_mam4_wetscav_np4_vs_np1
429:shoc_cldfrac_p3_wetscav_np2_vs_np1
430:shoc_cldfrac_p3_wetscav_np3_vs_np1
431:shoc_cldfrac_p3_wetscav_np4_vs_np1
434:check_subcycling
445:homme_shoc_cld_p3_rrtmgp_np2_vs_np1
446:homme_shoc_cld_p3_rrtmgp_np3_vs_np1
447:homme_shoc_cld_p3_rrtmgp_np4_vs_np1
448:homme_shoc_cld_p3_rrtmgp_baseline_cmp
453:homme_shoc_cld_p3_rrtmgp_pg2_np2_vs_np1
454:homme_shoc_cld_p3_rrtmgp_pg2_np3_vs_np1
455:homme_shoc_cld_p3_rrtmgp_pg2_np4_vs_np1
456:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
460:restarted_vs_monolithic_check_np4
465:homme_shoc_cld_spa_p3_rrtmgp_np2_vs_np1
466:homme_shoc_cld_spa_p3_rrtmgp_np3_vs_np1
467:homme_shoc_cld_spa_p3_rrtmgp_np4_vs_np1
468:homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
473:homme_shoc_cld_spa_p3_rrtmgp_128levels_np2_vs_np1
474:homme_shoc_cld_spa_p3_rrtmgp_128levels_np3_vs_np1
475:homme_shoc_cld_spa_p3_rrtmgp_128levels_np4_vs_np1
480:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
485:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np2_vs_np1
486:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np3_vs_np1
487:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np4_vs_np1
488:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
493:homme_shoc_cld_p3_mam_optics_rrtmgp_np2_vs_np1
494:homme_shoc_cld_p3_mam_optics_rrtmgp_np3_vs_np1
495:homme_shoc_cld_p3_mam_optics_rrtmgp_np4_vs_np1
496:homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
501:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np2_vs_np1
502:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np3_vs_np1
503:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np4_vs_np1
504:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 89402 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins9580798397327977532.sh

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE
SCREAM_PullRequest_Autotester_Weaver # 5983 FAILED (click to see last 100 lines of console output) 

mam4_wetscav        =  24.91 sec*proc (2 tests)
nudging             =  12.44 sec*proc (2 tests)
p3                  = 117.73 sec*proc (12 tests)
p3_sk               =  38.83 sec*proc (2 tests)
physics             = 188.83 sec*proc (25 tests)
remap               =   3.47 sec*proc (1 test)
rrtmgp              =  58.66 sec*proc (11 tests)
shoc                =  63.60 sec*proc (12 tests)
spa                 =  12.18 sec*proc (4 tests)
surface_coupling    =   4.45 sec*proc (1 test)

Total Test time (real) = 797.72 sec


Testing '''cc9136229e65f9e327ffc1c1e938674ab02ac994''' for test '''full_sp_debug'''


RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/ctest-build/full_sp_debug


Testing '''cc9136229e65f9e327ffc1c1e938674ab02ac994''' for test '''release'''


RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/ctest-build/release


Testing '''cc9136229e65f9e327ffc1c1e938674ab02ac994''' for test '''full_debug'''


RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx/ctest-build/full_debug

Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

95:mam4_wetscav_standalone_baseline_cmp


Build type release failed at testing time. Here'''s a list of failed tests:

109:mam4_wetscav_standalone_baseline_cmp

146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx

weaver failed'


    

  • errors='Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:
    
95:mam4_wetscav_standalone_baseline_cmp
    • 



Build type release failed at testing time. Here'''s a list of failed tests:

109:mam4_wetscav_standalone_baseline_cmp

146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx

weaver failed'


    

  • SA_FAILURES_DETAILS+='Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:
    
95:mam4_wetscav_standalone_baseline_cmp
    • 



Build type release failed at testing time. Here'''s a list of failed tests:

109:mam4_wetscav_standalone_baseline_cmp

146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx

weaver failed'


    

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
######################################################
    
FAILS DETECTED:
    
SCREAM STANDALONE TESTING FAILED!
    
Build type full_sp_debug failed at testing time. Here's a list of failed tests:
    
95:mam4_wetscav_standalone_baseline_cmp
    • 



Build type release failed at testing time. Here's a list of failed tests:

109:mam4_wetscav_standalone_baseline_cmp

146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5983/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le


cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins8228526322610333545.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

@bartgol
Copy link
Contributor

bartgol commented Aug 6, 2024

Oh, I see. Our ETI strategy is conflicting with the inline declaration. That makes sense. I wonder if ETI makes sense for such small functions...

@tcclevenger
Copy link
Contributor Author

ETI probably doesn't make sense for these. Considered just adding the definitions to the .hpp declaration, but wouldn't want to clutter it up.

@bartgol
Copy link
Contributor

bartgol commented Aug 6, 2024

ETI probably doesn't make sense for these. Considered just adding the definitions to the .hpp declaration, but wouldn't want to clutter it up.

You can simply include the impl files, but, unlike other functions, you don't guard with any ifdef, so they always get included. Then, you can mark them inline.

@tcclevenger tcclevenger force-pushed the tcclevenger/refactor_shoc_function branch from 183413e to 26fab5c Compare August 6, 2024 19:20
@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 6, 2024

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5762
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 26fab5c
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 48e2575
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5989
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 26fab5c
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 48e2575
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: tcclevenger/refactor_shoc_function
  • SHA: 26fab5c
  • Mode: TEST_REPO

Pull Request Author: tcclevenger

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 6, 2024

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5762
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 26fab5c
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 48e2575
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5989
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 26fab5c
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 48e2575
TEST_REPO_ALIAS SCREAM

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 6, 2024

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 6, 2024

All Jobs Finished; status = PASSED, target_sha=6011eebd8b61000a1480ca8e5d7c0492ea3d219c, However Inspection must be performed before merge can occur...

@tcclevenger tcclevenger requested a review from bartgol August 6, 2024 20:36
@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 6, 2024

The base branch has been updated since the last successful testing.

  • last PASS base branch sha: 6011eeb
  • current base branch sha : 3ce053c
    The AutoTester will discard the last PASS, and re-test the PR from scratch

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 6, 2024

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5763
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 26fab5c
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 48e2575
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5990
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 26fab5c
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 48e2575
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: tcclevenger/refactor_shoc_function
  • SHA: 26fab5c
  • Mode: TEST_REPO

Pull Request Author: tcclevenger

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 6, 2024

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5763
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 26fab5c
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 48e2575
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5990
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;shoc
PULLREQUESTNUM 2932
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 26fab5c
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 48e2575
TEST_REPO_ALIAS SCREAM

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 6, 2024

All Jobs Finished; status = PASSED, target_sha=3ce053caccc70824251d3fe8a15bb50b03d264c4, However Inspection must be performed before merge can occur...

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Aug 6, 2024

Status Flag 'Pull Request AutoTester' - AutoMerge IS ENABLED, but the Label AT: AUTOMERGE is not set. Either set Label AT: AUTOMERGE or manually merge the PR...

@E3SM-Bot E3SM-Bot merged commit 05a500e into master Aug 7, 2024
13 checks passed
@E3SM-Bot E3SM-Bot deleted the tcclevenger/refactor_shoc_function branch August 7, 2024 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bartgol bartgol bartgol approved these changes

@bogensch bogensch Awaiting requested review from bogensch

Assignees

@tcclevenger tcclevenger

Labels
BFB Bit for bit shoc regarding shoc macrophysics
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

SHOC assumed_pdf routine not converted to CPP as intended
3 participants
@tcclevenger @E3SM-Bot @bartgol