Fix and cleanup vertical layer diagnostic #2837

bartgol · 2024-05-20T23:32:39Z

A few mods were needed, since there was an inconsistency in the handling of the surface value, as well as on how this diagnostic was used in the FieldAtHeight diagnostic. Summary:

Fix usage of surface value (the b.c. for the vertical integral)
Added "height" as a version of "z" but from surface
Reworked the unit tests, so that they are simpler, and they do not just copy+paste the implementation of the class. In particular, no need to ensure calculate_dz works correctly, since it's already tested elsewhere. We just need to test that the diag computes the correct diagnostic type, and has the right behavior w.r.t. level index.

Also, one thing we should do in all the diagnostics: we are not free to request a pack size. By the time diags are created, all fields have been allocated, so the diag does not have a saying in pack size. Instead, the diag should defer the creation of the output field until all inputs are set (i.e.., do it inside initialize_impl), and pick as a pack size for the output whatever can work with all its inputs (i.e., the smallest among the pack sizes of the inputs).

* Surface value for vertical integral was wrong * Ensure geopotential is indeed a potential * Use altitude for elevation above surface

… extra data

…stic

* Rename altitude with height * Clean up includes * Clean up unit tests

E3SM-Bot · 2024-05-20T23:46:03Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5450
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`4b12127`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5728
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`4b12127`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Bot · 2024-05-21T00:10:03Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5450
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`4b12127`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5728
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`4b12127`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5450 FAILED (click to see last 100 lines of console output)


    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240520_174952_50gxm1
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240520_174952_50gxm1
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240520_174952_50gxm1
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240520_174952_50gxm1
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240520_174952_50gxm1
FAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240520_174952_50gxm1
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240520_174952_50gxm1
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240520_174952_50gxm1
test-scheduler took 678.629378080368 seconds'
+ errors='Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240520_174952_50gxm1
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240520_174952_50gxm1
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240520_174952_50gxm1
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240520_174952_50gxm1
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240520_174952_50gxm1
FAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240520_174952_50gxm1
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240520_174952_50gxm1
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240520_174952_50gxm1
test-scheduler took 678.629378080368 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240520_174952_50gxm1
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240520_174952_50gxm1
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240520_174952_50gxm1
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240520_174952_50gxm1
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240520_174952_50gxm1
FAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240520_174952_50gxm1
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240520_174952_50gxm1
FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240520_174952_50gxm1
test-scheduler took 678.629378080368 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
SCREAM V1 TESTING FAILED!

Waiting for tests to finish

FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase RUN)

Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240520_174952_50gxm1

FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase RUN)

Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240520_174952_50gxm1

FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase RUN)

Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240520_174952_50gxm1

FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase RUN)

Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240520_174952_50gxm1

PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN

Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240520_174952_50gxm1

FAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase RUN)

Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240520_174952_50gxm1

PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN

Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240520_174952_50gxm1

FAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase RUN)

Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240520_174952_50gxm1

test-scheduler took 678.629378080368 seconds

######################################################

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 86632 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins7165211047355103976.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5728 FAILED (click to see last 100 lines of console output)


      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ^
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/YAKL/src/extensions/YAKL_netcdf.h: In member function '\''yakl::SimpleNetCDF::NcDim yakl::SimpleNetCDF::NcVar::getDim(int) const'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/YAKL/src/extensions/YAKL_netcdf.h:158:38: warning: comparison of integer expressions of different signedness: '\''std::vector::size_type'\'' {aka '\''long unsigned int'\''} and '\''int'\'' [-Wsign-compare]
  158 |         if (isNull() || dims.size() <= i) {
      |                        ~~~~~~~~~~~~~~^~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/YAKL/src/extensions/YAKL_netcdf.h: In member function '\''void yakl::SimpleNetCDF::NcVar::print()'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/YAKL/src/extensions/YAKL_netcdf.h:213:19: warning: comparison of integer expressions of different signedness: '\''int'\'' and '\''std::vector::size_type'\'' {aka '\''long unsigned int'\''} [-Wsign-compare]
  213 |         for (int i=0; i < dims.size(); i++) {
      |                 ~~^~~~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/YAKL/src/extensions/YAKL_netcdf.h: In member function '\''yakl::SimpleNetCDF::NcVar yakl::SimpleNetCDF::NcFile::addVar(std::string, int, std::vector&)'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/YAKL/src/extensions/YAKL_netcdf.h:302:19: warning: comparison of integer expressions of different signedness: '\''int'\'' and '\''std::vector::size_type'\'' {aka '\''long unsigned int'\''} [-Wsign-compare]
  302 |         for (int i=0; i < dims.size(); i++) { dimids[i] = dims[i].getId(); }
      |                 ~~^~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/YAKL/src/extensions/YAKL_netcdf.h: In instantiation of '\''void yakl::SimpleNetCDF::read(yakl::Array&, std::string) [with T = double; int rank = 2; int myMem = 1; int myStyle = 2; std::string = std::__cxx11::basic_string]'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx/src/physics/rrtmgp/rrtmgp_test_utils.cpp:205:8:   required from here
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/YAKL/src/extensions/YAKL_netcdf.h:531:19: warning: comparison of integer expressions of different signedness: '\''int'\'' and '\''std::vector::size_type'\'' {aka '\''long unsigned int'\''} [-Wsign-compare]
  531 |           for (int i=0; i < varDims.size(); i++) { dimSizes[i] = varDims[varDims.size()-1-i].getSize(); }
      |                 ~~^~~~~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/YAKL/src/extensions/YAKL_netcdf.h:535:19: warning: comparison of integer expressions of different signedness: '\''int'\'' and '\''std::vector::size_type'\'' {aka '\''long unsigned int'\''} [-Wsign-compare]
  535 |           for (int i=0; i < dimSizes.size(); i++) {
      |                 ~~^~~~~~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/YAKL/src/extensions/YAKL_netcdf.h:536:17: warning: comparison of integer expressions of different signedness: '\''__gnu_cxx::__alloc_traits, int>::value_type'\'' {aka '\''int'\''} and '\''yakl::index_t'\'' {aka '\''long unsigned int'\''} [-Wsign-compare]
  536 |             if (dimSizes[i] != arr.extent(i)) {
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/YAKL/src/extensions/YAKL_netcdf.h: In instantiation of '\''void yakl::SimpleNetCDF::write(const yakl::Array&, std::string, std::vector >) [with T = double; int rank = 2; int myMem = 1; int myStyle = 2; std::string = std::__cxx11::basic_string]'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx/src/physics/rrtmgp/rrtmgp_test_utils.cpp:248:9:   required from here
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/YAKL/src/extensions/YAKL_netcdf.h:406:19: warning: comparison of integer expressions of different signedness: '\''int'\'' and '\''std::vector::size_type'\'' {aka '\''long unsigned int'\''} [-Wsign-compare]
  406 |         for (int i=0; i < varDims.size(); i++) {
      |                 ~~^~~~~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx/src/dynamics/homme/eamxx_homme_process_interface.cpp: In static member function '\''static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::HommeDynamics::restart_homme_state()::; Tag = __nv_dl_tag; OpFuncR = void; OpFuncArgs = {int}; F1 = const int; F2 = const int; F3 = const int; F4 = Kokkos::View*****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F5 = Kokkos::View*****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F6 = Kokkos::View****, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >]'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx/src/dynamics/homme/eamxx_homme_process_interface.cpp:992:569: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  992 |   Kokkos::parallel_for(Kokkos::RangePolicy<>(0,nelem*qsize*NGP*NGP*npacks),
      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ^
nvcc_internal_extended_lambda_implementation: In static member function '\''static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = rrtmgpTest::dummy_clouds(CloudOptics&, real2d&, real2d&, real2d&, real2d&, real2d&, real2d&, real2d&)::; Tag = __nv_dl_tag&, yakl::Array&, yakl::Array&, yakl::Array&, yakl::Array&, yakl::Array&, yakl::Array&), rrtmgpTest::dummy_clouds, 1>; OpFuncR = void; OpFuncArgs = {int, int}; F1 = yakl::Array; F2 = yakl::Array; F3 = yakl::Array; F4 = yakl::Array; F5 = yakl::Array; F6 = yakl::Array; F7 = double; F8 = yakl::Array; F9 = double]'\'':
nvcc_internal_extended_lambda_implementation:955:19: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx/src/physics/rrtmgp/tests/generate_baseline.cpp:2:10: fatal error: physics/rrtmgp/mo_garand_atmos_io.h: No such file or directory
    2 | #include "physics/rrtmgp/mo_garand_atmos_io.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [src/physics/rrtmgp/tests/CMakeFiles/generate_baseline.dir/build.make:76: src/physics/rrtmgp/tests/CMakeFiles/generate_baseline.dir/generate_baseline.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:3954: src/physics/rrtmgp/tests/CMakeFiles/generate_baseline.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx/src/physics/spa/tests/spa_main_test.cpp: In lambda function:
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx/src/physics/spa/tests/spa_main_test.cpp:202:72: note: parameter passing for argument of type '\''std::pair'\'' when C++17 is enabled changed to match C++14 in GCC 10.1
  202 |   auto max_min = [&] (const Real v0, const Real v1) -> std::pair {
      |                                                                        ^~~~~~               
nvcc_internal_extended_lambda_implementation: In static member function '\''static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::spa::SPAFunctions >::perform_vertical_interpolation(scream::spa::SPAFunctions >::view_2d >&, scream::spa::SPAFunctions >::view_2d >&, const scream::spa::SPAFunctions >::SPAData&, const scream::spa::SPAFunctions >::SPAData&)::; Tag = __nv_dl_tag**, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >&, const Kokkos::View**, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >&, const scream::spa::SPAFunctions >::SPAData&, const scream::spa::SPAFunctions >::SPAData&), scream::spa::SPAFunctions >::perform_vertical_interpolation, 1>; OpFuncR = void; OpFuncArgs = {const Kokkos::Impl::CudaTeamMember&}; F1 = ekat::LinInterp >; F2 = const Kokkos::View**, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F3 = const Kokkos::View**, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >]'\'':
nvcc_internal_extended_lambda_implementation:337:19: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
nvcc_internal_extended_lambda_implementation: In member function '\''void scream::control::IntensiveObservationPeriod::set_fields_from_iop_data(scream::control::IntensiveObservationPeriod::field_mgr_ptr)'\'':
nvcc_internal_extended_lambda_implementation:1311:339: warning: '\''ps_iop'\'' may be used uninitialized in this function [-Wmaybe-uninitialized]
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx/src/control/intensive_observation_period.cpp:773:6: note: '\''ps_iop'\'' was declared here
  773 |   Real ps_iop;
      |      ^~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp: In copy constructor '\''Kokkos::MDRangePolicy, Kokkos::IndexType >::MDRangePolicy(const Kokkos::MDRangePolicy, Kokkos::IndexType >&)'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp:185:8: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  185 | struct MDRangePolicy : public Kokkos::Impl::PolicyTraits {
      |        ^~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx/src/share/field/field_alloc_prop.hpp: In copy constructor '\''scream::FieldAllocProp::FieldAllocProp(const scream::FieldAllocProp&)'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx/src/share/field/field_alloc_prop.hpp:89:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
   89 |   FieldAllocProp (const FieldAllocProp&) = default;
      | ^ ~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp: In copy constructor '\''Kokkos::MDRangePolicy, Kokkos::IndexType >::MDRangePolicy(const Kokkos::MDRangePolicy, Kokkos::IndexType >&)'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp:185:8: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  185 | struct MDRangePolicy : public Kokkos::Impl::PolicyTraits {
      |        ^~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx/src/share/field/field_alloc_prop.hpp: In copy constructor '\''scream::FieldAllocProp::FieldAllocProp(const scream::FieldAllocProp&)'\'':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx/src/share/field/field_alloc_prop.hpp:89:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
   89 |   FieldAllocProp (const FieldAllocProp&) = default;
      | ^ ~~~~~~~~~~~~
make: *** [Makefile:166: all] Error 2
Error(s) occurred during baselines generation phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5728/scream/components/eamxx

weaver failed'

errors=
SA_FAILURES_DETAILS+=
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

######################################################

FAILS DETECTED:

SCREAM STANDALONE TESTING FAILED!

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins7391933746164579448.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

AaronDonahue

Thank you for putting this fix in so quickly, and improving the functionality quite a bit. I had a few comments, but for the most part I think this is nearly ready to go in.

components/eamxx/src/diagnostics/tests/vertical_layer_tests.cpp

components/eamxx/src/diagnostics/field_at_height.cpp

AaronDonahue · 2024-05-21T21:05:22Z

components/eamxx/src/share/io/scream_io_utils.hpp

@@ -80,9 +80,9 @@ struct LongNames {
  std::map<std::string,std::string> name_2_longname = {


Would it make sense to do away with this approach to long names altogether and just use the same syntax for hya, hyb, lev, etc. as we do for the diagnostics? I.e. add longname as metadata?

Probably. But we can do it in a separate PR.

E3SM-Bot · 2024-05-21T22:15:13Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5454
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`2eff286`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5732
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`2eff286`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Bot · 2024-05-21T22:54:24Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5454
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`2eff286`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5732
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`2eff286`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5454 FAILED (click to see last 100 lines of console output)


Starting MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 55.230006 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 71.625155 seconds (PASS)
Starting MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 16 procs
Finished SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 70.039380 seconds (PASS)
Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 80.709942 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 171.799680 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 196.249387 seconds (PASS)
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 172.175631 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 with 1 proc on interactive node and 64 procs on compute nodes
Finished MODEL_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 173.877849 seconds (PASS)
Starting RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 1.092724 seconds (PEND). [COMPLETED 1 of 8]
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Finished RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 0.985940 seconds (PEND). [COMPLETED 2 of 8]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 16 procs
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 175.877608 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 0.985550 seconds (PEND). [COMPLETED 3 of 8]
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Finished MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 316.881976 seconds (PASS)
Starting RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 1 proc on interactive node and 16 procs on compute nodes
Finished RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 7.553532 seconds (PEND). [COMPLETED 4 of 8]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 298.017619 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 0.934756 seconds (PEND). [COMPLETED 5 of 8]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 377.699940 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 1.093292 seconds (PEND). [COMPLETED 6 of 8]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 459.847448 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 1.078438 seconds (PEND). [COMPLETED 7 of 8]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 546.643005 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 0.938480 seconds (PEND). [COMPLETED 8 of 8]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240521_162852_rv7opd
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240521_162852_rv7opd
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240521_162852_rv7opd
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240521_162852_rv7opd
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240521_162852_rv7opd
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240521_162852_rv7opd
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240521_162852_rv7opd
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240521_162852_rv7opd
test-scheduler took 816.8661646842957 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at testing time. Here's a list of failed tests:
244:field_at_height
275:p3_standalone_np1
276:p3_standalone_np2
277:p3_standalone_np3
278:p3_standalone_np4
279:p3_standalone_np2_vs_np1
280:p3_standalone_np3_vs_np1
281:p3_standalone_np4_vs_np1
282:p3_tend_check_np1
283:p3_tend_check_np2
284:p3_tend_check_np3
285:p3_tend_check_np4
286:p3_standalone_baseline_cmp
392:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
416:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 76635 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins8122466755501452020.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5732 FAILED (click to see last 100 lines of console output)


119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

50:field_at_height

73:p3_standalone_np1

74:p3_tend_check_np1

75:p3_standalone_baseline_cmp
Build type release failed at testing time. Here'''s a list of failed tests:

55:field_at_height

80:p3_standalone_np1

81:p3_tend_check_np1

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5732/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5732/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5732/scream/components/eamxx

weaver failed'

SA_FAILURES_DETAILS+='Build type full_debug failed at testing time. Here'''s a list of failed tests:

55:field_at_height

80:p3_standalone_np1

81:p3_tend_check_np1

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp

Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

50:field_at_height

73:p3_standalone_np1

74:p3_tend_check_np1

75:p3_standalone_baseline_cmp
Build type release failed at testing time. Here'''s a list of failed tests:

55:field_at_height

80:p3_standalone_np1

81:p3_tend_check_np1

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5732/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5732/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5732/scream/components/eamxx

weaver failed'

[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

######################################################

FAILS DETECTED:

SCREAM STANDALONE TESTING FAILED!

Build type full_debug failed at testing time. Here's a list of failed tests:

55:field_at_height

80:p3_standalone_np1

81:p3_tend_check_np1

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp

Build type full_sp_debug failed at testing time. Here's a list of failed tests:

50:field_at_height

73:p3_standalone_np1

74:p3_tend_check_np1

75:p3_standalone_baseline_cmp
Build type release failed at testing time. Here's a list of failed tests:

55:field_at_height

80:p3_standalone_np1

81:p3_tend_check_np1

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5732/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5732/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5732/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins18014386292220375598.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

E3SM-Bot · 2024-05-22T15:17:17Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5457
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`200d84e`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5735
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`200d84e`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Bot · 2024-05-22T15:56:06Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5457
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`200d84e`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5735
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`200d84e`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5457 FAILED (click to see last 100 lines of console output)


Finished SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 55.070781 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 55.341020 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 70.964316 seconds (PASS)
Starting MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 16 procs
Finished SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 70.089535 seconds (PASS)
Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 78.348285 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 168.323542 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 195.187907 seconds (PASS)
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 174.816785 seconds (PASS)
Starting RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 1.044308 seconds (PEND). [COMPLETED 1 of 8]
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 177.325084 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 0.979847 seconds (PEND). [COMPLETED 2 of 8]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 16 procs
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 179.768168 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 1.132824 seconds (PEND). [COMPLETED 3 of 8]
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Finished MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 299.061915 seconds (PASS)
Starting RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 1 proc on interactive node and 16 procs on compute nodes
Finished RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 4.320375 seconds (PEND). [COMPLETED 4 of 8]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 302.222007 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 0.926057 seconds (PEND). [COMPLETED 5 of 8]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 349.183552 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 0.955631 seconds (PEND). [COMPLETED 6 of 8]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 449.007728 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 1.145098 seconds (PEND). [COMPLETED 7 of 8]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 524.328461 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 1.080479 seconds (PEND). [COMPLETED 8 of 8]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240522_093023_6y5wwa
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240522_093023_6y5wwa
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240522_093023_6y5wwa
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240522_093023_6y5wwa
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240522_093023_6y5wwa
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240522_093023_6y5wwa
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240522_093023_6y5wwa
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240522_093023_6y5wwa
test-scheduler took 799.2406249046326 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at testing time. Here's a list of failed tests:
275:p3_standalone_np1
276:p3_standalone_np2
277:p3_standalone_np3
278:p3_standalone_np4
279:p3_standalone_np2_vs_np1
280:p3_standalone_np3_vs_np1
281:p3_standalone_np4_vs_np1
282:p3_tend_check_np1
283:p3_tend_check_np2
284:p3_tend_check_np3
285:p3_tend_check_np4
286:p3_standalone_baseline_cmp
392:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
416:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 65442 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins18382407210291459634.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5735 FAILED (click to see last 100 lines of console output)


Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5735/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5735/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5735/scream/components/eamxx
weaver failed'
+ errors='Build type full_debug failed at testing time. Here'\''s a list of failed tests:
80:p3_standalone_np1
81:p3_tend_check_np1
82:p3_standalone_baseline_cmp
119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

73:p3_standalone_np1

74:p3_tend_check_np1

75:p3_standalone_baseline_cmp
Build type release failed at testing time. Here'''s a list of failed tests:

80:p3_standalone_np1

81:p3_tend_check_np1

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5735/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5735/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5735/scream/components/eamxx

weaver failed'

SA_FAILURES_DETAILS+='Build type full_debug failed at testing time. Here'''s a list of failed tests:

80:p3_standalone_np1

81:p3_tend_check_np1

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp

Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

73:p3_standalone_np1

74:p3_tend_check_np1

75:p3_standalone_baseline_cmp
Build type release failed at testing time. Here'''s a list of failed tests:

80:p3_standalone_np1

81:p3_tend_check_np1

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5735/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5735/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5735/scream/components/eamxx

weaver failed'

[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

######################################################

FAILS DETECTED:

SCREAM STANDALONE TESTING FAILED!

Build type full_debug failed at testing time. Here's a list of failed tests:

80:p3_standalone_np1

81:p3_tend_check_np1

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp

Build type full_sp_debug failed at testing time. Here's a list of failed tests:

73:p3_standalone_np1

74:p3_tend_check_np1

75:p3_standalone_baseline_cmp
Build type release failed at testing time. Here's a list of failed tests:

80:p3_standalone_np1

81:p3_tend_check_np1

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5735/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5735/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5735/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins95458278904361488.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

Requesting field above sealevel requires phis, which this test doesn't have

bartgol · 2024-05-22T19:14:28Z

@AaronDonahue I fixed the only issue left, the p3_standalone test. The baseline_cmp tests are expected to fail, since we fixed the bug in the vertical integral. If on the next AT run the only fails are the baseline_cmp tests, we're good to merge if you don't have any more comments.

E3SM-Bot · 2024-05-22T19:47:15Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5458
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`6bb577c`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5736
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`6bb577c`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Bot · 2024-05-22T20:27:12Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5458
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`6bb577c`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5736
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: AUTOMERGE;bugfix;diagnostic
PULLREQUESTNUM	2837
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`6bb577c`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`f5b824a`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5458 FAILED (click to see last 100 lines of console output)


Starting SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 1 procs
Finished SETUP for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 2.586625 seconds (PASS)
Finished SETUP for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 2.597872 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 procs
Starting SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 procs
Finished SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 2.068000 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 1 procs
Finished SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 2.531631 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 1 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 55.638204 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics with 16 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 55.203739 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 55.544454 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 71.588371 seconds (PASS)
Starting MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 16 procs
Finished SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 70.018759 seconds (PASS)
Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 80.181047 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 171.058584 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 196.482547 seconds (PASS)
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 172.790151 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 0.993317 seconds (PEND). [COMPLETED 1 of 8]
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 176.840637 seconds (PASS)
Starting RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics with 1 proc on interactive node and 64 procs on compute nodes
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 172.828001 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 0.947905 seconds (PEND). [COMPLETED 2 of 8]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 16 procs
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 0.951119 seconds (PEND). [COMPLETED 3 of 8]
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Finished MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 313.949582 seconds (PASS)
Starting RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 1 proc on interactive node and 16 procs on compute nodes
Finished RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 2.483071 seconds (PEND). [COMPLETED 4 of 8]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 316.091765 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 3.113427 seconds (PEND). [COMPLETED 5 of 8]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 334.897713 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 0.937182 seconds (PEND). [COMPLETED 6 of 8]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 436.942297 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 1.077209 seconds (PEND). [COMPLETED 7 of 8]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 520.681392 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 0.924875 seconds (PEND). [COMPLETED 8 of 8]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240522_140028_qw32pm
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240522_140028_qw32pm
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240522_140028_qw32pm
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240522_140028_qw32pm
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240522_140028_qw32pm
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240522_140028_qw32pm
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240522_140028_qw32pm
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240522_140028_qw32pm
test-scheduler took 793.841254234314 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at testing time. Here's a list of failed tests:
286:p3_standalone_baseline_cmp
392:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
416:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 122043 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins5199681147962970585.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5736 FAILED (click to see last 100 lines of console output)


===============================================================================
RUN: taskset -c 0-51 sh -c '\''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='\''\'\'''\''ON'\''\'\'''\'' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '\''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx/ctest-build/full_debug
Build type full_debug failed at testing time. Here'\''s a list of failed tests:
82:p3_standalone_baseline_cmp
119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

75:p3_standalone_baseline_cmp
Build type release failed at testing time. Here'''s a list of failed tests:

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx

weaver failed'

errors='Build type full_debug failed at testing time. Here'''s a list of failed tests:

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp

Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

75:p3_standalone_baseline_cmp
Build type release failed at testing time. Here'''s a list of failed tests:

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx

weaver failed'

SA_FAILURES_DETAILS+='Build type full_debug failed at testing time. Here'''s a list of failed tests:

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp

Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

75:p3_standalone_baseline_cmp
Build type release failed at testing time. Here'''s a list of failed tests:

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx

weaver failed'

[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

######################################################

FAILS DETECTED:

SCREAM STANDALONE TESTING FAILED!

Build type full_debug failed at testing time. Here's a list of failed tests:

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp

Build type full_sp_debug failed at testing time. Here's a list of failed tests:

75:p3_standalone_baseline_cmp
Build type release failed at testing time. Here's a list of failed tests:

82:p3_standalone_baseline_cmp

119:homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp

128:homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5736/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins7974152702785946132.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

AaronDonahue

@bartgol , once you confirm the fails in the AT are the expected ones go ahead and manually merge.

bartgol · 2024-05-28T17:35:51Z

The failures are the expected ones. Merging.

bartgol added 6 commits May 20, 2024 14:23

EAMxx: fix VerticalLayer diagnostic

ede1ab9

* Surface value for vertical integral was wrong * Ensure geopotential is indeed a potential * Use altitude for elevation above surface

EAMxx: use the LongNames util only if "long_name" is not in the field…

08f484f

… extra data

EAMxx: use field extra data to set long_name for VerticalLayer diagno…

0fcadb8

…stic

EAMxx: simplify unit testing for VerticalLayer

0da3b22

EAMxx: cleanup the VerticalLayer diagnostic

2f4be63

* Rename altitude with height * Clean up includes * Clean up unit tests

EAMxx: fix pack size determination for VerticalLayer diag

4b12127

bartgol added AT: AUTOMERGE bugfix diagnostic labels May 20, 2024

bartgol requested review from AaronDonahue and tcclevenger May 20, 2024 23:32

bartgol self-assigned this May 20, 2024

AaronDonahue reviewed May 21, 2024

View reviewed changes

EAMxx: fix comment and altitude->height renaming

2eff286

EAMxx: fix field_at_height unit test

200d84e

EAMxx: change output field for p3_standalone

6bb577c

Requesting field above sealevel requires phis, which this test doesn't have

AaronDonahue approved these changes May 28, 2024

View reviewed changes

bartgol merged commit 6557feb into master May 28, 2024
5 of 6 checks passed

bartgol deleted the bartgol/eamxx/fix-vert-lev-diag branch May 28, 2024 17:36

		@@ -80,9 +80,9 @@ struct LongNames {
		std::map<std::string,std::string> name_2_longname = {

Fix and cleanup vertical layer diagnostic #2837

Fix and cleanup vertical layer diagnostic #2837

Conversation

bartgol commented May 20, 2024

E3SM-Bot commented May 20, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Bot commented May 21, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

AaronDonahue left a comment

Choose a reason for hiding this comment

AaronDonahue May 21, 2024

Choose a reason for hiding this comment

bartgol May 21, 2024

Choose a reason for hiding this comment

E3SM-Bot commented May 21, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Bot commented May 21, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

E3SM-Bot commented May 22, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Bot commented May 22, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

bartgol commented May 22, 2024

E3SM-Bot commented May 22, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Bot commented May 22, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

AaronDonahue left a comment

Choose a reason for hiding this comment

bartgol commented May 28, 2024