EAMxx: enable kokkos bounds checks in full_debug standalone build

[bartgol]

Fixes #2828

[jgfouca]

Several of the other TestProperty subclasses use the DBG.CMAKE_ARGS list.

SP: DBG.CMAKE_ARGS + [("SCREAM_DOUBLE_PRECISION", "False")],
FPE: DBG.CMAKE_ARGS + [("SCREAM_PACK_SIZE", "1"), ("SCREAM_FPE","True")]

So this turns on bounds checking for DBG, SP, and FPE. Is that the intent?

[bartgol]

Ah, no I didn't want to do that. Although, should we? I def don't want it on in the valg build, so I need to edit that. I would argue that bounds can be checked in just one build, since access patterns do not depend on precision or FPE handling... I will make sure other builds don't inherit that.

[jgfouca]

Probably not. You can just explicitly disable it in the SP and FPE tests.

[jgfouca]

VALG does not use DBG.CMAKE_ARGS, so we are fine there.

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

• Build Num: 5435
• Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	d1e7f05
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

• Build Num: 5714
• Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	d1e7f05
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

• Branch: bartgol/eamxx/kokkos-enable-bounds-checks
• SHA: d1e7f05
• Mode: TEST_REPO

Pull Request Author: bartgol

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Error: Jenkins Jobs - A user has pushed a change to the PR before testing completed. NEW EVENT 'committed', ID C_kwDOCEfuetoAKDY4OTE3MThhMWY3ZmFkZGZjNDI2MzVlNWM3MjUyY2Y0MGY2ZmZhYjA... The Jenkins Jobs will be shutdown; Testing of this PR must occur again.

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

• Build Num: 5435
• Status: ERROR

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	d1e7f05
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

• Build Num: 5714
• Status: ERROR

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	d1e7f05
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5435 ERROR (click to see last 100 lines of console output)

test_opt_details (__main__.TestTestAllScream)
Test the 'opt' test in test-all-scream. It should set certain CMake values ... ok
test_pylint (__main__.TestTestAllScream) ... ok
test_run_fail_captured (__main__.TestTestAllScream)
Test that a failure from ctest in the testing phase is caught by test-all-scream ... ok
test_sp_details (__main__.TestTestAllScream)
Test the 'sp' test in test-all-scream. It should set certain CMake values ... ok
test_doctests (__main__.TestUtils) ... ok
test_full (__main__.TestUtils) ... ok
test_git_utils (__main__.TestUtils) ... ok
test_pylint (__main__.TestUtils) ... ok

Ran 33 tests in 231.853s
OK (skipped=4)

Warning: repo is not clean: M foo

[[ 0 != 0 ]]
./scripts/cime-nml-tests

test_append (main.TestBuildnml)

Test that var+=value syntax behaves as expected ... ok

test_atmchanges_are_lost_with_hack_xml (main.TestBuildnml)

Test that atmchanges are lost if SCREAM_HACK_XML=TRUE ... ok

test_atmchanges_are_preserved (main.TestBuildnml)

Test that atmchanges are not lost when eamxx setup is called ... ok

test_atmchanges_are_preserved_testmod (main.TestBuildnml)

Test that atmchanges via testmod are preserved ... ok

test_atmchanges_for_atm_procs_add (main.TestBuildnml)

Test atmchanges that add atm procs ... ok

test_atmchanges_for_atm_procs_add_group (main.TestBuildnml)

Test atmchanges that add atm proc groups ... ok

test_atmchanges_for_atm_procs_add_invalid (main.TestBuildnml)

Test atmchanges that add atm procs ... ok

test_atmchanges_for_atm_procs_remove (main.TestBuildnml)

Test atmchanges that remove atm procs ... ok

test_atmchanges_on_all_matches (main.TestBuildnml)

Test that atmchange --all works ... ok

test_atmchanges_on_all_matches_plus_spec (main.TestBuildnml)

Test atmchange --all followed by an atmchange of one of them ... ok

test_atmchanges_on_arrays (main.TestBuildnml)

Test that atmchange works for array data ... ok

test_atmchanges_with_namespace (main.TestBuildnml)

Test that atmchange works when using 'namespace' syntax foo::bar ... ok

test_buffer_unchanged_with_bad_change_syntax (main.TestBuildnml)

Test atmchange does not change buffer if syntax was wrong ... ok

test_doctests (main.TestBuildnml)

Run doctests for all eamxx/cime_config python files and nml-related files in scripts ... ok

test_invalid_xml_option (main.TestBuildnml)

Test atmchange errors out with invalid param names ... ok

test_nml_defaults_append (main.TestBuildnml)

Test that the append attribute for array-type params in namelist defaults works as expected ... ok

test_pylint (main.TestBuildnml)

Run pylint on all eamxx/cime_config python files and nml-related files in scripts ... ok

test_reset_atmchanges_are_lost (main.TestBuildnml)

Test that atmchanges are lost when resetting ... ok

test_xmlchange_propagates_to_atmconfig (main.TestBuildnml)

Test that xmlchanges impact atm config files ... ok


Ran 19 tests in 212.487s
OK

[[ 0 != 0 ]]
[[ -z '' ]]
[[ 1 == 1 ]]
[[ 0 == 1 ]]
[[ 0 == 1 ]]
[[ 0 == 1 ]]
[[ 0 == 1 ]]
echo 'SCREAM v1 tests were skipped, since the Github label '''AT: Skip v1 Testing''' was found.\n'

SCREAM v1 tests were skipped, since the Github label 'AT: Skip v1 Testing' was found.\n
set +x

######################################################

FAILS DETECTED:

SCREAM STANDALONE TESTING FAILED!

######################################################

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 107606 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins13835523482247896020.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5714 ERROR (click to see last 100 lines of console output)

+++ export _ModuleTable002_
+++ _ModuleTable_Sz_=2
+++ export _ModuleTable_Sz_
++ SCREAM_MACHINE=weaver
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
++ whoami
+ [[ e3sm-jenkins == \e\3\s\m\-\j\e\n\k\i\n\s ]]
+ git config --local user.email [email protected]
+ git config --local user.name 'Jenkins Jenkins'
+ declare -i fails=0
+ BASELINES_DIR=AUTO
+ TAS_ARGS='--baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m $machine'
+ [[ weaver == \p\m\-\g\p\u ]]
+ set +e
+ '[' -n 2830 ']'
+ is_at_run=1
+ SA_FAILURES_DETAILS=
+ '[' 1 -eq 1 ']'
++ ./scripts/gather-all-data './scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m $machine' -l -m weaver
***Forced exclusive execution
<>
<>
Build was aborted
Aborted by Luca Bertagna
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins6210184384182344508.sh

this_output='Job <73131> is submitted to queue .

Note: no value passed for --make-parallel-level. Using the default for this machine: 160

Note: no value passed for --ctest-parallel-level. Using the default for this machine: 4

Test full_debug can use 52 jobs to compile, and 1 jobs for test

Test full_sp_debug can use 52 jobs to compile, and 1 jobs for test

Test release can use 52 jobs to compile, and 1 jobs for test

RUN: { source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && export CTEST_PARALLEL_LEVEL=4;  } > /dev/null && env | sort

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5714/scream/components/eamxx

Last commit on ref '''b9a62cfef71a1ee753589df198eecec16382ff6e''': b9a62cfef7 Merge commit '''d1e7f053ea4a8f1bd6c6d23e2d99eae0fc25675a''' into HEAD

Checking baselines directory: /home/projects/e3sm/scream/pr-autotester/master-baselines/weaver

-> Test full_debug appears to have baselines

-> Test full_sp_debug appears to have baselines

-> Test release appears to have baselines

-> Test full_debug baselines are expired because origin/master is ahead of the existing baseline commit 335335f22192598f13005fda97f391ad3a8a1c72 by 2

-> Test full_sp_debug baselines are expired because origin/master is ahead of the existing baseline commit 335335f22192598f13005fda97f391ad3a8a1c72 by 2

-> Test release baselines are expired because origin/master is ahead of the existing baseline commit 335335f22192598f13005fda97f391ad3a8a1c72 by 2

Merge of '''origin/master''' not necessary. Current HEAD is already ahead.

###############################################################################

Generating baselines from git ref origin/master

###############################################################################

Terminated

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5714/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5714/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5714/scream/components/eamxx

weaver failed'
[[ 1 != 0 ]]
fails=0+1
sa_fail=1
[[ 1 == 1 ]]

++ echo 'Job <73131> is submitted to queue .

Note: no value passed for --make-parallel-level. Using the default for this machine: 160

Note: no value passed for --ctest-parallel-level. Using the default for this machine: 4

Test full_debug can use 52 jobs to compile, and 1 jobs for test

Test full_sp_debug can use 52 jobs to compile, and 1 jobs for test

Test release can use 52 jobs to compile, and 1 jobs for test

RUN: { source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && export CTEST_PARALLEL_LEVEL=4;  } > /dev/null && env | sort

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5714/scream/components/eamxx

Last commit on ref '''b9a62cfef71a1ee753589df198eecec16382ff6e''': b9a62cfef7 Merge commit '''d1e7f053ea4a8f1bd6c6d23e2d99eae0fc25675a''' into HEAD

Checking baselines directory: /home/projects/e3sm/scream/pr-autotester/master-baselines/weaver

-> Test full_debug appears to have baselines

-> Test full_sp_debug appears to have baselines

-> Test release appears to have baselines

-> Test full_debug baselines are expired because origin/master is ahead of the existing baseline commit 335335f22192598f13005fda97f391ad3a8a1c72 by 2

-> Test full_sp_debug baselines are expired because origin/master is ahead of the existing baseline commit 335335f22192598f13005fda97f391ad3a8a1c72 by 2

-> Test release baselines are expired because origin/master is ahead of the existing baseline commit 335335f22192598f13005fda97f391ad3a8a1c72 by 2

Merge of '''origin/master''' not necessary. Current HEAD is already ahead.

###############################################################################

Generating baselines from git ref origin/master

###############################################################################

Terminated

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5714/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5714/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5714/scream/components/eamxx

weaver failed'

++ grep -m1 -A 100000 'Build type '
errors=
SA_FAILURES_DETAILS+=
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

######################################################

FAILS DETECTED:

SCREAM STANDALONE TESTING FAILED!

######################################################

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

• Build Num: 5436
• Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	6891718
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

• Build Num: 5715
• Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	6891718
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

• Branch: bartgol/eamxx/kokkos-enable-bounds-checks
• SHA: 6891718
• Mode: TEST_REPO

Pull Request Author: bartgol

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

• Build Num: 5436
• Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	6891718
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

• Build Num: 5715
• Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	6891718
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5436 FAILED (click to see last 100 lines of console output)

Test that var+=value syntax behaves as expected ... ok
test_atmchanges_are_lost_with_hack_xml (__main__.TestBuildnml)
Test that atmchanges are lost if SCREAM_HACK_XML=TRUE ... ok
test_atmchanges_are_preserved (__main__.TestBuildnml)
Test that atmchanges are not lost when eamxx setup is called ... ok
test_atmchanges_are_preserved_testmod (__main__.TestBuildnml)
Test that atmchanges via testmod are preserved ... ok
test_atmchanges_for_atm_procs_add (__main__.TestBuildnml)
Test atmchanges that add atm procs ... ok
test_atmchanges_for_atm_procs_add_group (__main__.TestBuildnml)
Test atmchanges that add atm proc groups ... ok
test_atmchanges_for_atm_procs_add_invalid (__main__.TestBuildnml)
Test atmchanges that add atm procs ... ok
test_atmchanges_for_atm_procs_remove (__main__.TestBuildnml)
Test atmchanges that remove atm procs ... ok
test_atmchanges_on_all_matches (__main__.TestBuildnml)
Test that atmchange --all works ... ok
test_atmchanges_on_all_matches_plus_spec (__main__.TestBuildnml)
Test atmchange --all followed by an atmchange of one of them ... ok
test_atmchanges_on_arrays (__main__.TestBuildnml)
Test that atmchange works for array data ... ok
test_atmchanges_with_namespace (__main__.TestBuildnml)
Test that atmchange works when using 'namespace' syntax foo::bar ... ok
test_buffer_unchanged_with_bad_change_syntax (__main__.TestBuildnml)
Test atmchange does not change buffer if syntax was wrong ... ok
test_doctests (__main__.TestBuildnml)
Run doctests for all eamxx/cime_config python files and nml-related files in scripts ... ok
test_invalid_xml_option (__main__.TestBuildnml)
Test atmchange errors out with invalid param names ... ok
test_nml_defaults_append (__main__.TestBuildnml)
Test that the append attribute for array-type params in namelist defaults works as expected ... ok
test_pylint (__main__.TestBuildnml)
Run pylint on all eamxx/cime_config python files and nml-related files in scripts ... ok
test_reset_atmchanges_are_lost (__main__.TestBuildnml)
Test that atmchanges are lost when resetting ... ok
test_xmlchange_propagates_to_atmconfig (__main__.TestBuildnml)
Test that xmlchanges impact atm config files ... ok

Ran 19 tests in 208.660s
OK

[[ 0 != 0 ]]
[[ -z '' ]]
[[ 1 == 1 ]]
[[ 0 == 1 ]]
[[ 0 == 1 ]]
[[ 0 == 1 ]]
[[ 0 == 1 ]]
echo 'SCREAM v1 tests were skipped, since the Github label '''AT: Skip v1 Testing''' was found.\n'

SCREAM v1 tests were skipped, since the Github label 'AT: Skip v1 Testing' was found.\n
set +x

######################################################

FAILS DETECTED:

SCREAM STANDALONE TESTING FAILED!

Build type full_debug failed at testing time. Here's a list of failed tests:

220:nudging_tests_np1

221:nudging_tests_np2

222:nudging_tests_np3

223:nudging_tests_np4

262:aerocom_cld

418:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1

419:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np2

420:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np3

421:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np4

422:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np2_vs_np1

423:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np3_vs_np1

424:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np4_vs_np1

425:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5436/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5436/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5436/scream/components/eamxx

mappy failed

######################################################

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 44108 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins13801253667566183334.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5715 FAILED (click to see last 100 lines of console output)

driver              =  41.08 sec*proc (8 tests)
dynamics            =   5.21 sec*proc (3 tests)
fail                =  27.07 sec*proc (5 tests)
io                  =  49.11 sec*proc (13 tests)
mam4_optics         =   3.85 sec*proc (1 test)
nudging             =  13.76 sec*proc (2 tests)
p3                  =  63.82 sec*proc (7 tests)
p3_sk               =  29.92 sec*proc (2 tests)
physics             = 120.16 sec*proc (19 tests)
remap               =   5.77 sec*proc (1 test)
rrtmgp              =  31.17 sec*proc (9 tests)
shoc                =  21.25 sec*proc (7 tests)
spa                 =  11.50 sec*proc (4 tests)
surface_coupling    =   5.14 sec*proc (1 test)
Total Test time (real) = 563.35 sec
The following tests FAILED:

49 - nudging_tests_np1 (Failed)

73 - aerocom_cld (Failed)

CMake Error at /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/cmake/ctest_script.cmake:76 (message):

Test had fails
===============================================================================

Testing '''f2a20ad2a7747aebe5975877eae3e9afa31f60a4''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''f2a20ad2a7747aebe5975877eae3e9afa31f60a4''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/ctest-build/release
Testing '''f2a20ad2a7747aebe5975877eae3e9afa31f60a4''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx/ctest-build/full_debug

Build type full_debug failed at testing time. Here'''s a list of failed tests:

49:nudging_tests_np1

73:aerocom_cld
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx

weaver failed'

errors='Build type full_debug failed at testing time. Here'''s a list of failed tests:

49:nudging_tests_np1

73:aerocom_cld

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx

weaver failed'

SA_FAILURES_DETAILS+='Build type full_debug failed at testing time. Here'''s a list of failed tests:

49:nudging_tests_np1

73:aerocom_cld

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx

weaver failed'

[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

######################################################

FAILS DETECTED:

SCREAM STANDALONE TESTING FAILED!

Build type full_debug failed at testing time. Here's a list of failed tests:

49:nudging_tests_np1

73:aerocom_cld

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5715/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins939582571231647419.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

• Build Num: 5439
• Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	144da57
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

• Build Num: 5718
• Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	144da57
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

• Branch: bartgol/eamxx/kokkos-enable-bounds-checks
• SHA: 144da57
• Mode: TEST_REPO

Pull Request Author: bartgol

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Error: Jenkins Jobs - A user has pushed a change to the PR before testing completed. NEW EVENT 'committed', ID C_kwDOCEfuetoAKGU1NzE4ZjQxZDc3MDc3MGFjZjBmZTE1NDU1NDhmNmQ2NTc4ZWQ0MmE... The Jenkins Jobs will be shutdown; Testing of this PR must occur again.

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

• Build Num: 5439
• Status: ERROR

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	144da57
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

• Build Num: 5718
• Status: ERROR

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	144da57
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5439 ERROR (click to see last 100 lines of console output)

 > git config -f .gitmodules --get submodule.externals/haero.path # timeout=10
 > git config --get submodule.externals/mam4xx.url # timeout=10
 > git remote # timeout=10
 > git config --get remote.origin.url # timeout=10
 > git config -f .gitmodules --get submodule.externals/mam4xx.path # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/elm/src/external_models/fates # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/elm/src/external_models/mpp # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/elm/src/external_models/sbetr # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/eam/src/physics/rrtmgp/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/scorpio # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/scorpio_classic # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/eam/src/physics/cosp2/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive cime # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/ekat # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/YAKL # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/cub # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/mpas-ocean/src/cvmix # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/mpas-ocean/src/BGC # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/mpas-ocean/src/MARBL # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/mpas-ocean/src/gotm # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/mct # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/mpas-albany-landice/src/SeaLevelModel # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
Setting http proxy: proxy.sandia.gov:80
 > git submodule update --init --recursive components/mpas-ocean/src/SHTNS # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
Setting http proxy: proxy.sandia.gov:80
 > git submodule update --init --recursive components/mpas-ocean/src/FFTW # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
Setting http proxy: proxy.sandia.gov:80
 > git submodule update --init --recursive components/mpas-ocean/src/ppr # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/ww3/src/WW3 # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/eam/src/physics/crm/pam/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/mpas-seaice/src/icepack # timeout=10
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 17898 killed;
[ssh-agent] Stopped.
Build was aborted
Aborted by Luca Bertagna
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins16495415420392725635.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: ABORTED

SCREAM_PullRequest_Autotester_Weaver # 5718 ERROR (click to see last 100 lines of console output)

 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/scorpio_classic.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/scorpio_classic.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.cosp2.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.cosp2.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.cime.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.cime.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/ekat.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/ekat.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/YAKL.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/YAKL.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/cub.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/cub.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-ocean/src/cvmix.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-ocean/src/cvmix.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-ocean/src/BGC.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-ocean/src/BGC.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.MARBL.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.MARBL.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-ocean/src/gotm.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-ocean/src/gotm.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/mct.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/mct.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-albany-landice/src/SeaLevelModel.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-albany-landice/src/SeaLevelModel.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-ocean/src/SHTNS.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-ocean/src/SHTNS.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-ocean/src/FFTW.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-ocean/src/FFTW.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-ocean/src/ppr.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-ocean/src/ppr.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/ww3/src/source/WW3.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/ww3/src/source/WW3.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/eam/src/physics/crm/pam/external.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/eam/src/physics/crm/pam/external.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-seaice/src/icepack.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-seaice/src/icepack.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/haero.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/haero.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/mam4xx.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/mam4xx.path # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/elm/src/external_models/fates # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/elm/src/external_models/mpp # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/elm/src/external_models/sbetr # timeout=10
Build was aborted
Aborted by Luca Bertagna
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins13317347175167761520.sh

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

• Build Num: 5440
• Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	e5718f4
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

• Build Num: 5719
• Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	e5718f4
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

• Branch: bartgol/eamxx/kokkos-enable-bounds-checks
• SHA: e5718f4
• Mode: TEST_REPO

Pull Request Author: bartgol

[bartgol]

This PR brought up an issue in EKAT. I submitted a PR there, but we then need to wait for an upstream merge, since ekat will have kokkos 4.2 in it now. So this PR is on hold until

• Point to ekat master branch, and fix usage of deprecated kokkos code in Homme E3SM#6421 is merged
• Upstream merge 2024-05-20 #2835 is merged
• Make index_and_shift guard against OOB access in debug mode EKAT#339 is merged
• and we update EKAT in the scream repo.

The good news is that enabling bounds check showed a few issues (all benign or limited to unit tests, fortunately).

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

• Build Num: 5440
• Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	e5718f4
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

• Build Num: 5719
• Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts;AT: Skip v1 Testing
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	e5718f4
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	335335f
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5440 FAILED (click to see last 100 lines of console output)

OK (skipped=4)
Warning: repo is not clean: M foo
+ [[ 0 != 0 ]]
+ ./scripts/cime-nml-tests
test_append (__main__.TestBuildnml)
Test that var+=value syntax behaves as expected ... ok
test_atmchanges_are_lost_with_hack_xml (__main__.TestBuildnml)
Test that atmchanges are lost if SCREAM_HACK_XML=TRUE ... ok
test_atmchanges_are_preserved (__main__.TestBuildnml)
Test that atmchanges are not lost when eamxx setup is called ... ok
test_atmchanges_are_preserved_testmod (__main__.TestBuildnml)
Test that atmchanges via testmod are preserved ... ok
test_atmchanges_for_atm_procs_add (__main__.TestBuildnml)
Test atmchanges that add atm procs ... ok
test_atmchanges_for_atm_procs_add_group (__main__.TestBuildnml)
Test atmchanges that add atm proc groups ... ok
test_atmchanges_for_atm_procs_add_invalid (__main__.TestBuildnml)
Test atmchanges that add atm procs ... ok
test_atmchanges_for_atm_procs_remove (__main__.TestBuildnml)
Test atmchanges that remove atm procs ... ok
test_atmchanges_on_all_matches (__main__.TestBuildnml)
Test that atmchange --all works ... ok
test_atmchanges_on_all_matches_plus_spec (__main__.TestBuildnml)
Test atmchange --all followed by an atmchange of one of them ... ok
test_atmchanges_on_arrays (__main__.TestBuildnml)
Test that atmchange works for array data ... ok
test_atmchanges_with_namespace (__main__.TestBuildnml)
Test that atmchange works when using 'namespace' syntax foo::bar ... ok
test_buffer_unchanged_with_bad_change_syntax (__main__.TestBuildnml)
Test atmchange does not change buffer if syntax was wrong ... ok
test_doctests (__main__.TestBuildnml)
Run doctests for all eamxx/cime_config python files and nml-related files in scripts ... ok
test_invalid_xml_option (__main__.TestBuildnml)
Test atmchange errors out with invalid param names ... ok
test_nml_defaults_append (__main__.TestBuildnml)
Test that the append attribute for array-type params in namelist defaults works as expected ... ok
test_pylint (__main__.TestBuildnml)
Run pylint on all eamxx/cime_config python files and nml-related files in scripts ... ok
test_reset_atmchanges_are_lost (__main__.TestBuildnml)
Test that atmchanges are lost when resetting ... ok
test_xmlchange_propagates_to_atmconfig (__main__.TestBuildnml)
Test that xmlchanges impact atm config files ... ok

Ran 19 tests in 208.059s
OK

[[ 0 != 0 ]]
[[ -z '' ]]
[[ 1 == 1 ]]
[[ 0 == 1 ]]
[[ 0 == 1 ]]
[[ 0 == 1 ]]
[[ 0 == 1 ]]
echo 'SCREAM v1 tests were skipped, since the Github label '''AT: Skip v1 Testing''' was found.\n'

SCREAM v1 tests were skipped, since the Github label 'AT: Skip v1 Testing' was found.\n
set +x

######################################################

FAILS DETECTED:

SCREAM STANDALONE TESTING FAILED!

Build type full_debug failed at testing time. Here's a list of failed tests:

418:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1

419:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np2

420:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np3

421:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np4

422:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np2_vs_np1

423:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np3_vs_np1

424:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np4_vs_np1

425:homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5440/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5440/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5440/scream/components/eamxx

mappy failed

######################################################

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 29224 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins2819686915287395290.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5719 PASSED (click to see last 100 lines of console output)

116/133 Test #116: shoc_p3_nudging_glob_novert .............................   Passed    2.81 sec
        Start 117: homme_shoc_cld_p3_rrtmgp_np1
117/133 Test #117: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed   12.00 sec
        Start 118: homme_shoc_cld_p3_rrtmgp_baseline_cmp
118/133 Test #118: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.10 sec
        Start 119: homme_shoc_cld_p3_rrtmgp_pg2_np1
119/133 Test #119: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed   12.24 sec
        Start 120: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
120/133 Test #120: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.09 sec
        Start 121: model_baseline
121/133 Test #121: model_baseline ..........................................   Passed   12.81 sec
        Start 122: model_initial
122/133 Test #122: model_initial ...........................................   Passed    5.64 sec
        Start 123: model_restart
123/133 Test #123: model_restart ...........................................   Passed    6.53 sec
        Start 124: restarted_vs_monolithic_check_np1
124/133 Test #124: restarted_vs_monolithic_check_np1 .......................   Passed    0.09 sec
        Start 125: homme_shoc_cld_spa_p3_rrtmgp_np1
125/133 Test #125: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed    5.99 sec
        Start 126: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
126/133 Test #126: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.12 sec
        Start 127: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
127/133 Test #127: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed    8.67 sec
        Start 128: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
128/133 Test #128: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.39 sec
        Start 129: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
129/133 Test #129: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    0.56 sec
        Start 130: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1
130/133 Test #130: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1 .................   Passed   19.53 sec
        Start 131: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
131/133 Test #131: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp ........   Passed    0.09 sec
        Start 132: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
132/133 Test #132: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 .................   Passed   18.05 sec
        Start 133: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
133/133 Test #133: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ........   Passed    0.19 sec
100% tests passed, 0 tests failed out of 133
Label Time Summary:

baseline_cmp        =  66.16 secproc (16 tests)

baseline_gen        = 183.99 secproc (18 tests)

bfbhash             =   0.71 secproc (1 test)

check               =   0.71 secproc (1 test)

cld                 =  19.07 secproc (3 tests)

cld_fraction        =   5.78 secproc (1 test)

cxx baseline_cmp    =   5.20 secproc (2 tests)

diagnostics         =  33.74 secproc (22 tests)

driver              =  42.59 secproc (8 tests)

dynamics            =   7.30 secproc (3 tests)

fail                =  27.13 secproc (5 tests)

io                  =  52.94 secproc (13 tests)

mam4_optics         =   9.47 secproc (1 test)

nudging             =  13.63 secproc (2 tests)

p3                  =  58.91 secproc (7 tests)

p3_sk               =  39.53 secproc (2 tests)

physics             = 126.70 secproc (19 tests)

remap               =   6.68 secproc (1 test)

rrtmgp              =  34.42 secproc (9 tests)

shoc                =  22.97 secproc (7 tests)

spa                 =  12.77 secproc (4 tests)

surface_coupling    =   5.61 secproc (1 test)
Total Test time (real) = 556.53 sec
Testing '''55a0e0f4f42082e6eae4fbeb59c076580c7fb77e''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''55a0e0f4f42082e6eae4fbeb59c076580c7fb77e''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/ctest-build/release
Testing '''55a0e0f4f42082e6eae4fbeb59c076580c7fb77e''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5719/scream/components/eamxx

Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins15061889114860212590.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: SUCCESS

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

• Build Num: 5525
• Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	de71513
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	6d57c6b
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

• Build Num: 5795
• Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	de71513
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	6d57c6b
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

• Branch: bartgol/eamxx/kokkos-enable-bounds-checks
• SHA: de71513
• Mode: TEST_REPO

Pull Request Author: bartgol

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

• Build Num: 5525
• Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	de71513
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	6d57c6b
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

• Build Num: 5795
• Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	de71513
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	6d57c6b
TEST_REPO_ALIAS	SCREAM

[E3SM-Bot]

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

[E3SM-Bot]

All Jobs Finished; status = PASSED, target_sha=a15130cfca7eebaaca5d632a1e65163e14fe1808, However Inspection must be performed before merge can occur...

[E3SM-Bot]

The base branch has been updated since the last successful testing.

• last PASS base branch sha: a15130c
• current base branch sha : 2783f00
  The AutoTester will discard the last PASS, and re-test the PR from scratch

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

• Build Num: 5528
• Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	de71513
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	6d57c6b
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

• Build Num: 5798
• Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	de71513
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	6d57c6b
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

• Branch: bartgol/eamxx/kokkos-enable-bounds-checks
• SHA: de71513
• Mode: TEST_REPO

Pull Request Author: bartgol

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

• Build Num: 5528
• Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	de71513
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	6d57c6b
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

• Build Num: 5798
• Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: AUTOMERGE;scripts
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	de71513
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	6d57c6b
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5528 FAILED (click to see last 100 lines of console output)

Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 2.383869 seconds (PEND). [COMPLETED 2 of 8]
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 16 procs
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 217.913964 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 4.053236 seconds (PEND). [COMPLETED 3 of 8]
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Finished MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 319.894169 seconds (PASS)
Starting RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 1 proc on interactive node and 16 procs on compute nodes
Finished RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 3.801976 seconds (PEND). [COMPLETED 4 of 8]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 5739.952177 seconds (FAIL). [COMPLETED 5 of 8]
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240612_120852_tobkba
    Errors were:
        Building test for ERS in directory /home/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240612_120852_tobkba
        ERROR: BUILD FAIL: build e3sm failed, cat /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240612_120852_tobkba/bld/e3sm.bldlog.240612-121318
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 5715.272519 seconds (FAIL). [COMPLETED 6 of 8]

Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240612_120852_tobkba

Errors were:

Building test for PET in directory /home/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240612_120852_tobkba

ERROR: BUILD FAIL: build e3sm failed, cat /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240612_120852_tobkba/bld/e3sm.bldlog.240612-121346
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 5799.256800 seconds (FAIL). [COMPLETED 7 of 8]

Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240612_120852_tobkba

Errors were:

Building test for ERP in directory /home/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240612_120852_tobkba

ERROR: BUILD FAIL: build e3sm failed, cat /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240612_120852_tobkba/bld/e3sm.bldlog.240612-121220
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 5750.097153 seconds (FAIL). [COMPLETED 8 of 8]

Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240612_120852_tobkba

Errors were:

Building test for ERP in directory /home/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240612_120852_tobkba

ERROR: BUILD FAIL: build e3sm failed, cat /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240612_120852_tobkba/bld/e3sm.bldlog.240612-121308'

./scream/components/eamxx/scripts/jenkins/jenkins_common.sh: line 7: 115737 Terminated              $JENKINS_SCRIPT_DIR/jenkins_common_impl.sh 2>&1

115738                       | tee JENKINS_$DATE_STAMP

FATAL: command execution failed

java.io.EOFException

at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2915)

at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3410)

at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:954)

at java.base/java.io.ObjectInputStream.(ObjectInputStream.java:392)

at hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:50)

at hudson.remoting.Command.readFrom(Command.java:142)

at hudson.remoting.Command.readFrom(Command.java:128)

at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)

at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)

Caused: java.io.IOException: Unexpected termination of the channel

at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:75)

Caused: java.io.IOException: Backing channel 'mappy' is disconnected.

at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215)

at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)

at jdk.proxy2/jdk.proxy2.$Proxy99.isAlive(Unknown Source)

at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1212)

at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1204)

at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:195)

at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:145)

at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)

at hudson.plugins.postbuildtask.PostbuildTask.perform(PostbuildTask.java:123)

at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)

at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)

at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:767)

at hudson.model.Build$BuildExecution.post2(Build.java:179)

at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:711)

at hudson.model.Run.execute(Run.java:1918)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)

at hudson.model.ResourceController.execute(ResourceController.java:101)

at hudson.model.Executor.run(Executor.java:442)

FATAL: Unable to delete script file /tmp/jenkins12243628873192426899.sh

java.io.EOFException

at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2915)

at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3410)

at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:954)

at java.base/java.io.ObjectInputStream.(ObjectInputStream.java:392)

at hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:50)

at hudson.remoting.Command.readFrom(Command.java:142)

at hudson.remoting.Command.readFrom(Command.java:128)

at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)

at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)

Caused: java.io.IOException: Unexpected termination of the channel

at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:75)

Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@5ee3810:mappy": Remote call on mappy failed. The channel is closing down or has closed down

at hudson.remoting.Channel.call(Channel.java:996)

at hudson.FilePath.act(FilePath.java:1230)

at hudson.FilePath.act(FilePath.java:1219)

at hudson.FilePath.delete(FilePath.java:1766)

at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:163)

at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)

at hudson.plugins.postbuildtask.PostbuildTask.perform(PostbuildTask.java:123)

at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)

at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)

at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:767)

at hudson.model.Build$BuildExecution.post2(Build.java:179)

at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:711)

at hudson.model.Run.execute(Run.java:1918)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)

at hudson.model.ResourceController.execute(ResourceController.java:101)

at hudson.model.Executor.run(Executor.java:442)

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: [email protected]

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5798 PASSED (click to see last 100 lines of console output)

        Start 117: shoc_p3_nudging_glob_novert
117/134 Test #117: shoc_p3_nudging_glob_novert .............................   Passed    2.78 sec
        Start 118: homme_shoc_cld_p3_rrtmgp_np1
118/134 Test #118: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed   12.83 sec
        Start 119: homme_shoc_cld_p3_rrtmgp_baseline_cmp
119/134 Test #119: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.11 sec
        Start 120: homme_shoc_cld_p3_rrtmgp_pg2_np1
120/134 Test #120: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed   11.92 sec
        Start 121: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
121/134 Test #121: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.10 sec
        Start 122: model_baseline
122/134 Test #122: model_baseline ..........................................   Passed   12.72 sec
        Start 123: model_initial
123/134 Test #123: model_initial ...........................................   Passed    5.65 sec
        Start 124: model_restart
124/134 Test #124: model_restart ...........................................   Passed    6.95 sec
        Start 125: restarted_vs_monolithic_check_np1
125/134 Test #125: restarted_vs_monolithic_check_np1 .......................   Passed    0.11 sec
        Start 126: homme_shoc_cld_spa_p3_rrtmgp_np1
126/134 Test #126: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed   12.30 sec
        Start 127: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
127/134 Test #127: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.17 sec
        Start 128: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
128/134 Test #128: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed   16.21 sec
        Start 129: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
129/134 Test #129: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.38 sec
        Start 130: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
130/134 Test #130: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    0.64 sec
        Start 131: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1
131/134 Test #131: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1 .................   Passed   20.27 sec
        Start 132: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
132/134 Test #132: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp ........   Passed    0.10 sec
        Start 133: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
133/134 Test #133: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 .................   Passed   18.28 sec
        Start 134: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
134/134 Test #134: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ........   Passed    0.17 sec
100% tests passed, 0 tests failed out of 134
Label Time Summary:

baseline_cmp        =  72.31 secproc (16 tests)

baseline_gen        = 204.78 secproc (18 tests)

bfbhash             =   0.72 secproc (1 test)

check               =   0.71 secproc (1 test)

cld                 =  13.98 secproc (3 tests)

cld_fraction        =   4.85 secproc (1 test)

cxx baseline_cmp    =   5.87 secproc (2 tests)

diagnostics         =  60.84 secproc (22 tests)

driver              =  46.15 secproc (8 tests)

dynamics            =   7.34 secproc (3 tests)

fail                =  28.81 secproc (5 tests)

io                  =  59.10 secproc (13 tests)

mam4_optics         =   9.14 secproc (1 test)

nudging             =  14.29 secproc (2 tests)

p3                  =  59.74 secproc (7 tests)

p3_sk               =  45.51 secproc (2 tests)

physics             = 140.42 secproc (19 tests)

remap               =   6.01 secproc (1 test)

rrtmgp              =  36.63 secproc (9 tests)

shoc                =  17.91 secproc (7 tests)

spa                 =  12.41 secproc (4 tests)

surface_coupling    =   5.35 secproc (1 test)
Total Test time (real) = 639.45 sec
Testing '''f3feccfae385324b435a9b31c1a30b20e7bfa4ce''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''f3feccfae385324b435a9b31c1a30b20e7bfa4ce''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/ctest-build/release
Testing '''f3feccfae385324b435a9b31c1a30b20e7bfa4ce''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5798/scream/components/eamxx

Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins4526639141761715465.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

• Build Num: 5533
• Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: RETEST;AT: AUTOMERGE;scripts
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	de71513
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	6d57c6b
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

• Build Num: 5803
• Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: RETEST;AT: AUTOMERGE;scripts
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	de71513
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	6d57c6b
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

• Branch: bartgol/eamxx/kokkos-enable-bounds-checks
• SHA: de71513
• Mode: TEST_REPO

Pull Request Author: bartgol

[E3SM-Bot]

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

• Build Num: 5533
• Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: RETEST;AT: AUTOMERGE;scripts
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	de71513
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	6d57c6b
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

• Build Num: 5803
• Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	testing;BFB;AT: RETEST;AT: AUTOMERGE;scripts
PULLREQUESTNUM	2830
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	de71513
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	6d57c6b
TEST_REPO_ALIAS	SCREAM