Running WTF in Cheyenne

README

@@ -38,20 +38,33 @@ The file structure of the WRF Test Framework is as follows:
      Runs/                       --  All WRF tests are performed here
      clean                       --  Script for cleaning out old tests. run `./clean` or view
                                      the script's comments for usage instructions
+     run_all_Cheyenne.csh        --  Script for running qsub for GNU and INTEL and screen for PGI.
      run_from_github.py          --  Script for running the test directly from code on Github.
                                      See below for instructions.
      qsub_this_to_run_all_compilers.pbs.csh
                                  --  Script for running tests for all compilers on Cheyenne. See
                                      below for instructions.
+     qsub_*.pbs                  --  Script for running tests for each specific compiler on Cheyenne.
 
 =================================================================================================
-                           TO RUN DIRECTLY FROM GITHUB ON CHEYENNE
+                       TO RUN DIRECTLY FROM GITHUB ON A SUPERCOMPUTER (CHEYENNE)
 =================================================================================================
 
-1)   View the script `scripts/run_all_for_qsub.csh` and make modifications if necessary. The most
-     common modification that may need to be made is to change the compiler versions. Note that 
-     this script sets up each compiler as necessary, then submits each individual test as 
-     described below.
+1)   If you are running on Cheyenne, view the next files and make modifications if necessary: 
+     *) `qsub_intel.pbs` and `scripts/run_intel.csh`
+     *) `qsub_gnu.pbs` and `scripts/run_gnu.csh`
+     in order to change the account to be charged. Other common modification that may need to be made is 
+     to change the compiler versions. 
+     Note that script `run_all_Cheyenne.csh` sets up each compiler as necessary, then submits each individual 
+     test as described below using the previous scripts. It is also important to view files:
+     *) `regTest_*_Cheyenne.wtf`
+     and make modifications if necessary. This file must end in "*.wtf"  (case sensitive).
+     Lengthy explanations are given in the existing WRF Test Files for each configuration.
+     The most important parameters to check are near the top of the file.
+
+     If you are not running on Cheyenne, view `qsub_this_to_run_all_compilers.pbs.csh`, 
+     `scripts/run_all_for_qsub1.csh`, `scripts/run_all_for_qsub2.csh`, and regTest_*_*.wtf 
+     for similar modifications. 
 
 2)   In the top-level directory, run the script:
 
@@ -60,35 +73,57 @@ The file structure of the WRF Test Framework is as follows:
      This will prompt you to enter the URL of the repository you want to test (for example, 
      https://github.com/wrf-model/WRF), and the name of the branch you want to test (for example, 
      master). It will download the code, pack it into a tar file, and submit a test job using the 
-     `qsub_this_to_run_all_compiler.csh` script described below.
+     `qsub_this_to_run_all_compiler.csh` script described below. Finally, it will ask if you are on 
+     Cheyenne. If the answer is yes, it will run qsub for GNU and INTEL and screen for PGI using the
+     script run_all_Cheyenne.csh. If not, it will run qsub for everything using the script 
+     qsub_this_to_run_all_compilers.pbs.csh.
 
-3)   When the test has completed, results will be summarized in the directory Runs/RESULTS.        
+3)   In order to follow the executions, one can look at files foo_* for each compiler. When the test 
+     has completed, results will be summarized in the directory Runs/RESULTS.        
 
 =================================================================================================
-                           TO RUN ON A LOCAL TAR FILE ON CHEYENNE
+                       TO RUN ON A LOCAL TAR FILE ON A SUPERCOMPUTER (CHEYENNE)
 =================================================================================================
 
 1)   Place one or more  WRF source tar files in the "tarballs" directory.  Each file must end in 
      "*.tar" (case sensitive), and the tarfile names should not contain spaces. Each WRF tarfile 
-     must create the directory "WRFV3" when unpacked. Otherwise, the script will fail. NOTE: IT 
+     must create the directory "WRF" when unpacked. Otherwise, the script will fail. NOTE: IT 
      IS NOT RECOMMENDED TO RUN FOR MORE THAN ONE TAR FILE AT A TIME, unless you are running 
      overnight, as the process can take a long time and use a significant fraction of Cheyenne's
      resources.
 
-2)   View the script `scripts/run_all_for_qsub.csh` and make modifications if necessary. Note
-     that this script sets up each compiler as necessary, then submits each individual test as
-     described belowcompiler  This file must end in "*.wtf"  (case sensitive).
+2)   If you are running on Cheyenne, view the next files and make modifications if necessary: 
+     *) `qsub_intel.pbs` and `scripts/run_intel.csh`
+     *) `qsub_gnu.pbs` and `scripts/run_gnu.csh`
+     For instance, the account usually has to be changed. Note that script `run_all_Cheyenne.csh` 
+     sets up each compiler as necessary, then submits each individual test as described below using 
+     the previous scripts. It is also important to view files:
+     *) `regTest_*_Cheyenne.wtf`
+     and make modifications if necessary. These files must end in "*.wtf" (case sensitive).
      Lengthy explanations are given in the existing WRF Test Files for each configuration.
      The most important parameters to check are near the top of the file.
+
+     If you are not running on Cheyenne, view `qsub_this_to_run_all_compilers.pbs.csh`, 
+     `scripts/run_all_for_qsub1.csh`, `scripts/run_all_for_qsub2.csh`, and regTest_*_*.wtf 
+     for similar modifications. 
+
+3)   If you are on Cheyenne, in the top-level directory, issue the command:
+
+      "./run_all_Cheyenne.csh >& cheyenne.log &"
+
+     This will run separate tests simultaneously using qsub for GNU and Intel, and screen for 
+     PGI compilers, as well as separate tests for WRFDA 4DVAR (since the configuration option 
+     numbers are different).
 
-3)   In the top-level directory, issue the command:
+     If you are not on Cheyenne, in the top-level directory, issue the command:
 
        "qsub < qsub_this_to_run_all_compilers.pbs.csh"
 
-     This will run separate tests simultaneously for GNU, Intel, and PGI compilers, as well as 
-     separate tests for WRFDA 4DVAR (since the configuration option numbers are different).
+     This will run separate tests simultaneously using qsub for GNU, Intel, and PGI compilers, 
+     as well as separate tests for WRFDA 4DVAR (since the configuration option numbers are different).
 
-4)   When the test has completed, results will be summarized in the directory Runs/RESULTS.        
+4)   In order to follow the executions, one can look at files foo_* for each compiler. When the test 
+     has completed, results will be summarized in the directory Runs/RESULTS.      
 
 =================================================================================================
                            TO RUN MANUALLY FOR A SINGLE COMPILER
@@ -118,7 +153,8 @@ The file structure of the WRF Test Framework is as follows:
      WRF runs have completed when all the WRF output files are checked for invalid values (NaNs).   
 
 
-5)   When the test has completed, results will be summarized in the directory Runs/RESULTS. 
+5)   In order to follow the executions, one can look at files foo_* for each compiler. When the test 
+     has completed, results will be summarized in the directory Runs/RESULTS. 
 
 
 ------------------------------------------------------------------------------------------------------

qsub_gnu.pbs

@@ -0,0 +1,46 @@
+#!/bin/csh -f
+#
+# PBS batch script
+#
+# Job name
+#PBS -N gnu_WTF_v04.08
+# queue: share, regular, economy
+#PBS -q economy
+# Combine error and output files
+#PBS -j oe
+# output filename
+#PBS -o regtest_gnu.out
+# error filename
+#PBS -e regtest_gnu.out
+# wallclock time hh:mm:ss
+#PBS -l walltime=12:00:00
+# Project charge code
+#PBS -A UCUD0004
+# Claim 18 cores for this script (individual jobs created by this script can use way more)
+#PBS -l select=1:ncpus=18:mem=60GB
+# Send email on abort or end of main job
+#PBS -m ae
+
+if ( ! -e Data  ) then
+   echo "ERROR ERROR ERROR"
+   echo ""
+   echo "'Data' directory not found"
+   echo "If on Cheyenne, link /glade/p/wrf/Data into your WTF directory"
+   echo ""
+   echo "Otherwise, it can be downloaded from http://www2.mmm.ucar.edu/wrf/tmp/"
+   exit 1
+endif
+
+if ( ! -e Data/v04.08  ) then
+   echo "ERROR ERROR ERROR"
+   echo ""
+   echo "'Data' directory is not the correct version (v04.08)"
+   echo "If on Cheyenne, link /glade/p/wrf/Data into your WTF directory"
+   echo ""
+   echo "Otherwise, it can be downloaded from http://www2.mmm.ucar.edu/wrf/tmp/data_v04.08.tar"
+   exit 2
+endif
+
+setenv TMPDIR /gpfs/fs1/scratch/$USER/tmp # CISL-recommended hack for Cheyenne builds
+
+scripts/run_gnu.csh

qsub_intel.pbs

@@ -0,0 +1,46 @@
+#!/bin/csh -f
+#
+# PBS batch script
+#
+# Job name
+#PBS -N intel_WTF_v04.08
+# queue: share, regular, economy
+#PBS -q economy
+# Combine error and output files
+#PBS -j oe
+# output filename
+#PBS -o regtest_intel.out
+# error filename
+#PBS -e regtest_intel.out
+# wallclock time hh:mm:ss
+#PBS -l walltime=12:00:00
+# Project charge code
+#PBS -A UCUD0004
+# Claim 18 cores for this script (individual jobs created by this script can use way more)
+#PBS -l select=1:ncpus=18:mem=60GB
+# Send email on abort or end of main job
+#PBS -m ae
+
+if ( ! -e Data  ) then
+   echo "ERROR ERROR ERROR"
+   echo ""
+   echo "'Data' directory not found"
+   echo "If on Cheyenne, link /glade/p/wrf/Data into your WTF directory"
+   echo ""
+   echo "Otherwise, it can be downloaded from http://www2.mmm.ucar.edu/wrf/tmp/"
+   exit 1
+endif
+
+if ( ! -e Data/v04.08  ) then
+   echo "ERROR ERROR ERROR"
+   echo ""
+   echo "'Data' directory is not the correct version (v04.08)"
+   echo "If on Cheyenne, link /glade/p/wrf/Data into your WTF directory"
+   echo ""
+   echo "Otherwise, it can be downloaded from http://www2.mmm.ucar.edu/wrf/tmp/data_v04.08.tar"
+   exit 2
+endif
+
+setenv TMPDIR /gpfs/fs1/scratch/$USER/tmp # CISL-recommended hack for Cheyenne builds
+
+scripts/run_intel.csh

qsub_pgi.pbs

@@ -0,0 +1,48 @@
+#!/bin/csh -f
+#
+# PBS batch script
+#
+# Job name
+#PBS -N pgi_WTF_v04.08
+# queue: share, regular, economy
+#PBS -q economy
+# Combine error and output files
+#PBS -j oe
+# output filename
+#PBS -o regtest_pgi.out
+# error filename
+#PBS -e regtest_pgi.out
+# wallclock time hh:mm:ss
+#PBS -l walltime=12:00:00
+# Project charge code
+#PBS -A UCUD0004
+# Claim 18 cores for this script (individual jobs created by this script can use way more)
+#PBS -l select=1:ncpus=18:mem=60GB
+# Send email on abort or end of main job
+#PBS -m ae
+# login-node environment
+#PBS -l inception=login
+
+if ( ! -e Data  ) then
+   echo "ERROR ERROR ERROR"
+   echo ""
+   echo "'Data' directory not found"
+   echo "If on Cheyenne, link /glade/p/wrf/Data into your WTF directory"
+   echo ""
+   echo "Otherwise, it can be downloaded from http://www2.mmm.ucar.edu/wrf/tmp/"
+   exit 1
+endif
+
+if ( ! -e Data/v04.08  ) then
+   echo "ERROR ERROR ERROR"
+   echo ""
+   echo "'Data' directory is not the correct version (v04.08)"
+   echo "If on Cheyenne, link /glade/p/wrf/Data into your WTF directory"
+   echo ""
+   echo "Otherwise, it can be downloaded from http://www2.mmm.ucar.edu/wrf/tmp/data_v04.08.tar"
+   exit 2
+endif
+
+setenv TMPDIR /gpfs/fs1/scratch/$USER/tmp # CISL-recommended hack for Cheyenne builds
+
+scripts/run_pgi.csh

qsub_this_to_run_all_compilers.pbs.csh

@@ -5,17 +5,17 @@
 # Job name
 #PBS -N WTF_v04.08
 # queue: share, regular, economy
-#PBS -q share
+#PBS -q economy
 # Combine error and output files
 #PBS -j oe
 # output filename
 #PBS -o regtest.out
 # error filename
 #PBS -e regtest.out
 # wallclock time hh:mm:ss
-#PBS -l walltime=06:00:00
+#PBS -l walltime=12:00:00
 # Project charge code
-#PBS -A P64000400
+#PBS -A UCUD0004
 # Claim 18 cores for this script (individual jobs created by this script can use way more)
 #PBS -l select=1:ncpus=18:mem=60GB
 # Send email on abort or end of main job
@@ -42,7 +42,7 @@ if ( ! -e Data/v04.08  ) then
    exit 2
 endif
 
-setenv TMPDIR /glade/scratch/$USER/tmp # CISL-recommended hack for Cheyenne builds
+setenv TMPDIR /gpfs/fs1/scratch/$USER/tmp # CISL-recommended hack for Cheyenne builds
 
 scripts/run_all_for_qsub1.csh
 scripts/run_all_for_qsub2.csh

regTest_gnu_Cheyenne.wtf

@@ -36,7 +36,7 @@ fi
 # idealized cases, and if any are specified, then "em_real" or "em_real8" (respectively) must also be included earlier in the list.
 
 export BUILD_TYPES="em_real em_real8 em_hill2d_x nmm_nest em_chem em_chem_kpp em_b_wave em_quarter_ss em_quarter_ss8 nmm_hwrf wrfda_3dvar em_move em_fire"
-export BUILD_TYPES="em_real em_real8 em_hill2d_x nmm_nest em_chem             em_b_wave em_quarter_ss em_quarter_ss8 nmm_hwrf             em_move em_fire"
+export BUILD_TYPES="em_real em_real8 em_hill2d_x          em_chem             em_b_wave em_quarter_ss em_quarter_ss8                      em_move em_fire"
 
 # Select one of the directories below.  This is the parent directory for all namelist.input files to be used for 
 # the tests.  Below this directory should be subdirectories "em_real", "nmm_nest", etc with namelist.input files 
@@ -86,13 +86,13 @@ export BATCH_QUEUE_TYPE=PBS
 #   Cheyenne: share, regular, economy
 # Ignored when BATCH_COMPILE and BATCH_TEST are false.
 
-export BUILD_QUEUE=share
-export TEST_QUEUE=share
+export BUILD_QUEUE=regular
+export TEST_QUEUE=regular
 #export TEST_QUEUE=economy
 
 # Account charge code string for queue-managed computers. Ignored when BATCH_COMPILE and BATCH_TEST are false.
 
-export BATCH_ACCOUNT=NMMM0054
+export BATCH_ACCOUNT=UCUD0004
 
 # Number of processors/tasks to use for building WRF. 
 # No reason to set larger than 6 due to diminishing returns on compile time.

regTest_gnu_Cheyenne_WRFDA.wtf

@@ -85,13 +85,13 @@ export BATCH_QUEUE_TYPE=PBS
 #   Cheyenne: share, regular, economy
 # Ignored when BATCH_COMPILE and BATCH_TEST are false.
 
-export BUILD_QUEUE=share
-export TEST_QUEUE=share
+export BUILD_QUEUE=regular
+export TEST_QUEUE=regular
 #export TEST_QUEUE=economy
 
 # Account charge code string for queue-managed computers. Ignored when BATCH_COMPILE and BATCH_TEST are false.
 
-export BATCH_ACCOUNT=NMMM0054
+export BATCH_ACCOUNT=UCUD0004
 
 # Number of processors/tasks to use for building WRF. 
 # No reason to set larger than 6 due to diminishing returns on compile time.

regTest_intel_Cheyenne.wtf

@@ -37,7 +37,7 @@ fi
 
 export BUILD_TYPES="em_real"
 export BUILD_TYPES="em_real em_real8 em_hill2d_x nmm_nest em_chem em_chem_kpp em_b_wave em_quarter_ss em_quarter_ss8 nmm_hwrf wrfda_3dvar em_move em_fire"
-export BUILD_TYPES="em_real em_real8 em_hill2d_x nmm_nest em_chem             em_b_wave em_quarter_ss em_quarter_ss8 nmm_hwrf             em_move em_fire"
+export BUILD_TYPES="em_real em_real8 em_hill2d_x          em_chem             em_b_wave em_quarter_ss em_quarter_ss8                      em_move em_fire"
 
 
 # Select one of the directories below.  This is the parent directory for all namelist.input files to be used for 
@@ -88,13 +88,13 @@ export BATCH_QUEUE_TYPE=PBS
 #   Cheyenne: share, regular, economy
 # Ignored when BATCH_COMPILE and BATCH_TEST are false.
 
-export BUILD_QUEUE=share
-export TEST_QUEUE=share
+export BUILD_QUEUE=regular
+export TEST_QUEUE=regular
 #export TEST_QUEUE=economy
 
 # Account charge code string for queue-managed computers. Ignored when BATCH_COMPILE and BATCH_TEST are false.
 
-export BATCH_ACCOUNT=NMMM0054
+export BATCH_ACCOUNT=UCUD0004
 
 # Number of processors/tasks to use for building WRF. 
 # No reason to set larger than 6 due to diminishing returns on compile time.

regTest_intel_Cheyenne_WRFDA.wtf

@@ -85,13 +85,13 @@ export BATCH_QUEUE_TYPE=PBS
 #   Cheyenne: share, regular, economy
 # Ignored when BATCH_COMPILE and BATCH_TEST are false.
 
-export BUILD_QUEUE=share
-export TEST_QUEUE=share
+export BUILD_QUEUE=regular
+export TEST_QUEUE=regular
 #export TEST_QUEUE=economy
 
 # Account charge code string for queue-managed computers. Ignored when BATCH_COMPILE and BATCH_TEST are false.
 
-export BATCH_ACCOUNT=NMMM0054
+export BATCH_ACCOUNT=UCUD0004
 
 # Number of processors/tasks to use for building WRF. 
 # No reason to set larger than 6 due to diminishing returns on compile time.