Exit code 143 on BigRed3

[Lauren-Nilsson]

@jadhao @kadupitiya
Periodically, jobs I am running are terminating with exit code 143 on BigRed3.
There is no error in the log, energy is conserved and no obvious code issues. Restarting the job manually will generally run past the point where the error was thrown without issue.

Here are two examples, where the job was submitted for 20 hours.

1. After apparent timeout:
   Slurm Job_id=254798 Name=MVM_2.15_T_298 Failed, Run time 20:00:06, FAILED, ExitCode 143

2. Prematurely:
   Slurm Job_id=257095 Name=MVM_1.95_T_298 Failed, Run time 01:59:54, FAILED, ExitCode 143

Both of these ended at the exact same time, along with two jobs that timed out and one more that had an apparent timeout.

Is there a bug in the code or job script that could be causing this? I have measured the memory consumption locally -- it seems it should be well below the memory limit on BR3.

[Lauren-Nilsson]

Code I am using is on my Fork.

[kadupitiya]

@Lauren-Nilsson: Can you send me the job script?
When you run locally, can you run pass this number of steps?

[Lauren-Nilsson]

@kadupitiya The job script I am using is scripts/iu_cluster_job_script.pbs on my fork.

I emailed HPS team to ask about possible slurm error per our conversation. Thanks