Compare AMIP/Atmos-only with diagnostic EDMF job is failing on benchmarks pipeline

[sriharshakandala]

The Compare AMIP/Atmos-only with diagnostic EDMF job is failing on benchmarks pipeline.
Please see https://buildkite.com/clima/climacoupler-cpu-gpu-benchmarks/builds/201 for the error.

cc: @juliasloan25 @szy21

[juliasloan25]

Thanks for flagging this! It should be fixed by #1054, which was merged on Monday. I had missed propagating some changes to the benchmark comparison code. The job should run and pass this Sunday

[sriharshakandala]

Thanks. I started a new build here: https://buildkite.com/clima/climacoupler-cpu-gpu-benchmarks/builds/202

[sriharshakandala]

The CPU ClimaAtmos without diagnostic EDMF job is now failing here:
https://buildkite.com/clima/climacoupler-cpu-gpu-benchmarks/builds/202

[juliasloan25]

The CPU ClimaAtmos without diagnostic EDMF job is now failing here: https://buildkite.com/clima/climacoupler-cpu-gpu-benchmarks/builds/202

The folder it's looking for is

/scratch/clima/slurm-buildkite/climacoupler-cpu-gpu-benchmarks/202/climacoupler-cpu-gpu-benchmarks/experiments/ClimaEarth/output/climaatmos/climaatmos/artifacts/output_active

It looks like there is an output folder output_0000, but no output_active:

[jsloan@clima artifacts]$ pwd
/scratch/clima/slurm-buildkite/climacoupler-cpu-gpu-benchmarks/202/climacoupler-cpu-gpu-benchmarks/experiments/ClimaEarth/output/climaatmos/climaatmos/artifacts
[jsloan@clima artifacts]$ ls
output_0000

The error is when we call ClimaAtmos.get_simulation here. @Sbozzolo do you know why output_active wouldn't appear?

[Sbozzolo]

This error is rather mysterious to me. The code in OutputPathGenerator is

    link_exists = islink(active_link)

    if link_exists
        target = readlink(active_link)

So, reaching the line where we read the link means that we passed the line where we checked if there's a link.

[Sbozzolo]

But the, I see this

/scratch/clima/slurm-buildkite/climacoupler-cpu-gpu-benchmarks/202/climacoupler-cpu-gpu-benchmarks/experiments/ClimaEarth/output/climaatmos/climaatmos/artifacts/ already contains some output data, but no active link

Is it possible that multiple jobs are using the same output folder?

[Sbozzolo]

Okay. I think I know what's happening. The first time the job was run it failed due to an intermitted filesystem issue/slowdown (the error is "output path not synced"). This left the folder structure in a partially broken state, so the second time the job was run, it failed again. Running the second time cleaned some things up and allowed the system reset for the third attempt.

[juliasloan25]

The linked build (https://buildkite.com/clima/climacoupler-cpu-gpu-benchmarks/builds/202) has passed, so we can close this issue