eamxx: PIO: FATAL ERROR: File create mode is inconsistent among processes when partial restart files exist

[ndkeen]

I was trying to run some ne1024 benchmark cases on pm-gpu with 512 nodes. Run 3 days with output, write restart on last day. Case completed OK once, but the timing was much slower than expected, so I simply resubmitted. Again the timing was slow (it's ongoing issue, but not the point here) and resubmitted. However, the next 4 jobs all failed with following error:

 908: PIO: FATAL ERROR: Aborting... FATAL ERROR: File create mode is inconsistent among processes. (file = cess-v2-cntl.ne1024pg2_ne1024pg2.F2010-SCREAMv1.se18-oct18.n0512t4x111XX1.allyamlhist.3d.wr.sk.dins.cice.r.2019-08-04-00000.nc) (/dvs_ro/cfs/cdirs/\
e3sm/ndk/repos/se18-oct18/externals/scorpio/src/clib/pioc_support.c: 3418)

/pscratch/sd/n/ndk/e3sm_scratch/pm-gpu/se18-oct18/cess-v2-cntl.ne1024pg2_ne1024pg2.F2010-SCREAMv1.se18-oct18.n0512t4x111XX1.allyamlhist.3d.wr.sk.dins

@mahf708 said he also hit same error (I think also with ne1024 cases on pm-gpu) and found that he needed to remove any partial restart files -- perhaps cice.r in particular.

Now in my case, I'm not trying to read from restarts -- only writing. But for the fails, there were some partial restarts.
I've already removed the restarts (and rpointers) that were there as I'm testing if next job will be ok.

Labelling as eamxx, but not sure if it's specific to eamxx.

[mahf708]

Yeah, needed to remove cice-related files to get around all of this. Iirc, it was r, rh, and h files needing to be removed, but I didn't take notes

I did hit this with ne256pg2 as well fwiw

[ndkeen]

OK I simply removed all of the partial restarts and resubmitted. Job was ok -- restarts written as expected. This suggests to me, that the code is perhaps looking at existing files and doing some logic before trying to open and write. Which is maybe not what we want.