Adds WOA 23 SSS restoring file script

[vanroekel]

This adds WOA23 surface salinity restoring for E3SM G-cases.

Checklist

• User's Guide has been updated
• Developer's Guide has been updated
• API documentation in the Developer's Guide (api.rst) has any new or modified class, method and/or functions listed
• Documentation has been built locally and changes look as expected

[vanroekel]

@xylar I've made progress on the salinity restoring we discussed. I was hoping to get a quick look from you to see if i'm on a good track. I've made a script to create the surface salinity climatology file as close as possible to the PHC2 version (variable names and such). I haven't hooked up to files for e3sm yet. I will do so tomorrow and test.

[vanroekel]

The restoring has been moved to its own test case and extrapolation and connection to files for e3sm is completed. I've run through for the IcoswISC30E3r5 mesh. Here is SSS WOA for November

Here is the field on the mpas mesh

[vanroekel]

the new lat, lon WOA23 SSS file has been place in the initial_condition_database on anvil -- woa_decav_0.25_sss_monthly_extrap.20241031.nc

[vanroekel]

@xylar I think this is ready for your look. I haven't done the documentation yet, if you'd rather wait for that let me know. I'll get to that soon

[xylar]

@vanroekel, this looks great and just needs a little clean up and documentation. Feel free to commit my suggestions from GitHub but please also make some arbitrary changes to whitespace locally, then make sure you source the compass load script and then commit the files. This should trigger pre-commit which will do some linting that seems to be missing.

[xylar]

Looks great! Testing this now.

[xylar]

The actual file in the initial condition database needs to be renamed woa --> woa23.

[xylar]

I can't do it because I don't have write permission. In general, if you can chmod all your files in mpas_standalone so they are group writable, that would be helpful.

[xylar]

Interesting! I can rename it without having write permission.

[vanroekel]

Sorry for the stupid question, but here you mean just make sure group has full rwx permissions after I put the file in the initial_condition_database and not figuring how to do this through the python script?

[xylar]

No, these files should not be executable. Just group readable and writable. (Folks add execute permission to files a lot that shouldn't have it and this is not a great idea.) To change permissions recursively, do something like:

chmod -R g+rwX .

The capital X indicates execute permission only gets added to directories, not files.

[xylar]

not figuring how to do this through the python script?

Correct, not in the python script.

[vanroekel]

I caught my 'x' and did not add it when I changed permissions. It should just be 'rw' for groups.

[xylar]

It isn't okay to put a continuation line in the middle of a string. Here's how to do that properly:

Suggested change

	self.woa_filename = f'woa23_decav_0.25_sss_monthly_extrap.\
	{datestring}.nc'
	self.woa_filename = f'woa23_decav_0.25_sss_monthly_extrap.' \
	f'{datestring}.nc'

[xylar]

Suggested change

	in_filename = 'WOA23_monthly_sfc_salin.20241030.nc'
	in_filename = 'woa23_decav_0.25_sss_monthly_extrap.20241031.nc'

[xylar]

@vanroekel, if you make my suggested changes, I'll test again.

[vanroekel]

@xylar Thanks for the testing. I've taken on your changes and run the full test top to bottom on a different machine and it worked. This should be ready for your retesting. I also changed permissions on the files I added to the initial_conditions_database.

[xylar]

Perfect, I'll re-test as well, just to be safe. Sounds like we're likely good to go.

[xylar]

I ran both tests just fine. I'm seeing:

I just want to make sure we're okay with the extrapolation artifacts at the north pole.

[xylar]

Presumably, this is partly because we extrapolate in lat/lon space, not in real distance.

[xylar]

@vanroekel, this looks great, thanks for your patience and hard work.

I'll merge as soon as I get your reaction to my north pole plot.

[vanroekel]

I'm not sure about the artifacts on the N. Pole. Looks like a fair bit of variability over the pole. I'd ask what @alicebarthel and @milenaveneziani think.

Does this same kind of artifact show up in the initial condition too?

[milenaveneziani]

Kat no longer has the RARE climatologies, so those would have to be recalculated. But of course they wouldn't work for the SO.

[vanroekel]

Thanks @milenaveneziani this is intriguing. For the arctic it looks like since we have weak AMOC we'd probably want to use PHC3. It still would be nice if we could get a sense of if PHC3 is closer to truth. Any thoughts on how to do that?

The story seems switched for the S. Ocean. There are more of those wave like structures in the SSS field and the WOA23 looks smoother.

I'm not sure what the best path is. I'll need to think on this. Let me know what you think after looking at these.

[milenaveneziani]

yes, I agree, and that's likely because of all the new data in the SH that went into making recent versions of WOA. I just don't understand why WOA is so weird in the Arctic..
Again, I can talk with Mike and I can try to recalculate the RARE climos. ASTE is another option for the Arctic, but their domain decomposition is a big put off for me..

[vanroekel]

you're right - the structures seem very odd in WOA23. Could some of the difference be vertical resolution? I recall WOA23 has higher near surface resolution, but don't know what PHC3 is.

What are your feelings on thinking about blending WOA and PHC? Seems scary to me, but I don't know how to pick up what we want in each hemisphere. The other alternative is two sss restoring files that are used for different cases, but I'm not a huge fan of that either. I'm also curious what @xylar an @irenavankova think too.

[proteanplanet]

Just to reiterate @vanroekel's earlier point that not even the authors of PHC recommend it now.

[milenaveneziani]

Those patterns in the Arctic in WOA are worrisome to me. I don't think we want to use WOA-SSS restoring for G-cases that have a north polar ocean focus.
I sent an email to Mike and report back when I hear.

About the vertical resolution, Luke: WOA has a 5m resolution in the upper 100m, PHC has 10m res. Are you thinking that the artifacts are related to vertical interpolation?

[irenavankova]

Looking at the Weddell and Ross patterns, they look much better to me in WOA. I am not sure what happens in Weddell in October though that the salinity increases suddenly over a large area. While the pattern in Weddell looks reasonable for April and July, the peak salinities in front of western FRIS front are too low. They need to reach 34.9 as that is very consistently observed at the sea floor for three decades now. What I see from the WOA plot is 34.5 instead.

When I was looking at the PHC earlier in the Southern Ocean I saw some places with a one cycle in a season and other places with two cycles in a season. I wasn't sure whether that was physical or not.

[xylar]

@vanroekel, (and everyone), I don't feel comfortable with PHC but I do share @milenaveneziani's concern that WOA23 looks weird enough in the middle of the Arctic that it could cause weird behavior there. My guess is that data is just sparse there and no product is well constrained. Even so, some may be more plausible than others.

I continue to feel that salinity restoring is a crude instrument, but particularly anywhere you have salinification (areas of sea ice production and evaporation). Whereas fresher water stays at the surface, saltier water sinks so there's less of a signature in the SSS. So my feeling is partly that, if you are restoring hard enough that the details of the SSS restoring field matter especially where you have sea ice cover, even having the correct SSS field doesn't relate to having the right salt fluxes.

[milenaveneziani]

@irenavankova, about this:

They need to reach 34.9 as that is very consistently observed at the sea floor for three decades now. What I see from the WOA plot is 34.5 instead.

Note that those salinities I plotted are SSS, not Sbot. I can also plot the bottom level (I actually thought about it, but I just didn't want to inundate this PR with plots). Maybe I'll just make a one month plot for the SH only.

[milenaveneziani]

Here is Mike's response, mingled with my own comments:

PHC is the best product for initializing a global model when the first year of the run is some time during the period 1950-1990. This is the period over which the classified Russian Soviet data were collected; these are included in PHC. During this time, non-Soviet data coverage in the Arctic was horrible. Also: the Soviet data that we used was "pre-smoothed" and interpolated, which is why it looks so nice.

Luke, Alice: this makes me think that PHC could work well for the new initial condition process that we discussed (1-year G-case, which of course would start in 1958), but perhaps not for the long, standard G-cases.

However, since 1990 the Arctic T and S fields have of course changed a lot. "Atlantification," freshening in the Beaufort, etc. So PHC is a poor choice for validation of Arctic Ocean simulations in recent years. Also it will drag your model solutions back to pre-1990s conditions if you use it for restoring.

I am less worried about this, and more worried about PHC in the SO.

My impression is that people in Europe have more money for earth science, and often produce better quality data sets. Specifically, NOAA's WOA is often lacking recent observations in the Arctic Seas, owing to a lack of human resources for collecting and QC'ing the data. Meanwhile, the UK Met Office's hydrographic product called EN4 includes both WOD profiles and other, more recent data, and so I believe it is the best product to use now for Arctic research (and probably global research). Although it is not perfect! We have found static instabilities and other errors in it. EN4 provides T and S profiles (like NOAA's WOD) as well as gridded T and S fields for long-term climatology (like NOAA's WOA) as well has monthly mean fields with interannual variability. The latter likely defaults to climatology often, since we don't have lots of data every month, but still it could be nice.

For some reason, I had forgotten about EN4. I'll see if I can find climatologies easily; I know I have downloaded it a while back, but I can look into it again.

My conclusion in all this is similar to yours, Luke: there is likely no perfect product for the whole globe. One option is to allow (within compass) for a couple of choices, say WOA and EN4, and then the user decides what to do when they are building their cases from scratch.

[irenavankova]

@irenavankova, about this:

They need to reach 34.9 as that is very consistently observed at the sea floor for three decades now. What I see from the WOA plot is 34.5 instead.

Note that those salinities I plotted are SSS, not Sbot. I can also plot the bottom level (I actually thought about it, but I just didn't want to inundate this PR with plots). Maybe I'll just make a one month plot for the SH only.

Yes, I am aware SSS was plotted. But I thought that for 34.9 to be observed on the sea floor, it must have been at least that high at the surface at some point prior to that.

[xylar]

Regarding EN4, I have used it in ISMIP6. If it has been updated since then to include newer (especially seal-based) Antarctic observations, it might be worth considering. @vanroekel tried their 1900 climatology as an initial condition and it didn't help whatever the issue was at that time (Compass still supports that).

We can give it a go...

[milenaveneziani]

@irenavankova : here is WOA bottom salinity (1500 m) for April (never mind the title, and also note the different colorbar levels than before):

values are more 34.7 than 34.9.

[xylar]

So it's basically missing the continental shelf?

[vanroekel]

I guess we can try EN4 and see if that's reasonable. What should we use for the climatology? 1993-2023? Getting that processed will be trickier since the data I'm aware of is by year. I can't seem to find the climatology Mike mentions.

[vanroekel]

Before changing this PR though, Milena can you make some plots of EN4? This is where I've found data - https://www.metoffice.gov.uk/hadobs/en4/download-en4-2-2.html

I'd prefer to find a product that is acceptable than try have multiple G-case configs.

[xylar]

From what I can tell, EN4's gridded products use a climatology from 1970-2000 wherever data is missing. That worries me for creating a climatology that's ostensibly supposed to be more recent.

In general, I worry about creating our own climatology.

[vanroekel]

I agree @xylar tinkering around with making our own climo is worrying. Is it worth exploring other editions of WOA? CESM uses WOA18.

As I've thought about this, I'd really prefer to use a single product and not blend or have a bunch of different options. Maybe the available data precludes this though

[xylar]

I am in the process of switching from WOA18 to 23 in MPAS-Analysis. The differences seem to be negligible.

[alicebarthel]

Thanks, that's helpful to hear Mike's comments about PHC. I agree with @vanroekel and @xylar's concern about making our own dataset. My main concern in bringing it up was to have a sense for the agreed-upon best estimates, or the uncertainty range associated with monthly values in the Arctic. If we are simply concerned about the polar features (and satisfied with the magnitude), would a small degree of smoothing for the SSS restoring fields be an option? SSS restoring is already a model-per-model decision, and it sounds like PHC was smoothed. That would avoid impacting the broader E3SM/B-case methodology.
The discussion as to which dataset is best for initialization is still valid, particularly if EN4 looks significantly better.

[vanroekel]

@alicebarthel I was thinking about smoothing as well. That seems like a potentially good option.

Agreed on initialization being a valid discussion, but that should be a different issue and not a part of this PR.

[xylar]

Sorry @vanroekel for continuing to hijack this PR. We should move this discussion somewhere else. Someone feel free to make a space for this and we can move it there.

I was actually thinking to do two climatologies: one pre-1990 and one 1990-2024. I don't see a problem with doing that, as far as we document what we did. Not sure why you are concerned?

My concern is with the strategy they use to fill in the gaps. They use a 1970-2000 climatology. That means if you do a 1990-2024 climatology, rather than having holes where there was no data, you will have 1970-2000 data. In my view, it would be better to have the holes and interpolate/extrapolate to fill them in rather than fill them in with data that is 20 years older than you think you have.

[vanroekel]

How should we finish this PR? Should we add some smoothing? @milenaveneziani do you want to plot SSS from EN4?

[milenaveneziani]

I thought people didn't want to compute climatologies with EN4?
I am still not against it (I understand the argument, but I feel that, for the specific purposes of this PR, it's not super relevant). But it's a bit more work dealing with EN4, so I want to make sure people agree.

[vanroekel]

my initial feeling is to just try some smoothing. I think where/when we have a more robust IC discussion, we may end up revisiting SSS restoring to be consistent with the IC chosen.

[milenaveneziani]

Yes, smoothing of the WOA field sounds good: hopefully it helps.
I would treat the whole EN4 issue as separate discussion.

[vanroekel]

Let's plan on trying smoothing in this PR and in the IC discussion if we switch from WOA we definitely want to reconsider SSS

[vanroekel]

@xylar I hope to return to this this week, do you have any advice on how to explore smoothing? I was thinking the smoothing should be in the step where the extrapolated WOA file is made.

[xylar]

@vanroekel, yes, I agree. It should be possible to call _extrap_with_halo() a few extra times once you're done extrapolating and let that do the smoothing for you. Unlike for extrapolation, you would use the full smoothed array, not just the portion that was previously invalid. Let me know if that's clear or if you need further details. The number of iterations of smoothing should be a config option.

[milenaveneziani]

Just to close the loop on the EN4 question: as per our slack exchange with Luke and Alice, we have decided against using EN4 salinity, since the climatology that I computed (using the same years that they use for filling the gaps: 1971-2000) showed that 1) EN4 SSS seems even fresher than WOA in the Eastern Arctic and 2) smoothing would still be necessary for EN4. Xylar: if you are interested in looking at EN4 SSS (or bottom level salinity) for the Southern Ocean, let me know, and I can send the plots to you and Irena via slack.