Multimodel fails for daily data with different calendars

[katjaweigel]

Describe the bug
A multi-model mean, which was running for the last core version now fails because:

Encountered the following exception: failed to merge into a single cube.
Coordinates in cube.aux_coords (scalar) differ by dtype or class (i.e. DimCoord vs AuxCoord).

Different to the case reported here: ESMValGroup/ESMValTool#2198
the recipe does not use "extract_level" but "extract_region" and "extract_season", could this be connected?

Please attach

• The recipe is attached here, it belongs to the PR PV capacity factor for ESMValTool GMD paper  ESMValTool#2153 There the multi-model mean is currently turned off.
  recipe_pv_capacity_factor.yml_coord_issue_multi_model_mean.txt

• The main_log_debug.txt file:
  run_main_log_debug_coord_issue_multi_model_mean.txt

[sloosvel]

I doubt it's a problem with extract_region, because this feature was added so the extract_region step would work on the fx vars. But I will take a closer look.

[sloosvel]

Sorry I just checked the recipe, and at no point fx variables are being used. So if the multimodel step is failing it must be due to something else, and it is not related to what was reported in ESMValGroup/ESMValTool#2198 with the extract_levels function. The most recent changes in the multimodel stats were made in #1150 .

[katjaweigel]

Ok, thanks a lot @sloosvel! I tested without extract_season but with multi model mean now, and there is a new error:

ERROR [32820] Failed to run multi_model_statistics({<esmvalcore.preprocessor.PreprocessorFile object at 0x2b713a5d4640>, <esmvalcore.prepr
ocessor.PreprocessorFile object at 0x2b713ab42c70>, <esmvalcore.preprocessor.PreprocessorFile object at 0x2b713ab42e80>, <esmvalcore.preproc
essor.PreprocessorFile object at 0x2b713a5d4ca0>, <esmvalcore.preprocessor.PreprocessorFile object at 0x2b713ab422b0>, <esmvalcore.preproces
sor.PreprocessorFile object at 0x2b713ab424c0>, <esmvalcore.preprocessor.PreprocessorFile object at 0x2b713ab510d0>, <esmvalcore.preprocesso
r.PreprocessorFile object at 0x2b713a5d4730>, <esmvalcore.preprocessor.PreprocessorFile object at 0x2b713ab42970>, <esmvalcore.preprocessor.
PreprocessorFile object at 0x2b713a5d4580>, <esmvalcore.preprocessor.PreprocessorFile object at 0x2b713ab513d0>, <esmvalcore.preprocessor.Pr
eprocessorFile object at 0x2b713ab515e0>}, {'span': 'overlap', 'statistics': ['mean'], 'output_products': {'mean': <esmvalcore.preprocessor.
PreprocessorFile object at 0x2b713a5d4610>}})

...

IndexError: index 9490 is out of bounds for axis 0 with size 9490

There are several Warnings (which are also there in the case above when the multi-model mean does not run with extract_season and extract_region).

WARNING [32819] /work/bd0854/b380216/anaconda3/envs/esmvaltool202106/lib/python3.9/site-packages/iris/coords.py:1979: UserWarning: Collapsing a non-contiguous coordinate. Metadata may not be fully descriptive for 'height'.
warnings.warn(msg.format(self.name()))

WARNING [32819] /work/bd0854/b380216/anaconda3/envs/esmvaltool202106/lib/python3.9/site-packages/iris/coords.py:1979: UserWarning: Collapsing a non-contiguous coordinate. Metadata may not be fully descriptive for 'time'.
warnings.warn(msg.format(self.name()))

WARNING [32820] /work/bd0854/b380216/anaconda3/envs/esmvaltool202106/lib/python3.9/site-packages/iris/coords.py:1979: UserWarning: Collapsing a non-contiguous coordinate. Metadata may not be fully descriptive for 'multi-model'.
warnings.warn(msg.format(self.name()))

[katjaweigel]

Here is the complete log and the recipe.
run_main_log_debug_coord_issue_multi_model_mean_noseason.txt
recipe_pv_capacity_factor.yml_coord_issue_multi_model_mean_noseason.txt

[valeriupredoi]

@katjaweigel thanks for the detailed error report and logs. The last error tells me that a certain cube from the cubes list has the time axis shorter by exactly one point than the first cube in the list, which should not happen since they must have the same length of the time axis. I'll investigate on JASMIN

[valeriupredoi]

OK a first pass explains why the merge is failing - ACCESS1-3 has AuxCoord height at 1.5m as compared to (a subset of) the other models that have it at 2m, I had to use a subset of the models since the code is extremely slow, and using all the models I'd wait until England-Germany football game ⚽

[valeriupredoi]

thanks very much @katjaweigel - I've opened an issue in Core related to this, I will revert to the older 2.2.0 module for now, the new module is unacceptably slow and does contain the bugs you found yourself. I'll ping you here when I have reverted to the old module so you can run (just tested it myself and runs fine and fast) 👍

[katjaweigel]

@valeriupredoi ok, thanks a lot!
That probably causes the

WARNING [32819] /work/bd0854/b380216/anaconda3/envs/esmvaltool202106/lib/python3.9/site-packages/iris/coords.py:1979: UserWarning: Collapsing a non-contiguous coordinate. Metadata may not be fully descriptive for 'height'.
warnings.warn(msg.format(self.name()))

Warning and following issues.

The reason for the

WARNING [32819] /work/bd0854/b380216/anaconda3/envs/esmvaltool202106/lib/python3.9/site-packages/iris/coords.py:1979: UserWarning: Collapsing a non-contiguous coordinate. Metadata may not be fully descriptive for 'time'.
warnings.warn(msg.format(self.name()))

Warning and the

IndexError: index 9490 is out of bounds for axis 0 with size 9490

Error is most probably, that there are is a different number of days in the 26 years the recipe tries to look at, either 9490 days or 9497 days (probably depending on if the calendar contains leap years or not).
I'll try if "regrid_time" would help.

[katjaweigel]

regrid_time:
  frequency: day

Doesn't solve the leap year issue, its fails with:

IndexError: list index out of range
2021-06-29 12:27:34,816 UTC [8169] INFO esmvalcore._main:444

and there are still cubes with 9490 and 9497 days.

[Peter9192]

I don't think the height coord is a problem after all. Will write the details in #1204.

The leap years issue also triggers a warning (I think this is quite nice, actually):

6-29 08:15:56,307 UTC [32819] WARNING esmvalcore.preprocessor._multimodel:117 Multimodel encountered (sub)daily data and inconsistent time units or calendars. Attempting to continue, but might produce unexpected results.

Last thing to mention: these annoying iris warnings:

WARNING [32819] /work/bd0854/b380216/anaconda3/envs/esmvaltool202106/lib/python3.9/site-packages/iris/coords.py:1979: UserWarning: Collapsing a non-contiguous coordinate. Metadata may not be fully descriptive for 'height'. warnings.warn(msg.format(self.name()))

are silenced in #968 (see this commit).

[Peter9192]

@katjaweigel can we change the title of this issue to something like "multimodel fails for daily data with different calendars"?

[katjaweigel]

@Peter9192 sure, you can change it, since there is an own tas issue (#1204) now (sorry somehow I overlooked this and only saw the PR).