Rethinking the infamous standard_names

[amdasilva]

Topic for discussion

Perceived Issue

I hope I am not starting a holy war, but for full disclosure, I really hate the standard_names.

While there is definitely much value in the standardization of variable names, I am probably not alone saying that the standard_names in CF is perhaps one of its most confusing aspects and the main impediment for its full adoption. While CF stands for Climate and Forecasting, many of the standardization provided by CF goes beyond the climate and forecast community. Except, perhaps, by the infamous standard_names. Some of the issues are:

• The CF standards_name tries to be many things to too many people (read: communities)
• It has a centralized governance model
• It does not allow for (decentralized) systematic extensions to serve specific communities.

The current standard_names confound 2 main attributes of a variable: a) fundamental data type (pressure, temperature, relative humidity, etc.) and b) what we could broadly call data context. For example, air_pressure_at_cloud_base has pressure as the fundamental data type, and air at cloud base as the context. In other words, the characterization of a variable is at least 2D: (fundamental data type, context).

Suggestion: break up the standard_name

While the fundamental data types are universal and easier to standardize, the data context is better standardized by specific communities. Thus the foundational CF standard should not go beyond the fundamental data types, and be extensible to multiple contexts managed by specific communities. These would be much smaller tables. A contextual table for the Climate-Forecast community would be in order, extensible by other communities, say with a global attribute:

 :Conventions = "CF-1.x" ;
 :DataContext_Conventions = "CF-1.x   Other-1.0"

So, this extends the CF data contexts with additional contexts by the Other-1.0 data context conventions.

[ChrisBarker-NOAA]

Thanks for bringing this up -- I do think more "structure" to the standard names would be helpful.

I also have noted in some of the discussion of new standard names that sometimes none of the CF committee folks involved in the discussion really have the expertise to evaluate the name in a particular field. For the most part, we trust the proposer, but that may not always be adequate.

However, one problem is that it's hard to find people with the time, interest and expertise for this kind of work -- so it may be a challenge to flesh out the committees for each "specific communities" -- who would do the work?

Perhaps an incremental solution would be to more formalize the "categories" of standard names, currently:

Atmospheric Chemistry Atmosphere Dynamics Carbon Cycle Cloud HydrologyOcean Dynamics Radiation Sea Ice Surface

(which, I note, aren't quite aligned with disciplines, at the moment.)

Once that more formal structure is established, it would be easier to change the governance for some or all of the categories.

Also -- is there any precedent for a community coming up with a set of standard names for a particular field, and maintaining it either independently, or propose it en-mass to the CF standard name table? I ask, because I have started such an effort for the oil spill modeling community. We are working on a set of standard names for the results from oil spill models. Some of them are quite field specific, so it may not make sense to mange them the same way as all the other standard names.

[sethmcg]

I don't disagree, but I'm not sure I agree either. There are definitely things I would want to do differently if we were (re-) designing the standard_name system from scratch, and I think there's a lot of appeal to the idea of distinguishing between the fundamental data type and the data context.

However, the more I think about it, the more complicated it gets. There are plenty of cases where it's easy to separate things that way, but there are also a lot where things are not as clear. air_pressure_at_cloud_base and air_pressure_at_cloud_top are clearly talking about the same fundamental quantity in different contexts. But is that the same fundamental quantity as sound_pressure_in_air? Or sound_pressure_in_water? There are senses in which ambient air pressure is the same thing as sound pressure in air, and likewise for sound pressure in air vs water, but while I can think of cases where comparing the first pair or the last pair makes sense, the idea that air_pressure_at_cloud_base and sound_pressure_in_water are the same geophysical quantity in different contexts sure seems wrong.

Ditto with, say, incoming solar radiation at TOA and the irradiance of a wildfire. They're both radiative energy fluxes, but are they really the same thing? And energy is energy; you can combine enthalpy and potential energy into dry static energy; is enthalpy diffusion the same thing as heat advection? In one sense, it's all energy moving from point A to point B, but in another, those are very different things.

I don't think there's a single right answer to the question of what the fundamental data types are, and that's what makes ontology difficulty and messy. Everything is context-dependent, which is how we have ended up where we are today. (Or if there is a single answer, it's already captured in the units attribute: everything with units of 'K' is fundamentally a temperature, and the standard_name is actually entirely data context.)

[amdasilva]

I agree, what exactly constitute a fundamental data type (FDT) requires some thought and some systematic criteria to handle the cases you mentioned. For example, I'd say that reflectance is a FDT while TOA and surface are contexts. Perhaps air_pressure and sound_pressure should be FDTs; clearly 2m, 10m, 50m are contexts, this is what we would be trying to disentangle. Units are not enough: temperature and brightness temperatures are both expressed in K but are different animals.

[sebvi]

The building block approach suggested here is exactly the approach used in GRIB2. I would add to this that sometimes the standard name is not enough, one must also look at the cell_methods attribute for additional post processing.

[sethmcg]

But all of that aside, probably the bigger barrier is backwards-compatibility. One of the principles of CF is not to invalidate all of the existing data that's out there, so if you want to make a sweeping redesign of the standard_name system, you'll need to figure out a way to do it that handles that problem to get any traction.

[sebvi]

I never really understood the issue of backward compatibility in CF. My understanding is that the global attribute "Conventions" is there to prevent misinterpretations due to changes in the convention. But if each new version of the conventions is systematically additive wrt the previous version you simply always use the latest available version of the convention to interpret your metadata and there is no need to specify a version at all in the metadata.
In other words you don't need a convention versioning if backward compatibility is enforced by design, you only need "Conventions: CF ".

[davidhassell]

Hi Sebastien, one facet of backwards compatibility is that a software library coded to version X will always be able to read datasets for all versions up to and including X (as well as many datasets with versions later than X). This is good, because if older CF datasets need different software to read them then newer versions, then it won't be long until these older versioned datasets are unreadable, once the older software stops being maintained.

But if each new version of the conventions is systematically additive wrt the previous version

Indeed - that is usually how we usually ensure backwards compatibility :)

In other words you don't need a convention versioning if backward compatibility is enforced by design

Not so - You still need to know which attributes are standardised at a given version, so you know whether or not to apply a CF interpretation to them. For instance, when faced with a geometry attribute, should you interpret its contents as the name of a geometry variable container, or treat it as an arbitrary value that needs no special treatment? You decide by seeing if the version is earlier than CF-1.8, or not. This example of geometry cells was a wholly additive change, but knowledge of the dataset version is still crucial.

Cheers,
David

[ChrisBarker-NOAA]

Also -- if you have software that supports, e.g. CF 1.9, it may not be able to read CF 1.12 files.

Backward compatibility is just that -- backward, not forward. This is the case with an awful lot of software as well.

Also -- there may come a day in the future, when we do create a non-backward compatible version -- that would require bumping the major version number -- e.g. CF 2.0 -- at least if we are following semantic versioning, or something like it.

[amdasilva]

This is the crux of the matter: if not full backward-compatibility, at least a well-defined process for nearly pain-free migrating from the old to the new conventions. At some point old ways need to be deprecated so that we can move forward.

[ChrisBarker-NOAA]

a lot of appeal to the idea of distinguishing between the fundamental data type and the data context.

there's more than one way to "group" things. One way is to use naming conventions, and a single namespace, which is exactly what CF standard names do -- that's what all the underscores are about, e.g. , to use the OP's example:

air_pressure_at_cloud_base has pressure as the fundamental data type, and air at cloud base as the context.

in the CF way, "air_pressure" is the fundamental data type -- is it fundamental? maybe. you could say pressure is the true fundamental type -- but that's also the unit, so maybe it is better to think of "air pressure" as a single thing.

As the standard names grew fairly organically, I don't think its entirely consistent, but it is there to a large extent. And we should (and do already, I think) make sure that any new names follow this scheme consistently.

Perhaps there could be more work don't to codify the "fundamental data types", but maybe we're OK with what we are doing.

I think the other part of this idea is that categories of "contexts" to use the OP's word, could be "standardized by specific communities". I think that can be done without making a change to the scheme -- if an organized group representing a particular community steps up.

[DocOtak]

CF has become, as far as I can tell, the de facto standard for anything climate related which does cross a lot of disciplines. I'm coming at this from an observational oceanographic perspective and having both models and observations use a common data format is basically a miracle. This miracle being enabled, I feel, by the very issues that are pointed out. I want the centralization of all this information within CF and its controlled vocabularies and governance. It brings the communities together around something that just "good enough" even if not "community specific". That is, the situation of every community disliking a common format is much better than communities developing their own formats, CF as a convention might as well not exist then.

Standard names are not trying to be an ontology.

[amdasilva]

The unifying power of CF as an overarching standard is undeniable and one of its main achievements. However, there is something to be said about separation of concerns. No one is proposing a different "format", but rather a systematic process to make extensions (not changes) to the baseline "standard names" that can be tailored to specific communities, perhaps even projects. BTW, this is happening anyway where we simply state that the files are CF-ish. Having a methodical way of introducing well documented extensions to the standard_names would, if anything, strengthen the utility and reach of the CF standard.

[DocOtak]

Why would a community extension need to involve the standard_name attribute at all? Why couldn't they be in other attributes? CF does not restrict adding more attributes that it does not control.

When I first came to CF, I'm not sure I liked the standard names either. They failed to capture nuance that was considered important in my field. Measurement techniques, sampling devices/techniques, even which analytical group (using the same techniques as another group) was considered important. However, we found that most of the data users who were not intimate with the analytical techniques didn't care about this nuance, they only wanted the "good data". We found that the standard names, as they exist right now, forced us to think about the broader reusability of the data we were managing. They seemed to strike this balance of not being too specific or broad and make us think about what the data were (or were trying to be) as a more basic level. To use an idiom, we couldn't see the CF forest for our data trees.

What would the broken up standard names look like? I could image having each component of what @davidhassell has bellow each come from a controlled vocabulary and be composable, but I would still want this to exist centralized in CF. So:

[surface] [component] [base_quantity] [at surface] [in medium] [due to process] [assuming condition]

Could each have their own table of allowed values.

[amdasilva]

Why do I have to involve the standard_name at all? Because unless my variables come annotated by the standard_names sanctioned by appropriated CF committee, I cannot state that my files are CF compliant. What I'd like to state that I started with CF and extended it with standard_names of my choosing. Extensibility is the name of the game: use CF as the basis, make additions that are fit for purpose, when and if needed.

Regarding the granularity of standard_names, see my replay to @davidhassell below.

[sethmcg]

Because unless my variables come annotated by the standard_names sanctioned by appropriated CF committee, I cannot state that my files are CF compliant.

That's incorrect. Section 3.3 says: "each variable may optionally be given a "standard name", whose meaning is defined by this convention." (Emphasis added.)

It is highly desirable to annotate your variables with standard_names (assuming suitable ones exist), but it is not required for data to be CF-compliant.

If you want to use a non-standard name of your choosing, you can do that. Just call it nonstandard_name, or other_standard_name or extended_standard_name or something like that.

[davidhassell]

I also agree with Andrew. The three issues given in the original post also act as advantages to others, particularly those for who regularly use data created by other people.

Just to note that most standard names do follow a general pattern:

[surface] [component] base_quantity [at surface] [in medium] [due to process] [assuming condition]

(see, for instance, CF Standard Names: Current status and Steps toward interoperability in the environmental sciences and Towards an explorer tool for visualisation of grammatical patterns in the CF Standard Names through decomposition into n-grams).

[amdasilva]

David, this general pattern

[surface] [component] base_quantity [at surface] [in medium] [due to process] [assuming condition]

IS the problem. You are trying to express in 1D a piece of information that is fundamentally multi-dimensional. Even acknowledging some sparseness, the combinatorial problem that ensues will make the standard_names table to be a very long table indeed. By breaking this information in multiple attributes you get a much more manageable system.

I am not caught up in the minimalistic fundamental data type/data context of my original post; the base_quantity is probably what I was calling fundamental data type. If you would like to capture the details above,

• surface
• component
• base_quantity [at surface]
• in medium
• due to process
• assuming condition

you have now the chance of standardize each of these. In other words, fully embrace (and standardize) the n-grams mentioned in the poster you referenced above. These would be much smaller tables that now can be convolved to express a much larger number of "variable descriptions".

[larsbarring]

What you describes seems to me as fairly similar to the I-ADOPT approach for mapping the components that together describe a quantity. I-ADOPT has come up several times when discussing standard names and interoperability, most recently at the 2024 CF Workshop, see here. Would this be something that you could build on?

[markusfiebig]

Hi @amdasilva ,

I don't really want to involve myself all too much in this discussion, but if you want an example of an I-ADOPT compliant vocabulary that implements what you are imagining, and that covers at least parts of the CF thematic area, have a look here: ACTRIS Vocabulary.

[JonathanGregory]

Dear all

The concept and construction of standard names has certainly been discussed more than once before, but I don't have a note of which old emails or trac tickets it appears in. In my presentation at the CF workshop in 2018, I tried to answer some questions and objections about the design of standard names. Here are some points from that presentation:

• Standard names answer the question, "What does this variable contain, in physical terms?"

• To prevent profileration of standard names, they don't include numerical metadata or identification of coordinate axes (because these things are done by coordinate variables and scalar coordinate variables), nor metadata about statistical processing and subgrid variation (which is the purpose of cell_methods). Therefore the standard name alone sometimes doesn't provide what other ontologies might regard as the complete description of "a variable", like the CMIP name tasmin, which is the daily minimum air temperature at 2 m above the ground.

• On the other hand, standard names do include a lot of different descriptive properties, which it has sometimes been argued should be in separate netCDF attributes, though I don't think Arlindo @amdasilva is suggesting that. Arguments against are: * it is more convenient to examine one attribute than many; * if a new distinction was introduced all existing programs would have to modified to check it as well as all the others; * some standard names indicate an order of operations; * some include more than one instance of a given concept.

The first point is most relevant to Arlindo's initial comment. I think it's an advantage and an intention of standard names that they should all be self-explanatory to a geoscientist of any field, who isn't necessarily a domain expert. At least then you can tell what the variable is about, even if you don't fully understand it.

One reason for this is interoperability. CF is used for datasets from many disciplines, and some individual datasets are of relevance to many e.g. the output from Earth system models. To make this efficient, or even possible, we all have to speak the same language. That is why standard names are a centrally controlled vocabulary. If they were delegated to separate communities, it is inevitable that some standard names would be jargon or shorthand familiar in the relevant field, but incomprehensible or ambiguous outside it, and some quantities would be named in more than one field, probably with different names. They wouldn't be "standard" names, that serve the CF purpose of allowing users to decide which variables are comparable in datasets from different sources.

The cost of this policy is that deciding on standard names is sometimes hard. It involves people from various fields understanding what the quantity is, and describing it in terms anyone can understand. But with this design, that cost arises only once. If we had standard names which were not generally self-explanatory, then many users would pay the price of trying to reconcile different vocabularies.

The design of standard names also follows the CF principle of addressing present use-cases, rather than trying to anticipate them. It aims to be "good enough", as @DocOtak said. Hence we don't have to address the philosophical question of what is a "fundamental" quantity, such as Seth mentioned. We decide as we go along whether two things are the same or not.

I agree, and we've often discussed, that it would be useful to provide tools that made it easier to propose new names, in the simpler cases at least. It's easy when a new name uses existing vocabulary and patterns, and usually easy if we just need new vocabulary. There is much more consistency in construction than the guidelines alone, because we follow precedent. For example, a long time (14 years) ago, I did an analysis of the standard names of the time into lexicon and syntax. Maybe machine learning could help with this task.

Best wishes

Jonathan

[graybeal]

Maybe this is a “yes-and” situation. Making it easier to extend standard names does not have to mean distributing the responsibility for approving them. If we accept that additional terms are being created in CF-ish vocabularies, it’s easy to imagine—maybe even implement —an API to receive those names and represent them as proposals. They clearly have a use case, and the API could enforce existence tests on required attributes. In an ideal world, the process could identify a matching pattern, or suggest (to submitters or curators) possible patterns to follow instead of what’s submitted, including pre-filling predefined text. Pretty sure LLM could do this pretty well, given a list of existing patterns or just some training on existing names. In the end curation would still be needed, maybe at unsupportable scale. But at least the process would be capturing more real-world use cases.John

[JonathanGregory]

Yes, John, that's what I was thinking as well. I tried asking ChatGPT to suggest standard names a while ago. It wasn't hopeless, but not right either. If it proposed its standard names, we'd still have to work on them, as we often do with proposals from humans. It's possible that the corpus of existing names is not sufficient for it to be able to predict the language.