How should we best add semantics to generated python (and pydantic in particular)

[cmungall]

Currently when compiling to pydantic, we translate a lot of the linkml metamodel

• ranges, cardinality (required, multivalued)
• descriptions become docstrings
• patterns become patterns
• some constraints like maximum value are translated
• any_ofs become unions
• etc

We prioritize things that would be useful for validation. However, there are some things we don't translate

• a lot of metadata that could be useful for documentation purposes, including aliases, provenance, status, ...
• some things that could be useful for runtime introspection such as getting the URI/CURIE of the element
• some things that could be used for validation, e.g. id_prefixes

Some of these could be compiled into existing elements; e.g. fields like aliases could be appended onto the docstring. id_prefixes could be compiled into patterns.

However, it would be better to have a first-class representation of some of this in pydantic, such that things are available for introspection. The obvious way to do this is ClassVars or ModelConfigs.

Note that this wouldn't affect the runtime behavior of pydantic (and would not require any addition dependencies).

Another option is to extend the pydantic metamodel; for example @cthoyt's semantic-pydantic extends Field:

https://cthoyt.com/2024/01/10/semantic-pydantic.html

This is an intriguing possibility. We could have a profile in pydanticgen that would make SemanticFields.

Note however that this would entail a runtime dependency. We have gone down this route previously with the dataclass generator, which adds lots of semantic goodies onto dataclasses, but does come with some downsides in managing dependencies.

There may also be other libraries that are out there that also aim to add semantic annotations onto pydantic.

Perhaps there is a way that allows for composability. This is common in the java universe, and is solved with annotations on the elements. Unfortunately we can't have decorators at the attribute level in python, AFAIK...

[sneakers-the-rat]

IMO it would be nice to go the same was as pydantic and just store this as a model attribute (eg. model_config ) :).

Eg. here I make an actual pydantic model (rather than just a template object) so it can be introspected at runtime:
https://github.com/p2p-ld/nwb-linkml/blob/eac5ef4c80b2ba1ed751ddbe3348a34edaac8a68/nwb_linkml/src/nwb_linkml/generators/pydantic.py#L63

and then have a method to populate it with whatever properties are needed: https://github.com/p2p-ld/nwb-linkml/blob/eac5ef4c80b2ba1ed751ddbe3348a34edaac8a68/nwb_linkml/src/nwb_linkml/generators/pydantic.py#L483

and i've extended the template to allow for passing extra classes because that in general is useful when trying to patch the build process:
https://github.com/p2p-ld/nwb-linkml/blob/eac5ef4c80b2ba1ed751ddbe3348a34edaac8a68/nwb_linkml/src/nwb_linkml/generators/pydantic.py#L675

which ends up looking like this: https://github.com/p2p-ld/nwb-linkml/blob/eac5ef4c80b2ba1ed751ddbe3348a34edaac8a68/nwb_linkml/src/nwb_linkml/models/pydantic/core/v2_6_0_alpha/core_nwb_base.py#L49

and that example is just for the totally trivial thing of storing a single tree_root value, but you could expand that for any property relatively easily, that generator method has access to the whole class definition (which seems like all the scope you would want for something like this).

being a ClassVar keeps it out of model_dumps, but makes it accessible when you want to introspect on the class itself, which is when you would want to have this information.

[cmungall]

Should there be a separate LinkML_Meta per package? Or import this from a runtime library?

[sneakers-the-rat]

I dont have strong feelings. I do think it would be good to split linkml generators and runtimes into a package per format, maybe as namespace packages within linkml as a monorepo, so in that way I think it would be good to import.

Since it is (should be, imo?) a thin wrapper for a dict, the main benefits of importing would be (aside from aesthetics) a) to be able to do type introspection to find or check for metadata beyond naming convention, and b) for testing? What else am I missing.

For a) we could override the subclass check method, but that makes it even uglier and harder to test, and b) ATM ya we would have to compile the string. In my draft its only a string because pydantic 2 attr errors if you try and use the pydantic 1 style root model syntax, but after 1 is deprecation that wont be a problem.

Basically ya if you want to start a linkml-pydantic package I am into it