Data Lineage in brightway and LCA workflows

[cswh]

This discussion continues the initial ideas from
https://github.com/Depart-de-Sentier/brightcon-2024-material/tree/main/hackathon/data-lineage

Here we can gather ideas, needs, things to consider, Use Cases, etc. for the integration of data lineage within brightway and LCA workflows in general.

The most important points so far:

• The OpenLineage project allows for cross-platform data lineage and provides a python client
• Via additional blinker signals within brightway it would become very easy to add data lineage events when "permanent" data is being changed.
• Using the included http tansport type for the client would make it easy to send data lineage events to a running marquez server. This would fit well in a not-only brightway workflow in a standard way especially thinking on orchestrated services/models which could be part of the sentier.dev platform at some point.
• However running a marquez server for a pure personal workflow on your local machine can become a barrier for some users.
• OpenLineage allows for custom transport types which would allow to store the data lineage events within the datastores of brightway. That way the data lineage could as well be part of a brightway project export and be used by other packages like the Activity Browser.

If we want to go that way, we should have some common understanding of the OpenLineage Object Model and probably some custom facets within the LCA context. Therefore this discussion group.

[marc-vdm]

Activity Browser team listening in 👂

@mrvisscher

[FRaymand]

I'd be interested in helping out!