[pipes] JsonSchema for externals protocol

[smackesey]

Summary & Motivation

Add a script that generates a JSON schema for the externals protocol.

The script uses pydantic and lives in top-level scripts. It writes the json schema to python_modules/dagster-ext/json_schema/{context,message}.json. The script requires pydantic v2 so it must be run through tox -e jsonschema (from dagster-externals) until core is updated.

I wasn't sure how to represent a combined schema for context and message, so I put them in separate schema files.

Also adds a BK step that generates the schema and diffs it against the checked-in version, ensuring nothing has changed.

The schema files are also included in the built dagster-pipes package.

How I Tested These Changes

New unit tests to ensure JSON schema is valid and that context/message objects satisfy it.

[smackesey]

• [pipes] JsonSchema for externals protocol #16009 👈 (View in Graphite)
• [ext] add protocol version to all messages #16633
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[schrockn]

In order to properly review this, I do think we need to see how this json schema would be consumed in other programming language.

E.g. if I'm writing ext in Scala and I want to use the json schema, what does that look like?

[schrockn]

synchronously doing I/O on module import very non-ideal. Can we restructure to do on demand and cache?

[smackesey]

done

[smackesey]

if I'm writing ext in Scala and I want to use the json schema, what does that look like?

Depends on what our Scala ext story integration story looks like. If we have a dedicated lib then we would include this schema in that lib. If we don't then we could provide a CLI method to access it. Either way it falls to whatever JSON schema libs are available in Scala to actually perform validation.

Alternatively we could publish the schema to a public URL and just expose that.

[github-actions]

Deploy preview for dagster-university ready!

✅ Preview
https://dagster-university-41f7eq2x0-elementl.vercel.app
https://sean-json-schema.dagster-university.dagster-docs.io

Built with commit 824b830.
This pull request is being automatically deployed with vercel-action

[schrockn]

Depends on what our Scala ext story integration story looks like. If we have a dedicated lib then we would include this schema in that lib. If we don't then we could provide a CLI method to access it. Either way it falls to whatever JSON schema libs are available in Scala to actually perform validation.

Right. I don't think we need to commit to this schema right now, and I think it make sense to do this when we actually build our first non-Python integration. So my proposal is that we resurrect this diff when we write our first prototype in another language.

[schrockn]

what is this script used for in this PR?

[schrockn]

One sec processing the PR summary

[schrockn]

Makes sense. I just think we need some more testing.

[schrockn]

We need more tests here. We should be building a payload using the APIs used by the framework and validating them against the json schema.

[smackesey]

I added a test to explicitly validate the output of build_external_execution_context_data. This is also already tested a bunch of times above in any test that uses the helper _make_external_execution_context, which creates various flavors of PipesContextData (e.g. with and without partition key).

In the external -> orchestration direction, the existing protocol only defines the high level PipesMessage rather than all the individual messages, and that is tested in test_message_json_schema_validation.

class PipesMessage(TypedDict):
    """A message sent from the external process to the orchestration process."""

    __dagster_pipes_version: str
    method: str
    params: Optional[Mapping[str, Any]]