-
-
Notifications
You must be signed in to change notification settings - Fork 280
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal for Unparsable Remote References (mainly to support XML.) #624
Comments
I think this proposal addresses, in a generic way, an issue that will likely continue to re-emerge. OpenAPI naturally uses JSON due to its focus on synchronous RESTful interactions. The async world is much more diverse in both protocols and data formats. Data formats include things like Avro, Protobuf, XML, EDI, and the inevitable "cool new format" that will emerge next year. It feels like we need an extensible way to accommodate that diversity without needing to explicitly include it in the spec. Longer term, we may need to have something like a "format binding" (a parallel to protocol bindings.) The format binding would that provides format specific fields. For example, an XML format binding could include a namespace. But that seems like a lengthy, heavy lift, and @damaru-inc proposal seems like a good first step that will get early adopters off the ground. Also, here is an example of what a protobuf implementation would look like:
|
Interesting concept. Maybe it would be enough to add a flag which would specify whether a given schema should be parsed/formatted or not. We could extend my proposal to define schemas in other formats in different places (currently it is possible only in message's payload) - #622 and add to Schema Object a messages:
myXmlMessage:
payload:
parse: false
schema:
$ref: "https://example.com/myschema.xsd"
contentType: "application/xml" which would mean that it should not be parsed and transformed to JSON, but only resolved/fetched. We can also use the
|
@jmenning-solace Could you create issue about (as you described it) |
Currently we have many objects defined in XSD. These historical objects can be repurposed as event attributes. This unlocks tremendous value(financial, technical) if re-use can be accomplished. Looking forward for this discussion and feature availability. |
@magicmatatjahu, my concern is this would create valid AsyncAPI documents that are invalid JSON documents. So where do we go from there? I see two options (would love to hear others):
My preference would be the second.
This is an interesting concept, and maybe that's the appropriate place for the payload binding. Let me think more on that. It seems extendable to these use cases, with the caution that Avro is an easier format to deal with because it's JSON. |
Thanks for chiming in @masterhead , it's nice to get an end user perspective. Can I ask you a couple questions about your use case?
|
@jmenning-solace
Do you know, that // some_xsd.xsd
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:tns="http://tempuri.org/PurchaseOrderSchema.xsd"
targetNamespace="http://tempuri.org/PurchaseOrderSchema.xsd"
elementFormDefault="qualified">
<xsd:element name="PurchaseOrder" type="tns:PurchaseOrderType"/>
<xsd:complexType name="PurchaseOrderType">
<xsd:sequence>
<xsd:element name="ShipTo" type="tns:USAddress" maxOccurs="2"/>
<xsd:element name="BillTo" type="tns:USAddress"/>
</xsd:sequence>
<xsd:attribute name="OrderDate" type="xsd:date"/>
</xsd:complexType>
<xsd:complexType name="USAddress">
<xsd:sequence>
<xsd:element name="name" type="xsd:string"/>
<xsd:element name="street" type="xsd:string"/>
<xsd:element name="city" type="xsd:string"/>
<xsd:element name="state" type="xsd:string"/>
<xsd:element name="zip" type="xsd:integer"/>
</xsd:sequence>
<xsd:attribute name="country" type="xsd:NMTOKEN" fixed="US"/>
</xsd:complexType>
</xsd:schema>
# asyncapi.yaml
asyncapi: '2.1.0'
info:
title: Account Service
version: 1.0.0
description: This service is in charge of processing user signups
channels:
user/signedup:
subscribe:
message:
$ref: '#/components/messages/UserSignedUp'
components:
messages:
UserSignedUp:
payload:
type: object
properties:
displayName:
type: string
description: Name of the user
email:
type: string
format: email
description: Email of the user
someCustomProp:
$ref: ./some_xsd.xsd and then after dereferencing I have: {
"asyncapi": "2.1.0",
"info": {
"title": "Account Service",
"version": "1.0.0",
"description": "This service is in charge of processing user signups"
},
...
"components": {
"messages": {
"UserSignedUp": {
"payload": {
"type": "object",
"properties": {
"displayName": {
"type": "string",
"description": "Name of the user",
"x-parser-schema-id": "<anonymous-schema-2>"
},
"email": {
"type": "string",
"format": "email",
"description": "Email of the user",
"x-parser-schema-id": "<anonymous-schema-3>"
}
},
"someCustomProp": "<xsd:schema xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:tns=\"http://tempuri.org/PurchaseOrderSchema.xsd\" targetNamespace=\"http://tempuri.org/PurchaseOrderSchema.xsd\" elementFormDefault=\"qualified\"> <xsd:element name=\"PurchaseOrder\" type=\"tns:PurchaseOrderType\"/> <xsd:complexType name=\"PurchaseOrderType\"> <xsd:sequence> <xsd:element name=\"ShipTo\" type=\"tns:USAddress\" maxOccurs=\"2\"/> <xsd:element name=\"BillTo\" type=\"tns:USAddress\"/> </xsd:sequence> <xsd:attribute name=\"OrderDate\" type=\"xsd:date\"/> </xsd:complexType>\n<xsd:complexType name=\"USAddress\"> <xsd:sequence> <xsd:element name=\"name\" type=\"xsd:string\"/> <xsd:element name=\"street\" type=\"xsd:string\"/> <xsd:element name=\"city\" type=\"xsd:string\"/> <xsd:element name=\"state\" type=\"xsd:string\"/> <xsd:element name=\"zip\" type=\"xsd:integer\"/> </xsd:sequence> <xsd:attribute name=\"country\" type=\"xsd:NMTOKEN\" fixed=\"US\"/> </xsd:complexType> </xsd:schema>",
"x-parser-schema-id": "<anonymous-schema-1>"
},
...
}
}
},
"x-parser-spec-parsed": true
} So the dereferencer even fetches something from the web it still treats it as a string value, and only if it's valid JSON, i.e. a value starting with If you are talking about this case with making references that should not be fetched, there is now a possibility to use e.g. an extension for this case: messages:
myXmlMessage:
$ref: "https://example.com/myschema.xsd" # it will be fetched and treated as string value
contentType: "application/xml"
x-remote-ref: "https://example.com/myschema.xsd" # point to this reference that can be used in generators Another possibility is to add each |
We would not want full text of XML schema in the AsyncAPI file. More on the lines of remote pointer, where our tools can parse to provide how a sample payload look for user understanding. If it is remotely hosted on URL https://domain/ssss/schema_def it works good. But some times the URL may be relative path to AsyncAPI file too ) eg
For historical reasons, currently majority of objects are definitions are already in the WSDL which have to liberated and defined as events. It would be ideal if spec can support XSD for any combination of multiple objects. (Could re-use existing infrastructure of already defined XSD as-is instead of manually editing and creating xsd of each event object)
|
I tried your example, and it does work. My concern here isn't so much with your proposal, but with the fact that at least our parser-js treats $ref differently depending on whether it's under message/payload or above it. And what should we call 'someCustomProp?' At least with remoteReference, we can put that somewhere like message/payload and then its purpose becomes clear, and that also gives us a standard name for a property that we can use elsewhere. |
Most probably you mean the situation when you make a reference to the schema, e.g. xml, but then get an error from the parser that it can't parse that? Here I have to tell you that our parser doesn't treat
If one only needs a reference and not a value from a reference then |
I agree that we need a way to keep the reference. The current parser always attaches its own internal schema-id to anything that parses as a schema, e.g.
so that would be the logical place to put whatever kind of schema id we want. cheers |
Hey @magicmatatjahu, I had a simple NodeJS application using the
What are your thoughts on this? |
@TamimiGitHub Hi! You have error, because you try pass the non JSON as argument to the {
"test": "test",
"reference": {
"$ref": "./sampleXML.xsd"
}
} and then after dereferencing I have: {
"test": "test",
"reference": "<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:tns="http://tempuri.org/PurchaseOrderSchema.xsd" targetNamespace="http://tempuri.org/PurchaseOrderSchema.xsd" elementFormDefault="qualified"> <xsd:element name="PurchaseOrder" type="tns:PurchaseOrderType"/> <xsd:complexType name="PurchaseOrderType"> <xsd:sequence> <xsd:element name="ShipTo" type="tns:USAddress" maxOccurs="2"/> <xsd:element name="BillTo" type="tns:USAddress"/> </xsd:sequence> <xsd:attribute name="OrderDate" type="xsd:date"/> </xsd:complexType> <xsd:complexType name="USAddress"> <xsd:sequence> <xsd:element name="name" type="xsd:string"/> <xsd:element name="street" type="xsd:string"/> <xsd:element name="city" type="xsd:string"/> <xsd:element name="state" type="xsd:string"/> <xsd:element name="zip" type="xsd:integer"/> </xsd:sequence> <xsd:attribute name="country" type="xsd:NMTOKEN" fixed="US"/> </xsd:complexType> </xsd:schema>"
} |
Many thanks to @magicmatatjahu , @TamimiGitHub , @masterhead and @damaru-inc for helping me understand the implications here. I've learned a lot thanks to you all. In an attempt to summarize, I wanted to walk through a couple scenarios and see if we are the on same page I want to have entire .xsd imported into AsyncAPI, as a string
For instance, if xsd has multiple root elements:
|
The described proposal/problem itself is related to my proposal, which I extended to use references to nested non-JSON schema objects - Proposal to allow defining schema format other than default one (AsyncAPI Schema) - please see section @jessemenning You may be interested in this :) |
This issue has been automatically marked as stale because it has not had recent activity 😴 It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation. There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model. Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here. Thank you for your patience ❤️ |
Still valid. @derberg Could you remove |
Any conclusion on this topic ? What needs to happen to get that into the next releases ? |
We definitely need a champion that wants to drive the change, come up with proposal, respond to feedback, and present it to others |
Apologies for letting this lie dormant for so long. Anyway, the proposal, as I see it, is what I wrote here with the added refinements that Jessie made. I think we've responded to feedback (tell me if I missed something.) Here it is presented to others. Is the next step, then, to merge Jessie's suggestions in with my original proposal and re-present it? Or are the next steps to actually do PRs against the spec and the parser? If the latter, I'd be happy to do a PR against the spec, but I'm not the best person to add features to the parser. |
This issue has been automatically marked as stale because it has not had recent activity 😴 It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation. There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model. Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here. Thank you for your patience ❤️ |
Solace has customers who love AsyncAPI but use XML payloads in their messaging systems. Also, the question of whether things like XSD schemas are supported has come up more than once in the Slack channels.
We would like to propose the notion of an Unparsable Remote Reference. These would be, at minimum, URLs represented by simple strings. By Unparsable we mean that in general, AsyncAPI parsers would not be expected to retrieve and/or parse the entities pointed to by these references. Code generators, on the other hand, could use these references.
The use case we are trying to solve immediately is how to provide a URL to an XSD schema, so that a code generator could created a model class from the schema and use it with XML libraries for serializing messages.
One simple way to do this is one that requires no change to the specification nor to the parser. A message could look like this:
(This fragment works fine with the current parser, you can try it in the playground.)
When this message is passed back from the parser, the payload contains an anonymous schema containing the field remoteReference with its value.
An improvement would be to create a parser plugin (similar to the avro parser. That would allow us to also specify the schemaFormat (currently the parser will fail if you try to set the schemaFormat to
application/xml
- that won't work now because there is no schema parser defined for that format.)Yet another improvement would be to allow an object representing a schema registry, in cases where it would be desirable to add more fields besides just a URL.
This mechanism would also allow users to use Avro files in their original form (the current avro parser translates to JSON schema), and it could also be used to support protobuf or any other kind of schema.
The name remoteReference was intended to be general enough to be applied to other use cases, not just non-JSON schemas.
Ideally it would be nice to have a standard, documented way to do this.
The text was updated successfully, but these errors were encountered: