Dereferencer should work with AsyncAPI "payload"

[nadjahagen]

Hello,
I wanted to test Apicurio and the Apicurio Data Models library with a similar AsyncAPI definition to this example:

{
  "asyncapi": "2.1.0",
  "info": {
    "title": "Account Servic",
    "version": "1.0.0",
    "description": "Manages user accounts in the system.\n",
    "license": {
      "name": "Apache 2.0",
      "url": "https://www.apache.org/licenses/LICENSE-2.0"
    }
  },
  "servers": {
    "production": {
      "url": "localhost:5672",
      "protocol": "amqp",
      "description": "Test RabbitMQ broker"
    }
  },
  "channels": {
    "user/signedup": {
      "subscribe": {
        "summary": "Receive information about user signed up",
        "operationId": "receivedUserSignedUp",
        "message": {
          "description": "An event describing that a user just signed up.",
          "contentType": "avro/binary",
          "schemaFormat": "application/vnd.apache.avro+json;version=1.9.0",
          "payload": {
            "$ref": "https://raw.githubusercontent.com/sijie/pulsar-avro-schema-example/master/src/main/avro/twitter.avsc"
          }
        }
      }
    }
  }
}

When checking the implementation of the Dereferencer class, I noticed that the references are only resolved for objects of type IReferenceNode. While AaiMessageBase is of type IReferenceNode, this is not the case for the payload object in AaiMessage which is only of type Object. Therefore, references within the payload object will not be resolved.

In contrast to that, a reference like that can be resolved by the AsyncAPI Playground. According to the AsyncAPI spec, the payload object is defaulting to a Schema object, but can also be a $ref to a remote file or any other type.
So payload should probably also be of type IReferenceNode to enable that external references are resolved?

I appreciate feedback and hope I got everything right. Thanks! :)

[EricWittmann]

This is a good observation. Because the payload can be anything, it is not currently modeled (even when it's actually a Schema object). I think it would be great to update our support for AsyncAPI such that our reader/writer will parse the payload as a Schema object if that's what it is. But if it's not that then we can leave it as an Object. And then our traverser/visitor implementation should traverse and visit the payload Schema when appropriate.

@SafarMirek any interest in this?

[DyspC]

I was looking into this and there's also a missing part in the Aai20IReferenceManipulationStrategy for message payloads because it only attaches the fetched nodes to #/components and some referenceable nodes cannot be put there (especially in the above example because the spec states message payloads can be of any type but schemas under #/components have to be JSON schemas (no avro))

---

I also listed the nodes from the AsyncAPI spec (as per 2.1.0) that are either something or a reference

• Parameter Object | Reference Object
• Message Object | Reference Object
• Schema Object | Reference Object
• Security Scheme Object | Reference Object
• Correlation ID Object | Reference Object
• • (Traits)
• Operation Trait Object | Reference Object
• Message Trait Object | Reference Object
• • (Bindings)
• Server Bindings Object | Reference Object
• Channel Bindings Object | Reference Object
• Operation Bindings Object | Reference Object
• Message Bindings Object | Reference Object

+ The Channel Object has a $ref field but it does not look like it should be dereferenced (but rather be an additional definition that should cause no conflict)

The Schema type ins't dereferenced yet in 1.1.16 (working on this) but there is also the AaiOperationBase class that implements IReferenceNode and I don't get why, perhaps this is because the specification snippets miss a message line (to be fixed soon in asyncapi/spec#620)

[EricWittmann]

What do you think @jsenko ?

[EricWittmann]

I know this is old, but I'm wondering @DyspC if you might be willing to revisit this for Apicurio Data Models 2.0. It's all new with a new codegen approach to the models. We might consider defining a union type for the payload property that could better handle this use-case.

[DyspC]

Erm it looks really old to me and I don't understand what I've written anymore nor do I remember if I created a PR with what I was working on so digging it out would prove complicated

Looking at the branches on my fork it could be this one but I'm no longer working on API design tooling (and actually pulling my hair out over some thrift tooling that gulps all my time)

[EricWittmann]

OK no problem!

[legas117]

Hello,

I've recently encounter this problem as well, the payload of the message doesn't accept anything, can't reference any data model.
Did you have an idea how to bypass the problem ?