Smart handling of references in avro parser

[janvyhnanek]

Hi,
currently it is only possible to configure avro parser to handle complex types either as references registered as separate schemas or as types defined in the same schema. But usually you want to share only some complex types between schemas and you don't want to register all complex types as separate schemas which might be quite numberous. This is the reason why it is almost imposible to use references to other schemas practically in any case.

It would be useful to enable behaviour known in other descriptors like openapi or xsd - so that you can either import a complex type or define it locally. It might be a bit tricky to implement such avro parser since there is no explicit construction for importing another schema in avro. Avro parser would probably have to interpret type as a reference to another schema only if it is not defined nowhere in local avro schema.

This is followup for issue: #3640

[EricWittmann]

While I agree this functionality would be interesting, is it actually possible to do in the Apache Avro specification? I mean, forgetting about Registry - is it possible to do e.g. locally?

[janvyhnanek]

As far as I know you cannot distinguish between locally and externally defined type in avro specification. The parser would have to scan the avro schema and check whether the type reference has its definition locally. If not, it would search for an external definition of this type in schema registry.

Honestly, I am not sure if the current functionality of dereferencing schemas could be useful for anybody. We had to disable schema dereferencing globally in all applications - if not, every complex type inside avro schemas would have to be registered separately in the registry and it would mean maybe tens of thousands of new schemas registered which would unaccaptably increase the complexity.

But once again, having possibility of defining complex type in a separate schema would be quite appreciated in some cases but without need to defined all complex type separately.

thanks for consideration