JSONSchema should works with generic (parameterized type) POJO/DTO. #18849

ducquoc · 2018-12-04T01:06:08Z

ducquoc
Dec 4, 2018

Expected behavior

JSONSchema should works with generic (parameterized type) POJO/DTO. No exception at runtime, or at least update documentation/errorMessage how to work-around/bypass the issue.

Actual behavior

Exception when running producer/consumer with JSONSchema.of(clazz) (or Schema.JSON(clazz) ) wiith generic/parameterized class:

Exception in thread "main" java.lang.NoSuchFieldError: KEY_VALUE
	at org.apache.pulsar.client.impl.schema.KeyValueSchema.<init>(KeyValueSchema.java:50)
	at org.apache.pulsar.client.api.Schema.<clinit>(Schema.java:128)

Steps to reproduce

Create Producer or Consumer with client API, using JSONSchema.of(clazz) (or Schema.JSON(clazz) ) wiith generic/parameterized class.
Compiled OK, but when running the client producer/consumer it throws exception.

Example (ducquoc@a0649dc8):
https://github.com/ducquoc/pulsar/blob/fix-client-schema_json-schema-generic-parameterized-type/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/pojo/BrokerMessage.java

** Work-around: use

private Object payload;

and respective getter/setter instead of

private T payload;

System configuration

Pulsar version: 2.2.0 (but seems to be also affected on latest master branch)

jerrypeng · 2018-12-04T22:03:09Z

jerrypeng
Dec 4, 2018
Collaborator

So we use Avro underneath to generate a schema for the class. Parameterized/generics in Java are currently not support for schema generation in Avro. Though there is discussion and a PR to add that:

https://issues.apache.org/jira/browse/AVRO-2248

apache/avro#356

Though not sure if that is ever going to get merged in.

0 replies

ducquoc · 2018-12-05T13:41:56Z

ducquoc
Dec 5, 2018
Author

Thank you for your quick response.

I hope the Avro fix will be merged soon (and not much performance overhead). Until then I will apply the work-around java.lang.Object - sufficient so far :)

0 replies

vamsi360 · 2019-07-17T11:24:04Z

vamsi360
Jul 17, 2019

Hi, I worked on AVRO-2248 and we stabilised it and use it in production in our company. We will resume the discussion and make the required changes to get it merged to master.

For now, if generic types are required, you can use the fork https://github.com/vamsi360/avro

0 replies

rohts-patil · 2019-11-29T14:26:10Z

rohts-patil
Nov 29, 2019

@vamsi360 ,

I cloned your fork and built it to version 1.8.3-ppe-9.10.
I still get the errror.

Caused by: org.apache.avro.AvroTypeException: Unknown type: T
	at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:312)
	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:720)
	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:870)
	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:686)
	at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:238)
	at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:235)
	at avro.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
	at avro.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
	at avro.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
	at avro.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
	... 6 more

0 replies

shan-96 · 2019-12-02T11:59:40Z

shan-96
Dec 2, 2019

@vamsi360 ,
I cloned your fork and built it to version 1.8.3-ppe-9.10.
I still get the errror.
Caused by: org.apache.avro.AvroTypeException: Unknown type: T
at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:312)
at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:720)
at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:870)
at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:686)
at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:238)
at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:235)
at avro.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at avro.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at avro.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at avro.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
... 6 more

I am facing the same problem. It seems we are creating the build in a wrong way

0 replies

vamsi360 · 2020-02-11T07:49:10Z

vamsi360
Feb 11, 2020

Hi @rohts-patil @shan-96
Could you tell me how to replicate the issue you are facing with a test project? I will debug this and replicate/fix

0 replies

shan-96 · 2020-02-13T14:24:06Z

shan-96
Feb 13, 2020

@vamsi360

We are trying to write to Avro Files using version 1.8.3-ppe-9.10

We are writing a java object from memory to a specified file using DataFileWriter.
While building the schema of the POJO at runtime, we call...

private static Schema getJsonSchema(Class c) {
    Schema s = ReflectData.AllowNull.get().getSchema(c);
    AvroSchema avroSchema = new AvroSchema(s);
    return avroSchema.getAvroSchema();
  }

However, we still get the same exception as mentioned above which we got through maven released version of Avro (1.8)

Our schema (POJO) looks like this:

public RawData extends Data<String>{
    private static final long serialVersionUID = 1L;
    private long field 1;
    private String field 2;
}

public abstract class Data<T> implements Serializable {
    private static final long serialVersionUID = 1L;
    private String dataType;
    private T id;
}

We want to define AvroSchema for RawData.class

cc: @rohts-patil

0 replies

frankjkelly · 2020-09-08T16:44:34Z

frankjkelly
Sep 8, 2020

We too have been hit by this issue trying to send generics

public class InteractionEvent<T>

0 replies

jiazhai · 2020-09-09T07:58:44Z

jiazhai
Sep 9, 2020
Collaborator

@gaoran10 @congbobo184 Which of you have time to take a look at this issue?

0 replies

sijie · 2020-09-22T03:56:46Z

sijie
Sep 22, 2020
Collaborator

@frankjkelly Which pulsar client are you using?

0 replies

frankjkelly · 2020-09-22T13:05:56Z

frankjkelly
Sep 22, 2020

@sijie I am using 2.6.1 Client with 2.6.1 Server

When I use the code as follows

    private Producer<SignalEvent> getSignalProducer(String tenant) throws PulsarClientException {
        if (signalProducer.get(tenant) == null) {
            signalProducer.put(tenant, client.newProducer(Schema.JSON(SignalEvent.class))
                    .topic(createTopicUri(tenant, Topic.SIGNAL))
                    .create());
        }
        return signalProducer.get(tenant);
    }

I get the error

Unknown type: T
org.apache.avro.AvroTypeException: Unknown type: T
	at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:410)
	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:663)
	at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:759)
	at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:86)
	at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:615)
	at org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:335)
	at org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:332)
	at java.base/java.lang.ClassValue.getFromHashMap(ClassValue.java:226)
	at java.base/java.lang.ClassValue.getFromBackup(ClassValue.java:208)
	at java.base/java.lang.ClassValue.get(ClassValue.java:114)
	at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:346)
	at org.apache.pulsar.client.impl.schema.StructSchema.extractAvroSchema(StructSchema.java:174)
	at org.apache.pulsar.client.impl.schema.StructSchema.createAvroSchema(StructSchema.java:161)
	at org.apache.pulsar.client.impl.schema.StructSchema.parseSchemaInfo(StructSchema.java:187)
	at org.apache.pulsar.client.impl.schema.JSONSchema.of(JSONSchema.java:96)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.pulsar.client.internal.DefaultImplementation.lambda$newJSONSchema$26(DefaultImplementation.java:242)
	at org.apache.pulsar.client.internal.ReflectionUtils.catchExceptions(ReflectionUtils.java:35)
	at org.apache.pulsar.client.internal.DefaultImplementation.newJSONSchema(DefaultImplementation.java:239)
	at org.apache.pulsar.client.api.Schema.JSON(Schema.java:260)
	at com.cogito.library.eventmanager.PulsarEventBroker.getSignalProducer(PulsarEventBroker.java:106)
	at com.cogito.library.eventmanager.PulsarEventBroker.sendEvent(PulsarEventBroker.java:184)

but this code works fine

    private Producer<String> getSignalProducer(String tenant) throws PulsarClientException {
        if (signalProducer.get(tenant) == null) {
            signalProducer.put(tenant, client.newProducer(Schema.STRING)
                    .topic(createTopicUri(tenant, Topic.SIGNAL))
                    .create());
        }
        return signalProducer.get(tenant);
    }

Thanks!

0 replies

sijie · 2020-09-22T14:52:18Z

sijie
Sep 22, 2020
Collaborator

@frankjkelly I see. So it is related to extracting the schema information using the AVRO library. We will take a look and circle back.

0 replies

congbobo184 · 2020-09-23T08:39:26Z

congbobo184
Sep 23, 2020
Collaborator

@frankjkelly I think, because you use generic type T, so use can't provide the specific type. we can't generate avro schema according to the json schema information.

0 replies

frankjkelly · 2020-09-23T13:00:24Z

frankjkelly
Sep 23, 2020

Yes - as per @jerrypeng comment above
"Parameterized/generics in Java are currently not support for schema generation in Avro. Though there is discussion and a PR to add that:

https://issues.apache.org/jira/browse/AVRO-2248

apache/avro#356

Though not sure if that is ever going to get merged in."

So it appears it's up to the folks on the Avro side?

0 replies

sijie · 2020-09-23T15:40:29Z

sijie
Sep 23, 2020
Collaborator

@frankjkelly Yeah, currently it is blocked by apache/avro#356. I would encourage people to leave a comment in the avro issue and pull request to see if AVRO community can accelerate the process on adopting that change.

0 replies

frankjkelly · 2020-09-23T15:41:52Z

frankjkelly
Sep 23, 2020

Thank you - will do!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JSONSchema should works with generic (parameterized type) POJO/DTO. #18849

{{title}}

Replies: 16 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

JSONSchema should works with generic (parameterized type) POJO/DTO. #18849

Expected behavior

Actual behavior

Steps to reproduce

System configuration

Replies: 16 comments

jerrypeng Dec 4, 2018 Collaborator

ducquoc Dec 5, 2018 Author

jiazhai Sep 9, 2020 Collaborator

sijie Sep 22, 2020 Collaborator

sijie Sep 22, 2020 Collaborator

congbobo184 Sep 23, 2020 Collaborator

sijie Sep 23, 2020 Collaborator

jerrypeng
Dec 4, 2018
Collaborator

ducquoc
Dec 5, 2018
Author

jiazhai
Sep 9, 2020
Collaborator

sijie
Sep 22, 2020
Collaborator

sijie
Sep 22, 2020
Collaborator

congbobo184
Sep 23, 2020
Collaborator

sijie
Sep 23, 2020
Collaborator