(Java/Scala) How do I specify deserialize behavior on schema change?

[yoohaemin]

Hi, we are looking into using fury for our new Scala project.

In our project, we need to ensure that the two versions of the application (versions right after and right before each other) can always talk to each other.

Obviously, when adding a new class, it can't be deserialized in an old version, no matter which serialization library we are using, so the deploys will need to be in two steps.

However, the modifications of existing classes are trickier. There can be a few cases:

• Field added
• Field made optional (Int -> Option[Int])
• Field removed
• Field type changed, with a mapping function
• Field renamed

These problems can be solved by specifying behaviors to fury on a field:

• specify a default value on deserialization
• specify a mapping function from a value of different type to the current type
• specify a renaming function

How do I express such changes to Fury, so it can handle these cases?

I will be using compatible mode.
Thanks for reading.

[chaokunyang]

Compatible mode supports adding/removing fields. Fields type change and fields rename are not supportted

[chaokunyang]

Fields values of changed types/name will be ignored when deserialization

[yoohaemin]

What is the behavior when a field is added but payload doesn't contain the field? Is it added as null?

I am guessing if a field exists in payload but not in schema, it will be ignored.

[chaokunyang]

It will be added as a null. We don't support scala default value yet. It's reported in #1683 too, but seems @LoranceChen has stopped working at it

[yoohaemin]

Interesting idea, but will this detection add overhead to Java users? Maybe performance wise, it's ok because a field missing won't be too frequent of a case.

I also think with the planned unroll annotation (Scala 3), users will have a workaround.

https://docs.scala-lang.org/sips/unroll-default-arguments.html

This sip is decided to be implemented.

[chaokunyang]

However, I think it's possible to provide a callback to let users handle such situations. It's possible to extend Fury to pass objects and incompatible fields to users to let them handle such cases

[yoohaemin]

I already made the codecbuilder to work, but the serializer is a bit tricky. Maybe I will already open a WIP PR so I can get input on what is a preferred approach.

[chaokunyang]

Yeah, a WIP PR would be awesome, I can provide more inputs based on your PR

[chaokunyang]

We have a big optimization for scala serializaiton in #1853, I believe it will improve scala collection serializaiton performance greatly

[yoohaemin]

@chaokunyang Here's the WIP PR. It lacks test or docs, and I haven't run benchmarks yet.
#1870
Sorry it took a long time, I only have weekend to work on this.

[chaokunyang]

This is great, I will take a look at it later