feat(kafka): support several schemas #32
Conversation
| @@ -43,14 +43,16 @@ find_schema_class_file() { | |||
| # Otherwise we fallback on finding the filename containing the schema code | |||
| # to end with Schema or is named InputModel | |||
| # We might want to improve this in the future | |||
| schema_class_file="$(grep -lr "^@schema" src | head -n 1 || return 0)" | |||
| @@ -43,14 +43,16 @@ find_schema_class_file() { | |||
| # Otherwise we fallback on finding the filename containing the schema code | |||
| # to end with Schema or is named InputModel | |||
| # We might want to improve this in the future | |||
| schema_class_file="$(grep -lr "^@schema" src | head -n 1 || return 0)" | |||
| schema_class_files="$(grep -lr "^@schema" src || return 0)" | |||
leo-dur
force-pushed
the
kafka-hooks-support-several-schema
branch
from
80adf35 to
a7f9367
Compare
There was a problem hiding this comment.
I did some minor change + I kept the previous behavior of using only one schemas.avsc file when there is only one schema.
We can re-discuss that change but we need to take into account the impact on the deployment: currently, at deployment time the only schema to upload to the schema registry is simply found by checking for schema-.avsc.
If we stop using that, either the schema must be always explicitly specified for each topic in the values file or the logic should be improved to check how many schema file are available and to pick the only existing one when there is only one available and only one topic configured.
We also need to update the 2 actions https://github.com/Kpler/github-actions/blob/main/actions/kafka/upload-kafka-schemas/ and https://github.com/Kpler/github-actions/blob/main/actions/kafka/check-kafka-schemas-compatibility/ to support the multiple schema case (shouldn't be very complicated for these two).
Changes
Why several schemas support is needed?
On kafka-stream repos, we have 0-n output topics and 0-n internal topics (to record state).
Each topics usually need 1 or 2 schemas (either only value, or both key and value)
Note: Internal topics are read and written by the same app.
Example
https://github.com/Kpler/static-serializer-stream/pull/105/files
On that repo we have 2 output topics:
We have one schema per topic for the value (so 2 schemas)
We also have 1 internal topic with a schema for the value and the key