Releases: memiiso/debezium-server-iceberg
Release latest
Release latest
Release 0.7.0.Final
- Debezium: 2.7.2.Final
- Apache Iceberg: 1.6.1
- Quarkus: 3.13.3
Breaking Changes
Equality deletes:
- Now Equality delete files are containing key fields only and they are written with record projection. Now delete files only includes key files, previously delete files were including full delete record.
Notable changes
- Various code improvement added. Now code base is more closely aligned with upstream iceberg-kafka-connect
- Now size of the files generated is defined
write.target-file-size-bytes
config, previously Long.MAX_VALUE was used - Iceberg upgraded from 1.6.0 to 1.6.1
- Debezium upgraded from 2.7.0.Final to 2.7.2.Final
- Quarkus upgraded from 3.13.0 to 3.13.3
What's Changed
- Bump version.quarkus from 3.13.0 to 3.13.2 by @dependabot in #400
- Bump io.smallrye:jandex-maven-plugin from 3.2.1 to 3.2.2 by @dependabot in #399
- Bump version.debezium from 2.7.0.Final to 2.7.1.Final by @dependabot in #398
- Bump version.testcontainers from 1.20.0 to 1.20.1 by @dependabot in #396
- Add code improvements by @ismailsimsek in #409
- Improve code, Improve class and method namings by @ismailsimsek in #410
- Improve deduplication code by @ismailsimsek in #411
- Use
write.target-file-size-bytes
config, previouslyLong.MAX_VALUE
was used by @ismailsimsek in #412 - Fix, Create GenericAppenderFactory respecting to identifier fields. And create it with table properties by @ismailsimsek in #413
- Explicitly set operationId as UUID when creating OutputFileFactory by @ismailsimsek in #414
- Use Key fields and Record Projection for deletes by @ismailsimsek in #408
- Rename variable equalityFieldIds to identifierFieldIds by @ismailsimsek in #415
- Bump org.awaitility:awaitility from 4.2.1 to 4.2.2 by @dependabot in #406
- Bump org.postgresql:postgresql from 42.7.3 to 42.7.4 by @dependabot in #407
- Bump org.apache.maven.plugins:maven-surefire-plugin from 3.3.1 to 3.5.0 by @dependabot in #405
- In case of exception, call writer.abort() to cleanup by @ismailsimsek in #417
- Upgrade debezium and Quarkus to latest minor version by @ismailsimsek in #418
Full Changelog: 0.6.0.Final...0.7.0.Final
Release 0.6.0.Final
- Debezium: 2.7.0.Final
- Apache Iceberg: 1.6.0
- Quarkus: 3.13.0
Notable changes
- new config added to control casing of iceberg table names. To change tables names uppercase or lowercase folloing configuration properties could be used, see documentation.
debezium.sink.iceberg.destination-uppercase-table-names=true
ordebezium.sink.iceberg.destination-lowercase-table-names=true
- Quarkus upgraded from 3.12.x to 3.13.x, See Quarkus release notes here
What's Changed
- Add option to change case of destination/iceberg table names by @ismailsimsek in #389
- Bump version.quarkus from 3.12.3 to 3.13.0 by @dependabot in #392
- Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.2.5 to 3.3.1 by @dependabot in #395
- Bump org.apache.maven.plugins:maven-surefire-plugin from 3.3.0 to 3.3.1 by @dependabot in #394
- Bump io.smallrye:jandex-maven-plugin from 3.2.0 to 3.2.1 by @dependabot in #393
Full Changelog: 0.5.0.Final...0.6.0.Final
Release 0.5.0.Final
- Debezium: 2.7.0.Final
- Apache Iceberg: 1.6.0
- Quarkus: 3.12.3
What's Changed
- Bump io.minio:minio from 8.5.7 to 8.5.11 by @dependabot in #372
- Bump org.codehaus.groovy:groovy-bom from 3.0.21 to 3.0.22 by @dependabot in #374
- Bump com.squareup.okio:okio from 3.4.0 to 3.9.0 by @dependabot in #376
- Upgrade quarkus from 3.10.0 to 3.10.2 by @ismailsimsek in #382
- Bump version.testcontainers from 1.19.8 to 1.20.0 by @dependabot in #387
- Bump org.apache.maven.plugins:maven-enforcer-plugin from 3.4.1 to 3.5.0 by @dependabot in #385
Full Changelog: 0.5.0.Beta...0.5.0.Final
Release 0.5.0.Beta
- Debezium: 2.7.0.Final
- Apache Iceberg: 1.5.2
- Quarkus: 3.10.0
Breaking Changes
Debezium version upgraded from 2.5.x to 2.7.x:
- For Debezium 2.6 Breaking Changes please see release notes
- For Debezium 2.7 Breaking Changes please see release notes
IcebergSchemaHistory wrong table location corrected:
IcebergSchemaHistory class was using wrong table namespace {catalog-name}.{table-name}
now this is corrected to use {table-namespace}.{table-name}
If you were using different catalog-name
please follow following steps for migration
- stop debezium server
- copy the IcebergSchemaHistory table to new location:
{table-namespace}.{table-name}
- start debezium server
Configuration change:
Configuration option debezium.sink.iceberg.upsert-op-column
, defaultValue = __op
is removed.
Notable changes
- Quarkus upgraded from 3.6.x to 3.10.x, See Quarkus release notes here
- Java release target upgraded from 11 to 17
- Full support for debezium nested events added. Now its possible to run consumer without event flattening.
- Debezium event processing improved, unnecessary schema de-serialization is avoided.
What's Changed
- Upgrade java to 17 by @ismailsimsek in #318
- Upgrade debezium to 2.7.0.Alpha1 by @ismailsimsek in #317
- Bump version.quarkus from 3.6.9 to 3.10.0 by @dependabot in #313
- Simplify example config, removed hadoop catalog example by @ismailsimsek in #321
- Minor, Improve naming of state tables used in tests by @ismailsimsek in #322
- Add mongodb flattening settings to example config by @ismailsimsek in #324
- Fix reading config values in debezium storage classes by @ismailsimsek in #327
- Minor improvements to icebergevents consumer, IcebergEventsChangeCons… by @ismailsimsek in #328
- Update codeql-analysis.yml to use java 17 by @ismailsimsek in #329
- Upgrade spark to 4.0.x version and enable Mysql tests by @ismailsimsek in #331
- Remove DynamicBatchSizeWait class by @ismailsimsek in #332
- Enforce maven version is greater than 3.8.6 by @ismailsimsek in #334
- Don't partition tables on append mode. by @ismailsimsek in #340
- Bump io.smallrye:jandex-maven-plugin from 3.1.7 to 3.2.0 by @dependabot in #338
- Bump org.postgresql:postgresql from 42.7.2 to 42.7.3 by @dependabot in #337
- Bump version.testcontainers from 1.19.7 to 1.19.8 by @dependabot in #336
- Improve IcebergChangeEvent class, Optimize event and schema deserialization by @ismailsimsek in #343
- Fail deduplication if event key is null by @ismailsimsek in #344
- Improve naming, rename JsonSchema to ChangeEventSchema by @ismailsimsek in #345
- Improve code and schema conversion tests by @ismailsimsek in #346
- Add more tests on schema conversion by @ismailsimsek in #348
- Improve conversion of debezium schema to iceberg schema by @ismailsimsek in #347
- Revert "Bump io.smallrye:jandex-maven-plugin from 3.1.7 to 3.2.0 (#338)" by @ismailsimsek in #349
- Improve setting identifier fields by @ismailsimsek in #350
- Allow nested fields to be set as identifier by @ismailsimsek in #351
- Lower minimum maven version enforcement to 3.6.3 by @ismailsimsek in #353
- Optimize test logs, decrease verbose logs by @ismailsimsek in #354
- Check if configuration contains event flattening by @ismailsimsek in #352
- Test Field Addition is working, automatic schema expansion by @ismailsimsek in #355
- Test consumer without event flattening, Add flag to control identifier-field creation by @ismailsimsek in #356
- Document new features and improve documentation by @ismailsimsek in #357
- Remove stage repo used for spark 4.0.0-preview1 dependency by @ismailsimsek in #358
- Update README.md by @ismailsimsek in #359
- Update dependabot to weekly interval by @ismailsimsek in #360
- Bump docker/build-push-action from 5 to 6 by @dependabot in #361
- Improve handling delete events and logging message by @ismailsimsek in #366
- Improve documentation, schema change behaviour by @ismailsimsek in #367
- Bump com.squareup.okhttp3:okhttp from 4.11.0 to 4.12.0 by @dependabot in #364
- Bump org.apache.maven.plugins:maven-surefire-plugin from 3.2.5 to 3.3.0 by @dependabot in #363
- Bump version.debezium from 2.7.0.Alpha1 to 2.7.0.Final by @dependabot in #371
Full Changelog: 0.4.0.Final...0.5.0.Beta
Release 0.4.1.Final
This release includes fix to debezium storage classes {IcebergSchemaHistory, IcebergOffsetBackingStore}. Now both classes can be used by icebergevents
consumer too.
- Debezium: 2.7.0.Alpha1
- Apache Iceberg: 1.5.2
- Quarkus: 3.10.0
Breaking Changes
IcebergSchemaHistory class was using wrong table name {catalog-name}.{table-name}
now this is corrected to {table-namespace}.{table-name}
If you were using different catalog-name
previously please follow following steps for migration
- stop debezium server
- move the table to new location:
{table-namespace}.{table-name}
- start the debezium server
What's Changed
- Simplify example config, removed hadoop catalog example by @ismailsimsek in #321
- Fix reading config values in debezium storage classes by @ismailsimsek in #327
- Minor improvements to icebergevents consumer, IcebergEventsChangeCons… by @ismailsimsek in #328
Release 0.4.0.Final
Release 0.4.0.Final
What's Changed
- Fix multiple log4j logger bindings by @ismailsimsek in #289
- Fix __source_ts_ms values with 0 conversion and enable mongodb test by @ismailsimsek in #292
- Remove maven stage repository by @ismailsimsek in #293
- Test upsert with tables without PK by @ismailsimsek in #295
- Bump softprops/action-gh-release from 1 to 2 by @dependabot in #301
- Add example setup using docker-compose by @ismailsimsek in #302
- Add iceberg-hive-metastore jar to distribution by @ismailsimsek in #308
- Update docker-compose example by @ismailsimsek in #309
- Bump io.smallrye:jandex-maven-plugin from 3.1.6 to 3.1.7 by @dependabot in #300
- Bump version.testcontainers from 1.19.6 to 1.19.7 by @dependabot in #299
- Bump org.apache.maven.plugins:maven-surefire-plugin from 3.2.2 to 3.2.5 by @dependabot in #297
- Bump org.awaitility:awaitility from 4.2.0 to 4.2.1 by @dependabot in #296
- Bump dev-drprasad/delete-tag-and-release from 1.0 to 1.1 by @dependabot in #310
- Bump org.apache.maven.plugins:maven-assembly-plugin from 3.6.0 to 3.7.1 by @dependabot in #311
- Update debezium version to 2.5.4.Final by @ismailsimsek in #315
- Upgrade iceberg to 1.5.2 by @ismailsimsek in #316
Full Changelog: 0.3.0.Final...0.4.0.Final
Release 0.4.0.Beta
This release includes dependency upgrades and improvements with dependency versions.
Breaking Changes
Removed iceberg-spark-runtime-*
dependency from the release in #114
Debezium version upgraded from 2.2.1 to 2.5.2
For Debezium 2.2 Breaking Changes please see release notes
For Debezium 2.3 Breaking Changes please see release notes
For Debezium 2.4 Breaking Changes please see release notes
For Debezium 2.5 Breaking Changes please see release notes
Iceberg upgraded from 1.3.x to 1.5.0
For Iceberg changes please see https://iceberg.apache.org/releases/
Quarkus upgraded from 3.3.2 to 3.6.9
For Quarkus changes please see https://quarkus.io/blog/tag/release/
What's Changed
- Improve exception messages and fix committing to IcebergSchemaHistory by @ismailsimsek in #234
- Upgrade iceberg to 1.4.0 by @ismailsimsek in #237
- Correct schema history parameter in application.properties.example file by @ismailsimsek in #238
- Upgrade debezium version to 2.4.x by @ismailsimsek in #235
- Bump org.apache.maven.plugins:maven-assembly-plugin from 3.4.2 to 3.6.0 by @dependabot in #225
- Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.0.0-M7 to 3.1.2 by @dependabot in #245
- Bump docker/build-push-action from 4 to 5 by @dependabot in #242
- Bump docker/login-action from 2 to 3 by @dependabot in #241
- Bump actions/checkout from 2 to 4 by @dependabot in #240
- Bump com.squareup.okhttp3:okhttp from 4.10.0 to 4.11.0 by @dependabot in #243
- Bump version.hadoop from 3.3.2 to 3.3.6 by @dependabot in #244
- Upgrade iceberg to 1.4.1 by @ismailsimsek in #247
- Bump actions/setup-java from 3 to 4 by @dependabot in #253
- Bump version.testcontainers from 1.19.0 to 1.19.1 by @dependabot in #250
- Bump com.google.cloud:google-cloud-nio from 0.124.19 to 0.127.7 by @dependabot in #254
- Bump org.apache.maven.plugins:maven-surefire-plugin from 3.0.0-M7 to 3.2.2 by @dependabot in #255
- Improve documentation by @ismailsimsek in #258
- Upgrade iceberg dependency 1.4.1 to 1.4.3 by @ismailsimsek in #260
- Upgrade quarkus 3.3.2 to 3.5.3 by @ismailsimsek in #261
- Upgrade debezium to 2.5.0.Final by @ismailsimsek in #262
- documentation typo fix by @robinsinghstudios in #272
- Bump org.postgresql:postgresql from 42.6.0 to 42.7.2 in /debezium-server-iceberg-sink by @dependabot in #270
- Bump github/codeql-action from 2 to 3 by @dependabot in #268
- Bump com.fasterxml.jackson:jackson-bom from 2.13.5 to 2.16.1 by @dependabot in #267
- Bump version.awssdk from 2.19.4 to 2.22.9 by @dependabot in #265
- Bump io.minio:minio from 8.5.1 to 8.5.7 by @dependabot in #264
- Bump version.groovy from 3.0.17 to 3.0.20 by @dependabot in #263
- Upgrade dependencies, Upgrade iceberg to 1.5.0 and debezium to 2.5.2.Final by @ismailsimsek in #273
- Use jackson version same as iceberg 2.14.2 by @ismailsimsek in #274
- Add example config for Hive metastore catalog by @ismailsimsek in #275
- Align package dependencies to iceberg and revert iceberg back to 1.4.3 by @ismailsimsek in #276
- Revert iceberg version to 1.4.3, It is changed to 1.4.2 by mistake by @ismailsimsek in #279
- Bump version.groovy from 3.0.20 to 3.0.21 by @dependabot in #280
- Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.1.2 to 3.2.5 by @dependabot in #284
- Bump version.testcontainers from 1.19.3 to 1.19.6 by @dependabot in #281
- Upgrade quarkus from 3.5.3 to 3.6.9 by @ismailsimsek in #285
- Upgrade iceberg to 1.5.0 by @ismailsimsek in #278
- Use iceberg-spark only for tests by @racevedoo in #114
- Switch to iceberg cloud bundle dependecies by @ismailsimsek in #288
New Contributors
- @robinsinghstudios made their first contribution in #272
Full Changelog: 0.3.0.Beta...0.4.0.Beta
Release 0.3.0.Final
Release 0.3.0.Final
Breaking Changes
Iceberg upgraded from 1.3.1 to 1.4.3
For Iceberg changes please see https://iceberg.apache.org/releases/
Quarkus upgraded from 3.3.2 to 3.5.3
For Quarkus changes please see https://quarkus.io/blog/tag/release/
What's Changed
- Improve exception messages and fix committing to IcebergSchemaHistory by @ismailsimsek in #234
- Upgrade iceberg to 1.4.0 by @ismailsimsek in #237
- Correct schema history parameter in application.properties.example file by @ismailsimsek in #238
- Upgrade debezium version to 2.4.x by @ismailsimsek in #235
- Bump org.apache.maven.plugins:maven-assembly-plugin from 3.4.2 to 3.6.0 by @dependabot in #225
- Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.0.0-M7 to 3.1.2 by @dependabot in #245
- Bump docker/build-push-action from 4 to 5 by @dependabot in #242
- Bump docker/login-action from 2 to 3 by @dependabot in #241
- Bump actions/checkout from 2 to 4 by @dependabot in #240
- Bump com.squareup.okhttp3:okhttp from 4.10.0 to 4.11.0 by @dependabot in #243
- Bump version.hadoop from 3.3.2 to 3.3.6 by @dependabot in #244
- Upgrade iceberg to 1.4.1 by @ismailsimsek in #247
- Bump actions/setup-java from 3 to 4 by @dependabot in #253
- Bump version.testcontainers from 1.19.0 to 1.19.1 by @dependabot in #250
- Bump com.google.cloud:google-cloud-nio from 0.124.19 to 0.127.7 by @dependabot in #254
- Bump org.apache.maven.plugins:maven-surefire-plugin from 3.0.0-M7 to 3.2.2 by @dependabot in #255
- Improve documentation by @ismailsimsek in #258
- Upgrade iceberg dependency 1.4.1 to 1.4.3 by @ismailsimsek in #260
- Upgrade quarkus 3.3.2 to 3.5.3 by @ismailsimsek in #261
New Contributors
- @robinsinghstudios made their first contribution in #272
Full Changelog: 0.3.0.Beta...0.3.0.Final
Release 0.3.0.Beta
This release includes major features and changes
Notable changes
- Support for all nested data types added in #232 #233
- New iceberg offset and database history classes added in #174 #175
Notable PRs
- Convert __source_ts_ms and __ts_ms to TimestampType.withZone type in #108
- Add ofset storage class in #174
- Add iceberg schema history storage in #175
- Remove aws-java-sdk-bundle from release in #204
- Add hive-metastore and Iceberg cloud dependencies to release in #198
- Add support to map data type in #232 #233
Breaking Changes
aws-java-sdk-bundle
dependency removed from the release in #204__source_ts_ms
and__ts_ms
fields are converted to TimestampType.withZone type in #108
Debezium version upgraded from 1.9.5.Final to 2.2.1.Final
For Debezium 1.9 Breaking Changes please see release notes
For Debezium 2.0 Breaking Changes please see release notes
For Debezium 2.1 Breaking Changes please see release notes
For Debezium 2.2 Breaking Changes please see release notes
Iceberg upgraded from 0.14.0 to 1.3.1
For Iceberg changes please see https://iceberg.apache.org/releases/
Quarkus upgraded from 2.11.3.Final to 3.3.2
For Quarkus changes please see https://quarkus.io/blog/tag/release/
What's Changed
- Upgrade iceberg to 0.14.1 by @ismailsimsek in #110
- Convert
__source_ts_ms
and__ts_ms
to TimestampType.withZone type by @ismailsimsek in #108 - Use upstream DebeziumMetrics by @ismailsimsek in #111
- Upgrade quarkus to 2.13.2.Final by @racevedoo in #115
- Configure dependabot to update versions automatically by @racevedoo in #116
- Remove usages of ObjectNode.put(String, JsonNode) as it is deprecated by @racevedoo in #113
- Bump mysql-connector-java from 8.0.28 to 8.0.30 by @dependabot in #125
- Bump version.groovy from 3.0.12 to 3.0.13 by @dependabot in #124
- Bump docker/login-action from 1 to 2 by @dependabot in #123
- Bump minio from 8.4.3 to 8.4.5 by @dependabot in #122
- Bump actions/checkout from 2 to 3 by @dependabot in #121
- Bump dev-drprasad/delete-tag-and-release from 0.1.3 to 0.2.0 by @dependabot in #119
- Bump docker/build-push-action from 2 to 3 by @dependabot in #118
- Bump version.hadoop from 3.3.3 to 3.3.4 by @dependabot in #120
- Bump version.spark from 3.2.2 to 3.3.0 by @dependabot in #117
- Upgrade iceberg-spark to use spark 3.3 by @racevedoo in #127
- Bump maven-failsafe-plugin from 3.0.0-M6 to 3.0.0-M7 by @dependabot in #132
- Bump version.testcontainers from 1.17.3 to 1.17.5 by @dependabot in #131
- Bump version.awssdk from 2.17.256 to 2.17.293 by @dependabot in #130
- Bump jackson-bom from 2.13.3 to 2.13.4.20221013 by @dependabot in #128
- Upgrade iceberg to 1.0.0 by @ismailsimsek in #112
- Bump google-cloud-storage from 2.11.3 to 2.13.1 by @dependabot in #135
- Bump version.spark from 3.3.0 to 3.3.1 by @dependabot in #137
- Bump version.quarkus from 2.13.2.Final to 2.13.3.Final by @dependabot in #136
- Bump okhttp from 4.9.2 to 4.10.0 by @dependabot in #141
- Bump version.awssdk from 2.17.293 to 2.18.7 by @dependabot in #140
- Bump maven-assembly-plugin from 3.3.0 to 3.4.2 by @dependabot in #138
- Bump maven-surefire-plugin from 3.0.0-M6 to 3.0.0-M7 by @dependabot in #139
- Bump gcs-connector from hadoop3-2.2.7 to hadoop3-2.2.8 by @dependabot in #142
- Bump google-cloud-nio from 0.124.14 to 0.124.19 by @dependabot in #144
- Bump version.quarkus from 2.13.3.Final to 2.14.0.Final by @dependabot in #143
- Bump google-cloud-storage from 2.13.1 to 2.14.0 by @dependabot in #146
- Bump version.debezium from 1.9.6.Final to 1.9.7.Final by @dependabot in #145
- Avoid logging secrets in jvm arguments by @ismailsimsek in #148
- Bump gcsio from 2.2.7 to 2.2.9 by @dependabot in #152
- Bump postgresql from 42.4.2 to 42.4.3 in /debezium-server-iceberg-sink by @dependabot in #153
- Bump google-cloud-storage from 2.14.0 to 2.15.1 by @dependabot in #156
- Update iceberg to 1.1.0 by @ismailsimsek in #158
- Bump jandex-maven-plugin from 1.0.8 to 1.2.3 by @dependabot in #150
- Bump google-cloud-storage from 2.15.1 to 2.16.0 by @dependabot in #161
- Improve github release workflows by @ismailsimsek in #167
- Improve deduplicateBatch logic by @ismailsimsek in #168
- Bump postgresql from 42.4.3 to 42.5.2 by @dependabot in #171
- Bump docker/build-push-action from 3 to 4 by @dependabot in #169
- Bump minio from 8.4.5 to 8.5.1 by @dependabot in #170
- Add ofset storage class by @ismailsimsek in #174
- Improve test dependencies and code by @ismailsimsek in #177
- Upgrade quarkus by @ismailsimsek in #178
- Add iceberg schema history storage by @ismailsimsek in #175
- Bump version.testcontainers from 1.17.5 to 1.17.6 by @dependabot in #179
- Upgrade iceberg to 1.2.0 by @ismailsimsek in #182
- Document IcebergOffsetBackingStore, IcebergSchemaHistory storage classes by @ismailsimsek in #183
- Bump dev-drprasad/delete-tag-and-release from 0.2.0 to 0.2.1 by @dependabot in #184
- Bump version.awssdk from 2.18.7 to 2.19.4 by @dependabot in #162
- Upgrade debezium to 2.x by @ismailsimsek in #176
- Use Append commit when there is no delete files by @ismailsimsek in #188
- Minor cleanup and code improvements by @ismailsimsek in #190
- Fix IcebergSchemaHistory and IcebergOffsetBackingStore by @ismailsimsek in #194
- Bump google-cloud-storage from 2.16.0 to 2.22.1 by @dependabot in #193
- Upgrade debezium to 2.2.x and Quarkus to 3.x by @ismailsimsek in #189
- Use new storage calesses for tests by @ismailsimsek in #195
- Bump version.groovy from 3.0.13 to 3.0.17 by @dependabot in #185
- Improve Offset and Database History Storage classes by @ismailsimsek in #199
- Improve tests and Add JDBC catalog with S3FileIO example config by @ismailsimsek in #203
- Remove aws-java-sdk-bundle from release by @ismailsimsek in #204
- Add hive-metastore and Iceberg cloud dependencies to release by @ismailsimsek in #198
- Make
debezium.sink.iceberg.table-prefix
config optional by @ismailsimsek in #212 - Add deduplicateBatch test by @ismailsimsek in https://github.com/memiiso/debezium-server-...