Dataflow Templates 2022-05-09-00_RC01

Release Week of 2022-05-09

Bug Fixes

[All Flex Templates] Resolve dependency conflicts.
[SpannerChangeStreams To BigQuery] Bug fixes.

Contributors

@pranavbhandari24
@an2x

Dataflow Templates 2022-05-04-00_RC00

Release Week of 2022-05-02

Improvements

[All Templates] Upgraded from Beam 2.37 to 2.38.
[Cloud Spanner Change Streams Template] Added ignoreFields param and more unit tests.
[JDBC to Pub/Sub Template] Javadoc updates.
Upgraded io.opencensus.opencensus-api to the latest version 0.31.0.

Bug Fixes

[All Spanner Templates] Minor bugfixes to support Beam 2.38.

Contributors

@an2x
@haikuo-google
@nancyxu123
@pabloem
@pranavbhandari24

Dataflow Templates 2022-04-25-00_RC00

Release Week of 2022-04-25

New Templates

[SpannerChangeStreamsToGcs] Implement Spanner Change Streams to BigQuery changelog table replication Dataflow template

Improvements

[PubSubToSplunk] Add parameter to allow users to specify if HTTP messages sent to HEC endpoint should be GZIP encoded.

Bug Fixes

[DatastreamToSQL] Use KV into the write

Contributors

@dhercher
@pranavbhandari24

Dataflow Templates 2022-04-18-00_RC00

Release Week of 2022-04-18

Improvements

• [JDBC to BigQuery] Add documentation for disabledAlgorithms option

Bug Fixes

• [Datastream to SQL] Prevent Oracle records from being marked stale when multiple records with the same timestamp and SCN are loaded

Contributors

@dhercher
@pranavbhandari24

Dataflow Templates 2022-04-11-00_RC00

Release Week of 2022-04-11

New Templates

• Spanner Change Streams to GCS (only available in repository)

Improvements

• [Datastream to BigQuery] Use stateful transform to handle Oracle rollbacks
• [Datastream to BigQuery] Use txn id and row id for unique rollback values
• [Dataplex JDBC Ingestion] Added writeDisposition option to allow setting a write strategy if file already exists
• [Spanner Import/Export] Support building change streams DDL from information schema

Bug Fixes

• [Datastream to BigQuery] Change DATETIME handling to account for local timezones in Oracle in replication
• [Dataplex Tiering] Fix TIME field handling

Contributors

• dhercher
• nancyxu123
• olegsa
• pranavbhandari
• 1 internal contributor

Dataflow Templates 2022-04-04-00_RC00

Release Week of 2022-04-04

Improvements

[DataStreamToSQL] Improved logging
[DatastreamToBigQuery] Added datastream logging for json failures

Bug Fixes

[All JDBC Templates] Fix bug that handles JDBC CLOB data converting to String
[Jdbc To BigQuery] Fix bug to get specific Java object from JDBC data type Date and timestamp

Contributors

dhercher
weiwenxu
zhoufek

Dataflow Templates 2022-03-28-00_RC00

Release Week of 2022-03-28

Improvements

Upgraded to Beam 2.37.
[DataStreamToSQL] Added failing SQL statement to error message logged.
[DataStreamToBigQuery] Added an option to add pane info to a windowed filename.
Added first integration test StreamingDataGeneratorIT.

Bug Fixes

Fixed flaky testAssetWithEntityJsonToParquetFailOnExistingFilesE2E test.

Contributors

dhercher
olegsa
pranavbhandari
zhoufek

Dataflow Templates 2022-03-21-00_RC01

Release Week of 2022-03-21

New Templates

Deprecated Datastream to Postgres template (replaced with Datastream to SQL template)

Improvements

[Datastream to SQL] Added support for specifying log level and logging SQL in debug mode
[Spanner to Text] Cleaned up and removed LocalSpannerIO

Bug Fixes

[Pub/Sub to Splunk] Fixed write latency metrics

Contributors

dhercher
pranavbhandari

Dataflow Templates 2022-03-14-00_RC00

Release Week of March 14, 2022 🥧

New Templates

N/A

Improvements

[Pub/Sub to Splunk] Added secret manager support

Bug Fixes

[Datastream Oracle] Map DATE -> DATE instead of DATE -> TIMESTAMP

New Regions

N/A

Contributors

pranavbhandari

Dataflow Templates 2022-03-07-00_RC00

Release Week of March 7, 2022

New Templates

N/A

Improvements

[Pub/Sub to Splunk] Improved metrics

Bug Fixes

N/A

New Regions

N/A

Contributors

pranavbhandari