Dataflow Templates 2024-07-16-00_RC00

Release Week of 2024-07-16

Note: This release is being rolled out. It may not be in your region yet.

New Templates

• [Terraform] [Cloud Datastream to Spanner] Added terraform template for MySQL sharded source to Spanner

Improvements

• [Spanner Change Streams to BigQuery] Add support for FLOAT32 columns
• [Cloud Datastream to Spanner] Add support for FLOAT32 columns
• [Terraform] [Cloud Datastream to Spanner] Add support for shared VPC

Bug Fixes

• [Cloud Spanner to BigQuery] Fix a NPE when one of the source fields contains a NULL value.

Contributors

@arawind
@damondouglas
@Polber
@manitgupta
@shreyakhajanchi
@an2x

Dataflow Templates 2024-07-09-00_RC00

Release Week of 2024-07-09

Note: This release is being rolled out. It may not be in your region yet.

New Templates

N/A

Improvements

• added fk and referenced tables
• Mongo to BQ - advanced filters
• Remove CustomMySQLResourceManager
• Terraform: Improvements to existing Terraform templates for live migrations
• Support FKs in bulk

Bug Fixes

• Minor fixes for custom transformation
• Fixing timeout handling for uniform partitioning and (temporarily) disabling splitting overhad

Contributors

• aksharauke
• darshan-sj
• manitgupta
• Polber
• shreyakhajanchi
• stankiewicz
• VardhanThigle

Full Changelog: 2024-07-01-00_RC00...2024-07-09-00_RC00

Dataflow Templates 2024-07-01-00_RC00

Release Week of 2024-07-01

Note: This release is being rolled out. It may not be in your region yet.

Improvements

• Updated all templates to Beam 2.57.0
• Added DLQDirectory parameter to Sourcedb To Spanner template
• PubSub to Datadog template now includes full PubSub message by default
• Add support for using pre-existing private connectivity configuration for Datastream to Spanner terraform module
• Various IT's added

Contributors

@AdibH
@aksharauke
@AnandInguva
@dtru-ddog
@Dippatel98
@Polber
@justinuang
@manitgupta
@shreyakhajanchi
@Abacn
@darshan-sj

Dataflow Templates 2024-06-18-00_RC01

Release week of 2024-06-17

Dataflow Templates 2024-06-18-00_RC00

Release week of 2024-06-17

Dataflow Templates 2024-06-11-00_RC00

Release Week of 2024-06-11

New Templates

N/A

Improvements

• Add terraform module to Any SourceDB to Cloud Spanner template
• Update bigtable-beam-import to 2.14.2 to support Data Boost
• Update Template groupName details
• Update the skipOptions in Template annotation for kafka template
• Allow flags to be used as jinja template variables

Bug Fixes

• Add TLS authentication support for Kafka Templates

Contributors

• bharadwaj-aditya
• justinuang
• shreyakhajanchi
• akashorabek
• AnandInguva
• Dippatel98
• robertwb

Dataflow Templates 2024-06-04-00_RC01

Release week of 2024-06-03

Dataflow Templates 2024-06-04-00_RC00

Release week of 2024-06-03

Dataflow Templates 2024-05-28-00_RC00

Release Week of 2024-05-28

New Templates

N/A

Improvements

[datastream-to-spanner] Tests for additional CUJs (#1556)
[sourcedb-to-spanner] Add new feature (#1577, #1600, #1604)
[googlecloud-to-neo4j] Use built-in batch grouping transform (#1418)
[All Kafka] Add commit offsets (#1597)
[gcs-to-sourcedb] Add Integration test (#1562)
[kafka-to-kafka] Add authentication with SSL (#1555)
[spanner-change-streams-to-sharded-file-sink] Add Reverse replication reader IT (#1575)

Bug Fixes

[sourcedb-to-spanner] Fixed value mapping for blob and year (1578)

Contributors

Aditya Bharadwaj
Akarys Shorabek
Akshara Uke
Anand Inguva
Cloud Teleport
Deep1998
Dippatel98
djagaluru
Florent Biville
Jeffrey Kinard
jian jin
Manit Gupta
sharan-malyala
Sharan Teja M
Shreya Khajanchi
surabhiad
Svetak Sundhar
Tamanna Kakkar
Tanu Sharma
Vardhan Thigle
Vitaly Terentyev

Dataflow Templates 2024-05-21-00_RC00

New Templates

N/A

Improvements

• [BULK] Misc changes across types, logging and options
• [DataStreamToSpanner] - Convert sharded migration ITs to use Avro
• [Avro] - Add support for interval type
• [Plugins] - Add GcsReadBucket type
• [KafkaToBigQuery] - Add support for writing Kafka keys
• [Plugins] - Add support for KAFKA_TOPIC template parameters.
• Add Spanner PR workflow and exclude Spanner from global workflow
• [GcsToSourceDB] - Spanner Reverse replication writer IT
• [Plugins] - Template parameters: update old types to new ones
• [Documentation ] - Add more docs on adding ITs, flex templates, and deprecating parameters
• optimized the processing for lean Spanner write intervals

Bug Fixes

• [BULK MISC FIXES] - Add error counters; Fix null string handlings

Contributors

@ahmedabu98
@aksharauke
@akvelon
@an2x
@damccorm
@darshan-sj
@Deep1998
@Dippatel98
@manitgupta
@RajKumarGupta
@shreyakhajanchi