Dataflow Templates 2024-02-20-00_RC00

Release Week of 2024-02-20

Note: This release is being rolled out. It may not be in your region yet.

Contributors

• darshan-sj

Dataflow Templates 2024-02-14-00_RC00

Release week of 2024-02-12

Dataflow Templates 2024-02-13-01_RC00

Release week of 2024-02-12

Dataflow Templates 2024-02-13-00_RC00

Release week of 2024-02-12

Dataflow Templates 2024-02-09-00_RC00

Release Week of 2/12/2024

Improvements

• Reduce startup time for YamlTemplate
• Expose fetchSize option to DataplexJdbcIngestion template
• Add RpcPriority flag to DatastreamToSpanner Template

Bug Fixes

• Fix default value for At Least Once in Template Metadata
• Fix BigQueryDynamicDestination to properly fetch TableRow field names
• Add RpcPriority flag to Spanner Transaction
• Enfore csvFormat and delimiter as required parameters for GCS_CSV_to_BigQuery template

Contributors

• Ahmed Abualsaud
• Dippatel98
• Jeffrey Kinard
• Manit Gupta
• Yi Hu
• aksharau
• keyliug

Dataflow Templates 2024-01-30-01_RC00

Release week of 2024-01-29

Dataflow Templates 2024-01-23-00_RC00

Release Week of 2024-01-23

New Templates

Cloud Bigtable to Vector Embeddings template added

Improvements

BigQuery At Least Once flag is now hidden in the UI

Contributors

Bruno Volpato
Ron Gal
Dip Patel
djagaluru

Dataflow Templates 2024-01-16-00_RC00

Release Week of 2024-01-16

Note: This release is being rolled out. It may not be in your region yet.

New Templates

N/A

Improvements

• Adding Maven artifacts handling/usage to README by @bvolpato in #1253
• Add support for streaming modes by @Dippatel98 in #1261
• Include file name when throwing by @ahmedabu98 in #1274

Bug Fixes

N/A

Contributors

• @ahmedabu98
• @bvolpato
• @Dippatel98

Full Changelog: 2024-01-09-00_RC01...2024-01-16-00_RC00

Dataflow Templates 2024-01-09-00_RC01

Release Week of 2024-01-09

Note: This release is being rolled out. It may not be in your region yet.

Improvements

• [FirestoreToBigQuery] Support CreateDisposition, withMethod, and schema
• [JdbcToBQ] Support create disposition and assign BQ schema
• [YamlTemplate] Change template parameters from --yaml to --yaml_pipeline and --yaml_pipeline_file
• Add Streaming flag to Template annotation
• Make schema unique to improve debugability of test racing condition
• Cleanup ModelColumn builder TODOs
• Match IT framework with Beam diffs (soon to be consolidated)
• Change JVM config to prefer IPv4 stack

Bug Fixes

• Fix cascading NPE when there was a failure launching the pipeline
• Fix pipeline name record in metrics
• [Security] Bump kafka-clients due to CVE-2021-38153
• [Security] Bump ch.qos.logback:logback-classic from 1.2.10 to 1.2.13 in /structured-logging

Contributors

@adrw-google @bvolpato @Dippatel98 @Polber @Abacn @aksharauke

Dataflow Templates 2024-01-02-00_RC00

Release Week of 2024-01-02

Note: This release is being rolled out. It may not be in your region yet.

New Templates

N/A

Improvements

[Bug template] Ask template version instead of Beam version
[Classic Templates] Build the graph with use_runner_v2 for classic templates if -DunifiedWorker=true is used
[Datastream-to-Spanner] Adding support for PG SPANNER.COMMIT_TIMESTAMP
[SpannerChangeStreams-to-ShardFileSink] Added secret manager support when reading shard file

Bug Fixes

[ElasticsearchIO] Disable certificate work when no certificate is passed

Contributors

@aksharau
@bvolpato