Dataflow Templates 2024-05-14-00_RC00

Release week of 2024-05-13

Dataflow Templates 2024-05-13-00_RC00

Release Week of 2024-05-13

New Templates

• Kafka to GCS template (#1448)
• Kafka To BigQuery Template (#1464)
• Python UDFs to MongoDB flex templates (#1497)
• Add Python UDF support to GCS to Splunk template (#1494)

Improvements

• [sourcedb-to-spanner] Unified Schema Mappings For Bulk Reader. (#1460)
• [jdbc] JDBC Table Schema Discovery (#1481)
• Upgrade beam version to 2.56.0 (#1504)
• [jdbc] Implementing JDBCIO.RowMapper to map ResultSet to SourceRow. (#1508)
• [bigtable-to-vectorembedding] Add custom App Profile support for Bigtable to Vector Embedding export (#1505)
• [spanner-migration] Add support for the unified custom logical types: JSON, NUMBER and VARCHAR for Spanner Migration Templates (#1509)
• Xlang Dockerfile speed improvements (#1499)
• [spanner-migration] Support UNION field shema types and improve debugging (#1524)
• [UI] Add parentName and parentTriggerValues attributes for templates (#1456)

Bug Fixes

• Parse secret name and password only when using Auth (#1512)

Contributors

Anand Inguva

Archana Gupta

Danny McCormick

Deep1998

Dippatel98

Jeffrey Kinard

Manit Gupta

Ron Gal

Sharan Teja M

Shreya Khajanchi

Supriya Koppa

Vardhan Thigle

Vardhan Vinay Thigle

Yi Hu

anikin

archanagupta03

fozzie15

gauravjainn

hugo-syn

sharan-malyala

surabhiad

vitaly.terentyev

Dataflow Templates 2024-05-08-00_RC00

Release week of 2024-05-06

Dataflow Templates 2024-04-23-00_RC00

Improvements

[YamlTemplate] speed improvements via distroless Xlang image
[JDBC] discover schema from information system tables

Bug Fixes

[Spanner] preserve null and empty arrays correctly

Contributors

Akshara Uke
Cloud Teleport
Damon
Damon Douglas
Danny McCormick
Deep1998
Jack McCluskey
Yi Hu
aksharauke

Dataflow Templates 2024-04-16-00_RC01

Release Week of 2024-04-16

Note: This release is being rolled out. It may not be in your region yet.

Improvements

• Beam version updated to Beam 2.55.1
• Support FLOAT32 type in v1 templates of Spanner
• YAML template performance improvements

Contributors

Aravind Pedapudi
Damon
Damon Douglas
Danny McCormick
Deep1998
Dippatel98
Dmitrii Burlutckii
Jeffrey Kinard
Manit Gupta
Vardhan Thigle
Yi Hu
djagaluru
fozzie15
sharantej
vitaly.terentyev

Dataflow Templates 2024-04-02-00_RC00

Release week of 2024-04-01

Dataflow Templates 2024-03-27-00_RC00

Release Week of 2024-03-27

Note: This release is being rolled out. It may not be in your region yet.

New Templates

N/A

Improvements

• [Python templates] Python UDF Support
• support vector_length column annotation in DataFlow Import/Export
• [Spanner-changestreams-to-sharded-file-sink] load test fixes
• [Flex templates] Extend Python UDF support to GCS flex templates and clean up code reuse
• Rollback YamlTemplate Beam version to 2.54
• [DataGenerator templates] Return a bit more accurate count for DataGenerator Template
• Apache BEAM SDK updates to handle vector length column annotation

Bug Fixes

• [googlecloud-to-neo4j] Escape fields in generated sql
• Fix a bug in Stream_GCS_Text_to_BigQuery classic template
• [Datastream-to-Spanner] Fix conversion to Long
• [Python UDFs] Change logic to not fail if no UDF is provided.

Contributors

@aksharauke
@ambujojha
@Deep1998
@Dippatel98
@fozzie15
@manitgupta
@Polber

Dataflow Templates 2024-03-22-02_RC00

Release week of 2024-03-18

Dataflow Templates 2024-03-12-00_RC00

Release week of 2024-03-11

Dataflow Templates 2024-03-06-00_RC00

Release Week of 2024-03-06

Note: This release is being rolled out. It may not be in your region yet.

New Templates

• SourceDB to Spanner

Improvements

• Upgraded to use Beam 2.54.0.
• Terraform modules specific to each template
• Added At-least Once mode support to Streaming Templates

Bug Fixes

N/A

Contributors

@Abacn
@aksharauke
@bharadwaj-aditya
@damondouglas
@damccorm
@Deep1998
@Dippatel98
@Polber
@shunping