chore: sync main-cbsi with upstream/master

Squashed commit of the following: commit f88f4b5 Author: Shane Glass <[email protected]> Date: Mon Oct 23 15:10:22 2023 -0400 feat: data_warehouse Add GenAI capabilities (terraform-google-modules#272) commit b7efc4d Author: Awais Malik <[email protected]> Date: Thu Oct 19 15:38:43 2023 -0700 fix: adds a null check for expiration time (terraform-google-modules#268) commit 405972a Author: Shane Glass <[email protected]> Date: Tue Oct 17 17:03:10 2023 -0400 fix: update workflow.tftpl (terraform-google-modules#266) commit 1046e0d Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Date: Tue Oct 17 09:29:48 2023 -0700 chore(deps): update cft/developer-tools docker tag to v1.17 (terraform-google-modules#265) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> commit ceef798 Author: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Date: Tue Oct 17 09:48:20 2023 -0600 chore(master): release 7.0.0 (terraform-google-modules#249) commit 1a7620b Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Date: Tue Oct 10 09:11:32 2023 -0700 chore(deps): Update module github.com/GoogleCloudPlatform/cloud-foundation-toolkit/infra/blueprint-test to v0.9.0 (terraform-google-modules#260) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> commit 7fd5bcb Author: Awais Malik <[email protected]> Date: Mon Oct 9 14:33:52 2023 -0700 fix: upgraded versions.tf to include minor bumps from tpg v5 (terraform-google-modules#261) Co-authored-by: Andrew Peabody <[email protected]> Co-authored-by: Jason Davenport <[email protected]> commit e97adfb Author: Shane Glass <[email protected]> Date: Mon Oct 9 15:27:46 2023 -0400 feat!: data_warehosue migrating to TheLook Ecommerce dataset (terraform-google-modules#257) commit 096ca4e Author: Lloyd Armstrong <[email protected]> Date: Mon Oct 9 18:28:35 2023 +0200 fix: upgrade hashicorp/google to 5.0 (terraform-google-modules#259) Co-authored-by: Lloyd Armstrong <[email protected]>
cbsi-dto · Oct 30, 2023 · b72b6e1 · b72b6e1
1 parent 702c453
commit b72b6e1
Show file tree

Hide file tree

Showing 53 changed files with 4,084 additions and 617 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,29 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [7.0.0](https://github.com/terraform-google-modules/terraform-google-bigquery/compare/v6.1.1...v7.0.0) (2023-10-10)
+
+
+### ⚠ BREAKING CHANGES
+
+* data_warehosue migrating to TheLook Ecommerce dataset ([#257](https://github.com/terraform-google-modules/terraform-google-bigquery/issues/257))
+
+### Features
+
+* data_warehosue migrating to TheLook Ecommerce dataset ([#257](https://github.com/terraform-google-modules/terraform-google-bigquery/issues/257)) ([e97adfb](https://github.com/terraform-google-modules/terraform-google-bigquery/commit/e97adfb1984592a6f9e4eea8f8bdc3d2969e3d2d))
+* data_warehouse add labels to objects ([#253](https://github.com/terraform-google-modules/terraform-google-bigquery/issues/253)) ([70962af](https://github.com/terraform-google-modules/terraform-google-bigquery/commit/70962afc68366e8ef1214b9ad747d3d15fa7f313))
+
+
+### Bug Fixes
+
+* data_warehouse add bigquery data policy api ([#254](https://github.com/terraform-google-modules/terraform-google-bigquery/issues/254)) ([f2aa4f3](https://github.com/terraform-google-modules/terraform-google-bigquery/commit/f2aa4f392e94603ca831c40c6c08dc3fbdae8af4))
+* data_warehouse api identity dependency for p/s ([#252](https://github.com/terraform-google-modules/terraform-google-bigquery/issues/252)) ([d2c1256](https://github.com/terraform-google-modules/terraform-google-bigquery/commit/d2c125676f176c8fa33eb9dad12b7ed992dee6ac))
+* **deps:** update terraform terraform-google-modules/project-factory/google to 14.3 ([#245](https://github.com/terraform-google-modules/terraform-google-bigquery/issues/245)) ([cf6869c](https://github.com/terraform-google-modules/terraform-google-bigquery/commit/cf6869c8f37999f4b765b7b9ac501c73a5ca3e36))
+* update architecture diagrams, fix integration test ([#241](https://github.com/terraform-google-modules/terraform-google-bigquery/issues/241)) ([bc5abf8](https://github.com/terraform-google-modules/terraform-google-bigquery/commit/bc5abf8cb55bfc2c9939a1be6d6adfa7212e3a17))
+* update workflow.tftpl bc subworkflow creates bq not biglake tables ([#246](https://github.com/terraform-google-modules/terraform-google-bigquery/issues/246)) ([fc7d4ae](https://github.com/terraform-google-modules/terraform-google-bigquery/commit/fc7d4ae15376a22a5f37c140fc4b1c1c4ad3aa52))
+* upgrade hashicorp/google to 5.0 ([#259](https://github.com/terraform-google-modules/terraform-google-bigquery/issues/259)) ([096ca4e](https://github.com/terraform-google-modules/terraform-google-bigquery/commit/096ca4ee651152d040f8c555832488991773ddb8))
+* upgraded versions.tf to include minor bumps from tpg v5 ([#261](https://github.com/terraform-google-modules/terraform-google-bigquery/issues/261)) ([7fd5bcb](https://github.com/terraform-google-modules/terraform-google-bigquery/commit/7fd5bcbb4eca4d71a0d1c9b7fb7f4d19ae21ddbf))
+
 ## [6.1.1](https://github.com/terraform-google-modules/terraform-google-bigquery/compare/v6.1.0...v6.1.1) (2023-07-20)
 
 

diff --git a/Makefile b/Makefile
@@ -18,7 +18,7 @@
 # Make will use bash instead of sh
 SHELL := /usr/bin/env bash
 
-DOCKER_TAG_VERSION_DEVELOPER_TOOLS := 1.16
+DOCKER_TAG_VERSION_DEVELOPER_TOOLS := 1.17
 DOCKER_IMAGE_DEVELOPER_TOOLS := cft/developer-tools
 REGISTRY_URL := gcr.io/cloud-foundation-cicd
 ENABLE_BPMETADATA := 1

diff --git a/README.md b/README.md
@@ -27,7 +27,7 @@ Basic usage of this module is as follows:
 ```hcl
 module "bigquery" {
   source  = "terraform-google-modules/bigquery/google"
-  version = "~> 6.1"
+  version = "~> 7.0"
 
   dataset_id                  = "foo"
   dataset_name                = "foo"

diff --git a/build/int.cloudbuild.yaml b/build/int.cloudbuild.yaml
@@ -53,4 +53,4 @@ tags:
 - 'integration'
 substitutions:
   _DOCKER_IMAGE_DEVELOPER_TOOLS: 'cft/developer-tools'
-  _DOCKER_TAG_VERSION_DEVELOPER_TOOLS: '1.16'
+  _DOCKER_TAG_VERSION_DEVELOPER_TOOLS: '1.17'
diff --git a/build/lint.cloudbuild.yaml b/build/lint.cloudbuild.yaml
@@ -21,4 +21,4 @@ tags:
 - 'lint'
 substitutions:
   _DOCKER_IMAGE_DEVELOPER_TOOLS: 'cft/developer-tools'
-  _DOCKER_TAG_VERSION_DEVELOPER_TOOLS: '1.16'
+  _DOCKER_TAG_VERSION_DEVELOPER_TOOLS: '1.17'
diff --git a/examples/basic_bq/versions.tf b/examples/basic_bq/versions.tf
@@ -17,8 +17,7 @@
 terraform {
   required_providers {
     google = {
-      source  = "hashicorp/google"
-      version = "~> 4.0"
+      source = "hashicorp/google"
     }
   }
   required_version = ">= 0.13"

diff --git a/examples/basic_view/versions.tf b/examples/basic_view/versions.tf
@@ -17,8 +17,7 @@
 terraform {
   required_providers {
     google = {
-      source  = "hashicorp/google"
-      version = "~> 4.0"
+      source = "hashicorp/google"
     }
   }
   required_version = ">= 0.13"

diff --git a/examples/data_warehouse/versions.tf b/examples/data_warehouse/versions.tf
@@ -17,8 +17,7 @@
 terraform {
   required_providers {
     google = {
-      source  = "hashicorp/google"
-      version = "~> 4.52"
+      source = "hashicorp/google"
     }
   }
   required_version = ">= 0.13"

diff --git a/examples/multiple_tables/versions.tf b/examples/multiple_tables/versions.tf
@@ -17,8 +17,7 @@
 terraform {
   required_providers {
     google = {
-      source  = "hashicorp/google"
-      version = "~> 4.0"
+      source = "hashicorp/google"
     }
   }
   required_version = ">= 0.13"

diff --git a/examples/scheduled_queries/versions.tf b/examples/scheduled_queries/versions.tf
@@ -18,8 +18,7 @@ terraform {
   required_providers {
 
     google = {
-      source  = "hashicorp/google"
-      version = "~> 4.0"
+      source = "hashicorp/google"
     }
   }
 }
diff --git a/main.tf b/main.tf
@@ -155,15 +155,15 @@ resource "google_bigquery_table" "main" {
   labels              = each.value["labels"]
   schema              = each.value["schema"]
   clustering          = each.value["clustering"]
-  expiration_time     = each.value["expiration_time"]
+  expiration_time     = each.value["expiration_time"] != null ? each.value["expiration_time"] : 0
   project             = var.project_id
   deletion_protection = each.value["deletion_protection"]
 
   dynamic "time_partitioning" {
     for_each = each.value["time_partitioning"] != null ? [each.value["time_partitioning"]] : []
     content {
       type                     = time_partitioning.value["type"]
-      expiration_ms            = time_partitioning.value["expiration_ms"]
+      expiration_ms            = time_partitioning.value["expiration_ms"] != null ? time_partitioning.value["expiration_ms"] : 0
       field                    = time_partitioning.value["field"]
       require_partition_filter = time_partitioning.value["require_partition_filter"]
     }
@@ -219,15 +219,15 @@ resource "google_bigquery_table" "materialized_view" {
   description         = each.value["description"]
   labels              = each.value["labels"]
   clustering          = each.value["clustering"]
-  expiration_time     = each.value["expiration_time"]
+  expiration_time     = each.value["expiration_time"] != null ? each.value["expiration_time"] : 0
   project             = var.project_id
   deletion_protection = false
 
   dynamic "time_partitioning" {
     for_each = each.value["time_partitioning"] != null ? [each.value["time_partitioning"]] : []
     content {
       type                     = time_partitioning.value["type"]
-      expiration_ms            = time_partitioning.value["expiration_ms"]
+      expiration_ms            = time_partitioning.value["expiration_ms"] != null ? time_partitioning.value["expiration_ms"] : 0
       field                    = time_partitioning.value["field"]
       require_partition_filter = time_partitioning.value["require_partition_filter"]
     }
@@ -265,7 +265,7 @@ resource "google_bigquery_table" "external_table" {
   table_id            = each.key
   description         = each.value["description"]
   labels              = each.value["labels"]
-  expiration_time     = each.value["expiration_time"]
+  expiration_time     = each.value["expiration_time"] != null ? each.value["expiration_time"] : 0
   project             = var.project_id
   deletion_protection = false
 

diff --git a/metadata.yaml b/metadata.yaml
@@ -24,7 +24,7 @@ spec:
     source:
       repo: https://github.com/terraform-google-modules/terraform-google-bigquery
       sourceType: git
-    version: 6.1.0
+    version: 7.0.0
     actuationTool:
       flavor: Terraform
       version: '>= 0.13'

diff --git a/modules/authorization/README.md b/modules/authorization/README.md
@@ -17,7 +17,7 @@ Example:
 ```hcl
 module "dataset" {
   source = "terraform-google-modules/bigquery/google"
-  version = "~> 6.1"
+  version = "~> 7.0"
 
   dataset_id                  = "example_dataset"
   dataset_name                = "example_dataset"

diff --git a/modules/authorization/metadata.yaml b/modules/authorization/metadata.yaml
@@ -25,7 +25,7 @@ spec:
       repo: https://github.com/terraform-google-modules/terraform-google-bigquery
       sourceType: git
       dir: /modules/authorization
-    version: 6.1.0
+    version: 7.0.0
     actuationTool:
       flavor: Terraform
       version: '>= 0.13'

diff --git a/modules/authorization/versions.tf b/modules/authorization/versions.tf
@@ -20,12 +20,12 @@ terraform {
 
     google = {
       source  = "hashicorp/google"
-      version = ">= 4.44, < 5.0"
+      version = ">= 4.44, < 6"
     }
   }
 
   provider_meta "google" {
-    module_name = "blueprints/terraform/terraform-google-bigquery:authorization/v6.1.1"
+    module_name = "blueprints/terraform/terraform-google-bigquery:authorization/v7.0.0"
   }
 
 }
diff --git a/modules/data_warehouse/README.md b/modules/data_warehouse/README.md
@@ -12,9 +12,10 @@ The resources/services/activations/deletions that this module will create/trigge
 - Creates a BigQuery Dataset
 - Creates a BigQuery Table
 - Creates a Google Cloud Storage bucket
-- Loads the Google Cloud Storage bucket with data from https://console.cloud.google.com/marketplace/product/city-of-new-york/nyc-tlc-trips
+- Loads the Google Cloud Storage bucket with data from [TheLook eCommerce Public Dataset](https://console.cloud.google.com/marketplace/product/bigquery-public-data/thelook-ecommerce)
 - Provides SQL examples
 - Creates and inferences with a BigQuery ML model
+- Creates a remote model and uses Generative AI to generate text through a BigQuery ML remote model
 - Creates a Looker Studio report
 
 ### preDeploy
@@ -27,7 +28,7 @@ To deploy this blueprint you must have an active billing account and billing per
 ## Usage
 
 Functional examples are included in the
-[examples](./examples/) directory.
+[examples](../../examples/data_warehouse/README.md) directory.
 
 <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
 ## Inputs
@@ -40,14 +41,15 @@ Functional examples are included in the
 | labels | A map of labels to apply to contained resources. | `map(string)` | <pre>{<br>  "data-warehouse": true<br>}</pre> | no |
 | project\_id | Google Cloud Project ID | `string` | n/a | yes |
 | region | Google Cloud Region | `string` | n/a | yes |
+| text\_generation\_model\_name | Name of the BigQuery ML GenAI remote model that connects to the LLM used for text generation | `string` | `"text_generate_model"` | no |
 
 ## Outputs
 
 | Name | Description |
 |------|-------------|
 | bigquery\_editor\_url | The URL to launch the BigQuery editor with the sample query procedure opened |
 | ds\_friendly\_name | Dataset name |
-| lookerstudio\_report\_url | The URL to create a new Looker Studio report displays a sample dashboard for the taxi data analysis |
+| lookerstudio\_report\_url | The URL to create a new Looker Studio report displays a sample dashboard for the e-commerce data analysis |
 | neos\_tutorial\_url | The URL to launch the in-console tutorial for the EDW solution |
 | raw\_bucket | Raw bucket name |
 
@@ -61,8 +63,8 @@ These sections describe requirements for using this module.
 
 The following dependencies must be available:
 
-- [Terraform][terraform] v0.13
-- [Terraform Provider for GCP][terraform-provider-gcp] plugin v3.0
+- [Terraform](https://github.com/hashicorp/terraform) v0.13
+- [Terraform Provider for GCP](https://github.com/hashicorp/terraform-provider-google) plugin v3.0
 
 ### Service Account
 
@@ -76,22 +78,27 @@ the resources of this module:
 - Pub/Sub Admin: `roles/pubsub.admin`
 - Dataplex Admin: `roles/dataplex.admin`
 
-The [Project Factory module][project-factory-module] and the
-[IAM module][iam-module] may be used in combination to provision a
+The [Project Factory module](./.terraform/modules/project-services/README.md) and the
+[IAM module](https://github.com/terraform-google-modules/terraform-google-iam) may be used in combination to provision a
 service account with the necessary roles applied.
 
 ### APIs
 
 A project with the following APIs enabled must be used to host the
 resources of this module:
 
+- Vertex AI API: `aiplatform.googleapis.com`
 - BigQuery API: `bigquery.googleapis.com`
+- BigQuery Connection API: `bigqueryconnection.googleapis.com`
 - BigQuery Data Policy API: `bigquerydatapolicy.googleapis.com`
+- BigQuery Data Transfer Service API: `bigquerydatatransfer.googleapis.com`
 - BigQuery Migration API: `bigquerymigration.googleapis.com`
-- BigQuery Storage API: `bigquerystorage.googleapis.com`
-- BigQuery Connection API: `bigqueryconnection.googleapis.com`
 - BigQuery Reservations API: `bigqueryreservation.googleapis.com`
-- BigQuery Data Transfer Service API: `bigquerydatatransfer.googleapis.com`
+- BigQuery Storage API: `bigquerystorage.googleapis.com`
+- Google Cloud APIs: `cloudapis.googleapis.com`
+- Cloud Build API: `cloudbuild.googleapis.com`
+- Compute Engine API: `compute.googleapis.com`
+- Infrastructure Manager API: `config.googleapis.com`
 - Data Catalog API: `datacatalog.googleapis.com`
 - Data Lineage API: `datalineage.googleapis.com`
 - Eventarc API: `eventarc.googleapis.com`
@@ -101,10 +108,10 @@ resources of this module:
 - Google Cloud Storage JSON API: `storage-api.googleapis.com`
 - Google Cloud Workflows API: `workflows.googleapis.com`
 
-The [Project Factory module][project-factory-module] can be used to
+The [Project Factory module](./.terraform/modules/project-services/README.md) can be used to
 provision a project with the necessary APIs enabled.
 
 
 ## Security Disclosures
 
-Please see our [security disclosure process](./SECURITY.md).
+Please see our [security disclosure process](../../SECURITY.md).
diff --git a/modules/data_warehouse/assets/data-warehouse-architecture.json b/modules/data_warehouse/assets/data-warehouse-architecture.json
diff --git a/modules/data_warehouse/assets/data-warehouse-architecture.png b/modules/data_warehouse/assets/data-warehouse-architecture.png
diff --git a/modules/data_warehouse/assets/data-warehouse-architecture.svg b/modules/data_warehouse/assets/data-warehouse-architecture.svg