Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update HuggingFace DLC for TGI URI to 2.3 (latest) #852

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Update HuggingFace DLC for TGI URI to 2.3 (latest)

4994c11
Select commit
Loading
Failed to load commit list.
Open

Update HuggingFace DLC for TGI URI to 2.3 (latest) #852

Update HuggingFace DLC for TGI URI to 2.3 (latest)
4994c11
Select commit
Loading
Failed to load commit list.
Google Cloud Build / pr-review-trigger-ap-clusters (gke-ai-eco-dev) succeeded Oct 28, 2024 in 58m 59s

Summary

Build Information

Trigger pr-review-trigger-ap-clusters
Build 0839c829-bc3e-4dad-919f-8dbfdd33023d
Start 2024-10-28T11:13:09-07:00
Duration 58m16.749s
Status SUCCESS

Steps

Step Status Duration
validate platform SUCCESS 51.888s
validate ray SUCCESS 16.367s
validate jupyterhub SUCCESS 16.371s
validate rag SUCCESS 17.65s
create gke cluster SUCCESS 11m29.518s
test ray cluster SUCCESS 5m25.88s
cleanup ray cluster SUCCESS 32.152s
test jupyterhub SUCCESS 4m7.813s
cleanup jupyterhub SUCCESS 39.223s
test rag SUCCESS 38m28.78s
cleanup rag SUCCESS 2m9.733s
cleanup gke cluster FAILURE 4m54.266s
check result SUCCESS 647ms

Details

starting build "0839c829-bc3e-4dad-919f-8dbfdd33023d"

FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: 	git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: 	git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://github.com/GoogleCloudPlatform/ai-on-gke
 * branch            4994c11918a8f3c3e80955f367596d841ac7e15f -> FETCH_HEAD
HEAD is now at 4994c11 Update HuggingFace DLC for TGI URI to 2.3 (latest)
SETUPBUILD
BUILD
Starting Step #0 - "validate platform"
Step #0 - "validate platform": Pulling image: gcr.io/gke-ai-eco-dev/terraform
Step #0 - "validate platform": Using default tag: latest
Step #0 - "validate platform": latest: Pulling from gke-ai-eco-dev/terraform
Step #0 - "validate platform": ddf874abf16c: Pulling fs layer
Step #0 - "validate platform": ecfb911cf53f: Pulling fs layer
Step #0 - "validate platform": ce689bb70bd5: Pulling fs layer
Step #0 - "validate platform": 7d2b54998739: Pulling fs layer
Step #0 - "validate platform": b1c39b09da0f: Pulling fs layer
Step #0 - "validate platform": cc032834b94b: Pulling fs layer
Step #0 - "validate platform": 0279610fca41: Pulling fs layer
Step #0 - "validate platform": 6f125b563dda: Pulling fs layer
Step #0 - "validate platform": 7c8b6e22fcff: Pulling fs layer
Step #0 - "validate platform": faaaa1840355: Pulling fs layer
Step #0 - "validate platform": 60a28c2199b1: Pulling fs layer
Step #0 - "validate platform": 4c7171ac56db: Pulling fs layer
Step #0 - "validate platform": 372d85ec2f1f: Pulling fs layer
Step #0 - "validate platform": 633b411e5bd5: Pulling fs layer
Step #0 - "validate platform": b1c39b09da0f: Waiting
Step #0 - "validate platform": 7c8b6e22fcff: Waiting
Step #0 - "validate platform": 0279610fca41: Waiting
Step #0 - "validate platform": faaaa1840355: Waiting
Step #0 - "validate platform": 6f125b563dda: Waiting
Step #0 - "validate platform": cc032834b94b: Waiting
Step #0 - "validate platform": 60a28c2199b1: Waiting
Step #0 - "validate platform": 4c7171ac56db: Waiting
Step #0 - "validate platform": 7d2b54998739: Waiting
Step #0 - "validate platform": 372d85ec2f1f: Waiting
Step #0 - "validate platform": 633b411e5bd5: Waiting
Step #0 - "validate platform": ecfb911cf53f: Verifying Checksum
Step #0 - "validate platform": ecfb911cf53f: Download complete
Step #0 - "validate platform": ce689bb70bd5: Verifying Checksum
Step #0 - "validate platform": ce689bb70bd5: Download complete
Step #0 - "validate platform": 7d2b54998739: Download complete
Step #0 - "validate platform": ddf874abf16c: Download complete
Step #0 - "validate platform": 0279610fca41: Verifying Checksum
Step #0 - "validate platform": 0279610fca41: Download complete
Step #0 - "validate platform": 6f125b563dda: Verifying Checksum
Step #0 - "validate platform": 6f125b563dda: Download complete
Step #0 - "validate platform": 7c8b6e22fcff: Verifying Checksum
Step #0 - "validate platform": 7c8b6e22fcff: Download complete
Step #0 - "validate platform": faaaa1840355: Verifying Checksum
Step #0 - "validate platform": faaaa1840355: Download complete
Step #0 - "validate platform": cc032834b94b: Download complete
Step #0 - "validate platform": 60a28c2199b1: Verifying Checksum
Step #0 - "validate platform": 60a28c2199b1: Download complete
Step #0 - "validate platform": 4c7171ac56db: Verifying Checksum
Step #0 - "validate platform": 4c7171ac56db: Download complete
Step #0 - "validate platform": 372d85ec2f1f: Verifying Checksum
Step #0 - "validate platform": 372d85ec2f1f: Download complete
Step #0 - "validate platform": ddf874abf16c: Pull complete
Step #0 - "validate platform": ecfb911cf53f: Pull complete
Step #0 - "validate platform": ce689bb70bd5: Pull complete
Step #0 - "validate platform": 7d2b54998739: Pull complete
Step #0 - "validate platform": 633b411e5bd5: Verifying Checksum
Step #0 - "validate platform": 633b411e5bd5: Download complete
Step #0 - "validate platform": b1c39b09da0f: Verifying Checksum
Step #0 - "validate platform": b1c39b09da0f: Download complete
Step #0 - "validate platform": b1c39b09da0f: Pull complete
Step #0 - "validate platform": cc032834b94b: Pull complete
Step #0 - "validate platform": 0279610fca41: Pull complete
Step #0 - "validate platform": 6f125b563dda: Pull complete
Step #0 - "validate platform": 7c8b6e22fcff: Pull complete
Step #0 - "validate platform": faaaa1840355: Pull complete
Step #0 - "validate platform": 60a28c2199b1: Pull complete
Step #0 - "validate platform": 4c7171ac56db: Pull complete
Step #0 - "validate platform": 372d85ec2f1f: Pull complete
Step #0 - "validate platform": 633b411e5bd5: Pull complete
Step #0 - "validate platform": Digest: sha256:202db292b551d530ff51210759dd25971cb64b5afe675a5616e0c244d48a452b
Step #0 - "validate platform": Status: Downloaded newer image for gcr.io/gke-ai-eco-dev/terraform:latest
Step #0 - "validate platform": gcr.io/gke-ai-eco-dev/terraform:latest
Step #0 - "validate platform": 
Step #0 - "validate platform": Initializing the backend...
Step #0 - "validate platform": Initializing modules...
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for cloud-nat...
Step #0 - "validate platform": - cloud-nat in .terraform/modules/cloud-nat
Step #0 - "validate platform": - custom-network in ../modules/gcp-network
Step #0 - "validate platform": - private-gke-autopilot-cluster in ../modules/gke-autopilot-private-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for private-gke-autopilot-cluster.gke...
Step #0 - "validate platform": - private-gke-autopilot-cluster.gke in .terraform/modules/private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #0 - "validate platform": - private-gke-standard-cluster in ../modules/gke-standard-private-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for private-gke-standard-cluster.gke...
Step #0 - "validate platform": - private-gke-standard-cluster.gke in .terraform/modules/private-gke-standard-cluster.gke/modules/private-cluster
Step #0 - "validate platform": - public-gke-autopilot-cluster in ../modules/gke-autopilot-public-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for public-gke-autopilot-cluster.gke...
Step #0 - "validate platform": - public-gke-autopilot-cluster.gke in .terraform/modules/public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #0 - "validate platform": - public-gke-standard-cluster in ../modules/gke-standard-public-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for public-gke-standard-cluster.gke...
Step #0 - "validate platform": - public-gke-standard-cluster.gke in .terraform/modules/public-gke-standard-cluster.gke
Step #0 - "validate platform": 
Step #0 - "validate platform": Initializing provider plugins...
Step #0 - "validate platform": - Finding hashicorp/google versions matching ">= 4.51.0, >= 5.40.0, != 5.44.0, < 6.0.0, < 7.0.0"...
Step #0 - "validate platform": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #0 - "validate platform": - Finding hashicorp/google-beta versions matching ">= 5.40.0, != 5.44.0, <= 6.1.0, < 7.0.0"...
Step #0 - "validate platform": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #0 - "validate platform": - Finding hashicorp/kubernetes versions matching "~> 2.10, 2.18.1"...
Step #0 - "validate platform": - Installing hashicorp/google-beta v6.1.0...
Step #0 - "validate platform": - Installed hashicorp/google-beta v6.1.0 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/helm v2.8.0...
Step #0 - "validate platform": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/kubernetes v2.18.1...
Step #0 - "validate platform": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/google v5.44.2...
Step #0 - "validate platform": - Installed hashicorp/google v5.44.2 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/random v3.6.3...
Step #0 - "validate platform": - Installed hashicorp/random v3.6.3 (signed by HashiCorp)
Step #0 - "validate platform": 
Step #0 - "validate platform": Terraform has created a lock file .terraform.lock.hcl to record the provider
Step #0 - "validate platform": selections it made above. Include this file in your version control repository
Step #0 - "validate platform": so that Terraform can guarantee to make the same selections by default when
Step #0 - "validate platform": you run "terraform init" in the future.
Step #0 - "validate platform": 
Step #0 - "validate platform": Terraform has been successfully initialized!
Step #0 - "validate platform": 
Step #0 - "validate platform": You may now begin working with Terraform. Try running "terraform plan" to see
Step #0 - "validate platform": any changes that are required for your infrastructure. All Terraform commands
Step #0 - "validate platform": should now work.
Step #0 - "validate platform": 
Step #0 - "validate platform": If you ever set or change modules or backend configuration for Terraform,
Step #0 - "validate platform": rerun this command to reinitialize your working directory. If you forget, other
Step #0 - "validate platform": commands will detect it and remind you to do so if necessary.
Step #0 - "validate platform": Success! The configuration is valid.
Step #0 - "validate platform": 
Finished Step #0 - "validate platform"
Starting Step #2 - "validate jupyterhub"
Starting Step #3 - "validate rag"
Starting Step #1 - "validate ray"
Step #3 - "validate rag": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #2 - "validate jupyterhub": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #1 - "validate ray": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Initializing the backend...
Step #2 - "validate jupyterhub": Initializing modules...
Step #2 - "validate jupyterhub": - gcs in ../../modules/gcs
Step #2 - "validate jupyterhub": - infra in ../../infrastructure
Step #3 - "validate rag": 
Step #3 - "validate rag": Initializing the backend...
Step #3 - "validate rag": Initializing modules...
Step #3 - "validate rag": - cloudsql in ../../modules/cloudsql
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #1 - "validate ray": 
Step #1 - "validate ray": Initializing the backend...
Step #1 - "validate ray": Initializing modules...
Step #1 - "validate ray": - gcs in ../../modules/gcs
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/sql-db/google 20.0.0 for cloudsql.cloudsql...
Step #1 - "validate ray": - infra in ../../infrastructure
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #2 - "validate jupyterhub": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #2 - "validate jupyterhub": - infra.custom-network in ../../modules/gcp-network
Step #2 - "validate jupyterhub": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.private-gke-autopilot-cluster.gke...
Step #1 - "validate ray": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #1 - "validate ray": - infra.custom-network in ../../modules/gcp-network
Step #1 - "validate ray": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.private-gke-autopilot-cluster.gke...
Step #3 - "validate rag": - cloudsql.cloudsql in .terraform/modules/cloudsql.cloudsql/modules/postgresql
Step #3 - "validate rag": - frontend in frontend
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for frontend.frontend-workload-identity...
Step #2 - "validate jupyterhub": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #2 - "validate jupyterhub": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.private-gke-standard-cluster.gke...
Step #2 - "validate jupyterhub": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #2 - "validate jupyterhub": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.public-gke-autopilot-cluster.gke...
Step #2 - "validate jupyterhub": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #2 - "validate jupyterhub": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.public-gke-standard-cluster.gke...
Step #1 - "validate ray": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #1 - "validate ray": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.private-gke-standard-cluster.gke...
Step #3 - "validate rag": - frontend.frontend-workload-identity in .terraform/modules/frontend.frontend-workload-identity/modules/workload-identity
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.5.0 for frontend.frontend-workload-identity.annotate-sa...
Step #2 - "validate jupyterhub": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #2 - "validate jupyterhub": - jupyterhub in ../../modules/jupyter
Step #2 - "validate jupyterhub": - jupyterhub.iap_auth in ../../modules/iap
Step #1 - "validate ray": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for jupyterhub.jupyterhub-workload-identity...
Step #1 - "validate ray": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.public-gke-autopilot-cluster.gke...
Step #1 - "validate ray": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #1 - "validate ray": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.public-gke-standard-cluster.gke...
Step #1 - "validate ray": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #1 - "validate ray": - kuberay-cluster in ../../modules/kuberay-cluster
Step #1 - "validate ray": - kuberay-cluster.iap_auth in ../../modules/iap
Step #1 - "validate ray": - kuberay-monitoring in ../../modules/kuberay-monitoring
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for kuberay-workload-identity...
Step #3 - "validate rag": - frontend.frontend-workload-identity.annotate-sa in .terraform/modules/frontend.frontend-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #3 - "validate rag": - frontend.frontend-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/frontend.frontend-workload-identity.annotate-sa
Step #3 - "validate rag": - frontend.iap_auth in ../../modules/iap
Step #3 - "validate rag": - gcs in ../../modules/gcs
Step #3 - "validate rag": - inference-server in ../../modules/inference-service
Step #3 - "validate rag": - infra in ../../infrastructure
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #3 - "validate rag": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #3 - "validate rag": - infra.custom-network in ../../modules/gcp-network
Step #3 - "validate rag": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.private-gke-autopilot-cluster.gke...
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity in .terraform/modules/jupyterhub.jupyterhub-workload-identity/modules/workload-identity
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.5.0 for jupyterhub.jupyterhub-workload-identity.annotate-sa...
Step #1 - "validate ray": - kuberay-workload-identity in .terraform/modules/kuberay-workload-identity/modules/workload-identity
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.5.0 for kuberay-workload-identity.annotate-sa...
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity.annotate-sa in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa
Step #2 - "validate jupyterhub": - namespace in ../../modules/kubernetes-namespace
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.5.0 for project-services...
Step #1 - "validate ray": - kuberay-workload-identity.annotate-sa in .terraform/modules/kuberay-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #1 - "validate ray": - kuberay-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/kuberay-workload-identity.annotate-sa
Step #1 - "validate ray": - namespace in ../../modules/kubernetes-namespace
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.5.0 for project-services...
Step #3 - "validate rag": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #3 - "validate rag": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.private-gke-standard-cluster.gke...
Step #3 - "validate rag": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #3 - "validate rag": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.public-gke-autopilot-cluster.gke...
Step #3 - "validate rag": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #3 - "validate rag": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.public-gke-standard-cluster.gke...
Step #2 - "validate jupyterhub": - project-services in .terraform/modules/project-services/modules/project_services
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Initializing provider plugins...
Step #2 - "validate jupyterhub": - terraform.io/builtin/terraform is built in to Terraform
Step #2 - "validate jupyterhub": - Finding hashicorp/null versions matching ">= 2.1.0"...
Step #3 - "validate rag": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #3 - "validate rag": - jupyterhub in ../../modules/jupyter
Step #3 - "validate rag": - jupyterhub.iap_auth in ../../modules/iap
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for jupyterhub.jupyterhub-workload-identity...
Step #2 - "validate jupyterhub": - Finding hashicorp/google versions matching ">= 3.39.0, >= 3.43.0, >= 3.53.0, >= 4.51.0, >= 5.40.0, != 5.44.0, < 6.0.0, < 7.0.0"...
Step #1 - "validate ray": - project-services in .terraform/modules/project-services/modules/project_services
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity in .terraform/modules/jupyterhub.jupyterhub-workload-identity/modules/workload-identity
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.5.0 for jupyterhub.jupyterhub-workload-identity.annotate-sa...
Step #1 - "validate ray": 
Step #1 - "validate ray": Initializing provider plugins...
Step #1 - "validate ray": - terraform.io/builtin/terraform is built in to Terraform
Step #1 - "validate ray": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity.annotate-sa in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa
Step #3 - "validate rag": - kuberay-cluster in ../../modules/kuberay-cluster
Step #3 - "validate rag": - kuberay-cluster.iap_auth in ../../modules/iap
Step #3 - "validate rag": - kuberay-monitoring in ../../modules/kuberay-monitoring
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for kuberay-workload-identity...
Step #2 - "validate jupyterhub": - Finding hashicorp/google-beta versions matching ">= 3.43.0, >= 5.40.0, != 5.44.0, < 6.0.0, <= 6.1.0, < 7.0.0"...
Step #1 - "validate ray": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #3 - "validate rag": - kuberay-workload-identity in .terraform/modules/kuberay-workload-identity/modules/workload-identity
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.5.0 for kuberay-workload-identity.annotate-sa...
Step #2 - "validate jupyterhub": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #3 - "validate rag": - kuberay-workload-identity.annotate-sa in .terraform/modules/kuberay-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #3 - "validate rag": - kuberay-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/kuberay-workload-identity.annotate-sa
Step #3 - "validate rag": - namespace in ../../modules/kubernetes-namespace
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.5.0 for project-services...
Step #1 - "validate ray": - Finding hashicorp/google-beta versions matching ">= 3.43.0, >= 5.40.0, != 5.44.0, < 6.0.0, <= 6.1.0, < 7.0.0"...
Step #2 - "validate jupyterhub": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #1 - "validate ray": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #2 - "validate jupyterhub": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #1 - "validate ray": - Finding hashicorp/google versions matching ">= 3.39.0, >= 3.43.0, >= 3.53.0, >= 4.51.0, >= 5.40.0, != 5.44.0, < 6.0.0, < 7.0.0"...
Step #2 - "validate jupyterhub": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #1 - "validate ray": - Finding hashicorp/null versions matching ">= 2.1.0"...
Step #1 - "validate ray": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #1 - "validate ray": - Finding hashicorp/time versions matching "0.11.1"...
Step #2 - "validate jupyterhub": - Installing hashicorp/helm v2.8.0...
Step #1 - "validate ray": - Installing hashicorp/external v2.3.4...
Step #1 - "validate ray": - Installed hashicorp/external v2.3.4 (signed by HashiCorp)
Step #3 - "validate rag": - project-services in .terraform/modules/project-services/modules/project_services
Step #3 - "validate rag": 
Step #3 - "validate rag": Initializing provider plugins...
Step #3 - "validate rag": - terraform.io/builtin/terraform is built in to Terraform
Step #3 - "validate rag": - Finding hashicorp/null versions matching ">= 2.1.0, ~> 3.1"...
Step #1 - "validate ray": - Installing hashicorp/google-beta v5.44.2...
Step #3 - "validate rag": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #3 - "validate rag": - Finding hashicorp/google-beta versions matching ">= 3.43.0, >= 5.12.0, >= 5.40.0, != 5.44.0, < 6.0.0, <= 6.1.0, < 7.0.0"...
Step #3 - "validate rag": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #2 - "validate jupyterhub": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #3 - "validate rag": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0, ~> 3.1"...
Step #3 - "validate rag": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #3 - "validate rag": - Finding hashicorp/time versions matching "0.11.1"...
Step #2 - "validate jupyterhub": - Installing hashicorp/kubernetes v2.18.1...
Step #3 - "validate rag": - Finding hashicorp/google versions matching ">= 3.39.0, >= 3.43.0, >= 3.53.0, >= 4.51.0, >= 5.12.0, >= 5.40.0, != 5.44.0, < 6.0.0, < 7.0.0"...
Step #3 - "validate rag": - Installing hashicorp/helm v2.8.0...
Step #2 - "validate jupyterhub": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/random v3.6.3...
Step #3 - "validate rag": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/google-beta v5.44.2...
Step #2 - "validate jupyterhub": - Installed hashicorp/random v3.6.3 (signed by HashiCorp)
Step #1 - "validate ray": - Installed hashicorp/google-beta v5.44.2 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/external v2.3.4...
Step #1 - "validate ray": - Installing hashicorp/kubernetes v2.18.1...
Step #2 - "validate jupyterhub": - Installed hashicorp/external v2.3.4 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/null v3.2.3...
Step #1 - "validate ray": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installed hashicorp/null v3.2.3 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/google v5.44.2...
Step #2 - "validate jupyterhub": - Installing hashicorp/google v5.44.2...
Step #3 - "validate rag": - Installed hashicorp/google-beta v5.44.2 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/external v2.3.4...
Step #3 - "validate rag": - Installed hashicorp/external v2.3.4 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/random v3.6.3...
Step #3 - "validate rag": - Installed hashicorp/random v3.6.3 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/kubernetes v2.18.1...
Step #1 - "validate ray": - Installed hashicorp/google v5.44.2 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installed hashicorp/google v5.44.2 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/google-beta v5.44.2...
Step #1 - "validate ray": - Installing hashicorp/null v3.2.3...
Step #1 - "validate ray": - Installed hashicorp/null v3.2.3 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/helm v2.8.0...
Step #3 - "validate rag": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/time v0.11.1...
Step #3 - "validate rag": - Installed hashicorp/time v0.11.1 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/google v5.44.2...
Step #1 - "validate ray": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/time v0.11.1...
Step #2 - "validate jupyterhub": - Installed hashicorp/google-beta v5.44.2 (signed by HashiCorp)
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Terraform has created a lock file .terraform.lock.hcl to record the provider
Step #2 - "validate jupyterhub": selections it made above. Include this file in your version control repository
Step #2 - "validate jupyterhub": so that Terraform can guarantee to make the same selections by default when
Step #2 - "validate jupyterhub": you run "terraform init" in the future.
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Terraform has been successfully initialized!
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": You may now begin working with Terraform. Try running "terraform plan" to see
Step #2 - "validate jupyterhub": any changes that are required for your infrastructure. All Terraform commands
Step #2 - "validate jupyterhub": should now work.
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": If you ever set or change modules or backend configuration for Terraform,
Step #2 - "validate jupyterhub": rerun this command to reinitialize your working directory. If you forget, other
Step #2 - "validate jupyterhub": commands will detect it and remind you to do so if necessary.
Step #1 - "validate ray": - Installed hashicorp/time v0.11.1 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/random v3.6.3
...
[Logs truncated due to log size limitations. For full logs, see https://storage.cloud.google.com/ai-on-gke-build-logs/log-0839c829-bc3e-4dad-919f-8dbfdd33023d.txt.]
...
tance_config {
Step #11 - "cleanup gke cluster":                   - enable_integrity_monitoring = true -> null
Step #11 - "cleanup gke cluster":                   - enable_secure_boot          = true -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - workload_metadata_config {
Step #11 - "cleanup gke cluster":                   - mode = "GKE_METADATA" -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - placement_policy {}
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - upgrade_settings {
Step #11 - "cleanup gke cluster":               - max_surge       = 1 -> null
Step #11 - "cleanup gke cluster":               - max_unavailable = 0 -> null
Step #11 - "cleanup gke cluster":               - strategy        = "SURGE" -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster":       - node_pool {
Step #11 - "cleanup gke cluster":           - initial_node_count          = 0 -> null
Step #11 - "cleanup gke cluster":           - instance_group_urls         = [] -> null
Step #11 - "cleanup gke cluster":           - managed_instance_group_urls = [] -> null
Step #11 - "cleanup gke cluster":           - max_pods_per_node           = 16 -> null
Step #11 - "cleanup gke cluster":           - name                        = "nap-hucg0sjz" -> null
Step #11 - "cleanup gke cluster":           - node_count                  = 0 -> null
Step #11 - "cleanup gke cluster":           - node_locations              = [
Step #11 - "cleanup gke cluster":               - "us-east4-a",
Step #11 - "cleanup gke cluster":               - "us-east4-c",
Step #11 - "cleanup gke cluster":             ] -> null
Step #11 - "cleanup gke cluster":           - version                     = "1.30.5-gke.1355000" -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - autoscaling {
Step #11 - "cleanup gke cluster":               - location_policy      = "BALANCED" -> null
Step #11 - "cleanup gke cluster":               - max_node_count       = 1000 -> null
Step #11 - "cleanup gke cluster":               - min_node_count       = 0 -> null
Step #11 - "cleanup gke cluster":               - total_max_node_count = 0 -> null
Step #11 - "cleanup gke cluster":               - total_min_node_count = 0 -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - management {
Step #11 - "cleanup gke cluster":               - auto_repair  = true -> null
Step #11 - "cleanup gke cluster":               - auto_upgrade = true -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - network_config {
Step #11 - "cleanup gke cluster":               - create_pod_range     = false -> null
Step #11 - "cleanup gke cluster":               - enable_private_nodes = false -> null
Step #11 - "cleanup gke cluster":               - pod_ipv4_cidr_block  = "10.122.128.0/17" -> null
Step #11 - "cleanup gke cluster":               - pod_range            = "gke-ml-4994c11-852-0839c829-cluster-pods-c64b71c3" -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - node_config {
Step #11 - "cleanup gke cluster":               - disk_size_gb                = 250 -> null
Step #11 - "cleanup gke cluster":               - disk_type                   = "pd-balanced" -> null
Step #11 - "cleanup gke cluster":               - effective_taints            = [
Step #11 - "cleanup gke cluster":                   - {
Step #11 - "cleanup gke cluster":                       - effect = "NO_SCHEDULE"
Step #11 - "cleanup gke cluster":                       - key    = "cloud.google.com/compute-class"
Step #11 - "cleanup gke cluster":                       - value  = "Performance"
Step #11 - "cleanup gke cluster":                     },
Step #11 - "cleanup gke cluster":                   - {
Step #11 - "cleanup gke cluster":                       - effect = "NO_SCHEDULE"
Step #11 - "cleanup gke cluster":                       - key    = "cloud.google.com/gke-ephemeral-storage-local-ssd"
Step #11 - "cleanup gke cluster":                       - value  = "true"
Step #11 - "cleanup gke cluster":                     },
Step #11 - "cleanup gke cluster":                 ] -> null
Step #11 - "cleanup gke cluster":               - enable_confidential_storage = false -> null
Step #11 - "cleanup gke cluster":               - image_type                  = "COS_CONTAINERD" -> null
Step #11 - "cleanup gke cluster":               - labels                      = {
Step #11 - "cleanup gke cluster":                   - "cloud.google.com/compute-class" = "Performance"
Step #11 - "cleanup gke cluster":                   - "cloud.google.com/pod-isolation" = "9"
Step #11 - "cleanup gke cluster":                   - "cloud.google.com/pod-slots"     = "1"
Step #11 - "cleanup gke cluster":                 } -> null
Step #11 - "cleanup gke cluster":               - local_ssd_count             = 0 -> null
Step #11 - "cleanup gke cluster":               - logging_variant             = "DEFAULT" -> null
Step #11 - "cleanup gke cluster":               - machine_type                = "c3-standard-22-lssd" -> null
Step #11 - "cleanup gke cluster":               - metadata                    = {
Step #11 - "cleanup gke cluster":                   - "disable-legacy-endpoints" = "true"
Step #11 - "cleanup gke cluster":                 } -> null
Step #11 - "cleanup gke cluster":               - oauth_scopes                = [
Step #11 - "cleanup gke cluster":                   - "https://www.googleapis.com/auth/cloud-platform",
Step #11 - "cleanup gke cluster":                   - "https://www.googleapis.com/auth/userinfo.email",
Step #11 - "cleanup gke cluster":                 ] -> null
Step #11 - "cleanup gke cluster":               - preemptible                 = false -> null
Step #11 - "cleanup gke cluster":               - resource_labels             = {} -> null
Step #11 - "cleanup gke cluster":               - resource_manager_tags       = {} -> null
Step #11 - "cleanup gke cluster":               - service_account             = "tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":               - spot                        = false -> null
Step #11 - "cleanup gke cluster":               - tags                        = [] -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - ephemeral_storage_config {
Step #11 - "cleanup gke cluster":                   - local_ssd_count = 4 -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - gvnic {
Step #11 - "cleanup gke cluster":                   - enabled = true -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - reservation_affinity {
Step #11 - "cleanup gke cluster":                   - consume_reservation_type = "NO_RESERVATION" -> null
Step #11 - "cleanup gke cluster":                   - values                   = [] -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - shielded_instance_config {
Step #11 - "cleanup gke cluster":                   - enable_integrity_monitoring = true -> null
Step #11 - "cleanup gke cluster":                   - enable_secure_boot          = true -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - workload_metadata_config {
Step #11 - "cleanup gke cluster":                   - mode = "GKE_METADATA" -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - placement_policy {}
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - upgrade_settings {
Step #11 - "cleanup gke cluster":               - max_surge       = 1 -> null
Step #11 - "cleanup gke cluster":               - max_unavailable = 0 -> null
Step #11 - "cleanup gke cluster":               - strategy        = "SURGE" -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - node_pool_defaults {
Step #11 - "cleanup gke cluster":           - node_config_defaults {
Step #11 - "cleanup gke cluster":               - logging_variant = "DEFAULT" -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - gcfs_config {
Step #11 - "cleanup gke cluster":                   - enabled = true -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - notification_config {
Step #11 - "cleanup gke cluster":           - pubsub {
Step #11 - "cleanup gke cluster":               - enabled = false -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - pod_security_policy_config {
Step #11 - "cleanup gke cluster":           - enabled = false -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - private_cluster_config {
Step #11 - "cleanup gke cluster":           - enable_private_endpoint = false -> null
Step #11 - "cleanup gke cluster":           - enable_private_nodes    = false -> null
Step #11 - "cleanup gke cluster":           - private_endpoint        = "10.128.0.2" -> null
Step #11 - "cleanup gke cluster":           - public_endpoint         = "34.48.87.44" -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - master_global_access_config {
Step #11 - "cleanup gke cluster":               - enabled = false -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - protect_config {
Step #11 - "cleanup gke cluster":           - workload_vulnerability_mode = "DISABLED" -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - workload_config {
Step #11 - "cleanup gke cluster":               - audit_mode = "DISABLED" -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - release_channel {
Step #11 - "cleanup gke cluster":           - channel = "REGULAR" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - secret_manager_config {
Step #11 - "cleanup gke cluster":           - enabled = false -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - security_posture_config {
Step #11 - "cleanup gke cluster":           - mode               = "DISABLED" -> null
Step #11 - "cleanup gke cluster":           - vulnerability_mode = "VULNERABILITY_DISABLED" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - service_external_ips_config {
Step #11 - "cleanup gke cluster":           - enabled = false -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - timeouts {
Step #11 - "cleanup gke cluster":           - create = "45m" -> null
Step #11 - "cleanup gke cluster":           - delete = "45m" -> null
Step #11 - "cleanup gke cluster":           - update = "45m" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - vertical_pod_autoscaling {
Step #11 - "cleanup gke cluster":           - enabled = true -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - workload_identity_config {
Step #11 - "cleanup gke cluster":           - workload_pool = "gke-ai-eco-dev.svc.id.goog" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-metric_writer[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_project_iam_member" "cluster_service_account-metric_writer" {
Step #11 - "cleanup gke cluster":       - etag    = "BwYljiXiMcI=" -> null
Step #11 - "cleanup gke cluster":       - id      = "gke-ai-eco-dev/roles/monitoring.metricWriter/serviceAccount:tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member  = "serviceAccount:tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - role    = "roles/monitoring.metricWriter" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_project_iam_member" "cluster_service_account-nodeService_account" {
Step #11 - "cleanup gke cluster":       - etag    = "BwYljiXiMcI=" -> null
Step #11 - "cleanup gke cluster":       - id      = "gke-ai-eco-dev/roles/container.defaultNodeServiceAccount/serviceAccount:tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member  = "serviceAccount:tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - role    = "roles/container.defaultNodeServiceAccount" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-resourceMetadata-writer[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_project_iam_member" "cluster_service_account-resourceMetadata-writer" {
Step #11 - "cleanup gke cluster":       - etag    = "BwYljiXiMcI=" -> null
Step #11 - "cleanup gke cluster":       - id      = "gke-ai-eco-dev/roles/stackdriver.resourceMetadata.writer/serviceAccount:tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member  = "serviceAccount:tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - role    = "roles/stackdriver.resourceMetadata.writer" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-autopilot-cluster[0].module.gke.google_service_account.cluster_service_account[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_service_account" "cluster_service_account" {
Step #11 - "cleanup gke cluster":       - account_id   = "tf-gke-ml-4994c11-852--3ghl" -> null
Step #11 - "cleanup gke cluster":       - disabled     = false -> null
Step #11 - "cleanup gke cluster":       - display_name = "Terraform-managed service account for cluster ml-4994c11-852-0839c829-cluster" -> null
Step #11 - "cleanup gke cluster":       - email        = "tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - id           = "projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member       = "serviceAccount:tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - name         = "projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project      = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - unique_id    = "104124915920319948434" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-autopilot-cluster[0].module.gke.random_string.cluster_service_account_suffix[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "random_string" "cluster_service_account_suffix" {
Step #11 - "cleanup gke cluster":       - id          = "3ghl" -> null
Step #11 - "cleanup gke cluster":       - length      = 4 -> null
Step #11 - "cleanup gke cluster":       - lower       = true -> null
Step #11 - "cleanup gke cluster":       - min_lower   = 0 -> null
Step #11 - "cleanup gke cluster":       - min_numeric = 0 -> null
Step #11 - "cleanup gke cluster":       - min_special = 0 -> null
Step #11 - "cleanup gke cluster":       - min_upper   = 0 -> null
Step #11 - "cleanup gke cluster":       - number      = true -> null
Step #11 - "cleanup gke cluster":       - numeric     = true -> null
Step #11 - "cleanup gke cluster":       - result      = "3ghl" -> null
Step #11 - "cleanup gke cluster":       - special     = false -> null
Step #11 - "cleanup gke cluster":       - upper       = false -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Plan: 0 to add, 0 to change, 10 to destroy.
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Changes to Outputs:
Step #11 - "cleanup gke cluster":   - ca_certificate   = (sensitive value) -> null
Step #11 - "cleanup gke cluster":   - cluster_location = "us-east4" -> null
Step #11 - "cleanup gke cluster":   - cluster_name     = "ml-4994c11-852-0839c829-cluster" -> null
Step #11 - "cleanup gke cluster":   - endpoint         = (sensitive value) -> null
Step #11 - "cleanup gke cluster":   - private_cluster  = false -> null
Step #11 - "cleanup gke cluster":   - project_id       = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Destroying... [id=projects/gke-ai-eco-dev/locations/us-east4/clusters/ml-4994c11-852-0839c829-cluster]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-metric_writer[0]: Destroying... [id=gke-ai-eco-dev/roles/monitoring.metricWriter/serviceAccount:tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0]: Destroying... [id=gke-ai-eco-dev/roles/container.defaultNodeServiceAccount/serviceAccount:tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-resourceMetadata-writer[0]: Destroying... [id=gke-ai-eco-dev/roles/stackdriver.resourceMetadata.writer/serviceAccount:tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0]: Destruction complete after 8s
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-metric_writer[0]: Destruction complete after 8s
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-resourceMetadata-writer[0]: Destruction complete after 8s
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 1m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 1m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 1m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 1m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 1m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 1m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 2m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 2m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 2m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 2m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 2m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 2m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 3m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 3m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 3m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 3m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 3m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 3m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-4994c11-852-0839c829-cluster, 4m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Destruction complete after 4m2s
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_service_account.cluster_service_account[0]: Destroying... [id=projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-4994c11-852--3ghl@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_service_account.cluster_service_account[0]: Destruction complete after 0s
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.random_string.cluster_service_account_suffix[0]: Destroying... [id=3ghl]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.random_string.cluster_service_account_suffix[0]: Destruction complete after 0s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_service_networking_connection.private_service_access[0]: Destroying... [id=https%3A%2F%2Fwww.googleapis.com%2Fcompute%2Fv1%2Fprojects%2Fgke-ai-eco-dev%2Fglobal%2Fnetworks%2Fml-4994c11-852-0839c829-true:servicenetworking.googleapis.com]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_service_networking_connection.private_service_access[0]: Destruction complete after 0s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-4994c11-852-0839c829-true"]: Destroying... [id=projects/gke-ai-eco-dev/regions/us-east4/subnetworks/ml-4994c11-852-0839c829-true]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_global_address.google-managed-services-range[0]: Destroying... [id=projects/gke-ai-eco-dev/global/addresses/google-managed-services-ml-4994c11-852-0839c829-true]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-4994c11-852-0839c829-true"]: Still destroying... [id=projects/gke-ai-eco-dev/regions/us-east...bnetworks/ml-4994c11-852-0839c829-true, 10s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_global_address.google-managed-services-range[0]: Still destroying... [id=projects/gke-ai-eco-dev/global/addresse...-services-ml-4994c11-852-0839c829-true, 10s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-4994c11-852-0839c829-true"]: Still destroying... [id=projects/gke-ai-eco-dev/regions/us-east...bnetworks/ml-4994c11-852-0839c829-true, 20s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_global_address.google-managed-services-range[0]: Still destroying... [id=projects/gke-ai-eco-dev/global/addresse...-services-ml-4994c11-852-0839c829-true, 20s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_global_address.google-managed-services-range[0]: Destruction complete after 21s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-4994c11-852-0839c829-true"]: Still destroying... [id=projects/gke-ai-eco-dev/regions/us-east...bnetworks/ml-4994c11-852-0839c829-true, 30s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-4994c11-852-0839c829-true"]: Destruction complete after 31s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_network.network: Destroying... [id=projects/gke-ai-eco-dev/global/networks/ml-4994c11-852-0839c829-true]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_network.network: Still destroying... [id=projects/gke-ai-eco-dev/global/networks/ml-4994c11-852-0839c829-true, 10s elapsed]
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Error: Error waiting for Deleting Network: The network resource 'projects/gke-ai-eco-dev/global/networks/ml-4994c11-852-0839c829-true' is already being used by 'projects/gke-ai-eco-dev/global/firewalls/ml-4994c11-852-0839c829-true-odm2l5xrgkhkgaqqdwsg4qbt'
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": 
Finished Step #11 - "cleanup gke cluster"
ERROR: step exited with non-zero status: 1
Starting Step #12 - "check result"
Step #12 - "check result": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Finished Step #12 - "check result"
PUSH
DONE

Build Log: https://storage.cloud.google.com/ai-on-gke-build-logs/log-0839c829-bc3e-4dad-919f-8dbfdd33023d.txt