Skip to content

Latest commit

 

History

History
164 lines (135 loc) · 5.74 KB

README.md

File metadata and controls

164 lines (135 loc) · 5.74 KB
Raw

image

Workshop Deployment via terraform

IMPORTANT TO KNOW FOR THE WORKSHOP: We run in AWS, Azure and Google Cloud. Currently we do support 24 Regions. Please be aware that the cluster and the Flink Pool need to be in the same Cloud-Provider-Region.

This is the deployment of confluent cloud infrastructure resources to run the Flink SQL Hands-on Workshop. We will deploy with terraform:

  • Environment:
    • Name: flink_hands-on+UUID
    • with enabled Schema Registry (essentials) in AWS region (eu-central-1)
  • Confluent Cloud Basic Cloud: cc_handson_cluster
    • in AWS in region (eu-central-1)
  • Connectors:
    • Datagen for shoe_products
    • Datagen for shoe_customers
    • Datagen for show_orders
  • Service Accounts
    • app_manager-XXXX with Role EnvironmentAdmin
    • sr-XXXX with Role EnvironmentAdmin
    • clients-XXXX with Role CloudClusterAdmin
    • connectors-XXXX

image

Pre-requisites

  • User account on Confluent Cloud
  • Local install of Terraform (details below)
  • Local install of jq (details below)
  • Local install Confluent CLI, install the cli
  • Create API Key in Confluent Cloud via CLI: 
    confluent login
confluent api-key create --resource cloud --description "API for terraform"
# It may take a couple of minutes for the API key to be ready.
# Save the API key and secret. The secret is not retrievable later.
#+------------+------------------------------------------------------------------+
#| API Key    | <your generated key>                                             |
#| API Secret | <your generated secret>                                          |
#+------------+------------------------------------------------------------------+
  • Or visit the Cloud API Key page to create a Cloud API Key for your user, if you don't have any yet.

Installation (only need to do that once)

Install Terraform on MacOS

brew tap hashicorp/tap
brew install hashicorp/tap/terraform
brew update
brew upgrade hashicorp/tap/terraform

If you are running Windows, please use this guide
If you are running on Ubuntu (or WSL2 with Ubuntu), please use this guide

This tutorial was tested with Terraform v1.6.4 and confluent terraform provider 1.55.0 . To upgrade terraform on MacOS use

brew upgrade terraform

Or download new version from website

Install jq

brew install jq

If you are running Windows, download from here
If you are running on Ubuntu (or WSL2 with Ubuntu), please follow the instructions here

Install Confluent Cli

Please install the Confluent CLI, with these instructions

# For Mac
brew install confluentinc/tap/cli

This tutorial was developed with Confluent CLI v3.41.0 and run on lates: v4.9.8

Provision services for the demo

Clone the repo on your desktop.

cd $HOME # or where-ever directory you want to use
git clone https://github.com/griga23/shoe-store.git
cd shoe-store

Create Confluent Cloud Resource management API Keys

Go to the Confluent Console via the url https://confluent.cloud/settings/api-keys then, use the "+ Add API key" button on the right side of the page. Select "My account", >> "Next" then select the "Cloud resource management" tile, complete and download the API key. It will be use to get Terraform plan and provision resources.

Set environment variables

  • Add your API key to the Terraform variables by creating a tfvars file
cat > $PWD/terraform/terraform.tfvars <<EOF
confluent_cloud_api_key = "{Confluent Cloud API Key}"
confluent_cloud_api_secret = "{Confluent Cloud API Key Secret}"
cc_cloud_provider = "{the_selected_cloud_provider}"
cc_cloud_region= "{the_selected_region_of_the_confluent_cloud}"
EOF

Optional: Prefix your resources

In some cases you may want to prefix your resources, to do so use:

cat >> $PWD/terraform/terraform.tfvars <<EOF
use_prefix = "{choose your prefix}"
EOF

Deploy via terraform

run the following commands:

cd ./terraform
terraform init
terraform plan
terraform apply
# Apply shows you what will be provision, enter yes to continue provisioning 
terraform output -json
# for sensitive data
terraform output -raw SRSecret
terraform output -raw AppManagerSecret
terraform output -raw ClientSecret

Please check whether the terraform execution went without errors.

You can copy the login instruction also from the UI.

To continue with the UI:

You deployed:

image

You are ready to start with LAB1

Destroy the hands.on infrastructure

terraform destroy

There could be a conflict destroying everything with our Tags. In this case destroy again via terraform.

#╷
#│ Error: error deleting Tag "<tagID>/PII": 409 Conflict
#│ 
#│ 
#╵
#╷
#│ Error: error deleting Tag "<tagID>/Public": 409 Conflict
#│ 
#│ 
#╵
# destroy again
terraform destroy