From 0f71472c0c7764d3c86459ba0180ef02dbeddf3b Mon Sep 17 00:00:00 2001 From: Jon Chen Date: Mon, 1 Jul 2024 01:24:45 -0400 Subject: [PATCH] update docs --- README.md | 59 ++++--- docs/CLI_USAGE.md | 56 +++---- docs/DEVELOPMENT.md | 28 +++- docs/OVERVIEW.md | 336 ++++++++++++++++++++++++++++++++++++++ docs/guides/cloudshell.md | 93 ++++++----- docs/guides/skaffold.md | 0 docs/guides/templates.md | 21 --- 7 files changed, 468 insertions(+), 125 deletions(-) create mode 100644 docs/OVERVIEW.md delete mode 100644 docs/guides/skaffold.md delete mode 100644 docs/guides/templates.md diff --git a/README.md b/README.md index 8d02901a..0b1a4479 100644 --- a/README.md +++ b/README.md @@ -16,13 +16,11 @@ Test Automation, CI/CD, etc, with built-in modules including: - Modules and templates based on [Copier](https://copier.readthedocs.io/en/stable/). - CI/CD deployment templates. -## Build a solution by adding modules +### Build a solution by adding modules -Solutions Builder is designed with modules and templates. When creating a new solution -folder, it starts from a base template with a terraform foundation and project skeleton. -Then, you can start adding modules into this solution folder. +Solutions Builder is designed with modules and templates. It starts from a base project skeleton, then start adding modules into this solution folder. -For example, these are examples of creating a new solution: +For example, here are the steps when creating a new solution: - Start a new solution with a base (skeleton) code from a "root template" - Add a Cloud Run microservice with APIs @@ -31,36 +29,41 @@ For example, these are examples of creating a new solution: ![alt text](docs/assets/adding_modules.png) -Check out the [Quick Start](#quick-start---create-a-new-solution) section to create a sample solution. +See [Quick Start](#quick-start---create-a-new-solution) section to create a sample solution. + +### Concepts and Design Details + +- See [Overview](docs/OVERVIEW.md) for overall concepts and design details. ## Prerequisite -| Tool | Required Version | Installation | -| ---------- | ---------------- | ---------------------------------------------------- | -| Python | >= 3.11 | | -| gcloud CLI | Latest | https://cloud.google.com/sdk/docs/install | -| Terraform | >= v1.3.7 | https://developer.hashicorp.com/terraform/downloads | -| Skaffold | >= v2.4.0 | https://skaffold.dev/docs/install/#standalone-binary | +Required dependencies: -[Optional] For deploying to a GKE cluster, please install the following: +- Python (3.10 or later) +- [gcloud CLI](https://cloud.google.com/sdk/docs/install) (Latest) +- [Terraform](https://www.terraform.io/) (v1.4.0 or later) +- [Skaffold](https://skaffold.dev/docs/install/#standalone-binary) (v2.4.0 or later) -| Tool | Required Version | Installation | -| --------- | ---------------- | ---------------------------------------------------------- | -| Kustomize | >= v5.0.0 | https://kubectl.docs.kubernetes.io/installation/kustomize/ | +Optional dependencies for GKE cluster deployment: -Install all packages: +- [Kustomize](https://kubectl.docs.kubernetes.io/installation/kustomize) (v5.0.1 or later) -Mac: +Install dependencies with package managers: -``` -brew install gcloud terraform skaffold -``` +- Mac: -Windows: + ``` + brew install gcloud terraform skaffold + ``` -``` -choco install gcloud terraform skaffold -``` +- Windows: + + ``` + choco install gcloud terraform skaffold + ``` + +- Others + - Google Cloud Shell - See [docs/guides/cloudshell.md](docs/guides/cloudshell.md) ## Install Solutions Builder CLI @@ -131,7 +134,11 @@ Copying from template Complete. New solution folder created at ./my-solution-folder. ``` -Add a Sample Cloud Run microservice using `blank_service` module template: +- At this point, you'll see a solution folder created at `./my-solution-folder`, and a `sb.yaml` + file with all configuration about this solution folder. +- See [overview.md](docs/OVERVIEW.md) for more details about `sb.yaml`. + +Next, add a Sample Cloud Run microservice using `blank_service` module template: ``` cd my-solution-folder diff --git a/docs/CLI_USAGE.md b/docs/CLI_USAGE.md index 9695487d..d255e0d1 100644 --- a/docs/CLI_USAGE.md +++ b/docs/CLI_USAGE.md @@ -1,31 +1,5 @@ # Solutions Builder CLI Usage -## Concept - -### Copier-based templates - -- All templates and modules are created using [Copier](https://copier.readthedocs.io/en/stable/). -- Each template folder (e.g. in /modules) has a `copier.yaml` to define questions and variables. - -### sb.yaml - -- Once creating a solution folder, it will create a `sb.yaml` file in the solution folder. -- When adding a new component/module, it will add the configuration for the component to the `sb.yaml` file. - -### Components / Modules - -- A component (or module) can be a Terraform module, a microservice (Python) module, or a frontend module. -- A template must have a `copier.yaml` to define variables - and other configurations. You can still proceed if there's no - `copier.yaml` in the template folder, but it will just copy - entire folder without any modification. - -There are three ways of adding a component: - -- Add a component with built-in module template -- Add a component with a template in local folder -- Add a component with a template in remote Git repo - ## Installation With `pip`: @@ -109,10 +83,11 @@ sb new my-solution-folder -t /path/to/my-template-folder To create a new solution folder with a template a remote Git repo, add `-t ` like below: ``` -sb new my-solution-folder -t git@github.com:GoogleCloudPlatform/solutions-builder.git/modules/blank_service +sb new my-solution-folder -t git@github.com:GoogleCloudPlatform/solutions-builder.git/modules/root_template ``` - This will download the Git repo automatically, and use the "subfolder" as the template path. +- In this case, the repo path is `git@github.com:GoogleCloudPlatform/solutions-builder.git`, with the subfolder `modules/root_template`. ## Add a component @@ -202,6 +177,8 @@ To create a component using a template from a remote Git repo: sb add component my_service -t git@github.com:GoogleCloudPlatform/solutions-builder.git/modules/blank_service ``` +- The git repo path is `git@github.com:GoogleCloudPlatform/solutions-builder.git` with the subfolder `modules/blank_service`. + ## Terraform modules If you create a solution folder using `sb new `, it will create the Terraform base module for you. @@ -234,10 +211,25 @@ sb terraform apply --all - Optionally, add `--yes` to the end of the command to skip the confirmation prompt. E.g.: ``` + # This will init and apply all Terraform stages. sb terraform apply --all --yes ``` -- This will init and apply all Terraform stages. +What's behind the `sb terraform apply` command? + +- `sb terraform` runs `terraform` command behind the scene. The `sb terraform apply --all` + runs the following steps: + ``` + cd $SOLUTION_FOLDER/terraform/stages/1-bootstrap + terraform init + terraform apply -auto-approve + cd $SOLUTION_FOLDER/terraform/stages/2-foundation + terraform init + terraform apply -auto-approve + cd $SOLUTION_FOLDER/terraform/stages/3-application + terraform init + terraform apply -auto-approve + ``` ### Apply a specific Terraform stage @@ -245,6 +237,14 @@ sb terraform apply --all sb terraform apploy 3-application ``` +- Alternatively, this command is equivalent to the following steps: + + ``` + cd $SOLUTION_FOLDER/terraform/stages/3-application + terraform init + terraform apply -auto-approve + ``` + ### Destroy a specific Terraform stage ``` diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index 46084cdd..b66c2eb8 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -9,7 +9,7 @@ git clone https://github.com/GoogleCloudPlatform/solutions-builder cd solutions-builder ``` -To development locally with editable files: +### To development locally ``` poetry lock && poetry install @@ -35,22 +35,34 @@ poetry build - If encountering any errors, run `poetry build -vvv` to troubleshoot. -Test package upload to Test-PyPI: +### Test package upload to Test-PyPI + +Publish to Test-PyPI: ``` -# Publish poetry config repositories.test-pypi https://test.pypi.org/legacy/ poetry config pypi-token.test-pypi $TEST_PYPI_TOKEN -poetry publish -r test-pypi +poetry build && poetry publish -r test-pypi + +``` -# Install and test +Install and test from Test-PyPI + +``` pip install -U --index-url https://test.pypi.org/simple/ solutions-builder ``` -Publish to official PyPI: +### Publish to official PyPI + +Publish to PyPI ``` -# Publish to Test-PyPI poetry config pypi-token.pypi $PYPI_TOKEN -poetry publish +poetry build && poetry publish +``` + +Test with the published package: + +``` +pip install -U solutions-builder ``` diff --git a/docs/OVERVIEW.md b/docs/OVERVIEW.md new file mode 100644 index 00000000..ad2ba5b9 --- /dev/null +++ b/docs/OVERVIEW.md @@ -0,0 +1,336 @@ +# Solutions Builder Overview + +## Concept + +### Copier and Jinja templates + +- All templates and modules are created using [Copier](https://copier.readthedocs.io/en/stable/). +- Each template folder (e.g. in /modules) has a `copier.yaml` to define questions and variables. +- [Copier](https://copier.readthedocs.io/en/stable/) uses Jinja for template syntax. + +### sb.yaml + +- Once creating a solution folder, it will create a `sb.yaml` file in the solution folder. +- When adding a new component/module, it will add the configuration for the component to the `sb.yaml` file. + +## Components + +- A component can be a Terraform module, a microservice (Python) module, or a frontend module, etc. +- A template folder must have a `copier.yaml` to define variables + and other configurations. You can still proceed if there's no + `copier.yaml` in the template folder, but it will just copy + entire folder without any modification. + +There are three ways of adding a component: + +- Add a component with built-in module template +- Add a component with a template in local folder +- Add a component with a template in remote Git repo + +### Adding a component with built-in module template + +Run the following to add a component with built-in module template: + +``` +sb add component my_component_name -t blank_service +``` + +- This will create a new component called `my_component_name` with `blank_service` as the template. +- The `blank_service` module is one of the built-in modules in Solutions Builder. + +### Adding a component with a template in local folder + +Run the following to add a component with a template in local folder: + +``` +sb add component my_component_name -t /path/to/module +``` + +- This will create a new component called `my_component_name` with the module template in `/path/to/module`. +- The folder `/path/to/module` requires a `copier.yaml`. See the [Copier template] section below. + +### Adding a component with a template in remote Git repo + +Run the following to add a component with a template in local folder: + +``` +sb add component my_component_name -t git@github.com:GoogleCloudPlatform/solutions-builder.git/modules/blank_service +``` + +- This will create a new component called `my_component_name` with the module template in remote git repo `git@github.com:GoogleCloudPlatform/solutions-builder.git` with subfolder `modules/blank_service`. + +## Copier template + +Solutions Builder is built with [Copier](https://copier.readthedocs.io/en/stable/), which uses Jinja for template syntax. Each module folder has a `copier.yaml` that defines the questions and variables +that Copier will replaces with. + +For example, a microservice module folder has following files: + +``` +├── copier.yaml +├── skaffold.yaml.patch +├── terraform +│ └── stages +│ └── 3-application +│ └── {{component_name}}_cloudrun.tf +└── components + └── {{component_name}} + ├── Dockerfile + ├── README.md + ├── requirements.txt + ├── skaffold.yaml + ├── src + └── manifests + └── cloudrun-service.yaml +``` + +`copier.yaml` defines the questions (variables) in this module: + +``` +component_name: + type: str + help: What is the name of this component (snake_case)? + default: my_service + validator: "{% if not component_name %}Required{% endif %}" + +region: + type: str + help: Which Google Cloud region? + default: us-central1 + +deploy_method: + type: str + help: Default deploy method? (cloudrun or gke) + choices: + Cloud Run: cloudrun + GKE: gke + default: cloudrun + +... + +``` + +When adding a module from a Copier template, it will ask questions for +these variables: + +``` + +🎤 What is the name of this component (snake_case)? + test_service +🎤 Which Google Cloud region? + us-central1 +🎤 Default deploy method? (cloudrun or gke) + Cloud Run + +``` + +These variables (e.g. `component_name`, `region`, etc) will be used in Jinja templates like below: + +``` +# terraform/stages/3-application/{{component_name}}\_cloudrun.tf + +resource "google_cloud_run_v2_service" "{{component_name}}" { + depends_on = [ + null_resource.{{component_name}}_build_container + ] + name = "{{resource_name}}" + project = var.project_id + location = var.region + ingress = "INGRESS_TRAFFIC_ALL" + + template { + containers { + image = "us-docker.pkg.dev/${var.project_id}/default/{{resource_name}}:latest" + } + } +} + +``` + +- Variables like `{{component_name}}`, `{{region}}`, etc. will be replaced with the value given by the user. + +In `copier.yaml`, it also defines other configurations: + +- `_answers_file` defines where to put the answer.yaml file. + + ``` + _answers_file: ".sb/component_answers/{{component_name}}.yaml" + ``` + + - This YAML stores the answered values for each variables from `copier.yaml`. + +- `_templates_suffix` is set as empty string so Copier will process all file with any extensions. +- `_patch` is used by Solutions Builder to "patch" files with delta instead of overwriting. + ``` + _patch: + - "skaffold.yaml" + ``` + - In the example above, this template will patches `skaffold.yaml` by appending content to the end of the file. +- `_exclude` defines a list of files (regex) to exclude. See [Copier docs](https://copier.readthedocs.io/en/stable/configuring/#exclude) for more details. + +- `_jinja_extensions` defines list of extensions for additional Jinja functions. In Solutions Builder, it uses a custom extension file with several helper functions. + ``` + _jinja_extensions: + - jinja2_time.TimeExtension + - jinja2_strcase.StrcaseExtension + - copier_templates_extensions.TemplateExtensionLoader + - ../../copier_extensions/sb_helpers.py:SolutionsTemplateHelpersExtension + ``` + - `copier_extensions/sb_helpers.py` contains the custom helper functions. See [Copier docs](https://copier.readthedocs.io/en/stable/configuring/#jinja_extensions) for more details. + +## Building & deploying with Skaffold + +Solutions Builder uses [Skaffold](https://skaffold.dev) as the deployment orchestration tool. See https://skaffold.dev for more details. + +In a nutshell, Skaffold simplifies the process of building and deploying steps: + +- Builds Docker images +- Push container images to Artifact Registry +- Deploy images to Cloud Run or a Kubernetes cluster. +- Optionally, deploy a image with local port forwarding for local troubleshooting. + +Traditionally, each step requires one or a few commands, and requires a numbers of parameters. +Skaffold simplifies all the steps to one command like below: + +``` +skaffold run -p cloudrun --default-repo="us-docker.pkg.dev/my-project-id/default" +``` + +- This command will build, upload, and deploy image to Cloud Run. + +### Root skaffold.yaml + +In Solutions Builder, it uses 2-layer Skaffold configuration files for overall solution vs. individual component. + +E.g. the file structure looks like below: + +``` +├── sb.yaml +├── skaffold.yaml # root skaffold.yaml +├── components +│ └── test_service +│ ├── Dockerfile +│ ├── skaffold.yaml # component skaffold.yaml +│ └── src +│ ├── ... +└── ... + +``` + +- A root `skaffold.yaml` in the root solution folder. +- Each component (e.g. a microservice folder) has its own `skaffold.yaml`. + +In the root `skaffold.yaml`, it defines where to find other components: + +``` +apiVersion: skaffold/v4beta1 +kind: Config +metadata: + name: all-services +requires: +- configs: + - test_service + path: ./components/test_service +``` + +- `requires` defines a list of configs with name and path for each component. + +When adding a new component, it will patch a new config and path to the end of the requires list. This is defined in the `copier.yaml` in a module: + +``` +_patch: + - "skaffold.yaml" +``` + +A `skaffold.yaml.path` in a module template looks like this: + +``` +requires: + - configs: + - {{component_name}} + path: ./{{destination_path}}/{{component_name}} + +``` + +- When adding a component, Solutions Builder will merge the patch file to the root `skaffold.yaml`. +- See [blank_service/skaffold.yaml](../../solutions_builder/modules/blank_service/skaffold.yaml.patch) for an example. + +### skaffold.yaml in modules + +Skaffold uses `skaffold.yaml` to define all steps with several sections. E.g.: + +``` +apiVersion: skaffold/v4beta1 +kind: Config +metadata: + name: test_service + +# This section defines how it builds Docker images. +build: + ... + +# This section defines a list of profiles for different deploy methods. +profiles: +- name: cloudrun + ... + +- name: gke + ... +``` + +In the `build` section: + +``` +build: + artifacts: + - image: test-service + sync: + infer: + - '**/*.py' + - '**/*.json' + docker: + cacheFrom: + - test-service + - test-service:latest + googleCloudBuild: {} + +``` + +- `artifacts` sub-section defines the following: + - `image` defines the image name that will be used and uploaded to the + Artifact Registry. + - `sync` is for remote hot-reload, useful for remote troubleshooting. See [File Sync](https://skaffold.dev/docs/filesync/) for more info. + - `docker` defines how it build Docker images using Dockerfile and usage of caching. +- When `googleCloudBuild` presents, it will use GCP Cloud Build to build the docker images. + - It leaves `{}` to set default Cloud Build configuration. + +In the `profiles` section, it defines one or more "profiles": + +``` +profiles: + +# Profile for Cloud Run deployment, building images via CloudBuild +- name: cloudrun + manifests: + rawYaml: + - manifests/cloudrun-service.yaml + deploy: + cloudrun: + projectid: my-project-id + region: us-central1 + portForward: + - resourceType: service + resourceName: test-service + port: 80 + localPort: 9001 # For local port forwarding + +``` + +In the `cloudrun` profile, it defines where to find the manifest YAML, how to deploy, and optionally the local port-forwarding config. + +- `name` - Profile name. +- `manifests` - Cloud Run service YAML in `manifests/cloudrun-service.yaml` +- `deploy` - Project ID and region. +- `portForward` - Local port forwarding config. + +See [Skaffold - Cloud Run](https://skaffold.dev/docs/deployers/cloudrun/) for more info. diff --git a/docs/guides/cloudshell.md b/docs/guides/cloudshell.md index 9f1e70ae..2df31a09 100644 --- a/docs/guides/cloudshell.md +++ b/docs/guides/cloudshell.md @@ -1,42 +1,51 @@ -# Getting started a new Solution using Cloud Shell - -## Create a new Solution - -- Create a new GCP project -- Open up Cloud Shell -- Run the following in the Cloud Shell - ``` - export PROJECT_ID=$DEVSHELL_PROJECT_ID - sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin - ``` - -- Ensure `kustomize` version to be >= v5.0.0 - > kustomize - ``` - cd ~ - wget https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh - sudo rm /usr/local/bin/kustomize - sudo ./install_kustomize.sh 5.0.0 /usr/local/bin - ``` - -- Ensure `skaffold` version to be >= v2.4.0 - ``` - skaffold version - # v2.4.0 - ``` - -- Generate skeleton code in a new folder: - ``` - cookiecutter https://github.com/GoogleCloudPlatform/solutions-builder.git - ``` - - Provide the required variables to Cookiecutter prompt. - -- Build and deploy services to CloudRun: - ``` - cd - - # Choose the microservice deployment options: "gke", "cloudrun" or "gke|cloudrun" - export TEMPLATE_FEATURES="cloudrun" - source setup/init_env_var.sh - bash setup/setup_all.sh - ``` +# Using Solutions Builder in Google Cloud Shell + +## Install dependencies + +By default, Google Cloud Shell has the following installed (as of Jun 30th, 2024): + +- Python 3.10 +- Terraform v1.5.7 +- Skaffold v2.11.1 + +Optionally, install the following for GKE cluster deployment. + +``` +wget https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh +sudo rm /usr/local/bin/kustomize +sudo ./install_kustomize.sh 5.0.0 /usr/local/bin +``` + +Install Solutions Builder: + +``` +pip3 install -U solutions-builder +``` + +- Make sure to run with python3 pip. + +## Create a new GCP project + +It's highly recommended to create a new GCP project before running Solutions Builder in a Google Cloud Shell. + +- To avoid overriding the GCP resources in other existing project. + +## Create a new solution + +The following steps are the same as running Solutions Builder in your local environment. + +``` +sb new my-project-id +``` + +Then, add a component with the `blank_service` template: + +``` +sb add component test_service -t blank_service +``` + +Run the following to apply all terraforms, build and deploy to Cloud Run. + +``` +sb terraform apply --all --yes +``` diff --git a/docs/guides/skaffold.md b/docs/guides/skaffold.md deleted file mode 100644 index e69de29b..00000000 diff --git a/docs/guides/templates.md b/docs/guides/templates.md deleted file mode 100644 index 601f54e4..00000000 --- a/docs/guides/templates.md +++ /dev/null @@ -1,21 +0,0 @@ -# Working with Template (component/module) - -## Component/module template - -- A component (or module) can be a Terraform module, a microservice (Python) module, or a frontend module. -- A template must have a `copier.yaml` to define variables - and other configurations. You can still proceed if there's no - `copier.yaml` in the template folder, but it will just copy - entire folder without any modification. - -There are three ways of adding a component: - -- Add a component with built-in module template -- Add a component with a template in local folder -- Add a component with a template in remote Git repo - -## Create a Copier template - -Solutions Builder uses Copier as templates. - -To learn how to create a Copier template, see https://copier.readthedocs.io/en/stable/creating/.