Skip to content

Latest commit

 

History

History
490 lines (393 loc) · 18.8 KB

DEVELOPMENT.md

File metadata and controls

490 lines (393 loc) · 18.8 KB
Raw

Development

Table of Content

  1. Project Requirements
  2. Code Submission Process
  3. Local Environment Setup
  4. Develop and Test on a GKE cluster
  5. Advanced Skaffold commands
  6. Debugging
  7. Testing
  8. Running Jupyter Notebooks

This doc explains the development workflow, so you can get started contributing code to Core Solution Services

1. Project Requirements

Install the following based on the README.md#Prerequisites

  • skaffold
  • kustomize
  • kubectx
  • kubens

2. Code Submission Process

2.1. For the first-time setup:

  • Create a fork of a Git repository (using the button on the right corner of the page)
  • Choose your own GitHub profile to create this fork under your name.
  • You may want to slightly alter the name of your forked repo to make it easier to distinguish from the original repo (e.g., gps-core-solution-services is forked from the original repo core-solution-services)
  • Clone the forked repo to your local computer. 
    export YOUR_GITHUB_USERNAME=<your-github-username>
export YOUR_FORKED_REPO_NAME=<your-forked-repo-name>
cd ~/workspace
git clone https://github.com/${YOUR_GITHUB_USERNAME}/${YOUR_FORKED_REPO_NAME}.git
cd ${YOUR_FORKED_REPO_NAME}
  • Verify if the local git copy has the right remote endpoint. 
    git remote -v
# This will display the detailed remote list like below.
origin  https://github.com/<your-github-username>/<your-forked-repo-name>.git (fetch)
origin  https://github.com/<your-github-username>/<your-forked-repo-name>.git (push)
    • If for some reason your local git copy does not have the correct remotes, run the following: 
      git remote add origin https://github.com/${YOUR_GITHUB_USERNAME}/${YOUR_FORKED_REPO_NAME}.git
# Or, to reset the URL if origin remote exists
git remote set-url origin https://github.com/${YOUR_GITHUB_USERNAME}/${YOUR_FORKED_REPO_NAME}.git
  • Add the upstream repo to the remote list as upstream. 
    git remote add upstream https://github.com/GoogleCloudPlatform/core-solution-services.git
    • In default case, upstream will be the repo that you make the fork from.

2.2. When making code changes

  • Sync your fork with the latest commits in upstream/master branch. (more info) 
    # In your local fork repo folder.
git checkout -f main
git pull upstream main
  • Create a new local branch from main to start a new task (e.g. working on a feature or a bug fix): 
    # This will create a new branch.
git branch # verify output as "* main"
git checkout -b feature_xyz
  • After making changes, commit the local change to this custom branch and push to your fork repo on GitHub. Alternatively, you can use editors like VSCode to commit the changes easily. 
    git commit -a -m 'Your description'
git push
# Or, if it doesn’t push to the origin remote by default.
git push --set-upstream origin $YOUR_BRANCH_NAME
    • This will submit the changes to your fork repo on GitHub.
  • Go to your GitHub fork repo web page, click the Compare & Pull Request in the notification. In the Pull Request form, make sure that:
    • The upstream repo name is correct.
    • The destination branch is set to main.
    • The source branch is your custom branch. (e.g. feature_xyz in the example above)
    • You may also pick specific reviewers for this pull request.
  • Once the pull request is created, it will appear on the Pull Request list of the upstream origin repository, which will automatically run basic tests and checks via the CI/CD.
  • If any tests failed, fix the code in your local branch, re-commit and push the changes to the same custom branch. 
    # after fixing the code…
git commit -a -m 'another fix'
git push
    • This will update the pull request and re-run all necessary tests automatically.
    • If all tests passed, you will need to wait for the reviewers’ approval.
  • Once the request has been approved, the reviewer or Repo Admin will merge the pull request back to the upstream main branch.

2.3. (For Repo Admins) Reviewing a Pull Request

For code reviewers, go to the Pull Requests page of the origin repo on GitHub.

  • Go to the specific pull request, review and comment on the request branch.
  • Alternatively, you can use GitHub CLI gh to check out a PR and run the code locally: https://cli.github.com/manual/gh_pr_checkout
  • If all goes well with tests passed, click "Merge pull request" to merge the changes to main.

3. Local Environment Setup

3.1. VS Code Setup

Copy the following and paste to the settings.json of your VS Code:

{
  "python.linting.enabled": true,
  "python.linting.pylintPath": "pylint",
  "editor.formatOnSave": true,
  "python.formatting.provider": "yapf",
  "python.formatting.yapfArgs": [
    "--style={based_on_style: pep8, indent_width: 2}"
  ],
  "python.linting.pylintEnabled": true,
  "terminal.integrated.env.osx": {
    "PYTHONPATH": "${workspaceFolder}/components/common/src/"
  },
  "python.analysis.extraPaths": [
    "${workspaceFolder}/components/common/src/"
  ],
  "python.autoComplete.extraPaths": [
    "${workspaceFolder}/components/common/src/"
  ]
}

You may need to reload VS Code for these to take effect:

  • CMD + SHIFT + P
  • Developer: Reload Window

If VS Code asks you to install tools like pylint, etc. go ahead and do so.

3.2. Set up VirtualEnv for components/<microservice_folder>

Each component folder (non-common) in ./components represents a standalone Docker container and has its own Python dependencies defined in requirements.txt.

These microservice components also depend on ./components/common, so additional setup is required for the IDE's code-completion to register the common modules.

  • Make sure you aren't in a VirtualEnv

    deactivate

  • Set up VirtualEnv just for this microservice component

    cd components/<component_name>

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip

# install requirements from common
pip install -r ../common/requirements.txt

# install requirements from common_ml
pip install -r ../common_ml/requirements.txt

# install microservice requirements
pip install -r requirements.txt

  • (For VS Code only) Set up Python interpreter:

    • Open Command Palette (CMD + SHIFT + P)
    • Type Python: Select Interpreter
    • Choose your new interpreter
      • will look something like ./common/.venv/bin/python3

  • (For VS Code Only) Add the following to the settings.json in VS Code. The modules should load when you run from command line, and you should see code completion hints.

    {
  "terminal.integrated.env.osx": {
    "PYTHONPATH": "${workspaceFolder}/components/common/src/"
  },
  "python.analysis.extraPaths": [
    "${workspaceFolder}/components/common/src/"
  ]
}

  • You should now be able to load modules and test them locally:

    cd components/<component_name>/src
python main.py

  • Once you finish development for this component, run the following to exit the VirtualEnv:

    deactivate

3.3. Set up VirtualEnv for components/common

Follow the steps below to create a new VirtualEnv for components/common and install required python packages defined in the requirements.txt.

  • Create a VirtualEnv for a specific component
cd components/<component_name> # e.g. llm_service

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

  • (For VSCode only) Set up Python interpreter:

    • Open Command Palette (CMD + SHIFT + P)
    • Type Python: Select Interpreter
    • Choose your new interpreter
      • will look something like ./common/.venv/bin/python3

  • You should now be able to load modules and test them locally:

    # In ./components/<component_name>
cd src
python

    In REPL (Copy and paste the code)

    from common.models import User
user = User()

  • Once you finish development for this component, run the following to exit the VirtualEnv:

    deactivate

4. Develop and Test on a GKE cluster

4.1. Double-check the current gcloud config

  • Please make sure the gcloud command set to the current project. 
    export PROJECT_ID=<your-dev-project-id>
gcloud config set project ${PROJECT_ID}

4.2. Setup a context for your cluster

  • By default, skaffold builds with CloudBuild and runs in kubernetes cluster set in your local kubeconfig, using the namespace set in SKAFFOLD_NAMESPACE. If it is set to your GKE cluster, it will deploy to the cluster.

  • Set the following variables to your kubernetes cluster name and region. If you are not sure of your cluster name or region check the GCP cloud console.

    export REGION=$(gcloud container clusters list --filter=main-cluster --format="value(location)")
export CLUSTER_NAME=main-cluster

  • Create a kubernetes context for your cluster.

    gcloud container clusters get-credentials $CLUSTER_NAME --region $REGION --project $PROJECT_ID

  • Check your current context to verify that is it pointing to the GKE cluster you will use for development:

    kubectl config current-context

# Or you can use kubectx tool:
kubectx

4.3. Set up Kubernetes Namespace (optional)

You only need to set up a separate namespace if you are developing on the same cluster with multiple developers. Setting up a namespace allows you to deploy your own copy of the CSS services, running in your "namespace" on the Kubernetes cluster, that don't conflict with other developers who have their own own copy of the services in their own namespaces.

If you are in a small team (2-3 developers) you may also not need to set up your own namespaces. You can use the "default" namespace, as long as you don't have multiple developers testing deployments of a set of services that implement a new feature simultaneously.

  • Run the following to create a Kubernetes namespace for your development

    export SKAFFOLD_NAMESPACE=$YOUR_GITHUB_USERNAME
kubectl create ns $SKAFFOLD_NAMESPACE

  • Set the kubernetes context to your namespace

    kubectl config set-context --current --namespace=$SKAFFOLD_NAMESPACE

  • Run the following to create a Kubernetes Service Account (gke-sa) in your namespace and bind it to the GCP service account used for GKE:

    export PROJECT_ID=<your-dev-project-id>
bash ./setup/setup_ksa.sh
    • This will create a service account in the GKE cluster.
    • It will also bind the GKE service account with the regular GCP service account named "gke-sa".
    • You can verify it on the GCP Console > Service Accounts > "gke-sa" service account > Permissions.

4.4. Build and deploy all microservices

For development the skaffold dev command is very useful as it will deploy a set of services with local ports (defined in the skaffold config for each service), as well as tailing the logs locally, and performing a live reload when any file is changed. This allows you to rapidly test changes to the microservices.

  • Set the default repo for containers built by skaffold

    export SKAFFOLD_DEFAULT_REPO=us-docker.pkg.dev/${PROJECT_ID}/default

  • Run with hot reload and live logs:

    skaffold dev -p gke

  • Or just deploy to the default namespace with skaffold:

    skaffold run -p gke

5. Advanced Skaffold commands

5.1. Build and run with specific microservice(s)

skaffold dev -m <service1>,<service2>

5.2. Build and run microservices with a different Skaffold profile

# Using HPA (Horizontal Pod Autoscaler) profile
skaffold dev -p gke,gke-hpa

5.3. Skaffold profiles

By default, the Skaffold YAML contains the following pre-defined profiles ready to use.

  • gke - This is the default profile for local development, which will be activated automatically with the kubectl context set to the default cluster of this GCP project.
    • The corresponding kustomize YAML is in the ./components/<component_name>/kustomize/base folder.
  • gke-hpa - This is the profile for building and deploying to the Prod environment, e.g. to a customer's Prod environment. Adds Horizontal Pod Autoscaler (HPA)
    • The corresponding kustomize YAML is in the ./components/<component_name>/kustomize/hpa folder.

6. Debugging

6.1. Debugging locally (VS Code)

Mileage will vary - you may need to create a "Debug Current File" Debug configuration in VS Code, particularly if you are in a multi-folder Workspace.

6.1.1. Debugging components/common

By default, VS Code will use the Python interpreter you've selected with the Python extensions (CMD + SHIFT + P -> Select Interpreter) so clicking Debug and running without a configuration should work, so long as you have shifted the interpreter over and activated the "Common" VirtualEnv.

6.1.2. Debugging components/<component_name>

This should also just work, so long as you have selected the right interpreter, are in the microservice folder, and have entered your VirtualEnv.

Mileage will vary - you may need to create a "Debug Current File" Debug configuration in VS Code, particularly if you are in a multi-folder Workspace.

6.2. Debugging with Skaffold + Cloud Code (on GKE cluster)

NOTE: You don't need a VirtualEnv for this option.

First, install Cloud Code

  • From Command Palette (CMD + SHIFT + P): Cloud Code: Debug on Kubernetes
  • Select the root skaffold.yaml
  • Run All dependencies or the module that you want
  • Select a profile (dev for GKE)
  • Select context - the GKE cluster you're using
  • You will need to edit the launch.json file and add /src to the "sourceFileMap" field

When you're done, make sure to fully disconnect the debugger, so it removes the running services.

7. Testing

7.1. Unit testing

Unit tests make use of the firestore emulator. Tests currently assume that the emulator is running in the background, so you must launch the emulator before running tests. If the emulator us not running the test infrastructure will report this as an error.

  • Install NodeJS required by Firebase Emulator:

    Follow instructions on https://nodejs.org/en/download or https://formulae.brew.sh/formula/node

  • Install OpenJDK required by Firebase Emulator:

    Follow instructions on https://openjdk.org/install/ or https://formulae.brew.sh/formula/openjdk

  • Install Firebase CLI and emulator: We install firebase CLI with our own script, to pin the version of the emulator, as emulator updates have broken our tests in the past.

    utils/install_firebase.sh v13.1.0

    To install the latest version of firebase CLI and the emulators, run this command:

    curl -sL https://firebase.tools | bash
firebase setup:emulators:firestore

  • Launch emulator in background (should remain running until you restart your laptop)

    firebase emulators:start --only firestore --project fake-project &

  • Set up quota project:

    gcloud auth application-default set-quota-project $PROJECT_ID

  • Install Virtualenv and pip requirements

    # Start in the root folder
export BASE_DIR=$(pwd)
export PROJECT_ID=<your-dev-project-id>

# If you need to test the llm_service microservice:
cd components/llm_service
python3 -m venv .venv
source .venv/bin/activate
pip install -r ../common_ml/requirements.txt
pip install -r ../common_ml/requirements-test.txt
pip install -r requirements.txt
pip install -r requirements-test.txt

# If you need to test any other specific microservice, other than llm_service:
cd components/<component_name>
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -r requirements-test.txt

# If this component depends on the common folder:
pip install -r ../common/requirements.txt
pip install -r ../common/requirements-test.txt

  • Run unit tests locally for the entire codebase:

    cd components/<component_name>/src
PYTHONPATH=../../common/src python -m pytest

    Or run the unit tests with code coverage:

    cd components/<component_name>/src
PYTEST_ADDOPTS="--cache-clear --cov . " PYTHONPATH=../../common/src python -m pytest

  • Run unit tests locally for a particular test file

    cd components/<component_name>/src
PYTHONPATH=../../common/src python -m pytest path/to/target_test.py

  • Run unit tests with logging

    cd components/<component_name>/src
PYTHONPATH=../../common/src python -m pytest --log-cli-level=INFO path/to/target_test.py

7.2. Test filename convention and format

All unit test files follow the filename format:

  • Python:

    <original_filename>_test.py

  • Typescript/Javascript:

    <original_filename>.spec.ts
<original_filename>.spec.js

7.3. Code style and linter

Run linter locally:

python -m pylint $(git ls-files '*.py') --rcfile=$BASE_DIR/.pylintrc

8. Running Jupyter Notebooks

Complete the steps below in your virtual environment:

8.1. Installation

Install jupyterlab and notebook:

pip install jupyterlab notebook

8.2. Run Jupyter Notebook

Run Jupyter Notebook:

jupyter notebook

This will open a UI in your browser where you can navigate to the notebook and develop/run it.