Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy to production 2024-08-22 / 1 #54

Merged
merged 8 commits into from
Aug 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions .devcontainer/scripts/post_create.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,14 @@
#!/bin/bash
# this script install system dependencies for vectorizing and sets up conda

sudo apt-get update
sudo apt-get install -y build-essential python-dev libagg-dev libpotrace-dev pkg-config libffi-dev libcairo2-dev
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

# install system dependencies
bash "$SCRIPT_DIR"/../../scripts/install_system_dependencies.sh

# setup conda
conda env create -n dev -f envs/dev.yaml
conda init
conda init

# setup pre-commit
pre-commit install --install-hooks
176 changes: 176 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
### Python template
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
.idea/

# Other files
heroku.yml
docker-compose.yml
.pre-commit-config.yaml
.vscode/
.gitignore
.github/
.devcontainer/
.env.example
LICENSE
README.md
Dockerfile
.dockerignore
.git/
56 changes: 56 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
name: ci
on: push

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

permissions:
contents: read
id-token: write
packages: read

jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: "3.11"
- uses: pre-commit/[email protected]

test:
name: Test
runs-on: ubuntu-latest
env:
PORT: "5000"
S3_BUCKET: "-"
S3_TEST_BUCKET: "kittl-uploads-storage-staging"
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: 'eu-central-1'
role-to-assume: 'arn:aws:iam::339713006905:role/vectorizing-github-actions'
- name: Set up Python 3.11
uses: actions/setup-python@v2
with:
python-version: 3.11
- name: Set up system dependencies
run: |
bash scripts/install_system_dependencies.sh
- name: Set up `dev` conda environment
uses: conda-incubator/setup-miniconda@v2
with:
activate-environment: dev
environment-file: envs/dev.yaml
auto-activate-base: false
- name: Run tests
shell: bash -el {0}
run: |
conda activate dev
python -m pytest vectorizing/tests/test.py
43 changes: 8 additions & 35 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
@@ -1,42 +1,15 @@
name: deploy

on:
push:
branches:
- production
- main
-
jobs:
publish-docker-image:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v2
- name: Login to GitHub Container Registry
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: eu-central-1
- name: Publish the Docker image
run: |
docker build . --tag ghcr.io/kittl/vectorizing:${{ github.ref_name }}
docker push ghcr.io/kittl/vectorizing:${{ github.ref_name }}
- name: Deploy vectorizing in staging cluster
uses: kodermax/kubectl-aws-eks@master
env:
KUBE_CONFIG_DATA: ${{ secrets.KUBE_CONFIG }}
if: github.ref_name == 'main'
with:
args: rollout restart deployment/vectorizing --context staging
- name: Deploy vectorizing in production cluster
uses: kodermax/kubectl-aws-eks@master
env:
KUBE_CONFIG_DATA: ${{ secrets.KUBE_CONFIG }}
if: github.ref_name == 'production'
with:
args: rollout restart deployment/vectorizing --context production
deploy-py-server-argo:
name: Deploy PY server Argo
uses: ./.github/workflows/helpers-deploy-argo.yaml
secrets:
GH_PAT: ${{ secrets.GH_PAT }}
ECR_ROLE_ARN: ${{ github.ref == 'refs/heads/production' && secrets.PRODUCTION_ECR_ROLE_ARN || secrets.STAGING_ECR_ROLE_ARN }}
93 changes: 93 additions & 0 deletions .github/workflows/helpers-build-docker-image.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# This callable workflow builds and publish a package docker image

name: helpers-build-docker-image

on:
workflow_call:
inputs:
aws-region:
default: eu-central-1
description: The AWS region to use for ECR
required: false
type: string
docker_file_path:
description: >
"The path to the Dockerfile to use for building the image. For example:
`path/to/project/Dockerfile`"
type: string
required: true
extra-image-tags:
default: ''
description: Extra tags to use for the image (one per line)
required: false
type: string
platforms:
default: linux/amd64
description: The platforms to build for
required: false
type: string
secrets:
ECR_ROLE_ARN:
description: The ECR role ARN
required: true
outputs:
image-tag:
description: The docker image tag
value: ${{ jobs.build.outputs.image-tag }}

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

permissions:
contents: read
id-token: write
packages: read

jobs:
build:
name: Build docker image and push to ECR
runs-on: ubuntu-latest
outputs:
image-tag: ${{ steps.get-image-tag.outputs.image-tag }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ inputs.aws-region }}
role-to-assume: ${{ secrets.ECR_ROLE_ARN }}
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v2
- name: Get image tag output
shell: bash
id: get-image-tag
run: echo "image-tag=sha-${GITHUB_SHA:0:7}" >> $GITHUB_OUTPUT
- name: Docker meta tags
id: meta
uses: docker/metadata-action@v5
with:
flavor: |
latest=false
images: |
${{ format('{0}/{1}', steps.login-ecr.outputs.registry, github.event.repository.name) }}
tags: |
type=semver,pattern={{version}}
type=sha
${{ inputs.extra-image-tags }}
- name: Build and push
uses: docker/build-push-action@v6
with:
cache-from: type=gha
cache-to: type=gha,mode=max
context: .
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
platforms: ${{ inputs.platforms }}
provenance: false
push: true
file: ${{ inputs.docker_file_path }}
Loading
Loading