Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor workflows to allow migration from ecr to ghcr and onto gihthub hosted runner #1398

Draft
wants to merge 33 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
577f479
add new workflows to replace old build for migration from ecr to ghcr
Emterry Nov 28, 2024
fc335f9
amend docker compose build
Emterry Nov 28, 2024
73dc2dd
amend for Image_Name
Emterry Nov 28, 2024
a0de990
debug
Emterry Nov 28, 2024
e43eedd
debug
Emterry Dec 3, 2024
47c36e2
test-change
Emterry Dec 3, 2024
79a0e1e
Merge branch 'main' into migrate-to-ghcr
Emterry Dec 3, 2024
84bccaa
remove test workflow
Emterry Dec 3, 2024
1493325
Merge branch 'migrate-to-ghcr' of github.com:ministryofjustice/analyt…
Emterry Dec 3, 2024
3808146
debug test
Emterry Dec 3, 2024
d75971c
change to latest tag
Emterry Dec 3, 2024
f95741d
debug
Emterry Dec 3, 2024
704b1d6
add build step
Emterry Dec 3, 2024
182923b
refactor add credentials to debug
Emterry Dec 4, 2024
4958252
test something
Emterry Dec 4, 2024
6fab6dc
push to ghcr
Emterry Dec 4, 2024
207151a
update apline base
Emterry Dec 5, 2024
6efcde5
trivy
Emterry Dec 5, 2024
e7cedf3
gitleaks
Emterry Dec 5, 2024
856b2a2
fix scan
Emterry Dec 5, 2024
fd74edb
Merge branch 'main' into migrate-to-ghcr
Emterry Dec 6, 2024
5b6c581
pr review suggestions
Emterry Dec 6, 2024
42d6421
Merge branch 'migrate-to-ghcr' of github.com:ministryofjustice/analyt…
Emterry Dec 6, 2024
cdfd11d
newlines
Emterry Dec 6, 2024
7da7bd8
newlines
Emterry Dec 6, 2024
588a8a2
Merge branch 'main' into migrate-to-ghcr
Emterry Dec 6, 2024
db82931
pr changes
Emterry Dec 10, 2024
679057a
Merge branch 'migrate-to-ghcr' of github.com:ministryofjustice/analyt…
Emterry Dec 10, 2024
5d14c59
pr changes and trivy
Emterry Dec 10, 2024
6c7b3e7
test remove dependancy
Emterry Dec 12, 2024
49b8a24
Merge branch 'main' into migrate-to-ghcr
Emterry Dec 12, 2024
cb8149c
add setuptool dependancy to resolve trivy
Emterry Dec 12, 2024
f256548
trivy
Emterry Dec 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* @ministryofjustice/analytical-platform
31 changes: 31 additions & 0 deletions .github/workflows/dependency-review.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
name: 🔍 Dependency Review

on:
pull_request:
branches:
- main
types:
- edited
- opened
- reopened
- synchronize

permissions: {}

jobs:
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Checkout
id: checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

- name: Dependency Review
id: dependency_review
uses: actions/dependency-review-action@3b139cfc5fae8b618d3eae3675e383bb1769c019 # v4.5.0
with:
fail-on-severity: critical
91 changes: 91 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
---
name: 🔖 Release

on:
push:
tags:
- "*"

permissions: {}

jobs:
release:
name: Release
runs-on: ubuntu-latest
permissions:
actions: read
attestations: write
contents: write
id-token: write
packages: write
steps:
- name: Checkout
id: checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

- name: Install cosign
id: install_cosign
uses: sigstore/cosign-installer@dc72c7d5c4d10cd6bcb8cf6e3fd625a9e5e537da # v3.7.0

- name: Log in to GitHub Container Registry
id: ghcr_login
uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3.3.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Build and Push
id: build_and_push
uses: docker/build-push-action@48aba3b46d1b1fec4febb7c5d0c644b249a11355 # v6.10.0
with:
push: true
tags: ghcr.io/${{ github.repository }}:${{ github.ref_name }}

- name: Sign
id: sign
shell: bash
run: |
cosign sign --yes ghcr.io/${{ github.repository }}@${{ steps.build_and_push.outputs.digest }}

- name: Generate SBOM
id: generate_sbom
uses: anchore/sbom-action@55dc4ee22412511ee8c3142cbea40418e6cec693 # v0.17.8
with:
image: ghcr.io/${{ github.repository }}:${{ github.ref_name }}
format: cyclonedx-json
output-file: "sbom.cyclonedx.json"

- name: Attest
uses: actions/attest-build-provenance@ef244123eb79f2f7a7e75d99086184180e6d0018 # v1.4.4
id: attest
with:
subject-name: ghcr.io/${{ github.repository }}
subject-digest: ${{ steps.build_and_push.outputs.digest }}
push-to-registry: true

- name: Attest SBOM
uses: actions/attest-sbom@5026d3663739160db546203eeaffa6aa1c51a4d6 # v1.4.1
id: attest_sbom
with:
subject-name: ghcr.io/${{ github.repository }}
subject-digest: ${{ steps.build_and_push.outputs.digest }}
sbom-path: sbom.cyclonedx.json
push-to-registry: true

- name: cosign Verify
id: cosign_verify
shell: bash
run: |
cosign verify \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com \
--certificate-identity=https://github.com/${{ github.workflow_ref }} \
ghcr.io/${{ github.repository }}@${{ steps.build_and_push.outputs.digest }}

- name: GitHub Attestation Verify
id: gh_attestation_verify
shell: bash
env:
GH_TOKEN: ${{ github.token }}
run: |
gh attestation verify oci://ghcr.io/${{ github.repository }}:${{ github.ref_name }} --repo ${{ github.repository }}
40 changes: 40 additions & 0 deletions .github/workflows/scan.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
---
name: 🩻 Scan

on:
pull_request:
branches:
- main

permissions: {}

jobs:
scan:
name: Scan
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Checkout
id: checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

- name: Build
id: build
shell: bash
env:
IMAGE_NAME: ghcr.io/${{ github.repository }}
IMAGE_TAG: ${{ github.sha }}
run: |
make build

- name: Scan
id: scan
uses: aquasecurity/trivy-action@18f2510ee396bbf400402947b394f2dd8c87dbb0 # v0.29.0
env:
TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db:2
TRIVY_JAVA_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-java-db:1
with:
image-ref: ghcr.io/${{ github.repository }}:${{ github.sha }}
severity: HIGH,CRITICAL
exit-code: 1
77 changes: 0 additions & 77 deletions .github/workflows/test-and-push-docker-image.yaml

This file was deleted.

35 changes: 35 additions & 0 deletions .github/workflows/test.yml
Emterry marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Test

on:
pull_request:
branches:
- main

permissions: {}

jobs:
test:
runs-on: ubuntu-latest
env:
LOGS_BUCKET_NAME: moj-analytics-s3-logs
Emterry marked this conversation as resolved.
Show resolved Hide resolved
permissions:
contents: read

steps:
- name: Checkout
id: checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

- name: Build image
id: build_image
run: make build
env:
NETWORK: host
IMAGE_TAG: ${{ github.sha }}

- name: Run Python tests
id: test
run: make clean && make test
env:
NETWORK: default
IMAGE_TAG: ${{ github.sha }}
6 changes: 6 additions & 0 deletions .trivyignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Helm
CVE-2024-34156 # stdlib - helm binary

# Python

CVE-2024-33663 # python-jose needs patching/replacing abandonware no fix
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ RUN /node_modules/.bin/jest

FROM public.ecr.aws/docker/library/python:3.12-alpine3.18 AS base

ARG HELM_VERSION=3.14.1
ARG HELM_VERSION=3.16.0
ARG HELM_TARBALL=helm-v${HELM_VERSION}-linux-amd64.tar.gz
ARG HELM_BASEURL=https://get.helm.sh

Expand All @@ -26,7 +26,7 @@ ENV DJANGO_SETTINGS_MODULE="controlpanel.settings" \
RUN addgroup -g 1000 controlpanel \
&& adduser -G controlpanel -u 1000 controlpanel -D

RUN apk update \
RUN apk update && apk upgrade \
&& apk add --no-cache \
postgresql-client \
wget \
Expand Down
32 changes: 17 additions & 15 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,48 +1,50 @@
REPOSITORY?=controlpanel
VIRTUAL_ENV ?= venv
BIN=${VIRTUAL_ENV}/bin
IMAGE_TAG ?= local
IMAGE_TAG ?= latest
DOCKER_BUILDKIT?=1
REGISTRY?=593291632749.dkr.ecr.eu-west-1.amazonaws.com
MAKEFLAGS += -j2
IMAGE_NAME ?= ghcr.io/ministryofjustice/analytics-platform-control-panel


include Makefile.local.mk
export


.PHONY: clean build help test test-python dev-up

clean:
docker-compose down --volumes --remove-orphans
docker compose down --volumes --remove-orphans

build:
@docker-compose build frontend
@docker compose build frontend

test-python: DJANGO_SETTINGS_MODULE=controlpanel.settings.test
test-python:
@echo
@echo "> Running Python Tests (In Docker)..."
@docker-compose run --rm -e KUBECONFIG=tests/kubeconfig \
frontend sh -c "pytest tests --color=yes"
@docker compose run --rm -e KUBECONFIG=tests/kubeconfig \
frontend sh -c "pytest -v --tb=line tests --color=yes"

## test: Run tests in Docker container
test: test-python

prepare-up:
@docker-compose up -d db
@docker-compose run --rm --no-deps frontend sh -c "do sleep 2;done"
@docker-compose up migration
@docker-compose run --rm --no-deps frontend sh -c "do sleep 2;done"
@docker compose up -d db
@docker compose run --rm --no-deps frontend sh -c "do sleep 2;done"
@docker compose up migration
@docker compose run --rm --no-deps frontend sh -c "do sleep 2;done"

up: prepare-up
@docker-compose up -d frontend
@docker-compose logs -f
@docker compose up -d frontend
@docker compose logs -f

enter:
docker-compose run --rm --no-deps --entrypoint sh worker
docker compose run --rm --no-deps --entrypoint sh worker
logs:
@docker-compose logs -f
@docker compose logs -f
push:
docker-compose push frontend
docker compose push frontend

help: Makefile
@echo
Expand Down
Loading
Loading