build: turbo cache, generic build dockerfile, dependency caching in g…

…ithub actions (#1116)

Makes various build improvements

- utilises turbo repo remote cache in build and lint pipeline
- produces an automated Dockerfile.build which is capable of building
any node based target in the monorepo without specifying manually the
dependencies to bundle in the output
- updates README to explain Dockerfile.build
- utilises `npm i` cache in the Github Actions

.dockerignore

@@ -3,4 +3,5 @@ docs/
 node_modules/
 tests/
 .env
+Dockerfile.build
 out

.github/workflows/build-lint.yml

@@ -11,6 +11,9 @@ jobs:
     name: Build and Test
     timeout-minutes: 15
     runs-on: ubuntu-latest
+    env:
+      TURBO_TEAM: ${{ vars.TURBO_TEAM }}
+      TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
 
     steps:
       - name: Check out code
@@ -24,9 +27,33 @@ jobs:
           node-version: 20.9.0
           cache: 'npm'
 
+      - name: Setup node cache
+        id: node-cache
+        uses: actions/cache@v3
+        with:
+          path: "**/node_modules"
+          key: npm-${{ hashFiles('package-lock.json') }}-${{ hashFiles('package.json') }}
+          restore-keys: npm-
+
       - name: Install dependencies
-        run: npm install
+        if: steps.node-cache.outputs.cache-hit != 'true'
+        run: npm i
 
+      - name: Configure Turborepo Remote Cache
+        run: |
+          mkdir -p .turbo
+          if [ -n "${{ vars.TURBO_API_URL }}" ]; then
+            echo '{"apiurl": "${{ vars.TURBO_API_URL }}"}' > .turbo/config.json
+            echo "Turborepo config file created at .turbo/config.json"
+          else
+            echo "TURBO_API_URL not set. Skipping Turborepo cache configuration."
+          fi
+          if [ -n "$TURBO_TEAM" ] && [ -n "$TURBO_TOKEN" ]; then
+            echo "TURBO_TEAM and TURBO_TOKEN environment variables set"
+          else
+            echo "TURBO_TEAM and/or TURBO_TOKEN not set. Remote caching may not be available."
+          fi
+      
       - name: Lint
         run: npm run lint
 

.github/workflows/publish-conductor.yml

@@ -51,12 +51,21 @@ jobs:
             type=raw,value=latest,enable={{is_default_branch}}
             type=semver,pattern={{version}},enable=${{ github.event_name == 'release' || (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')) }}
 
-
+      # Build and push Docker image using docker/build-push-action
       - name: Build and push Docker image
         uses: docker/build-push-action@v4
         with:
+          # Set the build context to the root of the repository (for monorepo build)
           context: .
-          file: ./api/BuildDockerfile
+          # Specify the location of the Dockerfile
+          file: Dockerfile.build
+          # Push the image to the container registry
           push: true
+          # Use the tags generated by the docker/metadata-action
           tags: ${{ steps.meta.outputs.tags }}
+          # Use the labels generated by the docker/metadata-action
           labels: ${{ steps.meta.outputs.labels }}
+          # Specify build arguments to build the faims3 api project in the monorepo
+          build-args: |
+            PACKAGE_NAME=@faims3/api
+            NODE_RUN_DIR=api/build/src

Dockerfile.build

@@ -0,0 +1,93 @@
+# syntax=docker/dockerfile:1
+
+# BASED ON https://turbo.build/repo/docs/guides/tools/docker
+# AND https://github.com/ThaddeusJiang/turborepo-starter/blob/main/apps/api/Dockerfile
+
+# Use Node 20 base
+FROM node:20-alpine AS alpine
+
+# What is the @faims3/... name of the package to build
+ARG PACKAGE_NAME
+ENV PACKAGE_NAME=${PACKAGE_NAME}
+
+# What directory relative to the monorepo will the node build folder containing
+# index.js be built to. For example API -> api/build/src
+ARG NODE_RUN_DIR
+ENV NODE_RUN_DIR=${NODE_RUN_DIR}
+
+# install some dependencies
+RUN apk update && apk add --no-cache libc6-compat
+
+# create this layer to use to avoid repeatedly installing
+FROM alpine AS base
+
+# PRUNE TO GET DEPENDENCIES [pruner]
+# ===================================
+
+# Use turbo to prune package dependencies
+FROM base AS pruner
+
+# work in /app folder
+WORKDIR /app
+
+# install turbo global
+RUN npm install turbo@^2.0.11 -g
+
+# copy everything into pruner
+COPY . .
+
+# prune using turbo for only faims3 and it's dependencies. This produces an
+# /app/out folder which has /full (all needed deps) or /json (package*.json
+# only)
+RUN turbo prune ${PACKAGE_NAME} --docker
+
+
+# INSTALL PACKAGES [builder]
+# ============================
+
+# add lockfiles dependencies
+FROM base AS builder
+
+WORKDIR /app
+
+# Install dependencies
+COPY .gitignore .gitignore
+COPY --from=pruner /app/out/json/ .
+
+# install turbo global
+RUN npm install turbo@^2.0.11 -g
+RUN npm install
+
+
+# Build the project 
+COPY --from=pruner /app/out/full/ .
+COPY --from=pruner /app/out/json/ json
+
+# build packages using turbo
+RUN turbo run build --filter=${PACKAGE_NAME}
+
+# Package all necessary files to run app into pruner out
+COPY bundle.sh .
+RUN sh bundle.sh json . out && cp -R json/* /app/out/
+
+# PRODUCTION RUNNER [runner]
+# ==========================
+
+FROM alpine AS runner
+WORKDIR /app
+
+# Don't run production as root
+RUN addgroup --system --gid 1001 nodejs
+RUN adduser --system --uid 1001 faims
+USER faims
+
+# Copy all files which are required to run the app
+COPY --from=builder /app/out .
+
+# Run the node app
+WORKDIR /app/${NODE_RUN_DIR}
+
+CMD ["node", "index.js"]
+
+
+

README.md

@@ -39,7 +39,7 @@ source ~/.bashrc
 You can then setup Node v20 and activate it
 
 ```bash
-nvm install 20 
+nvm install 20
 nvm use 20
 ```
 
@@ -215,3 +215,52 @@ Run tests inside the conductor instance:
 
 docker compose exec conductor npm run test
 ```
+
+## Production docker builds
+
+This repo contains a `Dockerfile.build` which is designed to build and package a Node.js application in a multi-stage process, optimized for monorepo structures using Turborepo.
+
+### Stages:
+
+1. **Base**: Sets up the Alpine Node.js 20 environment with necessary dependencies.
+2. **Pruner**: Uses Turborepo to prune the monorepo, isolating the specified package and its dependencies.
+3. **Builder**: Installs dependencies, builds the project, and bundles necessary files.
+4. **Runner**: Creates a minimal production image to run the built application.
+
+### Usage:
+
+To build a package (e.g., `@faims3/api`) with the build output in a specific directory (e.g., `/api/build/src`):
+
+```bash
+docker build -f Dockerfile.build \
+             --build-arg PACKAGE_NAME=@faims3/api \
+             --build-arg NODE_RUN_DIR=api/build/src \
+             -t your-image-name .
+```
+
+### bundle.sh
+
+`bundle.sh` is a script which utilises the output of the `turbo prune <target> --docker` command. It leverages the existence of `package.json` files at directories to determine necessary build paths to copy when bundling a Node based Dockerimage. It currently copies the following folder
+
+- node_modules
+- build
+- dist
+- views (for the API specifically)
+
+`realpath` could not be used because `alpine` images do not support `--relative-to` meaning a custom function is included which finds relative paths.
+
+I would not recommend using this script for local work, it is predominately to help the Docker build be able to resolve dependencies without any manual intervention while still building a pruned output image.
+
+```bash
+# Helper script used for docker builds.
+
+# bundle.sh: A script to copy specific directories from a source to an output location
+# based on the presence of package.json files in a JSON dump directory.
+
+# Usage: ./bundle.sh <json_dump_path> <source_path> <output_path>
+#
+# Arguments:
+#   json_dump_path: Path to the directory containing package.json files
+#   source_path: Path to the source directory containing files to be copied
+#   output_path: Path to the output directory where files will be copied
+```

api/package.json

@@ -5,7 +5,7 @@
     "main": "build/src/index.js",
     "packageManager": "[email protected]",
     "engines": {
-        "node": "20.9.0"
+        "node": "^20.9.0"
     },
     "scripts": {
         "lint": "gts lint",

app/package.json

@@ -16,7 +16,7 @@
         ]
     },
     "engines": {
-        "node": "20.9.0"
+        "node": "^20.9.0"
     },
     "scripts": {
         "start": "vite",