From 9ab3b89836700f0a41679052d3fbfecc916cc573 Mon Sep 17 00:00:00 2001
From: Tomas Baca <klaxalk@gmail.com>
Date: Tue, 3 Dec 2024 15:36:14 +0100
Subject: [PATCH] more of docker

---
 .../30-docker/70-workspace-build.md           | 110 +++++++++++++++
 .../30-docker/70-workspace-caching.md         | 131 ------------------
 2 files changed, 110 insertions(+), 131 deletions(-)
 create mode 100644 docs/10-prerequisities/30-docker/70-workspace-build.md
 delete mode 100644 docs/10-prerequisities/30-docker/70-workspace-caching.md

diff --git a/docs/10-prerequisities/30-docker/70-workspace-build.md b/docs/10-prerequisities/30-docker/70-workspace-build.md
new file mode 100644
index 00000000..af6e9d7c
--- /dev/null
+++ b/docs/10-prerequisities/30-docker/70-workspace-build.md
@@ -0,0 +1,110 @@
+---
+title: ROS Workspace build
+pagination_label: Building and caching of ROS workspace in Docker
+description: How to build and cache a ROS workspace into a Docker image
+---
+
+# ROS Workspace caching with Docker
+
+This page describes the process of compiling **custom ROS packages** against dependencies from **base docker image**.
+This task is not trivial in general, however, we have prepared a set of scripts that will make the process straightforward.
+The following **assumption** apply for our situation:
+
+* We have a **base docker image**, which covers our dependencies, e.g., `ctumrs/mrs_uav_system:1.5.0`,
+* We need to **cache the build artifacts** (`./build`, `./devel`, `./.catkin_tools` within the workspace) for future rebuilds of the same sotware.
+* We need to transport the resulting compiled software to an offline machine (a.k.a. the **robot**),
+* The **robot** has the **base docker image** loaded in its docker daemon.
+
+Our process comprises of the following docker build stages:
+
+1. **Stage 1: build the workspace**
+    * Load the build cache (`./cache`) from the previous build
+    * Compile the catkin workspace
+2. **Stage 2: save the cache**
+    * Encapsulation of the whole workspace into a transport image (`alpine:latest`)
+    * Export of the image into a local directory (`./cache`)
+3. **Stage 3: export the transport image**
+    * Copy of the build neccessary build artefacts into a transport image (`alpine:latest`)
+    * Export of the transport image
+
+The compiled workspace is **transported** into the **robot** within an minimalistic `alpine`-based image, which makes it relatively small.
+The overhead of the transport image is only around 5 MB.
+On the other hand, if the workspace would be packed in a image based on the **base image**, the size would be offset by hundreds of megabytes.
+That is not a problem when the tranport occurs through a **docker registry**.
+However, since the [Portainer](/docs/prerequisities/portainer) interface makes the upload of **archived** images very simple, we prefer to bundle the whole image into a `.tar.gz` file.
+This approach complicates the deployment in one simple way: The workspace needs to **extracted** from the transport image and placed into a **shared volume** during runtime.
+
+## Pre-configured build pipeline
+
+A set of scripts that facilitate the build is provided at [ctu-mrs/mrs_docker](https://github.com/ctu-mrs/mrs_docker) under the `catkin_workspace_builder` folder.
+
+<Button label="🔗 ctu-mrs/mrs_docker repository" link="https://github.com/ctu-mrs/mrs_docker" block /><br />
+
+### Configuration
+
+* The sources of your ROS packages are supposed to be placed (or linked) into the `./src` folder.
+* The file `./common_vars.sh` configures the process by setting up the following environment variables:
+```bash
+# tag for the source image agains which the catkin workspace will be built
+export BASE_IMAGE=ctumrs/mrs_uav_system:1.5.0
+
+# tag for the 'transport' image used for packing the workspace
+export TRANSPORT_IMAGE=alpine:latest
+
+# tag for the resulting image in which the workspace will be packged
+export OUTPUT_IMAGE=catkin_workspace:1.0.0
+
+# location for the exported docker images using the `./export_image.sh` script
+export EXPORT_PATH=~/docker
+
+# CPU architecture for the output image
+export ARCH=amd64
+
+# path to the catkin workspace in the docker image
+export WORKSPACE_PATH=etc/docker/catkin_workspace
+
+# local path to the build cache
+export CACHE_PATH=cache
+```
+
+### Execution
+
+* Run `./build_image.sh` to compile the ROS packages into the image.
+* Run `./export_image.sh` to export the transport image with the workspace to the export destination.
+* Run `./clean.sh` to clean the build cache.
+
+## Using the workspace on a robot
+
+The ROS catkin workspace needs to be first extracted from the **transport image** and coppied into a shared docker volume.
+Then, a container can be started using the **base image** and the workspace can be sourced and used to start your packages.
+
+Here is an example of a minimalistic compose session.
+The full example can be found at [ctu-mrs/mrs_docker/compose/custom_workspace](https://github.com/ctu-mrs/mrs_docker/compose/custom_workspace)
+```yaml
+volumes:
+
+  catkin_workspace:
+
+services:
+
+  # will copy user's ROS catkin workspace from the 'transport' alpine image to a shared volume
+  copy_catkin_workspace:
+    image: <TRANSPORT_IMAGE>
+    volumes:
+      - catkin_workspace:/tmp/docker/catkin_workspace:consistent
+    tty: true
+    command: sh -c "rm -rvf /tmp/docker/catkin_workspace/*; mkdir -pv /tmp/docker/catkin_workspace; cp -rv /etc/docker/catkin_workspace/* /tmp/docker/catkin_workspace/"
+
+  # starts the HW API for connecting the MRS UAV System to PX4
+  custom_package:
+    image: <BASE_IMAGE>
+    network_mode: host
+    depends_on:
+      - copy_catkin_workspace
+    volumes:
+      - catkin_workspace:/etc/docker/catkin_workspace:consistent
+    env_file:
+      - ./stack.env
+    tty: true
+    command: bash -c "source /etc/docker/catkin_workspace/devel/setup.bash && waitForRos && roslaunch <my_package> <my_launchfile>.launch"
+```
diff --git a/docs/10-prerequisities/30-docker/70-workspace-caching.md b/docs/10-prerequisities/30-docker/70-workspace-caching.md
deleted file mode 100644
index dac4f526..00000000
--- a/docs/10-prerequisities/30-docker/70-workspace-caching.md
+++ /dev/null
@@ -1,131 +0,0 @@
----
-title: ROS Workspace caching
-pagination_label: Building and caching of ROS workspace in Docker
-description: How to build and chache a ROS workspace into a Docker image
----
-
-# ROS Workspace caching with Docker
-
-## Folder contents
-
-<details>
-<summary>Dockerfile</summary>
-<p>
-
-```dockerfile
-ARG WORKSPACE_PATH
-ARG BASE_IMAGE=ctumrs/mrs_uav_system:1.5.0_hailo_ai
-
-#############################################################################
-
-# FIRST STAGE: BUILD THE WORKSPACE
-FROM $BASE_IMAGE AS stage_build
-
-ARG WORKSPACE_PATH
-
-COPY ./cache/${WORKSPACE_PATH}/ /${WORKSPACE_PATH}/
-
-# create catkin workspace
-RUN [ ! -e /${WORKSPACE_PATH}/.catkin_tools ] && cd /${WORKSPACE_PATH} && catkin init && catkin config --profile reldeb --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=ON && catkin profile set reldeb && catkin config --extend /opt/ros/noetic || echo "[Docker]: catkin workspace already exists"
-
-# copy the sources from the local subfolder
-COPY src/ /${WORKSPACE_PATH}/src/
-
-RUN cd /${WORKSPACE_PATH} && catkin build -s
-
-#############################################################################
-
-# SECOND STAGE: COPY THE WORKSPACE TO A BLANK APLINE IMAGE
-FROM alpine AS stage_cache_workspace
-
-ARG WORKSPACE_PATH
-
-COPY --from=stage_build /${WORKSPACE_PATH} /${WORKSPACE_PATH}
-
-#############################################################################
-
-# THIRD STAGE: copy the workspace to a final blank ROS-equipped base image
-FROM alpine AS stage_finalization
-
-ARG WORKSPACE_PATH
-
-COPY ./cache/${WORKSPACE_PATH}/src /${WORKSPACE_PATH}/src
-COPY ./cache/${WORKSPACE_PATH}/.catkin_tools /${WORKSPACE_PATH}/.catkin_tools
-COPY ./cache/${WORKSPACE_PATH}/devel /${WORKSPACE_PATH}/devel
-
-CMD ["sh"]
-```
-
-</p>
-</details>
-<details>
-<summary>build_image.sh</summary>
-<p>
-
-```bash
-#!/bin/bash
-
-set -e
-
-trap 'last_command=$current_command; current_command=$BASH_COMMAND' DEBUG
-trap 'echo "$0: \"${last_command}\" command failed with exit code $?"' ERR
-
-# get the path to this script
-MY_PATH=`dirname "$0"`
-MY_PATH=`( cd "$MY_PATH" && pwd )`
-
-cd ${MY_PATH}
-
-## --------------------------------------------------------------
-## |                            setup                           |
-## --------------------------------------------------------------
-
-LOCAL_TAG=catkin_workspace:yolo
-
-ARCH=arm64 # robofly
-# ARCH=amd64
-
-source ./paths.sh
-
-## --------------------------------------------------------------
-## |                            build                           |
-## --------------------------------------------------------------
-
-# initialize the cache
-[ ! -e ${CACHE_PATH}/${WORKSPACE_PATH} ] && mkdir -p ./${CACHE_PATH}/${WORKSPACE_PATH}
-
-PASS_TO_DOCKER_BUILD="Dockerfile src ${CACHE_PATH}/${WORKSPACE_PATH}"
-
-docker buildx use default
-
-echo ""
-echo "$0: building the user's workspace"
-echo ""
-
-# this first build compiles the contents of "src" and storest the intermediate
-tar -czh $PASS_TO_DOCKER_BUILD 2>/dev/null | docker build - --no-cache --target stage_cache_workspace --output ./${CACHE_PATH} --build-arg WORKSPACE_PATH=${WORKSPACE_PATH} --file Dockerfile --platform=linux/$ARCH
-
-echo ""
-echo "$0: packing the workspace into a docker image"
-echo ""
-
-# this second build takes the resulting workspace and storest in in a final image
-# that can be deployed to a drone
-docker build . --no-cache --target stage_finalization --file Dockerfile --build-arg WORKSPACE_PATH=${WORKSPACE_PATH} --tag $LOCAL_TAG --platform=linux/$ARCH
-
-echo ""
-echo "$0: workspace was packed into '$LOCAL_TAG'"
-echo ""
-```
-
-</p>
-</details>
-<details>
-<summary>paths.sh</summary>
-<p>
-```bash
-export WORKSPACE_PATH=etc/docker/catkin_workspace
-export CACHE_PATH=cache
-```
-</p>
-</details>