Integrates GitLab Runner into DTaaS

deploy/services/gitlab/.env

@@ -1,2 +1,2 @@
-GITLAB_HOME='/Users/<Username>/DTaaS/deploy/services/gitlab'
+DTAAS_DIR='/Users/<username>/DTaaS'
 SERVER_DNS='foo.com'

deploy/services/gitlab/INTEGRATION.md

@@ -66,7 +66,7 @@ the following files:
    _deploy/config/client/env.local.js_.
 1. _deploy/docker/.env.local_ Add localpath and username.
 
-If the DTaaS application is hosted at <https://localhost>, then configure
+If the DTaaS application is hosted at <https://foo.com/>, then configure
 the following files:
 
 1. **DTaaS Client Authorization** token in

deploy/services/gitlab/README.md

@@ -29,7 +29,7 @@ Edit the `.env` file available in this directory to contain the following variab
 
 | Variable    | Example Value                                | Explanation                                                                                                                  |
 | :---------- | :------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------- |
-| GITLAB_HOME | '/home/Desktop/DTaaS/deploy/services/gitlab' | Full path to the DTaaS gitlab directory. This is an absolute path with no trailing slash.                                    |
+| DTAAS_DIR | '/Users/<username>/DTaaS' | Full path to the DTaaS directory. This is an absolute path with no trailing slash.                                    |
 | SERVER_DNS  | either `foo.com` or `localhost`                               | The server DNS, if you are deploying with a dedicated server. Remember not use _http(s)_ at the beginning of the DNS string. |
 
 **NOTE**: The DTaaS client uses the `react-oidc-context` node package, which

deploy/services/gitlab/compose.gitlab.yml

@@ -15,9 +15,9 @@ services:
         nginx['redirect_http_to_https'] = false
         letsencrypt['enable'] = false
     volumes:
-      - '${GITLAB_HOME}/config:/etc/gitlab'
-      - '${GITLAB_HOME}/logs:/var/log/gitlab'
-      - '${GITLAB_HOME}/data:/var/opt/gitlab'
+      - '${DTAAS_DIR}/deploy/services/gitlab/config:/etc/gitlab'
+      - '${DTAAS_DIR}/deploy/services/gitlab/logs:/var/log/gitlab'
+      - '${DTAAS_DIR}/deploy/services/gitlab/data:/var/opt/gitlab'
     shm_size: '256m'
     labels:
       - "traefik.enable=true"

deploy/services/runner/GITLAB-RUNNER.md

@@ -0,0 +1,108 @@
+# GitLab Runners
+
+GitLab Runners are agents that execute the tasks defined in GitLab CI/CD
+pipelines. A runner is connected to a GitLab instance (may be
+`https://gitlab.com` or a local instance such as `https://foo.com/gitlab`),
+and executes jobs for a single project, or any project within a GitLab group.
+
+This document outlines the steps needed to create a Docker container named
+`gitlab-runner` which will contain a single runner that will be responsible for
+the execution of Digital Twins. There are two installation scenarios:
+
+1. Localhost Installation - You are using the integrated runner locally with
+   a GitLab instance hosted at `https://localhost/gitlab`.
+2. Server Installation - You are using the integrated runner with a GitLab
+   instance hosted on a production server. This server may be a remote server
+   and not necessarily your own, and may have TLS enabled with a self-signed
+   certificate.
+
+Following the steps below sets up the integrated runner which can be used to
+execute digital twins from the Digital Twins Preview Page.
+
+## Obtaining A Registration Token
+
+First, we will obtain the token necessary to register the runner for the GitLab
+instance.
+
+1. On the __Admin__ dashboard, navigate to __CI / CD > Runners__.
+1. Select __New instance runner__.
+1. Under __Platform__, select the Linux operating system.
+1. Under __Tags__, add a `linux` tag.
+1. Select __Create runner__. A runner authentication token will be generated,
+   be sure to save it somewhere.
+
+You should see the following screen:
+
+![Runner Registration Screen](./runner-registration.png)
+
+## Configuring the Runner
+
+Depending on your installation scenario, the runner setup reads certain
+configurations settings:
+
+1. Localhost Installation - uses `deploy/docker/.env.local`
+1. Server Installation - uses `deploy/docker/.env.server`
+
+It is assumed you have set these up as part of the installation of the overall
+DTaaS software. If not, ensure these are properly set up.
+
+We need to register the runner with the GitLab instance so that they may
+communicate with each other. In this directory, the file `runner-config.toml`
+has the following template:
+
+```toml
+[[runners]]
+  name = "dtaas-runner-1"
+  url = "https://foo.com/gitlab/" # Edit this
+  token = "xxx" # Edit this
+  executor = "docker"
+  [runners.docker]
+    tls_verify = false
+    image = "ruby:2.7"
+    privileged = false
+    disable_entrypoint_overwrite = false
+    oom_kill_disable = false
+    volumes = ["/cache"]
+    network_mode = "host" # Disable this in secure contexts
+```
+
+1. Set the `url` variable to the URL of your GitLab instance.
+1. Set the `token` variable to the runner registration token you obtained earlier.
+1. If you are following the server installation scenario, remove the line
+   `network_mode = "host"`.
+
+## Start the GitLab Runner
+
+You may use the following commands to start and stop the `gitlab-runner`
+container respectively, depending on your installation scenario:
+
+1. Localhost Installation
+
+   ```bash
+   docker compose -f deploy/services/runner/compose.runner.local.yml --env-file deploy/docker/.env.local up -d
+   docker compose -f deploy/services/runner/compose.runner.local.yml --env-file deploy/docker/.env.local down
+   ```
+
+2. Server Installation
+
+   ```bash
+   docker compose -f deploy/services/runner/compose.runner.server.yml --env-file deploy/docker/.env.server up -d
+   docker compose -f deploy/services/runner/compose.runner.server.yml --env-file deploy/docker/.env.server down
+   ```
+
+Once the container starts, the runner within it will run automatically. You can
+tell if the runner is correctly configured by navigating to
+`CI/CD > Runners` on your Admin dashboard and seeing something like this:
+
+![Status indicator under Admin Area > Runners](./runner-activation.png)
+
+You will now have a GitLab runner ready to accept jobs for the GitLab instance.
+
+## Resource Allocation
+
+By default, the runner executor will pick up as many jobs as it can (limited
+by the number of threads on the machine). To limit the number of jobs that may
+be run concurrently, you can set the `limit` variable in `config.toml`.
+
+A list of advanced configuration options is provided on the
+[GitLab documentation page](https://docs.gitlab.com/runner/configuration/advanced-configuration.html).

deploy/services/runner/compose.runner.local.yml

@@ -0,0 +1,15 @@
+# Environment variables taken from deploy/docker/.env.local
+
+services:
+  gitlab-runner:
+    container_name: gitlab-runner
+    # Runner image version is independent of the gitlab-ce image version
+    image: gitlab/gitlab-runner:alpine-v17.5.3
+    volumes:
+      - "./runner-config.toml:/etc/gitlab-runner/config.toml:ro"
+      - "/var/run/docker.sock:/var/run/docker.sock:ro"
+      # Maps the self-signed certificate for localhost to the container
+      - "${DTAAS_DIR}/deploy/docker/certs/localhost/fullchain.pem:/etc/gitlab-runner/certs/localhost.crt:ro"
+
+    # To make https://localhost accessible from the container
+    network_mode: host

deploy/services/runner/compose.runner.server.yml

@@ -0,0 +1,13 @@
+# Environment variables taken from deploy/docker/.env.server
+
+services:
+  gitlab-runner:
+    container_name: gitlab-runner
+    # Runner image version is independent of the gitlab-ce image version
+    image: gitlab/gitlab-runner:alpine-v17.5.3
+    volumes:
+      - "./runner-config.toml:/etc/gitlab-runner/config.toml:ro"
+      - "/var/run/docker.sock:/var/run/docker.sock:ro"
+      # Maps the self-signed certificate for your server to the container
+      # Remove this if you are not using a self-signed certificate
+      - "${DTAAS_DIR}/deploy/docker/certs/${SERVER_DNS}/fullchain.pem:/etc/gitlab-runner/certs/${SERVER_DNS}.crt:ro"

deploy/services/runner/runner-config.toml

@@ -0,0 +1,13 @@
+[[runners]]
+  name = "dtaas-runner-1"
+  url = "https://foo.com/gitlab/" # Edit this
+  token = "xxx" # Edit this
+  executor = "docker"
+  [runners.docker]
+    tls_verify = false
+    image = "ruby:2.7"
+    privileged = false
+    disable_entrypoint_overwrite = false
+    oom_kill_disable = false
+    volumes = ["/cache"]
+    network_mode = "host" # Disable this in secure contexts

docs/admin/gitlab/runner.md

@@ -1,74 +1,139 @@
-# Gitlab Runner Integration
+# GitLab Runner Integration
 
-To properly use the __Digital Twins Preview Page__, you need to
-configure at least one project runner in your GitLab profile.
-The first step is to configure the CI/CD pipeline in gitlab project.
-The second step is to install the runner and integrate it
-with the selected gitlab project.
+This document outlines the steps needed to create a Docker container named
+`gitlab-runner` which will contain a single runner that will be responsible for
+the execution of Digital Twins. There are two installation scenarios:
 
-## Configure Gitlab Project
+1. __Localhost Installation__ - You are using the integrated runner locally with
+   a GitLab instance hosted at `https://localhost/gitlab`.
+2. __Server Installation__ - You are using the integrated runner with a GitLab
+   instance hosted on a production server. This server may be a remote server
+   and not necessarily your own, and may have TLS enabled with a self-signed
+   certificate.
 
-Follow the steps below:
+Following the steps below sets up the integrated runner which can be used to
+execute digital twins from the Digital Twins Preview Page.
 
-1. Navigate to the _DTaaS_ group and select the project named after your
-   GitLab username.
-1. In the project menu, go to Settings and select CI/CD.
-1. Expand the __Runners__ section and click on _New project runner_. Follow the
-   configuration instructions carefully:
-   - Add __linux__ as a tag during configuration.
-   - Click on _Create runner_. A runner authentication token is generated.
-     This token will be used later for registering a runner.
+## Runner Scopes
 
-## Runner
+A GitLab Runner can be configured for three different scopes:
 
-### Install Runner
+| Runner Scope    | Description |
+|-----------------|-------------|
+| Instance Runner | Available to all groups and projects in a GitLab instance. |
+| Group Runner    | Available to all projects and subgroups in a group. |
+| Project Runner  | Associated with one specific project. |
 
-A detailed guide on installation of
-[gitlab runners](https://docs.gitlab.com/runner/install/)
-on Linux OS is available on
-[gitlab website](https://docs.gitlab.com/runner/install/linux-repository.html).
-Remember to use `linux` as tag for the runner.
+We suggest creating instance runners as they are the most straightforward, but
+any type will work. More about these three types can be found on
+[the official GitLab documentation page](https://docs.gitlab.com/ee/ci/runners/runners_scope.html).
 
-### Register Runner
+## Obtaining A Registration Token
 
-Please see this [gitlab guide](https://docs.gitlab.com/runner/register/)
-on registering a runner.
+First, we will obtain the token necessary to register the runner for the GitLab
+instance. Open your GitLab instance (remote or local) and depending on your
+choice of runner scope, follow the steps given below:
 
-Remember to choose _docker_ as executor and _ruby:2.7_ as
-the default docker image.
+| Runner Scope    | Steps |
+|-----------------|-------|
+| Instance Runner |1. On the __Admin__ dashboard, navigate to __CI/CD > Runners__.<br>2. Select __New instance runner__.|
+| Group Runner    |1. On the __DTaaS__ group page, navigate to __Settings > CI/CD > Runners__.<br>2. Ensure the __Enable shared runners for this group__ option is enabled.<br>3. On the __DTaaS__ group page, navigate to __Build > Runners__.<br>4. Select __New group runner__.|
+| Project Runner  |1. On the __DTaaS__ group page, select the project named after your GitLab username.<br>2. Navigate to __Settings > CI/CD > Runners__.<br>3. Select __New project runner__.|
 
-```bash
-$sudo gitlab-runner register  --url https://gitlab.foo.com \
-  --token xxxxx
-```
+For any scope you have chosen, you will be directed to a page to create a
+runner:
 
-Or, you can also register the runner in non-interactive mode by running
+1. Under __Platform__, select the Linux operating system.
+1. Under __Tags__, add a `linux` tag.
+1. Select __Create runner__.
 
-```bash
-$sudo gitlab-runner register \
-  --non-interactive \
-  --url "https://gitlab.foo.com/" \
-  --token "xxxx" \
-  --executor "docker" \
-  --docker-image ruby:2.7 \
-  --description "docker-runner"
-```
+You should then see the following screen:
+
+![Runner Registration Screen](./runner-registration.png)
+
+Be sure to save the generated runner authentication token.
+
+## Configuring the Runner
+
+Depending on your installation scenario, the runner setup reads certain
+configurations settings:
 
-### Start Runner
+1. __Localhost Installation__ - uses `deploy/docker/.env.local`
+1. __Server Installation__ - uses `deploy/docker/.env.server`
 
-You can manually verify that the runner is available to pick up jobs by running
-the following command:
+It is assumed you have set these up as part of the installation of the overall
+DTaaS software. If not, ensure these are properly set up.
 
-```bash
-$sudo gitlab-runner run
+We need to register the runner with the GitLab instance so that they may
+communicate with each other. `deploy/services/runner/runner-config.toml`
+has the following template:
+
+```toml
+[[runners]]
+  name = "dtaas-runner-1"
+  url = "https://foo.com/gitlab/" # Edit this
+  token = "xxx" # Edit this
+  executor = "docker"
+  [runners.docker]
+    tls_verify = false
+    image = "ruby:2.7"
+    privileged = false
+    disable_entrypoint_overwrite = false
+    oom_kill_disable = false
+    volumes = ["/cache"]
+    network_mode = "host" # Disable this in secure contexts
 ```
 
-It can also be used to reactivate offline runners during subsequent sessions.
+1. Set the `url` variable to the URL of your GitLab instance.
+1. Set the `token` variable to the runner registration token you obtained earlier.
+1. If you are following the server installation scenario, remove the line
+   `network_mode = "host"`.
+
+## Start the GitLab Runner
+
+You may use the following commands to start and stop the `gitlab-runner`
+container respectively, depending on your installation scenario:
+
+1. Localhost Installation
+
+    ```bash
+    docker compose -f deploy/services/runner/compose.runner.local.yml --env-file deploy/docker/.env.local up -d
+    docker compose -f deploy/services/runner/compose.runner.local.yml --env-file deploy/docker/.env.local down
+    ```
+
+1. Server Installation
+
+    ```bash
+    docker compose -f deploy/services/runner/compose.runner.server.yml --env-file deploy/docker/.env.server up -d
+    docker compose -f deploy/services/runner/compose.runner.server.yml --env-file deploy/docker/.env.server down
+    ```
+
+Once the container starts, the runner within it will run automatically. You can
+tell if the runner is up and running by navigating to the page where
+you created the runner. For example, an Instance Runner would look like this:
+
+![Status indicator under Admin Area > Runners](./runner-activation.png)
+
+You will now have a GitLab runner ready to accept jobs for the GitLab instance.
 
 ## Pipeline Trigger Token
 
-You also need to create a
-[pipeline trigger token](https://archives.docs.gitlab.com/16.4/ee/ci/triggers/index.html).
-This token is required to trigger pipelines by using the API.
-You can create this token in your GitLab project's CI/CD settings under
-the *Pipeline trigger tokens* section.
+The Digital Twins Preview Page uses the GitLab API which requires a
+[Pipeline Trigger Token](https://docs.gitlab.com/ee/api/pipeline_triggers.html).
+Go to your project in the __DTaaS__ group and navigate to
+__Settings > CI/CD > Pipeline trigger tokens__. Add a new token with any
+description of your choice.
+
+![Creating a Pipeline Trigger Token](./pipeline-token.PNG)
+
+You can now use the Digital Twins Preview Page to manage and execute your
+digital twins.
+
+## Resource Allocation
+
+By default, the runner executor will pick up as many jobs as it can (limited
+by the number of threads on the machine). To limit the number of jobs that may
+be run concurrently, you can set the `limit` variable in `config.toml`.
+
+A list of advanced configuration options is provided on the
+[GitLab documentation page](https://docs.gitlab.com/runner/configuration/advanced-configuration.html).