From d3012463682042407196b81451c93713abe5d9f6 Mon Sep 17 00:00:00 2001 From: Uchechukwu Orji Date: Wed, 7 Aug 2024 14:32:21 +0100 Subject: [PATCH] improve documentation of scheduler settings --- dev/README.md | 53 +++++++++++++++++++++++------------------ dev/docker-compose.yaml | 4 ++-- 2 files changed, 32 insertions(+), 25 deletions(-) diff --git a/dev/README.md b/dev/README.md index 1445333..60450ef 100644 --- a/dev/README.md +++ b/dev/README.md @@ -29,10 +29,12 @@ This container is responsible for starting the task containers, setting up a wir ```sh docker compose up --build ``` + The above command only starts the `backend`, `postgresqldb` and `scheduler` services. To start the `worker-manager`, [you need to register a worker](#registering-a-worker). Afterwards, run: + ```sh docker compose --profile worker up --build ``` @@ -42,40 +44,43 @@ docker compose --profile worker up --build - [Get wireguard configuration files](#get-wireguard-configuration-files) - [Create a test worker](#create-a-test-worker) - **NOTE:** Unless otherwise stated, all files and commands are with respective to the `dev` directory. - ### Get wireguard configuration files -- Move the [Mullvad](https://mullvad.net/) configuration files into the `data` directory. - Configuration files should be named in the format `.conf` where - `` is the ISO 3166-1 alpha-2 code of the country. - On start, the `worker-manager` will select a random file to start up the `wireguard` container. - On receiving a task from the backend scheduler, it will search for a configuration file belonging to that task and reconfigure the wireguard container. - If the configuration file does not exist, it will skip the test. +- Move the [Mullvad](https://mullvad.net/) configuration files into the `data` directory. + Configuration files should be named in the format `.conf` where + `` is the ISO 3166-1 alpha-2 code of the country. + On start, the `worker-manager` will select a random file to start up the `wireguard` container. + On receiving a task from the backend scheduler, it will search for a configuration file belonging to that task and reconfigure the wireguard container. + If the configuration file does not exist, it will skip the test. ## create a test worker - Generate a private key: - ```sh - openssl genrsa -out id_rsa 2048 - ``` - The key name `id_rsa` is used as a bind mount in the compose file for the worker container. + + ```sh + openssl genrsa -out id_rsa 2048 + ``` + + The key name `id_rsa` is used as a bind mount in the compose file for the worker container. - Generate a public key for creating the worker on the database. - ```sh - openssl rsa -in id_rsa -pubout -out pubkey.pem - ``` + + ```sh + openssl rsa -in id_rsa -pubout -out pubkey.pem + ``` - Assuming the backend service is up (`docker compose up backend`), create a worker and assign them a list of countries to test for. - If no countries are provided, all available countries in the DB wiil be assigned to the worker. You can update the countries using `mirrors-qa-backend update-worker`. + If no countries are provided, all available countries in the DB wiil be assigned to the worker. You can update the countries using `mirrors-qa-backend update-worker`. + + In this example, we create a worker named `test` to test for mirrors in France, United States and Canada using the public key file + named `pubkey.pem`. + + ```sh + docker exec -i mirrors-qa-backend mirrors-qa-backend create-worker --countries=us,fr,ca test < ./pubkey.pem + ``` - In this example, we create a worker named `test` to test for mirrors in France, United States and Canada using the public key file - named `pubkey.pem`. - ```sh - docker exec -i mirrors-qa-backend mirrors-qa-backend create-worker --countries=us,fr,ca test < ./pubkey.pem - ``` - Set the name of the worker to the `WORKER_ID` variable in the `.env` file. - Start the services with the worker enabled using `docker compose --profile worker up --build` @@ -103,12 +108,14 @@ The `backend` code houses the `scheduler` and the `RESTful API`. The following e ### scheduler - `SCHEDULER_SLEEP_DURATION`: how long the scheduler should sleep after creating tests for idle workers -- `IDLE_WORKER_DURATION`: mark a worker as idle if it hasn't been seen within duration +- `IDLE_WORKER_DURATION`: duration since a worker was last seen to be considered "idle". Generally, + we want to set this to the same value as `SCHEDULER_SLEEP_DURATION` since a worker that hasn't + submitted tests throughout `SCHEDULER_SLEEP_DURATION` is idle. - `EXPIRE_TEST_DURATION`: expire tests whose results are still pending after duration ### worker-manager -- `SLEEP_DURATION`: how long the manager should sleep before polling the REST API +- `SLEEP_DURATION`: how long the manager should sleep before polling the REST API for pending tests. - `BACKEND_API_URI` - `DOCKER_SOCKET` - `PRIVATE_KEY_FILE`: name of private key file diff --git a/dev/docker-compose.yaml b/dev/docker-compose.yaml index 9137b01..c7c619a 100644 --- a/dev/docker-compose.yaml +++ b/dev/docker-compose.yaml @@ -44,8 +44,8 @@ services: container_name: mirrors-qa-scheduler environment: POSTGRES_URI: postgresql+psycopg://mirrors_qa:mirrors_qa@postgresdb:5432/mirrors_qa - IDLE_WORKER_DURATION: 10m - SCHEDULER_SLEEP_DURATION: 30m + IDLE_WORKER_DURATION: 15m + SCHEDULER_SLEEP_DURATION: 15m EXPIRE_TEST_DURATION: 1h DEBUG: true command: mirrors-qa-backend scheduler