Merge pull request #77 from Project-OMOTES/54-update-to-newest-versio…

…n-of-omotes-system-to-keep-integration-test-support

54: Rename computation-engine to omotes-system, fix integration with new setup approach for integration_tests and fix issue in job_submitter that an error result still led to a succeeding test.

.gitmodules

@@ -1,3 +1,3 @@
-[submodule "computation_engine"]
-	path = computation-engine-at-orchestrator
-	url = [email protected]:Project-OMOTES/computation-engine.git
+[submodule "omotes-system"]
+	path = omotes-system-at-orchestrator
+	url = [email protected]:Project-OMOTES/omotes-system.git

README.md

@@ -11,7 +11,7 @@ The following directory structure is used:
 
 - `ci/`: Contains all CI & other development scripts to help standardize the development workflow
   for both Linux and Windows.
-- `computation-engine-at-orchestrator/`: Submodule link to the latest `computation-engine` release.
+- `omotes-system-at-orchestrator/`: Submodule link to the latest `omotes-system` release.
   Necessary for infrastructure that is necessary during the integration test.
 - `config/`: Contains orchestrator workflow definitions configuration. The `workflow_config.json`
   file will be overwritten by a volume mount when deploying via docker.
@@ -60,25 +60,25 @@ A typical development workflow would be:
 5. run `lint`, `test_unit`, and `typecheck` to check for code quality issues.
 
 One of the example development workflows can be that you need to run and develop the orchestrator
-locally (without docker) while having other computation-engine infrastructure
+locally (without docker) while having other omotes-system infrastructure
 (e.g. postgres, rabbitmq, workers, etc.) started by dockers. The steps could be:
 
-1. comment out the `orchestrator` parts in the `docker-compose.yml` file in the `computation-engine`
+1. comment out the `orchestrator` parts in the `docker-compose.yml` file in the `omotes-system`
    repository.
-2. start computation-engine infrastructure via running `setup` and `start` scripts in
-   the `computation-engine` repository.
+2. start omotes-system infrastructure via running `setup` and `start` scripts in
+   the `omotes-system` repository.
 3. run `run.sh` or `run_windows.sh` in the `orchestrator` repository to develop and interact with
-   computation-engine infrastructure.
+   omotes-system infrastructure.
 
 All these scripts are expected to run from the root of the repository
 
-## Working with computation-engine submodule
+## Working with omotes-system submodule
 
-The [computation-engine](https://github.com/Project-OMOTES/computation-engine/) is available
-as a submodule at `computation-engine-at-orchestrator`. The name of this path is chosen
-to make sure starting the `computation-engine` with `docker compose` uses the
-`computation-engine-at-orchestrator` project name instead of `computation-engine`. If a developer
-is both developing the `orchestrator` and non-submodule `computation-engine` otherwise the
+The [omotes-system](https://github.com/Project-OMOTES/omotes-system) is available
+as a submodule at `omotes-system-at-orchestrator`. The name of this path is chosen
+to make sure starting the `omotes-system` with `docker compose` uses the
+`omotes-system-at-orchestrator` project name instead of `omotes-system`. If a developer
+is both developing the `orchestrator` and non-submodule `omotes-system` otherwise the
 environments may conflict in docker.
 
 To make the submodule available after cloning this repository:

integration_test/README.md

@@ -3,6 +3,6 @@ Submits a large number of jobs from multiples processes/SDKs to check if all of
 succeed successfully.
 
 # Setup and run the test
-First, ensure that the `computation-engine-at-orchestrator` repository is available as a submodule.
+First, ensure that the `omotes-system-at-orchestrator` repository is available as a submodule.
 
 1. Run `./ci/test_integration.sh` to setup and start the test.

integration_test/ci/_config.sh

@@ -1,15 +1,16 @@
 #!/bin/bash
 
-export CURRENT_WORKDIR=$PWD
-COMPUTATION_ENGINE="../computation-engine-at-orchestrator"
-ENV_FILE="${CURRENT_WORKDIR}/.env.test"
-DOCKER_COMPOSE_FILE="${COMPUTATION_ENGINE}/docker-compose.yml"
-DOCKER_COMPOSE_OVERRIDE_FILE="./docker-compose.override.yml"
+export ORCHESTRATOR_TEST_DIR=$PWD
+export OMOTES_SYSTEM="../omotes-system-at-orchestrator"
+export ORCHESTRATOR_DIR="${ORCHESTRATOR_TEST_DIR}/../"
+export TEST_WORKER_DIR="${ORCHESTRATOR_TEST_DIR}/test_worker/"
+export INTEGRATION_TESTS_DIR="${ORCHESTRATOR_TEST_DIR}/integration_tests/"
+export ENV_FILE="${ORCHESTRATOR_TEST_DIR}/.env.test"
 
-export COMPOSE_PROJECT_NAME=omotes_orchestrator_integration_tests
+export DOCKER_COMPOSE_SETUP_TEST_OVERRIDE="${OMOTES_SYSTEM}/docker-compose.yml -f ${OMOTES_SYSTEM}/docker-compose.override.setup.yml -f ${ORCHESTRATOR_TEST_DIR}/docker-compose.override.yml"
+export DOCKER_COMPOSE_TEST_OVERRIDE="${OMOTES_SYSTEM}/docker-compose.yml -f ${ORCHESTRATOR_TEST_DIR}/docker-compose.override.yml"
 
-export ORCHESTRATOR_DIR="${CURRENT_WORKDIR}/../"
-export TEST_WORKER_DIR="${CURRENT_WORKDIR}/test_worker/"
-export INTEGRATION_TESTS_DIR="${CURRENT_WORKDIR}/integration_tests/"
+export COMPOSE_PROJECT_NAME=omotes_orchestrator_integration_tests
 
-echo "Using docker compose files: ${DOCKER_COMPOSE_FILE} ${DOCKER_COMPOSE_OVERRIDE_FILE}"
+echo "Using docker compose files for setup: ${DOCKER_COMPOSE_SETUP_TEST_OVERRIDE}"
+echo "Using docker compose files for test: ${DOCKER_COMPOSE_TEST_OVERRIDE}"

integration_test/ci/_setup.sh

@@ -2,10 +2,11 @@
 
 . ci/_config.sh
 
-cp ${COMPUTATION_ENGINE}/.env.template ${ENV_FILE}
+cp ${OMOTES_SYSTEM}/.env.template ${ENV_FILE}
 sed -i 's/LOG_LEVEL=[a-z]*/LOG_LEVEL=WARNING  /gi' ${ENV_FILE}
 
-docker compose -f ${DOCKER_COMPOSE_FILE} -f ${DOCKER_COMPOSE_OVERRIDE_FILE} --env-file ${ENV_FILE} down -v
+docker compose -f ${DOCKER_COMPOSE_TEST_OVERRIDE} --env-file ${ENV_FILE} down -v
 
-${COMPUTATION_ENGINE}/scripts/setup_orchestrator_postgres_db.sh ${ENV_FILE} ${DOCKER_COMPOSE_FILE}
-${COMPUTATION_ENGINE}/scripts/setup_rabbitmq.sh ${ENV_FILE} ${DOCKER_COMPOSE_FILE}
+echo "Setting up OMOTES system"
+${OMOTES_SYSTEM}/scripts/setup.sh "${ENV_FILE}" "${DOCKER_COMPOSE_SETUP_TEST_OVERRIDE}"
+echo "Setup for OMOTES system finished."

integration_test/ci/remove_test_integration.sh

@@ -2,4 +2,4 @@
 
 . ci/_config.sh
 
-docker compose -f ${DOCKER_COMPOSE_FILE} -f ${DOCKER_COMPOSE_OVERRIDE_FILE} --env-file ${ENV_FILE} down -v
+docker compose -f ${DOCKER_COMPOSE_TEST_OVERRIDE} --env-file ${ENV_FILE} down -v

integration_test/ci/run_only_integration.sh

@@ -2,4 +2,4 @@
 
 . ci/_config.sh
 
-docker compose -f ${DOCKER_COMPOSE_FILE} -f ${DOCKER_COMPOSE_OVERRIDE_FILE} --env-file ${ENV_FILE} up --build integration_tests
+docker compose -f ${DOCKER_COMPOSE_TEST_OVERRIDE} --env-file ${ENV_FILE} up --build integration_tests

integration_test/ci/start_without_tests.sh

@@ -2,4 +2,4 @@
 
 . ci/_setup.sh
 
-docker compose -f ${DOCKER_COMPOSE_FILE} -f ${DOCKER_COMPOSE_OVERRIDE_FILE} --env-file ${ENV_FILE} up --build rabbitmq omotes_influxdb orchestrator test_worker test_hard_crash_worker test_long_sleep_worker
+docker compose -f ${DOCKER_COMPOSE_TEST_OVERRIDE} --env-file ${ENV_FILE} up --build rabbitmq omotes_influxdb orchestrator test_worker test_hard_crash_worker test_long_sleep_worker

integration_test/ci/test_integration.sh

@@ -2,4 +2,5 @@
 
 . ci/_setup.sh
 
-docker compose -f ${DOCKER_COMPOSE_FILE} -f ${DOCKER_COMPOSE_OVERRIDE_FILE} --env-file ${ENV_FILE} up --build --attach-dependencies --abort-on-container-exit integration_tests
+echo "Starting integration test."
+docker compose -f ${DOCKER_COMPOSE_TEST_OVERRIDE} --env-file ${ENV_FILE} up --build --attach-dependencies --abort-on-container-exit integration_tests

integration_test/docker-compose.override.yml

@@ -1,13 +1,9 @@
-networks:
-  omotes:
-    external: true
-
 services:
   orchestrator:
     build: ${ORCHESTRATOR_DIR}
     image: !reset
     volumes:
-      - "${CURRENT_WORKDIR}/orchestrator_workflow_config.json:/config/workflow_config.json"
+      - "${ORCHESTRATOR_TEST_DIR}/orchestrator_workflow_config.json:/config/workflow_config.json"
 
   orchestrator_postgres_db:
     ports:

integration_test/integration_tests/requirements.txt

@@ -1,4 +1,4 @@
-omotes-sdk-python == 3.2.2
+omotes-sdk-python == 3.2.3
 python-dotenv ~= 1.0.0
 pytest
 psycopg[binary,pool] ~= 3.2.1

integration_test/integration_tests/src/integration_tests/job_submitter.py

@@ -36,6 +36,20 @@
 PROCESS_COUNT = 5
 TIMEOUT_IN_WHICH_ALL_JOBS_MUST_FINISH_PER_PROCESS_SECONDS = 60.0
 
+DUMMY_ESDL = """<?xml version='1.0' encoding='UTF-8'?>
+<esdl:EnergySystem xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:esdl="http://www.tno.nl/esdl" id="a7889fc1-a6b5-4467-9a0d-b24fa496e0d3" description="" esdlVersion="v2401" name="Dummy ESDL" version="2">
+  <instance xsi:type="esdl:Instance" id="5275d2de-585c-4d26-a518-95588b4e910a" name="Untitled Instance">
+    <area xsi:type="esdl:Area" name="Untitled Area" id="f07f600e-0b0b-4047-81d5-e3e37ed5c2c3">
+      <asset xsi:type="esdl:HeatingDemand" name="HeatingDemand_48e0" id="48e0b6eb-db00-472a-918f-5ec5b70c5b20">
+        <port xsi:type="esdl:InPort" name="In" id="9a346418-9244-4402-9006-766ad5fa8555"/>
+        <port xsi:type="esdl:OutPort" name="Out" id="dab0f764-a112-41c0-bb7d-64c51a10e7db"/>
+        <geometry xsi:type="esdl:Point" lat="52.18338560942813" lon="4.788015455836084" CRS="WGS84"/>
+      </asset>
+    </area>
+  </instance>
+</esdl:EnergySystem>
+"""
+
 
 class JobSubmitter:
     process_number: int
@@ -75,6 +89,9 @@ def handle_on_finished(self, job: Job, result: JobResult):
                 )
                 self.errors.append(error)
 
+            if result.result_type != JobResult.ResultType.SUCCEEDED:
+                self.errors.append(f"Job {job.id} did not succeed: {result.result_type}")
+
             all_counted = False
             if JOB_COUNT_PER_PROCESS == len(self.result_jobs):
                 LOG.debug("Received the expected amount of result!")
@@ -113,7 +130,7 @@ def run(self):
 
             for i in range(0, JOB_COUNT_PER_PROCESS):
                 job_ref = omotes_if.submit_job(
-                    esdl="input-esdl-value",
+                    esdl=DUMMY_ESDL,
                     params_dict={
                         "key1": "value1",
                         "key2": ["just", "a", "list", "with", "an", "integer", 3],

integration_test/test_worker/requirements.txt

@@ -1 +1 @@
- omotes-sdk-python == 3.2.2
+ omotes-sdk-python == 3.2.3

integration_test/test_worker/test_worker.py

@@ -40,7 +40,7 @@ def test_worker_task(
     elif WORKER_TYPE == "LONG_SLEEP":
         time.sleep(100)
     update_progress_handler(0.7, f"After log {WORKER_TYPE}")
-    return f"Hello from test worker {WORKER_TYPE}!"
+    return input_esdl
 
 
 if __name__ == "__main__":