Executor testing

[wouterdb]

Validate that the new executor works, starting with a small setup (quickstart or lsm quickstart) and moving up to complete and complex models

tests

test	status iteration 1	status iteration 2
quickstart	done once (x4)
systemtenant	done (x2)	done
what if we have 150 agent with each 2 executors?	done, back to the drawing board

results

remark	follow up	status
too much memory consumption	https://inmanta.slack.com/archives/CKRF0C8R3/p1720173075389819	done
worker processes have not clear name (in process listing and logs)	#7999	todo
also rewrite process name of resourcetracker	will be difficult, as we have no entry point into it	WONTFIX
we are having zombies in the container	https://github.com/inmanta/inmanta-service-orchestrator/issues/504	TODO
shutdown failures	#7845	ongoing, waiting for more data

On the shutdown issues

We seem to have issue when shutting down the executors

1. At first, we had a hang, in the executor testcases,
2. this was addressed in Issue/investigate hanging #7822 which should make the crash more apparent
3. then it started to fail on test_executor_server and test_executor_server_dirty_shutdown https://jenkins.inmanta.com/job/core/job/inmanta-core/job/master/2960/console
4. attempt to reproduce with More exceptions on shutdown #7834
5. we suspect it was the process reaper, we disabled it on Revert child reaper #7845 is this fixes the instability, the issue is now the zombies in docker

On memory test

1. set up orchestrator using the NG container
2. use this model

import std
import std::testing

entity Loader:
    string name
end

index Loader(name)

implement Loader using loader

implementation loader for Loader:
    std::AgentConfig(agentname=name, autostart=true)
    for i in std::sequence(5):
        std::testing::NullResource(    
            agentname = self.name,
            name=f"{self.name} {i}"
        )
    end
end

for i in std::sequence(int(std::get_env("TEST_SIZE", "5"))):
    Loader(name=f"Loader nr {i}")
end

3. use this ugly script

inmanta-cli --host 172.30.0.3 project create -n test     
rm .inmanta
INM_ENV=$(inmanta-cli --host 172.30.0.3 environment show --format "{id}" scale_test )
inmanta-cli --host 172.30.0.3 environment delete $INM_ENV


inmanta-cli --host 172.30.0.3 environment create -p test -n scale_test --save
INM_ENV=$(inmanta-cli --host 172.30.0.3 environment show --format "{id}" scale_test )

for i in 5 10 15 20 50 100 150 200
do
  export  TEST_SIZE=$i
  inmanta export
  inmanta-cli monitor -e $INM_ENV
  memusage=$( docker container stats inmanta_orchestrator --no-stream --format=json | jq ".MemUsage" -r | cut -d "/" -f 1)
  echo "$TEST_SIZE $memusage" >>results.txt
done

executors	memory usage
5	542.4MiB
10	797.7MiB
15	1.027GiB
20	1.275GiB
50	2.751GiB
100	5.215GiB
150	7.689GiB
200	9.583GiB

results Iteration One (prior to rework for less memory usage)

remark	follow up	status
worker processes are hard to distinguish	#7746	resolved
partial venv creation can lead to isseus	#7748	ongoing
unclear log line: 2024-06-18 14:23:41,652 ERROR inmanta.loader Unable to load module inmanta_plugins.yang.gnmi.client	#7749	resolved
further improve process id: We could make it the concatenation of all known model versions (with a limit), collapsed into ranges? e.g. v1-10,12-20.		todo
code loading fail when loading multiple files	#7755	resolved
AttributeError: Can't pickle local object 'WeakSet.__init__.<locals>._remove unpickleable exceptions	#7756	resolved
we don't log the cause for loading excpetions on all code paths	#7785	review
also rewrite process name of forkserver	#7787	done
also rewrite process name of resourcetracker		will be difficult, as we have no entry point into it
we still have reloading issues	#7784	done
we are having zombies in the container	https://github.com/inmanta/inmanta-service-orchestrator/issues/504	re-opened, previous solution did not work
new agents hang sometimes	#7832	done
too much memory consumption	https://inmanta.slack.com/archives/CKRF0C8R3/p1720173075389819	ongoing
shutdown failures	#7845	ongoing, waiting for more data