Design Gurobi optimization worker #8

lfse-slafleur · 2023-11-09T15:17:07Z

Ask Martijn to lead the purchase order for license.

lfse-slafleur · 2024-01-18T09:03:22Z

Mark is going to start this up again. Sebastiaan will send the startup info to Mark. We need someone from the (TNO) business to tag along as TNO will need to buy a license for its mapeditor environment. We will ask TPG to tag along as well. Stephan is going to ask around to see whom from TPG will accompany this process.

Reason we will join: Gurobi has multiple license forms (standalone containers and each container has a license, embed Gurobi in one of our containers, use a central calculation server) and we need to see which ones we need and can support.

lfse-slafleur · 2024-09-30T13:14:58Z

We have evaluated Gurobi with a WSL license. In short, the license file allows a number of instances so this file may be shared by multiple containers. Also, overlap between usages (e.g. rolling updates, cancelling one task and starting a new one but the first session will overlap the next session) is allowed for a short time to. This solves any issues we may have for short overlaps.

Main tasks that are still necessary to fit Gurobi in our architecture is:

Finalize the code to integrate Gurobi into the optimizer worker.
Allow each worker instance to handle multiple types of tasks so we can have a set of 'Gurobi' optimizer workers which is limited by the amount of licenses available but each worker can do any optimization task. Otherwise we would need worker types * worker per type == # of licenses which doesn't scale for expensive licenses. <-- This requires design work. Consequence is that for Gurobi workers we cannot scale per worker type as we are able to do currently with opensource solvers.

lfse-slafleur · 2024-10-02T11:30:18Z

Proposed design to handle multiple types of tasks at a single worker:

At the internal SDK:

Extend https://github.com/Project-OMOTES/omotes-sdk-python/blob/main/src/omotes_sdk/internal/worker/worker.py to utilize a dict of workflow-types (string id) and a function to handle the task.
Workers can still only handle a single task at a time.
Leverage Celery to connect to multiple 'task queue's' at once.

At the orchestrator:

No changes at orchestrator necessary.

At the optimizer worker:

Luckily the Docker container used by optimizer-worker with HIGHs as the configured solver may be used also by the optimizer-worker with Gurobi as the configured solver. (see also https://github.com/Project-OMOTES/optimizer-worker/pull/75/files)
Use an env var called OPTIMIZER_SOLVER= with the options highs and gurobi at optimizer worker to be able to spin it up with either solver.
~~Also, extend the GROW_TASK_TYPE= env var with an option all to listen to all workflows (with the configured solver type).~~ <-- See below. Also include solver type in OPTIMIZER_SOLVER

At omotes system:

Change https://github.com/Project-OMOTES/omotes-system/blob/main/config/workflow_config_nwn.json by adding the '-highs' postfix as the solver for the existing optimizer workflows and adding another set with gurobi as the solver. (similarly update https://github.com/Project-OMOTES/orchestrator/blob/main/config/workflow_config.json)
This allows us to add the 'optimizer-gurobi-worker' with GROW_TASK_TYPE="all" and OPTIMIZER_SOLVER="gurobi" at https://github.com/Project-OMOTES/omotes-system/blob/main/docker-compose.yml . This worker will also require a gurobi.lic file in case the Gurobi WSL approach is used.

lfse-slafleur · 2024-10-02T12:11:52Z

@MichielTukker Could you provide a review of this proposal? Thanks!

MichielTukker · 2024-10-04T07:09:07Z

As discussed offline. While it's a change in the worker function (from 1 workflow per worker to possibly multiple workflows per worker), this allows the party hosting the omotes backend to provide all Gurobi workflows with the minimum number of Gurobi licenses.

1 suggestion: I think the OPTIMIZER_SOLVER= env var is not necessary, as long as the worker supports reading multiple workflow names from GROW_TASK_TYPE= (something like a ;-separated string). This prevents introducing another configuration variable. It also makes the configuration of the task-types more explicit

lfse-slafleur · 2024-10-09T07:47:21Z

As discussed offline. While it's a change in the worker function (from 1 workflow per worker to possibly multiple workflows per worker), this allows the party hosting the omotes backend to provide all Gurobi workflows with the minimum number of Gurobi licenses.

1 suggestion: I think the OPTIMIZER_SOLVER= env var is not necessary, as long as the worker supports reading multiple workflow names from GROW_TASK_TYPE= (something like a ;-separated string). This prevents introducing another configuration variable. It also makes the configuration of the task-types more explicit

Good idea! Makes it more flexible as well as you could potentially have both highs & gurobi workflows in the same workflow (although it would be nonsensical in our situation) and it makes it simpler. This would be a pattern that is reusable across other workers as well.

lfse-slafleur · 2024-10-09T07:53:31Z

@MichielTukker Notes from Gurobi eval including the logs you requested (license expired & license missing). RuntimeError is used for all Gurobi exceptions caught/generated by Casadi in Python. We are going to need to parse the message partially to figure out which error happened...

How to get it to work

Change the Dockerfile to download Gurobi, set the necessary env vars for both Gurobi (e.g. GUROBI_HOME, GRB_LICENSE_FILE and update PATH and LD_LIBRARY_PATH) & Casadi (e.g. GUROBI_VERSION and it hooks into the Gurobi-specific env vars)
Ensure the worker uses the mesido Gurobi version of each workflow problem e.g. EndScenarioSizingDiscountedStagedGurobi instead of EndScenarioSizingDiscountedStagedHIGHS
Ensure the license is installed in the location specified by GRB_LICENSE_FILE
Download gurobi (not the ARM version, but the x86_64 version for linux) from https://www.gurobi.com/downloads/gurobi-software/

Test performed with:

Casadi 3.6.3
Gurobi 11.0.3
Mesido 0.1.7

Questions:

How can we trigger a 'Failed to create GUROBI environment. Flag: 10030:Too many sessions, 3 active sessions for a baseline of 1' on purpose for testing code?
Gurobi can reuse tokens across cancellations.

Running too many instances

Gurobi appears to provide a LOT of leeway, where it would allow a single license to utilize multiple sessions at once.
Nevertheless, been able to break it once (and once only!):

Licensing sessions are 2xlicensed amount + 1 (so use 4 instances to break it!) and 30 minutes for 2x licensed amount.

Support for centralized calculation server.

grow_worker-1  | 2024-09-25 12:07:51,066 [rtctools][MainThread][util.py:125][ERROR]: .../casadi/core/function_internal.cpp:153: Error calling GurobiInterface::finalize for 'mysolver':
grow_worker-1  | .../casadi/interfaces/gurobi/gurobi_interface.cpp:159: Assertion "!flag && m->env" failed:
grow_worker-1  | Failed to create GUROBI environment. Flag: 10030:Too many sessions, 3 active sessions for a baseline of 1
grow_worker-1  | CasADi - 2024-09-25 12:07:40 WARNING("CasADi was not compiled with WITH_OPENMP=ON. Falling back to serial evaluation.") [.../casadi/core/map.cpp:403]
grow_worker-1  | 2024-09-25 12:07:51,114 [omotes_sdk_internal][MainThread][worker.py:191][ERROR]: Failure detected for job fc92a8d3-b7be-46f6-a81d-13562f6c52e5 celery task d5709ac8-b695-43af-aa6c-94ee8595680e
grow_worker-1  | 2024-09-25 12:07:51,114 [celery.app.trace][MainThread][trace.py:270][ERROR]: Task grow_optimizer_default[d5709ac8-b695-43af-aa6c-94ee8595680e] raised unexpected: RuntimeError('.../casadi/core/function_internal.cpp:153: Error calling GurobiInterface::finalize for \'mysolver\':\n.../casadi/interfaces/gurobi/gurobi_interface.cpp:159: Assertion "!flag && m->env" failed:\nFailed to create GUROBI environment. Flag: 10030:Too many sessions, 3 active sessions for a baseline of 1')
grow_worker-1  | multiprocessing.pool.RemoteTraceback: 
grow_worker-1  | """
grow_worker-1  | Traceback (most recent call last):
grow_worker-1  |   File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 125, in worker
grow_worker-1  |     result = (True, func(*args, **kwds))
grow_worker-1  |   File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
grow_worker-1  |     return list(map(*args))
grow_worker-1  |   File "/app/grow_worker/worker.py", line 65, in run_mesido
grow_worker-1  |     solution: GROWProblem = mesido_func(
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/mesido/workflows/grow_workflow.py", line 652, in run_end_scenario_sizing
grow_worker-1  |     solution = run_optimization_problem(
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/rtctools/util.py", line 122, in run_optimization_problem
grow_worker-1  |     prob.optimize()
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/rtctools/optimization/single_pass_goal_programming_mixin.py", line 366, in optimize
grow_worker-1  |     success = super().optimize(
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/rtctools/optimization/optimization_problem.py", line 144, in optimize
grow_worker-1  |     solver = casadi_solver('nlp', my_solver, nlp, nlpsol_options)
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/rtctools/optimization/single_pass_goal_programming_mixin.py", line 610, in __call__
grow_worker-1  |     return Solver()
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/rtctools/optimization/single_pass_goal_programming_mixin.py", line 580, in __init__
grow_worker-1  |     self._solver = ca.conic("mysolver",
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/casadi/casadi.py", line 26898, in conic
grow_worker-1  |     return _casadi.conic(*args)
grow_worker-1  | RuntimeError: .../casadi/core/function_internal.cpp:153: Error calling GurobiInterface::finalize for 'mysolver':
grow_worker-1  | .../casadi/interfaces/gurobi/gurobi_interface.cpp:159: Assertion "!flag && m->env" failed:
grow_worker-1  | Failed to create GUROBI environment. Flag: 10030:Too many sessions, 3 active sessions for a baseline of 1
grow_worker-1  | """
grow_worker-1  | 
grow_worker-1  | The above exception was the direct cause of the following exception:
grow_worker-1  | 
grow_worker-1  | Traceback (most recent call last):
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 477, in trace_task
grow_worker-1  |     R = retval = fun(*args, **kwargs)
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 760, in __protected_call__
grow_worker-1  |     return self.run(*args, **kwargs)
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/omotes_sdk/internal/worker/worker.py", line 212, in wrapped_worker_task
grow_worker-1  |     task.output_esdl = WORKER_TASK_FUNCTION(input_esdl, params_dict, task_util.update_progress)
grow_worker-1  |   File "/app/grow_worker/worker.py", line 142, in grow_worker_task
grow_worker-1  |     output_esdl = pool.map(run_mesido, [input_esdl])[0]
grow_worker-1  |   File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 364, in map
grow_worker-1  |     return self._map_async(func, iterable, mapstar, chunksize).get()
grow_worker-1  |   File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 771, in get
grow_worker-1  |     raise self._value
grow_worker-1  | RuntimeError: .../casadi/core/function_internal.cpp:153: Error calling GurobiInterface::finalize for 'mysolver':
grow_worker-1  | .../casadi/interfaces/gurobi/gurobi_interface.cpp:159: Assertion "!flag && m->env" failed:
grow_worker-1  | Failed to create GUROBI environment. Flag: 10030:Too many sessions, 3 active sessions for a baseline of 1
grow_worker-1  | 2024-09-25 12:07:51,115 [omotes_sdk_internal][MainThread][worker.py:140][INFO]: Job fc92a8d3-b7be-46f6-a81d-13562f6c52e5 (celery task id d5709ac8-b695-43af-aa6c-94ee8595680e) failed.
grow_worker-1  | 2024-09-25 12:07:51,115 [omotes_sdk_internal][MainThread][worker.py:158][DEBUG]: Sending result for job fc92a8d3-b7be-46f6-a81d-13562f6c52e5

No license installed

2024-10-09 09:32:47,617 [rtctools][MainThread][util.py:125][ERROR]: .../casadi/core/function_internal.cpp:153: Error calling GurobiInterface::finalize for 'mysolver':
.../casadi/interfaces/gurobi/gurobi_interface.cpp:159: Assertion "!flag && m->env" failed:
Failed to create GUROBI environment. Flag: 10009:Unable to open Gurobi license file '/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/gurobi.lic'
2024-10-09 09:32:47,672 [omotes_sdk_internal][MainThread][worker.py:191][ERROR]: Failure detected for job 22d0ee98-9dc6-47bd-8fdb-e08c3b6b54ca celery task 6fc78cba-ad78-451f-9938-03e52d4a9d83
2024-10-09 09:32:47,673 [celery.app.trace][MainThread][trace.py:270][ERROR]: Task grow_optimizer_default[6fc78cba-ad78-451f-9938-03e52d4a9d83] raised unexpected: RuntimeError('.../casadi/core/function_internal.cpp:153: Error calling GurobiInterface::finalize for \'mysolver\':\n.../casadi/interfaces/gurobi/gurobi_interface.cpp:159: Assertion "!flag && m->env" failed:\nFailed to create GUROBI environment. Flag: 10009:Unable to open Gurobi license file \'/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/gurobi.lic\'')
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/src/grow_worker/worker.py", line 63, in run_mesido
    solution: GROWProblem = mesido_func(
  File "/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/.venv/lib64/python3.8/site-packages/mesido/workflows/grow_workflow.py", line 652, in run_end_scenario_sizing
    solution = run_optimization_problem(
  File "/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/.venv/lib64/python3.8/site-packages/rtctools/util.py", line 122, in run_optimization_problem
    prob.optimize()
  File "/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/.venv/lib64/python3.8/site-packages/rtctools/optimization/single_pass_goal_programming_mixin.py", line 366, in optimize
    success = super().optimize(
  File "/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/.venv/lib64/python3.8/site-packages/rtctools/optimization/optimization_problem.py", line 144, in optimize
    solver = casadi_solver('nlp', my_solver, nlp, nlpsol_options)
  File "/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/.venv/lib64/python3.8/site-packages/rtctools/optimization/single_pass_goal_programming_mixin.py", line 610, in __call__
    return Solver()
  File "/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/.venv/lib64/python3.8/site-packages/rtctools/optimization/single_pass_goal_programming_mixin.py", line 580, in __init__
    self._solver = ca.conic("mysolver",
  File "/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/.venv/lib64/python3.8/site-packages/casadi/casadi.py", line 26898, in conic
    return _casadi.conic(*args)
RuntimeError: .../casadi/core/function_internal.cpp:153: Error calling GurobiInterface::finalize for 'mysolver':
.../casadi/interfaces/gurobi/gurobi_interface.cpp:159: Assertion "!flag && m->env" failed:
Failed to create GUROBI environment. Flag: 10009:Unable to open Gurobi license file '/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/gurobi.lic'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/.venv/lib64/python3.8/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/.venv/lib64/python3.8/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/.venv/lib64/python3.8/site-packages/omotes_sdk/internal/worker/worker.py", line 212, in wrapped_worker_task
    task.output_esdl = WORKER_TASK_FUNCTION(input_esdl, params_dict, task_util.update_progress)
  File "/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/src/grow_worker/worker.py", line 140, in grow_worker_task
    output_esdl = pool.map(run_mesido, [input_esdl])[0]
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib64/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
RuntimeError: .../casadi/core/function_internal.cpp:153: Error calling GurobiInterface::finalize for 'mysolver':
.../casadi/interfaces/gurobi/gurobi_interface.cpp:159: Assertion "!flag && m->env" failed:
Failed to create GUROBI environment. Flag: 10009:Unable to open Gurobi license file '/home/sebastiaan/repositories/nieuwe-warmte-nu/optimizer-worker/gurobi.lic'

Expired license installed

grow_worker-1  | Set parameter WLSAccessID
grow_worker-1  | Set parameter WLSSecret
grow_worker-1  | Set parameter LicenseID to value 2559744
grow_worker-1  | 2024-10-09 07:38:08,965 [rtctools][MainThread][util.py:125][ERROR]: .../casadi/core/function_internal.cpp:153: Error calling GurobiInterface::finalize for 'mysolver':
grow_worker-1  | .../casadi/interfaces/gurobi/gurobi_interface.cpp:159: Assertion "!flag && m->env" failed:
grow_worker-1  | Failed to create GUROBI environment. Flag: 10009:License 2559744 has expired
grow_worker-1  | CasADi - 2024-10-09 07:38:04 WARNING("CasADi was not compiled with WITH_OPENMP=ON. Falling back to serial evaluation.") [.../casadi/core/map.cpp:403]
grow_worker-1  | 2024-10-09 07:38:09,033 [omotes_sdk_internal][MainThread][worker.py:191][ERROR]: Failure detected for job 383ad7bc-6dbb-44d8-b6fe-75fc74e5f1ac celery task 5135d9db-8794-4d98-b103-d156b66f1e79
grow_worker-1  | 2024-10-09 07:38:09,033 [celery.app.trace][MainThread][trace.py:270][ERROR]: Task grow_optimizer_default[5135d9db-8794-4d98-b103-d156b66f1e79] raised unexpected: RuntimeError('.../casadi/core/function_internal.cpp:153: Error calling GurobiInterface::finalize for \'mysolver\':\n.../casadi/interfaces/gurobi/gurobi_interface.cpp:159: Assertion "!flag && m->env" failed:\nFailed to create GUROBI environment. Flag: 10009:License 2559744 has expired')
grow_worker-1  | multiprocessing.pool.RemoteTraceback: 
grow_worker-1  | """
grow_worker-1  | Traceback (most recent call last):
grow_worker-1  |   File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 125, in worker
grow_worker-1  |     result = (True, func(*args, **kwds))
grow_worker-1  |   File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
grow_worker-1  |     return list(map(*args))
grow_worker-1  |   File "/app/grow_worker/worker.py", line 65, in run_mesido
grow_worker-1  |     solution: GROWProblem = mesido_func(
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/mesido/workflows/grow_workflow.py", line 652, in run_end_scenario_sizing
grow_worker-1  |     solution = run_optimization_problem(
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/rtctools/util.py", line 122, in run_optimization_problem
grow_worker-1  |     prob.optimize()
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/rtctools/optimization/single_pass_goal_programming_mixin.py", line 366, in optimize
grow_worker-1  |     success = super().optimize(
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/rtctools/optimization/optimization_problem.py", line 144, in optimize
grow_worker-1  |     solver = casadi_solver('nlp', my_solver, nlp, nlpsol_options)
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/rtctools/optimization/single_pass_goal_programming_mixin.py", line 610, in __call__
grow_worker-1  |     return Solver()
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/rtctools/optimization/single_pass_goal_programming_mixin.py", line 580, in __init__
grow_worker-1  |     self._solver = ca.conic("mysolver",
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/casadi/casadi.py", line 26898, in conic
grow_worker-1  |     return _casadi.conic(*args)
grow_worker-1  | RuntimeError: .../casadi/core/function_internal.cpp:153: Error calling GurobiInterface::finalize for 'mysolver':
grow_worker-1  | .../casadi/interfaces/gurobi/gurobi_interface.cpp:159: Assertion "!flag && m->env" failed:
grow_worker-1  | Failed to create GUROBI environment. Flag: 10009:License 2559744 has expired
grow_worker-1  | """
grow_worker-1  | 
grow_worker-1  | The above exception was the direct cause of the following exception:
grow_worker-1  | 
grow_worker-1  | Traceback (most recent call last):
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 477, in trace_task
grow_worker-1  |     R = retval = fun(*args, **kwargs)
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 760, in __protected_call__
grow_worker-1  |     return self.run(*args, **kwargs)
grow_worker-1  |   File "/usr/local/lib/python3.8/site-packages/omotes_sdk/internal/worker/worker.py", line 212, in wrapped_worker_task
grow_worker-1  |     task.output_esdl = WORKER_TASK_FUNCTION(input_esdl, params_dict, task_util.update_progress)
grow_worker-1  |   File "/app/grow_worker/worker.py", line 142, in grow_worker_task
grow_worker-1  |     output_esdl = pool.map(run_mesido, [input_esdl])[0]
grow_worker-1  |   File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 364, in map
grow_worker-1  |     return self._map_async(func, iterable, mapstar, chunksize).get()
grow_worker-1  |   File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 771, in get
grow_worker-1  |     raise self._value
grow_worker-1  | RuntimeError: .../casadi/core/function_internal.cpp:153: Error calling GurobiInterface::finalize for 'mysolver':
grow_worker-1  | .../casadi/interfaces/gurobi/gurobi_interface.cpp:159: Assertion "!flag && m->env" failed:
grow_worker-1  | Failed to create GUROBI environment. Flag: 10009:License 2559744 has expired

Resource usage comparison

PoC tutorial (small ESDL)

### Gurobi
Runtime: 19.3 seconds
CPU: Always utilizes a single core fully
Memory usage: Peak 0.187 GB

### HIGHS
Runtime: 19.1 seconds
CPU: Always utilizes a single core fully
Memory usage: Peak 0.187 GB

Ronald (medium ESDL)

### Gurobi
Runtime: 1 minute and 37 seconds
CPU: Always utilizes a single core fully
Memory usage: Peak: 0.593 GB

### HIGHS
Runtime: 2 minutes and 25 seconds
CPU: Always utilizes a single core fully
Memory usage: Peak: 0.69 GB

graph_hdemand (larger ESDL)

### Gurobi
Runtime: 4 minutes and 20 seconds
CPU: Always utilizes a single core fully
Memory usage: Peak: 1.014 GB

### HIGHS
Runtime: 8 minutes and 45 seconds
CPU: Always utilizes a single core fully
Memory usage: Peak:  1.50 GB

Conclusions

Gurobi appears to significantly require less memory (33% for larger models, 14% for medium models)
Using Gurobi for our jobs make the jobs twice as fast.
- In our current setup, we appear to only use a single core.
- Significant time appears to also be spent in rtctools & mesido ('Transcribing problem' and between 'Done transcribing problem'-'start of solver')
- Very little time appears to be spend inside the gurobi solver (usually < 1 second. Couple of seconds for graph_hdemand)
- Conclusion: Faster than HIGHs
Mesido uses multiple calls to the solver.
- No issues were found that a license from an earlier call caused issues in next call within a job.
Only been able to trigger the fail safe once. All the other badgering appear to have been allowed. Checking with Ronald from Gurobi if this is normal.

MichielTukker · 2024-10-09T09:07:25Z

Good to know. At least the RuntimeError message format is predictable, "Flag: : " can be parsed to present an informative error message for the user

lfse-slafleur assigned MarkTNO Jan 18, 2024

lfse-slafleur assigned MichielTukker and unassigned MarkTNO Feb 1, 2024

lfse-slafleur added this to the Now milestone Apr 25, 2024

lfse-slafleur mentioned this issue Oct 2, 2024

v0.5 checklist #34

Closed

57 tasks

lfse-slafleur added the design ready for review label Oct 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Design Gurobi optimization worker #8

Design Gurobi optimization worker #8

lfse-slafleur commented Nov 9, 2023

lfse-slafleur commented Jan 18, 2024

lfse-slafleur commented Sep 30, 2024

lfse-slafleur commented Oct 2, 2024 •

edited

Loading

lfse-slafleur commented Oct 2, 2024

MichielTukker commented Oct 4, 2024

lfse-slafleur commented Oct 9, 2024

lfse-slafleur commented Oct 9, 2024

MichielTukker commented Oct 9, 2024

Design Gurobi optimization worker #8

Design Gurobi optimization worker #8

Comments

lfse-slafleur commented Nov 9, 2023

lfse-slafleur commented Jan 18, 2024

lfse-slafleur commented Sep 30, 2024

lfse-slafleur commented Oct 2, 2024 • edited Loading

lfse-slafleur commented Oct 2, 2024

MichielTukker commented Oct 4, 2024

lfse-slafleur commented Oct 9, 2024

lfse-slafleur commented Oct 9, 2024

How to get it to work

Questions:

Running too many instances

No license installed

Expired license installed

Resource usage comparison

PoC tutorial (small ESDL)

Ronald (medium ESDL)

graph_hdemand (larger ESDL)

Conclusions

MichielTukker commented Oct 9, 2024

lfse-slafleur commented Oct 2, 2024 •

edited

Loading