Supervisor: Make ZeroMQ socket timeout configurable, and/or increase default timeout #5620
Assignees
Labels
area:kallichore
Issues related to the new kernel supervisor
area: workbench
Issues related to Workbench category.
Milestone
Comments
|
Moving this up to 2025.01.0 - I keep running into this for new workspaces on PTD. I have to restart the runtime and hope it starts in less than 20 seconds. |
jmcphers
added a commit
that referenced
this issue
Dec 10, 2024
Addresses #5620 by adding an option to specify the number of seconds to wait for the kernel to connect before giving up. This change is mostly on the supervisor side; the client changes here just add the new option and pass it along to the supervisor. <img width="507" alt="image" src="https://github.com/user-attachments/assets/89ae4191-3865-4637-b301-e5660e61cd4d"> The default was 20 (hardcoded) and is now 30 (configurable). While I was updating the API, I also added some methods/types that will later support #5226 (but aren't currently implemented on the client or server side). ### QA Notes A really easy way to test this is to set the timeout at 1 second, since most kernels do not start up that quickly.
Verified FixedPositron Version(s) : 2025.01.0-71 Test scenario(s)Tested several intervals.. works as expected. 1s timeout triggers as expected. Link(s) to TestRail test cases run or created: |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Currently, the kernel supervisor waits up to 20 seconds for ZeroMQ socket connections. If it is not able to connect to the kernel after 20 seconds, it shows an error like this one:
20 seconds is a long time to wait for startup, but some systems are pretty slow, and in our own environments (which aren't necessarily the slowest of any we would support) we've observed start times of > 15s even when everything is working correctly. See e.g. traces in #5340.
It would be great if there were some way for us to know whether the kernel was working correctly (and slowly) or legitimately hung when we are waiting for a socket connection. However, since the sockets are how we talk to the kernel in the first place, we would need to establish some sort of side channel or heuristic to figure this out.
We should, at a minimum:
The text was updated successfully, but these errors were encountered: