Workbench: Positron Python console slow or fails to start

[jthomasmock]

System details:

Positron and OS details:

Positron Version: 2024.11.0 build 140
Code - OSS Version: 1.93.0
Commit: e0d844b
Date: 2024-10-31T14:13:54.986Z
Browser: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36

Interpreter details:

Python 3.12.4

Describe the issue:

Re-opening an existing project often fails to start a Python kernel

Steps to reproduce the issue:

1. Create a test folder mkdir python3-test
2. python -m venv .venv and then source and install ipykernel
3. Close and kill the Workbench session, then start a new session and try to select that existing folder.
4. Note that the Python console appears to be starting up, but sometimes will fail to startup and become functional

Expected or desired behavior:

Python consoles should start up consistently.

Were there any error messages in the UI, Output panel, or Developer Tools console?

[petetronic]

We need to prepare an environment to measure how long these activities are taking and capture more information when it doesn't work / is unacceptably slow.

[jmcphers]

Did some analysis of this on Palm. Indeed, starting Python is pretty slow. It can take a full 15 seconds from the time we start the Python process until its sockets come online. Here are the supervisor logs from a session on Palm, note :41 -> :56 elapses while we are waiting:

18:53:41 [DEBUG] (2) kcserver::kernel_state: [session python-dc6beb23] status 'idle' => 'starting' (start API called)
18:53:41 [DEBUG] (2) kcserver::kernel_session: Starting kernel for session python-dc6beb23: ["/home/jonathan/pandas-example/.venv/bin/python", "/usr/lib/rstudio-server/bin/positron-server/extensions/positron-python/python_files/positron/positron_language_server.py", "-f", "/tmp/connection_python-dc6beb23.json", "--logfile", "/tmp/kernel-fVk9kC/kernel.log", "--loglevel=error", "--session-mode=console"]
18:53:56 [DEBUG] (3) kcserver::kernel_session: [session python-dc6beb23] All ZeroMQ sockets connected successfully
18:53:56 [INFO] [session python-dc6beb23] Connected to heartbeat socket at tcp://127.0.0.1:15544.

Python itself boots in less than 1/10 of a second, so this isn't caused by Python itself being slow to open.

$ time python -c 'print("hello world")'
hello world

real    0m0.066s
user    0m0.024s
sys     0m0.005s

So this delay is being caused by the LSP and kernel startup. During this period, no output is emitted, so it's difficult to tell what is going on.

[jmcphers]

I pulled a trace and discovered that the Python startup time on Workbench is entirely dominated by reading Python packages at startup. No individual package takes a long time, but there are a lot of packages to read. Here are some examples from a trace run on a local workbench env with a pyenv:

• the site packages for the python env (e.g. /home/jmcphers/.pyenv/versions/3.12.6/lib/python3.12/site-packages/jedi)
• the vendored packages in the extension (e.g. usr/lib/rstudio-server/bin/positron-server/extensions/positron-python/python_files/positron/positron_ipykernel/_vendor/pydantic/dataclasses.py)
• the pycached files (e.g. /home/jmcphers/.positron-server/User/globalStorage/ms-python.python/pycache/usr/lib/rstudio-server/bin/positron-server/extensions/positron-python/python_files/positron/positron_ipykernel/_vendor/pygls/constants.cpython-312.pyc)

As another interesting datapoint, using Python in VS Code notebooks in VS Code with the Jupyter extension (which also uses ipykernel) is pretty snappy, restarting in 1-2s vs 15s for Positron (on similar hardware).

[wesm]

See related #5596

[jmcphers]

I think we can consider this resolved for the time being with a combination of the performance improvements made to resolve #5596 along with the additional patience added in #5620. Let's verify that Python starts reliably on PTD now.