Fix Windows Threading Issues #385

FrsECM · 2024-12-04T22:59:18Z

Before submitting

Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)

⚠️ How does this PR impact the user? ⚠️
As a user, i need to serve a model with multiple worker per device on a windows machine

What does this PR do?

This PR propose a fix to bug from Issue #384.

Uvicorn reference: Source

For example, with this simple server :

import litserve as ls
class SimpleLitAPI(ls.LitAPI):
    def setup(self, device):
        self.model1 = lambda x: x**2
    def decode_request(self, request):
        return request["input"] 
    def predict(self, x):
        squared = self.model1(x)
        output = squared
        return {"output": output}

    def encode_response(self, output):
        return {"output": output} 

# (STEP 2) - START THE SERVER
if __name__ == "__main__":
    # scale with advanced features (batching, GPUs, etc...)
    server = ls.LitServer(SimpleLitAPI(), accelerator="auto", max_batch_size=1,workers_per_device=2)
    server.run(port=8000)

Bellow, you'll find the difference between before and after the PR :

Before the PR

(litserve_win) PS > python .\dummy_server.py
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
C:\Users\F296849\AppData\Local\miniforge3\envs\iris2_win\lib\site-packages\litserve\server.py:475: UserWarning: Windows does not support forking. Using threads api_server_worker_type will be set to 'thread'
  warnings.warn(
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
Swagger UI is available at http://0.0.0.0:8000/docs
INFO:     Started server process [35312]
INFO:     Started server process [35312]
INFO:     Waiting for application startup.
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Application startup complete.
Accept failed on a socket
socket: <asyncio.TransportSocket fd=608, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 8000)>
Traceback (most recent call last):
  File "C:\Users\F296849\AppData\Local\miniforge3\envs\iris2_win\lib\asyncio\proactor_events.py", line 841, in loop
    f = self._proactor.accept(sock)
  File "C:\Users\F296849\AppData\Local\miniforge3\envs\iris2_win\lib\asyncio\windows_events.py", line 563, in accept
    self._register_with_iocp(listener)
  File "C:\Users\F296849\AppData\Local\miniforge3\envs\iris2_win\lib\asyncio\windows_events.py", line 732, in _register_with_iocp      
    _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)
OSError: [WinError 87] Paramètre incorrect
Task exception was never retrieved
future: <Task finished name='Task-7' coro=<IocpProactor.accept.<locals>.accept_coro() done, defined at C:\Users\F296849\AppData\Local\miniforge3\envs\iris2_win\lib\asyncio\windows_events.py:577> exception=OSError(22, 'L’opération d’entrée/sortie a été abandonnée en raison de l’arrêt d’un thread ou à la demande d’une application', None, 995, None)>
Traceback (most recent call last):
  File "C:\Users\F296849\AppData\Local\miniforge3\envs\iris2_win\lib\asyncio\windows_events.py", line 580, in accept_coro
    await future
OSError: [WinError 995] L’opération d’entrée/sortie a été abandonnée en raison de l’arrêt d’un thread ou à la demande d’une application
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
Setup complete for worker 1.
Setup complete for worker 0.

After the PR

(litserve_win) PS > python .\dummy_server.py
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
C:\Users\F296849\AppData\Local\miniforge3\envs\iris2_win\lib\site-packages\litserve\server.py:475: UserWarning: Windows does not support forking. Using threads api_server_worker_type will be set to 'thread'
  warnings.warn(
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
Swagger UI is available at http://0.0.0.0:8000/docs
INFO:     Started server process [24816]
INFO:     Waiting for application startup.
INFO:     Started server process [24816]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Application startup complete.
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
Setup complete for worker 0.
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
Setup complete for worker 1.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

for more information, see https://pre-commit.ci

codecov · 2024-12-04T23:08:38Z

Codecov Report

Attention: Patch coverage is 70.37037% with 8 lines in your changes missing coverage. Please review.

Project coverage is 87%. Comparing base (ca9d8c9) to head (63b1963).

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #385   +/-   ##
===================================
- Coverage    88%    87%   -1%     
===================================
  Files        25     25           
  Lines      1774   1791   +17     
===================================
+ Hits       1560   1564    +4     
- Misses      214    227   +13

aniketmaurya · 2024-12-04T23:26:27Z

hi @FrsECM, thank you so much for the PR! Could you also add additional information about why setting config.workers fixes this issue for reference?

FrsECM · 2024-12-04T23:48:47Z

hi @FrsECM, thank you so much for the PR! Could you also add additional information about why setting config.workers fixes this issue for reference?

Sure.
It's to make sure the code go through this portion of uvicorn :
encode/uvicorn@858f1c5

By default, the number of worker is set to 1 and it crashes when we have multiples worker per devices on windows.

for more information, see https://pre-commit.ci

FrsECM · 2024-12-05T00:41:11Z

I have to investigate a little more on another issue that cause [WinError 10022].

It seems that in some case (not completely reproductible), the socket is not ready to listen when we start uvicorn servers.
A workarround is to force listening before setting up uvicorn servers. I don't have any idea about why it behaves like that.
If you have any idea.

FrsECM · 2024-12-06T00:05:31Z

@aniketmaurya did you have a look on the PR ?

It seems to fix my problem on windows.

It doesn't fix issue number #372 but it allows to make multiple worker on windows.

aniketmaurya

LGTM! thanks for creating the fix 🚀

src/litserve/server.py

… into bugfix/windows_multiple_workers

for more information, see https://pre-commit.ci

FrsECM · 2024-12-08T17:31:24Z

@aniketmaurya i got an idea for #372.

Unlike on Linux, i discovered that inference workers were stopped first instead of uvicorn's.

A fix is to join on inference worker on windows and to cleanly give a signal to uvicorn in order to end threads properly.
In order to do it, i need to keep uvicorn's server in a class variable.
It's because currently only workers are returned by method _start_server().

I renamed some variables to be a little more explicit about their content.

src/litserve/server.py

Fix Type hint Co-authored-by: Aniket Maurya <[email protected]>

Remove windows comment. Co-authored-by: Aniket Maurya <[email protected]>

for more information, see https://pre-commit.ci

aniketmaurya · 2024-12-12T19:08:53Z

@FrsECM it seems like the Windows tests are stuck and have timed out.

FrsECM · 2024-12-12T19:10:46Z

Hum. Curious. On my windows machine I don’t have issues. I don’t know well the test stack. I'll try to have a look tomorrow. Envoyé à partir de Outlook pour iOS<https://aka.ms/o0ukef>

…

________________________________ De : Aniket Maurya ***@***.***> Envoyé : Thursday, December 12, 2024 8:09:16 PM À : Lightning-AI/LitServe ***@***.***> Cc : François Ponchon ***@***.***>; Mention ***@***.***> Objet : Re: [Lightning-AI/LitServe] Fix Windows Threading Issues (PR #385) @FrsECM<https://github.com/FrsECM> it seems like the Windows tests are stuck and have timed out. — Reply to this email directly, view it on GitHub<#385 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGG5F7H2TWURUOQHJ6OVXRT2FHNNZAVCNFSM6AAAAABTBG2HB6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZZHAYDQMJZHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

FrsECM · 2024-12-13T09:29:26Z

@aniketmaurya i did a first try and suspected httpx>=0.28.0. But i was not up to date.

On python 3.10.15, i have no issues on my computer, every tests are running.

(litserve) PS C:\BUSCODE\packages\LitServe> python -m pytest
C:\Users\F296849\AppData\Local\miniforge3\envs\litserve\lib\site-packages\pytest_asyncio\plugin.py:207: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"

  warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
==================================================================== test session starts ====================================================================
platform win32 -- Python 3.10.15, pytest-8.3.4, pluggy-1.5.0
rootdir: C:\BUSCODE\packages\LitServe
configfile: pytest.ini
plugins: anyio-4.6.2.post1, asyncio-0.25.0, cov-6.0.0
asyncio: mode=strict, asyncio_default_fixture_loop_scope=None
collected 169 items

tests\e2e\test_e2e.py .............                                                                                                                    [  7%]
tests\test_auth.py ....                                                                                                                                [ 10%]
tests\test_batch.py ...........                                                                                                                        [ 16%]
tests\test_callbacks.py ....                                                                                                                           [ 18%]
tests\test_cli.py ..                                                                                                                                   [ 20%] 
tests\test_compression.py .                                                                                                                            [ 20%]
tests\test_connector.py s..ssssssss.                                                                                                                   [ 27%] 
tests\test_docker_builder.py ..                                                                                                                        [ 28%]
tests\test_examples.py ..........                                                                                                                      [ 34%]
tests\test_form.py ...                                                                                                                                 [ 36%]
tests\test_lit_server.py .........sssss..ss...........                                                                                                 [ 53%]
tests\test_litapi.py ..................                                                                                                                [ 64%]
tests\test_logger.py ........                                                                                                                          [ 69%]
tests\test_logging.py ...                                                                                                                              [ 71%] 
tests\test_loops.py .............                                                                                                                      [ 78%]
tests\test_middlewares.py ...                                                                                                                          [ 80%]
tests\test_pydantic.py .                                                                                                                               [ 81%]
tests\test_readme.py s                                                                                                                                 [ 81%] 
tests\test_schema.py ..                                                                                                                                [ 82%]
tests\test_simple.py .......                                                                                                                           [ 86%]
tests\test_specs.py ..................                                                                                                                 [ 97%]
tests\test_torch.py .s                                                                                                                                 [ 98%]
tests\test_utils.py ..                                                                                                                                 [100%]
================================================= 151 passed, 18 skipped, 50 warnings in 204.37s (0:03:24) ==================================================

On python 3.11.11, i have no issues on my computer, every tests are running.

(litserve3.11) PS C:\BUSCODE\packages\LitServe> python -m pytest
C:\Users\F296849\AppData\Local\miniforge3\envs\litserve3.11\Lib\site-packages\pytest_asyncio\plugin.py:207: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"

  warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
==================================================================== test session starts ====================================================================
platform win32 -- Python 3.11.11, pytest-8.3.4, pluggy-1.5.0
rootdir: C:\BUSCODE\packages\LitServe
configfile: pytest.ini
plugins: anyio-4.7.0, asyncio-0.25.0, cov-6.0.0, retry-1.6.3
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=None
collected 169 items

tests\e2e\test_e2e.py .............                                                                                                                    [  7%]
tests\test_auth.py ....                                                                                                                                [ 10%]
tests\test_batch.py ...........                                                                                                                        [ 16%]
tests\test_callbacks.py ....                                                                                                                           [ 18%]
tests\test_cli.py ..                                                                                                                                   [ 20%]
tests\test_compression.py .                                                                                                                            [ 20%]
tests\test_connector.py s..ssssssss.                                                                                                                   [ 27%] 
tests\test_docker_builder.py ..                                                                                                                        [ 28%] 
tests\test_examples.py ..........                                                                                                                      [ 34%]
tests\test_form.py ...                                                                                                                                 [ 36%]
tests\test_lit_server.py .........sssss..ss...........                                                                                                 [ 53%]
tests\test_litapi.py ..................                                                                                                                [ 64%]
tests\test_logger.py ........                                                                                                                          [ 69%]
tests\test_logging.py ...                                                                                                                              [ 71%] 
tests\test_loops.py .............                                                                                                                      [ 78%]
tests\test_middlewares.py ...                                                                                                                          [ 80%]
tests\test_pydantic.py .                                                                                                                               [ 81%]
tests\test_readme.py s                                                                                                                                 [ 81%] 
tests\test_schema.py ..                                                                                                                                [ 82%]
tests\test_simple.py .......                                                                                                                           [ 86%]
tests\test_specs.py ..................                                                                                                                 [ 97%]
tests\test_torch.py .s                                                                                                                                 [ 98%]
tests\test_utils.py ..                                                                                                                                 [100%] 
================================================= 151 passed, 18 skipped, 48 warnings in 205.77s (0:03:25) ==================================================

FrsECM · 2024-12-13T09:42:17Z

@aniketmaurya I tried to increase the timeout. The usage of threading instead of processes and the absence of uvloop on windows makes it slower. It may also depend on the runner.
But i need your agreement to rerun the ci.

aniketmaurya · 2024-12-13T11:18:22Z

.github/workflows/ci-testing.yml

@@ -48,7 +48,7 @@ jobs:
          pip list

      - name: Tests
-        timeout-minutes: 10
+        timeout-minutes: 30


Let's switch it back after figuring out the reason CI is stuck since we don't want to run tests for 30 mins.

Suggested change

timeout-minutes: 30

timeout-minutes: 10

It seems CI is stuck since this commit 2bbed42

Maybe due to version 3.11 of python for a reason i ignore.

yes, since it's only specific to Python 3.11 on Windows latest, it probably means that something is not working as expected.

Fix bug on windows with uvicorn when multiple workers.

134963b

FrsECM requested review from lantiga, aniketmaurya, ethanwharris, Andrei-Aksionov and Borda as code owners December 4, 2024 22:59

[pre-commit.ci] auto fixes from pre-commit.com hooks

cb8e349

for more information, see https://pre-commit.ci

Francois Ponchon and others added 2 commits December 5, 2024 00:30

Force socket to listen before starting server

47efcc6

[pre-commit.ci] auto fixes from pre-commit.com hooks

fd977d4

for more information, see https://pre-commit.ci

aniketmaurya approved these changes Dec 6, 2024

View reviewed changes

src/litserve/server.py Outdated Show resolved Hide resolved

aniketmaurya and others added 4 commits December 6, 2024 10:12

Update src/litserve/server.py

a2c269a

Fix Ctrl+C on windows

0f5c29b

Merge remote-tracking branch 'origin/bugfix/windows_multiple_workers'…

2b27f14

… into bugfix/windows_multiple_workers

[pre-commit.ci] auto fixes from pre-commit.com hooks

494fcf9

for more information, see https://pre-commit.ci

Fix comments - Ctrl+C on Windows

f9c3141

FrsECM changed the title ~~Fix bug on windows with uvicorn when multiple workers.~~ Fix Windows Threading Issues Dec 8, 2024

Merge branch 'main' into bugfix/windows_multiple_workers

583a249

aniketmaurya reviewed Dec 8, 2024

View reviewed changes

src/litserve/server.py Outdated Show resolved Hide resolved

src/litserve/server.py Outdated Show resolved Hide resolved

FrsECM and others added 4 commits December 8, 2024 22:13

Update src/litserve/server.py

d05e664

Fix Type hint Co-authored-by: Aniket Maurya <[email protected]>

Update src/litserve/server.py

cbc9be5

Remove windows comment. Co-authored-by: Aniket Maurya <[email protected]>

Fix threading import Thread

2d598ed

[pre-commit.ci] auto fixes from pre-commit.com hooks

d84e445

for more information, see https://pre-commit.ci

aniketmaurya self-requested a review December 8, 2024 23:39

Merge branch 'main' into bugfix/windows_multiple_workers

3d7ff88

Merge branch 'main' into bugfix/windows_multiple_workers

3d51d28

Borda approved these changes Dec 11, 2024

View reviewed changes

Merge branch 'main' into bugfix/windows_multiple_workers

2478dcd

Francois Ponchon and others added 2 commits December 13, 2024 09:36

Increase test timeout => 30mn

3103bb8

Merge branch 'main' into bugfix/windows_multiple_workers

63b1963

FrsECM requested a review from Borda December 13, 2024 09:48

aniketmaurya reviewed Dec 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Windows Threading Issues #385

Fix Windows Threading Issues #385

FrsECM commented Dec 4, 2024 •

edited by aniketmaurya

Loading

codecov bot commented Dec 4, 2024 •

edited

Loading

aniketmaurya commented Dec 4, 2024

FrsECM commented Dec 4, 2024 •

edited

Loading

FrsECM commented Dec 5, 2024

FrsECM commented Dec 6, 2024

aniketmaurya left a comment

FrsECM commented Dec 8, 2024 •

edited

Loading

aniketmaurya commented Dec 12, 2024

FrsECM commented Dec 12, 2024 via email

FrsECM commented Dec 13, 2024

FrsECM commented Dec 13, 2024 •

edited

Loading

aniketmaurya Dec 13, 2024 •

edited

Loading

FrsECM Dec 13, 2024

aniketmaurya Dec 13, 2024

Fix Windows Threading Issues #385

Are you sure you want to change the base?

Fix Windows Threading Issues #385

Conversation

FrsECM commented Dec 4, 2024 • edited by aniketmaurya Loading

What does this PR do?

PR review

Did you have fun?

codecov bot commented Dec 4, 2024 • edited Loading

Codecov Report

aniketmaurya commented Dec 4, 2024

FrsECM commented Dec 4, 2024 • edited Loading

FrsECM commented Dec 5, 2024

FrsECM commented Dec 6, 2024

aniketmaurya left a comment

Choose a reason for hiding this comment

FrsECM commented Dec 8, 2024 • edited Loading

aniketmaurya commented Dec 12, 2024

FrsECM commented Dec 12, 2024 via email

FrsECM commented Dec 13, 2024

FrsECM commented Dec 13, 2024 • edited Loading

aniketmaurya Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

FrsECM Dec 13, 2024

Choose a reason for hiding this comment

aniketmaurya Dec 13, 2024

Choose a reason for hiding this comment

FrsECM commented Dec 4, 2024 •

edited by aniketmaurya

Loading

codecov bot commented Dec 4, 2024 •

edited

Loading

FrsECM commented Dec 4, 2024 •

edited

Loading

FrsECM commented Dec 8, 2024 •

edited

Loading

FrsECM commented Dec 13, 2024 •

edited

Loading

aniketmaurya Dec 13, 2024 •

edited

Loading