random failures when running test_conn with Python 3.13

[kulikjak]

❓ I'm submitting a ...

• 🐞 bug report
• 🐣 feature request
• ❓ question about the decisions made in the repository

🐞 Describe the bug. What is the current behavior?
And one more failure with Python 3.13: when running cheroot/test/test_conn.py, I am seeing one of the tests fail with the following error:

E                   pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function IOBase.__del__ at 0x1ffd0ae81cea20>
E                   
E                   Traceback (most recent call last):
E                     File "/usr/lib/python3.13/_pyio.py", line 418, in __del__
E                       self.close()
E                       ~~~~~~~~~~^^
E                     File "/usr/lib/python3.13/_pyio.py", line 1313, in close
E                       self.flush()
E                       ~~~~~~~~~~^^
E                     File "/usr/lib/python3.13/_pyio.py", line 1274, in flush
E                       self._flush_unlocked()
E                       ~~~~~~~~~~~~~~~~~~~~^^
E                     File "/scratch/userland-gate/components/python/cheroot/cheroot-10.0.0/cheroot/makefile.py", line 33, in _flush_unlocked
E                       n = self.raw.write(bytes(self._write_buf))
E                     File "/usr/lib/python3.13/socket.py", line 737, in write
E                       return self._sock.send(b)
E                              ~~~~~~~~~~~~~~~^^^
E                   OSError: [Errno 9] Bad file number

I saw the following three fail: test_streaming_10[True], test_keepalive[HTTP/1.0], test_keepalive[HTTP/1.1] with this error, although it's always just one for each run (and it seems random).

This again seems to be related to the following io change:
https://docs.python.org/3/whatsnew/3.13.html#io

I wasn't able to find a fix for this one.

💡 To Reproduce
Run the test suite with Python 3.13.0.

📋 Details
Full traceback:

==================================== ERRORS ====================================
__________________ ERROR at setup of test_keepalive[HTTP/1.0] __________________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_and_report.<locals>.<lambda> at 0x1fffe71a7c2020>
when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: Callable[[], TResult],
        when: Literal["collect", "setup", "call", "teardown"],
        reraise: type[BaseException] | tuple[type[BaseException], ...] | None = None,
    ) -> CallInfo[TResult]:
        """Call func, wrapping the result in a CallInfo.
    
        :param func:
            The function to call. Called without arguments.
        :type func: Callable[[], _pytest.runner.TResult]
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: TResult | None = func()

cls        = <class '_pytest.runner.CallInfo'>
duration   = 0.10653769504278898
excinfo    = <ExceptionInfo PytestUnraisableExceptionWarning('Exception ignored in: <function IOBase.__del__ at 0x1fffe71f1cea20>\n...in write\n    return self._sock.send(b)\n           ~~~~~~~~~~~~~~~^^^\nOSError: [Errno 9] Bad file number\n') tblen=8>
func       = <function call_and_report.<locals>.<lambda> at 0x1fffe71a7c2020>
precise_start = 7332079.8773448
precise_stop = 7332079.983882495
reraise    = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
result     = None
start      = 1730815279.3573403
stop       = 1730815279.463881
when       = 'setup'

/usr/lib/python3.13/vendor-packages/_pytest/runner.py:341: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.13/vendor-packages/_pytest/runner.py:242: in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
        item       = <Function test_keepalive[HTTP/1.0]>
        kwds       = {}
        runtest_hook = <HookCaller 'pytest_runtest_setup'>
/usr/lib/python3.13/vendor-packages/pluggy/_hooks.py:513: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
        firstresult = False
        kwargs     = {'item': <Function test_keepalive[HTTP/1.0]>}
        self       = <HookCaller 'pytest_runtest_setup'>
/usr/lib/python3.13/vendor-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
        firstresult = False
        hook_name  = 'pytest_runtest_setup'
        kwargs     = {'item': <Function test_keepalive[HTTP/1.0]>}
        methods    = [<HookImpl plugin_name='runner', plugin=<module '_pytest.runner' from '/usr/lib/python3.13/vendor-packages/_pytest/run...ugin=<module '_pytest.unraisableexception' from '/usr/lib/python3.13/vendor-packages/_pytest/unraisableexception.py'>>]
        self       = <_pytest.config.PytestPluginManager object at 0x1fffe72020c590>
/usr/lib/python3.13/vendor-packages/_pytest/unraisableexception.py:90: in pytest_runtest_setup
    yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def unraisable_exception_runtest_hook() -> Generator[None]:
        with catch_unraisable_exception() as cm:
            try:
                yield
            finally:
                if cm.unraisable:
                    if cm.unraisable.err_msg is not None:
                        err_msg = cm.unraisable.err_msg
                    else:
                        err_msg = "Exception ignored in"
                    msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                    msg += "".join(
                        traceback.format_exception(
                            cm.unraisable.exc_type,
                            cm.unraisable.exc_value,
                            cm.unraisable.exc_traceback,
                        )
                    )
>                   warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E                   pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function IOBase.__del__ at 0x1fffe71f1cea20>
E                   
E                   Traceback (most recent call last):
E                     File "/usr/lib/python3.13/_pyio.py", line 418, in __del__
E                       self.close()
E                       ~~~~~~~~~~^^
E                     File "/usr/lib/python3.13/_pyio.py", line 1313, in close
E                       self.flush()
E                       ~~~~~~~~~~^^
E                     File "/usr/lib/python3.13/_pyio.py", line 1274, in flush
E                       self._flush_unlocked()
E                       ~~~~~~~~~~~~~~~~~~~~^^
E                     File "/scratch/userland-gate/components/python/cheroot/cheroot-10.0.0/cheroot/makefile.py", line 33, in _flush_unlocked
E                       n = self.raw.write(bytes(self._write_buf))
E                     File "/usr/lib/python3.13/socket.py", line 737, in write
E                       return self._sock.send(b)
E                              ~~~~~~~~~~~~~~~^^^
E                   OSError: [Errno 9] Bad file number

cm         = <_pytest.unraisableexception.catch_unraisable_exception object at 0x1fffe71f1de090>
err_msg    = 'Exception ignored in'
msg        = 'Exception ignored in: <function IOBase.__del__ at 0x1fffe71f1cea20>\n\nTraceback (most recent call last):\n  File "/u...line 737, in write\n    return self._sock.send(b)\n           ~~~~~~~~~~~~~~~^^^\nOSError: [Errno 9] Bad file number\n'

📋 Environment

• Cheroot version: tested with 10.0.0 and 11.0.0b3
• Python version: 3.13.0
• OS: Oracle Solaris (but this doesn't seem to be platform dependent)

[webknjaz]

Thanks for reporting! Sounds like pytest's warning tracking feature detects an exception that was being ignored silently in older CPython releases. A short-term workaround for the tests could be adding an ignore entry to filterwarnings in pytest.ini. But long-term, we need to inspect why this is happening and track down what exactly seems to close the socket before flushing. One way to do this could be using https://docs.python.org/3/using/cmdline.html#envvar-PYTHONTRACEMALLOC.

[kulikjak]

I tried adding PYTHONTRACEMALLOC, but unfortunately the trace looks the same. It also passes roughly half the time I run the test suite with it, so the failure seems time dependent?

Adding ignore::pytest.PytestUnraisableExceptionWarning makes the test suite green.

[webknjaz]

Yeah, it sounds like there's a race condition somewhere.

If you send a PR with that but a more granular ignore matching the specific message, I'm open to accepting it as an immediate workaround.

[webknjaz]

Let's keep this open until the real fix can be implemented.

[The-Compiler]

For what it's worth, this isn't only an issue in cheroot's tests - I have a test HTTP server I run as a subprocess for qutebrowser tests, and I see this pop up there as well.

I think a prerequisite to trigger it is that the client aborts the connection, I've been able to reproduce it when spamming F5 in a browser (which seems to have cancelled requests for e.g. favicon.ico), but not trivially with curl or requests.

However, setting a requests timeout seems to reproduce it nicely, here is what I use to get it happen within 0.5s or so:

import time
import requests
import requests.exceptions
import threading
from cheroot import wsgi

def timeout_app(environ, start_response):
    time.sleep(0.05)
    status = '200 OK'
    headers = [('Content-type', 'text/plain; charset=utf-8')]
    start_response(status, headers)
    return [b""]

port = 1234
server = wsgi.Server(('localhost', port), timeout_app)
url = f"http://localhost:{port}"
print(url)

thread = threading.Thread(target=server.safe_start)
thread.start()

while not server.ready:
    time.sleep(0.1)

while True:
    try:
        requests.get(url, timeout=0.01)
    except requests.exceptions.ReadTimeout:
        print("!", end="", flush=True)
    else:
        print(".", end="", flush=True)

[The-Compiler]

Here is a workaround that works outside of pytest and hides those warnings:

import sys
import errno


def unraisable_hook(unraisable: "sys.UnraisableHookArgs") -> None:
    if (
        sys.version_info[:2] == (3, 13)
        and isinstance(unraisable.exc_value, OSError)
        and (
            unraisable.exc_value.errno == errno.EBADF
            or (
                sys.platform == "win32"
                and unraisable.exc_value.winerror == errno.WSAENOTSOCK
            )
        )
        and unraisable.object.__qualname__ == "IOBase.__del__"
    ):
        # WORKAROUND for bogus exceptions with cheroot:
        # https://github.com/cherrypy/cheroot/issues/734
        return
    sys.__unraisablehook__(unraisable)


sys.unraisablehook = unraisable_hook