How do I use a long-lived ClientSession?

[the-vindicar]

So I need a wrapper class over some REST API, and I want to implement it using aiohttp.
I will need a ClientSession to make requests, and it's stated in the docs that

1. you shouldn't make a new session for every request.
2. you should make sure your ClientSession is created in async code.

This complicates things a bit. Obviously, I can't just use async with ClientSession() as session:, which is recommended in the docs. But I came up with the following:

import asyncio
import typing

import aiohttp


class APIWrapper:
    def __init__(self, url: str):
        self.url: str = url
        self.session: typing.Union[aiohttp.ClientSession, None] = None

    async def __aenter__(self) -> 'APIWrapper':
        if self.session is None:
            self.session = aiohttp.ClientSession()
            await self.session.__aenter__()
        return self

    async def __aexit__(self, exc_type, exc_val, exc_tb):
        if self.session is not None:
            await self.session.__aexit__(exc_type, exc_val, exc_tb)
            self.session = None

    async def do_stuff(self) -> typing.Dict[str, typing.Any]:
        async with self.session.get(self.url) as resp:
            return await resp.json()

I then use this code in a relatively simple manner:

async def main():
    async with APIWrapper('https://dummy.restapiexample.com/api/v1/employee/1') as api:
        print(await api.do_stuff())

asyncio.run(main())

And it works, but I get the infamous RuntimeError: Event loop is closed at the end.
Why?! I'm closing every resource I use! What am I missing?
Running on Python 3.10, Windows 10.

Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x00000226A6814310>
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "C:\Program Files\Python310\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "C:\Program Files\Python310\lib\asyncio\base_events.py", line 750, in call_soon
    self._check_closed()
  File "C:\Program Files\Python310\lib\asyncio\base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

[Dreamsorcerer]

At a glance, it looks like it should work.

I'd just pass through the aenter/aexit methods myself:

    async def __aenter__(self):
        self. _session = ClientSession()
        await self._session.__aenter__()
        return self

    async def __aexit__(self, ...):
        return await self._session.__exit__(...)

If that's a full, working example, I'll try and run it tomorrow.

[the-vindicar]

Nope, changes nothing. Give me a moment, I will find some open API to use, and make a self-contained example.

EDIT: Done. Should be runnable now.

[the-vindicar]

Went and checked it on a Debian... no error.
After a more through search, looks like issue #4324 either came back, or wasn't really fixed in first place. =(

[Dreamsorcerer]

Ah, Windows. Yes, that bug should be fixed in aiohttp 4, which is not released.

However, maybe you should also report a bug to cpython. As you can see in the traceback, aiohttp is nowhere in the traceback, so asyncio is causing the exception, not us:

Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000017FAC405048>
Traceback (most recent call last):
  File "C:\Programs\Python37\lib\asyncio\proactor_events.py", line 95, in __del__
    self.close()
  File "C:\Programs\Python37\lib\asyncio\proactor_events.py", line 86, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "C:\Programs\Python37\lib\asyncio\base_events.py", line 683, in call_soon
    self._check_closed()
  File "C:\Programs\Python37\lib\asyncio\base_events.py", line 475, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Specifically, you can see that _ProactorBasePipeTransport.__del__ is called because of garbage collection. This should produce a warning, but on ProactorEventLoop appears it's triggering an exception. That's an asyncio bug.

Aiohttp 4/master simply fixes the warning, by making sure the object is cleaned up when the ClientSession exits (which requires a backwards-incompatible change to the API).

[BabakAmini]

@Dreamsorcerer I have a similar situation where I need to access the created session in various sections of my code without using a context manager (which creates and destroys session instance every time), but rather by directly referencing it. Here is my class definition:

class MyClass():
    def __init__(self):
        self.session = aiohttp.ClientSession()
        self.loop = asyncio.new_event_loop() # I need this loop reference for later use in other class methods
        asyncio.set_event_loop(self.loop)
        atexit.register(self.cleanup)

    def cleanup(self):
        self.loop.create_task(self.session.close(), name='session_close')
        pending = asyncio.all_tasks(loop=self.loop)
        group = asyncio.gather(*pending)
        self.loop.run_until_complete(group)
        self.loop.close()

async def main():
    mc = MyClass()
    async with mc.session.get(url) as resp:
        return await resp.text()

asyncio.run(main())

This works without any issue and there is no warning like the one @the-vindicar discussed. Is this enough to ensure that everything is properly cleaned up?

Aiohttp 4/master simply fixes the warning, by making sure the object is cleaned up when the ClientSession exits.

Can I remove my cleanup procedure after upgrading to V4?

[Dreamsorcerer]

Your example doesn't look any different, it should still use a context manager as far as I can see, again passing __aenter__ and __aexit__ methods through so you can use MyClass as the context manager. Using atexit to do that cleanup is very bad practice for asyncio applications and I'd avoid it at all costs (and using any of the low-level asyncio APIs, or storing a reference to the loop ideally).

Can I remove my cleanup procedure after upgrading to V4?

No, the session still needs to exit/close.

[BabakAmini]

Your example doesn't look any different, it should still use a context manager as far as I can see.

I don't want to create a new session instance whenever needed. In my code, I create one instance of MyClass and then the only one instance of the session is available to send all requests. This is different from the code at the start of this discussion.

[Dreamsorcerer]

No, the session exists as long as the class is used, exactly the same as what you just described.
The original code is:

async def main():
    async with APIWrapper('https://dummy.restapiexample.com/api/v1/employee/1') as api:
        print(await api.do_stuff())

Your code can also use a context manager:

async def main():
    async with MyClass() as mc:
        async with mc.session.get(url) as resp:
            return await resp.text()