something wrong in my app with error "close cannot be used while an asynchronous query is underway"

[msean]

Renctly I tried to redeploy my project used with sanic frame ,some errors happened when I do a test under high-concurrency.
info in the log:

close cannot be used while an asynchronous query is underway

and the traceback is below:

Traceback (most recent call last):
  File "/home/mwh/ad_sanic/handlers/oprs.py", line 17, in post
    _parsed_req = await ReqParser(_req_body,'parse',req_ip=(request.headers['Remote-Addr']).split(':')[0]).parser()
  File "/home/mwh/ad_sanic/utils/req_parse.py", line 49, in parser
    _mapped['sim_country_iso'] = await self.geo()
  File "/home/mwh/ad_sanic/utils/req_parse.py", line 75, in geo
    db_country = (await self.db.get(DbIp.select(DbIp.country).where((DbIp.ip_from<=self._req_ip)&(DbIp.ip_to>=self._req_ip)))).country
  File "/home/mwh/py3.5_virtuanlenv/lib/python3.5/site-packages/peewee_async.py", line 169, in get
    result = yield from self.execute(query.limit(1))
  File "/home/mwh/py3.5_virtuanlenv/lib/python3.5/site-packages/peewee_async.py", line 269, in execute
    return (yield from execute(query))
  File "/home/mwh/py3.5_virtuanlenv/lib/python3.5/site-packages/peewee_async.py", line 428, in execute
    return (yield from coroutine(query))
  File "/home/mwh/py3.5_virtuanlenv/lib/python3.5/site-packages/peewee_async.py", line 569, in select
    cursor = yield from _execute_query_async(query)
  File "/home/mwh/py3.5_virtuanlenv/lib/python3.5/site-packages/peewee_async.py", line 1491, in _execute_query_async
    return (yield from _run_sql(query.database, *query.sql()))
  File "/home/mwh/py3.5_virtuanlenv/lib/python3.5/site-packages/peewee_async.py", line 1476, in _run_sql
    cursor = yield from database.cursor_async()
  File "/home/mwh/py3.5_virtuanlenv/lib/python3.5/site-packages/peewee_async.py", line 902, in cursor_async
    return (yield from self._async_conn.cursor(conn=conn))
  File "/home/mwh/py3.5_virtuanlenv/lib/python3.5/site-packages/peewee_async.py", line 1067, in cursor
    conn = yield from self.acquire()
  File "/home/mwh/py3.5_virtuanlenv/lib/python3.5/site-packages/peewee_async.py", line 1036, in acquire
    return (yield from self.pool.acquire())
  File "/home/mwh/py3.5_virtuanlenv/lib/python3.5/site-packages/aiopg/utils.py", line 67, in __iter__
    resp = yield from self._coro
  File "/home/mwh/py3.5_virtuanlenv/lib/python3.5/site-packages/aiopg/pool.py", line 180, in _acquire
    yield from self._cond.wait()
  File "/usr/local/lib/python3.5/asyncio/locks.py", line 326, in wait
    yield from fut
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 385, in __iter__
    yield self  # This tells Task to wait for completion.
  File "uvloop/future.pyx", line 434, in uvloop.loop.BaseTask._fast_wakeup (uvloop/loop.c:112710)
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 266, in result
    raise CancelledError
concurrent.futures._base.CancelledError

what caused this problem?

[rudyryk]

Probably that's some kind of edge case with uvloop and aiopg. Does this error happen without uvloop?

[zaabjuda]

I have this bug without uvloop.

[dennypenta]

Yes. I have the same issue.
I wanna testing my model with aiohttp and unit test.
I would try to solve it for an other driver, local threads parameters or override event loop.
But looks like its not helpful, manage connections has no effect too.
It's just testing, what if that issue will happen in a production.

[dennypenta]

You should explicity call objects.close() before close event loop

[ahopkins]

I get this while performing unit tests.

I have a series of simple tests to grab an entity with a simple manager context query. If I try to execute more than one during a test, I get this error.

_common/common/db/getters.py:82: in get_generic
    return await entity_getter(node_tag=node_tag, client_code=client_code)
_common/common/db/getters.py:17: in get_store
    Client.code ** client_code
/home/adam/.virtualenvs/mxtest/lib/python3.6/site-packages/peewee_async.py:169: in get
    result = yield from self.execute(query)
/home/adam/.virtualenvs/mxtest/lib/python3.6/site-packages/peewee_async.py:269: in execute
    return (yield from execute(query))
/home/adam/.virtualenvs/mxtest/lib/python3.6/site-packages/peewee_async.py:433: in execute
    return (yield from coroutine(query))
/home/adam/.virtualenvs/mxtest/lib/python3.6/site-packages/peewee_async.py:574: in select
    cursor = yield from _execute_query_async(query)
/home/adam/.virtualenvs/mxtest/lib/python3.6/site-packages/peewee_async.py:1555: in _execute_query_async
    return (yield from _run_sql(query.database, *query.sql()))
/home/adam/.virtualenvs/mxtest/lib/python3.6/site-packages/peewee_async.py:1542: in _run_sql
    return cursor
/home/adam/.virtualenvs/mxtest/lib/python3.6/site-packages/peewee.py:3656: in __exit__
    reraise(new_type, new_type(*exc_args), traceback)
/home/adam/.virtualenvs/mxtest/lib/python3.6/site-packages/peewee.py:135: in reraise
    raise value.with_traceback(tb)
/home/adam/.virtualenvs/mxtest/lib/python3.6/site-packages/peewee_async.py:1539: in _run_sql
    yield from cursor.release
/usr/lib/python3.6/asyncio/coroutines.py:210: in coro
    res = func(*args, **kw)
/home/adam/.virtualenvs/mxtest/lib/python3.6/site-packages/peewee_async.py:1107: in release_cursor
    cursor.close()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <aiopg.cursor.Cursor object at 0x7fc48e910fd0>

    def close(self):
        """Close the cursor now."""
>       self._impl.close()
E       peewee.ProgrammingError: close cannot be used while an asynchronous query is underway

/home/adam/.virtualenvs/mxtest/lib/python3.6/site-packages/aiopg/cursor.py:51: ProgrammingError

[hexazero]

I was getting this same issue.

peewee.ProgrammingError: close cannot be used while an asynchronous query is underway

After tracing through the issue, I determined it was due to my setup with pytest-asyncio since that library does the following in the default event_loop fixture:

Creates and injects a new instance of the default asyncio event loop. By default, the loop will be closed at the end of the test (i.e. the default fixture scope is function).

To fix the issue, I created a conftest.py file at the root of my tests that has the following replacement fixture.

import pytest
import asyncio


@pytest.fixture(scope='session')
def event_loop():
    return asyncio.new_event_loop()

Maybe this will help someone else who lands here from google.