Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes aiopg hanging or raising psycopg2.OperationalError suddenly in gh-actions CI #2138

Merged

Conversation

pcrespov
Copy link
Member

@pcrespov pcrespov commented Feb 5, 2021

Problem

Some jobs in our gh-actions CI suddenly experienced long delays or failures as

    async def cancel():
        self._waiter = create_future(self._loop)
        self._cancelling = True
        self._cancellation_waiter = self._waiter
>       self._conn.cancel()
E       psycopg2.OperationalError: asynchronous connection attempt underway

/opt/hostedtoolcache/Python/3.6.12/x64/lib/python3.6/site-packages/aiopg/connection.py:195: OperationalError

Found a report of a bug in aiopg with similar behaviour caused by problems resolving the hostname.

Noticed that the VM deployed in gh-actions before and after the problem had just been updated. Checking the /etc/hosts revealed multiple maps of 127.0.0.1 which might have been a reason why aiopg could not resolve it (again a pr in aiopg to address that problem)

What do these changes do?

Replacing localhost by 127.0.0.1 in all settings related with pg hostname

How to test

Run all tests and no more jobs hanging or tests raising OperationalError

@codecov
Copy link

codecov bot commented Feb 5, 2021

Codecov Report

Merging #2138 (2d66bbc) into master (8670901) will increase coverage by 7.3%.
The diff coverage is 100.0%.

Impacted file tree graph

@@           Coverage Diff            @@
##           master   #2138     +/-   ##
========================================
+ Coverage    67.2%   74.6%   +7.3%     
========================================
  Files         409     419     +10     
  Lines       15811   16316    +505     
  Branches     1608    1612      +4     
========================================
+ Hits        10633   12176   +1543     
+ Misses       4763    3686   -1077     
- Partials      415     454     +39     
Flag Coverage Δ
integrationtests 65.4% <ø> (+<0.1%) ⬆️
unittests 68.7% <100.0%> (+13.2%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ages/service-library/src/servicelib/aiopg_utils.py 91.3% <100.0%> (ø)
.../simcore-sdk/src/simcore_sdk/node_data/__init__.py 0.0% <0.0%> (-100.0%) ⬇️
...core-sdk/src/simcore_sdk/node_data/data_manager.py 0.0% <0.0%> (-95.3%) ⬇️
...re-sdk/src/simcore_sdk/node_ports_v2/port_utils.py 78.9% <0.0%> (-15.8%) ⬇️
...es/simcore-sdk/src/simcore_sdk/node_ports/_item.py 79.5% <0.0%> (-14.8%) ⬇️
...ore-sdk/src/simcore_sdk/node_ports/_schema_item.py 87.5% <0.0%> (-12.5%) ⬇️
.../simcore-sdk/src/simcore_sdk/node_ports_v2/port.py 85.2% <0.0%> (-11.5%) ⬇️
...-sdk/src/simcore_sdk/node_ports_v2/nodeports_v2.py 89.7% <0.0%> (-10.3%) ⬇️
...mcore-sdk/src/simcore_sdk/node_ports/_data_item.py 80.9% <0.0%> (-9.6%) ⬇️
...mcore-sdk/src/simcore_sdk/node_ports/exceptions.py 68.5% <0.0%> (-9.3%) ⬇️
... and 148 more

@pcrespov pcrespov self-assigned this Feb 5, 2021
@pcrespov pcrespov added the bug buggy, it does not work as expected label Feb 5, 2021
@pcrespov pcrespov added this to the The White Rabbit milestone Feb 5, 2021
@pcrespov pcrespov marked this pull request as ready for review February 5, 2021 20:12
@pcrespov pcrespov merged commit 3e051b5 into ITISFoundation:master Feb 5, 2021
@pcrespov pcrespov deleted the fix/aiopg-hanging-git-actions branch February 5, 2021 20:31
@sanderegg sanderegg mentioned this pull request Feb 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug buggy, it does not work as expected
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant