Make more tests robust and independent #546
Labels
ci
Continuous integration
Comments
|
|
The same error at test_connect_message::AssertionError-2> assert msg in captured.out
E assert "connected to database 'jdbc:hsqldb:mem://test_connect_message_1' (user: ixmp)..." in ''
E + where '' = CaptureResult(out='', err='').out
ixmp\tests\backend\test_jdbc.py:277: AssertionError
---------------------------- Captured stdout call -----------------------------
2024-09-30 05:17:54,257 INFO at.ac.iiasa.ixmp.Platform:182 - closed the connection to database 'jdbc:hsqldb:mem://test_connect_message_0'
2024-09-30 05:17:54,445 INFO at.ac.iiasa.ixmp.Platform:165 - Welcome to the IX modeling platform!
2024-09-30 05:17:54,445 INFO at.ac.iiasa.ixmp.Platform:166 - connected to database 'jdbc:hsqldb:mem://test_connect_message_1' (user: ixmp)...
------------------------------ Captured log call ------------------------------
INFO ixmp.backend.jdbc:jdbc.py:288 launching ixmp.Platform connected to jdbc:hsqldb:mem://test_connect_message_1And another slight variation of this today: ____________________________ test_connect_message _____________________________
[gw1] win32 -- Python 3.9.13 C:\hostedtoolcache\windows\Python\3.9.13\x64\python.exe
capfd = <_pytest.capture.CaptureFixture object at 0x0000017F77F31760>
caplog = <_pytest.logging.LogCaptureFixture object at 0x0000017F7916B850>
request = <FixtureRequest for <Function test_connect_message>>
def test_connect_message(capfd, caplog, request):
msg = (
f"connected to database 'jdbc:hsqldb:mem://{request.node.name}_0' "
"(user: ixmp)..."
)
ixmp.Platform(
backend="jdbc",
driver="hsqldb",
url=f"jdbc:hsqldb:mem://{request.node.name}_0",
log_level="INFO",
)
# Java code via JPype does not log to the standard Python logger
assert not any(msg in m for m in caplog.messages)
# NB cannot inspect capfd here. Depending on the order in which the tests were run,
# a previous run may have left the Java log level higher than INFO, in which
# case the Java Platform object would not write to stderr before set_log_level()
# in the above call. Try again now that the level is INFO:
msg = (
f"connected to database 'jdbc:hsqldb:mem://{request.node.name}_1' "
"(user: ixmp)..."
)
ixmp.Platform(
backend="jdbc",
driver="hsqldb",
url=f"jdbc:hsqldb:mem://{request.node.name}_1",
)
# Instead, log messages are printed to stdout
captured = capfd.readouterr()
> assert msg in captured.out
E assert "connected to database 'jdbc:hsqldb:mem://test_connect_message_1' (user: ixmp)..." in ''
E + where '' = CaptureResult(out='', err="2024-12-04 05:17:25,543 INFO at.ac.iiasa.ixmp.Platform:182 - closed the connection to database 'jdbc:hsqldb:mem://test_connect_message_0'\r\n2024-12-04 05:17:25,759 INFO at.ac.iiasa.ixmp.Platform:165 - Welcome to the IX modeling platform!\r\n2024-12-04 05:17:25,759 INFO at.ac.iiasa.ixmp.Platform:166 - connected to database 'jdbc:hsqldb:mem://test_connect_message_1' (user: ixmp)...\r\n").out
ixmp\tests\backend\test_jdbc.py:276: AssertionError
------------------------------ Captured log call ------------------------------
INFO ixmp.backend.jdbc:jdbc.py:288 launching ixmp.Platform connected to jdbc:hsqldb:mem://test_connect_message_1 |
|
Thanks for continuing to track these. I'll remove the If particular users report failures of the code in real-world usage that correspond to the flaky tests on GHA, then those are definitely |
|
|
|
|
|
|
|
|
Even this is reappearing, making me wonder how many tests were fixed in the original PR at all:
|
|
These errors never come alone, likely because they happen at the setup stage. Once there is one, there are around 12, all running into the same error message.
|
|
These tests were only flaky once on Ubuntu-latest with Python 3.13, they might only be flaky in combination with the error above.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Unfortunately, despite our recent improvements in #543, we still see some apparent flakiness. Its main source seems to be the setup-graphviz action on the windows runners, which sometimes fails like this:
In addition, there was a new flaky test today:
test_connect_message::AssertionError
Which almost seems like
outanderrgot mixed up in theCaptureResult. If this originates from one test changing the logging level without changing it back again, we should figure out a way to change the level just for this one test, without affecting the whole environment.The text was updated successfully, but these errors were encountered: