🔨 disk-usage pytest plugin

[pcrespov]

What do these changes do?

After running out of space in the gh-action machines the CI stops with messages as

       XMinioStorageFull: Storage backend has reached its minimum free drive threshold. Please delete a few objects to proceed.

This PR implements an optional plugin to monitor disk usage during test execution, identifying tests that do not properly clean up resources. This helps prevent potential issues when running continuous integration (CI) pipelines on external systems, such as GitHub Actions.

The plugin is activated by using the --disk-usage option, and it can be configured with a custom threshold using the --disk-usage-threshold option.

Warnings are generated if disk usage increases beyond the specified threshold, allowing for targeted investigation of resource management in specific tests, modules, or the entire test session.

Currently, ONLY activated in integration tests for

Related issue/s

• @sanderegg, @matusdrobuliak66 and me agreed that it would be good to have some fixtures to assist finding which tests are not cleaning up artifacts behind them.

How to test

• In simcore-sdk/tests add test_foo.py

import random

def test_it():
    with open(f"/tmp/test_{random.random()}_file.txt", "wb") as f:
        f.write(b"\0" * 1024 * 1024)

• then pytest --disk-usage --disk-usage-threshold=1 --log-cli-level=WARNING tests/unit/test_foo.py
  

Example

in this CI failure

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.1%. Comparing base (cafbf96) to head (d4cc986).
Report is 442 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #6196      +/-   ##
=========================================
+ Coverage    84.5%   88.1%    +3.5%     
=========================================
  Files          10    1463    +1453     
  Lines         214   60731   +60517     
  Branches       25    1484    +1459     
=========================================
+ Hits          181   53547   +53366     
- Misses         23    6871    +6848     
- Partials       10     313     +303     

Flag	Coverage Δ
integrationtests	64.8% <ø> (?)
unittests	86.1% <ø> (+1.5%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 1418 files with indirect coverage changes

[pcrespov]

I think the culprit is here https://github.com/ITISFoundation/osparc-simcore/actions/runs/10408869816/job/28827529734?pr=6196#step:13:241

This test-suite (i.e. test_node_data_manager) seems to accumulate a lot of data after the test

Check all the modules and tests in the CI log starting with WARNING Disk usage increased

[sanderegg]

very nice addition! thanks!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud