🎨 EFS Guardian: adding size monitoring

[matusdrobuliak66]

What do these changes do?

• ✨ Adding size monitoring. EFS Guardian subscribes to a queue that listens for heartbeats when a dynamic service is running. Each minute, a heartbeat triggers a monitoring function that checks the current size. If the size exceeds the threshold, write permissions on all files are removed.
• ✨ Prepare the groundwork for a background task that will handle removing data from EFS for projects that have not been accessed for some time.

Related issue/s

• relates Filesystem - Large File Handling (Caching) #6411

How to test

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)

[codecov]

Codecov Report

Attention: Patch coverage is 85.26316% with 28 lines in your changes missing coverage. Please review.

Project coverage is 88.1%. Comparing base (cafbf96) to head (bd5eb5e).
Report is 626 commits behind head on master.

Files with missing lines	Patch %	Lines
..._service_efs_guardian/services/process_messages.py	35.7%	18 Missing ⚠️
...service_efs_guardian/services/efs_manager_utils.py	73.9%	6 Missing ⚠️
...ls-library/src/models_library/rabbitmq_messages.py	77.7%	1 Missing and 1 partial ⚠️
...ce_efs_guardian/services/process_messages_setup.py	94.8%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #6502      +/-   ##
=========================================
+ Coverage    84.5%   88.1%    +3.5%     
=========================================
  Files          10    1546    +1536     
  Lines         214   63324   +63110     
  Branches       25    2059    +2034     
=========================================
+ Hits          181   55807   +55626     
- Misses         23    7199    +7176     
- Partials       10     318     +308     

Flag	Coverage Δ
integrationtests	64.7% <42.8%> (?)
unittests	86.1% <85.2%> (+1.5%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...c/simcore_service_dynamic_sidecar/core/rabbitmq.py	100.0% <100.0%> (ø)
...dynamic_sidecar/modules/resource_tracking/_core.py	96.2% <100.0%> (ø)
...c/simcore_service_efs_guardian/core/application.py	97.2% <100.0%> (ø)
.../src/simcore_service_efs_guardian/core/settings.py	100.0% <100.0%> (ø)
..._service_efs_guardian/services/background_tasks.py	100.0% <100.0%> (ø)
...ce_efs_guardian/services/background_tasks_setup.py	100.0% <100.0%> (ø)
...mcore_service_efs_guardian/services/efs_manager.py	100.0% <100.0%> (ø)
...ore_service_efs_guardian/services/modules/redis.py	100.0% <100.0%> (ø)
...ls-library/src/models_library/rabbitmq_messages.py	89.4% <77.7%> (ø)
...ce_efs_guardian/services/process_messages_setup.py	94.8% <94.8%> (ø)
... and 2 more

... and 1484 files with indirect coverage changes

[GitHK]

All good, please check some of my suggestions

[pcrespov]

This PR

• the dy-sidecar notifies DynamicServiceRunningMessage and the efs-guardian listens
• efs-guardian:
  • prepares background modules (still not fully implemented) which needs redis for distributed lock mechanism
  • EfsManager operates on EFS mounted folder. I understand is a "single drive" and you only manipulate one project/node at a time?
  • process-messages reacts to a service running. If size is over threshold, it remove write premssions. Uses redis as a distributed lock mechanims here as well.

Generally, I think you should give a deeper thought about including good testing and error handling.

[sonarcloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud