refactor progress based basebackup metrics

[sebinsunny]

This PR refactors the basebackups monitoring introduced in PR #615. Previously, we reset the basebackup progress file whenever a new basebackup request was made, which resulted in not catching a few cases where pghoard restarts. Now, the progress file is only reset when a backup is successful, and we also record the total bytes uploaded in the file for the previous basebackup. If there is a retry due to a pghoard restart or a failed backup request, we check if progress has been made; if it has not exceeded the bytes uploaded in the previous state, we emit a stalled metric. Also, added logging for upload progress for each file and snapshot stages in a basebackup operation.

[SRE-7476]

About this change - What it does

Resolves: #xxxxx

Why this way

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 84.21053% with 3 lines in your changes missing coverage. Please review.

Project coverage is 90.80%. Comparing base (5505b86) to head (56098ae).
Report is 11 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #622      +/-   ##
==========================================
- Coverage   91.01%   90.80%   -0.21%     
==========================================
  Files          31       31              
  Lines        4917     4968      +51     
==========================================
+ Hits         4475     4511      +36     
- Misses        442      457      +15     

Files	Coverage Δ
pghoard/basebackup/base.py	92.25% <100.00%> (ø)
pghoard/basebackup/delta.py	90.87% <0.00%> (-0.34%)	⬇️
pghoard/transfer.py	94.51% <87.50%> (-1.28%)	⬇️

... and 4 files with indirect coverage changes