upload event tracker #605
Merged
upload event tracker #605
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kathia-barahona
commented
Oct 4, 2023
kathia-barahona
commented
Oct 4, 2023
kathia-barahona
force-pushed
the
track_wal_file_upload_progress
branch
4 times, most recently
from
e7f57aa to
8cfc277
Compare
kathia-barahona
commented
Oct 4, 2023
kathia-barahona
force-pushed
the
track_wal_file_upload_progress
branch
2 times, most recently
from
68f3061 to
c2bd30d
Compare
kathia-barahona
force-pushed
the
track_wal_file_upload_progress
branch
from
c2bd30d to
c8a449d
Compare
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #605 +/- ##
==========================================
- Coverage 91.33% 91.13% -0.21%
==========================================
Files 32 32
Lines 4731 4814 +83
==========================================
+ Hits 4321 4387 +66
- Misses 410 427 +17
|
rikonen
reviewed
Oct 5, 2023
kathia-barahona
force-pushed
the
track_wal_file_upload_progress
branch
from
c8a449d to
11bfcd7
Compare
kathia-barahona
force-pushed
the
track_wal_file_upload_progress
branch
3 times, most recently
from
7f64f28 to
7efc655
Compare
rdunklau
reviewed
Oct 19, 2023
rikonen
reviewed
Oct 29, 2023
kathia-barahona
force-pushed
the
track_wal_file_upload_progress
branch
from
7efc655 to
f0e10b8
Compare
rikonen
previously requested changes
Nov 26, 2023
kathia-barahona
force-pushed
the
track_wal_file_upload_progress
branch
from
f0e10b8 to
6dfb7e1
Compare
rikonen
reviewed
Nov 30, 2023
pghoard/transfer.py
Outdated
|
|
||
| self._tracked_events[file_key].increments.append(TransferIncrement(total_bytes_uploaded=total_bytes_uploaded)) | ||
|
|
||
| self.metrics.increase(**metric_data) |
There was a problem hiding this comment.
Looks like it's possible for metric_data to be empty dict, should skip the call in that case.
There was a problem hiding this comment.
mmm true, tho should it be...? Otherwise we are uploading files that we are not tracking 🤔
kathia-barahona
force-pushed
the
track_wal_file_upload_progress
branch
2 times, most recently
from
d3d2fb1 to
ab97d26
Compare
alexole
reviewed
Jan 3, 2024
alexole
reviewed
Jan 3, 2024
|
Tiny comments, otherwise looks reasonable to me. |
Introduces UploadEventProgressTracker thread for monitoring the progress for each individual UploadEvent and logs a warning if an event has no progress for a specified period of time. It helps ensure that file uploads are making progress and do not get stuck.
sebinsunny
force-pushed
the
track_wal_file_upload_progress
branch
from
ab97d26 to
f64ffb5
Compare
alexole
approved these changes
Jan 23, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
About this change - What it does
Introduces
UploadEventProgressTrackerthread for monitoring the progress for each individual UploadEvent and logs a warning if an event has no progress for a specified period of time. It helps ensure that file uploads are makingprogress and do not get stuck.
Resolves: #BF-2157
Why this way
We currently just track the amount of increments, but cannot identify if an upload got stuck.