[v15] Prevent deadlocking event-handler on missing recordings #44772
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
If the number of session.upload events without a corresponding session recording is higher than the configured concurrency, then the event-handler could deadlock while attempting to find said recording. To prevent this from occurring processing of session recordings has been modified such that: - if a recording does not exist and the session.upload event is from more than 48 hours in the past it is assumed to be lost and no more attempts to process the recording will happen - all processing of recordings that were not found is now done in a separate background routine, instead of inline with event processing The storage directory will now contain information about sessions with a missing recording. The background routine will process them periodically, capping the number of attempts per recording at 3.
rosstimothy
changed the title
tigrato
approved these changes
Jul 29, 2024
marcoandredinis
approved these changes
Jul 29, 2024
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport #44659 to branch/v15
Changelog: Reduced the probability that the event-handler deadlocks when encountering errors processing session recordings