widen both sides of the witness loader window #599
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andymck
force-pushed
the
andymck/extend-witness-loader-window
branch
from
d2441b0
to
264a908
Compare
andymck
force-pushed
the
andymck/extend-witness-loader-window
branch
from
264a908
to
1fc76a1
Compare
andymck
changed the title
bbalser
approved these changes
Aug 16, 2023
jeffgrunewald
approved these changes
Aug 17, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change originates from an investigation into a community reported missing witness. The community member was able to demonstrate a witness being reported to a received beacon but the same witness was not on the selected or unselected lists for the corresponding beacon.
I was able to locate the specific beacon and witness reports from within the ingestor s3 files.
ingestor receive times:
beacon: 1691723926989 ( Friday, 11 August 2023 03:18:46.989 )
witness: 1691723930291 ( Friday, 11 August 2023 03:18:50.291 )
ingestor output files:
the beacon ended up in a beacon roll up file outputted by the ingestor to s3 with a timestamp of
1691723570113 ( Friday, 11 August 2023 03:12:50.113 )
the witness ended up in a witness roll up file outputted by the ingestor to s3 with a timestamp of
witness is in 1691723930291 ( Friday, 11 August 2023 03:18:50.291 )
Note the 6 min difference in the beacon and witness roll up files.
Summary
Whilst processing these files from S3, the iot verifier oracle utilises a sliding window with a width of 5 mins. It will first process all beacon files within this window width and generate an xor filter.
It then takes with same window width and widens it by extending it 5 mins backwards and 5 mins forward. So the witness loading window width is now 15 mins. All witnesses for the previously
processed beacons are expected to be in an S3 roll up file within the time period defined by this window. If so they will be paired with the beacons via the xor filter. If they are outside of this window they will end up being dropped.
The witness in question was outside of the witness window width, ie it was in a witness roll up file with a timestamp of 6mins later than the beacon window end time and thus beyond the 15 mins witness window. This was due to an edge case in the way we write files to S3 with the witness being processed by the ingestor just after a roll up file was completed ( the ingestor writes roll up files to s3 at 5 min intervals ) and thus forcing it into the next roll up file. The pushed it out to being > 5 mins of the beacon file.
The fix is to extend the witness window width on both sides to include a buffer so that it can incorporate up to two rollups of witness reports beyond that of the beacon report. With this change the witness window width will be extended from 15 mins to 19 mins.