Delta Exporter action: Handle vacuumed objects correctly #8409
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jonathan-Rosenberg
added
include-changelog
E2E Test Results - DynamoDB Local - Local Block Adapter |
E2E Test Results - Quickstart |
guy-har
approved these changes
Dec 10, 2024
| if entry.remove ~= nil then | ||
| -- If the object is not found, and the entry is a remove entry, we can assume it was vacuumed | ||
| print(string.format("Object with path '%s' of a `remove` entry wasn't found. Assuming vacuum.", unescaped_path)) | ||
| unfound_paths[unescaped_path] = nil |
There was a problem hiding this comment.
If it was an add entry (meaning the clause beneath) and a stat object failed, there might still be a remove entry with the same path further ahead which will cause a vacuum to delete it, and in that case we don't want to fail the process, thus we add it to the unfound_paths table.
This line indicates that if a remove occurred and a stat object failed, we should treat any previously added entries as vacuumed objects.
The test on line 155 checks if there were any failed add entry path stat object requests (meaning that they weren't removed by a remove entry), and if that's the case throw an error.
Jonathan-Rosenberg
deleted the
fix/delta-exporter/ignore-unfound-removed-entries
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When traversing a delta lake log, we might come across entries of objects that were vacuumed. Prior to this change, we would stat all of the entry paths to get their physical address, even if those paths were vacuumed. This would cause a 404 "not found" status code to be returned, and the hook would error.
This fix handles such cases.
The resulting exported delta log will include logical paths of vacuumed objects but that doesn't matter for either reading a table, nor future vacuums.
How was it tested?
Manually: created a table, added data, altered the table such that the delta log included a path under an
addentry (in the0000...0.jsonfile) and the same path under aremoveentry (in the0000...1.jsonfile). I've deleted the path manually, thus mimicking vacuum, and committed which made the hook run.Finally, verified that the resulting log is as expected, and read the table using Spark.