Addresses #29: Support missing version-hint.txt and provide additional options #63
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This intends to address #29 by providing some additional options when finding metadata files to use.
This PR essentially adds two parameters to the iceberg functions:
version
- Either a version number of the table to use, or a "hint" filename that should be read to get the latest version.Default value:
'version-hint.text'
(preserves the previous behavior of looking for aversion-hint.text
fileExamples:
version='12'
: Explicitly load table version 12.version='my-custom-version-file.txt'
: Use version frommy-custom-file.txt
instead ofversion-hint.text
version_name_format
- comma-delimited list of format strings that should be tried to find the metadata file name based on the supplied or hinted version.Default value:
'v%s%s.metadata.json,%s%s.metadata.json
(extends the previous behavior by also checking for a version file without the prefixed "v" in the event the default behavior failed).Examples:
'version-%smetadata.json%s'
would look forversion-42.metadata.json.gz
when specified with(..., version='42', medata_compression_codec='gzip')
As was suggested by @lamb-russell, this PR will (eventually) do the following:
"v"
prefix being added to the version in"version-hint.txt"
with a more flexible printf-style template string that can be specified as a parameter."version-hint.txt"
"version-hint.txt"
)