[format] Format prefix in option keys are processed in each format #4245
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JingsongLi
reviewed
Sep 24, 2024
|
|
||
| public AvroFileFormat(FormatContext context) { | ||
| super(IDENTIFIER); | ||
| this.context = context; | ||
|
|
||
| this.options = getIdentifierPrefixOptions(context.options(), false); |
There was a problem hiding this comment.
Can you also modify AVRO_OUTPUT_CODEC and AVRO_ROW_NAME_MAPPING to add prefix?
wwj6591812
reviewed
Sep 24, 2024
| String prefix = formatIdentifier.toLowerCase() + "."; | ||
| for (String key : options.keySet()) { | ||
| if (key.toLowerCase().startsWith(prefix)) { | ||
| String substr = key.substring(prefix.length()); |
There was a problem hiding this comment.
Change substr to another better name?
| @@ -62,11 +62,14 @@ public class AvroFileFormat extends FileFormat { | |||
| public static final ConfigOption<Map<String, String>> AVRO_ROW_NAME_MAPPING = | |||
There was a problem hiding this comment.
In AvroFileFormat, IDENTIFIER && AVRO_OUTPUT_CODEC && AVRO_ROW_NAME_MAPPING should be private.
| public static FileFormat fromIdentifier(String identifier, Options options) { | ||
| return fromIdentifier(identifier, new FormatContext(options, 1024, 1024)); | ||
| return fromIdentifier( |
There was a problem hiding this comment.
This called method fromIdentifier should be private.
wwj6591812
reviewed
Sep 24, 2024
| @@ -47,8 +47,8 @@ public void testAbsent() { | |||
| @Test | |||
| public void testPresent() { | |||
| Options options = new Options(); | |||
| options.setString("haha", "1"); | |||
| options.setString("compress", "zlib"); | |||
| options.setString("orc.haha", "1"); | |||
There was a problem hiding this comment.
Change haha to other normal orc option?
|
Good work! |
|
+1 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Currently, format prefix in option keys are removed before sending into the constructor of each format. However, most formats (orc, parquet) still recover these prefixes in their code, so it would be better to let the format itself deal with these prefixes.
Also, we're going to introduce a special
compacted changelogformat, which wraps other file formats. However with the current code, each format can only see the options related to itself. If option keys are processed in each format, we can see options for other formats and keep what we need.This PR process option keys in each format.
Tests
Existing tests should cover this change.
API and Format
No format changes.
Documentation
No new feature.