Fix bugs and add features for hive integration #1565
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vegetableysm
changed the title
This comment was marked as resolved.
This comment was marked as resolved.
vegetableysm
force-pushed
the
hive-integration
branch
from
d919332 to
ef541f4
Compare
This comment was marked as resolved.
This comment was marked as resolved.
vegetableysm
force-pushed
the
hive-integration
branch
3 times, most recently
from
3c83f7a to
9c12349
Compare
… process will overwrite the content of table instead of appending. Fix bug: spark will rename a empty file and it will trigle a exception because the objectID is invalid. Support char and varchar. Fix bug that getSplits will try to open a dir. Signed-off-by: vegetableysm <[email protected]>
Fix the bug of renaming table(Including partitioned table). Fixed a bug that compound queries don't work. Signed-off-by: vegetableysm <[email protected]>
… when the file is deleted. Signed-off-by: vegetableysm <[email protected]>
Signed-off-by: vegetableysm <[email protected]>
Signed-off-by: vegetableysm <[email protected]>
Refactor setValue and getValue. Refactor string, varchar, char and binary type. Signed-off-by: vegetableysm <[email protected]>
Signed-off-by: vegetableysm <[email protected]>
Signed-off-by: vegetableysm <[email protected]>
Signed-off-by: vegetableysm <[email protected]>
Signed-off-by: vegetableysm <[email protected]>
Signed-off-by: vegetableysm <[email protected]>
Signed-off-by: vegetableysm <[email protected]>
vegetableysm
force-pushed
the
hive-integration
branch
from
9c12349 to
198b553
Compare
Signed-off-by: vegetableysm <[email protected]>
vegetableysm
force-pushed
the
hive-integration
branch
from
198b553 to
9465f4c
Compare
sighingnow
requested changes
Nov 13, 2023
| + tableFilePath | ||
| + ", content: " | ||
| + new String(buffer, StandardCharsets.UTF_8)); | ||
| break; |
There was a problem hiding this comment.
Don't use instanceof to check the exception type, rather, use Multiple-exception catches instead.
| + new String(buffer, StandardCharsets.UTF_8)); | ||
| break; | ||
| } | ||
| throw e; |
| case CHAR: | ||
| case VARCHAR: | ||
| return Types.MinorType.LARGEVARCHAR.getType(); | ||
| return Types.MinorType.VARCHAR.getType(); |
There was a problem hiding this comment.
Use large var char for string types to avoid exceeding the size limit of arrow buffers in VarCharArray.
| meta.addMember("buffer_", buffer.seal(client)); | ||
| meta.addMember("null_bitmap_", BufferBuilder.empty(client)); | ||
| meta.addMember("data_buffer_", dataBufferBuilder.seal(client)); | ||
| meta.addMember("validity_buffer_", validityBufferBuilder.seal(client)); |
There was a problem hiding this comment.
Don't rename these key names. They MUST be kept consistent with the builders/resolvers in existing C++/Python/Go/Rust SDKs.
| .resolve( | ||
| meta.getMemberMeta( | ||
| "buffer_" + String.valueOf(i) + "_"))) | ||
| .getBuffer()); |
There was a problem hiding this comment.
Can you refactor this part (e.g., use some temporary variables) to make it looks better?
| public ObjectTransformer() {} | ||
|
|
||
| public Object defaultTransform(Object object) { | ||
| return object; |
There was a problem hiding this comment.
defaultTransform -> transform.
|
|
||
| public int intTransform(Object object) { | ||
| return (int) object; | ||
| } |
There was a problem hiding this comment.
- intTransform -> transformInt
- longTransform -> transformLong
- ...
|
Once ready for review, remember removing the "WIP" in the RP's title. |
vegetableysm
changed the title
Signed-off-by: vegetableysm <[email protected]>
sighingnow
approved these changes
Nov 14, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What do these changes do?
Fix bugs and add features based on test results
Related issue number
Fixes #issue number