Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] cdc table sync with Hive Catalog caseSensitive incorrect #2481

Closed
2 tasks done
gfunc opened this issue Dec 11, 2023 · 2 comments
Closed
2 tasks done

[Bug] cdc table sync with Hive Catalog caseSensitive incorrect #2481

gfunc opened this issue Dec 11, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@gfunc
Copy link

gfunc commented Dec 11, 2023

Search before asking

  • I searched in the issues and found nothing similar.

Paimon version

master branch f68743c

Compute Engine

flink

Minimal reproduce step

Run a KafkaSyncTableAction test with a hive catalog.

What doesn't meet your expectations?

parser created with caseSensitive hard coded to true, not according to the provided catalog's caseSensitive() implementation

https://github.com/apache/incubator-paimon/blob/f68743c1bf429db4b28fccd3e24e23bf0a5266f6/paimon-flink/paimon-flink-cdc/src/main/java/org/apache/paimon/flink/action/cdc/MessageQueueSchemaUtils.java#L53

This will cause case validations to fail due to inconsistency since the hive catalog's caseSensitive() would return false.

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!
@gfunc gfunc added the bug Something isn't working label Dec 11, 2023
@yuzelin
Copy link
Contributor

yuzelin commented Dec 13, 2023

In latest master branch (c9ef520), the following line will handle the case-sensetivity: https://github.com/apache/incubator-paimon/blob/c9ef52026c366b90c539d01a3da909ea3900b154/paimon-flink/paimon-flink-cdc/src/main/java/org/apache/paimon/flink/action/cdc/SyncTableActionBase.java#L101

The modification is from 2c401ac. It reorganized the building flow: first we retrieve schema without caseSensitive; then we build Paimon table schema with caseSensitive. Ant We also add many tests with case-insensitive catalog in that PR. So I think this bug is solved.

@gfunc
Copy link
Author

gfunc commented Dec 14, 2023

Tested OK with SASL enabled kafka and Hive catalog.

@gfunc gfunc closed this as completed Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants