Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [Devops] When using Dinky dev version ,data sources that support HDFS to manage paimon #3872

Closed
2 of 3 tasks
dsanww opened this issue Oct 15, 2024 · 5 comments
Closed
2 of 3 tasks
Assignees
Labels
Bug Something isn't working Need Reproduce Need to provide reproduction steps

Comments

@dsanww
Copy link

dsanww commented Oct 15, 2024

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

When using Dinky dev version(commit:114cc18179bb857c93c34324b32438f97ef8af1c) ,data sources that support HDFS to manage paimon
An error was reported while submitting the paimon task:
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.paimon.utils.HadoopUtils.readHadoopXml(HadoopUtils.java:220) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.utils.HadoopUtils.addHadoopConfIfFound(HadoopUtils.java:175) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.utils.HadoopUtils.getHadoopConfiguration(HadoopUtils.java:113) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.catalog.CatalogContext.(CatalogContext.java:53) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.catalog.CatalogContext.create(CatalogContext.java:73) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.FlinkCatalogFactory.createPaimonCatalog(FlinkCatalogFactory.java:81) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.action.ActionBase.initPaimonCatalog(ActionBase.java:78) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.action.ActionBase.(ActionBase.java:65) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.action.cdc.SynchronizationActionBase.(SynchronizationActionBase.java:77) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.action.cdc.SyncDatabaseActionBase.(SyncDatabaseActionBase.java:68) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseAction.(MySqlSyncDatabaseAction.java:109) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseActionFactory.createAction(MySqlSyncDatabaseActionFactory.java:52) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseActionFactory.createAction(MySqlSyncDatabaseActionFactory.java:31) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.action.cdc.SynchronizationActionFactoryBase.create(SynchronizationActionFactoryBase.java:45) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.action.cdc.SyncDatabaseActionFactoryBase.create(SyncDatabaseActionFactoryBase.java:47) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.action.ActionFactory.createAction(ActionFactory.java:87) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at org.apache.paimon.flink.action.FlinkActions.main(FlinkActions.java:43) ~[paimon-flink-1.18-1.0-20241009.jar:1.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.client.program.PackagedProgramUtils.getPipelineFromProgram(PackagedProgramUtils.java:145) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.18.0]
at org.dinky.trans.dml.ExecuteJarOperation.getStreamGraph(ExecuteJarOperation.java:118) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?]
at org.dinky.trans.dml.ExecuteJarOperation.getStreamGraph(ExecuteJarOperation.java:84) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?]
at org.dinky.app.flinksql.Submitter.executeJarJob(Submitter.java:263) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?]
at org.dinky.app.flinksql.Submitter.submit(Submitter.java:142) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?]
at org.dinky.app.MainApp.main(MainApp.java:53) ~[dinky-app-1.18-1.2.0-SNAPSHOT-jar-with-dependencies.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.18.0.jar:1.18.0]

What you expected to happen

no error

How to reproduce

1、Set dinky data source paimon to HDFS and specify the hdfs configuration path
2、run paimon with flink application job on dinky

Anything else

No response

Version

dev

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@dsanww dsanww added Bug Something isn't working Waiting for reply Waiting for reply labels Oct 15, 2024
@dsanww dsanww changed the title [Bug] [Module Name] paimon [Bug] [DEv] When using Dinky dev version ,data sources that support HDFS to manage paimon Oct 15, 2024
@dsanww dsanww changed the title [Bug] [DEv] When using Dinky dev version ,data sources that support HDFS to manage paimon [Bug] [Devops] When using Dinky dev version ,data sources that support HDFS to manage paimon Oct 15, 2024
Copy link

Hello @dsanww, this issue is about CDC/CDCSOURCE, so I assign it to @aiwenmo. If you have any questions, you can comment and reply.

你好 @dsanww, 这个 issue 是关于 CDC/CDCSOURCE 的,所以我把它分配给了 @aiwenmo。如有任何问题,可以评论回复。

@aiwenmo aiwenmo removed the Waiting for reply Waiting for reply label Oct 18, 2024
@aiwenmo
Copy link
Contributor

aiwenmo commented Oct 21, 2024

What's you sql?

@dsanww
Copy link
Author

dsanww commented Oct 21, 2024

What's you sql?

paimon's DatabaseSync Action Jar

@Zzm0809 Zzm0809 added the Need Reproduce Need to provide reproduction steps label Oct 23, 2024
Copy link

Hello @dsanww, we need you to provide reproduction steps so that we can help you troubleshoot the problem. Issues are marked `Need Reproduce`.

你好 @dsanww, 我们需要你提供复现步骤以便于我们帮你排查问题,Issues 标注为 `Need Reproduce` 。

Copy link

github-actions bot commented Nov 3, 2024

Hi @, Because this issue was marked as requiring detailed reproduction steps, no response was received for 10 days. The issue is now closed. If you have any questions, please comment and reply.

Hi @, 由于该 issue 被标记为需要提供详细复现步骤,却 10 天未收到回应。现关闭 issue,若有任何问题,可评论回复。

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working Need Reproduce Need to provide reproduction steps
Projects
None yet
Development

No branches or pull requests

3 participants