Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DevelopmentServer does not work with clickhouse connector #13027

Closed
guiyanakuang opened this issue Jun 29, 2022 · 3 comments
Closed

DevelopmentServer does not work with clickhouse connector #13027

guiyanakuang opened this issue Jun 29, 2022 · 3 comments

Comments

@guiyanakuang
Copy link
Member

It looks like the dependencies with classifier cannot be excluded from airlift/reslover and I have made a pr here. Please help me review it.

The details are as follows：

In the trino 386 version, ck connector. we define the following in the root pom.xml

trino/pom.xml

Lines 1020 to 1031 in c9f881f

<dependency>
<groupId>com.clickhouse</groupId>
<artifactId>clickhouse-jdbc</artifactId>
<version>0.3.2-patch3</version>
<classifier>all</classifier>
<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>

From the logs during debugging development using DevelopmentServer we can see that.

2022-06-26T13:33:12.850+0800	DEBUG	startup-0	io.trino.server.PluginManager	    /Users/user/.m2/repository/com/clickhouse/clickhouse-grpc-client/0.3.2-patch3/clickhouse-grpc-client-0.3.2-patch3-netty.jar
2022-06-26T13:33:12.850+0800	DEBUG	startup-0	io.trino.server.PluginManager	    /Users/user/.m2/repository/com/clickhouse/clickhouse-http-client/0.3.2-patch3/clickhouse-http-client-0.3.2-patch3-shaded.jar

clickhouse-grpc-client-0.3.2-patch3-netty.jar and clickhouse-http-client-0.3.2-patch3-shaded.jar are not excluded because they have a classifier set.

clickhouse-jdbc-0.3.2-patch3-all.jar and clickhouse-grpc-client-0.3.2-patch3-netty.jar have classes with the same name but different contents. If com.clickhouse.client.data.ClickHouseLZ4InputStream in clickhouse-grpc-client-0.3.2-patch3-netty.jar is loaded first, an exception will be thrown, which blocks trino local development.

com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2053)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3966)
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4863)
	at io.trino.collect.cache.EvictableCache.get(EvictableCache.java:108)
	at io.trino.plugin.jdbc.CachingJdbcClient.get(CachingJdbcClient.java:752)
	at io.trino.plugin.jdbc.CachingJdbcClient.getSchemaNames(CachingJdbcClient.java:148)
	at io.trino.plugin.jdbc.CachingJdbcClient.schemaExists(CachingJdbcClient.java:141)
	at io.trino.plugin.jdbc.DefaultJdbcMetadata.schemaExists(DefaultJdbcMetadata.java:116)
	at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.schemaExists(ClassLoaderSafeConnectorMetadata.java:174)
	at io.trino.metadata.MetadataManager.lambda$schemaExists$1(MetadataManager.java:224)
	at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958)
	at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127)
	at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
	at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:528)
	at io.trino.metadata.MetadataManager.schemaExists(MetadataManager.java:224)
	at io.trino.sql.rewrite.ShowQueriesRewrite$Visitor.visitShowColumns(ShowQueriesRewrite.java:474)
	at io.trino.sql.rewrite.ShowQueriesRewrite$Visitor.visitShowColumns(ShowQueriesRewrite.java:216)
	at io.trino.sql.tree.ShowColumns.accept(ShowColumns.java:68)
	at io.trino.sql.tree.AstVisitor.process(AstVisitor.java:27)
	at io.trino.sql.rewrite.ShowQueriesRewrite.rewrite(ShowQueriesRewrite.java:213)
	at io.trino.sql.rewrite.StatementRewrite.rewrite(StatementRewrite.java:53)
	at io.trino.sql.analyzer.Analyzer.analyze(Analyzer.java:76)
	at io.trino.sql.analyzer.Analyzer.analyze(Analyzer.java:71)
	at io.trino.execution.SqlQueryExecution.analyze(SqlQueryExecution.java:269)
	at io.trino.execution.SqlQueryExecution.<init>(SqlQueryExecution.java:193)
	at io.trino.execution.SqlQueryExecution$SqlQueryExecutionFactory.createQueryExecution(SqlQueryExecution.java:808)
	at io.trino.dispatcher.LocalDispatchQueryFactory.lambda$createDispatchQuery$0(LocalDispatchQueryFactory.java:135)
	at io.trino.$gen.Trino_dev____20220626_050612_2.call(Unknown Source)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory
	at com.clickhouse.client.data.ClickHouseLZ4InputStream.<clinit>(ClickHouseLZ4InputStream.java:20)
	at com.clickhouse.client.http.ClickHouseHttpConnection.getResponseInputStream(ClickHouseHttpConnection.java:245)
	at com.clickhouse.client.http.HttpUrlConnectionImpl.buildResponse(HttpUrlConnectionImpl.java:65)
	at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:203)
	at com.clickhouse.client.http.ClickHouseHttpClient.postRequest(ClickHouseHttpClient.java:88)
	at com.clickhouse.client.http.ClickHouseHttpClient.execute(ClickHouseHttpClient.java:115)
	at com.clickhouse.client.ClickHouseRequest.execute(ClickHouseRequest.java:1375)
	at com.clickhouse.jdbc.ClickHouseDatabaseMetaData.query(ClickHouseDatabaseMetaData.java:71)
	at com.clickhouse.jdbc.ClickHouseDatabaseMetaData.query(ClickHouseDatabaseMetaData.java:54)
	at com.clickhouse.jdbc.ClickHouseDatabaseMetaData.getSchemas(ClickHouseDatabaseMetaData.java:1215)
	at io.trino.plugin.jdbc.BaseJdbcClient.listSchemas(BaseJdbcClient.java:153)
	at io.trino.plugin.jdbc.BaseJdbcClient.getSchemaNames(BaseJdbcClient.java:142)
	at io.trino.plugin.jdbc.ForwardingJdbcClient.getSchemaNames(ForwardingJdbcClient.java:72)
	at io.trino.plugin.jdbc.jmx.StatisticsAwareJdbcClient.lambda$getSchemaNames$1(StatisticsAwareJdbcClient.java:92)
	at io.trino.plugin.jdbc.jmx.JdbcApiStats.wrap(JdbcApiStats.java:35)
	at io.trino.plugin.jdbc.jmx.StatisticsAwareJdbcClient.getSchemaNames(StatisticsAwareJdbcClient.java:92)
	at io.trino.plugin.jdbc.CachingJdbcClient.lambda$getSchemaNames$1(CachingJdbcClient.java:148)
	at io.trino.collect.cache.EmptyCache.get(EmptyCache.java:66)
	at io.trino.plugin.jdbc.CachingJdbcClient.get(CachingJdbcClient.java:752)
	at io.trino.plugin.jdbc.CachingJdbcClient.getSchemaNames(CachingJdbcClient.java:148)
	at io.trino.plugin.jdbc.CachingJdbcClient.lambda$getSchemaNames$1(CachingJdbcClient.java:148)
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4868)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
	... 40 more
Caused by: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4Factory
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
	at io.trino.server.PluginClassLoader.loadClass(PluginClassLoader.java:126)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 66 more
@ebyhr
Copy link
Member

ebyhr commented Jun 29, 2022

I would recommend using ClickHouseQueryRunner.main() for local development if you don't have special reason to use DevelopmentServer.

@guiyanakuang
Copy link
Member Author

Thank you for the recommendation, I get it.

Implemented some custom redirection optimization rules between different engines based on ApplyTableScanRedirection, so had to test some loading of several different plugins.

@ebyhr
Copy link
Member

ebyhr commented Mar 27, 2023

Closing as we don't have ClickHosue config properties in testing/trino-server-dev/etc/catalog. Please feel free to send a PR to add the file.

@ebyhr ebyhr closed this as not planned Won't fix, can't repro, duplicate, stale Mar 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@guiyanakuang @ebyhr