Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](tvf) fix azure tvf: can not build s3() (#45872) #46382

Merged
merged 1 commit into from
Jan 4, 2025

Conversation

kaijchen
Copy link
Contributor

@kaijchen kaijchen commented Jan 3, 2025

backport #45872

### What problem does this PR solve?

Issue Number: DORIS-17644

Related PR: apache#37240

Problem Summary:

`properties` is an immutable map, trying to modify it will cause
`UnsupportedOperationException` to be thrown.

```
2024-12-24 18:01:22,287 WARN (mysql-nio-pool-0|490) [StmtExecutor.executeByNereids():836] Nereids plan query failed:
SELECT * FROM S3 ( ... )
org.apache.doris.nereids.exceptions.AnalysisException: Can not build s3(): null
        at org.apache.doris.nereids.trees.expressions.functions.table.S3.toCatalogFunction(S3.java:55) ~[doris-fe.jar:1.2-SNAPSHOT]
        at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:186) ~[guava-33.2.1-jre.jar:?]
        at org.apache.doris.nereids.trees.expressions.functions.table.TableValuedFunction.lambda$new$0(TableValuedFunction.java:55) ~[doris-fe.jar:1.2-SNAPSHOT]
        at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:186) ~[guava-33.2.1-jre.jar:?]
        at org.apache.doris.nereids.trees.expressions.functions.table.TableValuedFunction.getTable(TableValuedFunction.java:103) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.logical.LogicalTVFRelation.computeOutput(LogicalTVFRelation.java:105) ~[doris-fe.jar:1.2-SNAPSHOT]
        at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:186) ~[guava-33.2.1-jre.jar:?]
        at org.apache.doris.nereids.properties.LogicalProperties.getOutput(LogicalProperties.java:104) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.AbstractPlan.getOutput(AbstractPlan.java:169) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.util.PlanUtils.fastGetChildrenOutputs(PlanUtils.java:161) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.rules.analysis.BindExpression.buildSimpleExprAnalyzer(BindExpression.java:1241) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.rules.analysis.BindExpression.bindProject(BindExpression.java:626) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.pattern.PatternMatcher$1.transform(PatternMatcher.java:92) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.rules.AppliedAwareRule.transform(AppliedAwareRule.java:55) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.jobs.rewrite.PlanTreeRewriteJob.rewrite(PlanTreeRewriteJob.java:57) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.jobs.rewrite.PlanTreeRewriteBottomUpJob.rewriteThis(PlanTreeRewriteBottomUpJob.java:91) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.jobs.rewrite.PlanTreeRewriteBottomUpJob.execute(PlanTreeRewriteBottomUpJob.java:75) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.jobs.scheduler.SimpleJobScheduler.executeJobPool(SimpleJobScheduler.java:44) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.jobs.rewrite.RootPlanTreeRewriteJob.execute(RootPlanTreeRewriteJob.java:66) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.jobs.executor.AbstractBatchJobExecutor.execute(AbstractBatchJobExecutor.java:139) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.jobs.executor.Analyzer.analyze(Analyzer.java:87) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.lambda$analyze$4(NereidsPlanner.java:361) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.keepOrShowPlanProcess(NereidsPlanner.java:888) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.analyze(NereidsPlanner.java:361) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.planWithoutLock(NereidsPlanner.java:250) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.planWithLock(NereidsPlanner.java:224) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:145) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:830) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:609) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:572) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:562) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:347) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:250) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:209) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:237) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:417) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.lang.UnsupportedOperationException
        at com.google.common.collect.ImmutableMap.put(ImmutableMap.java:814) ~[guava-33.2.1-jre.jar:?]
        at org.apache.doris.tablefunction.S3TableValuedFunction.<init>(S3TableValuedFunction.java:66) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.expressions.functions.table.S3.toCatalogFunction(S3.java:53) ~[doris-fe.jar:1.2-SNAPSHOT]
        ... 39 more
```
@Thearas
Copy link
Contributor

Thearas commented Jan 3, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaijchen
Copy link
Contributor Author

kaijchen commented Jan 3, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41153 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9809f163e5e420d3f415afdd5a537ace9aba2628, data reload: false

------ Round 1 ----------------------------------
q1	17739	7671	7234	7234
q2	2047	162	174	162
q3	10642	1109	1137	1109
q4	10866	764	803	764
q5	8100	2907	2925	2907
q6	235	150	153	150
q7	991	623	638	623
q8	9473	1999	2100	1999
q9	6786	6478	6491	6478
q10	8749	2295	2301	2295
q11	461	272	264	264
q12	417	216	211	211
q13	17787	3007	2997	2997
q14	237	218	216	216
q15	553	524	519	519
q16	677	620	625	620
q17	990	534	518	518
q18	7466	6755	6736	6736
q19	1409	1085	1073	1073
q20	477	199	196	196
q21	3999	3222	3081	3081
q22	1112	1008	1001	1001
Total cold run time: 111213 ms
Total hot run time: 41153 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7306	7265	7229	7229
q2	332	238	243	238
q3	3224	3015	3007	3007
q4	2041	1840	1799	1799
q5	5691	5723	5742	5723
q6	224	139	141	139
q7	2266	1892	1814	1814
q8	3412	3531	3521	3521
q9	8897	8913	8913	8913
q10	3617	3574	3536	3536
q11	600	527	487	487
q12	809	598	604	598
q13	6480	3057	3008	3008
q14	298	259	261	259
q15	574	516	512	512
q16	702	664	665	664
q17	1802	1592	1555	1555
q18	7855	7631	7353	7353
q19	1645	1623	1545	1545
q20	2058	1809	1828	1809
q21	5449	5299	5242	5242
q22	1112	1045	1011	1011
Total cold run time: 66394 ms
Total hot run time: 59962 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191428 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9809f163e5e420d3f415afdd5a537ace9aba2628, data reload: false

query1	977	374	379	374
query2	6508	2138	2058	2058
query3	6703	218	215	215
query4	33931	23348	23429	23348
query5	4333	462	460	460
query6	280	191	180	180
query7	4628	319	312	312
query8	291	228	236	228
query9	9759	2728	2697	2697
query10	487	272	266	266
query11	18254	15246	15201	15201
query12	154	103	101	101
query13	1640	442	416	416
query14	10334	7328	6376	6376
query15	290	178	182	178
query16	8190	470	450	450
query17	1638	559	532	532
query18	2155	309	309	309
query19	363	150	146	146
query20	115	106	113	106
query21	61	44	45	44
query22	4367	4612	4269	4269
query23	34780	34127	34043	34043
query24	11180	2873	2850	2850
query25	674	413	406	406
query26	1245	168	172	168
query27	2726	296	301	296
query28	8091	2455	2460	2455
query29	866	456	433	433
query30	330	174	174	174
query31	1015	796	814	796
query32	101	59	59	59
query33	787	287	299	287
query34	952	508	517	508
query35	907	718	730	718
query36	1097	943	973	943
query37	142	77	71	71
query38	4208	3865	3845	3845
query39	1464	1451	1432	1432
query40	148	86	85	85
query41	56	51	54	51
query42	117	102	103	102
query43	535	474	472	472
query44	1258	821	803	803
query45	186	170	173	170
query46	1168	718	726	718
query47	1983	1853	1862	1853
query48	483	390	382	382
query49	1183	394	403	394
query50	811	408	405	405
query51	7365	7198	7067	7067
query52	113	86	96	86
query53	263	185	181	181
query54	1314	471	473	471
query55	75	76	79	76
query56	284	274	256	256
query57	1260	1126	1110	1110
query58	252	209	221	209
query59	3167	2969	3049	2969
query60	301	271	267	267
query61	136	130	156	130
query62	854	652	685	652
query63	213	188	183	183
query64	5650	734	654	654
query65	3330	3201	3195	3195
query66	1473	318	317	317
query67	15966	15719	15565	15565
query68	4450	578	573	573
query69	451	267	262	262
query70	1092	1132	1111	1111
query71	327	255	252	252
query72	6513	4248	4092	4092
query73	749	346	350	346
query74	10261	9023	9019	9019
query75	3376	2652	2688	2652
query76	2799	1038	1004	1004
query77	445	285	266	266
query78	10532	9667	9743	9667
query79	2438	620	613	613
query80	1187	432	423	423
query81	545	253	242	242
query82	819	116	121	116
query83	233	153	141	141
query84	241	80	83	80
query85	1428	305	304	304
query86	440	302	297	297
query87	4484	4334	4292	4292
query88	4236	2398	2368	2368
query89	397	290	301	290
query90	2022	185	181	181
query91	190	153	170	153
query92	61	49	51	49
query93	1617	550	553	550
query94	943	298	303	298
query95	356	249	251	249
query96	602	285	281	281
query97	3352	3221	3218	3218
query98	218	199	195	195
query99	1542	1300	1271	1271
Total cold run time: 304313 ms
Total hot run time: 191428 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9809f163e5e420d3f415afdd5a537ace9aba2628, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.49	0.50	0.51
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.51	0.51
query10	0.55	0.56	0.56
query11	0.14	0.11	0.13
query12	0.14	0.11	0.10
query13	0.62	0.59	0.59
query14	3.06	2.95	2.94
query15	0.87	0.83	0.82
query16	0.39	0.36	0.39
query17	1.09	1.00	0.96
query18	0.22	0.22	0.22
query19	1.83	1.87	1.97
query20	0.02	0.01	0.01
query21	15.36	0.56	0.58
query22	2.75	2.75	2.01
query23	16.83	1.09	0.77
query24	3.26	1.90	1.26
query25	0.19	0.11	0.07
query26	0.74	0.13	0.13
query27	0.05	0.04	0.03
query28	9.75	1.10	1.07
query29	12.58	3.24	3.22
query30	0.24	0.06	0.06
query31	2.85	0.38	0.38
query32	3.27	0.46	0.47
query33	2.97	2.99	3.07
query34	17.11	4.41	4.47
query35	4.51	4.46	4.52
query36	0.68	0.48	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.03	0.04	0.03
Total cold run time: 106.77 s
Total hot run time: 33.21 s

@yiguolei yiguolei merged commit 9161c05 into apache:branch-3.0 Jan 4, 2025
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants