Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Opt](multi-catalog)Improve performance by introducing cache of list directory files when getting split for each query. #43913

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[Opt](multi-catalog)Improve performance by introducing cache of list directory files when getting split for each query. #43913

wants to merge 1 commit into from

Conversation

kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented Nov 13, 2024 •
edited
Loading

What problem does this PR solve?

Refer to trino to implement the cache mechanism of multiple hive tables at the query level to obtain the file split list of each partition.
Because files within a query should have the same visibility, the split list of partitions that see the same table should be consistent across the query scope. So this cache is reasonable and should be enabled by default.
The mechanism in Trino is transactional level. A transaction can see the same table, so the command is TransactionScopeCachingDirectoryLister. This name is retained for Doris to expand to the transaction concept in the future.
In addition, for this scenario, because the caffeine cache currently used by doris has an elimination phase strategy, the existing cache items in the window area may be eliminated immediately after the weight is updated. Therefore, EvictableCache which based on guava was introduced and eliminated based on segment LRU.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen kaka11chen force-pushed the query_directory_list_cache branch from 65ef2e1 to ecb2339 Compare November 27, 2024 02:46
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39843 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ecb2339563e8dce502d7aa821e01f8dcbb5dabb0, data reload: false

------ Round 1 ----------------------------------
q1	17585	7397	7259	7259
q2	2042	167	161	161
q3	10633	1075	1176	1075
q4	10505	794	724	724
q5	7587	2709	2736	2709
q6	242	148	145	145
q7	988	612	594	594
q8	9238	1851	1922	1851
q9	6528	6355	6398	6355
q10	7016	2311	2371	2311
q11	488	261	253	253
q12	430	230	227	227
q13	17785	3066	3017	3017
q14	249	223	214	214
q15	563	529	504	504
q16	663	591	606	591
q17	988	503	519	503
q18	7308	6775	6733	6733
q19	1353	1038	1063	1038
q20	475	187	184	184
q21	4021	3152	3079	3079
q22	387	316	319	316
Total cold run time: 107074 ms
Total hot run time: 39843 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7229	7263	7260	7260
q2	319	232	224	224
q3	2942	2862	2966	2862
q4	2087	1823	1822	1822
q5	5738	5697	5672	5672
q6	221	144	143	143
q7	2313	1875	1804	1804
q8	3437	3513	3511	3511
q9	8831	8947	8891	8891
q10	3597	3561	3540	3540
q11	595	505	507	505
q12	838	594	618	594
q13	9853	3352	3260	3260
q14	310	278	283	278
q15	568	528	522	522
q16	686	646	643	643
q17	1867	1659	1642	1642
q18	8306	7684	7675	7675
q19	1683	1542	1537	1537
q20	2121	1911	1858	1858
q21	5592	5375	5321	5321
q22	622	601	558	558
Total cold run time: 69755 ms
Total hot run time: 60122 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196094 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ecb2339563e8dce502d7aa821e01f8dcbb5dabb0, data reload: false

query1	1244	942	953	942
query2	6222	2070	2066	2066
query3	10811	4054	3901	3901
query4	67692	29319	23522	23522
query5	4933	460	465	460
query6	408	185	178	178
query7	5523	310	291	291
query8	316	233	231	231
query9	8581	2808	2741	2741
query10	467	263	254	254
query11	17051	15231	16000	15231
query12	168	104	105	104
query13	1448	424	441	424
query14	10573	7925	7855	7855
query15	212	199	199	199
query16	7093	444	456	444
query17	1125	565	567	565
query18	1870	288	300	288
query19	201	177	156	156
query20	125	111	109	109
query21	210	103	102	102
query22	4702	4428	4485	4428
query23	34862	34764	34554	34554
query24	5614	2510	2493	2493
query25	496	404	405	404
query26	648	153	147	147
query27	1874	284	293	284
query28	4696	2517	2490	2490
query29	672	429	418	418
query30	237	153	154	153
query31	1014	857	848	848
query32	67	56	61	56
query33	417	280	285	280
query34	942	518	508	508
query35	852	739	746	739
query36	1132	958	1010	958
query37	129	78	87	78
query38	4573	4410	4506	4410
query39	1528	1465	1441	1441
query40	206	102	103	102
query41	50	42	45	42
query42	108	102	100	100
query43	555	514	503	503
query44	1202	841	835	835
query45	191	173	166	166
query46	1137	727	719	719
query47	2026	1907	1913	1907
query48	435	315	331	315
query49	734	406	424	406
query50	825	401	399	399
query51	7453	7240	7079	7079
query52	99	87	87	87
query53	250	185	182	182
query54	507	398	395	395
query55	78	74	78	74
query56	258	232	242	232
query57	1331	1141	1157	1141
query58	220	214	215	214
query59	3271	3181	2908	2908
query60	284	252	262	252
query61	109	108	105	105
query62	800	666	663	663
query63	216	190	193	190
query64	1417	671	635	635
query65	3293	3224	3219	3219
query66	678	301	308	301
query67	15989	15796	15482	15482
query68	3935	596	565	565
query69	422	257	257	257
query70	1239	1167	1150	1150
query71	337	252	248	248
query72	6464	3985	2557	2557
query73	773	352	367	352
query74	10091	8997	9115	8997
query75	3416	2677	2669	2669
query76	1934	1123	1092	1092
query77	487	269	304	269
query78	10547	9503	9456	9456
query79	1908	594	627	594
query80	1406	447	438	438
query81	522	233	231	231
query82	1316	119	115	115
query83	270	145	150	145
query84	281	70	70	70
query85	1102	303	314	303
query86	419	320	288	288
query87	4824	4653	4516	4516
query88	3773	2265	2240	2240
query89	420	314	297	297
query90	1912	187	189	187
query91	137	101	107	101
query92	68	54	60	54
query93	2307	531	547	531
query94	781	326	287	287
query95	348	238	246	238
query96	629	284	277	277
query97	2840	2664	2698	2664
query98	219	195	196	195
query99	1693	1306	1315	1306
Total cold run time: 321270 ms
Total hot run time: 196094 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ecb2339563e8dce502d7aa821e01f8dcbb5dabb0, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.24	0.08	0.06
query4	1.62	0.10	0.10
query5	0.43	0.41	0.40
query6	1.18	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.51	0.51
query10	0.55	0.54	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	2.85	2.75	2.71
query15	0.91	0.84	0.82
query16	0.39	0.38	0.37
query17	1.04	1.02	1.04
query18	0.22	0.21	0.22
query19	1.82	1.88	1.91
query20	0.01	0.01	0.01
query21	15.36	0.62	0.61
query22	2.59	1.92	1.86
query23	17.23	0.87	0.78
query24	3.25	1.76	0.79
query25	0.24	0.19	0.05
query26	0.54	0.15	0.14
query27	0.06	0.04	0.03
query28	10.07	1.10	1.08
query29	12.61	3.21	3.20
query30	0.25	0.07	0.06
query31	2.88	0.38	0.37
query32	3.30	0.46	0.47
query33	3.01	3.04	3.08
query34	17.00	4.45	4.40
query35	4.57	4.42	4.47
query36	0.65	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.12	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.98 s
Total hot run time: 32.26 s

@kaka11chen kaka11chen force-pushed the query_directory_list_cache branch from ecb2339 to 9c415d4 Compare November 27, 2024 11:20
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39811 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9c415d458d11468625786fde91be97bab3d7f60c, data reload: false

------ Round 1 ----------------------------------
q1	17594	7436	7279	7279
q2	2051	171	166	166
q3	10567	1129	1145	1129
q4	10226	713	716	713
q5	7582	2672	2625	2625
q6	235	149	145	145
q7	993	624	602	602
q8	9237	1855	1892	1855
q9	6576	6411	6351	6351
q10	6937	2258	2277	2258
q11	461	266	259	259
q12	415	228	220	220
q13	17779	2992	3044	2992
q14	247	204	203	203
q15	574	540	517	517
q16	638	586	593	586
q17	972	588	529	529
q18	7502	6726	6701	6701
q19	1345	1039	953	953
q20	468	187	177	177
q21	3990	3230	3251	3230
q22	386	323	321	321
Total cold run time: 106775 ms
Total hot run time: 39811 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7246	7278	7226	7226
q2	324	227	229	227
q3	2883	2769	2974	2769
q4	2160	1866	1849	1849
q5	5534	5673	5677	5673
q6	222	142	138	138
q7	2178	1817	1815	1815
q8	3385	3514	3459	3459
q9	8832	8806	8798	8798
q10	3640	3542	3542	3542
q11	596	522	493	493
q12	811	602	600	600
q13	11906	3196	3189	3189
q14	297	263	264	263
q15	580	531	532	531
q16	664	654	630	630
q17	1854	1629	1553	1553
q18	8040	7416	7383	7383
q19	1653	1571	1481	1481
q20	2044	1858	1807	1807
q21	5261	5192	5199	5192
q22	639	583	546	546
Total cold run time: 70749 ms
Total hot run time: 59164 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190713 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9c415d458d11468625786fde91be97bab3d7f60c, data reload: false

query1	966	380	382	380
query2	6520	2162	2079	2079
query3	6703	211	216	211
query4	34281	23541	23508	23508
query5	4368	462	454	454
query6	310	191	186	186
query7	4619	297	302	297
query8	309	221	225	221
query9	9697	2702	2705	2702
query10	479	277	246	246
query11	18268	15250	15331	15250
query12	154	104	105	104
query13	1664	426	415	415
query14	10578	6857	7007	6857
query15	271	181	182	181
query16	8224	484	492	484
query17	1713	557	532	532
query18	2135	287	289	287
query19	358	143	138	138
query20	115	110	109	109
query21	209	100	110	100
query22	4581	4197	4200	4197
query23	34930	33920	34321	33920
query24	11419	2469	2449	2449
query25	685	368	371	368
query26	1811	144	150	144
query27	2886	278	275	275
query28	8242	2448	2406	2406
query29	1034	400	429	400
query30	304	144	145	144
query31	1021	794	822	794
query32	98	54	57	54
query33	765	273	281	273
query34	993	504	522	504
query35	924	730	731	730
query36	1112	929	940	929
query37	277	92	71	71
query38	4302	4146	4279	4146
query39	1484	1450	1429	1429
query40	274	98	98	98
query41	48	45	45	45
query42	106	99	97	97
query43	531	496	491	491
query44	1332	798	803	798
query45	178	168	170	168
query46	1139	700	685	685
query47	1961	1834	1901	1834
query48	407	318	305	305
query49	1285	402	388	388
query50	791	393	378	378
query51	7384	7166	7218	7166
query52	99	91	89	89
query53	260	178	179	178
query54	1140	394	412	394
query55	86	81	75	75
query56	246	245	240	240
query57	1314	1207	1224	1207
query58	246	215	216	215
query59	3391	3198	3008	3008
query60	271	242	246	242
query61	110	110	110	110
query62	938	677	675	675
query63	216	183	179	179
query64	5134	671	629	629
query65	3274	3162	3215	3162
query66	1427	320	303	303
query67	15951	15628	15848	15628
query68	5002	555	565	555
query69	436	250	255	250
query70	1197	1175	1146	1146
query71	351	257	245	245
query72	6391	4058	4005	4005
query73	763	364	356	356
query74	10500	9171	8941	8941
query75	3468	2674	2657	2657
query76	2904	1155	1040	1040
query77	505	285	275	275
query78	10386	9364	9389	9364
query79	1583	593	599	593
query80	1112	433	429	429
query81	542	240	227	227
query82	941	121	115	115
query83	272	149	148	148
query84	240	67	72	67
query85	1348	301	296	296
query86	447	310	305	305
query87	4680	4634	4591	4591
query88	3451	2231	2186	2186
query89	400	295	302	295
query90	2089	188	186	186
query91	138	102	104	102
query92	65	50	52	50
query93	1117	525	522	522
query94	1116	285	286	285
query95	360	252	248	248
query96	604	289	279	279
query97	2819	2674	2689	2674
query98	215	198	201	198
query99	1566	1339	1322	1322
Total cold run time: 305356 ms
Total hot run time: 190713 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9c415d458d11468625786fde91be97bab3d7f60c, data reload: false

query1	0.03	0.03	0.03
query2	0.10	0.03	0.02
query3	0.23	0.08	0.06
query4	1.64	0.11	0.11
query5	0.43	0.39	0.41
query6	1.17	0.67	0.64
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.58	0.51	0.51
query10	0.56	0.56	0.56
query11	0.14	0.10	0.10
query12	0.13	0.11	0.10
query13	0.60	0.60	0.60
query14	2.73	2.77	2.75
query15	0.90	0.82	0.85
query16	0.39	0.37	0.37
query17	1.01	1.07	1.00
query18	0.23	0.21	0.20
query19	1.98	1.75	1.96
query20	0.01	0.01	0.01
query21	15.36	0.58	0.60
query22	2.56	2.74	1.30
query23	17.08	0.97	0.79
query24	2.93	0.78	1.65
query25	0.22	0.21	0.07
query26	0.53	0.14	0.14
query27	0.05	0.04	0.04
query28	10.46	1.10	1.08
query29	12.54	3.26	3.20
query30	0.25	0.07	0.06
query31	2.86	0.38	0.38
query32	3.28	0.47	0.47
query33	3.06	2.96	2.99
query34	17.12	4.45	4.44
query35	4.48	4.45	4.47
query36	0.67	0.49	0.50
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.15	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.03	0.04	0.03
Total cold run time: 106.83 s
Total hot run time: 31.61 s

@kaka11chen kaka11chen force-pushed the query_directory_list_cache branch from 9c415d4 to 2156ce4 Compare November 27, 2024 14:58
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40191 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2156ce4b15f7666aa4e366336d871a179075b8a3, data reload: false

------ Round 1 ----------------------------------
q1	17732	7597	7321	7321
q2	2050	176	168	168
q3	10631	1150	1207	1150
q4	10303	775	680	680
q5	7615	2786	2759	2759
q6	243	151	148	148
q7	1004	628	606	606
q8	9247	1889	1949	1889
q9	6528	6393	6447	6393
q10	7005	2282	2353	2282
q11	462	252	246	246
q12	410	217	211	211
q13	17803	2981	3007	2981
q14	238	217	207	207
q15	580	534	510	510
q16	647	573	582	573
q17	1003	600	566	566
q18	7434	6760	6823	6760
q19	1338	1015	1088	1015
q20	472	183	180	180
q21	4090	3368	3234	3234
q22	386	312	326	312
Total cold run time: 107221 ms
Total hot run time: 40191 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7275	7262	7336	7262
q2	330	228	234	228
q3	2958	2896	3124	2896
q4	2114	1844	1820	1820
q5	5674	5687	5693	5687
q6	221	141	136	136
q7	2194	1815	1884	1815
q8	3414	3541	3548	3541
q9	8916	9019	9041	9019
q10	3636	3579	3533	3533
q11	601	501	534	501
q12	823	616	635	616
q13	12827	3164	3156	3156
q14	304	278	286	278
q15	551	524	502	502
q16	675	635	622	622
q17	1815	1604	1561	1561
q18	7856	7492	7453	7453
q19	1684	1418	1508	1418
q20	2037	1851	1818	1818
q21	5487	5190	5295	5190
q22	634	557	549	549
Total cold run time: 72026 ms
Total hot run time: 59601 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191340 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2156ce4b15f7666aa4e366336d871a179075b8a3, data reload: false

query1	970	367	372	367
query2	6518	2173	2102	2102
query3	6712	215	203	203
query4	34460	23589	23389	23389
query5	4315	466	449	449
query6	279	186	180	180
query7	4618	292	305	292
query8	294	224	234	224
query9	9665	2743	2719	2719
query10	465	250	251	250
query11	18259	15272	15179	15179
query12	162	113	105	105
query13	1666	416	417	416
query14	9773	7486	7140	7140
query15	305	183	180	180
query16	8111	459	495	459
query17	1829	581	579	579
query18	2122	294	301	294
query19	390	153	149	149
query20	123	149	114	114
query21	207	103	107	103
query22	4470	4164	4184	4164
query23	35138	34277	34304	34277
query24	11096	2450	2361	2361
query25	663	367	385	367
query26	1628	151	145	145
query27	2771	280	275	275
query28	8064	2457	2463	2457
query29	952	405	404	404
query30	299	148	150	148
query31	1064	813	799	799
query32	101	56	55	55
query33	796	278	285	278
query34	978	500	523	500
query35	910	762	748	748
query36	1119	968	979	968
query37	148	72	76	72
query38	4449	4275	4285	4275
query39	1495	1440	1426	1426
query40	284	99	98	98
query41	45	43	42	42
query42	106	93	97	93
query43	535	491	503	491
query44	1276	788	794	788
query45	186	166	164	164
query46	1137	699	685	685
query47	1961	1858	1889	1858
query48	409	302	319	302
query49	1244	384	372	372
query50	807	383	385	383
query51	7337	7160	7247	7160
query52	96	91	88	88
query53	252	188	179	179
query54	1230	390	413	390
query55	74	77	74	74
query56	272	231	245	231
query57	1346	1209	1173	1173
query58	239	216	231	216
query59	3183	3127	3064	3064
query60	280	246	242	242
query61	110	106	104	104
query62	897	663	683	663
query63	208	188	180	180
query64	5078	646	658	646
query65	3277	3211	3217	3211
query66	1138	322	311	311
query67	16028	15554	15688	15554
query68	5093	535	560	535
query69	422	251	258	251
query70	1120	1159	1165	1159
query71	348	251	243	243
query72	6429	4046	3967	3967
query73	778	364	372	364
query74	10402	9098	8970	8970
query75	3431	2639	2658	2639
query76	3005	1123	1133	1123
query77	569	286	283	283
query78	10460	9398	9427	9398
query79	2365	591	610	591
query80	1065	423	430	423
query81	557	233	234	233
query82	651	118	126	118
query83	250	155	145	145
query84	233	75	67	67
query85	1927	300	298	298
query86	493	307	306	306
query87	4762	4502	4545	4502
query88	4166	2235	2204	2204
query89	408	292	290	290
query90	2135	190	193	190
query91	137	102	104	102
query92	66	52	49	49
query93	2142	552	541	541
query94	1073	272	287	272
query95	357	247	252	247
query96	672	277	286	277
query97	2878	2670	2675	2670
query98	229	190	196	190
query99	1570	1336	1319	1319
Total cold run time: 306612 ms
Total hot run time: 191340 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2156ce4b15f7666aa4e366336d871a179075b8a3, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.08	0.07
query4	1.60	0.10	0.11
query5	0.43	0.39	0.42
query6	1.13	0.69	0.70
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.58	0.51	0.53
query10	0.55	0.56	0.56
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.62	0.61	0.61
query14	2.71	2.86	2.83
query15	0.96	0.84	0.85
query16	0.40	0.40	0.38
query17	1.08	1.02	1.09
query18	0.22	0.21	0.20
query19	2.00	1.77	1.99
query20	0.01	0.01	0.01
query21	15.36	0.62	0.60
query22	2.70	2.57	1.10
query23	17.05	0.89	0.78
query24	3.12	1.82	1.07
query25	0.22	0.14	0.14
query26	0.52	0.14	0.13
query27	0.06	0.04	0.04
query28	10.40	1.12	1.11
query29	12.54	3.27	3.23
query30	0.25	0.06	0.06
query31	2.87	0.40	0.40
query32	3.25	0.49	0.49
query33	3.02	3.05	3.08
query34	16.99	4.55	4.58
query35	4.58	4.52	4.57
query36	0.68	0.51	0.50
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.03	0.02
query40	0.16	0.13	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 107.05 s
Total hot run time: 32.37 s

@kaka11chen kaka11chen force-pushed the query_directory_list_cache branch 2 times, most recently from adc3ee6 to d0f5f61 Compare November 28, 2024 03:02
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40086 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d0f5f61a4043bd2069ef63cfa2d757556831a297, data reload: false

------ Round 1 ----------------------------------
q1	17905	7596	7308	7308
q2	2047	178	171	171
q3	10631	1182	1130	1130
q4	10232	692	776	692
q5	7631	2724	2708	2708
q6	241	148	149	148
q7	988	624	611	611
q8	9233	1848	1943	1848
q9	6645	6350	6389	6350
q10	7004	2293	2335	2293
q11	452	267	261	261
q12	424	226	224	224
q13	17795	3057	3067	3057
q14	249	224	209	209
q15	569	547	514	514
q16	650	572	586	572
q17	1000	608	623	608
q18	7490	6899	6757	6757
q19	1365	1010	1016	1010
q20	494	188	185	185
q21	4088	3120	3282	3120
q22	382	311	310	310
Total cold run time: 107515 ms
Total hot run time: 40086 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7298	7232	7260	7232
q2	328	232	227	227
q3	2980	2784	2800	2784
q4	1968	1696	1703	1696
q5	5416	5420	5425	5420
q6	225	138	141	138
q7	2139	1742	1705	1705
q8	3271	3426	3434	3426
q9	8558	8563	8557	8557
q10	3521	3436	3456	3436
q11	593	504	495	495
q12	792	559	593	559
q13	10996	3044	3026	3026
q14	299	259	266	259
q15	563	499	512	499
q16	671	637	641	637
q17	1809	1603	1583	1583
q18	7779	7531	7525	7525
q19	1670	1462	1511	1462
q20	2028	1809	1791	1791
q21	5370	5110	5182	5110
q22	611	569	539	539
Total cold run time: 68885 ms
Total hot run time: 58106 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191103 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d0f5f61a4043bd2069ef63cfa2d757556831a297, data reload: false

query1	977	385	370	370
query2	6511	2128	2126	2126
query3	6712	207	214	207
query4	33988	23558	23715	23558
query5	4403	458	441	441
query6	271	188	176	176
query7	4621	308	304	304
query8	299	236	224	224
query9	9602	2737	2718	2718
query10	480	262	255	255
query11	18164	15342	15175	15175
query12	161	104	104	104
query13	1694	420	418	418
query14	11694	6987	7515	6987
query15	304	170	167	167
query16	8012	446	466	446
query17	2191	561	556	556
query18	2150	279	285	279
query19	361	151	140	140
query20	119	110	112	110
query21	209	99	102	99
query22	4457	4329	4286	4286
query23	35028	35298	34224	34224
query24	11200	2395	2464	2395
query25	587	387	384	384
query26	1287	146	151	146
query27	2735	292	281	281
query28	7885	2473	2443	2443
query29	664	398	408	398
query30	315	146	148	146
query31	1059	781	815	781
query32	96	58	56	56
query33	764	300	278	278
query34	1001	504	524	504
query35	871	774	734	734
query36	1122	948	979	948
query37	206	76	71	71
query38	4332	4223	4243	4223
query39	1504	1422	1434	1422
query40	284	101	98	98
query41	47	45	46	45
query42	112	103	96	96
query43	543	488	480	480
query44	1189	802	791	791
query45	182	163	161	161
query46	1143	713	697	697
query47	1955	1853	1828	1828
query48	412	309	306	306
query49	1253	391	391	391
query50	824	395	378	378
query51	7304	7025	7035	7025
query52	103	94	93	93
query53	253	179	179	179
query54	1184	393	392	392
query55	82	82	81	81
query56	254	242	230	230
query57	1349	1168	1154	1154
query58	232	215	218	215
query59	3269	3004	2977	2977
query60	276	244	239	239
query61	113	113	112	112
query62	886	681	688	681
query63	214	189	185	185
query64	4989	647	614	614
query65	3302	3200	3282	3200
query66	1256	315	326	315
query67	16044	15676	16133	15676
query68	4584	544	554	544
query69	421	251	251	251
query70	1193	1141	1160	1141
query71	343	243	232	232
query72	6075	4050	4003	4003
query73	772	425	357	357
query74	9394	8984	9082	8984
query75	3424	2637	2665	2637
query76	2878	1064	994	994
query77	518	267	275	267
query78	10358	9411	9401	9401
query79	2524	595	599	595
query80	1205	420	425	420
query81	552	230	244	230
query82	807	118	115	115
query83	251	152	153	152
query84	234	73	73	73
query85	1386	290	303	290
query86	445	302	303	302
query87	4869	4806	4562	4562
query88	3599	2251	2202	2202
query89	397	297	303	297
query90	2224	189	192	189
query91	134	108	101	101
query92	67	54	51	51
query93	1854	544	540	540
query94	1043	289	279	279
query95	360	242	237	237
query96	619	271	275	271
query97	2863	2768	2664	2664
query98	207	199	202	199
query99	1571	1318	1304	1304
Total cold run time: 304261 ms
Total hot run time: 191103 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.49 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d0f5f61a4043bd2069ef63cfa2d757556831a297, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.04	0.03
query3	0.23	0.08	0.07
query4	1.62	0.10	0.10
query5	0.43	0.41	0.40
query6	1.16	0.65	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.51
query10	0.55	0.56	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	2.84	2.74	2.72
query15	0.90	0.82	0.82
query16	0.38	0.38	0.38
query17	0.98	1.06	1.07
query18	0.22	0.21	0.21
query19	1.93	1.78	1.91
query20	0.02	0.02	0.01
query21	15.36	0.60	0.59
query22	2.32	2.40	1.62
query23	17.05	1.05	0.78
query24	2.74	1.14	1.46
query25	0.30	0.05	0.13
query26	0.44	0.14	0.13
query27	0.05	0.04	0.04
query28	10.44	1.09	1.07
query29	12.54	3.21	3.23
query30	0.25	0.07	0.06
query31	2.86	0.38	0.38
query32	3.26	0.47	0.48
query33	3.00	3.07	3.07
query34	17.03	4.49	4.47
query35	4.53	4.51	4.45
query36	0.67	0.49	0.48
query37	0.09	0.06	0.05
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 106.2 s
Total hot run time: 32.49 s

@kaka11chen kaka11chen marked this pull request as ready for review November 28, 2024 07:02
@kaka11chen kaka11chen force-pushed the query_directory_list_cache branch from d0f5f61 to 6be0114 Compare November 29, 2024 11:48
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40430 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6be0114be0562d79dc9a8b73ee1809a658975fb9, data reload: false

------ Round 1 ----------------------------------
q1	17760	7644	7358	7358
q2	2051	184	187	184
q3	10604	1153	1139	1139
q4	10266	772	693	693
q5	7621	2748	2689	2689
q6	243	151	151	151
q7	988	642	620	620
q8	9282	1853	1929	1853
q9	6583	6514	6451	6451
q10	6962	2315	2322	2315
q11	459	271	263	263
q12	422	230	222	222
q13	17790	3108	3039	3039
q14	236	210	214	210
q15	569	547	525	525
q16	669	611	594	594
q17	969	675	567	567
q18	7558	6714	6780	6714
q19	1333	1038	990	990
q20	462	187	183	183
q21	4013	3375	3359	3359
q22	381	315	311	311
Total cold run time: 107221 ms
Total hot run time: 40430 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7319	7261	7297	7261
q2	329	227	232	227
q3	2930	2933	3067	2933
q4	2161	1867	1823	1823
q5	5632	5671	5638	5638
q6	222	141	142	141
q7	2192	1767	1809	1767
q8	3357	3497	3480	3480
q9	9008	8994	9005	8994
q10	3620	3556	3571	3556
q11	609	510	504	504
q12	833	658	629	629
q13	10416	3246	3290	3246
q14	294	283	271	271
q15	566	525	522	522
q16	681	633	625	625
q17	1847	1595	1578	1578
q18	7943	7535	7523	7523
q19	1642	1532	1559	1532
q20	2014	1812	1823	1812
q21	5609	5430	5287	5287
q22	618	549	556	549
Total cold run time: 69842 ms
Total hot run time: 59898 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191182 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6be0114be0562d79dc9a8b73ee1809a658975fb9, data reload: false

query1	992	382	360	360
query2	6519	2097	2026	2026
query3	6705	214	219	214
query4	33803	23670	23383	23383
query5	4380	485	449	449
query6	285	190	205	190
query7	4615	310	309	309
query8	302	227	222	222
query9	9345	2706	2700	2700
query10	487	252	253	252
query11	18248	15149	15297	15149
query12	153	101	100	100
query13	1672	425	409	409
query14	9125	6713	7752	6713
query15	313	182	187	182
query16	8165	454	480	454
query17	1712	567	560	560
query18	2138	290	293	290
query19	371	152	146	146
query20	122	108	110	108
query21	204	103	99	99
query22	4432	4351	4269	4269
query23	35865	34396	34355	34355
query24	10366	2468	2431	2431
query25	538	393	373	373
query26	856	148	155	148
query27	2800	285	284	284
query28	7960	2458	2449	2449
query29	641	399	409	399
query30	290	153	156	153
query31	1025	804	810	804
query32	133	56	58	56
query33	775	304	293	293
query34	937	510	512	510
query35	866	760	739	739
query36	1102	967	977	967
query37	127	76	74	74
query38	4299	4233	4239	4233
query39	1478	1418	1455	1418
query40	277	106	99	99
query41	48	45	44	44
query42	111	97	96	96
query43	543	482	489	482
query44	1339	837	818	818
query45	189	162	169	162
query46	1173	693	705	693
query47	1928	1811	1823	1811
query48	408	314	331	314
query49	1167	400	405	400
query50	836	384	382	382
query51	7376	7138	7095	7095
query52	98	90	87	87
query53	255	180	177	177
query54	1208	401	407	401
query55	81	76	79	76
query56	265	233	279	233
query57	1241	1141	1091	1091
query58	221	207	217	207
query59	3187	2887	2914	2887
query60	280	240	241	240
query61	116	117	106	106
query62	897	660	665	660
query63	212	187	182	182
query64	5191	709	813	709
query65	3293	3192	3247	3192
query66	1311	314	312	312
query67	15987	15732	15806	15732
query68	3527	562	557	557
query69	398	256	257	256
query70	1191	1124	1142	1124
query71	323	253	250	250
query72	5961	4209	4110	4110
query73	756	357	362	357
query74	9956	9054	9018	9018
query75	3377	2639	2696	2639
query76	1995	1101	1049	1049
query77	493	286	272	272
query78	10289	9378	9392	9378
query79	1317	586	584	584
query80	1060	444	428	428
query81	515	248	228	228
query82	1286	125	124	124
query83	266	164	157	157
query84	241	75	79	75
query85	1402	374	363	363
query86	375	311	302	302
query87	4855	4595	4570	4570
query88	3262	2329	2184	2184
query89	407	293	301	293
query90	2047	192	189	189
query91	152	103	104	103
query92	65	52	51	51
query93	1065	534	538	534
query94	1136	301	299	299
query95	381	253	261	253
query96	659	299	277	277
query97	2803	2731	2689	2689
query98	227	195	193	193
query99	1558	1362	1318	1318
Total cold run time: 296729 ms
Total hot run time: 191182 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6be0114be0562d79dc9a8b73ee1809a658975fb9, data reload: false

query1	0.04	0.02	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.40	0.41	0.42
query6	1.14	0.65	0.65
query7	0.02	0.02	0.01
query8	0.05	0.04	0.03
query9	0.58	0.50	0.51
query10	0.55	0.56	0.54
query11	0.14	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.61	0.61
query14	2.71	2.78	2.70
query15	0.91	0.84	0.83
query16	0.37	0.37	0.37
query17	1.02	1.02	1.09
query18	0.22	0.20	0.21
query19	1.95	1.88	1.99
query20	0.02	0.02	0.01
query21	15.36	0.58	0.59
query22	2.60	2.00	1.53
query23	16.98	0.91	0.95
query24	3.09	0.72	0.87
query25	0.23	0.09	0.04
query26	0.53	0.15	0.14
query27	0.05	0.04	0.04
query28	10.89	1.11	1.08
query29	12.55	3.27	3.27
query30	0.24	0.06	0.06
query31	2.87	0.39	0.40
query32	3.25	0.47	0.47
query33	3.01	3.01	3.03
query34	17.27	4.47	4.55
query35	4.56	4.55	4.57
query36	0.64	0.51	0.48
query37	0.09	0.07	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.12	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 107.39 s
Total hot run time: 32.25 s

@kaka11chen kaka11chen force-pushed the query_directory_list_cache branch from 6be0114 to 7333767 Compare December 3, 2024 06:01
@doris-robot
Copy link

TPC-DS: Total hot run time: 198081 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 73337677726752322a51c1b86abef9d141a9156d, data reload: false

query1	1249	969	938	938
query2	6231	2166	2112	2112
query3	10983	4381	4555	4381
query4	66649	29049	23482	23482
query5	4908	470	470	470
query6	418	183	187	183
query7	5559	302	307	302
query8	321	248	243	243
query9	8870	2713	2704	2704
query10	446	258	242	242
query11	17265	15289	15966	15289
query12	156	107	110	107
query13	1470	423	441	423
query14	11128	7460	7454	7454
query15	215	185	191	185
query16	6870	457	494	457
query17	1246	603	594	594
query18	1586	296	313	296
query19	207	156	147	147
query20	114	116	111	111
query21	205	104	99	99
query22	4810	4759	4830	4759
query23	34943	34713	34427	34427
query24	5410	2529	2559	2529
query25	502	397	388	388
query26	655	148	151	148
query27	1836	284	286	284
query28	4380	2456	2468	2456
query29	667	416	411	411
query30	211	152	148	148
query31	996	856	839	839
query32	68	52	54	52
query33	422	289	301	289
query34	941	510	526	510
query35	921	766	746	746
query36	1096	969	977	969
query37	115	73	73	73
query38	4529	4581	4426	4426
query39	1536	1471	1608	1471
query40	197	114	102	102
query41	47	42	63	42
query42	113	103	95	95
query43	544	514	502	502
query44	1181	821	826	821
query45	189	171	168	168
query46	1161	738	723	723
query47	2069	1966	1956	1956
query48	422	305	328	305
query49	729	386	409	386
query50	850	374	415	374
query51	7378	7105	7232	7105
query52	101	88	88	88
query53	254	183	180	180
query54	511	395	395	395
query55	77	76	76	76
query56	252	246	229	229
query57	1287	1097	1106	1097
query58	211	207	215	207
query59	3387	3025	3036	3025
query60	264	242	236	236
query61	113	107	113	107
query62	779	658	671	658
query63	214	191	196	191
query64	1407	668	646	646
query65	3266	3195	3200	3195
query66	665	310	296	296
query67	16061	15707	15797	15707
query68	4173	566	550	550
query69	420	260	255	255
query70	1198	1091	1141	1091
query71	329	247	253	247
query72	6381	4069	4042	4042
query73	755	369	353	353
query74	10151	9218	8949	8949
query75	3401	2671	2703	2671
query76	1930	1072	1091	1072
query77	477	272	363	272
query78	10505	9485	9550	9485
query79	1829	592	610	592
query80	1469	428	437	428
query81	507	293	229	229
query82	1284	118	119	118
query83	267	141	141	141
query84	276	67	72	67
query85	1026	304	300	300
query86	409	294	310	294
query87	4789	4587	4530	4530
query88	3553	2214	2202	2202
query89	435	299	293	293
query90	1886	190	191	190
query91	134	106	102	102
query92	63	48	51	48
query93	2071	542	543	542
query94	761	286	304	286
query95	371	254	252	252
query96	627	276	283	276
query97	2891	2649	2653	2649
query98	218	192	191	191
query99	1621	1324	1308	1308
Total cold run time: 320070 ms
Total hot run time: 198081 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 73337677726752322a51c1b86abef9d141a9156d, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.11
query5	0.43	0.40	0.41
query6	1.16	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.59	0.52	0.50
query10	0.55	0.56	0.57
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.61	0.62	0.59
query14	2.70	2.88	2.74
query15	0.90	0.83	0.83
query16	0.38	0.41	0.40
query17	1.09	1.06	1.03
query18	0.21	0.20	0.19
query19	1.97	1.93	1.97
query20	0.01	0.02	0.02
query21	15.36	0.59	0.58
query22	2.06	2.95	1.80
query23	16.92	1.04	0.84
query24	2.67	1.45	1.75
query25	0.17	0.15	0.04
query26	0.68	0.14	0.13
query27	0.05	0.05	0.05
query28	10.05	1.09	1.07
query29	12.54	3.26	3.19
query30	0.24	0.06	0.05
query31	2.87	0.38	0.37
query32	3.27	0.46	0.46
query33	2.98	3.01	3.07
query34	16.74	4.41	4.49
query35	4.52	4.48	4.43
query36	0.66	0.50	0.48
query37	0.08	0.05	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.16 s
Total hot run time: 32.99 s

@kaka11chen kaka11chen force-pushed the query_directory_list_cache branch from 7333767 to 5fd4a75 Compare December 30, 2024 07:06
@kaka11chen
Copy link
Contributor Author

run buildall

@kaka11chen kaka11chen force-pushed the query_directory_list_cache branch from 5fd4a75 to c6e693c Compare December 30, 2024 07:36
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32568 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c6e693cb242e94252bda8c2648824acc28e7386b, data reload: false

------ Round 1 ----------------------------------
q1	17613	6212	6069	6069
q2	2050	330	186	186
q3	10387	1263	766	766
q4	10231	886	440	440
q5	7844	2211	1981	1981
q6	205	188	154	154
q7	895	785	602	602
q8	9235	1416	1206	1206
q9	5192	4875	5055	4875
q10	6747	2330	1878	1878
q11	483	285	259	259
q12	358	368	220	220
q13	17780	3621	2998	2998
q14	238	242	212	212
q15	576	507	500	500
q16	637	622	580	580
q17	567	847	328	328
q18	7217	6430	6317	6317
q19	3565	973	558	558
q20	304	307	182	182
q21	2812	2167	1948	1948
q22	355	330	309	309
Total cold run time: 105291 ms
Total hot run time: 32568 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6522	6840	6196	6196
q2	233	334	227	227
q3	2281	2652	2321	2321
q4	1385	1811	1346	1346
q5	4356	5053	4926	4926
q6	187	183	148	148
q7	2139	1949	1831	1831
q8	2608	2844	2754	2754
q9	7281	7301	7262	7262
q10	3106	3311	2740	2740
q11	593	522	507	507
q12	664	775	621	621
q13	3406	3669	3111	3111
q14	306	323	283	283
q15	556	513	515	513
q16	657	699	642	642
q17	1219	1733	1266	1266
q18	7791	7526	7413	7413
q19	840	1170	1042	1042
q20	2004	2056	1899	1899
q21	5726	5207	4987	4987
q22	613	651	606	606
Total cold run time: 54473 ms
Total hot run time: 52641 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197007 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c6e693cb242e94252bda8c2648824acc28e7386b, data reload: false

query1	1316	943	953	943
query2	6491	2315	2235	2235
query3	10977	4783	4844	4783
query4	32898	23806	23475	23475
query5	3973	632	453	453
query6	289	210	189	189
query7	3994	499	306	306
query8	307	258	244	244
query9	9429	2761	2755	2755
query10	448	313	259	259
query11	17841	15589	15217	15217
query12	164	111	105	105
query13	1585	572	429	429
query14	10512	7276	7953	7276
query15	265	202	197	197
query16	8464	653	493	493
query17	1625	831	615	615
query18	2061	437	331	331
query19	231	236	161	161
query20	119	115	108	108
query21	204	130	123	123
query22	4726	4532	4718	4532
query23	34980	33412	33656	33412
query24	6675	2305	2272	2272
query25	463	484	410	410
query26	868	273	158	158
query27	2154	473	332	332
query28	5578	2500	2524	2500
query29	620	552	436	436
query30	209	191	158	158
query31	996	934	854	854
query32	76	62	57	57
query33	484	349	291	291
query34	781	851	526	526
query35	828	834	763	763
query36	1033	1098	954	954
query37	116	107	78	78
query38	4138	4254	4101	4101
query39	1548	1545	1461	1461
query40	213	125	102	102
query41	45	45	43	43
query42	129	101	103	101
query43	519	514	491	491
query44	1320	834	830	830
query45	189	175	171	171
query46	903	1022	659	659
query47	1981	1988	1947	1947
query48	405	413	332	332
query49	726	475	428	428
query50	642	682	404	404
query51	7316	7357	7239	7239
query52	108	105	102	102
query53	243	265	190	190
query54	498	511	425	425
query55	90	79	77	77
query56	260	260	252	252
query57	1262	1237	1197	1197
query58	244	224	236	224
query59	3008	3366	3086	3086
query60	290	258	271	258
query61	116	111	107	107
query62	843	792	776	776
query63	239	197	196	196
query64	3212	1045	654	654
query65	3311	3317	3227	3227
query66	851	442	305	305
query67	16402	15951	15585	15585
query68	9587	771	526	526
query69	478	294	257	257
query70	1204	1132	1108	1108
query71	428	279	267	267
query72	6214	3815	3867	3815
query73	675	747	372	372
query74	10449	9180	8875	8875
query75	4695	3138	2724	2724
query76	5264	1215	775	775
query77	972	384	296	296
query78	10066	10430	9312	9312
query79	2810	877	610	610
query80	627	513	448	448
query81	490	270	245	245
query82	639	147	126	126
query83	168	162	144	144
query84	242	87	75	75
query85	790	357	303	303
query86	355	299	294	294
query87	4572	4564	4414	4414
query88	3813	2221	2202	2202
query89	449	334	300	300
query90	1946	188	189	188
query91	138	134	115	115
query92	68	58	55	55
query93	960	903	566	566
query94	649	366	291	291
query95	341	276	254	254
query96	477	597	277	277
query97	2782	2827	2716	2716
query98	228	201	196	196
query99	2009	1512	1461	1461
Total cold run time: 299416 ms
Total hot run time: 197007 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c6e693cb242e94252bda8c2648824acc28e7386b, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.42	0.42	0.41
query6	1.15	0.64	0.65
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.60	0.52	0.51
query10	0.54	0.55	0.55
query11	0.14	0.10	0.11
query12	0.13	0.12	0.11
query13	0.60	0.61	0.59
query14	2.71	2.86	2.82
query15	0.90	0.82	0.83
query16	0.39	0.40	0.39
query17	1.02	1.06	1.04
query18	0.21	0.21	0.21
query19	1.87	1.87	2.02
query20	0.02	0.01	0.01
query21	15.37	0.93	0.57
query22	0.76	0.81	0.66
query23	15.28	1.46	0.51
query24	3.08	1.68	2.06
query25	0.23	0.19	0.11
query26	0.29	0.14	0.14
query27	0.05	0.08	0.04
query28	14.42	1.48	1.05
query29	12.59	3.92	3.30
query30	0.25	0.09	0.06
query31	2.83	0.62	0.39
query32	3.23	0.54	0.47
query33	3.12	3.10	3.12
query34	16.82	5.07	4.50
query35	4.46	4.47	4.43
query36	0.66	0.49	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.13	0.12
query41	0.07	0.04	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.66 s
Total hot run time: 32.17 s

starocean999
starocean999 previously approved these changes Jan 2, 2025
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 2, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 2, 2025

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 2, 2025

PR approved by anyone and no changes requested.

morningman
morningman previously approved these changes Jan 3, 2025
View reviewed changes
@morningman morningman force-pushed the query_directory_list_cache branch from c6e693c to 855a349 Compare January 3, 2025 08:54
@morningman
Copy link
Contributor

run buildall

@kaka11chen
[Opt](multi-catalog)Improve performance by introducing cache of list …
4aea992 
…directory files when getting split for each query.
@kaka11chen kaka11chen dismissed stale reviews from morningman and starocean999 via 4aea992 January 3, 2025 09:25
@kaka11chen kaka11chen force-pushed the query_directory_list_cache branch from 855a349 to 4aea992 Compare January 3, 2025 09:25
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 3, 2025
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32975 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4aea992fe7991282efa47bc29cdeaa7b4b1d8c7b, data reload: false

------ Round 1 ----------------------------------
q1	17587	6122	6064	6064
q2	2058	304	177	177
q3	10420	1294	736	736
q4	10212	880	440	440
q5	7491	2274	2018	2018
q6	213	182	158	158
q7	903	747	594	594
q8	9220	1386	1245	1245
q9	5259	4881	4970	4881
q10	6743	2318	1852	1852
q11	484	282	263	263
q12	350	365	216	216
q13	17772	3722	3086	3086
q14	227	234	225	225
q15	570	504	503	503
q16	634	640	598	598
q17	580	855	340	340
q18	7050	6499	6433	6433
q19	1568	1002	578	578
q20	324	335	191	191
q21	2787	2299	2058	2058
q22	366	337	319	319
Total cold run time: 102818 ms
Total hot run time: 32975 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6384	6287	6273	6273
q2	241	325	245	245
q3	2265	2689	2343	2343
q4	1401	1855	1328	1328
q5	4400	4784	4929	4784
q6	192	176	148	148
q7	2116	1985	1803	1803
q8	2611	2796	2675	2675
q9	7338	7205	7148	7148
q10	3094	3352	2818	2818
q11	608	524	493	493
q12	648	767	597	597
q13	3457	3898	3191	3191
q14	299	304	277	277
q15	564	509	512	509
q16	670	703	645	645
q17	1250	1737	1273	1273
q18	7785	7454	7189	7189
q19	848	861	1116	861
q20	1983	2016	1853	1853
q21	5385	5098	4829	4829
q22	651	610	600	600
Total cold run time: 54190 ms
Total hot run time: 51882 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190500 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4aea992fe7991282efa47bc29cdeaa7b4b1d8c7b, data reload: false

query1	1006	379	361	361
query2	6517	2476	2325	2325
query3	6719	214	212	212
query4	34073	24119	23461	23461
query5	4371	663	484	484
query6	297	212	174	174
query7	4632	510	306	306
query8	294	240	233	233
query9	9605	2671	2653	2653
query10	436	341	266	266
query11	18079	15515	15216	15216
query12	151	104	108	104
query13	1651	528	397	397
query14	9699	6793	6832	6793
query15	239	197	203	197
query16	7954	612	445	445
query17	1563	726	563	563
query18	1884	390	306	306
query19	185	176	155	155
query20	120	114	110	110
query21	220	156	100	100
query22	4276	4392	4203	4203
query23	34498	33527	33632	33527
query24	7362	2277	2282	2277
query25	513	444	375	375
query26	1220	283	154	154
query27	2636	462	328	328
query28	5657	2440	2429	2429
query29	683	541	427	427
query30	231	184	150	150
query31	965	872	820	820
query32	92	58	60	58
query33	492	359	291	291
query34	763	838	511	511
query35	782	811	744	744
query36	1001	1046	937	937
query37	121	100	79	79
query38	4215	4160	4220	4160
query39	1477	1444	1503	1444
query40	210	114	99	99
query41	49	49	48	48
query42	121	102	101	101
query43	530	531	501	501
query44	1339	839	813	813
query45	181	173	171	171
query46	889	1045	666	666
query47	1870	1926	1850	1850
query48	390	404	325	325
query49	784	493	399	399
query50	644	654	393	393
query51	7296	7081	7106	7081
query52	100	105	89	89
query53	232	252	185	185
query54	480	489	407	407
query55	82	77	78	77
query56	271	271	252	252
query57	1209	1157	1133	1133
query58	257	233	245	233
query59	3241	3274	2998	2998
query60	299	288	275	275
query61	139	127	130	127
query62	903	824	727	727
query63	233	199	205	199
query64	4247	1082	719	719
query65	3316	3241	3271	3241
query66	1073	418	317	317
query67	16061	15802	15635	15635
query68	7858	716	516	516
query69	461	289	256	256
query70	1215	1102	1139	1102
query71	455	285	251	251
query72	6470	3904	3949	3904
query73	676	778	363	363
query74	9967	9260	8938	8938
query75	3884	3170	2639	2639
query76	3774	1196	797	797
query77	777	384	271	271
query78	10173	10155	9365	9365
query79	2279	833	600	600
query80	624	536	441	441
query81	456	280	235	235
query82	445	159	126	126
query83	183	167	145	145
query84	250	89	69	69
query85	786	369	300	300
query86	358	293	304	293
query87	4499	4391	4403	4391
query88	4405	2188	2153	2153
query89	417	334	306	306
query90	1886	188	192	188
query91	138	137	105	105
query92	73	60	54	54
query93	1435	891	555	555
query94	657	420	268	268
query95	337	266	249	249
query96	490	620	273	273
query97	2898	3006	2857	2857
query98	230	195	199	195
query99	1649	1549	1459	1459
Total cold run time: 293330 ms
Total hot run time: 190500 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.84 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4aea992fe7991282efa47bc29cdeaa7b4b1d8c7b, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.07	0.07
query4	1.61	0.10	0.11
query5	0.41	0.42	0.41
query6	1.15	0.65	0.65
query7	0.03	0.02	0.01
query8	0.04	0.03	0.02
query9	0.56	0.51	0.48
query10	0.55	0.57	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.60	0.60
query14	2.84	2.81	2.81
query15	0.89	0.82	0.81
query16	0.38	0.38	0.41
query17	1.06	1.05	1.01
query18	0.24	0.21	0.22
query19	1.93	1.85	1.97
query20	0.02	0.01	0.01
query21	15.37	0.94	0.59
query22	0.76	0.82	0.61
query23	15.31	1.48	0.58
query24	3.04	0.48	1.65
query25	0.20	0.11	0.12
query26	0.23	0.14	0.14
query27	0.04	0.06	0.06
query28	14.01	1.53	1.05
query29	12.57	3.98	3.30
query30	0.25	0.10	0.05
query31	2.81	0.59	0.38
query32	3.75	0.54	0.46
query33	3.09	3.10	3.14
query34	16.49	5.12	4.43
query35	4.46	4.47	4.44
query36	0.67	0.50	0.50
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.16	0.14	0.12
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.47 s
Total hot run time: 30.84 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starocean999 starocean999 starocean999 left review comments

@morningman morningman morningman left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
dev/2.1.x-experimental dev/3.0.x-experimental reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kaka11chen @doris-robot @morningman @starocean999