Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat](metrics) Unify metrics of thread pool #43144

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

[feat](metrics) Unify metrics of thread pool #43144

wants to merge 13 commits into from

Conversation

zhiqiang-hhhh
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh commented Nov 2, 2024 •
edited
Loading

What problem does this PR solve?

Add metrics for all thread pool, more specifically, for all ThreadPool objects.
All thread pool will have following metrics:

  1. thread_pool_active_threads
  2. thread_pool_queue_size
  3. thread_pool_max_queue_size
  4. thread_pool_max_threads
  5. task_execution_time_ns_avg_in_last_1000_times
  6. task_wait_worker_ns_avg_in_last_1000_times

A new class IntervalHistogramStat is created for interval histogram calculation.

Metrics is updated by hook method when they are needed by prometheus.

Check List (For Committer)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No colde files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

  • Release note

    None

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Nov 2, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/exec/scan/scanner_scheduler.cpp Outdated Show resolved Hide resolved
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (9821/25967)
Line Coverage: 28.99% (81627/281564)
Region Coverage: 28.25% (42144/149158)
Branch Coverage: 24.83% (21381/86106)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b36415796af4881788f26f82f81791a16e8e4608_b36415796af4881788f26f82f81791a16e8e4608/report/index.html

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as draft November 4, 2024 00:59
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

1 similar comment
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (9820/25967)
Line Coverage: 28.99% (81616/281564)
Region Coverage: 28.25% (42141/149158)
Branch Coverage: 24.83% (21379/86106)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b36415796af4881788f26f82f81791a16e8e4608_b36415796af4881788f26f82f81791a16e8e4608/report/index.html

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (9821/25970)
Line Coverage: 28.97% (81596/281672)
Region Coverage: 28.24% (42131/149203)
Branch Coverage: 24.82% (21381/86140)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7eb8642f9a587895a5c757f61d356507495daed1_7eb8642f9a587895a5c757f61d356507495daed1/report/index.html

@zhiqiang-hhhh
Copy link
Contributor Author

image

Fail of test has nothing to do with this pr.

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as ready for review November 4, 2024 12:51
wangbo
wangbo previously approved these changes Nov 6, 2024
View reviewed changes
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 6, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024

PR approved by anyone and no changes requested.

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as draft November 6, 2024 09:32
github-actions[bot]
github-actions bot reviewed Dec 17, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/util/interval_histogram.h Show resolved Hide resolved
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh changed the title [opt](metrics) More metrics for scanner [feat](metrics) Unify metrics of thread pool Dec 17, 2024
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as ready for review December 19, 2024 12:29
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

1 similar comment
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-DS: Total hot run time: 197417 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 080155eef8dcd7e8123eb8c624004292f020f8d4, data reload: false

query1	1299	944	957	944
query2	6369	2406	2295	2295
query3	11118	4797	4811	4797
query4	33252	23842	23333	23333
query5	4392	647	465	465
query6	299	220	174	174
query7	3975	524	309	309
query8	300	228	222	222
query9	9630	2727	2706	2706
query10	443	318	251	251
query11	18004	15420	15670	15420
query12	170	109	106	106
query13	1645	560	413	413
query14	11006	7590	8239	7590
query15	258	208	198	198
query16	7982	696	446	446
query17	1503	794	588	588
query18	2040	435	322	322
query19	207	192	158	158
query20	120	115	119	115
query21	231	128	106	106
query22	4825	4670	4414	4414
query23	34897	33639	33695	33639
query24	6438	2282	2345	2282
query25	483	477	399	399
query26	810	295	154	154
query27	2065	497	331	331
query28	5674	2458	2433	2433
query29	643	576	429	429
query30	216	185	148	148
query31	1002	912	866	866
query32	78	67	67	67
query33	504	374	310	310
query34	801	908	525	525
query35	841	871	780	780
query36	1018	1076	967	967
query37	123	108	82	82
query38	4273	4255	4182	4182
query39	1540	1475	1505	1475
query40	213	124	104	104
query41	47	44	44	44
query42	117	101	104	101
query43	515	521	490	490
query44	1384	839	839	839
query45	185	175	169	169
query46	946	1126	655	655
query47	2006	2037	1975	1975
query48	396	445	334	334
query49	738	492	438	438
query50	663	703	403	403
query51	7326	7263	7328	7263
query52	99	109	93	93
query53	232	265	191	191
query54	501	493	422	422
query55	87	82	87	82
query56	272	274	248	248
query57	1272	1217	1185	1185
query58	243	232	234	232
query59	3096	3350	3006	3006
query60	280	294	290	290
query61	112	115	106	106
query62	887	829	752	752
query63	236	190	192	190
query64	3378	1060	672	672
query65	3271	3278	3339	3278
query66	704	452	315	315
query67	16539	15866	15454	15454
query68	8316	767	516	516
query69	475	295	246	246
query70	1188	1127	1094	1094
query71	432	286	245	245
query72	6246	4167	3891	3891
query73	662	818	366	366
query74	10124	9179	8957	8957
query75	3977	3191	2675	2675
query76	3619	1335	810	810
query77	768	367	278	278
query78	10184	10242	9427	9427
query79	3556	927	580	580
query80	748	543	440	440
query81	491	271	220	220
query82	628	151	125	125
query83	211	172	239	172
query84	282	86	72	72
query85	807	382	300	300
query86	348	324	304	304
query87	4527	4451	4424	4424
query88	4634	2241	2226	2226
query89	433	330	289	289
query90	1815	194	189	189
query91	140	139	109	109
query92	69	56	55	55
query93	1921	918	539	539
query94	675	398	287	287
query95	337	276	260	260
query96	486	668	278	278
query97	2755	2854	2653	2653
query98	228	209	191	191
query99	1688	1603	1427	1427
Total cold run time: 298989 ms
Total hot run time: 197417 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 080155eef8dcd7e8123eb8c624004292f020f8d4, data reload: false

query1	0.04	0.05	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.42	0.44	0.39
query6	1.16	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.51	0.52
query10	0.56	0.58	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.62	0.59
query14	2.72	2.80	2.77
query15	0.90	0.83	0.84
query16	0.38	0.39	0.37
query17	1.06	1.04	1.05
query18	0.23	0.21	0.20
query19	1.97	1.96	1.85
query20	0.01	0.01	0.01
query21	15.36	0.90	0.59
query22	0.75	0.78	0.92
query23	15.08	1.44	0.59
query24	3.05	1.46	0.96
query25	0.18	0.31	0.12
query26	0.24	0.16	0.13
query27	0.08	0.05	0.07
query28	13.54	1.52	1.04
query29	12.58	4.01	3.29
query30	0.24	0.09	0.07
query31	2.83	0.59	0.38
query32	3.23	0.54	0.46
query33	3.06	3.17	3.14
query34	16.76	5.13	4.49
query35	4.55	4.51	4.49
query36	0.64	0.49	0.48
query37	0.09	0.06	0.05
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.57 s
Total hot run time: 31.61 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32716 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cc29307c604aa6f643a9308946d420d83dfbaa57, data reload: false

------ Round 1 ----------------------------------
q1	17646	6241	6081	6081
q2	2057	318	181	181
q3	10399	1249	765	765
q4	10200	893	449	449
q5	7508	2188	2022	2022
q6	207	183	143	143
q7	914	752	598	598
q8	9259	1392	1190	1190
q9	5269	4932	4971	4932
q10	6775	2331	1864	1864
q11	479	283	253	253
q12	351	366	216	216
q13	17769	3621	2942	2942
q14	230	246	212	212
q15	554	503	496	496
q16	645	635	579	579
q17	573	866	334	334
q18	7168	6550	6448	6448
q19	1614	973	535	535
q20	304	323	192	192
q21	2812	2182	1979	1979
q22	357	334	305	305
Total cold run time: 103090 ms
Total hot run time: 32716 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6225	6204	6190	6190
q2	233	325	231	231
q3	2211	2648	2311	2311
q4	1403	1866	1342	1342
q5	4373	4766	4780	4766
q6	188	179	143	143
q7	2095	1977	1830	1830
q8	2617	2823	2721	2721
q9	7374	7519	7359	7359
q10	3092	3352	2761	2761
q11	571	503	488	488
q12	676	783	652	652
q13	3351	3754	3110	3110
q14	276	302	309	302
q15	564	515	502	502
q16	667	680	663	663
q17	1250	1750	1264	1264
q18	7704	7602	7318	7318
q19	886	1078	1132	1078
q20	2008	2014	1955	1955
q21	5656	5302	5041	5041
q22	639	579	587	579
Total cold run time: 54059 ms
Total hot run time: 52606 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196835 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cc29307c604aa6f643a9308946d420d83dfbaa57, data reload: false

query1	1298	932	924	924
query2	6507	2319	2323	2319
query3	11023	4686	4942	4686
query4	32945	23575	23494	23494
query5	3813	614	456	456
query6	273	204	205	204
query7	3976	475	295	295
query8	292	237	227	227
query9	9505	2708	2721	2708
query10	455	332	244	244
query11	18078	15333	15198	15198
query12	155	105	101	101
query13	1576	518	406	406
query14	9152	7796	6761	6761
query15	263	204	195	195
query16	7955	622	420	420
query17	1588	740	585	585
query18	2077	430	331	331
query19	218	192	165	165
query20	133	118	119	118
query21	212	126	110	110
query22	4761	4601	4515	4515
query23	35275	33877	33424	33424
query24	6553	2253	2278	2253
query25	472	465	413	413
query26	921	254	154	154
query27	2061	450	342	342
query28	5697	2536	2503	2503
query29	540	525	423	423
query30	217	186	149	149
query31	988	915	846	846
query32	67	62	56	56
query33	474	352	296	296
query34	769	851	538	538
query35	811	851	730	730
query36	1004	1064	975	975
query37	117	94	77	77
query38	4254	4305	4362	4305
query39	1529	1443	1426	1426
query40	212	118	102	102
query41	45	42	44	42
query42	119	109	98	98
query43	514	514	499	499
query44	1285	829	819	819
query45	189	178	168	168
query46	882	1073	667	667
query47	2056	2066	2013	2013
query48	385	432	336	336
query49	757	494	409	409
query50	645	679	394	394
query51	7329	7358	7264	7264
query52	107	98	87	87
query53	222	247	185	185
query54	484	494	423	423
query55	85	75	76	75
query56	257	256	243	243
query57	1264	1228	1155	1155
query58	241	224	256	224
query59	3301	3248	3148	3148
query60	275	262	257	257
query61	112	108	106	106
query62	863	838	748	748
query63	225	194	190	190
query64	3767	1046	663	663
query65	3294	3289	3218	3218
query66	892	439	319	319
query67	16520	15997	15516	15516
query68	7969	760	521	521
query69	481	312	266	266
query70	1227	1193	1160	1160
query71	439	310	254	254
query72	6217	3876	3762	3762
query73	645	751	357	357
query74	9857	9005	8854	8854
query75	3959	3149	2676	2676
query76	3658	1188	760	760
query77	764	345	271	271
query78	10117	10043	11003	10043
query79	2464	821	599	599
query80	700	532	435	435
query81	546	279	239	239
query82	227	143	120	120
query83	205	159	143	143
query84	296	96	74	74
query85	795	378	320	320
query86	357	292	306	292
query87	4509	4458	4415	4415
query88	4051	2223	2224	2223
query89	406	317	293	293
query90	1883	184	188	184
query91	132	132	107	107
query92	68	54	50	50
query93	969	739	534	534
query94	648	391	287	287
query95	330	260	244	244
query96	489	602	284	284
query97	2768	2779	2702	2702
query98	227	213	202	202
query99	1715	1563	1424	1424
Total cold run time: 293449 ms
Total hot run time: 196835 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cc29307c604aa6f643a9308946d420d83dfbaa57, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.42	0.41	0.43
query6	1.17	0.65	0.64
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.59	0.51	0.53
query10	0.57	0.59	0.56
query11	0.16	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.61
query14	2.85	2.73	2.75
query15	0.89	0.82	0.82
query16	0.38	0.39	0.38
query17	1.03	1.07	1.06
query18	0.21	0.21	0.21
query19	1.95	1.79	1.96
query20	0.01	0.01	0.01
query21	15.35	0.91	0.58
query22	0.74	0.73	0.64
query23	15.41	1.37	0.51
query24	2.93	1.43	0.45
query25	0.18	0.20	0.10
query26	0.34	0.17	0.14
query27	0.06	0.06	0.05
query28	13.19	1.53	1.05
query29	12.60	3.93	3.21
query30	0.24	0.09	0.06
query31	2.81	0.59	0.38
query32	3.22	0.54	0.47
query33	3.09	3.14	3.09
query34	16.70	5.03	4.46
query35	4.49	4.53	4.47
query36	0.65	0.51	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.03
query40	0.16	0.14	0.12
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 105.5 s
Total hot run time: 30.65 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33106 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5e27cf55d6ca7d5a38c27ed4ad03a9665acaa154, data reload: false

------ Round 1 ----------------------------------
q1	17593	6286	6299	6286
q2	2048	326	195	195
q3	10374	1339	730	730
q4	10219	884	436	436
q5	7546	2202	2026	2026
q6	203	182	153	153
q7	905	757	622	622
q8	9250	1391	1181	1181
q9	5346	4955	4978	4955
q10	6787	2292	1858	1858
q11	488	297	271	271
q12	345	361	224	224
q13	17770	3627	2977	2977
q14	250	240	223	223
q15	574	517	496	496
q16	656	621	595	595
q17	604	871	331	331
q18	7180	6591	6501	6501
q19	2935	981	554	554
q20	303	324	190	190
q21	2851	2284	1984	1984
q22	370	340	318	318
Total cold run time: 104597 ms
Total hot run time: 33106 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6302	6229	6212	6212
q2	240	334	228	228
q3	2245	2692	2349	2349
q4	1446	1888	1325	1325
q5	4398	4782	4672	4672
q6	180	170	138	138
q7	1995	1870	1752	1752
q8	2516	2713	2633	2633
q9	6916	6856	6906	6856
q10	2966	3256	2720	2720
q11	594	528	498	498
q12	628	716	559	559
q13	3249	3572	2989	2989
q14	275	290	278	278
q15	550	502	488	488
q16	661	698	678	678
q17	1209	1742	1252	1252
q18	7494	7284	7140	7140
q19	826	1154	1061	1061
q20	1945	2005	1861	1861
q21	5425	5089	4800	4800
q22	624	617	598	598
Total cold run time: 52684 ms
Total hot run time: 51087 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190439 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5e27cf55d6ca7d5a38c27ed4ad03a9665acaa154, data reload: false

query1	1010	379	388	379
query2	6514	2324	2252	2252
query3	6710	211	217	211
query4	33766	23952	23543	23543
query5	4316	612	453	453
query6	281	205	192	192
query7	4639	497	301	301
query8	311	252	248	248
query9	9578	2790	2789	2789
query10	477	346	257	257
query11	18592	15294	15248	15248
query12	175	108	112	108
query13	1672	550	421	421
query14	12168	7663	7170	7170
query15	248	193	189	189
query16	8096	584	423	423
query17	1563	764	578	578
query18	2135	441	291	291
query19	224	180	146	146
query20	115	110	110	110
query21	209	123	105	105
query22	4535	4612	4341	4341
query23	34637	33276	33297	33276
query24	6140	2250	2198	2198
query25	502	435	386	386
query26	1207	273	151	151
query27	2031	448	358	358
query28	5062	2492	2456	2456
query29	751	556	415	415
query30	222	181	146	146
query31	1083	901	816	816
query32	88	61	59	59
query33	500	357	298	298
query34	783	855	522	522
query35	805	806	739	739
query36	995	1053	967	967
query37	115	101	71	71
query38	4216	4269	4126	4126
query39	1512	1420	1418	1418
query40	203	128	105	105
query41	47	46	45	45
query42	118	104	100	100
query43	518	530	496	496
query44	1347	807	802	802
query45	181	172	167	167
query46	858	1030	648	648
query47	1905	1892	1878	1878
query48	391	404	329	329
query49	781	479	404	404
query50	627	654	400	400
query51	7227	7093	7109	7093
query52	106	99	93	93
query53	221	253	181	181
query54	477	480	420	420
query55	83	76	77	76
query56	248	277	260	260
query57	1177	1162	1128	1128
query58	243	229	231	229
query59	3051	3244	2942	2942
query60	277	258	250	250
query61	109	110	111	110
query62	857	830	751	751
query63	227	192	192	192
query64	4592	1032	658	658
query65	3296	3193	3268	3193
query66	836	419	304	304
query67	15917	15892	15647	15647
query68	9830	769	530	530
query69	450	286	265	265
query70	1243	1151	1029	1029
query71	444	295	259	259
query72	5798	3830	3872	3830
query73	680	741	366	366
query74	10091	9014	8825	8825
query75	4648	3091	2659	2659
query76	5539	1208	778	778
query77	1109	364	275	275
query78	10240	10167	9327	9327
query79	3392	904	588	588
query80	690	513	431	431
query81	462	279	229	229
query82	629	154	127	127
query83	194	164	224	164
query84	292	88	75	75
query85	782	362	306	306
query86	355	317	303	303
query87	4349	4587	4453	4453
query88	4453	2235	2241	2235
query89	400	333	310	310
query90	1904	189	188	188
query91	132	146	105	105
query92	64	56	52	52
query93	1207	892	544	544
query94	651	399	286	286
query95	336	257	247	247
query96	492	609	291	291
query97	2724	2786	2700	2700
query98	235	210	199	199
query99	1679	1544	1405	1405
Total cold run time: 299397 ms
Total hot run time: 190439 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.68 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5e27cf55d6ca7d5a38c27ed4ad03a9665acaa154, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.08	0.07
query4	1.62	0.11	0.10
query5	0.42	0.43	0.42
query6	1.14	0.65	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.53
query10	0.56	0.59	0.54
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.61	0.60
query14	2.70	2.78	2.72
query15	0.90	0.82	0.82
query16	0.39	0.38	0.39
query17	1.05	0.98	1.10
query18	0.23	0.21	0.22
query19	1.98	1.83	1.99
query20	0.01	0.01	0.01
query21	15.36	0.92	0.58
query22	0.75	0.79	0.77
query23	15.56	1.40	0.56
query24	3.10	1.20	1.92
query25	0.26	0.17	0.15
query26	0.20	0.15	0.14
query27	0.06	0.05	0.04
query28	14.31	1.50	1.05
query29	12.58	4.05	3.30
query30	0.25	0.09	0.06
query31	2.83	0.62	0.38
query32	3.24	0.54	0.46
query33	3.07	3.13	3.11
query34	16.72	5.11	4.49
query35	4.48	4.44	4.45
query36	0.64	0.49	0.50
query37	0.09	0.06	0.07
query38	0.04	0.03	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.07	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.78 s
Total hot run time: 31.68 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32531 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 747b37a13ad50634e58d004df5a66d666f98be0d, data reload: false

------ Round 1 ----------------------------------
q1	17568	6091	6040	6040
q2	2052	312	172	172
q3	10419	1261	747	747
q4	10247	875	442	442
q5	8346	2196	2016	2016
q6	217	180	147	147
q7	899	756	612	612
q8	9254	1379	1156	1156
q9	5256	4905	4972	4905
q10	6752	2307	1855	1855
q11	482	271	256	256
q12	348	365	219	219
q13	17775	3542	2884	2884
q14	256	222	216	216
q15	556	514	496	496
q16	636	617	588	588
q17	579	859	333	333
q18	7006	6615	6406	6406
q19	2824	969	558	558
q20	298	314	182	182
q21	2804	2183	2003	2003
q22	357	343	298	298
Total cold run time: 104931 ms
Total hot run time: 32531 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6390	6191	6233	6191
q2	239	333	233	233
q3	2246	2623	2366	2366
q4	1381	1821	1341	1341
q5	4384	4764	4951	4764
q6	194	180	152	152
q7	2070	1937	1861	1861
q8	2643	2882	2697	2697
q9	7355	7335	7344	7335
q10	3098	3351	2748	2748
q11	598	511	495	495
q12	672	765	628	628
q13	3410	3781	3157	3157
q14	280	317	301	301
q15	570	510	499	499
q16	645	692	663	663
q17	1248	1741	1270	1270
q18	7688	7474	7393	7393
q19	851	1125	1145	1125
q20	2039	2071	1890	1890
q21	5836	5272	4841	4841
q22	658	625	605	605
Total cold run time: 54495 ms
Total hot run time: 52555 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197785 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 747b37a13ad50634e58d004df5a66d666f98be0d, data reload: false

query1	1317	936	929	929
query2	6411	2505	2392	2392
query3	10995	4708	4778	4708
query4	33339	23619	23364	23364
query5	4615	636	451	451
query6	329	191	178	178
query7	3982	495	310	310
query8	311	262	237	237
query9	9323	2638	2639	2638
query10	447	310	249	249
query11	17931	15333	15105	15105
query12	157	102	110	102
query13	1554	521	406	406
query14	10481	7611	7833	7611
query15	252	205	201	201
query16	8282	609	486	486
query17	1580	781	580	580
query18	2090	423	317	317
query19	215	182	163	163
query20	120	113	128	113
query21	204	126	112	112
query22	4781	4611	4458	4458
query23	34096	33403	33474	33403
query24	6553	2379	2297	2297
query25	490	493	399	399
query26	1050	270	149	149
query27	2044	475	336	336
query28	5544	2459	2419	2419
query29	640	557	433	433
query30	210	211	152	152
query31	990	921	819	819
query32	69	61	59	59
query33	492	364	303	303
query34	759	865	526	526
query35	816	866	772	772
query36	1053	1072	984	984
query37	117	112	85	85
query38	4192	4371	4227	4227
query39	1607	1466	1462	1462
query40	211	125	112	112
query41	49	51	46	46
query42	128	105	116	105
query43	521	538	503	503
query44	1371	852	868	852
query45	188	186	172	172
query46	880	1082	688	688
query47	2025	2025	1953	1953
query48	408	410	347	347
query49	758	475	412	412
query50	635	684	400	400
query51	7240	7262	7201	7201
query52	108	102	92	92
query53	229	252	183	183
query54	501	501	422	422
query55	84	81	77	77
query56	261	267	263	263
query57	1257	1250	1177	1177
query58	242	232	252	232
query59	3252	3361	3209	3209
query60	275	279	261	261
query61	115	108	112	108
query62	880	821	769	769
query63	229	188	189	188
query64	3548	1023	664	664
query65	3367	3262	3292	3262
query66	968	418	317	317
query67	16523	15756	15463	15463
query68	7647	750	512	512
query69	470	301	257	257
query70	1187	1139	1135	1135
query71	430	281	251	251
query72	6173	3828	3888	3828
query73	658	744	364	364
query74	9929	9317	9047	9047
query75	4052	3132	2658	2658
query76	3675	1198	764	764
query77	765	359	265	265
query78	10285	10078	10497	10078
query79	2843	840	599	599
query80	707	514	421	421
query81	488	273	232	232
query82	340	145	119	119
query83	221	164	153	153
query84	278	85	67	67
query85	779	385	303	303
query86	367	325	303	303
query87	4808	4420	4453	4420
query88	4291	2251	2235	2235
query89	408	319	293	293
query90	1895	189	191	189
query91	135	129	108	108
query92	70	53	61	53
query93	964	756	522	522
query94	668	371	284	284
query95	337	271	253	253
query96	492	617	286	286
query97	2784	2774	2753	2753
query98	231	198	194	194
query99	1722	1566	1426	1426
Total cold run time: 295790 ms
Total hot run time: 197785 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 747b37a13ad50634e58d004df5a66d666f98be0d, data reload: false

query1	0.03	0.04	0.04
query2	0.06	0.04	0.03
query3	0.24	0.07	0.07
query4	1.63	0.10	0.11
query5	0.42	0.41	0.40
query6	1.19	0.65	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.55	0.50	0.49
query10	0.56	0.58	0.56
query11	0.15	0.10	0.11
query12	0.13	0.11	0.11
query13	0.62	0.61	0.59
query14	2.73	2.83	2.74
query15	0.89	0.83	0.82
query16	0.36	0.38	0.37
query17	1.04	1.05	1.05
query18	0.23	0.21	0.22
query19	1.94	1.89	1.99
query20	0.02	0.02	0.01
query21	15.36	0.92	0.59
query22	0.74	0.81	0.73
query23	15.25	1.44	0.52
query24	3.27	1.71	1.37
query25	0.20	0.26	0.12
query26	0.22	0.15	0.14
query27	0.08	0.05	0.06
query28	14.41	1.51	1.05
query29	12.59	4.05	3.29
query30	0.25	0.10	0.06
query31	2.83	0.57	0.40
query32	3.22	0.54	0.45
query33	3.08	3.07	3.12
query34	16.68	5.06	4.48
query35	4.50	4.42	4.45
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 106.68 s
Total hot run time: 31.81 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@yiguolei yiguolei yiguolei left review comments

@wangbo wangbo wangbo left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@zhiqiang-hhhh @doris-robot @yiguolei @wangbo