Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] fix be core in highly concurrent queries #47411

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[Bug] fix be core in highly concurrent queries #47411

wants to merge 1 commit into from
+25 −9

Conversation

yongjinhou
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #47410

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@yongjinhou
Copy link
Contributor Author

run buildall

@yongjinhou
Copy link
Contributor Author

run buildall

@yongjinhou
Copy link
Contributor Author

run buildall

@yongjinhou
Copy link
Contributor Author

run compile

@doris-robot
Copy link

TPC-H: Total hot run time: 32262 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2e29b0cb49202e398cf0dc9b18d963fe99be574a, data reload: false

------ Round 1 ----------------------------------
q1	17599	5578	5391	5391
q2	2043	293	164	164
q3	10519	1286	733	733
q4	10224	951	536	536
q5	7560	2424	2169	2169
q6	187	165	138	138
q7	926	760	629	629
q8	9238	1391	1162	1162
q9	5271	4910	4906	4906
q10	6842	2348	1887	1887
q11	468	270	275	270
q12	333	357	226	226
q13	17785	3649	3086	3086
q14	231	221	217	217
q15	514	482	456	456
q16	645	611	589	589
q17	560	880	326	326
q18	7141	6394	6446	6394
q19	2733	967	540	540
q20	306	312	183	183
q21	2817	2192	1948	1948
q22	362	331	312	312
Total cold run time: 104304 ms
Total hot run time: 32262 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5597	5455	5630	5455
q2	239	323	237	237
q3	2287	2646	2283	2283
q4	1478	1792	1380	1380
q5	4349	4758	4648	4648
q6	169	166	132	132
q7	2133	2003	1842	1842
q8	2650	2799	2670	2670
q9	7324	7219	7251	7219
q10	2980	3224	2781	2781
q11	571	506	494	494
q12	686	759	632	632
q13	3559	3883	3174	3174
q14	291	283	268	268
q15	514	480	479	479
q16	647	672	650	650
q17	1209	1716	1249	1249
q18	7614	7563	7434	7434
q19	780	982	1134	982
q20	2013	2004	1908	1908
q21	5601	5209	4989	4989
q22	574	571	554	554
Total cold run time: 53265 ms
Total hot run time: 51460 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184860 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2e29b0cb49202e398cf0dc9b18d963fe99be574a, data reload: false

query1	974	389	373	373
query2	6523	2113	2045	2045
query3	6789	209	207	207
query4	33391	23771	23110	23110
query5	4256	606	455	455
query6	272	190	182	182
query7	4600	497	300	300
query8	321	246	236	236
query9	9350	2598	2566	2566
query10	471	325	245	245
query11	18242	15065	15029	15029
query12	153	106	108	106
query13	1685	530	407	407
query14	10411	6700	6507	6507
query15	240	195	188	188
query16	7985	619	468	468
query17	1605	732	576	576
query18	2108	402	315	315
query19	225	183	156	156
query20	118	113	110	110
query21	215	124	105	105
query22	4599	4569	4387	4387
query23	33930	33221	33066	33066
query24	6466	2298	2389	2298
query25	494	451	388	388
query26	791	272	151	151
query27	1985	449	330	330
query28	5461	2433	2416	2416
query29	656	536	461	461
query30	230	181	159	159
query31	949	887	805	805
query32	84	62	59	59
query33	499	371	295	295
query34	740	844	493	493
query35	803	815	738	738
query36	983	1004	936	936
query37	121	102	77	77
query38	4060	4101	4037	4037
query39	1426	1375	1441	1375
query40	198	112	107	107
query41	53	53	52	52
query42	123	107	101	101
query43	517	533	473	473
query44	1323	815	808	808
query45	175	172	167	167
query46	854	1024	642	642
query47	1828	1868	1812	1812
query48	377	399	314	314
query49	752	474	405	405
query50	607	659	406	406
query51	4197	4158	4168	4158
query52	105	105	92	92
query53	223	260	192	192
query54	482	512	408	408
query55	82	80	83	80
query56	264	266	235	235
query57	1223	1180	1068	1068
query58	246	222	238	222
query59	2923	3250	2810	2810
query60	273	270	258	258
query61	115	114	112	112
query62	789	719	638	638
query63	220	197	189	189
query64	3765	1002	638	638
query65	3200	3172	3165	3165
query66	870	402	319	319
query67	15984	15636	15633	15633
query68	4912	840	527	527
query69	484	290	268	268
query70	1229	1111	1112	1111
query71	379	291	254	254
query72	5287	3799	3852	3799
query73	650	755	371	371
query74	10015	9104	8735	8735
query75	3182	3175	2669	2669
query76	3095	1164	748	748
query77	512	357	272	272
query78	10021	10226	9309	9309
query79	2439	808	612	612
query80	1720	526	460	460
query81	576	265	237	237
query82	350	145	121	121
query83	215	171	156	156
query84	245	92	73	73
query85	765	354	299	299
query86	487	301	293	293
query87	4324	4513	4432	4432
query88	3901	2198	2154	2154
query89	398	330	293	293
query90	1711	193	194	193
query91	133	142	111	111
query92	67	56	58	56
query93	1808	887	530	530
query94	707	398	309	309
query95	317	261	253	253
query96	488	606	290	290
query97	2784	2896	2770	2770
query98	215	203	203	203
query99	1278	1371	1291	1291
Total cold run time: 280673 ms
Total hot run time: 184860 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.9 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2e29b0cb49202e398cf0dc9b18d963fe99be574a, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.23	0.07	0.06
query4	1.61	0.11	0.11
query5	0.40	0.41	0.42
query6	1.15	0.66	0.65
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.59	0.50	0.51
query10	0.56	0.57	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.71	2.72	2.72
query15	0.89	0.83	0.83
query16	0.39	0.41	0.38
query17	1.05	1.05	0.99
query18	0.24	0.21	0.20
query19	1.91	1.88	2.00
query20	0.02	0.01	0.02
query21	15.37	0.88	0.56
query22	0.77	0.79	0.60
query23	15.42	1.43	0.50
query24	3.07	1.27	1.61
query25	0.25	0.15	0.09
query26	0.17	0.14	0.13
query27	0.06	0.04	0.04
query28	14.35	1.00	0.42
query29	12.58	4.04	3.36
query30	0.25	0.09	0.08
query31	3.14	0.58	0.38
query32	3.38	0.57	0.48
query33	3.04	3.00	3.04
query34	16.66	5.19	4.47
query35	4.53	4.54	4.53
query36	0.66	0.49	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 107 s
Total hot run time: 30.9 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.98% (10950/26081)
Line Coverage: 32.24% (92455/286759)
Region Coverage: 31.37% (47379/151013)
Branch Coverage: 27.44% (24008/87486)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2e29b0cb49202e398cf0dc9b18d963fe99be574a_2e29b0cb49202e398cf0dc9b18d963fe99be574a/report/index.html

yiguolei
yiguolei reviewed Jan 25, 2025
View reviewed changes
be/src/runtime/fragment_mgr.h
@@ -216,6 +216,7 @@ class FragmentMgr : public RestMonitorIface {
scoped_refptr<Thread> _cancel_thread;
// every job is a pool
std::unique_ptr<ThreadPool> _thread_pool;
std::unique_ptr<FifoThreadPool> _thread_pool_for_prepare;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why it will core in the future？？？
You should explain it in the PR description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiguolei yiguolei yiguolei left review comments

@Gabriel39 Gabriel39 Awaiting requested review from Gabriel39

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] fix be core in highly concurrent queries
3 participants
@yongjinhou @doris-robot @yiguolei