[Improvement](scheduler) Use a separate eager queue to execute cancel… #45614

Gabriel39 · 2024-12-18T13:44:04Z

…ed tasks

What problem does this PR solve?

Now, a pipeline task which is canceled will be put in runnable queue and release memory once it ran. However, a runnable queue may have too much tasks which leads to a unacceptable delay for this canceled task. So this PR use a separate queue to process all canceled tasks.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…ed tasks

hello-stephen · 2024-12-18T13:44:10Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Gabriel39 · 2024-12-18T13:44:13Z

run buildall

github-actions · 2024-12-18T13:44:58Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions · 2024-12-18T13:49:23Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-12-19T02:04:41Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

Gabriel39 · 2024-12-19T02:06:06Z

run buildall

github-actions · 2024-12-19T02:09:47Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-19T02:35:26Z

TeamCity be ut coverage result:
Function Coverage: 38.87% (10133/26066)
Line Coverage: 29.78% (85168/285986)
Region Coverage: 28.83% (43691/151539)
Branch Coverage: 25.37% (22197/87492)
Coverage Report: http://coverage.selectdb-in.cc/coverage/56171ee35e2ea084e2311b6db2ef278f5d6ec05d_56171ee35e2ea084e2311b6db2ef278f5d6ec05d/report/index.html

github-actions · 2024-12-19T02:59:27Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

Gabriel39 · 2024-12-19T03:02:28Z

run buildall

github-actions · 2024-12-19T03:02:39Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions

clang-tidy made some suggestions

be/src/pipeline/task_queue.cpp

doris-robot · 2024-12-19T03:33:52Z

TeamCity be ut coverage result:
Function Coverage: 38.88% (10134/26067)
Line Coverage: 29.78% (85183/285996)
Region Coverage: 28.84% (43698/151544)
Branch Coverage: 25.37% (22202/87496)
Coverage Report: http://coverage.selectdb-in.cc/coverage/95971717ad6b94ae0e15d953ea17a8687f0a71bb_95971717ad6b94ae0e15d953ea17a8687f0a71bb/report/index.html

Gabriel39 · 2024-12-19T03:47:03Z

run buildall

github-actions · 2024-12-19T03:47:11Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

doris-robot · 2024-12-19T04:20:35Z

TeamCity be ut coverage result:
Function Coverage: 38.88% (10134/26067)
Line Coverage: 29.79% (85213/286000)
Region Coverage: 28.84% (43708/151547)
Branch Coverage: 25.37% (22200/87498)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e624dc5091fd94105e858c34d93632cdf06f4737_e624dc5091fd94105e858c34d93632cdf06f4737/report/index.html

Gabriel39 · 2024-12-19T07:24:33Z

run buildall

github-actions · 2024-12-19T07:24:57Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

doris-robot · 2024-12-19T08:12:31Z

TeamCity be ut coverage result:
Function Coverage: 38.87% (10133/26067)
Line Coverage: 29.78% (85171/285998)
Region Coverage: 28.83% (43691/151545)
Branch Coverage: 25.37% (22195/87496)
Coverage Report: http://coverage.selectdb-in.cc/coverage/42532ad7b23925dc5c5ec8051beca939e2ce4b0d_42532ad7b23925dc5c5ec8051beca939e2ce4b0d/report/index.html

github-actions · 2024-12-19T08:32:53Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

Gabriel39 · 2024-12-19T08:33:23Z

run buildall

doris-robot · 2024-12-19T09:02:37Z

TeamCity be ut coverage result:
Function Coverage: 38.88% (10134/26067)
Line Coverage: 29.78% (85173/285997)
Region Coverage: 28.83% (43692/151545)
Branch Coverage: 25.37% (22195/87496)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6e2fe8b6325172cb7a79b2b3a6c829d20f225962_6e2fe8b6325172cb7a79b2b3a6c829d20f225962/report/index.html

be/src/pipeline/pipeline_task.cpp

be/src/pipeline/task_queue.h

be/src/pipeline/pipeline_task.h

be/src/pipeline/task_queue.cpp

github-actions · 2024-12-20T01:52:11Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions · 2024-12-20T01:53:41Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions

clang-tidy made some suggestions

github-actions · 2024-12-20T01:56:40Z

be/src/pipeline/task_queue.cpp

@@ -172,7 +178,7 @@ PipelineTask* MultiCoreTaskQueue::take(int core_id) {
    return task;
 }

-PipelineTask* MultiCoreTaskQueue::_steal_take(int core_id) {
+std::shared_ptr<PipelineTask> MultiCoreTaskQueue::_steal_take(int core_id) {


warning: method '_steal_take' can be made const [readability-make-member-function-const]

Suggested change

std::shared_ptr<PipelineTask> MultiCoreTaskQueue::_steal_take(int core_id) {

std::shared_ptr<PipelineTask> MultiCoreTaskQueue::_steal_take(int core_id) const {

be/src/pipeline/task_queue.h:126:

- std::shared_ptr<PipelineTask> _steal_take(int core_id); + std::shared_ptr<PipelineTask> _steal_take(int core_id) const;

Gabriel39 · 2024-12-20T01:57:43Z

run buildall

doris-robot · 2024-12-20T02:54:07Z

TeamCity be ut coverage result:
Function Coverage: 38.87% (10131/26062)
Line Coverage: 29.80% (85212/285993)
Region Coverage: 28.91% (43479/150398)
Branch Coverage: 25.44% (22168/87154)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d1c6313a84d8dec05d6c3caef6e5b4d00c05a2d3_d1c6313a84d8dec05d6c3caef6e5b4d00c05a2d3/report/index.html

Gabriel39 · 2024-12-20T08:04:21Z

run buildall

doris-robot · 2024-12-20T08:32:50Z

TPC-H: Total hot run time: 39918 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 08b123bfdd25c2b71e27e18afe2e396b80466034, data reload: false

------ Round 1 ----------------------------------
q1	17806	7571	7279	7279
q2	2052	182	180	180
q3	10650	1170	1192	1170
q4	10322	728	715	715
q5	7589	2750	2721	2721
q6	236	149	155	149
q7	969	618	611	611
q8	9257	1894	1922	1894
q9	6675	6517	6407	6407
q10	7014	2300	2277	2277
q11	480	262	276	262
q12	419	215	223	215
q13	17793	2946	2958	2946
q14	247	227	205	205
q15	547	513	506	506
q16	652	606	579	579
q17	990	482	586	482
q18	7732	6660	6671	6660
q19	1370	965	916	916
q20	484	191	190	190
q21	4239	3243	3357	3243
q22	385	311	317	311
Total cold run time: 107908 ms
Total hot run time: 39918 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7277	7222	7244	7222
q2	332	232	231	231
q3	2946	2819	2840	2819
q4	2007	1744	1674	1674
q5	5360	5402	5458	5402
q6	221	137	140	137
q7	2173	1711	1688	1688
q8	3250	3415	3431	3415
q9	8715	8649	8632	8632
q10	3508	3479	3475	3475
q11	605	491	504	491
q12	772	581	550	550
q13	12011	2892	2975	2892
q14	296	268	260	260
q15	551	503	487	487
q16	678	624	649	624
q17	1801	1596	1563	1563
q18	7935	7328	7524	7328
q19	1716	1516	1534	1516
q20	2032	1865	1834	1834
q21	5528	5285	5272	5272
q22	654	589	547	547
Total cold run time: 70368 ms
Total hot run time: 58059 ms

doris-robot · 2024-12-20T08:42:08Z

TeamCity be ut coverage result:
Function Coverage: 38.87% (10131/26062)
Line Coverage: 29.80% (85271/286101)
Region Coverage: 28.92% (43529/150496)
Branch Coverage: 25.44% (22169/87152)
Coverage Report: http://coverage.selectdb-in.cc/coverage/08b123bfdd25c2b71e27e18afe2e396b80466034_08b123bfdd25c2b71e27e18afe2e396b80466034/report/index.html

doris-robot · 2024-12-20T08:44:56Z

TPC-DS: Total hot run time: 189933 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 08b123bfdd25c2b71e27e18afe2e396b80466034, data reload: false

query1	1000	402	404	402
query2	6528	2556	2391	2391
query3	6719	215	211	211
query4	33782	23350	23325	23325
query5	4391	469	456	456
query6	288	212	191	191
query7	4630	301	306	301
query8	313	244	229	229
query9	9692	2725	2716	2716
query10	487	256	249	249
query11	18006	15220	15061	15061
query12	163	106	105	105
query13	1694	414	412	412
query14	10837	6873	6914	6873
query15	307	191	179	179
query16	8307	460	482	460
query17	1754	583	606	583
query18	2148	317	318	317
query19	380	148	147	147
query20	119	117	115	115
query21	211	102	99	99
query22	4575	4408	4459	4408
query23	34546	33772	33192	33192
query24	11494	2514	2354	2354
query25	682	391	382	382
query26	1787	156	152	152
query27	2988	327	323	323
query28	7769	2419	2429	2419
query29	1045	403	408	403
query30	303	147	143	143
query31	1020	800	804	800
query32	101	59	58	58
query33	762	303	278	278
query34	994	527	517	517
query35	875	752	735	735
query36	1112	942	951	942
query37	282	80	75	75
query38	4124	4181	4018	4018
query39	1510	1408	1435	1408
query40	283	101	99	99
query41	47	43	48	43
query42	117	103	101	101
query43	542	520	499	499
query44	1257	796	795	795
query45	183	166	167	166
query46	1166	704	699	699
query47	1923	1815	1853	1815
query48	416	323	321	321
query49	1295	390	378	378
query50	794	395	395	395
query51	7229	7037	7124	7037
query52	102	95	90	90
query53	264	183	184	183
query54	1277	414	417	414
query55	89	82	77	77
query56	264	263	241	241
query57	1258	1141	1082	1082
query58	236	219	226	219
query59	3291	3362	3127	3127
query60	288	250	242	242
query61	108	111	111	111
query62	915	660	730	660
query63	218	196	189	189
query64	5068	674	681	674
query65	3252	3191	3227	3191
query66	1427	316	310	310
query67	15783	15673	15438	15438
query68	5835	550	556	550
query69	436	280	250	250
query70	1245	1114	1112	1112
query71	450	253	255	253
query72	6351	4123	4106	4106
query73	776	370	367	367
query74	10336	8868	8868	8868
query75	3470	2676	2689	2676
query76	3523	1187	1158	1158
query77	563	270	286	270
query78	11900	9998	9341	9341
query79	1111	596	595	595
query80	718	453	433	433
query81	543	228	233	228
query82	325	127	125	125
query83	262	213	142	142
query84	238	69	70	69
query85	776	307	299	299
query86	425	305	303	303
query87	4497	4472	4412	4412
query88	3729	2225	2205	2205
query89	406	294	294	294
query90	2249	197	194	194
query91	137	102	103	102
query92	66	55	53	53
query93	1291	543	547	543
query94	1026	283	297	283
query95	358	254	254	254
query96	626	281	280	280
query97	2838	2646	2663	2646
query98	214	203	197	197
query99	1508	1322	1300	1300
Total cold run time: 305045 ms
Total hot run time: 189933 ms

doris-robot · 2024-12-20T08:50:27Z

ClickBench: Total hot run time: 32.85 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 08b123bfdd25c2b71e27e18afe2e396b80466034, data reload: false

query1	0.03	0.03	0.03
query2	0.09	0.04	0.03
query3	0.23	0.07	0.08
query4	1.60	0.10	0.11
query5	0.42	0.39	0.42
query6	1.16	0.65	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.50	0.51
query10	0.55	0.58	0.56
query11	0.14	0.11	0.11
query12	0.14	0.12	0.11
query13	0.60	0.60	0.59
query14	2.87	2.71	2.84
query15	0.89	0.82	0.82
query16	0.38	0.38	0.38
query17	1.06	1.06	1.00
query18	0.23	0.21	0.21
query19	1.95	1.78	1.98
query20	0.01	0.01	0.01
query21	15.35	0.58	0.56
query22	2.41	2.07	2.57
query23	17.11	1.01	0.71
query24	3.03	1.15	1.88
query25	0.17	0.23	0.05
query26	0.60	0.14	0.13
query27	0.04	0.05	0.05
query28	9.89	1.11	1.07
query29	12.57	3.22	3.22
query30	0.25	0.06	0.06
query31	2.85	0.39	0.39
query32	3.25	0.47	0.46
query33	3.03	3.10	3.10
query34	17.16	4.48	4.55
query35	4.54	4.53	4.48
query36	0.68	0.48	0.49
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.12	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.03
query43	0.03	0.02	0.03
Total cold run time: 106.41 s
Total hot run time: 32.85 s

Gabriel39 · 2024-12-20T14:37:35Z

run buildall

doris-robot · 2024-12-20T15:11:48Z

TPC-H: Total hot run time: 39954 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8011705f5621ae333b9a538477e31119ed66c2ab, data reload: false

------ Round 1 ----------------------------------
q1	17594	7420	7289	7289
q2	2043	174	169	169
q3	10552	1145	1175	1145
q4	10234	730	730	730
q5	7615	2743	2620	2620
q6	246	154	155	154
q7	993	634	613	613
q8	9255	1826	1936	1826
q9	6577	6481	6388	6388
q10	6978	2281	2326	2281
q11	473	277	270	270
q12	429	217	227	217
q13	17782	2942	3017	2942
q14	262	209	208	208
q15	563	522	500	500
q16	659	590	597	590
q17	990	525	545	525
q18	7391	6778	6822	6778
q19	1343	942	947	942
q20	475	186	186	186
q21	4031	3279	3324	3279
q22	370	302	316	302
Total cold run time: 106855 ms
Total hot run time: 39954 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7298	7210	7238	7210
q2	334	231	231	231
q3	2894	2741	2970	2741
q4	2092	1860	1849	1849
q5	5571	5680	5674	5674
q6	222	144	144	144
q7	2212	1792	1906	1792
q8	3395	3469	3497	3469
q9	8879	8935	8874	8874
q10	3638	3595	3544	3544
q11	599	507	510	507
q12	793	639	597	597
q13	12873	3108	3037	3037
q14	299	302	266	266
q15	584	499	512	499
q16	685	642	651	642
q17	1865	1637	1616	1616
q18	7815	7535	7318	7318
q19	1683	1474	1551	1474
q20	2073	1822	1888	1822
q21	5418	5256	5298	5256
q22	662	564	574	564
Total cold run time: 71884 ms
Total hot run time: 59126 ms

doris-robot · 2024-12-20T15:23:51Z

TPC-DS: Total hot run time: 189600 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8011705f5621ae333b9a538477e31119ed66c2ab, data reload: false

query1	969	389	391	389
query2	6538	2375	2292	2292
query3	6705	229	220	220
query4	33636	23449	23376	23376
query5	4375	453	461	453
query6	282	186	187	186
query7	4622	311	313	311
query8	308	236	240	236
query9	9353	2697	2689	2689
query10	471	256	253	253
query11	17985	15068	15302	15068
query12	165	106	106	106
query13	1764	425	433	425
query14	11869	6769	7135	6769
query15	325	190	194	190
query16	8252	450	401	401
query17	1760	606	578	578
query18	2123	285	289	285
query19	365	167	148	148
query20	128	109	109	109
query21	211	103	103	103
query22	4767	4513	4184	4184
query23	34787	33391	33420	33391
query24	11015	2482	2436	2436
query25	681	376	388	376
query26	1859	159	158	158
query27	2904	338	320	320
query28	7942	2437	2434	2434
query29	1041	404	416	404
query30	304	148	152	148
query31	1038	826	812	812
query32	94	57	60	57
query33	778	295	286	286
query34	1006	508	528	508
query35	898	754	729	729
query36	1100	932	933	932
query37	271	76	75	75
query38	4234	4058	4110	4058
query39	1505	1418	1412	1412
query40	300	101	100	100
query41	47	45	50	45
query42	130	101	106	101
query43	546	511	490	490
query44	1281	805	813	805
query45	184	167	162	162
query46	1171	716	718	716
query47	1945	1850	1828	1828
query48	409	316	327	316
query49	1278	383	399	383
query50	794	380	385	380
query51	7119	7198	7131	7131
query52	104	92	95	92
query53	255	185	187	185
query54	1195	400	408	400
query55	78	82	83	82
query56	268	246	248	246
query57	1265	1098	1129	1098
query58	240	218	228	218
query59	3173	2988	2924	2924
query60	296	254	255	254
query61	112	110	117	110
query62	885	674	690	674
query63	217	190	186	186
query64	5043	677	681	677
query65	3272	3216	3243	3216
query66	1419	312	303	303
query67	15964	15556	15610	15556
query68	5736	564	561	561
query69	427	255	257	255
query70	1236	1048	1153	1048
query71	449	255	257	255
query72	6439	4036	4035	4035
query73	780	351	361	351
query74	9272	8881	8778	8778
query75	3411	2721	2654	2654
query76	3602	1088	1054	1054
query77	551	286	274	274
query78	10240	9450	9386	9386
query79	1680	611	590	590
query80	1043	481	417	417
query81	527	233	235	233
query82	923	115	121	115
query83	244	163	145	145
query84	235	70	72	70
query85	1264	302	313	302
query86	394	302	296	296
query87	4420	4419	4438	4419
query88	3348	2236	2199	2199
query89	424	297	293	293
query90	2167	197	192	192
query91	134	105	114	105
query92	62	49	59	49
query93	1282	535	545	535
query94	1151	285	298	285
query95	357	256	252	252
query96	611	277	287	277
query97	2805	2692	2688	2688
query98	223	205	196	196
query99	1570	1306	1320	1306
Total cold run time: 304427 ms
Total hot run time: 189600 ms

doris-robot · 2024-12-20T15:29:22Z

ClickBench: Total hot run time: 32.22 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8011705f5621ae333b9a538477e31119ed66c2ab, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.44	0.41	0.40
query6	1.16	0.66	0.66
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.54	0.56	0.56
query11	0.15	0.10	0.11
query12	0.13	0.11	0.11
query13	0.61	0.62	0.59
query14	2.84	2.86	2.75
query15	0.89	0.83	0.82
query16	0.41	0.39	0.37
query17	1.03	1.05	1.01
query18	0.22	0.21	0.21
query19	1.87	1.78	2.02
query20	0.02	0.00	0.01
query21	15.35	0.61	0.60
query22	3.08	2.40	1.35
query23	17.00	1.04	1.01
query24	3.42	0.81	1.44
query25	0.32	0.06	0.19
query26	0.47	0.15	0.14
query27	0.06	0.05	0.04
query28	10.32	1.11	1.07
query29	12.60	3.22	3.23
query30	0.24	0.06	0.06
query31	2.84	0.38	0.40
query32	3.24	0.48	0.47
query33	3.11	3.24	3.15
query34	16.83	4.45	4.50
query35	4.51	4.53	4.48
query36	0.69	0.50	0.49
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.12	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 107.47 s
Total hot run time: 32.22 s

doris-robot · 2024-12-20T15:41:47Z

TeamCity be ut coverage result:
Function Coverage: 38.88% (10133/26064)
Line Coverage: 29.81% (85300/286148)
Region Coverage: 28.93% (43553/150530)
Branch Coverage: 25.44% (22181/87178)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8011705f5621ae333b9a538477e31119ed66c2ab_8011705f5621ae333b9a538477e31119ed66c2ab/report/index.html

[Improvement](scheduler) Use a separate eager queue to execute cancel…

e718439

…ed tasks

update

56171ee

Gabriel39 force-pushed the imp_1218 branch from f4fda83 to 9597171 Compare December 19, 2024 03:01

github-actions bot reviewed Dec 19, 2024

View reviewed changes

be/src/pipeline/task_queue.cpp Show resolved Hide resolved

be/src/pipeline/task_queue.cpp Show resolved Hide resolved

use shared ptr

42532ad

Gabriel39 force-pushed the imp_1218 branch from e624dc5 to 42532ad Compare December 19, 2024 07:24

update

6e2fe8b

yiguolei reviewed Dec 19, 2024

View reviewed changes

be/src/pipeline/pipeline_task.cpp Outdated Show resolved Hide resolved

yiguolei reviewed Dec 19, 2024

View reviewed changes

be/src/pipeline/task_queue.h Outdated Show resolved Hide resolved

yiguolei reviewed Dec 19, 2024

View reviewed changes

be/src/pipeline/task_queue.h Outdated Show resolved Hide resolved

yiguolei reviewed Dec 19, 2024

View reviewed changes

be/src/pipeline/pipeline_task.h Outdated Show resolved Hide resolved

yiguolei reviewed Dec 19, 2024

View reviewed changes

be/src/pipeline/task_queue.cpp Outdated Show resolved Hide resolved

update

4a17d43

update

d1c6313

github-actions bot reviewed Dec 20, 2024

View reviewed changes

update

08b123b

update

8011705

	std::shared_ptr<PipelineTask> MultiCoreTaskQueue::_steal_take(int core_id) {
	std::shared_ptr<PipelineTask> MultiCoreTaskQueue::_steal_take(int core_id) const {

[Improvement](scheduler) Use a separate eager queue to execute cancel… #45614

Are you sure you want to change the base?

[Improvement](scheduler) Use a separate eager queue to execute cancel… #45614

Conversation

Gabriel39 commented Dec 18, 2024 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

hello-stephen commented Dec 18, 2024

Gabriel39 commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

sh-checker report

github-actions bot commented Dec 18, 2024

github-actions bot commented Dec 19, 2024

sh-checker report

Gabriel39 commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

doris-robot commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

sh-checker report

Gabriel39 commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

sh-checker report

github-actions bot left a comment

Choose a reason for hiding this comment

doris-robot commented Dec 19, 2024

Gabriel39 commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

sh-checker report

doris-robot commented Dec 19, 2024

Gabriel39 commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

sh-checker report

doris-robot commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

sh-checker report

Gabriel39 commented Dec 19, 2024

doris-robot commented Dec 19, 2024

github-actions bot commented Dec 20, 2024

sh-checker report

github-actions bot commented Dec 20, 2024

sh-checker report

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Dec 20, 2024

Choose a reason for hiding this comment

Gabriel39 commented Dec 20, 2024

doris-robot commented Dec 20, 2024

Gabriel39 commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

Gabriel39 commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

doris-robot commented Dec 20, 2024

Gabriel39 commented Dec 18, 2024 •

edited

Loading

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`