Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](compaction) fix compaction producer hold for permits leak #45664

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

qzsee
Copy link
Contributor

@qzsee qzsee commented Dec 19, 2024

What problem does this PR solve?

compaction producer hold
Clipboard_Screenshot_1734609076

compaction threads is free
Clipboard_Screenshot_1734609095
so judge permits has alread leak

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copy link
Collaborator

@TangSiyang2001 TangSiyang2001 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by anyone and no changes requested.

@TangSiyang2001
Copy link
Collaborator

run buildall

garenshi added 3 commits December 20, 2024 11:05
@dqz123
Copy link

dqz123 commented Dec 20, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39698 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c8f3a060048f536c339b6910422e8e59e4978fd8, data reload: false

------ Round 1 ----------------------------------
q1	17581	7438	7252	7252
q2	2043	188	162	162
q3	10607	1067	1186	1067
q4	10577	753	750	750
q5	7614	2685	2666	2666
q6	243	147	145	145
q7	979	657	597	597
q8	9603	1865	1939	1865
q9	6664	6457	6476	6457
q10	7068	2294	2307	2294
q11	473	255	252	252
q12	435	217	219	217
q13	17781	2939	2953	2939
q14	264	215	218	215
q15	558	517	494	494
q16	664	583	582	582
q17	978	499	556	499
q18	7257	6683	6601	6601
q19	1347	1041	1055	1041
q20	488	182	182	182
q21	3979	3111	3176	3111
q22	375	310	317	310
Total cold run time: 107578 ms
Total hot run time: 39698 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7226	7221	7210	7210
q2	330	227	231	227
q3	2881	2819	2902	2819
q4	2032	1781	1841	1781
q5	5680	5656	5646	5646
q6	226	147	146	146
q7	2269	1788	1809	1788
q8	3394	3531	3467	3467
q9	8857	8963	8992	8963
q10	3547	3517	3511	3511
q11	604	491	522	491
q12	828	589	573	573
q13	12961	3056	3152	3056
q14	305	283	274	274
q15	563	509	514	509
q16	688	633	641	633
q17	1837	1618	1609	1609
q18	8260	7873	7700	7700
q19	1769	1450	1540	1450
q20	2102	1864	1862	1862
q21	5542	5415	5447	5415
q22	654	577	585	577
Total cold run time: 72555 ms
Total hot run time: 59707 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195864 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c8f3a060048f536c339b6910422e8e59e4978fd8, data reload: false

query1	1283	979	930	930
query2	6242	2339	2379	2339
query3	10980	4557	4438	4438
query4	33400	23380	23387	23380
query5	4223	465	453	453
query6	282	191	185	185
query7	3993	301	308	301
query8	301	235	239	235
query9	9556	2745	2708	2708
query10	453	250	247	247
query11	18003	15097	15189	15097
query12	160	104	107	104
query13	1564	427	451	427
query14	8648	7041	7593	7041
query15	287	206	197	197
query16	8254	438	437	437
query17	1769	633	587	587
query18	2207	322	327	322
query19	385	175	164	164
query20	123	115	113	113
query21	218	109	113	109
query22	4761	4476	4365	4365
query23	34733	33715	33684	33684
query24	10381	2491	2512	2491
query25	620	403	400	400
query26	1228	158	159	158
query27	2401	354	353	353
query28	7725	2494	2444	2444
query29	843	412	423	412
query30	229	153	180	153
query31	1044	853	842	842
query32	90	56	56	56
query33	765	296	287	287
query34	1162	558	540	540
query35	909	754	759	754
query36	1117	934	947	934
query37	147	76	79	76
query38	4295	4084	4368	4084
query39	1480	1643	1451	1451
query40	208	103	100	100
query41	45	42	45	42
query42	113	101	110	101
query43	541	485	514	485
query44	1269	836	825	825
query45	194	169	167	167
query46	1162	726	717	717
query47	2055	1901	1894	1894
query48	416	330	329	329
query49	923	387	408	387
query50	830	407	385	385
query51	7431	7291	7177	7177
query52	100	91	93	91
query53	261	182	178	178
query54	1106	411	454	411
query55	88	83	78	78
query56	287	237	247	237
query57	1295	1183	1167	1167
query58	234	220	227	220
query59	3288	3053	3070	3053
query60	277	267	268	267
query61	113	106	106	106
query62	907	698	699	698
query63	220	185	188	185
query64	4094	718	650	650
query65	3254	3264	3274	3264
query66	822	309	298	298
query67	16602	15512	15542	15512
query68	5633	542	538	538
query69	504	265	243	243
query70	1198	1115	1122	1115
query71	474	252	249	249
query72	6939	4103	4082	4082
query73	802	359	361	359
query74	10202	8796	8876	8796
query75	3639	2756	2704	2704
query76	3736	1077	1076	1076
query77	606	273	273	273
query78	10153	9364	9444	9364
query79	1972	593	601	593
query80	1288	432	428	428
query81	511	241	231	231
query82	262	120	127	120
query83	200	155	139	139
query84	286	67	71	67
query85	1019	318	296	296
query86	383	306	303	303
query87	4430	4625	4270	4270
query88	4172	2209	2200	2200
query89	444	289	287	287
query90	2168	186	189	186
query91	138	106	102	102
query92	66	48	50	48
query93	4711	545	535	535
query94	917	286	285	285
query95	354	247	250	247
query96	665	279	281	279
query97	2840	2617	2643	2617
query98	217	197	201	197
query99	1608	1313	1324	1313
Total cold run time: 306933 ms
Total hot run time: 195864 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c8f3a060048f536c339b6910422e8e59e4978fd8, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.10	0.12
query5	0.44	0.41	0.39
query6	1.38	0.67	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.50	0.52
query10	0.55	0.55	0.55
query11	0.13	0.10	0.10
query12	0.14	0.11	0.10
query13	0.60	0.60	0.60
query14	2.72	2.73	2.87
query15	0.89	0.84	0.83
query16	0.38	0.39	0.39
query17	1.06	1.06	0.98
query18	0.23	0.20	0.21
query19	1.85	1.87	2.02
query20	0.01	0.01	0.02
query21	15.37	0.58	0.60
query22	2.60	2.69	2.01
query23	16.95	0.95	0.77
query24	2.65	1.02	1.76
query25	0.19	0.27	0.10
query26	0.42	0.14	0.15
query27	0.04	0.04	0.05
query28	10.38	1.13	1.06
query29	12.68	3.26	3.30
query30	0.25	0.07	0.06
query31	2.86	0.38	0.38
query32	3.25	0.47	0.45
query33	3.08	3.21	3.18
query34	17.16	4.47	4.47
query35	4.42	4.47	4.45
query36	0.68	0.49	0.51
query37	0.10	0.06	0.05
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.45 s
Total hot run time: 32.92 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.90% (10138/26063)
Line Coverage: 29.82% (85304/286101)
Region Coverage: 28.94% (43557/150489)
Branch Coverage: 25.45% (22176/87146)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c8f3a060048f536c339b6910422e8e59e4978fd8_c8f3a060048f536c339b6910422e8e59e4978fd8/report/index.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants