Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[refactor](hms)Refactor the logic of locking when reading hive acid tables. #46731

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

hubgeter
Copy link
Contributor

@hubgeter hubgeter commented Jan 9, 2025

What problem does this PR solve?

Problem Summary:

  1. Refactor the logic of locking when reading hive acid tables. See the description of HiveAcidTransactionMgr.java for details.
  2. Solve the problem that hms table locks are not released when reading hive acid multi-table join scenarios.
  3. The global QueryCallBackMgr is introduced to release resources that are acquired during the planning process and need to be released after the query ends.

Release note

Solve the problem that hms table locks are not released when reading hive acid multi-table join scenarios.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

hubgeter commented Jan 9, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32293 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cfdc438efbac73caaca4e6fbc5b1e5794b2805bb, data reload: false

------ Round 1 ----------------------------------
q1	17613	6670	6073	6073
q2	2037	305	169	169
q3	10419	1239	706	706
q4	10218	853	432	432
q5	7503	2127	1915	1915
q6	206	180	145	145
q7	882	731	614	614
q8	9231	1342	1172	1172
q9	5169	4845	4721	4721
q10	6752	2276	1857	1857
q11	494	295	254	254
q12	340	354	220	220
q13	17757	3684	3110	3110
q14	228	232	208	208
q15	559	509	494	494
q16	637	629	602	602
q17	558	852	338	338
q18	7003	6601	6301	6301
q19	1241	937	527	527
q20	305	320	188	188
q21	2817	2233	1940	1940
q22	366	327	307	307
Total cold run time: 102335 ms
Total hot run time: 32293 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6179	6190	6215	6190
q2	234	330	236	236
q3	2268	2645	2334	2334
q4	1363	1798	1328	1328
q5	4292	4691	4662	4662
q6	182	176	144	144
q7	2107	1912	1821	1821
q8	2606	2787	2681	2681
q9	7174	7073	7208	7073
q10	3054	3297	2805	2805
q11	596	521	510	510
q12	713	760	681	681
q13	3391	3781	3318	3318
q14	292	297	274	274
q15	559	524	487	487
q16	634	689	649	649
q17	1220	1724	1268	1268
q18	7701	7416	7061	7061
q19	745	1054	1025	1025
q20	1900	1992	1884	1884
q21	5415	5079	4755	4755
q22	588	613	594	594
Total cold run time: 53213 ms
Total hot run time: 51780 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187390 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cfdc438efbac73caaca4e6fbc5b1e5794b2805bb, data reload: false

query1	982	368	359	359
query2	6515	2439	2345	2345
query3	6710	215	216	215
query4	33598	23386	23173	23173
query5	4413	617	456	456
query6	304	230	211	211
query7	4605	477	292	292
query8	283	231	215	215
query9	9669	2698	2659	2659
query10	471	296	237	237
query11	17835	15372	14910	14910
query12	155	113	106	106
query13	1688	517	389	389
query14	9643	6965	6904	6904
query15	256	196	192	192
query16	7663	607	426	426
query17	1599	725	579	579
query18	1381	374	317	317
query19	220	183	166	166
query20	118	125	110	110
query21	206	129	115	115
query22	4261	4416	4284	4284
query23	34694	32496	33827	32496
query24	7681	2311	2323	2311
query25	450	433	375	375
query26	1060	255	146	146
query27	2494	445	340	340
query28	5585	2423	2429	2423
query29	490	526	414	414
query30	230	182	150	150
query31	1045	847	799	799
query32	88	61	60	60
query33	503	347	286	286
query34	723	839	501	501
query35	798	785	728	728
query36	1013	998	931	931
query37	135	97	75	75
query38	4020	4011	4064	4011
query39	1480	1448	1394	1394
query40	205	120	99	99
query41	51	50	49	49
query42	128	104	105	104
query43	523	533	497	497
query44	1330	806	821	806
query45	175	176	159	159
query46	858	1025	624	624
query47	1825	1859	1750	1750
query48	380	385	317	317
query49	756	471	391	391
query50	641	650	389	389
query51	7123	6847	6889	6847
query52	105	100	92	92
query53	219	261	177	177
query54	470	469	407	407
query55	80	75	82	75
query56	260	246	233	233
query57	1183	1171	1072	1072
query58	235	239	242	239
query59	2971	3200	3043	3043
query60	271	246	248	246
query61	117	110	111	110
query62	826	773	725	725
query63	224	192	181	181
query64	4602	1006	662	662
query65	3221	3137	3171	3137
query66	897	433	316	316
query67	15854	15654	15387	15387
query68	8917	687	511	511
query69	465	282	254	254
query70	1232	1125	1130	1125
query71	432	284	243	243
query72	6264	3837	3821	3821
query73	668	744	360	360
query74	10128	9314	8825	8825
query75	4206	3144	2651	2651
query76	4147	1159	748	748
query77	789	345	296	296
query78	9889	10074	9404	9404
query79	3264	785	602	602
query80	686	509	420	420
query81	456	272	235	235
query82	599	152	129	129
query83	196	164	147	147
query84	284	85	77	77
query85	791	350	300	300
query86	429	310	299	299
query87	4296	4301	4186	4186
query88	4417	2191	2132	2132
query89	397	325	295	295
query90	1851	185	185	185
query91	133	133	172	133
query92	67	54	55	54
query93	1209	877	528	528
query94	662	411	288	288
query95	329	265	247	247
query96	481	596	286	286
query97	2877	2893	2803	2803
query98	234	195	195	195
query99	1727	1501	1377	1377
Total cold run time: 293116 ms
Total hot run time: 187390 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.54 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cfdc438efbac73caaca4e6fbc5b1e5794b2805bb, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.11	0.10
query5	0.41	0.44	0.40
query6	1.14	0.65	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.58	0.50	0.50
query10	0.55	0.55	0.56
query11	0.14	0.10	0.11
query12	0.13	0.10	0.11
query13	0.60	0.61	0.60
query14	2.77	2.76	2.71
query15	0.90	0.83	0.81
query16	0.38	0.38	0.37
query17	1.04	1.05	1.08
query18	0.23	0.21	0.21
query19	1.93	1.85	1.97
query20	0.02	0.02	0.01
query21	15.37	0.93	0.59
query22	0.75	0.75	0.69
query23	15.29	1.40	0.54
query24	3.35	1.19	1.57
query25	0.27	0.10	0.10
query26	0.19	0.15	0.14
query27	0.07	0.05	0.04
query28	14.60	1.50	1.04
query29	12.61	3.93	3.28
query30	0.25	0.09	0.06
query31	2.83	0.60	0.39
query32	3.23	0.54	0.45
query33	3.25	3.12	3.11
query34	16.72	5.07	4.50
query35	4.48	4.47	4.46
query36	0.64	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 107.23 s
Total hot run time: 31.54 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants