Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Opt](load) only MoW table need table commit lock when commit transaction for cloud mode (#45220) #46045

Open
wants to merge 2 commits into
base: branch-3.0
Choose a base branch
from

Conversation

liaoxin01
Copy link
Contributor

cherry-pick from #45220

@Thearas
Copy link
Contributor

Thearas commented Dec 26, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01
Copy link
Contributor Author

run buildall

@liaoxin01
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41082 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a47c875c25998ad07d6e04988014d9e92f380b02, data reload: false

------ Round 1 ----------------------------------
q1	17571	8252	7310	7310
q2	2060	175	175	175
q3	10723	1110	1156	1110
q4	10571	784	802	784
q5	7749	2858	2872	2858
q6	237	145	148	145
q7	980	614	624	614
q8	9360	1944	2022	1944
q9	6581	6464	6418	6418
q10	7039	2272	2332	2272
q11	468	266	282	266
q12	407	221	212	212
q13	17805	3037	3010	3010
q14	246	220	207	207
q15	566	528	520	520
q16	706	626	603	603
q17	997	650	522	522
q18	7289	6715	6642	6642
q19	1402	1067	1110	1067
q20	487	210	206	206
q21	4013	3227	3202	3202
q22	1126	999	995	995
Total cold run time: 108383 ms
Total hot run time: 41082 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7267	7247	7202	7202
q2	325	255	230	230
q3	3044	2924	2981	2924
q4	2108	1879	1780	1780
q5	5719	5721	5744	5721
q6	227	144	144	144
q7	2242	1827	1826	1826
q8	3358	3587	3417	3417
q9	8914	8945	8880	8880
q10	3595	3591	3524	3524
q11	602	501	501	501
q12	817	636	646	636
q13	9360	3203	3167	3167
q14	303	279	272	272
q15	584	530	522	522
q16	712	675	681	675
q17	1853	1638	1585	1585
q18	8271	7772	7706	7706
q19	1714	1517	1656	1517
q20	2091	1911	1870	1870
q21	5534	5351	5492	5351
q22	1137	1086	1043	1043
Total cold run time: 69777 ms
Total hot run time: 60493 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199187 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a47c875c25998ad07d6e04988014d9e92f380b02, data reload: false

query1	1305	924	910	910
query2	6215	2098	2123	2098
query3	10969	4364	4463	4364
query4	67238	29008	23479	23479
query5	4967	458	446	446
query6	403	173	179	173
query7	5506	309	306	306
query8	312	240	237	237
query9	8834	2710	2694	2694
query10	437	255	261	255
query11	17136	15216	15796	15216
query12	153	117	100	100
query13	1501	450	445	445
query14	10572	7845	8010	7845
query15	212	191	193	191
query16	6644	491	495	491
query17	1282	581	591	581
query18	1000	351	333	333
query19	211	167	163	163
query20	118	114	117	114
query21	65	45	45	45
query22	4761	4676	4600	4600
query23	34712	35096	34579	34579
query24	6254	2900	2845	2845
query25	529	414	417	414
query26	664	168	166	166
query27	1839	304	302	302
query28	4346	2537	2509	2509
query29	721	472	433	433
query30	242	166	165	165
query31	1015	818	855	818
query32	66	56	54	54
query33	412	282	268	268
query34	913	502	503	502
query35	888	769	759	759
query36	1085	984	979	979
query37	135	75	71	71
query38	4206	4049	4092	4049
query39	1527	1465	1491	1465
query40	142	85	83	83
query41	51	48	45	45
query42	108	97	100	97
query43	556	539	512	512
query44	1186	846	826	826
query45	192	169	167	167
query46	1175	743	749	743
query47	2029	1925	1931	1925
query48	465	376	367	367
query49	718	386	396	386
query50	833	415	420	415
query51	7293	7165	7300	7165
query52	100	92	85	85
query53	251	179	179	179
query54	552	441	446	441
query55	75	75	76	75
query56	267	248	243	243
query57	1172	1103	1153	1103
query58	212	201	205	201
query59	3426	3247	2925	2925
query60	278	251	247	247
query61	113	105	123	105
query62	778	682	670	670
query63	211	192	197	192
query64	1430	637	637	637
query65	3260	3212	3192	3192
query66	695	291	297	291
query67	16000	15734	15768	15734
query68	4037	598	580	580
query69	418	258	264	258
query70	1166	1112	1099	1099
query71	365	252	258	252
query72	6252	4037	4053	4037
query73	745	335	343	335
query74	10218	9020	9203	9020
query75	3361	2620	2662	2620
query76	1816	1043	1044	1043
query77	483	270	275	270
query78	10655	9759	9602	9602
query79	1354	606	596	596
query80	865	432	444	432
query81	505	247	240	240
query82	1265	117	119	117
query83	222	146	151	146
query84	281	82	84	82
query85	898	290	289	289
query86	349	302	298	298
query87	4447	4531	4323	4323
query88	3732	2410	2347	2347
query89	412	290	290	290
query90	2002	183	185	183
query91	185	145	149	145
query92	65	50	48	48
query93	1579	550	548	548
query94	783	294	283	283
query95	356	256	255	255
query96	603	281	280	280
query97	3356	3224	3171	3171
query98	218	201	199	199
query99	1571	1297	1284	1284
Total cold run time: 317527 ms
Total hot run time: 199187 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.88 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a47c875c25998ad07d6e04988014d9e92f380b02, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.11
query5	0.53	0.52	0.51
query6	1.13	0.74	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.57	0.50	0.49
query10	0.54	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.12	0.12
query13	0.61	0.60	0.60
query14	3.03	3.09	2.99
query15	0.92	0.83	0.81
query16	0.39	0.37	0.37
query17	1.07	1.08	1.09
query18	0.24	0.22	0.22
query19	1.92	1.99	1.93
query20	0.01	0.01	0.01
query21	15.37	0.61	0.58
query22	2.52	2.96	1.89
query23	16.88	1.06	0.73
query24	3.04	1.36	0.71
query25	0.24	0.05	0.15
query26	0.52	0.14	0.13
query27	0.05	0.04	0.04
query28	10.81	1.10	1.07
query29	12.60	3.26	3.23
query30	0.24	0.06	0.06
query31	2.89	0.39	0.37
query32	3.27	0.47	0.46
query33	3.01	3.02	3.04
query34	17.05	4.46	4.54
query35	4.49	4.56	4.51
query36	0.64	0.50	0.50
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 107.35 s
Total hot run time: 32.88 s

…tion for cloud mode (apache#45220)

table commit lock only used for MoW table, other table modes do not need
this lock.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants