Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [Optimize](Variant) optimize schema update performance #45480 #45730

Open
wants to merge 2 commits into
base: branch-3.0
Choose a base branch
from

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #45480

When update schema with high concurrency, updaing schemas cost is
expensive.
1. update schema only when rows is not 0
2. copy_from is expensive, use copy constructor
@Thearas
Copy link
Contributor

Thearas commented Dec 20, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Dec 20, 2024
@Thearas
Copy link
Contributor

Thearas commented Dec 20, 2024

run buildall

@eldenmoon
Copy link
Member

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40443 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 28aa84658149fef5266a5f03c3f02a6c32267942, data reload: false

------ Round 1 ----------------------------------
q1	17578	7479	7296	7296
q2	2046	181	168	168
q3	10676	1101	1136	1101
q4	10567	705	730	705
q5	7753	2836	2764	2764
q6	236	150	149	149
q7	964	609	603	603
q8	9575	1912	2017	1912
q9	8093	6431	6447	6431
q10	7010	2332	2310	2310
q11	452	258	253	253
q12	398	216	207	207
q13	17765	2940	2977	2940
q14	264	204	218	204
q15	555	513	526	513
q16	669	614	617	614
q17	976	663	527	527
q18	7270	6487	6581	6487
q19	2517	1095	1008	1008
q20	464	200	194	194
q21	4057	3153	3084	3084
q22	1081	1014	973	973
Total cold run time: 110966 ms
Total hot run time: 40443 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7457	7347	7228	7228
q2	325	229	227	227
q3	3008	2867	2895	2867
q4	2068	1800	1784	1784
q5	5621	5788	5729	5729
q6	228	146	142	142
q7	2197	1789	1773	1773
q8	3352	3626	3500	3500
q9	8976	8963	8943	8943
q10	3571	3479	3499	3479
q11	602	493	509	493
q12	809	596	631	596
q13	16564	3145	3152	3145
q14	291	284	275	275
q15	580	525	523	523
q16	721	671	677	671
q17	1881	1648	1607	1607
q18	8248	8026	7400	7400
q19	4093	1670	1543	1543
q20	2101	1857	1834	1834
q21	5565	5392	5386	5386
q22	1137	1034	1023	1023
Total cold run time: 79395 ms
Total hot run time: 60168 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195053 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 28aa84658149fef5266a5f03c3f02a6c32267942, data reload: false

query1	1222	934	913	913
query2	6246	2136	2107	2107
query3	11015	4223	4200	4200
query4	66440	29123	23480	23480
query5	5347	470	447	447
query6	461	175	172	172
query7	5686	317	305	305
query8	314	229	231	229
query9	9151	2677	2664	2664
query10	514	267	259	259
query11	17860	15144	15658	15144
query12	154	107	102	102
query13	1549	437	427	427
query14	10114	6480	7512	6480
query15	210	186	184	184
query16	7308	526	533	526
query17	1092	543	554	543
query18	1786	307	311	307
query19	216	147	142	142
query20	120	113	107	107
query21	66	41	44	41
query22	4530	4514	4453	4453
query23	34739	33893	33982	33893
query24	6178	2871	2914	2871
query25	516	385	385	385
query26	691	160	158	158
query27	1939	291	289	289
query28	4717	2563	2525	2525
query29	690	446	418	418
query30	253	166	158	158
query31	1007	839	771	771
query32	67	52	54	52
query33	447	284	285	284
query34	927	505	509	505
query35	844	744	737	737
query36	1088	944	931	931
query37	117	74	73	73
query38	4086	3998	3978	3978
query39	1512	1431	1472	1431
query40	149	84	82	82
query41	49	49	49	49
query42	113	96	99	96
query43	539	496	505	496
query44	1174	803	797	797
query45	196	173	179	173
query46	1163	751	696	696
query47	1975	1895	1881	1881
query48	472	375	366	366
query49	744	399	387	387
query50	846	414	414	414
query51	7323	7375	7123	7123
query52	105	90	92	90
query53	252	189	184	184
query54	562	452	461	452
query55	76	76	75	75
query56	262	245	238	238
query57	1211	1086	1079	1079
query58	220	206	207	206
query59	3246	3087	2953	2953
query60	289	285	263	263
query61	106	103	103	103
query62	793	653	662	653
query63	208	179	184	179
query64	1803	653	602	602
query65	3226	3131	3124	3124
query66	733	303	311	303
query67	15941	15272	15313	15272
query68	4574	560	545	545
query69	457	249	247	247
query70	1119	1054	1070	1054
query71	435	255	247	247
query72	6604	3976	3887	3887
query73	778	332	332	332
query74	10440	8983	8972	8972
query75	3304	2613	2632	2613
query76	2549	1099	1138	1099
query77	518	280	260	260
query78	10697	9905	9447	9447
query79	7317	588	598	588
query80	1727	414	402	402
query81	545	241	243	241
query82	1290	120	115	115
query83	249	144	143	143
query84	295	80	78	78
query85	1622	312	288	288
query86	469	303	294	294
query87	4440	4256	4206	4206
query88	5689	2355	2404	2355
query89	417	282	283	282
query90	2043	188	185	185
query91	196	142	146	142
query92	65	47	47	47
query93	6227	530	519	519
query94	841	294	280	280
query95	351	243	248	243
query96	645	274	280	274
query97	3411	3152	3163	3152
query98	216	214	203	203
query99	1746	1307	1286	1286
Total cold run time: 336271 ms
Total hot run time: 195053 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants