Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix](flexible partial update) Fix some problems in flexible partial update #41701

Open
wants to merge 43 commits into
base: master
Choose a base branch
from

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Oct 11, 2024

  1. Fix incorrect result when there are insert after delete for same key in one load. In flexible partial update, if there is an insert after a delete on the same key in one batch, the final result will be always delete that row in table, which is incorrect. The reason is that when doing aggregation for rows with same keys in memtable, we treat delete sign column as an ordinary column. So if there is a row with delete sign in that batch, the final aggregated row will always have a delete sign and any insert on the same key after it will be lost.
    In the above situation, we should keep those rows and don't merge them when doing aggregation in memtable. When flush the memtable to segment, we should first apply the semantic of delete if exists and then insert the row after it.
  2. fix flexible partial update don't use row store column to read data
  3. Fix flexible partial update incorrect read index when resolving conflict in publish phase
  4. add some default value cases and fix on update current timestamp
  5. fix publish phase alignment problem when table has sequence column and the rowset to read has multi segments.
  6. add some publish alignment case when table has sequence column
  7. revert [Fix](partial-update) Correct the alignment process when the table has sequence column and add cases #25346, we use the sequence value in flush phase as its final value and will not do alignment in publish phase if the partial update load doesn't specified sequence column.
  8. fix auto-increment column value is wrongly updated when it's not specified.
  9. Fix the problem that the value of sequence column may decrease.
  10. Only read seq col and seq map col when the previous row is deleted in partial update and flexible partial update when doing alignment.
  11. Fix the value of auto-increment column is not consistent between replicas.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@bobhan1
Copy link
Contributor Author

bobhan1 commented Oct 11, 2024

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp Outdated Show resolved Hide resolved
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.39% (9659/25830)
Line Coverage: 28.70% (80276/279695)
Region Coverage: 28.09% (41447/147527)
Branch Coverage: 24.70% (21109/85464)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8dcd7ae197fa48902b27ffb384aab71195d11193_8dcd7ae197fa48902b27ffb384aab71195d11193/report/index.html

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp Outdated Show resolved Hide resolved
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp Outdated Show resolved Hide resolved
@bobhan1 bobhan1 force-pushed the handle-delete-in-one-batch-for-flexible-partial-update branch from 24069a4 to 378ef2f Compare October 11, 2024 12:01
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp Outdated Show resolved Hide resolved
@bobhan1 bobhan1 force-pushed the handle-delete-in-one-batch-for-flexible-partial-update branch from d21f1b1 to d2e1b4c Compare October 11, 2024 12:49
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp Outdated Show resolved Hide resolved
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp Outdated Show resolved Hide resolved
@bobhan1 bobhan1 force-pushed the handle-delete-in-one-batch-for-flexible-partial-update branch 3 times, most recently from 6428fa3 to f177418 Compare October 12, 2024 07:08
@bobhan1
Copy link
Contributor Author

bobhan1 commented Oct 12, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.39% (9666/25854)
Line Coverage: 28.68% (80375/280236)
Region Coverage: 28.06% (41451/147730)
Branch Coverage: 24.68% (21113/85560)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f177418f2be6c8d0161d7df4ecbf5451fd3791c0_f177418f2be6c8d0161d7df4ecbf5451fd3791c0/report/index.html

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp Outdated Show resolved Hide resolved
@bobhan1
Copy link
Contributor Author

bobhan1 commented Oct 12, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40999 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 79fe26c74dbc3442b2b7b76045ec77054fcb9a10, data reload: false

------ Round 1 ----------------------------------
q1	17607	7641	7313	7313
q2	2020	296	297	296
q3	12122	1077	1199	1077
q4	10576	807	829	807
q5	7771	2904	2832	2832
q6	235	150	149	149
q7	1028	649	614	614
q8	9349	1971	1961	1961
q9	6530	6450	6356	6356
q10	7010	2337	2311	2311
q11	438	248	263	248
q12	427	222	214	214
q13	17783	2989	3005	2989
q14	236	223	214	214
q15	581	526	523	523
q16	654	579	591	579
q17	986	508	536	508
q18	7282	6628	6697	6628
q19	1351	1013	1015	1013
q20	483	204	195	195
q21	3925	3196	3232	3196
q22	1089	1019	976	976
Total cold run time: 109483 ms
Total hot run time: 40999 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7298	7248	7274	7248
q2	325	235	235	235
q3	3022	2971	2954	2954
q4	2105	1846	1823	1823
q5	5795	5762	5737	5737
q6	229	141	149	141
q7	2299	1838	1847	1838
q8	3345	3584	3425	3425
q9	8951	8890	8869	8869
q10	3572	3543	3556	3543
q11	587	514	497	497
q12	826	621	652	621
q13	11172	3210	3135	3135
q14	312	286	283	283
q15	579	546	532	532
q16	705	653	632	632
q17	1861	1592	1607	1592
q18	8275	7796	7444	7444
q19	1703	1412	1541	1412
q20	2098	1877	1896	1877
q21	5577	5413	5509	5413
q22	1146	1061	1057	1057
Total cold run time: 71782 ms
Total hot run time: 60308 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.42% (9681/25871)
Line Coverage: 28.70% (80449/280325)
Region Coverage: 28.10% (41532/147820)
Branch Coverage: 24.68% (21115/85568)
Coverage Report: http://coverage.selectdb-in.cc/coverage/79fe26c74dbc3442b2b7b76045ec77054fcb9a10_79fe26c74dbc3442b2b7b76045ec77054fcb9a10/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 192400 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 79fe26c74dbc3442b2b7b76045ec77054fcb9a10, data reload: false

query1	947	404	412	404
query2	6268	2185	2044	2044
query3	8684	191	200	191
query4	33760	23696	23760	23696
query5	3616	500	468	468
query6	272	168	179	168
query7	4212	305	315	305
query8	290	226	226	226
query9	9293	2679	2676	2676
query10	463	278	266	266
query11	17811	15387	15216	15216
query12	156	106	96	96
query13	1583	472	437	437
query14	9590	7462	7457	7457
query15	246	166	178	166
query16	8042	454	489	454
query17	1634	604	559	559
query18	2088	318	321	318
query19	369	146	145	145
query20	121	107	112	107
query21	213	110	103	103
query22	4946	4362	4529	4362
query23	34854	34245	34311	34245
query24	10977	2867	2848	2848
query25	651	406	400	400
query26	1511	160	158	158
query27	2589	307	292	292
query28	7684	2410	2409	2409
query29	954	426	428	426
query30	262	158	155	155
query31	1023	795	793	793
query32	100	53	55	53
query33	768	303	314	303
query34	908	516	501	501
query35	879	738	736	736
query36	1109	940	932	932
query37	148	84	85	84
query38	4035	3939	3881	3881
query39	1545	1449	1402	1402
query40	255	99	98	98
query41	48	45	44	44
query42	120	95	96	95
query43	556	496	493	493
query44	1217	806	840	806
query45	200	166	159	159
query46	1143	731	713	713
query47	1933	1843	1850	1843
query48	417	358	341	341
query49	981	421	417	417
query50	812	415	414	414
query51	7139	6944	6842	6842
query52	99	86	88	86
query53	253	184	181	181
query54	1401	470	481	470
query55	77	80	79	79
query56	307	272	261	261
query57	1236	1187	1191	1187
query58	256	251	303	251
query59	3281	3221	3028	3028
query60	317	304	287	287
query61	125	119	118	118
query62	854	665	666	665
query63	225	197	190	190
query64	5112	669	601	601
query65	3306	3168	3203	3168
query66	1381	295	326	295
query67	16016	15729	15562	15562
query68	4419	564	562	562
query69	508	295	302	295
query70	1135	1066	1125	1066
query71	369	264	272	264
query72	7516	3927	3928	3927
query73	761	334	344	334
query74	10190	9073	9087	9073
query75	4011	2674	2715	2674
query76	3237	832	878	832
query77	647	330	315	315
query78	10462	9627	9513	9513
query79	1486	590	592	590
query80	2654	432	452	432
query81	596	245	247	245
query82	732	137	138	137
query83	321	142	141	141
query84	277	84	75	75
query85	1698	299	285	285
query86	449	293	299	293
query87	4520	4378	4278	4278
query88	3215	2396	2368	2368
query89	410	288	284	284
query90	2149	185	180	180
query91	138	105	106	105
query92	64	47	48	47
query93	1172	526	538	526
query94	1238	286	295	286
query95	354	255	251	251
query96	608	278	281	278
query97	3248	3130	3122	3122
query98	219	205	190	190
query99	1534	1256	1307	1256
Total cold run time: 304005 ms
Total hot run time: 192400 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.6 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 79fe26c74dbc3442b2b7b76045ec77054fcb9a10, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.65	0.10	0.10
query5	0.52	0.50	0.50
query6	1.14	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.05	0.03
query9	0.56	0.52	0.51
query10	0.56	0.54	0.56
query11	0.14	0.11	0.11
query12	0.14	0.10	0.10
query13	0.62	0.60	0.59
query14	2.71	2.72	2.85
query15	0.92	0.82	0.85
query16	0.38	0.38	0.38
query17	0.96	1.04	1.00
query18	0.19	0.20	0.19
query19	1.92	1.75	1.89
query20	0.01	0.02	0.01
query21	15.35	0.57	0.55
query22	2.99	2.21	1.95
query23	17.01	0.92	0.79
query24	2.77	2.07	1.00
query25	0.28	0.08	0.12
query26	0.50	0.15	0.13
query27	0.05	0.04	0.04
query28	10.10	1.09	1.08
query29	12.59	3.24	3.19
query30	0.24	0.06	0.05
query31	2.89	0.37	0.38
query32	3.29	0.46	0.47
query33	3.01	3.00	3.02
query34	16.68	4.51	4.45
query35	4.51	4.46	4.50
query36	0.65	0.48	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.2 s
Total hot run time: 32.6 s

@bobhan1 bobhan1 force-pushed the handle-delete-in-one-batch-for-flexible-partial-update branch from 79fe26c to 02cb8af Compare October 12, 2024 11:29
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/memtable.cpp Show resolved Hide resolved
be/src/olap/memtable.cpp Show resolved Hide resolved
@bobhan1
Copy link
Contributor Author

bobhan1 commented Oct 12, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.42% (9681/25873)
Line Coverage: 28.63% (80267/280324)
Region Coverage: 28.09% (41525/147830)
Branch Coverage: 24.67% (21110/85580)
Coverage Report: http://coverage.selectdb-in.cc/coverage/02cb8af6bdf15fd2a892f3a48fa8c8b4446761a7_02cb8af6bdf15fd2a892f3a48fa8c8b4446761a7/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40756 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 02cb8af6bdf15fd2a892f3a48fa8c8b4446761a7, data reload: false

------ Round 1 ----------------------------------
q1	17571	7956	7273	7273
q2	2024	288	271	271
q3	12151	1041	1145	1041
q4	10583	773	706	706
q5	7767	2876	2827	2827
q6	246	156	147	147
q7	1002	633	624	624
q8	9341	1933	2020	1933
q9	6562	6454	6439	6439
q10	6975	2329	2321	2321
q11	449	247	245	245
q12	414	215	216	215
q13	17796	2981	3010	2981
q14	241	209	218	209
q15	568	534	525	525
q16	665	585	588	585
q17	985	571	555	555
q18	7248	6579	6839	6579
q19	1351	964	1057	964
q20	484	201	199	199
q21	3980	3108	3294	3108
q22	1083	1009	1026	1009
Total cold run time: 109486 ms
Total hot run time: 40756 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7263	7218	7239	7218
q2	330	236	233	233
q3	3063	2920	2952	2920
q4	2084	1878	1825	1825
q5	5785	5785	5763	5763
q6	233	141	144	141
q7	2257	1868	1794	1794
q8	3398	3570	3412	3412
q9	8954	8889	8869	8869
q10	3616	3524	3482	3482
q11	580	493	513	493
q12	815	636	658	636
q13	11153	3195	3185	3185
q14	301	284	277	277
q15	595	528	525	525
q16	694	656	649	649
q17	1857	1648	1640	1640
q18	8189	7842	7503	7503
q19	1712	1512	1449	1449
q20	2108	1879	1928	1879
q21	5411	5409	5414	5409
q22	1127	1044	1049	1044
Total cold run time: 71525 ms
Total hot run time: 60346 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192006 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 02cb8af6bdf15fd2a892f3a48fa8c8b4446761a7, data reload: false

query1	921	404	388	388
query2	6258	2109	2109	2109
query3	8694	190	208	190
query4	34238	23596	23396	23396
query5	3520	478	474	474
query6	266	165	167	165
query7	4185	299	305	299
query8	287	249	212	212
query9	9532	2654	2645	2645
query10	416	281	284	281
query11	17678	15208	15604	15208
query12	152	97	93	93
query13	1569	459	450	450
query14	9381	7692	7240	7240
query15	242	168	169	168
query16	7296	440	433	433
query17	1667	624	595	595
query18	1393	319	320	319
query19	298	160	154	154
query20	121	113	115	113
query21	206	103	107	103
query22	4746	4496	4326	4326
query23	35012	34207	34207	34207
query24	11103	2856	2916	2856
query25	606	421	410	410
query26	1068	171	168	168
query27	2354	301	300	300
query28	7389	2415	2409	2409
query29	761	447	429	429
query30	259	159	156	156
query31	1058	800	794	794
query32	96	56	56	56
query33	765	299	303	299
query34	896	504	503	503
query35	877	750	728	728
query36	1084	945	951	945
query37	153	92	88	88
query38	3992	3835	3803	3803
query39	1474	1426	1434	1426
query40	206	98	97	97
query41	47	44	46	44
query42	125	102	97	97
query43	533	499	502	499
query44	1217	826	806	806
query45	193	163	168	163
query46	1175	714	726	714
query47	1942	1855	1837	1837
query48	443	364	349	349
query49	928	417	411	411
query50	821	428	423	423
query51	7133	6943	6961	6943
query52	103	91	86	86
query53	256	181	182	181
query54	1241	476	480	476
query55	83	78	75	75
query56	309	280	273	273
query57	1214	1152	1171	1152
query58	242	234	246	234
query59	3392	3107	3079	3079
query60	303	273	266	266
query61	103	99	102	99
query62	878	681	674	674
query63	220	194	190	190
query64	4340	641	610	610
query65	3249	3292	3208	3208
query66	1266	311	306	306
query67	16017	15554	15634	15554
query68	4427	567	535	535
query69	508	306	310	306
query70	1188	1111	1120	1111
query71	368	270	283	270
query72	7185	4027	3992	3992
query73	781	352	354	352
query74	10401	8950	8863	8863
query75	3392	2704	2724	2704
query76	3046	921	865	865
query77	573	298	286	286
query78	10437	9609	9527	9527
query79	3470	613	600	600
query80	2328	455	458	455
query81	587	237	232	232
query82	725	139	137	137
query83	322	129	132	129
query84	281	79	78	78
query85	1122	295	294	294
query86	324	302	306	302
query87	4513	4426	4268	4268
query88	4078	2396	2362	2362
query89	416	287	290	287
query90	2083	187	182	182
query91	151	107	104	104
query92	61	46	48	46
query93	1683	520	533	520
query94	886	306	298	298
query95	354	256	252	252
query96	619	276	273	273
query97	3245	3109	3098	3098
query98	216	217	190	190
query99	1510	1324	1304	1304
Total cold run time: 301272 ms
Total hot run time: 192006 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.62 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 02cb8af6bdf15fd2a892f3a48fa8c8b4446761a7, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.52	0.52	0.50
query6	1.14	0.73	0.73
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.56	0.51	0.49
query10	0.55	0.55	0.56
query11	0.14	0.11	0.11
query12	0.13	0.11	0.11
query13	0.62	0.61	0.60
query14	2.86	2.71	2.88
query15	0.90	0.83	0.83
query16	0.38	0.39	0.38
query17	1.00	1.08	1.02
query18	0.19	0.20	0.20
query19	1.96	1.87	2.00
query20	0.02	0.01	0.00
query21	15.35	0.59	0.60
query22	3.09	1.82	1.83
query23	17.28	1.01	1.04
query24	3.43	1.69	2.09
query25	0.31	0.18	0.24
query26	0.60	0.14	0.14
query27	0.05	0.05	0.03
query28	9.16	1.10	1.08
query29	12.58	3.25	3.16
query30	0.24	0.06	0.06
query31	2.90	0.37	0.37
query32	3.28	0.46	0.45
query33	3.04	3.05	3.08
query34	17.08	4.41	4.50
query35	4.53	4.46	4.52
query36	0.66	0.48	0.48
query37	0.08	0.06	0.05
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 107.05 s
Total hot run time: 33.62 s

bobhan1 and others added 17 commits January 16, 2025 15:10
…ence column may decrease (apache#198)

* fix partial update that will cause seq col value not increasing problem

* fix flexible partial update that will cause seq col value not increasing problem

* fix existing case
… the previous row is deleted in partial update and flexible partial update when doing alignment (apache#206)

* only read seq col and seq map col from old rows if old rows if deleted when do alignment for flexible partial update

* let partial update only read seq col if the previous row is deleted in flush phase

* add flexible partial update publish case

* fix partial update publish and add caser
…is not consistent between replicas (apache#207)

* add reproduce case

* fix and update case

* fix auto-increment value not consistent between replicas

* add case
@bobhan1 bobhan1 force-pushed the handle-delete-in-one-batch-for-flexible-partial-update branch from 9cc0eab to 507956a Compare January 16, 2025 07:13
@bobhan1
Copy link
Contributor Author

bobhan1 commented Jan 16, 2025

run buildall

@bobhan1
Copy link
Contributor Author

bobhan1 commented Jan 16, 2025

run buildall

@bobhan1
Copy link
Contributor Author

bobhan1 commented Jan 16, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32482 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 21ec0541fbd8c9eab4a761c02f69ddb72c4856a6, data reload: false

------ Round 1 ----------------------------------
q1	17588	5541	5446	5446
q2	2058	307	176	176
q3	10405	1271	740	740
q4	10225	966	529	529
q5	7543	2410	2236	2236
q6	197	168	137	137
q7	928	772	610	610
q8	9239	1387	1201	1201
q9	5247	4908	4915	4908
q10	6836	2342	1897	1897
q11	474	271	252	252
q12	343	356	221	221
q13	17777	3677	3059	3059
q14	241	238	209	209
q15	527	467	484	467
q16	635	614	584	584
q17	563	899	330	330
q18	6866	6436	6375	6375
q19	1211	966	555	555
q20	307	337	209	209
q21	2819	2167	2031	2031
q22	371	333	310	310
Total cold run time: 102400 ms
Total hot run time: 32482 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5554	5546	5507	5507
q2	248	335	231	231
q3	2250	2606	2312	2312
q4	1393	1875	1371	1371
q5	4370	4764	4700	4700
q6	168	159	124	124
q7	2080	1994	1832	1832
q8	2660	2851	2680	2680
q9	7334	7190	7303	7190
q10	2982	3308	2813	2813
q11	606	555	497	497
q12	671	756	628	628
q13	3492	3918	3273	3273
q14	275	295	264	264
q15	525	474	458	458
q16	678	672	652	652
q17	1259	1743	1260	1260
q18	7797	7530	7221	7221
q19	806	1223	1051	1051
q20	1988	2025	1885	1885
q21	5621	5365	4876	4876
q22	599	594	582	582
Total cold run time: 53356 ms
Total hot run time: 51407 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187654 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 21ec0541fbd8c9eab4a761c02f69ddb72c4856a6, data reload: false

query1	1000	389	384	384
query2	6520	2040	2048	2040
query3	6795	212	213	212
query4	33399	23280	23469	23280
query5	4373	632	468	468
query6	293	197	187	187
query7	4603	508	319	319
query8	301	236	239	236
query9	9326	2586	2580	2580
query10	479	305	262	262
query11	17680	15296	14891	14891
query12	157	104	109	104
query13	1664	526	414	414
query14	10319	6528	7102	6528
query15	238	198	175	175
query16	8075	640	449	449
query17	1590	725	542	542
query18	2124	394	305	305
query19	209	187	155	155
query20	111	108	109	108
query21	204	120	100	100
query22	4248	4273	4371	4273
query23	34127	32921	32961	32921
query24	6559	2182	2226	2182
query25	509	460	385	385
query26	1200	267	161	161
query27	2025	463	335	335
query28	5268	2472	2415	2415
query29	715	545	417	417
query30	229	184	155	155
query31	953	858	791	791
query32	96	64	62	62
query33	550	372	304	304
query34	749	862	486	486
query35	806	821	753	753
query36	1001	1018	965	965
query37	126	104	84	84
query38	4095	4228	3989	3989
query39	1483	1435	1425	1425
query40	205	111	102	102
query41	53	53	52	52
query42	133	104	101	101
query43	532	531	492	492
query44	1270	850	796	796
query45	181	172	177	172
query46	852	1037	647	647
query47	1811	1837	1802	1802
query48	378	394	311	311
query49	782	483	394	394
query50	641	648	394	394
query51	6832	6868	6885	6868
query52	101	103	90	90
query53	221	248	186	186
query54	472	480	422	422
query55	79	79	82	79
query56	269	256	243	243
query57	1176	1136	1088	1088
query58	242	242	253	242
query59	2988	3135	3120	3120
query60	293	286	308	286
query61	126	118	120	118
query62	801	717	651	651
query63	214	193	187	187
query64	4523	1052	653	653
query65	3279	3140	3168	3140
query66	1085	416	301	301
query67	16036	15564	15644	15564
query68	2174	816	554	554
query69	418	304	261	261
query70	1204	1179	1157	1157
query71	312	283	262	262
query72	6040	3859	3854	3854
query73	631	746	355	355
query74	9684	9254	8752	8752
query75	3130	3186	2695	2695
query76	2205	1183	777	777
query77	349	362	282	282
query78	10084	10252	9351	9351
query79	988	868	614	614
query80	1175	554	437	437
query81	521	281	237	237
query82	357	153	127	127
query83	176	176	151	151
query84	239	92	73	73
query85	765	350	294	294
query86	335	327	316	316
query87	4530	4418	4424	4418
query88	3663	2137	2109	2109
query89	399	331	286	286
query90	1830	193	188	188
query91	130	139	109	109
query92	57	59	56	56
query93	977	850	544	544
query94	691	398	307	307
query95	354	271	266	266
query96	484	613	278	278
query97	2801	2879	2741	2741
query98	221	203	193	193
query99	1284	1416	1267	1267
Total cold run time: 277982 ms
Total hot run time: 187654 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 21ec0541fbd8c9eab4a761c02f69ddb72c4856a6, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.08	0.07
query4	1.61	0.11	0.11
query5	0.43	0.41	0.42
query6	1.17	0.65	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.49	0.53
query10	0.53	0.56	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.59	0.59
query14	2.69	2.71	2.79
query15	0.89	0.83	0.83
query16	0.39	0.38	0.38
query17	1.07	1.04	1.05
query18	0.22	0.20	0.21
query19	1.84	1.76	2.00
query20	0.02	0.01	0.01
query21	15.42	0.92	0.58
query22	0.75	0.67	0.62
query23	15.44	1.40	0.50
query24	2.95	0.58	1.06
query25	0.20	0.17	0.07
query26	0.42	0.16	0.15
query27	0.08	0.05	0.05
query28	13.13	1.08	0.44
query29	12.95	3.97	3.29
query30	0.25	0.10	0.06
query31	2.85	0.60	0.39
query32	3.23	0.55	0.47
query33	3.01	2.98	3.08
query34	16.84	5.10	4.45
query35	4.52	4.49	4.52
query36	0.63	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.14	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 105.94 s
Total hot run time: 30.02 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.50% (10823/26078)
Line Coverage: 31.88% (91443/286814)
Region Coverage: 31.03% (46863/151005)
Branch Coverage: 27.13% (23735/87494)
Coverage Report: http://coverage.selectdb-in.cc/coverage/21ec0541fbd8c9eab4a761c02f69ddb72c4856a6_21ec0541fbd8c9eab4a761c02f69ddb72c4856a6/report/index.html

@bobhan1
Copy link
Contributor Author

bobhan1 commented Jan 17, 2025

run external

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants