[opt](rowset meta) truncate segments key bounds if too large to avoid `RowsetMetaCloudPB` exceeds fdb's 100KB limits #45287

bobhan1 · 2024-12-11T07:51:29Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-12-11T07:51:34Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

github-actions · 2024-12-11T07:56:47Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-12-11T09:27:21Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-12-11T09:40:46Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-12-11T10:41:46Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions

clang-tidy made some suggestions

github-actions · 2024-12-11T12:13:28Z

be/test/olap/segments_key_bounds_truncation_test.cpp

+// specific language governing permissions and limitations
+// under the License.
+
+#include <gen_cpp/olap_file.pb.h>


warning: 'gen_cpp/olap_file.pb.h' file not found [clang-diagnostic-error]

#include <gen_cpp/olap_file.pb.h> ^

be/src/olap/base_tablet.h

zhannngchen · 2024-12-12T06:58:55Z

be/src/util/slice.cpp

@@ -27,4 +27,23 @@ Slice::Slice(const faststring& s)
          data((char*)(s.data())),
          size(s.size()) {}

+bool Slice::origin_is_strictly_less_than(Slice X, bool X_is_truncated, Slice Y,
+                                         bool Y_is_truncated) {


Y_is_truncated is useless?

github-actions

clang-tidy made some suggestions

be/test/olap/segments_key_bounds_truncation_test.cpp

github-actions

clang-tidy made some suggestions

be/test/olap/segments_key_bounds_truncation_test.cpp

github-actions

clang-tidy made some suggestions

be/test/olap/segments_key_bounds_truncation_test.cpp

github-actions

clang-tidy made some suggestions

be/test/olap/segments_key_bounds_truncation_test.cpp

gensrc/proto/olap_file.proto

bobhan1 · 2024-12-18T08:20:06Z

run buildall

doris-robot · 2024-12-18T10:06:51Z

TPC-H: Total hot run time: 40571 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 59d64f429a274bf1f8dfcff61f71bab681cfeade, data reload: false

------ Round 1 ----------------------------------
q1	17599	7532	7261	7261
q2	2037	191	181	181
q3	10528	1194	1211	1194
q4	10241	760	717	717
q5	7628	2748	2755	2748
q6	237	142	144	142
q7	1022	633	615	615
q8	9240	1895	1937	1895
q9	6673	6445	6549	6445
q10	6990	2341	2346	2341
q11	460	267	262	262
q12	404	233	228	228
q13	17785	2996	2901	2901
q14	238	216	215	215
q15	575	499	483	483
q16	668	620	580	580
q17	995	640	673	640
q18	7460	6840	6734	6734
q19	1336	1064	1071	1064
q20	491	188	191	188
q21	4139	3441	3419	3419
q22	391	318	320	318
Total cold run time: 107137 ms
Total hot run time: 40571 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7331	7236	7208	7208
q2	333	231	236	231
q3	2909	2903	3020	2903
q4	2069	1903	1778	1778
q5	5607	5730	5615	5615
q6	223	138	141	138
q7	2171	1769	1771	1769
q8	3407	3549	3494	3494
q9	8945	8876	8931	8876
q10	3566	3578	3582	3578
q11	616	501	505	501
q12	806	652	609	609
q13	13999	3039	3074	3039
q14	295	270	260	260
q15	552	528	503	503
q16	694	650	644	644
q17	1810	1596	1590	1590
q18	7948	7537	7393	7393
q19	1733	1634	1636	1634
q20	2114	1847	1820	1820
q21	5521	5306	5324	5306
q22	639	574	554	554
Total cold run time: 73288 ms
Total hot run time: 59443 ms

doris-robot · 2024-12-18T10:18:54Z

TPC-DS: Total hot run time: 190947 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 59d64f429a274bf1f8dfcff61f71bab681cfeade, data reload: false

query1	983	399	379	379
query2	6528	2540	2395	2395
query3	6717	226	221	221
query4	33837	23805	23313	23313
query5	4331	518	492	492
query6	304	207	191	191
query7	4633	302	305	302
query8	311	247	249	247
query9	9694	2738	2773	2738
query10	482	291	272	272
query11	17917	15118	15188	15118
query12	159	112	115	112
query13	1679	424	426	424
query14	10419	6682	6731	6682
query15	286	180	200	180
query16	8315	469	468	468
query17	1641	610	559	559
query18	2130	300	303	300
query19	393	190	162	162
query20	127	118	119	118
query21	205	104	111	104
query22	4631	4398	4190	4190
query23	34607	33641	33668	33641
query24	10429	2350	2399	2350
query25	655	408	402	402
query26	1197	155	155	155
query27	2899	326	329	326
query28	8032	2413	2408	2408
query29	850	413	427	413
query30	303	153	149	149
query31	1025	824	863	824
query32	102	75	74	74
query33	781	316	311	311
query34	951	499	513	499
query35	879	751	741	741
query36	1089	928	961	928
query37	146	93	92	92
query38	4307	4139	4080	4080
query39	1484	1464	1431	1431
query40	203	110	108	108
query41	51	44	52	44
query42	125	110	115	110
query43	545	501	500	500
query44	1278	813	824	813
query45	186	171	179	171
query46	1164	706	706	706
query47	1984	1832	1866	1832
query48	407	335	323	323
query49	1151	442	396	396
query50	831	392	412	392
query51	7148	7089	7200	7089
query52	109	102	100	100
query53	262	185	189	185
query54	1060	422	458	422
query55	94	92	90	90
query56	275	256	265	256
query57	1254	1129	1163	1129
query58	274	255	264	255
query59	3302	3155	3272	3155
query60	301	270	280	270
query61	108	106	106	106
query62	887	698	687	687
query63	215	195	198	195
query64	4079	670	645	645
query65	3288	3206	3257	3206
query66	854	373	324	324
query67	15833	15474	15558	15474
query68	4944	531	533	531
query69	459	288	281	281
query70	1203	1162	1139	1139
query71	445	270	274	270
query72	6554	4093	4102	4093
query73	763	357	358	357
query74	10493	8859	8931	8859
query75	3531	2648	2673	2648
query76	2996	1085	1027	1027
query77	527	295	286	286
query78	10270	10102	9418	9418
query79	2177	606	608	606
query80	900	444	434	434
query81	548	244	233	233
query82	563	130	133	130
query83	251	150	153	150
query84	234	74	75	74
query85	1305	319	317	317
query86	505	298	286	286
query87	4563	4511	4496	4496
query88	4019	2237	2218	2218
query89	419	301	313	301
query90	2057	199	206	199
query91	139	109	111	109
query92	75	71	73	71
query93	1582	547	542	542
query94	910	297	291	291
query95	398	280	273	273
query96	629	289	283	283
query97	2852	2746	2738	2738
query98	237	209	198	198
query99	1591	1344	1315	1315
Total cold run time: 300865 ms
Total hot run time: 190947 ms

doris-robot · 2024-12-18T10:24:29Z

ClickBench: Total hot run time: 33.74 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 59d64f429a274bf1f8dfcff61f71bab681cfeade, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.05	0.03
query3	0.24	0.07	0.07
query4	1.63	0.10	0.11
query5	0.43	0.39	0.41
query6	1.15	0.65	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.50
query10	0.56	0.58	0.56
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.71	2.76	2.87
query15	0.91	0.83	0.83
query16	0.39	0.39	0.38
query17	1.04	1.08	1.06
query18	0.24	0.22	0.22
query19	1.93	1.99	1.86
query20	0.01	0.01	0.02
query21	15.36	0.59	0.57
query22	2.66	2.38	2.20
query23	16.98	1.01	0.87
query24	3.03	1.65	1.72
query25	0.27	0.13	0.06
query26	0.60	0.14	0.13
query27	0.04	0.04	0.04
query28	9.52	1.09	1.08
query29	12.55	3.23	3.23
query30	0.25	0.06	0.07
query31	2.86	0.39	0.40
query32	3.28	0.48	0.47
query33	3.17	3.02	3.08
query34	16.93	4.41	4.43
query35	4.46	4.47	4.47
query36	0.67	0.48	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.01 s
Total hot run time: 33.74 s

doris-robot · 2024-12-24T05:34:42Z

TPC-DS: Total hot run time: 196292 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b9c1e624feab236ef25265b35de0d2e967375082, data reload: false

query1	1292	967	913	913
query2	6243	2462	2428	2428
query3	10987	4708	4783	4708
query4	32797	23251	23310	23251
query5	4285	469	459	459
query6	280	176	166	166
query7	4021	301	304	301
query8	291	229	232	229
query9	9609	2750	2748	2748
query10	482	250	244	244
query11	18063	15071	15056	15056
query12	159	102	104	102
query13	1586	430	420	420
query14	9157	7429	6812	6812
query15	272	205	193	193
query16	8111	440	464	440
query17	1566	617	577	577
query18	2171	323	334	323
query19	352	170	177	170
query20	129	123	113	113
query21	209	110	114	110
query22	4710	4743	4396	4396
query23	34672	33561	33699	33561
query24	10445	2553	2438	2438
query25	623	392	385	385
query26	1062	152	151	151
query27	2362	341	333	333
query28	6837	2496	2471	2471
query29	823	436	431	431
query30	222	157	150	150
query31	1051	844	896	844
query32	90	57	59	57
query33	772	286	303	286
query34	1031	542	525	525
query35	891	778	766	766
query36	1121	940	988	940
query37	150	78	80	78
query38	4317	4398	4551	4398
query39	1504	1465	1456	1456
query40	207	104	103	103
query41	49	47	45	45
query42	120	100	100	100
query43	543	518	512	512
query44	1315	834	838	834
query45	187	172	177	172
query46	1194	728	733	728
query47	2073	1908	1932	1908
query48	440	338	331	331
query49	965	396	400	396
query50	838	393	430	393
query51	7334	7124	7158	7124
query52	113	97	90	90
query53	265	191	187	187
query54	1240	424	438	424
query55	79	81	84	81
query56	257	258	254	254
query57	1299	1206	1176	1176
query58	249	220	243	220
query59	3295	3170	3301	3170
query60	278	255	263	255
query61	112	108	110	108
query62	894	690	717	690
query63	226	202	198	198
query64	4147	667	657	657
query65	3325	3262	3224	3224
query66	785	307	318	307
query67	16631	15755	15528	15528
query68	5692	548	551	548
query69	495	262	252	252
query70	1199	1186	1089	1089
query71	487	252	248	248
query72	7075	4101	4104	4101
query73	869	381	369	369
query74	10193	8843	8842	8842
query75	3736	2646	2648	2646
query76	3683	1136	1048	1048
query77	585	278	286	278
query78	10889	9600	9375	9375
query79	1975	600	621	600
query80	912	416	433	416
query81	516	247	232	232
query82	368	117	126	117
query83	195	147	140	140
query84	294	73	66	66
query85	887	312	309	309
query86	360	297	293	293
query87	4674	4316	4342	4316
query88	3952	2240	2228	2228
query89	433	297	291	291
query90	2120	190	195	190
query91	137	104	105	104
query92	68	52	51	51
query93	2871	548	554	548
query94	882	299	246	246
query95	346	254	261	254
query96	640	283	283	283
query97	2872	2655	2662	2655
query98	231	198	196	196
query99	1589	1327	1317	1317
Total cold run time: 304430 ms
Total hot run time: 196292 ms

doris-robot · 2024-12-24T05:40:27Z

ClickBench: Total hot run time: 33.01 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b9c1e624feab236ef25265b35de0d2e967375082, data reload: false

query1	0.04	0.03	0.02
query2	0.09	0.05	0.04
query3	0.22	0.06	0.05
query4	1.65	0.08	0.08
query5	0.43	0.42	0.42
query6	1.15	0.65	0.67
query7	0.02	0.02	0.02
query8	0.04	0.05	0.05
query9	0.54	0.48	0.50
query10	0.56	0.58	0.56
query11	0.16	0.12	0.12
query12	0.16	0.12	0.12
query13	0.60	0.59	0.59
query14	2.84	2.80	2.85
query15	0.91	0.85	0.84
query16	0.38	0.37	0.38
query17	1.04	1.00	1.06
query18	0.19	0.18	0.19
query19	1.94	1.90	2.05
query20	0.01	0.01	0.01
query21	15.35	0.67	0.66
query22	4.31	8.15	1.58
query23	18.25	1.34	1.38
query24	2.20	0.22	0.23
query25	0.15	0.08	0.09
query26	0.28	0.19	0.19
query27	0.08	0.08	0.08
query28	13.25	1.17	1.14
query29	12.62	3.32	3.32
query30	0.25	0.09	0.06
query31	2.83	0.40	0.39
query32	3.23	0.49	0.47
query33	3.05	3.18	3.09
query34	16.81	4.48	4.48
query35	4.51	4.51	4.51
query36	0.66	0.47	0.47
query37	0.20	0.17	0.16
query38	0.16	0.15	0.15
query39	0.06	0.04	0.05
query40	0.17	0.13	0.13
query41	0.10	0.05	0.06
query42	0.07	0.06	0.05
query43	0.06	0.05	0.05
Total cold run time: 111.62 s
Total hot run time: 33.01 s

bobhan1 · 2024-12-24T15:11:55Z

run buildall

doris-robot · 2024-12-24T16:19:23Z

TPC-H: Total hot run time: 33372 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 621196e587787f1d7f740555ec42d412845b6542, data reload: false

------ Round 1 ----------------------------------
q1	17641	6590	6198	6198
q2	2057	313	171	171
q3	10414	1484	737	737
q4	10202	916	451	451
q5	7517	2294	2055	2055
q6	222	190	150	150
q7	933	772	607	607
q8	9252	1552	1314	1314
q9	5465	5091	5091	5091
q10	6795	2327	1879	1879
q11	523	283	259	259
q12	364	373	223	223
q13	17778	3707	3039	3039
q14	235	233	219	219
q15	566	524	510	510
q16	635	632	585	585
q17	584	928	355	355
q18	7090	6563	6355	6355
q19	1225	1063	583	583
q20	324	331	200	200
q21	2985	2323	2082	2082
q22	370	333	309	309
Total cold run time: 103177 ms
Total hot run time: 33372 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6436	6425	6415	6415
q2	247	331	229	229
q3	2288	2689	2262	2262
q4	1439	1877	1362	1362
q5	4395	4874	5008	4874
q6	211	181	142	142
q7	2134	1978	1831	1831
q8	2741	2912	2836	2836
q9	7491	7437	7139	7139
q10	3112	3375	2883	2883
q11	612	522	498	498
q12	679	765	608	608
q13	3480	3747	3120	3120
q14	303	299	295	295
q15	564	516	498	498
q16	694	700	649	649
q17	1313	1765	1262	1262
q18	7770	7695	7505	7505
q19	898	1284	1250	1250
q20	2036	2067	1890	1890
q21	5625	5260	5072	5072
q22	651	620	590	590
Total cold run time: 55119 ms
Total hot run time: 53210 ms

doris-robot · 2024-12-24T16:31:40Z

TPC-DS: Total hot run time: 197755 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 621196e587787f1d7f740555ec42d412845b6542, data reload: false

query1	1298	1007	932	932
query2	6246	2400	2393	2393
query3	10976	4851	4806	4806
query4	33209	23529	23552	23529
query5	4280	636	483	483
query6	280	214	191	191
query7	3997	507	310	310
query8	338	239	222	222
query9	9434	2749	2738	2738
query10	457	315	246	246
query11	17952	15330	15232	15232
query12	157	108	109	108
query13	1566	545	400	400
query14	10073	7744	7653	7653
query15	245	205	204	204
query16	7397	677	461	461
query17	1588	762	573	573
query18	1279	392	319	319
query19	211	190	161	161
query20	118	120	113	113
query21	204	123	106	106
query22	4711	4558	4414	4414
query23	34817	33866	33959	33866
query24	6846	2275	2334	2275
query25	458	465	411	411
query26	778	320	159	159
query27	2014	489	338	338
query28	5364	2556	2512	2512
query29	560	556	424	424
query30	211	191	153	153
query31	1001	944	856	856
query32	71	63	58	58
query33	483	351	305	305
query34	770	854	551	551
query35	822	854	780	780
query36	996	1061	956	956
query37	116	101	75	75
query38	4109	4279	4151	4151
query39	1538	1471	1471	1471
query40	231	129	113	113
query41	50	49	54	49
query42	115	108	109	108
query43	528	533	500	500
query44	1377	843	844	843
query45	211	184	168	168
query46	914	1079	688	688
query47	1985	1947	1919	1919
query48	408	417	334	334
query49	716	483	410	410
query50	649	693	393	393
query51	7328	7290	7299	7290
query52	100	101	97	97
query53	239	263	190	190
query54	481	483	403	403
query55	79	75	77	75
query56	256	264	271	264
query57	1198	1183	1111	1111
query58	243	223	227	223
query59	3017	3153	2922	2922
query60	274	268	249	249
query61	109	107	107	107
query62	872	796	737	737
query63	224	204	189	189
query64	3258	1001	661	661
query65	3326	3200	3202	3200
query66	832	407	296	296
query67	16170	16013	15502	15502
query68	9503	769	525	525
query69	479	295	264	264
query70	1211	1156	1123	1123
query71	452	289	256	256
query72	5898	3933	3842	3842
query73	660	740	374	374
query74	10077	9036	9206	9036
query75	4626	3178	2679	2679
query76	4946	1189	767	767
query77	849	381	286	286
query78	10186	10245	9481	9481
query79	2347	858	605	605
query80	700	516	460	460
query81	492	262	236	236
query82	314	151	125	125
query83	197	165	160	160
query84	282	98	75	75
query85	805	356	304	304
query86	359	314	298	298
query87	4420	4667	4311	4311
query88	3141	2280	2248	2248
query89	417	333	296	296
query90	1922	193	185	185
query91	129	203	110	110
query92	69	58	65	58
query93	954	898	536	536
query94	642	375	298	298
query95	341	267	250	250
query96	489	617	281	281
query97	2767	2815	2672	2672
query98	219	202	201	201
query99	1673	1532	1437	1437
Total cold run time: 293726 ms
Total hot run time: 197755 ms

doris-robot · 2024-12-24T16:37:04Z

ClickBench: Total hot run time: 31.41 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 621196e587787f1d7f740555ec42d412845b6542, data reload: false

query1	0.03	0.03	0.03
query2	0.09	0.04	0.05
query3	0.23	0.05	0.06
query4	1.64	0.08	0.09
query5	0.41	0.38	0.41
query6	1.17	0.66	0.65
query7	0.02	0.02	0.02
query8	0.05	0.05	0.05
query9	0.56	0.50	0.51
query10	0.55	0.57	0.57
query11	0.17	0.12	0.14
query12	0.16	0.12	0.13
query13	0.61	0.59	0.60
query14	2.86	2.73	2.75
query15	0.92	0.83	0.83
query16	0.38	0.38	0.37
query17	1.08	1.05	1.05
query18	0.23	0.21	0.21
query19	1.89	1.75	2.00
query20	0.01	0.01	0.01
query21	15.35	0.96	0.66
query22	0.77	0.78	0.70
query23	14.96	1.48	0.69
query24	2.23	0.36	0.23
query25	0.15	0.09	0.08
query26	0.29	0.19	0.18
query27	0.07	0.08	0.08
query28	13.36	1.81	1.13
query29	12.63	4.13	3.41
query30	0.25	0.08	0.06
query31	2.85	0.61	0.40
query32	3.22	0.57	0.48
query33	3.12	3.14	3.09
query34	16.78	5.14	4.50
query35	4.53	4.48	4.55
query36	0.62	0.48	0.50
query37	0.19	0.17	0.16
query38	0.16	0.16	0.15
query39	0.06	0.05	0.04
query40	0.17	0.14	0.13
query41	0.10	0.06	0.05
query42	0.07	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 105.04 s
Total hot run time: 31.41 s

doris-robot · 2024-12-24T16:49:07Z

TeamCity be ut coverage result:
Function Coverage: 38.81% (10100/26021)
Line Coverage: 29.80% (85211/285957)
Region Coverage: 28.92% (43508/150446)
Branch Coverage: 25.47% (22195/87152)
Coverage Report: http://coverage.selectdb-in.cc/coverage/621196e587787f1d7f740555ec42d412845b6542_621196e587787f1d7f740555ec42d412845b6542/report/index.html

bobhan1 · 2024-12-26T06:21:04Z

run buildall

zhannngchen

LGTM

github-actions · 2024-12-26T07:23:17Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-26T07:23:19Z

PR approved by anyone and no changes requested.

…append

bobhan1 · 2024-12-26T08:14:24Z

run buildall

bobhan1 · 2024-12-26T13:20:29Z

run buildall

doris-robot · 2024-12-26T14:31:09Z

TeamCity be ut coverage result:
Function Coverage: 38.91% (10126/26026)
Line Coverage: 29.91% (85570/286117)
Region Coverage: 29.05% (43727/150540)
Branch Coverage: 25.59% (22313/87198)
Coverage Report: http://coverage.selectdb-in.cc/coverage/90f50958ad30556bf527cb86647b33d157d1bf9a_90f50958ad30556bf527cb86647b33d157d1bf9a/report/index.html

doris-robot · 2024-12-26T15:04:43Z

TPC-H: Total hot run time: 32650 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 90f50958ad30556bf527cb86647b33d157d1bf9a, data reload: false

------ Round 1 ----------------------------------
q1	17586	6196	6054	6054
q2	2062	305	172	172
q3	10721	1233	721	721
q4	10197	873	426	426
q5	7486	2193	1970	1970
q6	209	177	150	150
q7	895	797	607	607
q8	9228	1421	1217	1217
q9	5230	4926	5005	4926
q10	6724	2304	1841	1841
q11	476	273	266	266
q12	358	353	219	219
q13	17760	3645	3047	3047
q14	234	233	217	217
q15	562	492	502	492
q16	640	638	602	602
q17	587	854	322	322
q18	7184	6431	6520	6431
q19	3058	967	538	538
q20	302	313	182	182
q21	2809	2162	1943	1943
q22	353	335	307	307
Total cold run time: 104661 ms
Total hot run time: 32650 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6333	6251	6246	6246
q2	236	327	233	233
q3	2248	2681	2321	2321
q4	1399	1778	1348	1348
q5	4373	4788	4874	4788
q6	186	178	140	140
q7	2101	1956	1818	1818
q8	2637	2807	2693	2693
q9	7348	7305	7406	7305
q10	3069	3379	2724	2724
q11	575	517	493	493
q12	672	758	605	605
q13	3425	3776	3134	3134
q14	280	296	282	282
q15	583	513	499	499
q16	664	696	668	668
q17	1225	1754	1259	1259
q18	7677	7528	7416	7416
q19	869	1149	1099	1099
q20	1995	2032	1916	1916
q21	5635	5314	4865	4865
q22	639	643	586	586
Total cold run time: 54169 ms
Total hot run time: 52438 ms

doris-robot · 2024-12-26T15:17:07Z

TPC-DS: Total hot run time: 196711 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 90f50958ad30556bf527cb86647b33d157d1bf9a, data reload: false

query1	1296	983	926	926
query2	6513	2339	2376	2339
query3	10955	4769	4745	4745
query4	33025	23804	23453	23453
query5	4177	606	455	455
query6	269	186	179	179
query7	4002	499	303	303
query8	314	254	239	239
query9	9284	2723	2720	2720
query10	470	305	244	244
query11	17971	15665	15214	15214
query12	170	108	106	106
query13	1590	549	427	427
query14	9878	7364	7219	7219
query15	247	208	185	185
query16	8175	650	460	460
query17	1553	782	596	596
query18	2094	431	326	326
query19	216	194	164	164
query20	123	125	123	123
query21	269	133	107	107
query22	4534	4739	4387	4387
query23	34488	33707	33338	33338
query24	6611	2403	2320	2320
query25	500	448	412	412
query26	775	286	155	155
query27	2076	470	337	337
query28	5285	2511	2460	2460
query29	639	567	447	447
query30	204	182	150	150
query31	951	885	845	845
query32	82	55	58	55
query33	477	352	309	309
query34	774	866	524	524
query35	831	854	780	780
query36	1022	1044	954	954
query37	111	92	78	78
query38	4239	4208	4157	4157
query39	1555	1633	1487	1487
query40	225	116	102	102
query41	48	48	43	43
query42	120	106	104	104
query43	537	504	515	504
query44	1359	847	829	829
query45	193	176	173	173
query46	906	1062	660	660
query47	2027	1995	1905	1905
query48	393	413	323	323
query49	728	466	376	376
query50	650	673	411	411
query51	7285	7170	7268	7170
query52	102	99	95	95
query53	232	251	185	185
query54	494	510	416	416
query55	84	82	76	76
query56	252	265	243	243
query57	1292	1244	1164	1164
query58	231	215	235	215
query59	3124	3142	3073	3073
query60	266	276	248	248
query61	110	108	104	104
query62	867	813	758	758
query63	220	220	193	193
query64	3518	1005	642	642
query65	3320	3273	3237	3237
query66	781	417	317	317
query67	16691	15816	15562	15562
query68	9787	758	515	515
query69	509	286	251	251
query70	1209	1123	1125	1123
query71	439	300	251	251
query72	6285	4093	4006	4006
query73	793	755	359	359
query74	10116	9170	8789	8789
query75	4672	3148	2651	2651
query76	5577	1183	800	800
query77	1011	361	290	290
query78	10094	10212	9784	9784
query79	4695	871	600	600
query80	674	515	433	433
query81	498	264	233	233
query82	210	142	118	118
query83	191	161	142	142
query84	282	92	74	74
query85	742	346	297	297
query86	345	291	307	291
query87	4427	4432	4323	4323
query88	4196	2222	2207	2207
query89	412	340	322	322
query90	2210	187	188	187
query91	137	137	103	103
query92	67	54	59	54
query93	2772	898	528	528
query94	683	396	290	290
query95	321	265	251	251
query96	481	613	283	283
query97	2737	2804	2700	2700
query98	233	197	209	197
query99	1692	1559	1406	1406
Total cold run time: 302297 ms
Total hot run time: 196711 ms

doris-robot · 2024-12-26T15:22:25Z

ClickBench: Total hot run time: 31.46 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 90f50958ad30556bf527cb86647b33d157d1bf9a, data reload: false

query1	0.04	0.03	0.02
query2	0.10	0.05	0.05
query3	0.23	0.06	0.06
query4	1.64	0.08	0.09
query5	0.42	0.41	0.40
query6	1.17	0.66	0.66
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.55	0.49	0.50
query10	0.58	0.57	0.56
query11	0.17	0.12	0.12
query12	0.16	0.13	0.13
query13	0.62	0.59	0.59
query14	2.84	2.74	2.85
query15	0.91	0.84	0.83
query16	0.37	0.39	0.37
query17	0.98	1.07	0.98
query18	0.18	0.18	0.19
query19	1.88	1.80	2.03
query20	0.02	0.01	0.01
query21	15.38	0.96	0.66
query22	0.77	0.80	0.71
query23	14.98	1.48	0.68
query24	2.10	0.38	0.22
query25	0.15	0.09	0.09
query26	0.28	0.19	0.18
query27	0.08	0.07	0.09
query28	13.44	1.71	1.13
query29	12.69	4.11	3.40
query30	0.24	0.08	0.06
query31	2.84	0.58	0.40
query32	3.23	0.59	0.48
query33	3.24	3.14	3.08
query34	16.43	5.32	4.51
query35	4.53	4.55	4.54
query36	0.62	0.51	0.49
query37	0.19	0.16	0.15
query38	0.16	0.16	0.15
query39	0.06	0.05	0.04
query40	0.17	0.13	0.13
query41	0.10	0.05	0.05
query42	0.07	0.05	0.05
query43	0.05	0.05	0.05
Total cold run time: 104.74 s
Total hot run time: 31.46 s

bobhan1 force-pushed the truncate-segmnet-key-bounds branch from f848d0f to 6754c7f Compare December 11, 2024 09:35

bobhan1 force-pushed the truncate-segmnet-key-bounds branch from 6754c7f to c6e01f4 Compare December 11, 2024 10:36

bobhan1 force-pushed the truncate-segmnet-key-bounds branch from c6e01f4 to a870868 Compare December 11, 2024 12:08

github-actions bot reviewed Dec 11, 2024

View reviewed changes

bobhan1 force-pushed the truncate-segmnet-key-bounds branch from b29e183 to 4de3fe4 Compare December 12, 2024 03:42

zhannngchen reviewed Dec 12, 2024

View reviewed changes

bobhan1 force-pushed the truncate-segmnet-key-bounds branch 7 times, most recently from 4b70373 to c2c2892 Compare December 17, 2024 07:59

github-actions bot reviewed Dec 17, 2024

View reviewed changes

be/test/olap/segments_key_bounds_truncation_test.cpp Show resolved Hide resolved

github-actions bot reviewed Dec 18, 2024

View reviewed changes

be/test/olap/segments_key_bounds_truncation_test.cpp Show resolved Hide resolved

bobhan1 force-pushed the truncate-segmnet-key-bounds branch from 286cb7a to 3c33012 Compare December 18, 2024 04:05

github-actions bot reviewed Dec 18, 2024

View reviewed changes

be/test/olap/segments_key_bounds_truncation_test.cpp Show resolved Hide resolved

github-actions bot reviewed Dec 18, 2024

View reviewed changes

be/test/olap/segments_key_bounds_truncation_test.cpp Show resolved Hide resolved

be/test/olap/segments_key_bounds_truncation_test.cpp Show resolved Hide resolved

bobhan1 marked this pull request as ready for review December 18, 2024 06:39

gavinchou reviewed Dec 18, 2024

View reviewed changes

gensrc/proto/olap_file.proto Outdated Show resolved Hide resolved

bobhan1 force-pushed the truncate-segmnet-key-bounds branch from 59f196e to 59d64f4 Compare December 18, 2024 08:00

bobhan1 force-pushed the truncate-segmnet-key-bounds branch from b9c1e62 to 621196e Compare December 24, 2024 15:11

bobhan1 force-pushed the truncate-segmnet-key-bounds branch from 621196e to b8e8935 Compare December 26, 2024 04:14

zhannngchen previously approved these changes Dec 26, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 26, 2024

github-actions bot added the reviewed label Dec 26, 2024

bobhan1 added 5 commits December 26, 2024 16:05

upadte

235ab38

remove reduntant config enable_segments_key_bounds_truncation

6b1182b

only set segments_key_bounds_truncated to true when actual truncate h…

1d23abb

…append

fix

904e46d

adjust

2d33fa3

bobhan1 dismissed zhannngchen’s stale review via 2d33fa3 December 26, 2024 08:13

bobhan1 force-pushed the truncate-segmnet-key-bounds branch from a0503e7 to 2d33fa3 Compare December 26, 2024 08:13

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 26, 2024

fix

90f5095

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[opt](rowset meta) truncate segments key bounds if too large to avoid `RowsetMetaCloudPB` exceeds fdb's 100KB limits #45287

[opt](rowset meta) truncate segments key bounds if too large to avoid `RowsetMetaCloudPB` exceeds fdb's 100KB limits #45287

bobhan1 commented Dec 11, 2024 •

edited

Loading

doris-robot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

github-actions bot left a comment

github-actions bot Dec 11, 2024

zhannngchen Dec 12, 2024

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

bobhan1 commented Dec 18, 2024

doris-robot commented Dec 18, 2024

doris-robot commented Dec 18, 2024

doris-robot commented Dec 18, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

bobhan1 commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

bobhan1 commented Dec 26, 2024

zhannngchen left a comment

github-actions bot commented Dec 26, 2024

github-actions bot commented Dec 26, 2024

bobhan1 commented Dec 26, 2024

bobhan1 commented Dec 26, 2024

doris-robot commented Dec 26, 2024

doris-robot commented Dec 26, 2024

doris-robot commented Dec 26, 2024

doris-robot commented Dec 26, 2024

[opt](rowset meta) truncate segments key bounds if too large to avoid RowsetMetaCloudPB exceeds fdb's 100KB limits #45287

Are you sure you want to change the base?

[opt](rowset meta) truncate segments key bounds if too large to avoid RowsetMetaCloudPB exceeds fdb's 100KB limits #45287

Conversation

bobhan1 commented Dec 11, 2024 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Dec 11, 2024

Choose a reason for hiding this comment

zhannngchen Dec 12, 2024

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

bobhan1 commented Dec 18, 2024

doris-robot commented Dec 18, 2024

doris-robot commented Dec 18, 2024

doris-robot commented Dec 18, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

bobhan1 commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

bobhan1 commented Dec 26, 2024

zhannngchen left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 26, 2024

github-actions bot commented Dec 26, 2024

bobhan1 commented Dec 26, 2024

bobhan1 commented Dec 26, 2024

doris-robot commented Dec 26, 2024

doris-robot commented Dec 26, 2024

doris-robot commented Dec 26, 2024

doris-robot commented Dec 26, 2024

[opt](rowset meta) truncate segments key bounds if too large to avoid `RowsetMetaCloudPB` exceeds fdb's 100KB limits #45287

[opt](rowset meta) truncate segments key bounds if too large to avoid `RowsetMetaCloudPB` exceeds fdb's 100KB limits #45287

bobhan1 commented Dec 11, 2024 •

edited

Loading