Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](index compaction)Optimize logic of picking columns for index compaction (#42051) #42285

Merged
merged 1 commit into from
Oct 23, 2024
Merged

[opt](index compaction)Optimize logic of picking columns for index compaction (#42051) #42285

merged 1 commit into from
Oct 23, 2024

Conversation

qidaye
Copy link
Contributor

@qidaye qidaye commented Oct 22, 2024

Proposed changes

bp #42051

@qidaye
[opt](index compaction)Optimize logic of picking columns for index co…
9592458 
…mpaction (apache#42051)

1. Rename `skip_inverted_index` to `columns_to_do_index_compaction` to
make it more clear.
2. Remove redundant rowid_conversion check in index compaction
3. Rename function `construct_skip_inverted_index` to
`construct_index_compaction_columns`
4. Check `index.col_unique_ids` empty before using to avoid crash and
log tablet schema for debug.
5. Avoid unnecessary bkd and variant index checks by advancing column
type checking before `has_inverted_index`
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Oct 22, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40530 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 95924584b41873b614c8bc7d39390af4eca7ef09, data reload: false

------ Round 1 ----------------------------------
q1	17618	7322	7284	7284
q2	2033	271	262	262
q3	12349	1149	1102	1102
q4	10567	740	699	699
q5	7762	2838	2798	2798
q6	235	150	149	149
q7	979	597	596	596
q8	9580	1898	1888	1888
q9	8025	6413	6791	6413
q10	7009	2259	2307	2259
q11	435	242	239	239
q12	403	215	212	212
q13	17773	2990	2956	2956
q14	230	217	217	217
q15	558	512	519	512
q16	688	606	596	596
q17	973	537	542	537
q18	7181	6578	6601	6578
q19	1403	903	969	903
q20	585	257	272	257
q21	3909	3260	3090	3090
q22	1067	983	991	983
Total cold run time: 111362 ms
Total hot run time: 40530 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7259	7244	7238	7238
q2	327	229	234	229
q3	3038	2854	2896	2854
q4	2056	1810	1737	1737
q5	5698	5708	5680	5680
q6	236	140	144	140
q7	2207	1803	1735	1735
q8	3329	3491	3392	3392
q9	8919	8906	8783	8783
q10	3552	3511	3501	3501
q11	572	467	476	467
q12	812	636	634	634
q13	15946	3133	3135	3133
q14	305	272	268	268
q15	587	500	514	500
q16	699	674	663	663
q17	1838	1621	1586	1586
q18	8238	7650	7517	7517
q19	4859	1510	1514	1510
q20	2109	1854	1836	1836
q21	5402	5131	5316	5131
q22	1101	1071	1038	1038
Total cold run time: 79089 ms
Total hot run time: 59572 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189009 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 95924584b41873b614c8bc7d39390af4eca7ef09, data reload: false

query1	904	394	381	381
query2	6256	2123	1995	1995
query3	8701	193	193	193
query4	34018	23279	23293	23279
query5	3744	453	452	452
query6	282	172	164	164
query7	4216	313	301	301
query8	265	207	207	207
query9	9264	2681	2681	2681
query10	460	297	294	294
query11	17815	15145	15037	15037
query12	148	100	102	100
query13	1520	404	424	404
query14	8285	7235	7072	7072
query15	230	171	174	171
query16	7814	504	502	502
query17	1632	560	552	552
query18	1978	322	315	315
query19	247	155	168	155
query20	128	120	115	115
query21	216	106	113	106
query22	4686	4582	4374	4374
query23	35226	33945	33793	33793
query24	10886	2839	2734	2734
query25	605	403	396	396
query26	728	162	162	162
query27	2069	295	288	288
query28	6446	2505	2463	2463
query29	739	428	429	428
query30	283	154	154	154
query31	1067	768	788	768
query32	98	56	54	54
query33	719	297	294	294
query34	937	481	505	481
query35	843	754	729	729
query36	1113	939	939	939
query37	143	88	86	86
query38	4108	3885	3777	3777
query39	1487	1436	1459	1436
query40	208	100	99	99
query41	50	55	47	47
query42	129	96	93	93
query43	513	478	470	470
query44	1150	763	770	763
query45	195	167	167	167
query46	1145	676	698	676
query47	1900	1793	1784	1784
query48	454	376	365	365
query49	925	384	409	384
query50	810	413	398	398
query51	6989	6876	6771	6771
query52	98	93	91	91
query53	257	182	180	180
query54	1219	466	479	466
query55	77	74	72	72
query56	285	275	259	259
query57	1157	1095	1101	1095
query58	222	307	231	231
query59	3037	2843	2982	2843
query60	292	279	270	270
query61	105	102	125	102
query62	836	664	682	664
query63	204	181	178	178
query64	3802	668	604	604
query65	3229	3151	3165	3151
query66	762	303	303	303
query67	15757	15251	15278	15251
query68	4535	535	537	535
query69	622	283	300	283
query70	1144	1087	1132	1087
query71	429	268	271	268
query72	7996	3885	3794	3794
query73	751	333	336	333
query74	10564	8984	8864	8864
query75	4764	2600	2570	2570
query76	4414	903	901	901
query77	778	276	278	276
query78	10022	9138	9202	9138
query79	6574	596	599	596
query80	1553	426	443	426
query81	592	239	240	239
query82	1223	143	144	143
query83	392	134	127	127
query84	289	73	81	73
query85	1952	302	294	294
query86	450	289	286	286
query87	4405	4192	4170	4170
query88	5429	2408	2441	2408
query89	533	280	294	280
query90	2227	186	193	186
query91	169	141	139	139
query92	63	47	46	46
query93	6367	531	522	522
query94	1115	295	292	292
query95	347	241	248	241
query96	621	283	288	283
query97	3268	3163	3121	3121
query98	224	201	195	195
query99	1827	1306	1284	1284
Total cold run time: 311752 ms
Total hot run time: 189009 ms

airborne12
airborne12 approved these changes Oct 23, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12 airborne12 merged commit ae5535e into apache:branch-3.0 Oct 23, 2024
21 of 26 checks passed
@qidaye qidaye deleted the pick_42051_3.0 branch October 23, 2024 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@airborne12 airborne12 airborne12 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@qidaye @doris-robot @airborne12