[fix](bloom filter)Fix drop column with bloom filter index #44361

qidaye · 2024-11-20T13:58:04Z

What problem does this PR solve?

When drop column with bloom filter, we modify the bloom filter column info
When replay editLog, we rebuild bloom filter info by table schema.

Related PR: #41369

Problem Summary:

Release note

Fix drop column with bloom filter

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

1. When drop column with bloom filter, we can not do light schema change. 2. Set light_schema_change to false, to log the operation in editLog.

doris-robot · 2024-11-20T13:58:09Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

qidaye · 2024-11-20T13:59:00Z

run buildall

fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java

doris-robot · 2024-11-20T14:18:02Z

TPC-H: Total hot run time: 39934 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b14cd87a133700eaeadc0a17dc78c69c1e1eb82e, data reload: false

------ Round 1 ----------------------------------
q1	17607	7448	7298	7298
q2	2035	184	177	177
q3	10530	1133	1141	1133
q4	10208	709	650	650
q5	7586	2756	2744	2744
q6	244	147	149	147
q7	984	629	612	612
q8	9245	1787	1944	1787
q9	6484	6420	6387	6387
q10	6957	2298	2305	2298
q11	460	260	264	260
q12	411	217	215	215
q13	17751	3067	3085	3067
q14	251	208	209	208
q15	577	532	533	532
q16	663	593	597	593
q17	974	644	585	585
q18	7535	6722	6718	6718
q19	1346	952	970	952
q20	454	189	179	179
q21	4554	3186	3076	3076
q22	387	325	316	316
Total cold run time: 107243 ms
Total hot run time: 39934 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7324	7316	7257	7257
q2	314	229	226	226
q3	2899	2739	2816	2739
q4	1968	1684	1688	1684
q5	5412	5404	5478	5404
q6	215	140	141	140
q7	2129	1785	1728	1728
q8	3266	3418	3420	3418
q9	8480	8501	8549	8501
q10	3511	3456	3457	3456
q11	593	507	520	507
q12	808	576	581	576
q13	10398	3065	3051	3051
q14	289	261	289	261
q15	566	515	510	510
q16	691	628	650	628
q17	1805	1615	1606	1606
q18	7853	7585	7388	7388
q19	1691	1597	1562	1562
q20	2108	1837	1833	1833
q21	5471	5400	5084	5084
q22	663	590	567	567
Total cold run time: 68454 ms
Total hot run time: 58126 ms

doris-robot · 2024-11-20T14:30:13Z

TPC-DS: Total hot run time: 191241 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b14cd87a133700eaeadc0a17dc78c69c1e1eb82e, data reload: false

query1	981	386	374	374
query2	6524	2241	2156	2156
query3	6727	215	216	215
query4	34278	23510	23787	23510
query5	4353	457	434	434
query6	277	175	173	173
query7	4623	297	294	294
query8	277	221	219	219
query9	9718	2678	2662	2662
query10	482	258	249	249
query11	18236	15233	15176	15176
query12	158	103	103	103
query13	1657	429	430	429
query14	10231	7161	7490	7161
query15	299	177	174	174
query16	8202	455	460	455
query17	1845	588	584	584
query18	2139	301	303	301
query19	364	158	154	154
query20	118	114	115	114
query21	212	103	102	102
query22	4516	4425	4104	4104
query23	35144	34102	34064	34064
query24	11621	2466	2544	2466
query25	672	400	384	384
query26	1794	149	151	149
query27	2869	281	270	270
query28	8026	2459	2466	2459
query29	1020	408	401	401
query30	302	148	150	148
query31	1008	796	844	796
query32	99	54	59	54
query33	772	286	289	286
query34	1011	501	522	501
query35	857	714	728	714
query36	1099	968	960	960
query37	274	73	84	73
query38	4425	4181	4286	4181
query39	1617	1456	1443	1443
query40	284	96	102	96
query41	46	43	44	43
query42	120	96	97	96
query43	562	501	508	501
query44	1268	801	813	801
query45	183	165	162	162
query46	1160	684	711	684
query47	1939	1801	1850	1801
query48	405	313	304	304
query49	1292	391	379	379
query50	815	381	391	381
query51	7343	7128	7018	7018
query52	98	91	88	88
query53	254	177	179	177
query54	1223	400	413	400
query55	78	75	85	75
query56	251	247	232	232
query57	1325	1166	1169	1166
query58	235	244	210	210
query59	3425	3238	3057	3057
query60	275	242	246	242
query61	140	104	109	104
query62	882	665	677	665
query63	211	195	186	186
query64	5145	644	672	644
query65	3329	3204	3242	3204
query66	1411	318	320	318
query67	16321	15830	15709	15709
query68	5006	545	576	545
query69	420	249	259	249
query70	1216	1079	1146	1079
query71	328	242	250	242
query72	6343	4030	4010	4010
query73	769	362	357	357
query74	10539	8990	9150	8990
query75	3483	2697	2677	2677
query76	2986	1157	1153	1153
query77	511	272	277	272
query78	10410	9464	9459	9459
query79	2247	615	606	606
query80	1118	423	443	423
query81	547	229	234	229
query82	908	116	117	116
query83	251	148	144	144
query84	237	73	72	72
query85	1382	293	288	288
query86	433	280	302	280
query87	4739	4646	4481	4481
query88	3779	2224	2179	2179
query89	400	291	295	291
query90	2164	188	191	188
query91	138	104	102	102
query92	62	50	53	50
query93	1602	543	555	543
query94	1124	284	296	284
query95	361	249	247	247
query96	621	280	283	280
query97	2891	2675	2653	2653
query98	220	211	215	211
query99	1547	1306	1318	1306
Total cold run time: 307629 ms
Total hot run time: 191241 ms

doris-robot · 2024-11-20T14:35:43Z

ClickBench: Total hot run time: 32.92 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b14cd87a133700eaeadc0a17dc78c69c1e1eb82e, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.08	0.07
query4	1.62	0.10	0.11
query5	0.44	0.40	0.40
query6	1.14	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.52	0.51
query10	0.56	0.56	0.56
query11	0.14	0.09	0.10
query12	0.14	0.12	0.11
query13	0.62	0.61	0.61
query14	2.81	2.73	2.73
query15	0.90	0.82	0.83
query16	0.39	0.37	0.39
query17	1.00	1.02	1.00
query18	0.24	0.21	0.22
query19	1.95	1.89	2.00
query20	0.02	0.02	0.01
query21	15.38	0.61	0.60
query22	2.76	2.28	1.80
query23	16.98	1.00	0.80
query24	3.24	1.21	2.13
query25	0.28	0.11	0.20
query26	0.50	0.15	0.14
query27	0.04	0.04	0.04
query28	9.62	1.12	1.08
query29	12.52	3.21	3.17
query30	0.26	0.06	0.06
query31	2.87	0.39	0.38
query32	3.27	0.46	0.46
query33	3.07	3.07	3.16
query34	17.04	4.45	4.48
query35	4.54	4.55	4.53
query36	0.64	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.14	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.4 s
Total hot run time: 32.92 s

qidaye · 2024-11-21T08:38:44Z

run buildall

doris-robot · 2024-11-21T09:04:16Z

TPC-H: Total hot run time: 40258 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f063ffe153f96436faa24cdbcdf3389146ebc60, data reload: false

------ Round 1 ----------------------------------
q1	17619	7564	7316	7316
q2	2049	184	189	184
q3	10527	1162	1220	1162
q4	10209	727	731	727
q5	7604	2854	2772	2772
q6	244	153	152	152
q7	1009	656	625	625
q8	9461	1917	1945	1917
q9	8272	6443	6454	6443
q10	7080	2330	2372	2330
q11	454	269	264	264
q12	544	222	220	220
q13	17787	3033	3002	3002
q14	255	209	207	207
q15	577	548	531	531
q16	695	602	595	595
q17	999	556	602	556
q18	7448	6694	6640	6640
q19	1359	1059	899	899
q20	476	186	188	186
q21	3967	3222	3217	3217
q22	396	313	324	313
Total cold run time: 109031 ms
Total hot run time: 40258 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7368	7269	7295	7269
q2	328	226	231	226
q3	2944	2804	2802	2802
q4	1994	1724	1761	1724
q5	5429	5444	5472	5444
q6	224	142	142	142
q7	2116	1693	1750	1693
q8	3308	3431	3424	3424
q9	8533	8557	8585	8557
q10	3518	3467	3467	3467
q11	603	496	509	496
q12	835	567	598	567
q13	11305	3013	3036	3013
q14	303	267	262	262
q15	563	514	505	505
q16	665	661	639	639
q17	1814	1593	1568	1568
q18	7710	7377	7592	7377
q19	1684	1499	1465	1465
q20	2046	1797	1839	1797
q21	5361	5272	5024	5024
q22	603	570	564	564
Total cold run time: 69254 ms
Total hot run time: 58025 ms

doris-robot · 2024-11-21T09:16:05Z

TPC-DS: Total hot run time: 191637 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f063ffe153f96436faa24cdbcdf3389146ebc60, data reload: false

query1	869	389	392	389
query2	4628	2085	2088	2085
query3	4213	222	207	207
query4	33406	23606	23470	23470
query5	3234	455	445	445
query6	246	177	175	175
query7	3840	297	308	297
query8	273	225	227	225
query9	8165	2670	2665	2665
query10	420	256	256	256
query11	18042	15198	15400	15198
query12	153	107	103	103
query13	1334	414	427	414
query14	9563	7406	7506	7406
query15	289	179	196	179
query16	7404	493	465	465
query17	1391	595	545	545
query18	1951	286	283	283
query19	319	145	146	145
query20	116	113	108	108
query21	215	106	101	101
query22	4607	4558	4455	4455
query23	34929	34020	34159	34020
query24	11403	2394	2490	2394
query25	593	387	379	379
query26	1616	143	149	143
query27	2601	269	288	269
query28	6911	2449	2431	2431
query29	917	412	405	405
query30	281	146	156	146
query31	1038	836	792	792
query32	74	60	56	56
query33	704	286	276	276
query34	942	516	534	516
query35	863	699	725	699
query36	1083	932	941	932
query37	254	80	72	72
query38	4451	4190	4254	4190
query39	1507	1429	1441	1429
query40	276	99	96	96
query41	48	44	44	44
query42	113	95	96	95
query43	535	491	481	481
query44	1218	835	814	814
query45	185	163	163	163
query46	1151	715	704	704
query47	1983	1906	1879	1879
query48	408	318	302	302
query49	1026	382	393	382
query50	835	374	382	374
query51	7268	7114	7117	7114
query52	96	87	92	87
query53	253	181	176	176
query54	740	408	404	404
query55	82	81	96	81
query56	263	248	244	244
query57	1329	1199	1182	1182
query58	236	219	212	212
query59	3250	3136	2926	2926
query60	293	238	250	238
query61	113	111	113	111
query62	840	675	678	675
query63	212	185	185	185
query64	4504	665	635	635
query65	3283	3205	3260	3205
query66	1153	342	309	309
query67	16176	15888	15686	15686
query68	4928	542	560	542
query69	417	253	257	253
query70	1150	1144	1149	1144
query71	349	252	250	250
query72	6221	4068	4230	4068
query73	765	356	372	356
query74	10338	9097	9007	9007
query75	3449	2658	2712	2658
query76	2707	1011	1097	1011
query77	464	316	278	278
query78	10586	9447	9529	9447
query79	1092	589	596	589
query80	721	427	447	427
query81	554	232	230	230
query82	654	118	121	118
query83	209	153	149	149
query84	251	69	68	68
query85	978	294	298	294
query86	325	306	296	296
query87	4666	4619	4495	4495
query88	3146	2258	2221	2221
query89	404	297	305	297
query90	2139	189	194	189
query91	139	105	104	104
query92	62	57	51	51
query93	1040	529	533	529
query94	943	284	258	258
query95	345	253	246	246
query96	618	280	280	280
query97	2859	2697	2697	2697
query98	213	188	192	188
query99	1556	1346	1328	1328
Total cold run time: 287002 ms
Total hot run time: 191637 ms

doris-robot · 2024-11-21T09:21:33Z

ClickBench: Total hot run time: 32.68 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9f063ffe153f96436faa24cdbcdf3389146ebc60, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.24	0.07	0.08
query4	1.59	0.10	0.11
query5	0.41	0.40	0.40
query6	1.14	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.51	0.48
query10	0.55	0.56	0.56
query11	0.14	0.11	0.10
query12	0.15	0.11	0.11
query13	0.61	0.61	0.60
query14	2.80	2.83	2.80
query15	0.89	0.83	0.82
query16	0.38	0.38	0.37
query17	1.04	0.95	0.99
query18	0.23	0.21	0.21
query19	1.98	1.77	1.83
query20	0.02	0.01	0.02
query21	15.40	0.63	0.59
query22	2.84	2.21	2.00
query23	16.90	1.10	0.84
query24	2.99	1.80	0.98
query25	0.25	0.19	0.14
query26	0.44	0.13	0.13
query27	0.04	0.04	0.05
query28	10.28	1.09	1.07
query29	12.57	3.28	3.31
query30	0.25	0.06	0.07
query31	2.88	0.38	0.38
query32	3.27	0.46	0.48
query33	3.01	2.96	3.01
query34	16.94	4.47	4.44
query35	4.53	4.46	4.42
query36	0.67	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.62 s
Total hot run time: 32.68 s

xiaokang

LGTM

github-actions · 2024-11-21T10:04:09Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-21T10:04:12Z

PR approved by anyone and no changes requested.

airborne12

LGTM

) Problem Summary: 1. When drop column with bloom filter, we modify the bloom filter column info 2. When replay editLog, we rebuild bloom filter info by table schema. Related PR: apache#41369 Fix drop column with bloom filter index

…44478) pick #44361

…44476) pick #44361

…44480) pick #44361

[fix](bloom filter)Fix drop column with bloom filter

b14cd87

1. When drop column with bloom filter, we can not do light schema change. 2. Set light_schema_change to false, to log the operation in editLog.

qidaye added dev/2.0.x dev/2.1.x dev/3.0.x labels Nov 20, 2024

airborne12 reviewed Nov 20, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java Outdated Show resolved Hide resolved

xiaokang changed the title ~~[fix](bloom filter)Fix drop column with bloom filter~~ [fix](bloom filter)Fix drop column with bloom filter index Nov 21, 2024

rebuild bf info when replay editlog

9f063ff

xiaokang approved these changes Nov 21, 2024

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Nov 21, 2024

airborne12 approved these changes Nov 21, 2024

View reviewed changes

qidaye merged commit 074bd7c into apache:master Nov 22, 2024
27 of 29 checks passed

qidaye deleted the fix_bf_drop_column branch November 22, 2024 02:14

github-actions bot added dev/3.0.x-conflict dev/2.1.x-conflict labels Nov 22, 2024

qidaye mentioned this pull request Nov 22, 2024

[fix](bloom filter)Fix drop column with bloom filter index (#44361) #44476

Merged

qidaye mentioned this pull request Nov 22, 2024

[fix](bloom filter)Fix drop column with bloom filter index (#44361) #44478

Merged

qidaye mentioned this pull request Nov 22, 2024

[fix](bloom filter)Fix drop column with bloom filter index (#44361) #44480

Merged

qidaye removed dev/3.0.x-conflict dev/2.1.x-conflict labels Nov 22, 2024

yiguolei pushed a commit that referenced this pull request Nov 22, 2024

[fix](bloom filter)Fix drop column with bloom filter index (#44361) (#…

964b40a

…44478) pick #44361

yiguolei added dev/2.1.8-merged and removed dev/2.1.x labels Nov 22, 2024

airborne12 pushed a commit that referenced this pull request Nov 22, 2024

[fix](bloom filter)Fix drop column with bloom filter index (#44361) (#…

7489aaf

…44476) pick #44361

airborne12 pushed a commit that referenced this pull request Nov 26, 2024

[fix](bloom filter)Fix drop column with bloom filter index (#44361) (#…

12fb61c

…44480) pick #44361

airborne12 added dev/2.0.16-merged dev/3.0.3-merged and removed dev/2.0.x dev/3.0.x labels Nov 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](bloom filter)Fix drop column with bloom filter index #44361

[fix](bloom filter)Fix drop column with bloom filter index #44361

qidaye commented Nov 20, 2024 •

edited

Loading

doris-robot commented Nov 20, 2024

qidaye commented Nov 20, 2024

doris-robot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

qidaye commented Nov 21, 2024

doris-robot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

xiaokang left a comment

github-actions bot commented Nov 21, 2024

github-actions bot commented Nov 21, 2024

airborne12 left a comment

[fix](bloom filter)Fix drop column with bloom filter index #44361

[fix](bloom filter)Fix drop column with bloom filter index #44361

Conversation

qidaye commented Nov 20, 2024 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented Nov 20, 2024

qidaye commented Nov 20, 2024

doris-robot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

doris-robot commented Nov 20, 2024

qidaye commented Nov 21, 2024

doris-robot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

doris-robot commented Nov 21, 2024

xiaokang left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 21, 2024

github-actions bot commented Nov 21, 2024

airborne12 left a comment

Choose a reason for hiding this comment

qidaye commented Nov 20, 2024 •

edited

Loading