Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [enchement](iceberg)support read iceberg partition evolution table. #45367 #45563

Merged
merged 1 commit into from
Dec 20, 2024

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #45367

…45367)

### What problem does this PR solve?
Problem Summary:
Supports reading iceberg tables after `ADD DROP REPLACE` partition.
Before this, reading such tables will cause be core.

https://iceberg.apache.org/docs/1.7.0/spark-ddl/#alter-table-add-partition-field

### Release note
[fix](iceberg) Supports reading Iceberg tables after partition evolution.
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Dec 17, 2024
@hello-stephen
Copy link
Contributor

run buildall

Copy link
Contributor Author

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40129 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4faccfbe01756f66e917ba20091c0fb1c2a51aaa, data reload: false

------ Round 1 ----------------------------------
q1	17614	7306	7219	7219
q2	2052	185	164	164
q3	10702	1048	1140	1048
q4	10548	759	718	718
q5	7716	2795	2714	2714
q6	233	151	149	149
q7	963	611	600	600
q8	9337	1882	1973	1882
q9	6571	6357	6358	6357
q10	6943	2254	2266	2254
q11	448	250	258	250
q12	415	215	217	215
q13	17774	2959	2987	2959
q14	242	209	208	208
q15	546	502	520	502
q16	649	606	595	595
q17	977	521	574	521
q18	7185	6514	6658	6514
q19	1378	1009	959	959
q20	464	196	188	188
q21	3874	3331	3152	3152
q22	1073	988	961	961
Total cold run time: 107704 ms
Total hot run time: 40129 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7290	7219	7241	7219
q2	317	231	232	231
q3	2836	2719	2840	2719
q4	2036	1757	1778	1757
q5	5635	5668	5675	5668
q6	229	146	147	146
q7	2183	1760	1759	1759
q8	3360	3524	3468	3468
q9	8653	8798	8799	8798
q10	3541	3465	3479	3465
q11	623	514	507	507
q12	807	598	581	581
q13	16207	3180	3122	3122
q14	310	280	267	267
q15	558	520	511	511
q16	698	671	666	666
q17	1808	1643	1628	1628
q18	8243	7772	7591	7591
q19	7473	1610	1458	1458
q20	2065	1868	1854	1854
q21	5343	5108	5201	5108
q22	1090	1022	1013	1013
Total cold run time: 81305 ms
Total hot run time: 59536 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195391 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4faccfbe01756f66e917ba20091c0fb1c2a51aaa, data reload: false

query1	1317	905	911	905
query2	6220	2119	2113	2113
query3	10815	4084	4064	4064
query4	66059	29823	23624	23624
query5	5822	453	436	436
query6	488	171	168	168
query7	6320	305	321	305
query8	334	236	227	227
query9	9275	2700	2678	2678
query10	542	274	271	271
query11	18160	15184	15799	15184
query12	161	105	101	101
query13	1619	440	462	440
query14	11340	7084	6470	6470
query15	223	179	174	174
query16	7727	494	513	494
query17	1080	573	589	573
query18	1941	332	323	323
query19	219	158	161	158
query20	113	112	120	112
query21	69	49	47	47
query22	4588	4500	4599	4500
query23	34244	33822	33958	33822
query24	5751	2914	2909	2909
query25	568	435	423	423
query26	695	172	172	172
query27	1799	301	304	301
query28	4646	2558	2546	2546
query29	710	452	468	452
query30	243	163	169	163
query31	1050	842	824	824
query32	63	54	53	53
query33	477	271	298	271
query34	911	503	516	503
query35	814	762	738	738
query36	1062	988	959	959
query37	116	71	75	71
query38	4044	4076	3967	3967
query39	1519	1469	1460	1460
query40	139	80	79	79
query41	51	46	47	46
query42	109	102	94	94
query43	533	509	500	500
query44	1233	797	816	797
query45	186	179	169	169
query46	1120	738	741	738
query47	1966	1870	1915	1870
query48	468	389	381	381
query49	728	373	392	373
query50	835	412	414	412
query51	7328	7316	6916	6916
query52	98	91	85	85
query53	254	178	179	178
query54	556	442	436	436
query55	74	72	74	72
query56	247	236	234	234
query57	1224	1134	1085	1085
query58	208	195	201	195
query59	3307	3026	3261	3026
query60	262	265	252	252
query61	106	103	106	103
query62	762	657	691	657
query63	205	184	196	184
query64	1868	674	621	621
query65	3232	3135	3190	3135
query66	741	294	300	294
query67	15670	15262	15277	15262
query68	4767	555	568	555
query69	439	251	255	251
query70	1187	1053	1038	1038
query71	414	261	243	243
query72	6642	3941	3945	3941
query73	750	344	344	344
query74	10361	8982	8853	8853
query75	3351	2619	2632	2619
query76	2804	1102	1074	1074
query77	506	269	272	269
query78	10569	9675	9537	9537
query79	7832	583	586	583
query80	1742	420	409	409
query81	551	243	241	241
query82	1241	117	114	114
query83	260	144	145	144
query84	286	86	82	82
query85	1729	315	290	290
query86	460	305	290	290
query87	4454	4232	4216	4216
query88	5828	2478	2479	2478
query89	413	290	289	289
query90	1984	186	183	183
query91	179	143	143	143
query92	67	47	47	47
query93	6451	550	549	549
query94	831	269	285	269
query95	346	249	240	240
query96	637	286	281	281
query97	3357	3141	3144	3141
query98	214	196	203	196
query99	1608	1287	1299	1287
Total cold run time: 338914 ms
Total hot run time: 195391 ms

@hubgeter
Copy link
Contributor

run p0

@morningman morningman merged commit 0671640 into branch-3.0 Dec 20, 2024
21 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-45367-branch-3.0 branch December 20, 2024 03:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants