Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat](iceberg)Supports using rest type catalog to read tables in unity catalog for 3.0 (#43525) #45161

Merged
merged 1 commit into from
Dec 9, 2024

Conversation

wuwenchi
Copy link
Contributor

@wuwenchi wuwenchi commented Dec 8, 2024

pb: #43525

…nity catalog (apache#43525)

### What problem does this PR solve?

1. We now support using the `rest` type catalog to read tables in the
unity catalog (https://github.com/unitycatalog/unitycatalog).
2. When reading the parquet file on the be side, we find the
corresponding column name based on the column id, which naturally
supports the column rename function.

example:
```
CREATE CATALOG `uc3`
PROPERTIES (
"type"  =  "iceberg",
"iceberg.catalog.type"  =  "rest",
"uri"  =  "http://127.0.0.1:8080/api/2.1/unity-catalog/iceberg",
"external_catalog.name" = "unity"  --- catalog name in unity catalog
);
```
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Dec 8, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40794 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b9ae487af3772bc49cd1866b68015fb82d61a8e9, data reload: false

------ Round 1 ----------------------------------
q1	17621	7443	7346	7346
q2	2076	169	173	169
q3	10693	1089	1167	1089
q4	10548	831	771	771
q5	7744	2856	2859	2856
q6	234	142	141	141
q7	974	608	599	599
q8	9353	1996	1972	1972
q9	6554	6479	6407	6407
q10	6996	2293	2277	2277
q11	467	284	274	274
q12	413	221	213	213
q13	17768	2996	3012	2996
q14	235	231	223	223
q15	573	530	523	523
q16	689	599	613	599
q17	975	589	562	562
q18	7198	6679	6575	6575
q19	1869	1071	1041	1041
q20	480	204	194	194
q21	3941	3112	3000	3000
q22	1078	995	967	967
Total cold run time: 108479 ms
Total hot run time: 40794 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7351	7254	7221	7221
q2	328	231	227	227
q3	2845	2900	2882	2882
q4	1990	1795	1765	1765
q5	5676	5703	5721	5703
q6	220	135	141	135
q7	2225	1757	1767	1757
q8	3332	3512	3510	3510
q9	8831	8867	8852	8852
q10	3546	3545	3529	3529
q11	608	493	494	493
q12	775	591	608	591
q13	16521	3166	3156	3156
q14	306	260	287	260
q15	583	514	541	514
q16	699	668	671	668
q17	1857	1632	1604	1604
q18	8109	7920	7555	7555
q19	3250	1703	1660	1660
q20	2081	1838	1876	1838
q21	5299	5381	5360	5360
q22	1067	1016	1019	1016
Total cold run time: 77499 ms
Total hot run time: 60296 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194732 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b9ae487af3772bc49cd1866b68015fb82d61a8e9, data reload: false

query1	1248	940	941	940
query2	6240	2087	2063	2063
query3	10823	3969	4045	3969
query4	66234	29501	23483	23483
query5	5339	444	430	430
query6	434	167	165	165
query7	5683	311	307	307
query8	292	219	228	219
query9	9047	2717	2714	2714
query10	517	268	247	247
query11	18008	15162	15750	15162
query12	162	102	106	102
query13	1567	449	426	426
query14	11043	7017	6814	6814
query15	217	183	181	181
query16	7351	502	462	462
query17	1137	580	579	579
query18	1890	337	342	337
query19	236	170	160	160
query20	119	119	118	118
query21	215	112	111	111
query22	4611	4593	4575	4575
query23	34447	34330	34149	34149
query24	6092	2857	2940	2857
query25	547	392	392	392
query26	689	170	164	164
query27	2130	308	312	308
query28	4187	2550	2525	2525
query29	673	465	432	432
query30	238	167	167	167
query31	993	817	837	817
query32	65	53	52	52
query33	471	282	271	271
query34	905	502	506	502
query35	851	733	716	716
query36	1078	955	963	955
query37	128	71	69	69
query38	4052	3974	4058	3974
query39	1533	1459	1468	1459
query40	216	99	100	99
query41	48	47	46	46
query42	115	98	95	95
query43	524	507	498	498
query44	1156	821	813	813
query45	186	168	175	168
query46	1155	748	745	745
query47	2033	1933	1912	1912
query48	472	378	394	378
query49	779	442	395	395
query50	856	417	408	408
query51	7362	6996	7197	6996
query52	97	89	88	88
query53	250	182	186	182
query54	570	461	456	456
query55	80	76	82	76
query56	267	243	238	238
query57	1164	1114	1134	1114
query58	218	206	202	202
query59	3140	2943	3052	2943
query60	284	260	266	260
query61	134	135	126	126
query62	767	646	663	646
query63	207	187	188	187
query64	1864	750	682	682
query65	3241	3201	3158	3158
query66	730	301	301	301
query67	16419	16022	15460	15460
query68	4688	552	572	552
query69	428	252	260	252
query70	1103	1079	1107	1079
query71	384	242	250	242
query72	6441	3953	2449	2449
query73	761	344	345	344
query74	10112	8873	8989	8873
query75	3306	2653	2607	2607
query76	2119	982	1092	982
query77	501	259	265	259
query78	10811	9707	9633	9633
query79	10357	612	611	611
query80	2139	419	414	414
query81	554	245	242	242
query82	1252	119	113	113
query83	304	141	166	141
query84	283	88	77	77
query85	1273	320	285	285
query86	451	273	279	273
query87	4472	4254	4335	4254
query88	5929	2399	2399	2399
query89	518	291	285	285
query90	2190	186	181	181
query91	195	149	154	149
query92	61	48	48	48
query93	6926	546	556	546
query94	963	277	286	277
query95	341	245	253	245
query96	643	278	272	272
query97	3301	3185	3125	3125
query98	216	197	223	197
query99	1581	1281	1281	1281
Total cold run time: 340429 ms
Total hot run time: 194732 ms

@morningman morningman merged commit 860f7d0 into apache:branch-3.0 Dec 9, 2024
18 of 21 checks passed
@wuwenchi wuwenchi deleted the fix-unity-catalog-3.0 branch December 9, 2024 02:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants