[opt](hms table)Some optimizations for hms external table #44909

wuwenchi · 2024-12-03T07:38:01Z

What problem does this PR solve?

Problem Summary:

Increase the schema cache to reduce the time to obtain the schema.
HoodieTableMetaClient is stored in HMSExternalTable to prevent redundant creation.
Cache HoodieTableFileSystemView to speed up getting FileGroup or FileSlice.
refresh timeline before get hudi schema to avoid throwing not found file error.
Add FSDataInputStreamWrapper to solve hudi conflict class.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-12-03T07:38:07Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

wuwenchi · 2024-12-03T07:38:36Z

run buildall

doris-robot · 2024-12-03T09:02:42Z

TPC-H: Total hot run time: 40334 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f5120edd70db2f580c199ffa66abcfe9eed14a56, data reload: false

------ Round 1 ----------------------------------
q1	17600	7578	7388	7388
q2	2048	173	166	166
q3	10578	1066	1161	1066
q4	10505	764	726	726
q5	7610	2744	2733	2733
q6	238	146	148	146
q7	1018	631	599	599
q8	9218	1848	1948	1848
q9	6665	6556	6533	6533
q10	7001	2302	2281	2281
q11	467	265	283	265
q12	419	225	223	223
q13	17767	3030	3044	3030
q14	253	215	212	212
q15	585	540	527	527
q16	668	585	595	585
q17	979	599	714	599
q18	7287	6735	6693	6693
q19	1331	1052	1032	1032
q20	447	177	179	177
q21	4066	3196	3268	3196
q22	372	323	309	309
Total cold run time: 107122 ms
Total hot run time: 40334 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7290	7273	7269	7269
q2	332	228	237	228
q3	2936	2790	2968	2790
q4	2108	1832	1854	1832
q5	5751	5694	5669	5669
q6	229	139	138	138
q7	2259	1800	1855	1800
q8	3450	3539	3487	3487
q9	8980	8998	9100	8998
q10	3636	3592	3570	3570
q11	599	517	496	496
q12	822	599	589	589
q13	10554	3258	3201	3201
q14	297	275	270	270
q15	579	532	530	530
q16	681	641	634	634
q17	1850	1641	1597	1597
q18	8308	7843	7569	7569
q19	1672	1620	1579	1579
q20	2123	1874	1911	1874
q21	5681	5454	5540	5454
q22	649	582	580	580
Total cold run time: 70786 ms
Total hot run time: 60154 ms

doris-robot · 2024-12-03T09:15:23Z

TPC-DS: Total hot run time: 198338 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f5120edd70db2f580c199ffa66abcfe9eed14a56, data reload: false

query1	1263	965	924	924
query2	6230	2083	2064	2064
query3	10999	4448	4609	4448
query4	67375	28402	23596	23596
query5	4934	462	453	453
query6	406	191	180	180
query7	5530	307	296	296
query8	322	239	249	239
query9	8527	2707	2706	2706
query10	442	264	265	264
query11	17248	15382	15890	15382
query12	157	111	108	108
query13	1460	443	446	443
query14	10130	7920	7626	7626
query15	219	188	191	188
query16	7163	504	503	503
query17	1092	641	550	550
query18	1816	303	303	303
query19	211	152	144	144
query20	114	111	109	109
query21	236	105	104	104
query22	4803	4547	4692	4547
query23	35598	34805	34751	34751
query24	5547	2582	2525	2525
query25	518	383	395	383
query26	672	155	149	149
query27	2046	281	283	281
query28	4554	2481	2465	2465
query29	688	428	422	422
query30	225	152	148	148
query31	1026	831	817	817
query32	68	62	62	62
query33	474	284	310	284
query34	945	538	541	538
query35	906	794	753	753
query36	1089	961	951	951
query37	118	80	71	71
query38	4664	4443	4482	4443
query39	1543	1490	1460	1460
query40	208	103	107	103
query41	45	47	43	43
query42	114	107	100	100
query43	554	501	506	501
query44	1201	827	832	827
query45	201	178	166	166
query46	1182	727	732	727
query47	2057	1963	1936	1936
query48	424	322	316	316
query49	720	397	417	397
query50	865	391	384	384
query51	7450	7251	6975	6975
query52	99	86	88	86
query53	252	180	187	180
query54	519	387	419	387
query55	74	76	78	76
query56	262	223	243	223
query57	1258	1105	1115	1105
query58	205	204	206	204
query59	3231	3010	2917	2917
query60	282	246	245	245
query61	114	107	101	101
query62	793	665	641	641
query63	214	185	192	185
query64	1710	644	640	640
query65	3328	3192	3187	3187
query66	718	301	326	301
query67	16147	15748	15539	15539
query68	3955	580	564	564
query69	433	262	251	251
query70	1196	1158	1148	1148
query71	356	247	254	247
query72	6198	4036	4085	4036
query73	804	362	365	362
query74	10316	9067	9069	9067
query75	3382	2637	2664	2637
query76	1900	1087	1052	1052
query77	494	274	268	268
query78	10411	9502	9464	9464
query79	1134	597	590	590
query80	844	530	446	446
query81	495	238	230	230
query82	214	122	117	117
query83	168	148	148	148
query84	282	74	70	70
query85	939	290	302	290
query86	367	291	296	291
query87	4706	4597	4679	4597
query88	3956	2211	2168	2168
query89	418	289	292	289
query90	2111	186	193	186
query91	138	101	104	101
query92	60	54	51	51
query93	1772	538	545	538
query94	872	298	263	263
query95	343	257	244	244
query96	623	278	280	278
query97	2905	2660	2720	2660
query98	221	192	200	192
query99	1593	1346	1321	1321
Total cold run time: 319191 ms
Total hot run time: 198338 ms

doris-robot · 2024-12-03T09:20:52Z

ClickBench: Total hot run time: 32.34 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f5120edd70db2f580c199ffa66abcfe9eed14a56, data reload: false

query1	0.03	0.03	0.04
query2	0.06	0.03	0.03
query3	0.23	0.08	0.06
query4	1.63	0.11	0.11
query5	0.43	0.40	0.41
query6	1.17	0.66	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.50
query10	0.55	0.57	0.55
query11	0.13	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	2.82	2.73	2.85
query15	0.90	0.83	0.82
query16	0.38	0.37	0.37
query17	1.05	1.06	1.00
query18	0.23	0.21	0.21
query19	1.95	1.70	1.95
query20	0.02	0.00	0.02
query21	15.35	0.60	0.62
query22	2.72	1.90	2.44
query23	16.95	1.00	0.90
query24	3.38	1.13	0.79
query25	0.26	0.15	0.05
query26	0.58	0.14	0.14
query27	0.04	0.04	0.06
query28	10.71	1.10	1.09
query29	12.54	3.24	3.25
query30	0.25	0.06	0.06
query31	2.85	0.39	0.39
query32	3.26	0.46	0.46
query33	2.99	3.07	3.01
query34	16.92	4.44	4.50
query35	4.54	4.44	4.46
query36	0.69	0.50	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 107.49 s
Total hot run time: 32.34 s

wuwenchi · 2024-12-04T09:56:43Z

run buildall

doris-robot · 2024-12-04T11:08:16Z

TPC-H: Total hot run time: 40246 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 201ca19bbe4fd95b191a46a9a8c17c463b722173, data reload: false

------ Round 1 ----------------------------------
q1	17616	7525	7275	7275
q2	2055	173	169	169
q3	10597	1129	1205	1129
q4	10569	759	772	759
q5	7629	2755	2756	2755
q6	240	148	146	146
q7	1004	637	602	602
q8	9234	1894	1944	1894
q9	6719	6569	6476	6476
q10	6982	2333	2331	2331
q11	455	271	265	265
q12	432	227	226	226
q13	17771	3026	3066	3026
q14	249	210	218	210
q15	569	536	529	529
q16	656	578	615	578
q17	976	522	519	519
q18	7335	6615	6703	6615
q19	1326	1009	1087	1009
q20	484	187	184	184
q21	4036	3258	3226	3226
q22	397	327	323	323
Total cold run time: 107331 ms
Total hot run time: 40246 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7246	7207	7349	7207
q2	325	222	224	222
q3	2952	2891	2969	2891
q4	2083	1887	2006	1887
q5	5668	5708	5704	5704
q6	232	141	141	141
q7	2250	1822	1854	1822
q8	3391	3563	3532	3532
q9	9001	9135	9046	9046
q10	3631	3555	3559	3555
q11	608	516	488	488
q12	829	638	630	630
q13	10777	3318	3248	3248
q14	311	271	287	271
q15	596	529	527	527
q16	690	655	656	655
q17	1863	1642	1611	1611
q18	8353	7818	7708	7708
q19	1705	1581	1556	1556
q20	2154	1900	1884	1884
q21	5585	5565	5603	5565
q22	650	600	613	600
Total cold run time: 70900 ms
Total hot run time: 60750 ms

doris-robot · 2024-12-04T11:20:56Z

TPC-DS: Total hot run time: 197319 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 201ca19bbe4fd95b191a46a9a8c17c463b722173, data reload: false

query1	1249	961	941	941
query2	6239	2059	1999	1999
query3	10972	4445	4468	4445
query4	67008	28409	23540	23540
query5	4898	482	462	462
query6	413	194	180	180
query7	5477	308	300	300
query8	315	229	225	225
query9	8323	2724	2718	2718
query10	441	243	251	243
query11	16897	15281	16044	15281
query12	162	104	105	104
query13	1432	448	410	410
query14	10610	6971	6791	6791
query15	212	186	194	186
query16	6999	431	503	431
query17	1054	582	595	582
query18	1705	320	322	320
query19	213	170	165	165
query20	123	117	116	116
query21	201	105	107	105
query22	4667	4380	4578	4380
query23	35060	34450	35589	34450
query24	5358	2662	2617	2617
query25	493	387	400	387
query26	665	154	156	154
query27	2116	279	283	279
query28	5080	2528	2514	2514
query29	675	427	415	415
query30	210	149	148	148
query31	1008	825	853	825
query32	70	58	56	56
query33	475	315	299	299
query34	954	516	533	516
query35	909	763	790	763
query36	1112	970	957	957
query37	126	73	75	73
query38	4601	4383	4456	4383
query39	1528	1498	1467	1467
query40	205	119	96	96
query41	43	41	42	41
query42	114	99	101	99
query43	548	501	498	498
query44	1204	834	845	834
query45	197	171	180	171
query46	1192	718	714	714
query47	2050	1927	1901	1901
query48	427	332	336	332
query49	738	423	382	382
query50	857	391	398	391
query51	7415	7175	7176	7175
query52	99	88	88	88
query53	255	184	173	173
query54	509	394	391	391
query55	77	74	72	72
query56	261	233	233	233
query57	1213	1128	1118	1118
query58	206	217	215	215
query59	3234	3186	2940	2940
query60	268	245	241	241
query61	109	108	104	104
query62	774	680	674	674
query63	220	225	200	200
query64	1476	674	650	650
query65	3308	3393	3222	3222
query66	719	300	296	296
query67	16089	15724	15684	15684
query68	3818	556	550	550
query69	434	264	245	245
query70	1207	1151	1099	1099
query71	370	245	245	245
query72	6215	4074	4072	4072
query73	790	361	359	359
query74	10219	9156	9072	9072
query75	3415	2701	2686	2686
query76	1893	1149	1002	1002
query77	480	277	278	277
query78	10504	9548	9501	9501
query79	1756	624	631	624
query80	1196	422	427	422
query81	505	249	234	234
query82	391	120	113	113
query83	177	146	143	143
query84	278	70	68	68
query85	967	291	295	291
query86	410	289	305	289
query87	4733	4647	4519	4519
query88	3757	2217	2180	2180
query89	423	285	293	285
query90	1974	214	183	183
query91	139	101	105	101
query92	59	49	52	49
query93	2551	546	551	546
query94	812	291	292	291
query95	344	253	246	246
query96	633	284	286	284
query97	2876	2674	2684	2674
query98	218	193	204	193
query99	1584	1314	1326	1314
Total cold run time: 318985 ms
Total hot run time: 197319 ms

doris-robot · 2024-12-04T11:26:26Z

ClickBench: Total hot run time: 33.21 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 201ca19bbe4fd95b191a46a9a8c17c463b722173, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.08	0.07
query4	1.61	0.11	0.10
query5	0.43	0.42	0.42
query6	1.15	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.51	0.51
query10	0.54	0.56	0.55
query11	0.14	0.11	0.11
query12	0.14	0.11	0.12
query13	0.60	0.60	0.59
query14	2.72	2.75	2.86
query15	0.91	0.82	0.84
query16	0.39	0.39	0.40
query17	1.05	1.04	1.03
query18	0.22	0.21	0.21
query19	1.84	1.75	1.99
query20	0.02	0.01	0.02
query21	15.35	0.60	0.57
query22	2.29	1.65	1.85
query23	17.29	0.84	0.76
query24	3.06	2.14	1.80
query25	0.12	0.10	0.15
query26	0.69	0.14	0.14
query27	0.05	0.05	0.03
query28	9.50	1.10	1.08
query29	12.54	3.32	3.26
query30	0.25	0.06	0.07
query31	2.87	0.39	0.39
query32	3.28	0.47	0.45
query33	3.09	3.08	3.02
query34	17.08	4.46	4.45
query35	4.52	4.51	4.51
query36	0.66	0.48	0.50
query37	0.08	0.06	0.05
query38	0.05	0.04	0.03
query39	0.02	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.84 s
Total hot run time: 33.21 s

wuwenchi · 2024-12-13T08:54:58Z

run buildall

github-actions · 2024-12-15T03:50:35Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-15T03:50:37Z

PR approved by anyone and no changes requested.

wuwenchi · 2024-12-16T02:02:47Z

run buildall

wuwenchi · 2024-12-16T07:03:09Z

run compile

wuwenchi · 2024-12-23T06:29:31Z

run buildall

doris-robot · 2024-12-23T06:48:49Z

TPC-H: Total hot run time: 40065 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4dd3f0d01353ff7e1aa4c5da2c38ef3b2c6bd1eb, data reload: false

------ Round 1 ----------------------------------
q1	17615	7424	7283	7283
q2	2050	185	177	177
q3	10536	1136	1228	1136
q4	10216	715	742	715
q5	7588	2711	2710	2710
q6	234	148	147	147
q7	1002	636	638	636
q8	9245	1828	1924	1828
q9	6654	6466	6412	6412
q10	6973	2268	2318	2268
q11	485	270	257	257
q12	425	232	223	223
q13	17771	2961	2997	2961
q14	251	211	220	211
q15	556	504	496	496
q16	638	574	582	574
q17	984	573	521	521
q18	7455	6827	6688	6688
q19	1350	1027	999	999
q20	484	181	182	181
q21	4427	3324	3402	3324
q22	381	318	328	318
Total cold run time: 107320 ms
Total hot run time: 40065 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7283	7255	7255	7255
q2	327	239	227	227
q3	2965	2807	3094	2807
q4	2142	1891	1858	1858
q5	5639	5718	5606	5606
q6	220	141	138	138
q7	2240	1796	1820	1796
q8	3395	3502	3500	3500
q9	8892	8883	8878	8878
q10	3594	3516	3554	3516
q11	606	509	515	509
q12	823	602	605	602
q13	12294	3102	3112	3102
q14	295	258	259	258
q15	565	504	504	504
q16	683	636	635	635
q17	1837	1636	1587	1587
q18	7889	7530	7449	7449
q19	1678	1507	1588	1507
q20	2022	1776	1768	1768
q21	5476	5277	5337	5277
q22	636	567	570	567
Total cold run time: 71501 ms
Total hot run time: 59346 ms

doris-robot · 2024-12-23T07:00:54Z

TPC-DS: Total hot run time: 190336 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4dd3f0d01353ff7e1aa4c5da2c38ef3b2c6bd1eb, data reload: false

query1	974	381	384	381
query2	6529	2376	2508	2376
query3	6709	232	223	223
query4	33610	23447	23338	23338
query5	4375	480	460	460
query6	295	183	191	183
query7	4642	309	307	307
query8	307	240	238	238
query9	9256	2703	2700	2700
query10	494	251	250	250
query11	18091	15168	15086	15086
query12	158	109	112	109
query13	1677	436	410	410
query14	10209	7724	6982	6982
query15	341	198	192	192
query16	8316	447	430	430
query17	1785	593	574	574
query18	2168	304	319	304
query19	363	148	146	146
query20	113	105	114	105
query21	215	107	115	107
query22	4462	4325	4331	4325
query23	34895	33507	36348	33507
query24	11406	2482	2450	2450
query25	546	377	385	377
query26	1134	155	156	155
query27	2840	330	319	319
query28	7680	2411	2428	2411
query29	654	407	409	407
query30	306	145	153	145
query31	1039	839	817	817
query32	99	86	58	58
query33	782	297	295	295
query34	988	514	518	514
query35	869	763	719	719
query36	1090	951	976	951
query37	203	74	75	74
query38	4324	4113	4303	4113
query39	1519	1445	1424	1424
query40	299	101	101	101
query41	53	45	44	44
query42	114	105	100	100
query43	554	487	492	487
query44	1284	828	827	827
query45	187	175	172	172
query46	1178	707	698	698
query47	1947	1837	1833	1833
query48	420	334	326	326
query49	1273	386	397	386
query50	813	394	384	384
query51	7263	7064	7087	7064
query52	108	93	97	93
query53	261	183	186	183
query54	1030	418	407	407
query55	84	80	79	79
query56	262	252	234	234
query57	1288	1137	1130	1130
query58	246	222	237	222
query59	3283	3055	2981	2981
query60	273	235	251	235
query61	108	107	110	107
query62	895	692	673	673
query63	216	188	197	188
query64	5058	678	639	639
query65	3268	3206	3209	3206
query66	1280	312	312	312
query67	15818	15597	15481	15481
query68	5640	559	553	553
query69	432	255	265	255
query70	1231	1259	1143	1143
query71	434	269	257	257
query72	6030	4155	4065	4065
query73	792	369	361	361
query74	10093	8928	8817	8817
query75	3424	2642	2654	2642
query76	3565	1140	1107	1107
query77	588	271	267	267
query78	10358	9390	9357	9357
query79	1137	598	608	598
query80	823	433	460	433
query81	529	226	240	226
query82	647	125	122	122
query83	259	149	149	149
query84	236	76	73	73
query85	1362	304	313	304
query86	436	299	310	299
query87	4596	4322	4468	4322
query88	3929	2247	2230	2230
query89	413	295	305	295
query90	2217	195	192	192
query91	149	108	109	108
query92	64	52	57	52
query93	1613	555	547	547
query94	1089	303	286	286
query95	352	249	256	249
query96	625	283	286	283
query97	2847	2667	2692	2667
query98	221	208	194	194
query99	1606	1340	1329	1329
Total cold run time: 302063 ms
Total hot run time: 190336 ms

doris-robot · 2024-12-23T07:06:26Z

ClickBench: Total hot run time: 33.52 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4dd3f0d01353ff7e1aa4c5da2c38ef3b2c6bd1eb, data reload: false

query1	0.04	0.04	0.04
query2	0.10	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.10	0.11
query5	0.40	0.38	0.40
query6	1.15	0.65	0.65
query7	0.03	0.02	0.02
query8	0.04	0.04	0.04
query9	0.57	0.51	0.51
query10	0.56	0.58	0.55
query11	0.15	0.11	0.10
query12	0.14	0.10	0.11
query13	0.61	0.60	0.58
query14	2.72	2.77	2.81
query15	0.90	0.82	0.83
query16	0.40	0.38	0.38
query17	1.06	1.04	1.03
query18	0.24	0.21	0.21
query19	1.95	1.78	1.99
query20	0.01	0.02	0.01
query21	15.36	0.58	0.59
query22	2.95	2.45	2.28
query23	17.06	1.13	0.89
query24	3.42	1.30	1.28
query25	0.23	0.24	0.05
query26	0.41	0.14	0.14
query27	0.04	0.05	0.05
query28	10.13	1.12	1.08
query29	12.59	3.22	3.20
query30	0.24	0.06	0.07
query31	2.86	0.38	0.37
query32	3.24	0.47	0.48
query33	3.12	3.26	3.12
query34	17.35	4.50	4.54
query35	4.51	4.51	4.53
query36	0.70	0.48	0.50
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.14
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 107.62 s
Total hot run time: 33.52 s

wuwenchi · 2024-12-24T02:08:24Z

run feut

wuwenchi · 2024-12-24T02:08:35Z

run p0

wuwenchi · 2024-12-25T08:35:12Z

run buildall

wuwenchi · 2024-12-26T02:00:02Z

run buildall

fix hudi hbase jar conflict

wuwenchi · 2024-12-26T03:18:00Z

run buildall

Problem Summary: 1. Increase the schema cache to reduce the time to obtain the schema. 2. `HoodieTableMetaClient` is stored in `HMSExternalTable` to prevent redundant creation. 3. Cache HoodieTableFileSystemView to speed up getting FileGroup or FileSlice. 4. Fix analyze path for `file:/abc`. 5. Add `FSDataInputStreamWrapper` to solve hudi conflict class.

…4909) (#46086) bp: #44909

…4909) (#46084) bp: #44909

wuwenchi marked this pull request as draft December 3, 2024 07:39

wuwenchi force-pushed the opt-hudi-scan branch 2 times, most recently from f3636a1 to 201ca19 Compare December 4, 2024 07:44

wuwenchi marked this pull request as ready for review December 5, 2024 08:51

wuwenchi marked this pull request as draft December 11, 2024 06:18

wuwenchi force-pushed the opt-hudi-scan branch from 4c121d9 to d74fea1 Compare December 13, 2024 04:19

wuwenchi marked this pull request as ready for review December 13, 2024 08:55

morningman added dev/2.1.x dev/3.0.x labels Dec 15, 2024

morningman previously approved these changes Dec 15, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 15, 2024

github-actions bot added the reviewed label Dec 15, 2024

suxiaogang223 approved these changes Dec 15, 2024

View reviewed changes

wuwenchi dismissed morningman’s stale review via 0f97eff December 16, 2024 02:02

wuwenchi force-pushed the opt-hudi-scan branch from 438f122 to 0f97eff Compare December 16, 2024 02:02

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 16, 2024

wuwenchi added 4 commits December 26, 2024 10:51

add fsview cache

ccc1d86

fix hudi hbase jar conflict

fix

3d7a282

fix

bc8deab

fix

1c4a204

wuwenchi force-pushed the opt-hudi-scan branch from 705bc0f to 1c4a204 Compare December 26, 2024 03:17

morningman approved these changes Dec 26, 2024

View reviewed changes

morningman merged commit ddb727f into apache:master Dec 26, 2024
23 checks passed

github-actions bot added dev/3.0.x-conflict dev/2.1.x-conflict labels Dec 26, 2024

wuwenchi mentioned this pull request Dec 27, 2024

[opt](hms table)Some optimizations for hms external table for 2.1 (#44909) #46084

Merged

wuwenchi mentioned this pull request Dec 27, 2024

[opt](hms table)Some optimizations for hms external table for 3.0 (#44909) #46086

Merged

morningman pushed a commit that referenced this pull request Dec 27, 2024

[opt](hms table)Some optimizations for hms external table for 3.0 (#4…

01dc5bc

…4909) (#46086) bp: #44909

morningman added dev/3.0.4-merged and removed dev/3.0.x labels Dec 27, 2024

CalvinKirs pushed a commit that referenced this pull request Dec 27, 2024

[opt](hms table)Some optimizations for hms external table for 2.1 (#4…

69961a3

…4909) (#46084) bp: #44909

morningman added dev/2.1.8-merged and removed dev/2.1.x dev/3.0.x-conflict dev/2.1.x-conflict labels Dec 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[opt](hms table)Some optimizations for hms external table #44909

[opt](hms table)Some optimizations for hms external table #44909

wuwenchi commented Dec 3, 2024 •

edited

Loading

doris-robot commented Dec 3, 2024

wuwenchi commented Dec 3, 2024

doris-robot commented Dec 3, 2024

doris-robot commented Dec 3, 2024

doris-robot commented Dec 3, 2024

wuwenchi commented Dec 4, 2024

doris-robot commented Dec 4, 2024

doris-robot commented Dec 4, 2024

doris-robot commented Dec 4, 2024

wuwenchi commented Dec 13, 2024

github-actions bot commented Dec 15, 2024

github-actions bot commented Dec 15, 2024

wuwenchi commented Dec 16, 2024

wuwenchi commented Dec 16, 2024

wuwenchi commented Dec 23, 2024

doris-robot commented Dec 23, 2024

doris-robot commented Dec 23, 2024

doris-robot commented Dec 23, 2024

wuwenchi commented Dec 24, 2024

wuwenchi commented Dec 24, 2024

wuwenchi commented Dec 25, 2024

wuwenchi commented Dec 26, 2024

wuwenchi commented Dec 26, 2024

[opt](hms table)Some optimizations for hms external table #44909

[opt](hms table)Some optimizations for hms external table #44909

Conversation

wuwenchi commented Dec 3, 2024 • edited Loading

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented Dec 3, 2024

wuwenchi commented Dec 3, 2024

doris-robot commented Dec 3, 2024

doris-robot commented Dec 3, 2024

doris-robot commented Dec 3, 2024

wuwenchi commented Dec 4, 2024

doris-robot commented Dec 4, 2024

doris-robot commented Dec 4, 2024

doris-robot commented Dec 4, 2024

wuwenchi commented Dec 13, 2024

github-actions bot commented Dec 15, 2024

github-actions bot commented Dec 15, 2024

wuwenchi commented Dec 16, 2024

wuwenchi commented Dec 16, 2024

wuwenchi commented Dec 23, 2024

doris-robot commented Dec 23, 2024

doris-robot commented Dec 23, 2024

doris-robot commented Dec 23, 2024

wuwenchi commented Dec 24, 2024

wuwenchi commented Dec 24, 2024

wuwenchi commented Dec 25, 2024

wuwenchi commented Dec 26, 2024

wuwenchi commented Dec 26, 2024

wuwenchi commented Dec 3, 2024 •

edited

Loading