[feat](mtmv)Paimon queries the data in the cache instead of querying the latest data #44911

zddr · 2024-12-03T07:50:25Z

What problem does this PR solve?

add PaimonMetadataCacheMgr in ExternalMetaCacheMgr to manage snapshotCache of paimon table
move paimonSchemaCache to PaimonMetadataCacheMgr, and add schemaId as part of key
PaimonExternalTable overrides the methods in ExternalTable and supports partition pruning
PaimonExternalTable implements the MvcTable interface, supporting the retrieval of snapshot data from the cache during queries to avoid cache refreshes that may result in different versions of metadata being used in a single query
MTMVTask retrieves snapshot data of mvccTable before the task starts to avoid cache refresh that may result in different versions of metadata being used in a single refresh task

behavior changes of query paimon table:

FE has just started and is query the latest data
Paimon data has changed, Doris is still query the previous data
After the snapshot cache expires, Doris will query the latest data
desc paimon; The schema corresponding to the snapshotId in the snapshot cache is displayed

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Paimon queries the data in the cache instead of querying the latest data

Release note

Paimon queries the data in the cache instead of querying the latest data

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
    The previous case was able to cover normal read and write operations, but querying fixed versions of cases requires the ability to manipulate Paimon data, which is currently not possible
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-12-03T07:50:30Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zddr · 2024-12-03T12:22:23Z

run buildall

doris-robot · 2024-12-03T12:46:27Z

TPC-H: Total hot run time: 40539 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 027609485cad47c91b79b0bd6a04cc6fc8a4fefa, data reload: false

------ Round 1 ----------------------------------
q1	17610	7534	7476	7476
q2	2053	177	163	163
q3	10564	1168	1199	1168
q4	10236	763	694	694
q5	7585	2769	2749	2749
q6	238	152	145	145
q7	1010	613	620	613
q8	9254	1906	1917	1906
q9	6683	6588	6518	6518
q10	7044	2327	2326	2326
q11	473	261	271	261
q12	444	220	221	220
q13	17796	3031	3069	3031
q14	250	220	230	220
q15	606	532	524	524
q16	648	573	581	573
q17	1006	563	532	532
q18	7564	6817	6857	6817
q19	1344	1021	1000	1000
q20	451	191	178	178
q21	4004	3195	3116	3116
q22	384	309	330	309
Total cold run time: 107247 ms
Total hot run time: 40539 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7314	7287	7239	7239
q2	332	231	230	230
q3	2967	2791	2828	2791
q4	1974	1741	1710	1710
q5	5372	5426	5432	5426
q6	220	140	138	138
q7	2147	1760	1729	1729
q8	3265	3424	3402	3402
q9	8649	8710	8673	8673
q10	3505	3496	3448	3448
q11	591	486	503	486
q12	784	603	583	583
q13	12771	3076	3107	3076
q14	310	263	257	257
q15	573	504	513	504
q16	655	651	630	630
q17	1803	1581	1575	1575
q18	7801	7583	7341	7341
q19	1662	1433	1587	1433
q20	2059	1855	1841	1841
q21	5421	5292	5233	5233
q22	626	552	560	552
Total cold run time: 70801 ms
Total hot run time: 58297 ms

doris-robot · 2024-12-03T12:58:28Z

TPC-DS: Total hot run time: 191071 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 027609485cad47c91b79b0bd6a04cc6fc8a4fefa, data reload: false

query1	958	389	367	367
query2	5246	2097	2042	2042
query3	6616	209	224	209
query4	33822	23658	23694	23658
query5	4341	465	467	465
query6	273	196	192	192
query7	4585	295	306	295
query8	301	244	239	239
query9	9621	2714	2713	2713
query10	468	251	243	243
query11	18234	15324	15361	15324
query12	165	99	104	99
query13	1647	424	421	421
query14	11168	7147	7978	7147
query15	308	179	194	179
query16	7305	459	449	449
query17	2157	580	570	570
query18	2001	289	285	285
query19	346	149	145	145
query20	119	107	116	107
query21	205	111	101	101
query22	4394	4327	4112	4112
query23	35311	34248	34206	34206
query24	11429	2444	2526	2444
query25	610	381	414	381
query26	1620	146	152	146
query27	2745	275	273	273
query28	7896	2418	2429	2418
query29	811	437	403	403
query30	298	151	154	151
query31	1100	813	836	813
query32	69	55	55	55
query33	740	287	307	287
query34	933	500	551	500
query35	897	754	759	754
query36	1088	932	952	932
query37	252	78	72	72
query38	4342	4196	4226	4196
query39	1463	1421	1408	1408
query40	277	98	98	98
query41	47	42	46	42
query42	105	96	96	96
query43	530	479	498	479
query44	1263	805	799	799
query45	191	159	165	159
query46	1165	694	731	694
query47	1983	1860	1852	1852
query48	396	319	320	319
query49	1255	379	393	379
query50	805	400	378	378
query51	7291	7129	6959	6959
query52	100	91	88	88
query53	257	192	175	175
query54	1241	401	403	401
query55	75	76	75	75
query56	284	237	242	237
query57	1218	1137	1098	1098
query58	223	203	204	203
query59	3167	3065	2873	2873
query60	276	246	246	246
query61	147	110	107	107
query62	877	671	672	671
query63	215	180	183	180
query64	3677	657	646	646
query65	3323	3233	3237	3233
query66	1176	312	309	309
query67	16231	15798	15576	15576
query68	5404	558	540	540
query69	405	247	246	246
query70	1188	1078	1110	1078
query71	326	248	261	248
query72	6122	4055	4055	4055
query73	767	368	366	366
query74	10548	8982	9105	8982
query75	3362	2650	2654	2650
query76	3014	1195	1105	1105
query77	484	266	270	266
query78	10429	9438	9514	9438
query79	1172	600	584	584
query80	1176	432	427	427
query81	529	226	238	226
query82	1051	122	116	116
query83	242	146	143	143
query84	237	73	69	69
query85	1307	297	294	294
query86	350	311	294	294
query87	4750	4559	4434	4434
query88	3394	2223	2194	2194
query89	397	300	290	290
query90	2095	185	183	183
query91	139	103	102	102
query92	62	49	51	49
query93	1070	531	537	531
query94	1087	293	299	293
query95	364	247	246	246
query96	601	273	272	272
query97	2841	2672	2669	2669
query98	224	202	196	196
query99	1543	1312	1332	1312
Total cold run time: 300359 ms
Total hot run time: 191071 ms

doris-robot · 2024-12-03T13:03:57Z

ClickBench: Total hot run time: 33.3 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 027609485cad47c91b79b0bd6a04cc6fc8a4fefa, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.41	0.43	0.39
query6	1.14	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.51	0.49
query10	0.57	0.56	0.55
query11	0.14	0.11	0.11
query12	0.15	0.11	0.12
query13	0.61	0.60	0.60
query14	2.71	2.71	2.75
query15	0.91	0.83	0.84
query16	0.39	0.40	0.39
query17	1.03	1.07	1.06
query18	0.22	0.21	0.20
query19	1.95	1.87	1.99
query20	0.01	0.02	0.01
query21	15.35	0.59	0.59
query22	2.92	1.94	1.79
query23	17.02	1.27	0.90
query24	2.83	1.62	1.96
query25	0.34	0.05	0.12
query26	0.48	0.14	0.14
query27	0.04	0.04	0.05
query28	9.70	1.09	1.08
query29	12.54	3.26	3.25
query30	0.25	0.07	0.07
query31	2.86	0.38	0.38
query32	3.26	0.47	0.47
query33	2.97	3.06	3.00
query34	17.07	4.48	4.41
query35	4.55	4.49	4.48
query36	0.68	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.15 s
Total hot run time: 33.3 s

fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMetadataCacheMgr.java

fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonSource.java

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneFileScanPartition.java

fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java

fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java

zddr · 2024-12-05T08:15:09Z

run buildall

doris-robot · 2024-12-05T08:54:36Z

TPC-H: Total hot run time: 39991 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5e5211e47fd68c4437c10d1cf5162f921e8a82b9, data reload: false

------ Round 1 ----------------------------------
q1	17610	7465	7297	7297
q2	2081	178	165	165
q3	10707	1077	1200	1077
q4	10551	794	729	729
q5	7609	2697	2750	2697
q6	235	148	148	148
q7	989	621	615	615
q8	9254	1846	1915	1846
q9	6712	6534	6508	6508
q10	7042	2314	2294	2294
q11	466	269	259	259
q12	424	227	223	223
q13	17775	3014	3066	3014
q14	232	224	212	212
q15	571	524	531	524
q16	665	579	596	579
q17	977	590	550	550
q18	7260	6646	6744	6646
q19	1335	1058	1005	1005
q20	462	176	179	176
q21	3981	3185	3111	3111
q22	371	316	317	316
Total cold run time: 107309 ms
Total hot run time: 39991 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7239	7317	7249	7249
q2	331	230	235	230
q3	2901	2794	2964	2794
q4	2096	1829	1831	1829
q5	5679	5657	5673	5657
q6	227	142	140	140
q7	2232	1814	1812	1812
q8	3385	3559	3468	3468
q9	8994	8967	9051	8967
q10	3629	3559	3544	3544
q11	600	507	500	500
q12	826	610	628	610
q13	13200	3313	3197	3197
q14	310	281	275	275
q15	577	524	522	522
q16	670	620	627	620
q17	1854	1628	1608	1608
q18	8257	7878	7527	7527
q19	1679	1604	1452	1452
q20	2143	1913	1866	1866
q21	5671	5579	5519	5519
q22	652	580	550	550
Total cold run time: 73152 ms
Total hot run time: 59936 ms

doris-robot · 2024-12-05T09:07:20Z

TPC-DS: Total hot run time: 198051 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5e5211e47fd68c4437c10d1cf5162f921e8a82b9, data reload: false

query1	1460	992	949	949
query2	6247	2080	1983	1983
query3	10937	4301	4508	4301
query4	66782	28740	23906	23906
query5	4950	473	486	473
query6	419	190	178	178
query7	5502	314	301	301
query8	328	253	247	247
query9	8752	2721	2721	2721
query10	446	248	259	248
query11	17209	15545	15904	15545
query12	154	105	108	105
query13	1440	408	400	400
query14	10996	6915	7656	6915
query15	225	187	187	187
query16	7126	498	473	473
query17	1235	602	593	593
query18	1876	320	320	320
query19	218	166	159	159
query20	126	113	114	113
query21	213	113	107	107
query22	4690	4495	4676	4495
query23	34675	34664	34540	34540
query24	5467	2545	2556	2545
query25	484	383	390	383
query26	651	151	153	151
query27	2033	292	289	289
query28	4441	2497	2479	2479
query29	688	430	434	430
query30	212	154	149	149
query31	1009	857	849	849
query32	69	53	60	53
query33	450	307	329	307
query34	920	520	530	520
query35	921	762	777	762
query36	1090	958	974	958
query37	121	79	72	72
query38	4646	4406	4522	4406
query39	1500	1537	1493	1493
query40	203	106	100	100
query41	45	42	45	42
query42	115	104	101	101
query43	546	504	512	504
query44	1220	828	829	828
query45	212	176	180	176
query46	1201	739	745	739
query47	2039	1932	1933	1932
query48	410	338	326	326
query49	727	394	400	394
query50	838	401	397	397
query51	7291	7216	7037	7037
query52	103	87	90	87
query53	263	183	182	182
query54	513	393	405	393
query55	76	72	80	72
query56	246	236	252	236
query57	1266	1104	1120	1104
query58	236	209	212	209
query59	3206	2960	3168	2960
query60	270	249	263	249
query61	109	106	121	106
query62	796	721	755	721
query63	216	189	189	189
query64	1342	654	646	646
query65	3308	3189	3181	3181
query66	701	299	301	299
query67	16225	15615	15597	15597
query68	4063	584	554	554
query69	435	254	249	249
query70	1199	1158	1147	1147
query71	366	249	253	249
query72	6475	4153	4006	4006
query73	756	363	365	363
query74	10453	9105	9189	9105
query75	3360	2649	2668	2649
query76	1921	1136	1050	1050
query77	464	278	290	278
query78	10522	9534	9372	9372
query79	2203	608	597	597
query80	1402	431	430	430
query81	511	231	237	231
query82	1227	127	117	117
query83	168	145	144	144
query84	283	107	73	73
query85	989	314	299	299
query86	414	299	292	292
query87	4657	4579	4660	4579
query88	3836	2210	2184	2184
query89	417	295	291	291
query90	1959	189	193	189
query91	140	107	103	103
query92	68	49	48	48
query93	2913	548	539	539
query94	811	294	286	286
query95	364	245	251	245
query96	636	280	277	277
query97	2870	2687	2671	2671
query98	233	200	192	192
query99	1654	1330	1332	1330
Total cold run time: 322200 ms
Total hot run time: 198051 ms

doris-robot · 2024-12-05T09:12:49Z

ClickBench: Total hot run time: 32.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5e5211e47fd68c4437c10d1cf5162f921e8a82b9, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.63	0.10	0.11
query5	0.44	0.40	0.40
query6	1.16	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.51	0.51
query10	0.56	0.57	0.58
query11	0.15	0.11	0.11
query12	0.13	0.11	0.11
query13	0.60	0.61	0.59
query14	2.76	2.83	2.78
query15	0.90	0.82	0.83
query16	0.39	0.40	0.38
query17	1.03	1.07	1.07
query18	0.22	0.20	0.20
query19	1.95	1.79	2.02
query20	0.02	0.01	0.02
query21	15.37	0.63	0.59
query22	2.61	1.81	1.72
query23	17.08	0.93	0.70
query24	3.26	1.32	2.19
query25	0.37	0.27	0.05
query26	0.45	0.14	0.15
query27	0.04	0.05	0.04
query28	9.64	1.08	1.08
query29	12.54	3.24	3.22
query30	0.25	0.07	0.06
query31	2.87	0.39	0.37
query32	3.27	0.47	0.47
query33	2.99	3.04	3.03
query34	17.23	4.43	4.51
query35	4.48	4.50	4.46
query36	0.67	0.47	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.03	0.02
query40	0.18	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.56 s
Total hot run time: 32.72 s

github-actions · 2024-12-05T10:06:25Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-05T10:06:27Z

PR approved by anyone and no changes requested.

doris-robot · 2024-12-09T11:59:12Z

ClickBench: Total hot run time: 32.62 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1de646f5174a149c70c6f94640df38f22b1dc79e, data reload: false

query1	0.03	0.04	0.04
query2	0.07	0.03	0.03
query3	0.23	0.08	0.07
query4	1.62	0.10	0.10
query5	0.42	0.39	0.42
query6	1.13	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.52	0.52
query10	0.54	0.57	0.54
query11	0.14	0.11	0.11
query12	0.14	0.10	0.11
query13	0.62	0.61	0.59
query14	2.81	2.83	2.76
query15	0.89	0.83	0.83
query16	0.37	0.37	0.37
query17	1.07	1.06	1.08
query18	0.23	0.23	0.21
query19	1.98	1.86	2.02
query20	0.02	0.01	0.02
query21	15.35	0.57	0.57
query22	2.20	1.89	2.33
query23	16.92	0.84	0.93
query24	2.87	0.86	1.02
query25	0.21	0.04	0.12
query26	0.50	0.14	0.13
query27	0.07	0.05	0.04
query28	11.19	1.10	1.07
query29	12.67	3.30	3.24
query30	0.24	0.06	0.06
query31	2.87	0.38	0.39
query32	3.26	0.46	0.46
query33	2.99	3.03	3.09
query34	17.34	4.46	4.53
query35	4.51	4.52	4.49
query36	0.67	0.48	0.49
query37	0.09	0.06	0.07
query38	0.06	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.31 s
Total hot run time: 32.62 s

zddr · 2024-12-10T02:58:39Z

run buildall

doris-robot · 2024-12-10T03:31:30Z

TPC-H: Total hot run time: 40026 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b1da13804a9c8a2d149de0ca7640a83ea6e517bf, data reload: false

------ Round 1 ----------------------------------
q1	17606	7380	7256	7256
q2	2049	173	166	166
q3	10639	1093	1167	1093
q4	10552	688	685	685
q5	7608	2690	2723	2690
q6	234	150	146	146
q7	1001	624	613	613
q8	9239	1875	1883	1875
q9	6655	6502	6475	6475
q10	6931	2331	2328	2328
q11	456	250	257	250
q12	422	224	230	224
q13	17766	3027	2990	2990
q14	239	202	214	202
q15	588	551	514	514
q16	645	573	593	573
q17	961	543	521	521
q18	7194	6910	6684	6684
q19	1333	1002	942	942
q20	453	196	181	181
q21	4303	3298	3298	3298
q22	380	322	320	320
Total cold run time: 107254 ms
Total hot run time: 40026 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7272	7208	7249	7208
q2	329	231	239	231
q3	2898	2801	2885	2801
q4	2050	1800	1775	1775
q5	5650	5630	5649	5630
q6	236	146	144	144
q7	2248	1822	1813	1813
q8	3355	3588	3488	3488
q9	8960	8899	8980	8899
q10	3591	3574	3551	3551
q11	612	515	527	515
q12	820	627	612	612
q13	11009	3253	3204	3204
q14	302	287	270	270
q15	562	513	513	513
q16	689	639	641	639
q17	1850	1644	1617	1617
q18	8320	7716	7793	7716
q19	1721	1599	1533	1533
q20	2085	1848	1923	1848
q21	5618	5566	5525	5525
q22	647	605	650	605
Total cold run time: 70824 ms
Total hot run time: 60137 ms

doris-robot · 2024-12-10T03:44:12Z

TPC-DS: Total hot run time: 197014 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b1da13804a9c8a2d149de0ca7640a83ea6e517bf, data reload: false

query1	1249	970	952	952
query2	6230	2112	2007	2007
query3	10956	4356	4424	4356
query4	66757	29132	23367	23367
query5	4978	469	459	459
query6	450	209	177	177
query7	5636	300	297	297
query8	332	238	245	238
query9	9356	2718	2698	2698
query10	463	241	236	236
query11	17340	15201	15858	15201
query12	142	101	105	101
query13	1601	415	399	399
query14	10431	6908	7209	6908
query15	219	184	202	184
query16	7202	465	496	465
query17	1314	638	585	585
query18	1841	312	326	312
query19	221	160	163	160
query20	120	117	112	112
query21	211	109	104	104
query22	4754	4455	4537	4455
query23	34651	35469	34681	34681
query24	5401	2533	2496	2496
query25	495	384	392	384
query26	656	150	157	150
query27	2036	292	278	278
query28	4283	2473	2486	2473
query29	693	429	409	409
query30	211	148	143	143
query31	1002	828	865	828
query32	63	55	62	55
query33	467	283	277	277
query34	920	521	526	521
query35	877	767	762	762
query36	1060	945	975	945
query37	132	76	70	70
query38	4667	4475	4417	4417
query39	1520	1460	1460	1460
query40	205	99	96	96
query41	44	41	41	41
query42	107	93	97	93
query43	578	514	515	514
query44	1184	846	823	823
query45	196	166	173	166
query46	1179	751	718	718
query47	2018	1965	1944	1944
query48	418	313	333	313
query49	734	391	411	391
query50	832	386	405	386
query51	7372	7117	7124	7117
query52	98	89	90	89
query53	254	174	187	174
query54	514	387	395	387
query55	79	77	74	74
query56	246	234	249	234
query57	1228	1097	1123	1097
query58	215	209	218	209
query59	3243	3124	3298	3124
query60	288	242	245	242
query61	110	105	142	105
query62	772	685	658	658
query63	207	187	199	187
query64	1367	652	636	636
query65	3282	3174	3178	3174
query66	692	290	304	290
query67	15954	15569	15575	15569
query68	4130	592	558	558
query69	419	261	272	261
query70	1214	1143	1141	1141
query71	347	249	259	249
query72	6084	4097	4019	4019
query73	776	359	364	359
query74	10167	8982	9024	8982
query75	3389	2668	2663	2663
query76	1802	1037	1097	1037
query77	473	279	273	273
query78	10462	9399	9431	9399
query79	2178	607	601	601
query80	1393	406	445	406
query81	563	231	229	229
query82	1273	114	120	114
query83	172	137	138	137
query84	283	74	66	66
query85	984	303	292	292
query86	398	293	284	284
query87	4758	4601	4492	4492
query88	3456	2214	2166	2166
query89	423	298	290	290
query90	2000	189	189	189
query91	140	102	104	102
query92	63	51	49	49
query93	2870	541	534	534
query94	829	307	295	295
query95	400	251	248	248
query96	627	280	281	280
query97	2853	2764	2779	2764
query98	241	204	212	204
query99	1589	1358	1382	1358
Total cold run time: 321109 ms
Total hot run time: 197014 ms

doris-robot · 2024-12-10T03:49:42Z

ClickBench: Total hot run time: 33.2 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b1da13804a9c8a2d149de0ca7640a83ea6e517bf, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.06
query4	1.64	0.11	0.10
query5	0.42	0.42	0.41
query6	1.15	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.50	0.50
query10	0.54	0.55	0.55
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.60	0.62
query14	2.85	2.90	2.84
query15	0.90	0.83	0.82
query16	0.39	0.37	0.38
query17	0.98	1.02	1.00
query18	0.22	0.22	0.22
query19	1.96	1.78	1.99
query20	0.02	0.01	0.01
query21	15.38	0.61	0.59
query22	3.06	2.28	1.96
query23	17.22	0.90	0.91
query24	3.30	1.28	2.00
query25	0.21	0.16	0.12
query26	0.56	0.14	0.14
query27	0.04	0.04	0.04
query28	9.58	1.10	1.08
query29	12.54	3.24	3.24
query30	0.25	0.07	0.06
query31	2.87	0.37	0.37
query32	3.29	0.44	0.45
query33	2.98	3.14	3.04
query34	17.08	4.47	4.45
query35	4.63	4.49	4.51
query36	0.65	0.48	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.07 s
Total hot run time: 33.2 s

zddr · 2024-12-10T04:48:10Z

run p0

github-actions · 2024-12-11T02:21:20Z

PR approved by at least one committer and no changes requested.

In the previous PR, a snapshot of the table was obtained and stored in the statementContext at the beginning of the query. The modification of this PR is to ensure that the same metadata is used during the query process. When calling the relevant interface, snapshot needs to be obtained from statementContext as a parameter and passed to the relevant method Related PR: #44911 #44673

…the latest data (apache#44911) Problem Summary: - add `PaimonMetadataCacheMgr` in `ExternalMetaCacheMgr` to manage snapshotCache of paimon table - move paimonSchemaCache to PaimonMetadataCacheMgr, and add schemaId as part of key - PaimonExternalTable overrides the methods in ExternalTable and supports partition pruning - PaimonExternalTable implements the MvcTable interface, supporting the retrieval of snapshot data from the cache during queries to avoid cache refreshes that may result in different versions of metadata being used in a single query - MTMVTask retrieves snapshot data of mvccTable before the task starts to avoid cache refresh that may result in different versions of metadata being used in a single refresh task Paimon queries the data in the cache instead of querying the latest data behavior changes of query paimon table: - FE has just started and is query the latest data - Paimon data has changed, Doris is still query the previous data - After the snapshot cache expires, Doris will query the latest data - desc paimon; The schema corresponding to the snapshotId in the snapshot cache is displayed

…44911 (#45660) pick: #44911 #43959 only pick code about paimon, not pick some code about MTMV REFRESH

…43959 apache#44911 (apache#45660) pick: apache#44911 apache#43959 only pick code about paimon, not pick some code about MTMV REFRESH

) pick: #44911 #43959 only pick code about paimon, not pick some code about MTMV REFRESH

zddr added 4 commits December 2, 2024 20:05

1

b5703b2

1

5674be5

1

6ad30e3

1

a8617f2

zddr added 6 commits December 3, 2024 18:38

1

f28e835

1

3912664

1

e45b918

1

54af29e

1

a9aca03

1

0276094

morrySnow reviewed Dec 5, 2024

View reviewed changes

zddr added 5 commits December 5, 2024 14:32

Merge branch 'master' into mv_paimon_new

e6527d8

resolve comment

17628ee

add case

cd7ebaa

add case

a7faa91

case

5e5211e

morrySnow previously approved these changes Dec 5, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 5, 2024

github-actions bot added the reviewed label Dec 5, 2024

move paimon schema cache to ExternalSchemaCache

b1da138

morningman approved these changes Dec 11, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 11, 2024

morrySnow added dev/2.1.x dev/3.0.x labels Dec 11, 2024

morrySnow merged commit 700719a into apache:master Dec 11, 2024
27 of 28 checks passed

morrySnow added the kind/behavior-changed label Dec 11, 2024

github-actions bot added dev/3.0.x-conflict dev/2.1.x-conflict labels Dec 11, 2024

zddr mentioned this pull request Dec 11, 2024

[feat](mtmv)use real snapshot instead of optional.empty() #45273

Merged

16 tasks

zddr mentioned this pull request Dec 19, 2024

[feat](mtmv)mtmv support paimon partition refresh(#43959,#44911) #45660

Merged

morrySnow pushed a commit that referenced this pull request Dec 24, 2024

branch-3.0: [feat](mtmv)mtmv support paimon partition refresh #43959 #…

f4e5727

…44911 (#45660) pick: #44911 #43959 only pick code about paimon, not pick some code about MTMV REFRESH

morrySnow added dev/3.0.4-merged and removed dev/3.0.x dev/3.0.x-conflict labels Dec 24, 2024

zddr mentioned this pull request Dec 24, 2024

[feat](mtmv)mtmv support paimon partition refresh(#43959,#44911) #45878

Merged

yiguolei pushed a commit that referenced this pull request Dec 25, 2024

[feat](mtmv)mtmv support paimon partition refresh(#43959,#44911) (#45878

fdb6519

) pick: #44911 #43959 only pick code about paimon, not pick some code about MTMV REFRESH

yiguolei added dev/2.1.8-merged and removed dev/2.1.x dev/2.1.x-conflict labels Dec 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat](mtmv)Paimon queries the data in the cache instead of querying the latest data #44911

[feat](mtmv)Paimon queries the data in the cache instead of querying the latest data #44911

zddr commented Dec 3, 2024 •

edited

Loading

doris-robot commented Dec 3, 2024

zddr commented Dec 3, 2024

doris-robot commented Dec 3, 2024

doris-robot commented Dec 3, 2024

doris-robot commented Dec 3, 2024

zddr commented Dec 5, 2024

doris-robot commented Dec 5, 2024

doris-robot commented Dec 5, 2024

doris-robot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

doris-robot commented Dec 9, 2024

zddr commented Dec 10, 2024

doris-robot commented Dec 10, 2024

doris-robot commented Dec 10, 2024

doris-robot commented Dec 10, 2024

zddr commented Dec 10, 2024

github-actions bot commented Dec 11, 2024

[feat](mtmv)Paimon queries the data in the cache instead of querying the latest data #44911

[feat](mtmv)Paimon queries the data in the cache instead of querying the latest data #44911

Conversation

zddr commented Dec 3, 2024 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented Dec 3, 2024

zddr commented Dec 3, 2024

doris-robot commented Dec 3, 2024

doris-robot commented Dec 3, 2024

doris-robot commented Dec 3, 2024

zddr commented Dec 5, 2024

doris-robot commented Dec 5, 2024

doris-robot commented Dec 5, 2024

doris-robot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

doris-robot commented Dec 9, 2024

zddr commented Dec 10, 2024

doris-robot commented Dec 10, 2024

doris-robot commented Dec 10, 2024

doris-robot commented Dec 10, 2024

zddr commented Dec 10, 2024

github-actions bot commented Dec 11, 2024

zddr commented Dec 3, 2024 •

edited

Loading