Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [opt](cloud) reduce cache hotspot table write amplification #45557 #45677

Merged
merged 1 commit into from
Dec 20, 2024

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #45557

1. batch insert cloud_cache_hotspot in FE
2. enlarge polling interval in FE
3. shrink bucket num to 1 for cloud_cache_hotspot table
4. ignore stable statistics only catch the dynamic in BE

Signed-off-by: zhengyu <[email protected]>
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Dec 19, 2024
@hello-stephen
Copy link
Contributor

run buildall

Copy link
Contributor Author

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40049 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 708df297eb3ec20f8ef33f743d41df1154436c91, data reload: false

------ Round 1 ----------------------------------
q1	17568	7331	7206	7206
q2	2047	164	165	164
q3	10746	1835	1129	1129
q4	10977	773	679	679
q5	7759	2810	2692	2692
q6	234	146	147	146
q7	965	615	596	596
q8	9578	1897	1941	1897
q9	8018	6329	6375	6329
q10	7028	2349	2335	2335
q11	470	274	266	266
q12	405	223	216	216
q13	17886	2949	2978	2949
q14	237	218	206	206
q15	545	521	513	513
q16	688	620	612	612
q17	940	523	552	523
q18	7101	6438	6524	6438
q19	1888	1046	1070	1046
q20	455	203	191	191
q21	3878	3320	2960	2960
q22	1064	956	977	956
Total cold run time: 110477 ms
Total hot run time: 40049 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7274	7165	7172	7165
q2	323	237	235	235
q3	2923	2801	2823	2801
q4	1990	1829	1815	1815
q5	5654	5680	5672	5672
q6	225	140	141	140
q7	2158	1749	1767	1749
q8	3329	3509	3351	3351
q9	8808	8758	8754	8754
q10	3525	3468	3484	3468
q11	589	490	496	490
q12	820	604	627	604
q13	16504	3131	3078	3078
q14	295	268	270	268
q15	552	514	507	507
q16	711	672	695	672
q17	1819	1645	1610	1610
q18	8184	7705	7503	7503
q19	3637	1643	1460	1460
q20	2042	1862	1838	1838
q21	5163	5260	5236	5236
q22	1104	1019	1023	1019
Total cold run time: 77629 ms
Total hot run time: 59435 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195466 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 708df297eb3ec20f8ef33f743d41df1154436c91, data reload: false

query1	1271	927	908	908
query2	6235	2073	2072	2072
query3	10835	4050	4103	4050
query4	65707	29474	23546	23546
query5	5430	445	439	439
query6	472	173	178	173
query7	5670	310	310	310
query8	316	231	230	230
query9	9290	2683	2672	2672
query10	494	275	265	265
query11	17708	15117	15640	15117
query12	169	97	97	97
query13	1558	440	416	416
query14	10809	7096	7297	7096
query15	211	184	180	180
query16	7342	518	495	495
query17	1170	578	580	578
query18	1880	336	318	318
query19	221	159	152	152
query20	119	116	110	110
query21	71	46	45	45
query22	4724	4476	4326	4326
query23	34924	34031	33948	33948
query24	6105	2883	2987	2883
query25	519	432	422	422
query26	689	179	172	172
query27	1782	310	298	298
query28	4566	2557	2543	2543
query29	699	467	458	458
query30	247	166	179	166
query31	1015	816	816	816
query32	70	54	60	54
query33	449	291	303	291
query34	927	512	512	512
query35	815	759	760	759
query36	1085	934	933	933
query37	118	73	72	72
query38	4069	4002	3957	3957
query39	1506	1451	1469	1451
query40	141	79	78	78
query41	49	45	45	45
query42	112	96	96	96
query43	514	486	488	486
query44	1120	792	792	792
query45	184	168	173	168
query46	1142	715	716	715
query47	1965	1916	1904	1904
query48	486	362	366	362
query49	731	382	368	368
query50	827	414	416	414
query51	7129	7072	7094	7072
query52	96	90	85	85
query53	272	185	183	183
query54	556	453	439	439
query55	79	72	72	72
query56	263	227	243	227
query57	1153	1069	1092	1069
query58	199	207	212	207
query59	2973	2976	2809	2809
query60	270	253	250	250
query61	109	108	108	108
query62	757	646	660	646
query63	214	196	188	188
query64	1781	667	626	626
query65	3240	3151	3159	3151
query66	640	280	313	280
query67	15663	15270	15260	15260
query68	4459	563	539	539
query69	411	249	255	249
query70	1154	1093	1152	1093
query71	393	251	268	251
query72	6556	3994	3929	3929
query73	754	338	339	338
query74	10228	8913	8909	8909
query75	3334	2595	2661	2595
query76	2411	1003	1090	1003
query77	526	257	269	257
query78	10749	9720	9484	9484
query79	8225	604	600	600
query80	2138	464	421	421
query81	552	249	246	246
query82	1321	128	117	117
query83	259	145	150	145
query84	280	80	74	74
query85	1731	308	296	296
query86	467	318	305	305
query87	4505	4263	4242	4242
query88	5575	2369	2388	2369
query89	423	284	289	284
query90	2113	179	182	179
query91	182	143	144	143
query92	65	49	50	49
query93	6674	528	532	528
query94	944	275	308	275
query95	351	256	252	252
query96	631	272	282	272
query97	3280	3121	3101	3101
query98	217	199	206	199
query99	1623	1294	1292	1292
Total cold run time: 336483 ms
Total hot run time: 195466 ms

@dataroaring dataroaring merged commit d38f410 into branch-3.0 Dec 20, 2024
21 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-45557-branch-3.0 branch December 20, 2024 07:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants