Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](routine load) reduce routine load task consume log (#42058) #42293

Merged
merged 1 commit into from
Oct 23, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Oct 22, 2024

pick (#42058)

There is too much routine load task log, five million logs were generated in 10 minutes.

grep 'consumer meet partition eof' be.INFO.log.20240930-164533 | wc -l 
5369624

There is too much routine load task log, five million logs were
generated in 10 minutes.

```
grep 'consumer meet partition eof' be.INFO.log.20240930-164533 | wc -l 
5369624
```
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Contributor Author

sollhui commented Oct 22, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40984 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd4125718a9d6b346322fa4b342a8b31ee0f887f, data reload: false

------ Round 1 ----------------------------------
q1	17577	7447	7337	7337
q2	2010	269	271	269
q3	12209	1376	1192	1192
q4	10886	732	758	732
q5	7763	2827	2824	2824
q6	239	153	157	153
q7	972	623	628	623
q8	9598	1906	1830	1830
q9	8058	6401	6389	6389
q10	7020	2292	2324	2292
q11	444	239	249	239
q12	410	232	221	221
q13	17767	2977	2974	2974
q14	232	206	214	206
q15	560	517	525	517
q16	679	607	611	607
q17	971	547	532	532
q18	7166	6501	6536	6501
q19	1892	1072	1064	1064
q20	549	268	271	268
q21	4007	3239	3242	3239
q22	1098	1016	975	975
Total cold run time: 112107 ms
Total hot run time: 40984 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7427	7291	7477	7291
q2	336	236	233	233
q3	3062	2820	2828	2820
q4	2048	1814	1774	1774
q5	5661	5707	5734	5707
q6	229	142	147	142
q7	2175	1813	1766	1766
q8	3345	3473	3382	3382
q9	8928	8914	8800	8800
q10	3529	3548	3511	3511
q11	581	490	483	483
q12	814	641	609	609
q13	16567	3138	3146	3138
q14	315	278	280	278
q15	562	519	527	519
q16	690	669	673	669
q17	1846	1609	1590	1590
q18	8095	7677	7549	7549
q19	3056	1531	1332	1332
q20	2126	1822	1829	1822
q21	5467	5326	5196	5196
q22	1120	1052	1000	1000
Total cold run time: 77979 ms
Total hot run time: 59611 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188907 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cd4125718a9d6b346322fa4b342a8b31ee0f887f, data reload: false

query1	863	392	376	376
query2	6246	2126	1983	1983
query3	8710	200	209	200
query4	33818	23413	23337	23337
query5	3720	463	460	460
query6	293	159	154	154
query7	4191	313	301	301
query8	263	211	229	211
query9	9579	2642	2644	2642
query10	434	284	315	284
query11	17840	15128	15095	15095
query12	145	101	98	98
query13	1530	444	404	404
query14	9319	7286	7192	7192
query15	231	175	164	164
query16	7915	469	506	469
query17	1576	600	580	580
query18	2089	339	326	326
query19	231	161	160	160
query20	127	130	123	123
query21	230	105	111	105
query22	4783	4415	4488	4415
query23	35006	33670	33679	33670
query24	11182	2867	2867	2867
query25	602	408	403	403
query26	728	167	171	167
query27	2062	293	304	293
query28	6501	2484	2469	2469
query29	712	428	436	428
query30	269	158	156	156
query31	1042	799	799	799
query32	93	54	56	54
query33	667	304	292	292
query34	888	515	485	485
query35	865	737	753	737
query36	1107	948	937	937
query37	140	89	86	86
query38	4028	3899	3911	3899
query39	1487	1415	1395	1395
query40	203	104	99	99
query41	48	47	47	47
query42	124	97	94	94
query43	527	487	496	487
query44	1115	772	781	772
query45	193	166	167	166
query46	1151	725	726	725
query47	1910	1833	1809	1809
query48	458	372	374	372
query49	917	400	413	400
query50	831	409	409	409
query51	7093	6845	7027	6845
query52	107	93	92	92
query53	271	184	182	182
query54	1194	475	475	475
query55	79	73	75	73
query56	285	260	265	260
query57	1180	1125	1097	1097
query58	246	238	241	238
query59	3187	2986	3082	2986
query60	300	278	301	278
query61	123	147	98	98
query62	849	653	763	653
query63	217	185	178	178
query64	3704	633	598	598
query65	3308	3219	3179	3179
query66	822	294	298	294
query67	15806	15566	15425	15425
query68	4604	557	543	543
query69	616	283	293	283
query70	1183	1117	1068	1068
query71	423	283	278	278
query72	7985	3888	2598	2598
query73	756	341	344	341
query74	10300	8770	8935	8770
query75	4548	2630	2680	2630
query76	4202	867	872	867
query77	769	291	294	291
query78	9882	9224	9122	9122
query79	2875	574	574	574
query80	1982	432	448	432
query81	586	260	245	245
query82	883	140	144	140
query83	376	136	134	134
query84	297	83	77	77
query85	2115	292	322	292
query86	474	293	311	293
query87	4307	4304	4307	4304
query88	3665	2443	2452	2443
query89	419	288	291	288
query90	2124	188	193	188
query91	173	139	141	139
query92	67	47	47	47
query93	3196	545	535	535
query94	1125	287	294	287
query95	350	248	249	248
query96	608	282	282	282
query97	3297	3192	3140	3140
query98	221	193	194	193
query99	1588	1300	1288	1288
Total cold run time: 303756 ms
Total hot run time: 188907 ms

@liaoxin01 liaoxin01 merged commit 1a30b13 into apache:branch-3.0 Oct 23, 2024
20 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants