Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhance](nereids)rewrite aggregate to limit when all group by key is uniform and not null, and there is no aggregate functions #46223

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

feiniaofeiafei
Copy link
Contributor

@feiniaofeiafei feiniaofeiafei commented Dec 31, 2024

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
select 1 c1 from test group by c; -> select 1 c1 from test limit 1;
c is uniform and aggregate can be eliminated

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 31, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@feiniaofeiafei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33135 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 58827db55b38f87444c8c78d1d9991fcc8bb58c9, data reload: false

------ Round 1 ----------------------------------
q1	17596	6270	6170	6170
q2	2057	308	166	166
q3	10411	1287	735	735
q4	10225	886	439	439
q5	7519	2268	2055	2055
q6	213	184	153	153
q7	937	750	625	625
q8	9244	1514	1315	1315
q9	5299	4997	5158	4997
q10	6801	2336	1900	1900
q11	491	284	263	263
q12	352	395	224	224
q13	17807	3617	2952	2952
q14	239	230	211	211
q15	562	507	503	503
q16	619	631	593	593
q17	592	887	345	345
q18	7306	6500	6370	6370
q19	1996	1004	583	583
q20	312	328	186	186
q21	2916	2194	2040	2040
q22	368	334	310	310
Total cold run time: 103862 ms
Total hot run time: 33135 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6423	6474	6457	6457
q2	244	326	230	230
q3	2285	2647	2282	2282
q4	1500	1855	1359	1359
q5	4357	4799	5089	4799
q6	196	173	140	140
q7	2151	2008	1831	1831
q8	2705	2842	2808	2808
q9	7266	7226	7285	7226
q10	3066	3333	2832	2832
q11	607	532	505	505
q12	669	760	612	612
q13	3368	3778	3061	3061
q14	282	311	291	291
q15	602	506	500	500
q16	673	687	657	657
q17	1269	1731	1252	1252
q18	7766	7505	7146	7146
q19	855	941	1258	941
q20	1917	1984	1821	1821
q21	5508	5216	4866	4866
q22	609	620	601	601
Total cold run time: 54318 ms
Total hot run time: 52217 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191379 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 58827db55b38f87444c8c78d1d9991fcc8bb58c9, data reload: false

query1	965	394	381	381
query2	6546	2357	2359	2357
query3	6706	207	214	207
query4	33362	23536	23347	23347
query5	4334	607	467	467
query6	287	206	198	198
query7	4624	488	300	300
query8	304	250	251	250
query9	9449	2728	2722	2722
query10	456	322	264	264
query11	18126	15450	15342	15342
query12	151	110	105	105
query13	1675	536	396	396
query14	10139	7731	7245	7245
query15	289	198	187	187
query16	8121	565	445	445
query17	1626	751	568	568
query18	1331	370	286	286
query19	213	173	162	162
query20	126	139	112	112
query21	202	117	106	106
query22	4506	4497	4398	4398
query23	34258	33695	33671	33671
query24	6348	2298	2249	2249
query25	496	440	404	404
query26	901	278	172	172
query27	1991	451	334	334
query28	5194	2444	2403	2403
query29	645	545	419	419
query30	235	180	150	150
query31	970	912	816	816
query32	87	59	58	58
query33	490	365	298	298
query34	758	831	509	509
query35	799	804	734	734
query36	1025	1051	965	965
query37	114	97	75	75
query38	4432	4165	4201	4165
query39	1488	1434	1455	1434
query40	203	112	98	98
query41	47	45	46	45
query42	118	99	101	99
query43	503	532	494	494
query44	1234	790	786	786
query45	183	183	174	174
query46	865	1035	634	634
query47	1931	1938	1904	1904
query48	382	407	321	321
query49	731	510	384	384
query50	624	635	387	387
query51	7131	7102	7009	7009
query52	102	97	89	89
query53	220	248	181	181
query54	480	471	404	404
query55	78	82	77	77
query56	263	252	242	242
query57	1213	1175	1145	1145
query58	240	224	227	224
query59	2993	3384	3246	3246
query60	264	262	252	252
query61	118	113	121	113
query62	890	799	776	776
query63	224	189	190	189
query64	3713	1004	652	652
query65	3288	3187	3216	3187
query66	945	422	305	305
query67	15917	15857	15577	15577
query68	9143	746	517	517
query69	471	291	251	251
query70	1229	1178	1100	1100
query71	427	281	256	256
query72	5923	3817	3901	3817
query73	664	752	355	355
query74	10252	9363	9279	9279
query75	4416	3174	2646	2646
query76	4098	1289	761	761
query77	773	365	281	281
query78	10162	10168	9377	9377
query79	3288	898	580	580
query80	728	524	442	442
query81	499	265	232	232
query82	660	150	125	125
query83	166	167	145	145
query84	238	90	74	74
query85	804	436	301	301
query86	405	310	273	273
query87	4417	4425	4430	4425
query88	4802	2229	2197	2197
query89	420	339	300	300
query90	1836	187	187	187
query91	136	150	104	104
query92	70	58	53	53
query93	1735	877	517	517
query94	665	392	288	288
query95	341	262	250	250
query96	480	611	285	285
query97	2764	2806	2661	2661
query98	233	200	197	197
query99	1643	1562	1408	1408
Total cold run time: 292557 ms
Total hot run time: 191379 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 58827db55b38f87444c8c78d1d9991fcc8bb58c9, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.07	0.07
query4	1.61	0.11	0.10
query5	0.42	0.42	0.39
query6	1.14	0.64	0.64
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.55	0.51	0.51
query10	0.55	0.53	0.55
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.62
query14	2.81	2.78	2.72
query15	0.88	0.83	0.82
query16	0.37	0.38	0.37
query17	1.02	1.05	1.02
query18	0.23	0.20	0.20
query19	1.96	1.80	2.03
query20	0.01	0.02	0.01
query21	15.37	0.91	0.57
query22	0.78	0.72	0.66
query23	15.33	1.45	0.55
query24	3.76	1.38	0.97
query25	0.16	0.18	0.12
query26	0.25	0.16	0.14
query27	0.07	0.05	0.04
query28	13.64	1.57	1.04
query29	12.58	3.94	3.28
query30	0.25	0.09	0.06
query31	2.83	0.60	0.39
query32	3.22	0.54	0.47
query33	3.06	3.05	3.16
query34	16.70	5.11	4.52
query35	4.59	4.50	4.52
query36	0.64	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.13
query41	0.07	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.57 s
Total hot run time: 31.27 s

@feiniaofeiafei feiniaofeiafei force-pushed the agg_to_limit branch 2 times, most recently from 533d8b5 to 9a61323 Compare January 2, 2025 04:03
@feiniaofeiafei
Copy link
Contributor Author

run buildall

1 similar comment
@feiniaofeiafei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32695 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9a613239a96f40f43ffea9cf4d355c02bc79de01, data reload: false

------ Round 1 ----------------------------------
q1	17595	6118	6062	6062
q2	2046	298	175	175
q3	10424	1244	766	766
q4	10516	864	433	433
q5	9236	2205	1963	1963
q6	212	186	149	149
q7	885	756	597	597
q8	9238	1355	1168	1168
q9	5249	4909	4960	4909
q10	6766	2312	1860	1860
q11	492	282	255	255
q12	348	355	222	222
q13	17767	3587	3018	3018
q14	244	241	228	228
q15	567	507	501	501
q16	648	614	595	595
q17	581	857	355	355
q18	6944	6438	6406	6406
q19	1569	964	545	545
q20	315	315	189	189
q21	2912	2179	1999	1999
q22	366	342	300	300
Total cold run time: 104920 ms
Total hot run time: 32695 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6234	6256	6200	6200
q2	241	323	226	226
q3	2269	2643	2327	2327
q4	1450	1829	1343	1343
q5	4402	4758	4862	4758
q6	192	195	145	145
q7	2051	1954	1814	1814
q8	2666	2853	2705	2705
q9	7284	7254	7320	7254
q10	3083	3366	2764	2764
q11	565	519	492	492
q12	676	813	634	634
q13	3358	3725	3073	3073
q14	278	321	269	269
q15	581	535	497	497
q16	681	694	653	653
q17	1249	1728	1288	1288
q18	7654	7386	7329	7329
q19	858	1225	1065	1065
q20	2010	2019	1902	1902
q21	5869	5357	4900	4900
q22	646	632	605	605
Total cold run time: 54297 ms
Total hot run time: 52243 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196823 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9a613239a96f40f43ffea9cf4d355c02bc79de01, data reload: false

query1	1281	957	943	943
query2	6485	2378	2365	2365
query3	11033	4644	4583	4583
query4	32984	23676	23454	23454
query5	4554	620	455	455
query6	278	203	176	176
query7	3981	497	308	308
query8	297	236	244	236
query9	9230	2631	2617	2617
query10	468	311	254	254
query11	17972	15475	15284	15284
query12	165	117	108	108
query13	1577	543	401	401
query14	9069	6897	8303	6897
query15	265	210	206	206
query16	8609	628	469	469
query17	1635	771	616	616
query18	2151	416	325	325
query19	201	184	166	166
query20	117	129	129	129
query21	209	129	111	111
query22	4485	4579	4623	4579
query23	34252	33527	33810	33527
query24	6466	2297	2334	2297
query25	492	468	394	394
query26	811	278	163	163
query27	2149	471	344	344
query28	5681	2439	2429	2429
query29	671	590	432	432
query30	209	184	152	152
query31	989	898	858	858
query32	87	62	66	62
query33	475	347	324	324
query34	788	864	536	536
query35	828	835	736	736
query36	1029	1077	984	984
query37	123	106	79	79
query38	4388	4409	4154	4154
query39	1527	1476	1472	1472
query40	221	119	101	101
query41	46	47	43	43
query42	123	107	111	107
query43	534	542	507	507
query44	1367	836	836	836
query45	187	187	172	172
query46	906	1066	649	649
query47	2012	1996	1951	1951
query48	399	433	325	325
query49	715	484	396	396
query50	655	688	399	399
query51	7241	7156	7162	7156
query52	103	109	97	97
query53	237	266	192	192
query54	488	507	416	416
query55	82	79	80	79
query56	263	264	252	252
query57	1279	1232	1223	1223
query58	246	235	231	231
query59	3222	3307	3310	3307
query60	282	268	267	267
query61	110	107	106	106
query62	884	798	774	774
query63	230	196	196	196
query64	3286	1039	665	665
query65	3309	3285	3306	3285
query66	921	420	308	308
query67	16654	15738	15511	15511
query68	9699	776	527	527
query69	476	295	249	249
query70	1192	1148	1151	1148
query71	452	320	258	258
query72	5997	3808	3801	3801
query73	668	754	368	368
query74	10339	9208	8978	8978
query75	4592	3105	2672	2672
query76	4500	1193	775	775
query77	915	445	285	285
query78	10307	10539	9363	9363
query79	5415	871	593	593
query80	724	517	436	436
query81	476	270	228	228
query82	320	149	122	122
query83	191	160	149	149
query84	281	89	75	75
query85	804	357	302	302
query86	357	317	281	281
query87	4644	4710	4508	4508
query88	3895	2254	2172	2172
query89	436	328	303	303
query90	2119	184	185	184
query91	127	137	103	103
query92	67	54	50	50
query93	3381	878	537	537
query94	671	380	293	293
query95	352	261	241	241
query96	502	612	273	273
query97	2779	2824	2728	2728
query98	232	208	204	204
query99	1645	1578	1441	1441
Total cold run time: 302568 ms
Total hot run time: 196823 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9a613239a96f40f43ffea9cf4d355c02bc79de01, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.04	0.03
query3	0.24	0.08	0.07
query4	1.60	0.11	0.11
query5	0.43	0.41	0.41
query6	1.17	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.50	0.54
query10	0.56	0.57	0.54
query11	0.15	0.10	0.10
query12	0.14	0.12	0.11
query13	0.60	0.59	0.59
query14	2.81	2.83	2.75
query15	0.89	0.81	0.82
query16	0.39	0.38	0.38
query17	1.06	1.06	0.99
query18	0.24	0.20	0.21
query19	1.94	1.89	2.03
query20	0.01	0.00	0.01
query21	15.36	0.90	0.57
query22	0.76	0.99	0.64
query23	15.17	1.43	0.57
query24	3.25	0.91	1.96
query25	0.20	0.26	0.11
query26	0.24	0.14	0.14
query27	0.05	0.07	0.04
query28	13.70	1.51	1.05
query29	12.55	3.96	3.23
query30	0.25	0.09	0.08
query31	2.81	0.59	0.40
query32	3.23	0.54	0.47
query33	3.10	3.08	3.08
query34	16.63	5.09	4.50
query35	4.45	4.43	4.45
query36	0.64	0.48	0.47
query37	0.10	0.07	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.86 s
Total hot run time: 31.21 s

@feiniaofeiafei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32273 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dac6f4ddec22f1c892f0263487d0d1a06986664c, data reload: false

------ Round 1 ----------------------------------
q1	17596	5504	5364	5364
q2	2048	303	172	172
q3	10409	1243	702	702
q4	10230	944	515	515
q5	7544	2377	2139	2139
q6	197	165	135	135
q7	902	757	612	612
q8	9230	1396	1209	1209
q9	5261	4976	4961	4961
q10	6838	2342	1875	1875
q11	478	277	256	256
q12	334	362	219	219
q13	17767	3675	3056	3056
q14	226	223	214	214
q15	513	471	470	470
q16	631	602	593	593
q17	553	855	323	323
q18	7022	6507	6452	6452
q19	1218	941	525	525
q20	308	332	200	200
q21	2768	2133	1958	1958
q22	361	323	333	323
Total cold run time: 102434 ms
Total hot run time: 32273 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5523	5454	5437	5437
q2	246	338	235	235
q3	2291	2674	2321	2321
q4	1383	1853	1379	1379
q5	4287	4729	4676	4676
q6	164	156	127	127
q7	2243	2006	1835	1835
q8	2664	2788	2667	2667
q9	7287	7243	7192	7192
q10	3039	3291	2769	2769
q11	581	501	503	501
q12	700	748	608	608
q13	3477	3843	3381	3381
q14	271	286	268	268
q15	518	482	465	465
q16	636	706	643	643
q17	1198	1727	1248	1248
q18	7537	7386	7527	7386
q19	740	964	1112	964
q20	1997	2032	1921	1921
q21	5608	5299	4889	4889
q22	585	603	555	555
Total cold run time: 52975 ms
Total hot run time: 51467 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188164 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dac6f4ddec22f1c892f0263487d0d1a06986664c, data reload: false

query1	997	383	371	371
query2	6515	2080	2026	2026
query3	6786	222	225	222
query4	33513	23156	23123	23123
query5	4363	618	472	472
query6	299	200	179	179
query7	4607	497	306	306
query8	293	238	233	233
query9	9427	2650	2643	2643
query10	450	321	256	256
query11	18271	15342	15193	15193
query12	150	109	103	103
query13	1631	504	391	391
query14	9350	6873	6914	6873
query15	209	186	185	185
query16	7902	609	415	415
query17	1597	738	544	544
query18	2046	386	296	296
query19	224	182	154	154
query20	116	109	105	105
query21	217	118	99	99
query22	4601	4623	4498	4498
query23	34074	33195	33077	33077
query24	6925	2296	2323	2296
query25	482	442	384	384
query26	1210	271	154	154
query27	2021	457	366	366
query28	5352	2429	2402	2402
query29	674	536	411	411
query30	238	186	160	160
query31	967	858	782	782
query32	69	64	59	59
query33	515	349	297	297
query34	723	851	505	505
query35	804	808	735	735
query36	1002	1046	971	971
query37	118	101	81	81
query38	4196	4111	4144	4111
query39	1440	1431	1379	1379
query40	209	117	100	100
query41	54	52	49	49
query42	118	100	101	100
query43	509	515	492	492
query44	1363	820	820	820
query45	179	170	164	164
query46	856	1043	635	635
query47	1850	1839	1774	1774
query48	381	388	317	317
query49	785	484	407	407
query50	621	657	372	372
query51	6866	6979	6673	6673
query52	104	101	92	92
query53	227	246	182	182
query54	481	494	399	399
query55	90	75	76	75
query56	256	251	252	251
query57	1199	1184	1079	1079
query58	247	247	245	245
query59	3025	3108	3065	3065
query60	274	271	250	250
query61	122	118	117	117
query62	813	731	630	630
query63	228	193	187	187
query64	4168	1092	721	721
query65	3222	3165	3166	3165
query66	1071	434	305	305
query67	15881	15609	15560	15560
query68	2483	818	551	551
query69	448	287	255	255
query70	1200	1100	1119	1100
query71	324	287	246	246
query72	5799	3809	3922	3809
query73	623	763	363	363
query74	10000	8907	8783	8783
query75	3118	3145	2687	2687
query76	2565	1189	757	757
query77	470	354	273	273
query78	10159	10319	9329	9329
query79	928	967	599	599
query80	1299	538	431	431
query81	555	274	242	242
query82	399	158	122	122
query83	325	190	154	154
query84	239	97	69	69
query85	1045	366	291	291
query86	398	308	301	301
query87	4358	4453	4297	4297
query88	3424	2206	2184	2184
query89	385	337	297	297
query90	1602	190	190	190
query91	133	142	108	108
query92	62	61	54	54
query93	925	865	531	531
query94	663	380	288	288
query95	341	269	248	248
query96	497	603	290	290
query97	2767	2828	2787	2787
query98	218	205	199	199
query99	1277	1398	1246	1246
Total cold run time: 278602 ms
Total hot run time: 188164 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.16 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dac6f4ddec22f1c892f0263487d0d1a06986664c, data reload: false

query1	0.03	0.03	0.02
query2	0.08	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.11
query5	0.43	0.41	0.43
query6	1.15	0.66	0.65
query7	0.03	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.50
query10	0.56	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.60	0.62	0.60
query14	2.73	2.85	2.74
query15	0.90	0.82	0.84
query16	0.39	0.39	0.37
query17	1.04	1.04	0.99
query18	0.22	0.21	0.22
query19	1.90	1.90	2.02
query20	0.01	0.01	0.01
query21	15.37	0.94	0.57
query22	0.76	0.96	0.84
query23	14.97	1.34	0.59
query24	3.10	1.17	1.17
query25	0.27	0.18	0.16
query26	0.20	0.14	0.13
query27	0.06	0.07	0.06
query28	14.30	1.03	0.43
query29	12.59	3.99	3.28
query30	0.25	0.09	0.06
query31	2.83	0.62	0.38
query32	3.22	0.54	0.46
query33	2.96	2.99	2.99
query34	16.45	5.15	4.51
query35	4.57	4.52	4.51
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.05	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.91 s
Total hot run time: 31.16 s

@feiniaofeiafei
Copy link
Contributor Author

run p0

@feiniaofeiafei
Copy link
Contributor Author

run external

@feiniaofeiafei
Copy link
Contributor Author

run p0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants