Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] (nereids)implement DropStoragePolicyCommand in nereids #44825

Merged
merged 1 commit into from
Dec 24, 2024

Conversation

msridhar78
Copy link
Contributor

@msridhar78 msridhar78 commented Dec 1, 2024

What problem does this PR solve?

Issue Number: close #42626

Related PR: #xxx

Problem Summary:
[Enhancement] (nereids)implement DropStoragePolicyCommand in nereids

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@msridhar78
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39995 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed0381532dfc0bdff1e83a04f27cac8232e05639, data reload: false

------ Round 1 ----------------------------------
q1	17607	7435	7271	7271
q2	2049	182	171	171
q3	10697	1054	1208	1054
q4	10522	727	662	662
q5	7622	2669	2676	2669
q6	239	152	150	150
q7	987	634	596	596
q8	9256	1836	1949	1836
q9	6614	6442	6486	6442
q10	6995	2336	2309	2309
q11	463	267	263	263
q12	423	225	223	223
q13	17780	3054	3039	3039
q14	254	216	228	216
q15	576	540	526	526
q16	668	606	586	586
q17	974	578	529	529
q18	7175	6704	6719	6704
q19	1363	1002	1025	1002
q20	458	184	179	179
q21	4160	3250	3268	3250
q22	385	327	318	318
Total cold run time: 107267 ms
Total hot run time: 39995 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7253	7279	7233	7233
q2	331	245	237	237
q3	2876	2790	2946	2790
q4	2131	1835	1828	1828
q5	5652	5687	5607	5607
q6	222	142	140	140
q7	2286	1826	1821	1821
q8	3424	3557	3486	3486
q9	8959	9128	9102	9102
q10	3620	3531	3578	3531
q11	607	514	504	504
q12	803	608	608	608
q13	10668	3211	3238	3211
q14	297	280	266	266
q15	589	526	535	526
q16	688	632	639	632
q17	1860	1607	1654	1607
q18	8205	7774	7528	7528
q19	1703	1614	1469	1469
q20	2131	1925	1853	1853
q21	5556	5568	5457	5457
q22	670	577	562	562
Total cold run time: 70531 ms
Total hot run time: 59998 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196381 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed0381532dfc0bdff1e83a04f27cac8232e05639, data reload: false

query1	1237	955	921	921
query2	6248	2080	2114	2080
query3	10981	4292	4397	4292
query4	66892	28836	23451	23451
query5	4947	456	461	456
query6	402	188	176	176
query7	5533	304	295	295
query8	319	238	248	238
query9	8791	2701	2665	2665
query10	428	253	250	250
query11	17258	15328	15831	15328
query12	155	99	101	99
query13	1448	440	404	404
query14	10667	7298	6746	6746
query15	217	186	197	186
query16	7103	482	473	473
query17	1158	593	602	593
query18	1809	334	318	318
query19	203	169	157	157
query20	131	118	111	111
query21	210	112	114	112
query22	4813	4643	4511	4511
query23	34675	34506	34392	34392
query24	5591	2497	2425	2425
query25	482	391	389	389
query26	656	162	153	153
query27	1847	283	289	283
query28	4343	2463	2442	2442
query29	674	462	447	447
query30	211	151	148	148
query31	1001	836	814	814
query32	69	55	55	55
query33	411	299	288	288
query34	926	515	523	515
query35	879	755	784	755
query36	1093	962	983	962
query37	128	76	68	68
query38	4461	4357	4443	4357
query39	1556	1455	1490	1455
query40	196	96	95	95
query41	42	41	41	41
query42	115	100	105	100
query43	553	481	498	481
query44	1186	821	801	801
query45	184	168	169	168
query46	1181	718	712	712
query47	2039	1958	1960	1958
query48	422	305	309	305
query49	724	397	432	397
query50	840	396	386	386
query51	7413	7116	7066	7066
query52	97	90	84	84
query53	256	173	177	173
query54	495	396	393	393
query55	76	71	74	71
query56	247	233	235	233
query57	1274	1154	1134	1134
query58	209	210	205	205
query59	3348	3117	3007	3007
query60	258	242	251	242
query61	110	105	103	103
query62	801	679	680	679
query63	213	187	193	187
query64	1346	684	650	650
query65	3318	3215	3181	3181
query66	701	304	353	304
query67	15932	15538	15662	15538
query68	4271	559	539	539
query69	424	249	247	247
query70	1213	1151	1123	1123
query71	371	252	250	250
query72	6387	4037	4066	4037
query73	750	347	357	347
query74	10151	8899	8962	8899
query75	3394	2650	2661	2650
query76	1816	973	996	973
query77	499	264	288	264
query78	10592	9516	9438	9438
query79	2290	604	624	604
query80	1393	415	422	415
query81	514	235	235	235
query82	1284	121	120	120
query83	254	144	196	144
query84	289	68	69	68
query85	986	302	293	293
query86	418	308	304	304
query87	4778	4603	4577	4577
query88	3793	2204	2182	2182
query89	414	302	285	285
query90	1955	191	192	191
query91	134	105	104	104
query92	66	47	49	47
query93	2942	551	525	525
query94	878	293	287	287
query95	352	245	242	242
query96	624	285	279	279
query97	2848	2717	2703	2703
query98	225	195	194	194
query99	1630	1327	1311	1311
Total cold run time: 321464 ms
Total hot run time: 196381 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.62 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ed0381532dfc0bdff1e83a04f27cac8232e05639, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.03	0.03
query3	0.22	0.07	0.07
query4	1.61	0.10	0.11
query5	0.41	0.40	0.38
query6	1.14	0.67	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.51	0.49
query10	0.54	0.56	0.55
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.60
query14	2.70	2.72	2.89
query15	0.90	0.82	0.85
query16	0.39	0.39	0.39
query17	1.05	1.00	1.00
query18	0.23	0.21	0.21
query19	1.91	1.86	1.90
query20	0.02	0.02	0.01
query21	15.36	0.59	0.59
query22	2.98	2.07	1.93
query23	17.15	0.88	0.80
query24	2.72	0.98	2.10
query25	0.30	0.10	0.16
query26	0.42	0.15	0.13
query27	0.05	0.05	0.05
query28	10.20	1.10	1.08
query29	12.53	3.27	3.19
query30	0.25	0.07	0.06
query31	2.86	0.38	0.37
query32	3.27	0.45	0.47
query33	3.00	3.04	3.12
query34	16.97	4.51	4.50
query35	4.55	4.42	4.49
query36	0.68	0.49	0.47
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.17	0.12	0.12
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.51 s
Total hot run time: 32.62 s

@LiBinfeng-01
Copy link
Collaborator

we can add some cases to make sure implement is right

@msridhar78
Copy link
Contributor Author

we can add some cases to make sure implement is right

done

@msridhar78
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40584 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a82b7baf4b2df3ff2267b77d3db1d0816aa6df33, data reload: false

------ Round 1 ----------------------------------
q1	18835	7468	7391	7391
q2	2050	175	165	165
q3	10973	1171	1236	1171
q4	10819	710	780	710
q5	7747	2691	2757	2691
q6	244	148	148	148
q7	1015	631	617	617
q8	9909	1865	1970	1865
q9	6940	6581	6679	6581
q10	7126	2312	2365	2312
q11	480	273	276	273
q12	477	235	231	231
q13	18267	3040	3036	3036
q14	262	223	214	214
q15	563	508	498	498
q16	691	587	589	587
q17	987	614	567	567
q18	7292	6893	6900	6893
q19	1390	991	836	836
q20	482	189	194	189
q21	4065	3388	3286	3286
q22	382	329	323	323
Total cold run time: 110996 ms
Total hot run time: 40584 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7408	7292	7312	7292
q2	325	229	233	229
q3	3069	2953	2958	2953
q4	2141	1820	1827	1820
q5	5609	5680	5648	5648
q6	235	146	143	143
q7	2232	1825	1802	1802
q8	3363	3501	3389	3389
q9	8952	8973	8899	8899
q10	3620	3594	3569	3569
q11	597	492	494	492
q12	822	662	578	578
q13	11264	3086	3085	3085
q14	311	283	268	268
q15	561	511	511	511
q16	680	669	645	645
q17	1844	1606	1584	1584
q18	8419	7713	7446	7446
q19	1727	1640	1563	1563
q20	2129	1957	1938	1938
q21	5655	5528	5414	5414
q22	652	567	562	562
Total cold run time: 71615 ms
Total hot run time: 59830 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196482 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a82b7baf4b2df3ff2267b77d3db1d0816aa6df33, data reload: false

query1	1382	972	1025	972
query2	6238	2321	2296	2296
query3	11085	4676	4792	4676
query4	33441	23297	23568	23297
query5	4849	473	454	454
query6	285	190	186	186
query7	4005	306	306	306
query8	304	240	251	240
query9	9523	2711	2686	2686
query10	492	240	260	240
query11	17959	15010	15049	15010
query12	153	105	99	99
query13	1572	419	413	413
query14	9320	7635	7470	7470
query15	276	193	194	193
query16	8015	505	475	475
query17	1592	605	616	605
query18	2174	314	328	314
query19	374	169	170	169
query20	133	123	117	117
query21	211	108	110	108
query22	4969	4555	4444	4444
query23	34557	33946	33661	33661
query24	10115	2540	2509	2509
query25	678	379	409	379
query26	725	159	155	155
query27	2145	342	342	342
query28	6867	2489	2477	2477
query29	749	434	431	431
query30	227	151	147	147
query31	1050	869	860	860
query32	80	56	54	54
query33	683	287	286	286
query34	968	507	527	507
query35	914	763	768	763
query36	1109	972	993	972
query37	123	73	76	73
query38	4287	4117	4178	4117
query39	1499	1452	1438	1438
query40	207	105	102	102
query41	53	43	47	43
query42	119	102	110	102
query43	534	511	497	497
query44	1218	853	848	848
query45	183	173	172	172
query46	1188	711	727	711
query47	2027	1927	1933	1927
query48	421	320	319	319
query49	898	385	407	385
query50	836	400	403	400
query51	7388	7144	7349	7144
query52	100	94	92	92
query53	267	187	186	186
query54	1128	410	421	410
query55	82	81	85	81
query56	265	244	251	244
query57	1279	1132	1158	1132
query58	229	221	240	221
query59	3474	3124	3189	3124
query60	289	248	251	248
query61	113	104	106	104
query62	894	686	670	670
query63	213	184	194	184
query64	3682	681	629	629
query65	3299	3221	3185	3185
query66	805	319	327	319
query67	15987	15601	15438	15438
query68	5172	564	551	551
query69	484	245	246	245
query70	1225	1140	1097	1097
query71	476	249	248	248
query72	6879	4189	4027	4027
query73	788	355	373	355
query74	10079	8883	9002	8883
query75	3753	2611	2627	2611
query76	3717	1026	1117	1026
query77	600	288	304	288
query78	10240	9493	9387	9387
query79	2509	623	625	623
query80	1397	421	422	421
query81	537	248	222	222
query82	1341	116	116	116
query83	240	144	139	139
query84	287	67	71	67
query85	1408	302	300	300
query86	456	298	303	298
query87	4722	4394	4414	4394
query88	3783	2217	2173	2173
query89	430	285	288	285
query90	2045	190	185	185
query91	137	103	107	103
query92	63	49	53	49
query93	2849	541	542	541
query94	889	293	311	293
query95	355	243	270	243
query96	629	276	276	276
query97	2869	2680	2722	2680
query98	227	200	190	190
query99	1666	1332	1308	1308
Total cold run time: 304558 ms
Total hot run time: 196482 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.15 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a82b7baf4b2df3ff2267b77d3db1d0816aa6df33, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.06	0.08
query4	1.61	0.10	0.10
query5	0.43	0.40	0.40
query6	1.22	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.51
query10	0.54	0.58	0.56
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.61
query14	2.83	2.75	2.73
query15	0.90	0.82	0.82
query16	0.38	0.39	0.38
query17	1.02	1.04	1.05
query18	0.22	0.21	0.20
query19	1.94	1.85	2.01
query20	0.02	0.01	0.01
query21	15.37	0.58	0.59
query22	2.97	1.79	1.79
query23	16.95	0.99	0.81
query24	3.15	1.35	0.55
query25	0.25	0.09	0.23
query26	0.37	0.14	0.14
query27	0.06	0.04	0.04
query28	10.67	1.10	1.07
query29	12.63	3.21	3.21
query30	0.25	0.06	0.07
query31	2.85	0.38	0.38
query32	3.26	0.46	0.45
query33	3.08	3.07	3.10
query34	16.88	4.47	4.53
query35	4.54	4.47	4.49
query36	0.68	0.49	0.47
query37	0.09	0.05	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.12	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 107.41 s
Total hot run time: 32.15 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 23, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@starocean999 starocean999 merged commit d0f09fb into apache:master Dec 24, 2024
24 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Enhancement] (nereids)implement DropStoragePolicyCommand in nereids
4 participants