Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement] (nereids)implement RecoverCommand in nereids #44071

Merged
merged 3 commits into from
Nov 20, 2024
Merged

[Enhancement] (nereids)implement RecoverCommand in nereids #44071

merged 3 commits into from
Nov 20, 2024

Conversation

Vallishp
Copy link
Contributor

@Vallishp Vallishp commented Nov 16, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #42550

Support recover table and recover partition command in nereids.
#42550
#42545
#42547

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • [ *] Regression test //added
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Vallishp
Copy link
Contributor Author

run buildall

@Vallishp Vallishp force-pushed the nereids_recover_all branch from c74ba5a to fe98347 Compare November 16, 2024 14:27
@Vallishp
Copy link
Contributor Author

run buildall

@Vallishp Vallishp force-pushed the nereids_recover_all branch from fe98347 to e7c1764 Compare November 17, 2024 15:31
@Vallishp
Copy link
Contributor Author

run buildall

@Vallishp Vallishp force-pushed the nereids_recover_all branch from e7c1764 to e5db130 Compare November 18, 2024 03:49
@Vallishp
Copy link
Contributor Author

run buildall

@Vallishp Vallishp force-pushed the nereids_recover_all branch from e5db130 to be7f092 Compare November 18, 2024 10:05
@Vallishp
Copy link
Contributor Author

run buildall

starocean999
starocean999 previously approved these changes Nov 19, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring previously approved these changes Nov 19, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Vallishp Vallishp dismissed stale reviews from dataroaring and starocean999 via cba8acb November 19, 2024 09:48
@Vallishp
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 19, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 45253 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cba8acb9402f4e4fdcab7d2660e02783393f836a, data reload: false

------ Round 1 ----------------------------------
q1	17940	7727	7358	7358
q2	2242	1176	1164	1164
q3	10063	1154	1240	1154
q4	10313	719	681	681
q5	7611	2758	2696	2696
q6	247	151	151	151
q7	997	640	617	617
q8	9366	2365	2357	2357
q9	6619	6390	6429	6390
q10	7004	2306	2345	2306
q11	461	265	270	265
q12	414	224	214	214
q13	17764	3024	3098	3024
q14	247	218	221	218
q15	579	540	527	527
q16	649	588	606	588
q17	984	562	480	480
q18	7297	6831	6799	6799
q19	1341	1056	963	963
q20	2917	2735	2708	2708
q21	4387	3267	3236	3236
q22	1396	1357	1366	1357
Total cold run time: 110838 ms
Total hot run time: 45253 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7322	7445	7450	7445
q2	352	241	244	241
q3	3091	3099	2993	2993
q4	2094	1911	1821	1821
q5	5648	5775	5817	5775
q6	226	148	142	142
q7	2216	1850	1801	1801
q8	3314	3562	3535	3535
q9	8914	8928	8962	8928
q10	3635	3607	3569	3569
q11	589	503	525	503
q12	832	632	604	604
q13	10217	3260	3294	3260
q14	293	260	296	260
q15	575	525	522	522
q16	670	636	639	636
q17	1885	1670	1610	1610
q18	8292	7709	7765	7709
q19	1734	1622	1624	1622
q20	2208	1885	1861	1861
q21	5618	5534	5408	5408
q22	653	572	583	572
Total cold run time: 70378 ms
Total hot run time: 60817 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cba8acb9402f4e4fdcab7d2660e02783393f836a, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.04	0.05
query3	0.23	0.06	0.07
query4	1.63	0.10	0.10
query5	0.41	0.43	0.40
query6	1.13	0.66	0.67
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.50	0.49
query10	0.56	0.54	0.56
query11	0.13	0.10	0.11
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	2.71	2.72	2.79
query15	0.92	0.84	0.82
query16	0.39	0.37	0.39
query17	1.07	1.06	1.02
query18	0.20	0.21	0.20
query19	1.87	1.87	2.03
query20	0.01	0.01	0.01
query21	15.35	0.57	0.58
query22	2.63	2.13	2.22
query23	17.10	0.83	0.87
query24	3.06	1.40	0.66
query25	0.35	0.05	0.04
query26	0.45	0.14	0.13
query27	0.04	0.05	0.04
query28	10.55	1.09	1.08
query29	12.57	3.27	3.30
query30	0.25	0.07	0.06
query31	2.85	0.38	0.38
query32	3.28	0.46	0.48
query33	3.02	3.04	3.03
query34	16.92	4.44	4.50
query35	4.55	4.53	4.61
query36	0.66	0.49	0.49
query37	0.10	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.03
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.87 s
Total hot run time: 32.65 s

@Vallishp
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 44985 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b950a7a4a76fbe3d8c86d6dd41036a65f8963398, data reload: false

------ Round 1 ----------------------------------
q1	17610	7416	7404	7404
q2	2202	1163	1183	1163
q3	9987	1156	1157	1156
q4	10219	716	731	716
q5	7612	2737	2698	2698
q6	238	149	145	145
q7	1002	633	608	608
q8	9374	2382	2342	2342
q9	6606	6516	6388	6388
q10	6976	2333	2307	2307
q11	460	255	250	250
q12	417	216	217	216
q13	17768	3052	2998	2998
q14	235	205	208	205
q15	570	534	523	523
q16	648	587	593	587
q17	970	501	563	501
q18	7326	6637	6777	6637
q19	1332	1048	1048	1048
q20	2899	2716	2675	2675
q21	3991	3081	3186	3081
q22	1410	1352	1337	1337
Total cold run time: 109852 ms
Total hot run time: 44985 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7261	7336	7317	7317
q2	360	239	241	239
q3	3092	2943	2895	2895
q4	2031	1808	1752	1752
q5	5610	5686	5428	5428
q6	218	133	136	133
q7	2133	1725	1680	1680
q8	3260	3422	3390	3390
q9	8489	8541	8481	8481
q10	3509	3444	3448	3444
q11	603	492	488	488
q12	790	587	588	587
q13	8763	3092	3061	3061
q14	287	255	273	255
q15	565	508	512	508
q16	683	610	625	610
q17	1805	1572	1568	1568
q18	7869	7353	7606	7353
q19	1671	1586	1605	1586
q20	2048	1818	1823	1818
q21	5430	5224	5245	5224
q22	624	560	555	555
Total cold run time: 67101 ms
Total hot run time: 58372 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b950a7a4a76fbe3d8c86d6dd41036a65f8963398, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.04	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.44	0.39	0.41
query6	1.16	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.53	0.50
query10	0.57	0.56	0.55
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.60	0.60
query14	2.70	2.74	2.72
query15	0.90	0.84	0.83
query16	0.39	0.38	0.38
query17	1.09	1.05	1.06
query18	0.24	0.21	0.22
query19	1.98	1.89	1.98
query20	0.02	0.01	0.01
query21	15.36	0.57	0.56
query22	2.66	1.92	1.70
query23	16.94	1.06	0.77
query24	2.58	1.70	1.81
query25	0.29	0.07	0.08
query26	0.47	0.16	0.14
query27	0.05	0.04	0.04
query28	9.84	1.10	1.08
query29	12.54	3.20	3.20
query30	0.25	0.07	0.07
query31	2.86	0.39	0.37
query32	3.24	0.46	0.47
query33	2.96	3.08	3.01
query34	16.97	4.41	4.46
query35	4.50	4.50	4.51
query36	0.65	0.48	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.14
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 105.59 s
Total hot run time: 33.12 s

@Vallishp
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 45254 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b950a7a4a76fbe3d8c86d6dd41036a65f8963398, data reload: false

------ Round 1 ----------------------------------
q1	17641	7450	7311	7311
q2	2215	1167	1170	1167
q3	9960	1161	1197	1161
q4	10247	771	778	771
q5	7618	2731	2684	2684
q6	235	150	148	148
q7	969	635	594	594
q8	9358	2361	2362	2361
q9	6712	6431	6457	6431
q10	7036	2291	2345	2291
q11	466	259	261	259
q12	489	228	222	222
q13	17794	3066	3069	3066
q14	248	208	209	208
q15	570	532	507	507
q16	668	591	587	587
q17	995	569	595	569
q18	7432	6817	6738	6738
q19	1344	1034	1018	1018
q20	2908	2713	2726	2713
q21	4024	3246	3115	3115
q22	1374	1350	1333	1333
Total cold run time: 110303 ms
Total hot run time: 45254 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7406	7326	7466	7326
q2	358	244	256	244
q3	3081	3029	2966	2966
q4	2095	1846	1789	1789
q5	5664	5691	5756	5691
q6	226	144	140	140
q7	2214	1840	1790	1790
q8	3330	3582	3586	3582
q9	8895	8814	8949	8814
q10	3604	3572	3581	3572
q11	600	503	499	499
q12	822	638	640	638
q13	11889	3252	3299	3252
q14	315	281	270	270
q15	582	519	522	519
q16	700	639	637	637
q17	1865	1663	1622	1622
q18	8248	7713	7743	7713
q19	1687	1670	1445	1445
q20	2187	1890	1895	1890
q21	5664	5532	5426	5426
q22	632	561	568	561
Total cold run time: 72064 ms
Total hot run time: 60386 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.15 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b950a7a4a76fbe3d8c86d6dd41036a65f8963398, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.43	0.43	0.41
query6	1.15	0.64	0.65
query7	0.02	0.01	0.01
query8	0.03	0.03	0.03
query9	0.59	0.49	0.51
query10	0.55	0.54	0.56
query11	0.13	0.10	0.11
query12	0.14	0.12	0.10
query13	0.62	0.60	0.60
query14	2.71	2.81	2.86
query15	0.90	0.82	0.83
query16	0.38	0.39	0.37
query17	1.05	1.06	1.00
query18	0.20	0.20	0.20
query19	1.96	1.80	2.01
query20	0.01	0.01	0.02
query21	15.36	0.57	0.57
query22	2.73	2.58	1.65
query23	17.14	0.90	0.84
query24	3.26	0.77	1.96
query25	0.21	0.13	0.05
query26	0.56	0.13	0.14
query27	0.05	0.04	0.04
query28	9.85	1.10	1.07
query29	12.56	3.22	3.23
query30	0.25	0.07	0.06
query31	2.85	0.39	0.38
query32	3.25	0.47	0.47
query33	2.92	2.97	3.03
query34	16.98	4.44	4.48
query35	4.57	4.58	4.56
query36	0.65	0.49	0.50
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.02	0.03
Total cold run time: 106.53 s
Total hot run time: 32.15 s

dataroaring
dataroaring approved these changes Nov 20, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 20, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@starocean999 starocean999 merged commit 2a4885e into apache:master Nov 20, 2024
26 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@starocean999 starocean999 starocean999 approved these changes

@CalvinKirs CalvinKirs Awaiting requested review from CalvinKirs CalvinKirs is a code owner

@morningman morningman Awaiting requested review from morningman morningman is a code owner

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Enhancement] (nereids)implement RecoverCommand in nereids
4 participants
@Vallishp @doris-robot @starocean999 @dataroaring