Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](cloud) Cloud enable fe deploy mode from master-observers to mul… #45255

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

[Fix](cloud) Cloud enable fe deploy mode from master-observers to mul… #45255

wants to merge 5 commits into from

Conversation

deardeng
Copy link
Contributor

@deardeng deardeng commented Dec 10, 2024 •
edited
Loading

…ti-follers

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

IN CLOUD

  1. add some cloud metaservice api regression test
  2. prohibit one node add to multi cluster repeatedly
  3. supports switching from master observer mode to multi follower mode, and compatible with upgrading from existing master-observers deployment mode to multi-follwers mode
  4. when adding an FE node(add fe cluster, add fe node), there is a 5-minute protection time and the FE node cannot be droped(drop fe cluster, drop fe node)
  5. drop the node. If the node cannot be found, an error will occur and the code will return 404 instead of the previous silent processing
  6. If the MS node starts with 127.0.0.1 IP and the prohibit_use_loopback_addresses is set to true(default), the MS node will not be able to start, ms log will find enable check prohibit use loopback addr, but localhost=127.0.0.1, so exit(-1)

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@deardeng
Copy link
Contributor Author

run buildall

@deardeng
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39925 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be63162b4fc91141f8f9ef1ced2c639c17c8b2fe, data reload: false

------ Round 1 ----------------------------------
q1	17573	7377	7373	7373
q2	2048	188	169	169
q3	10568	1095	1209	1095
q4	10552	717	730	717
q5	7601	2715	2715	2715
q6	234	148	152	148
q7	998	647	594	594
q8	9248	1836	1925	1836
q9	6617	6443	6559	6443
q10	6965	2309	2284	2284
q11	465	263	248	248
q12	431	231	223	223
q13	17784	2991	2963	2963
q14	241	204	215	204
q15	580	532	538	532
q16	663	603	581	581
q17	980	533	585	533
q18	7161	6782	6630	6630
q19	1341	1072	1013	1013
q20	467	180	189	180
q21	3991	3266	3124	3124
q22	381	323	320	320
Total cold run time: 106889 ms
Total hot run time: 39925 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7298	7238	7191	7191
q2	329	228	242	228
q3	2881	2741	2921	2741
q4	2095	1828	1792	1792
q5	5677	5650	5633	5633
q6	227	141	145	141
q7	2218	1770	1800	1770
q8	3340	3572	3494	3494
q9	8955	9122	9040	9040
q10	3597	3548	3527	3527
q11	619	517	522	517
q12	817	600	603	600
q13	11435	3285	3222	3222
q14	319	271	288	271
q15	578	521	537	521
q16	686	657	653	653
q17	1829	1628	1632	1628
q18	8278	7761	7554	7554
q19	1691	1609	1604	1604
q20	2089	1898	1834	1834
q21	5594	5565	5540	5540
q22	648	577	633	577
Total cold run time: 71200 ms
Total hot run time: 60078 ms

@deardeng
Copy link
Contributor Author

run cloudut

@doris-robot
Copy link

TPC-DS: Total hot run time: 197298 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be63162b4fc91141f8f9ef1ced2c639c17c8b2fe, data reload: false

query1	1255	946	933	933
query2	6212	2050	2034	2034
query3	11021	4591	4399	4399
query4	66485	28631	23660	23660
query5	4889	486	458	458
query6	412	183	178	178
query7	5479	302	294	294
query8	322	236	242	236
query9	8368	2703	2690	2690
query10	435	244	240	240
query11	16933	15265	15933	15265
query12	155	101	101	101
query13	1455	422	407	407
query14	10307	7178	7734	7178
query15	207	192	192	192
query16	7177	475	508	475
query17	1099	595	588	588
query18	1909	317	311	311
query19	204	167	165	165
query20	130	108	113	108
query21	213	106	112	106
query22	4646	4432	4382	4382
query23	35027	34301	34612	34301
query24	5389	2426	2513	2426
query25	484	387	414	387
query26	648	148	156	148
query27	1881	280	289	280
query28	4434	2429	2444	2429
query29	702	439	409	409
query30	221	159	165	159
query31	1028	849	841	841
query32	64	53	57	53
query33	441	289	299	289
query34	930	508	551	508
query35	871	748	772	748
query36	1079	963	959	959
query37	127	73	79	73
query38	4582	4414	4357	4357
query39	1514	1459	1477	1459
query40	229	103	99	99
query41	44	43	44	43
query42	117	103	96	96
query43	538	513	491	491
query44	1202	841	840	840
query45	195	171	175	171
query46	1190	731	728	728
query47	2006	1951	1950	1950
query48	405	324	310	310
query49	724	404	409	404
query50	845	395	393	393
query51	7344	7253	7278	7253
query52	102	86	84	84
query53	267	182	178	178
query54	532	412	394	394
query55	79	77	76	76
query56	247	233	233	233
query57	1269	1095	1107	1095
query58	212	209	208	208
query59	3189	2930	2953	2930
query60	263	237	235	235
query61	114	133	102	102
query62	781	671	697	671
query63	215	189	184	184
query64	1353	650	660	650
query65	3281	3205	3178	3178
query66	696	308	297	297
query67	15927	15659	15876	15659
query68	3935	573	562	562
query69	451	254	253	253
query70	1215	1149	1176	1149
query71	362	253	250	250
query72	6366	4200	4148	4148
query73	747	360	355	355
query74	10323	8963	9092	8963
query75	3382	2678	2672	2672
query76	1883	1037	1125	1037
query77	492	286	274	274
query78	10343	9443	9451	9443
query79	1380	594	591	591
query80	943	445	434	434
query81	484	251	226	226
query82	1305	122	118	118
query83	248	140	140	140
query84	287	71	70	70
query85	875	295	307	295
query86	348	294	270	270
query87	4708	4585	4650	4585
query88	3318	2201	2164	2164
query89	422	304	294	294
query90	2014	187	186	186
query91	139	104	101	101
query92	64	48	52	48
query93	1770	526	532	526
query94	809	299	285	285
query95	349	262	254	254
query96	608	278	278	278
query97	2831	2644	2661	2644
query98	215	193	195	193
query99	1589	1312	1325	1312
Total cold run time: 316306 ms
Total hot run time: 197298 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.39 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit be63162b4fc91141f8f9ef1ced2c639c17c8b2fe, data reload: false

query1	0.04	0.03	0.02
query2	0.09	0.04	0.04
query3	0.23	0.06	0.05
query4	1.66	0.08	0.08
query5	0.42	0.40	0.44
query6	1.17	0.67	0.67
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.57	0.51	0.50
query10	0.56	0.56	0.57
query11	0.16	0.12	0.12
query12	0.17	0.14	0.13
query13	0.62	0.60	0.60
query14	2.71	2.78	2.71
query15	0.93	0.85	0.85
query16	0.38	0.36	0.39
query17	1.01	1.03	1.01
query18	0.20	0.19	0.19
query19	2.01	1.90	2.06
query20	0.01	0.02	0.01
query21	15.35	0.67	0.66
query22	4.79	6.92	1.90
query23	18.30	1.46	1.38
query24	2.11	0.24	0.23
query25	0.16	0.09	0.08
query26	0.27	0.17	0.17
query27	0.08	0.08	0.08
query28	13.18	1.16	1.15
query29	12.63	3.45	3.41
query30	0.24	0.07	0.06
query31	2.87	0.41	0.39
query32	3.22	0.50	0.49
query33	3.01	3.04	3.20
query34	16.74	4.50	4.47
query35	4.56	4.53	4.54
query36	0.67	0.49	0.49
query37	0.21	0.18	0.16
query38	0.16	0.15	0.16
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 111.97 s
Total hot run time: 33.39 s

@deardeng
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@deardeng
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

gavinchou
gavinchou previously approved these changes Dec 23, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 23, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 27, 2024
@deardeng
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32741 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 920dbac97dcb6d7bddda90e53d9e724ebf67b8ae, data reload: false

------ Round 1 ----------------------------------
q1	17704	6130	6138	6130
q2	2056	321	175	175
q3	10496	1224	736	736
q4	10252	867	432	432
q5	8264	2199	1999	1999
q6	207	185	150	150
q7	888	762	597	597
q8	9247	1339	1174	1174
q9	5320	4942	4972	4942
q10	6780	2315	1854	1854
q11	476	276	264	264
q12	354	363	221	221
q13	17759	3553	2991	2991
q14	236	236	212	212
q15	560	502	496	496
q16	621	619	578	578
q17	582	835	333	333
q18	7101	6425	6369	6369
q19	2053	992	597	597
q20	304	320	187	187
q21	2843	2177	1999	1999
q22	364	345	305	305
Total cold run time: 104467 ms
Total hot run time: 32741 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6418	6273	6227	6227
q2	241	329	233	233
q3	2240	2652	2333	2333
q4	1392	1859	1364	1364
q5	4322	4735	4911	4735
q6	197	183	144	144
q7	2050	1969	1863	1863
q8	2642	2781	2666	2666
q9	7340	7250	7324	7250
q10	3097	3366	2762	2762
q11	571	510	489	489
q12	655	772	647	647
q13	3394	3765	3170	3170
q14	285	313	266	266
q15	583	531	517	517
q16	665	695	646	646
q17	1259	1760	1261	1261
q18	7739	7577	7264	7264
q19	836	1118	1127	1118
q20	2030	2107	1899	1899
q21	5756	5299	4893	4893
q22	607	621	587	587
Total cold run time: 54319 ms
Total hot run time: 52334 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197747 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 920dbac97dcb6d7bddda90e53d9e724ebf67b8ae, data reload: false

query1	1284	938	894	894
query2	6406	2270	2333	2270
query3	11030	4928	4685	4685
query4	33198	23887	23493	23493
query5	4351	601	482	482
query6	287	211	191	191
query7	3987	491	311	311
query8	314	263	247	247
query9	9418	2733	2725	2725
query10	481	316	260	260
query11	17988	15564	15622	15564
query12	168	111	120	111
query13	1574	539	411	411
query14	10152	7419	7298	7298
query15	246	204	187	187
query16	7726	653	499	499
query17	1555	773	626	626
query18	1873	403	362	362
query19	205	192	168	168
query20	127	120	114	114
query21	208	133	108	108
query22	4871	4513	4459	4459
query23	34014	33856	34249	33856
query24	6672	2335	2317	2317
query25	493	475	419	419
query26	812	279	160	160
query27	2006	460	339	339
query28	5934	2493	2477	2477
query29	608	561	443	443
query30	207	191	154	154
query31	1019	958	862	862
query32	89	60	59	59
query33	469	351	316	316
query34	777	870	521	521
query35	811	854	751	751
query36	1040	1056	976	976
query37	123	101	70	70
query38	4283	4186	4357	4186
query39	1537	1510	1459	1459
query40	223	117	108	108
query41	47	49	45	45
query42	120	105	110	105
query43	533	543	488	488
query44	1372	846	840	840
query45	191	175	170	170
query46	892	1060	686	686
query47	2029	2000	1909	1909
query48	384	426	345	345
query49	699	470	397	397
query50	649	684	422	422
query51	7211	7195	7078	7078
query52	104	110	96	96
query53	232	266	187	187
query54	476	535	411	411
query55	90	84	85	84
query56	263	272	250	250
query57	1221	1233	1160	1160
query58	241	231	230	230
query59	3157	3234	2961	2961
query60	292	286	248	248
query61	110	106	114	106
query62	856	812	782	782
query63	236	197	199	197
query64	3506	1073	677	677
query65	3350	3263	3275	3263
query66	723	416	333	333
query67	16454	15875	15567	15567
query68	9264	744	513	513
query69	492	285	258	258
query70	1236	1104	1131	1104
query71	439	280	262	262
query72	6360	3796	3835	3796
query73	652	753	363	363
query74	10586	9118	9228	9118
query75	4614	3116	2685	2685
query76	4746	1217	775	775
query77	848	388	285	285
query78	10051	10239	9420	9420
query79	3468	910	584	584
query80	701	519	462	462
query81	514	262	237	237
query82	676	150	122	122
query83	190	159	142	142
query84	282	90	66	66
query85	759	368	297	297
query86	337	325	308	308
query87	4420	4716	4448	4448
query88	4065	2220	2205	2205
query89	427	331	298	298
query90	1830	187	193	187
query91	136	136	102	102
query92	68	53	50	50
query93	1184	875	526	526
query94	638	402	275	275
query95	325	269	264	264
query96	492	605	280	280
query97	2746	2854	2702	2702
query98	217	202	195	195
query99	1694	1567	1443	1443
Total cold run time: 298461 ms
Total hot run time: 197747 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 920dbac97dcb6d7bddda90e53d9e724ebf67b8ae, data reload: false

query1	0.03	0.03	0.03
query2	0.09	0.04	0.04
query3	0.24	0.06	0.06
query4	1.64	0.07	0.08
query5	0.42	0.39	0.41
query6	1.17	0.66	0.65
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.54	0.49	0.53
query10	0.56	0.56	0.56
query11	0.16	0.13	0.13
query12	0.15	0.13	0.13
query13	0.61	0.61	0.59
query14	2.72	2.74	2.90
query15	0.91	0.84	0.83
query16	0.38	0.39	0.38
query17	1.07	1.06	1.08
query18	0.19	0.18	0.19
query19	1.93	1.88	2.05
query20	0.02	0.01	0.02
query21	15.35	0.96	0.65
query22	0.77	0.79	0.70
query23	14.94	1.48	0.69
query24	2.22	0.37	0.22
query25	0.15	0.09	0.09
query26	0.28	0.20	0.18
query27	0.08	0.08	0.08
query28	13.32	1.84	1.13
query29	12.68	4.09	3.45
query30	0.25	0.08	0.05
query31	2.84	0.61	0.41
query32	3.22	0.58	0.48
query33	3.14	3.08	3.10
query34	16.70	5.15	4.56
query35	4.55	4.57	4.59
query36	0.62	0.51	0.47
query37	0.21	0.16	0.17
query38	0.17	0.15	0.15
query39	0.05	0.05	0.04
query40	0.17	0.14	0.12
query41	0.10	0.05	0.05
query42	0.07	0.05	0.06
query43	0.05	0.05	0.05
Total cold run time: 104.83 s
Total hot run time: 31.71 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
dev/3.0.x reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@deardeng @doris-robot @gavinchou @dataroaring