Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](kerberos) pass kerberos_krb5_conf_path to be jvm option #47431

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Jan 24, 2025

What problem does this PR solve?

Pass config::kerberos_krb5_conf_path to jvm option and add p2 regression for spark hudi jni scanner

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 24, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223 suxiaogang223 marked this pull request as draft January 24, 2025 10:59
@suxiaogang223 suxiaogang223 marked this pull request as ready for review January 25, 2025 05:25
@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32647 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bb877c3aba60e232064370867e2b58189f098828, data reload: false

------ Round 1 ----------------------------------
q1	17642	5435	5445	5435
q2	2057	305	167	167
q3	10434	1242	755	755
q4	10727	975	537	537
q5	7583	2439	2151	2151
q6	185	173	133	133
q7	920	793	611	611
q8	9234	1388	1158	1158
q9	5267	4967	4911	4911
q10	6848	2349	1888	1888
q11	469	272	250	250
q12	349	366	228	228
q13	17763	3721	3125	3125
q14	226	227	215	215
q15	525	490	468	468
q16	624	609	590	590
q17	550	871	326	326
q18	7236	6614	6855	6614
q19	3933	965	533	533
q20	310	347	210	210
q21	2904	2224	2018	2018
q22	372	355	324	324
Total cold run time: 106158 ms
Total hot run time: 32647 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5877	5627	5583	5583
q2	257	332	243	243
q3	2493	2736	2381	2381
q4	1473	1862	1449	1449
q5	4474	4879	4984	4879
q6	167	163	128	128
q7	2090	1949	1849	1849
q8	2653	2856	2701	2701
q9	7238	7159	7239	7159
q10	3073	3180	2762	2762
q11	589	536	498	498
q12	665	780	639	639
q13	3491	3950	3330	3330
q14	287	294	265	265
q15	524	478	485	478
q16	648	700	630	630
q17	1215	1737	1286	1286
q18	7779	7493	7437	7437
q19	792	1148	1054	1054
q20	2016	2062	1904	1904
q21	5478	5296	5087	5087
q22	611	597	605	597
Total cold run time: 53890 ms
Total hot run time: 52339 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.06% (10971/26083)
Line Coverage: 32.34% (92744/286779)
Region Coverage: 31.48% (47544/151018)
Branch Coverage: 27.54% (24090/87486)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bb877c3aba60e232064370867e2b58189f098828_bb877c3aba60e232064370867e2b58189f098828/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 191019 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bb877c3aba60e232064370867e2b58189f098828, data reload: false

query1	1318	942	921	921
query2	6397	2158	2079	2079
query3	11002	4579	4313	4313
query4	60711	29156	23078	23078
query5	5523	601	465	465
query6	430	197	184	184
query7	5525	500	290	290
query8	330	242	238	238
query9	8458	2619	2605	2605
query10	452	313	260	260
query11	17690	14842	15564	14842
query12	171	115	112	112
query13	1448	577	393	393
query14	10385	7583	6857	6857
query15	213	192	198	192
query16	7263	650	487	487
query17	1103	729	576	576
query18	1845	397	307	307
query19	198	183	154	154
query20	113	118	116	116
query21	216	124	99	99
query22	4430	4563	4589	4563
query23	33842	33225	33422	33225
query24	5619	2392	2257	2257
query25	470	451	391	391
query26	632	284	157	157
query27	1733	488	338	338
query28	3968	2484	2401	2401
query29	533	568	418	418
query30	212	190	167	167
query31	933	916	809	809
query32	71	60	56	56
query33	427	358	292	292
query34	733	858	503	503
query35	823	859	766	766
query36	1017	1057	975	975
query37	125	105	80	80
query38	4277	4356	4276	4276
query39	1494	1474	1422	1422
query40	207	115	107	107
query41	56	55	51	51
query42	127	109	104	104
query43	538	519	527	519
query44	1357	831	839	831
query45	201	183	176	176
query46	876	1080	650	650
query47	1971	1911	1833	1833
query48	396	423	331	331
query49	728	501	414	414
query50	663	697	402	402
query51	4327	4256	4249	4249
query52	106	110	100	100
query53	241	253	203	203
query54	489	517	417	417
query55	86	82	82	82
query56	267	269	245	245
query57	1158	1214	1121	1121
query58	245	238	241	238
query59	3435	3300	3098	3098
query60	278	275	255	255
query61	129	122	164	122
query62	762	749	681	681
query63	233	202	196	196
query64	1270	1017	652	652
query65	3249	3174	3136	3136
query66	709	387	304	304
query67	15940	15611	15579	15579
query68	5032	833	528	528
query69	492	295	256	256
query70	1189	1144	1147	1144
query71	420	281	257	257
query72	6091	3704	3805	3704
query73	810	750	363	363
query74	10022	9093	8810	8810
query75	3253	3155	2720	2720
query76	3668	1194	752	752
query77	492	355	274	274
query78	9952	9939	9450	9450
query79	3401	798	589	589
query80	1619	530	443	443
query81	557	271	249	249
query82	358	146	123	123
query83	270	171	156	156
query84	285	94	72	72
query85	809	349	305	305
query86	455	289	302	289
query87	4522	4635	4404	4404
query88	4668	2179	2147	2147
query89	394	323	289	289
query90	1553	192	192	192
query91	134	131	108	108
query92	67	54	54	54
query93	2880	906	529	529
query94	768	397	303	303
query95	343	270	257	257
query96	486	610	282	282
query97	2872	2873	2729	2729
query98	225	196	204	196
query99	1270	1356	1285	1285
Total cold run time: 311531 ms
Total hot run time: 191019 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.14 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bb877c3aba60e232064370867e2b58189f098828, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.06	0.07
query4	1.62	0.10	0.10
query5	0.43	0.42	0.41
query6	1.15	0.65	0.66
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.59	0.50	0.50
query10	0.55	0.56	0.56
query11	0.15	0.11	0.11
query12	0.15	0.11	0.10
query13	0.63	0.59	0.60
query14	2.73	2.75	2.73
query15	0.90	0.83	0.81
query16	0.39	0.37	0.38
query17	1.01	0.99	1.07
query18	0.23	0.22	0.21
query19	1.94	1.89	2.02
query20	0.01	0.01	0.01
query21	15.36	0.89	0.58
query22	0.75	0.96	0.88
query23	14.96	1.37	0.57
query24	2.63	1.19	1.45
query25	0.19	0.09	0.12
query26	0.33	0.15	0.13
query27	0.05	0.05	0.06
query28	14.23	1.04	0.43
query29	12.60	3.94	3.29
query30	0.25	0.09	0.06
query31	2.83	0.61	0.38
query32	3.23	0.55	0.45
query33	2.95	3.00	3.14
query34	16.80	5.13	4.52
query35	4.48	4.52	4.54
query36	0.63	0.50	0.52
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.14	0.12
query41	0.07	0.02	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.65 s
Total hot run time: 31.14 s

@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32092 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 94be24d16de3a258ef44b3005653bbcb321ded35, data reload: false

------ Round 1 ----------------------------------
q1	17639	5426	5482	5426
q2	2049	306	167	167
q3	10798	1246	730	730
q4	10244	969	515	515
q5	8429	2361	2131	2131
q6	201	167	133	133
q7	907	772	601	601
q8	9239	1334	1119	1119
q9	5110	4868	4858	4858
q10	6822	2313	1889	1889
q11	483	269	249	249
q12	343	356	224	224
q13	17779	3687	3099	3099
q14	221	220	215	215
q15	516	490	471	471
q16	649	619	591	591
q17	547	848	320	320
q18	6859	6491	6335	6335
q19	1909	941	529	529
q20	290	309	187	187
q21	2802	2236	1989	1989
q22	365	336	314	314
Total cold run time: 104201 ms
Total hot run time: 32092 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5509	5464	5495	5464
q2	241	324	234	234
q3	2274	2675	2297	2297
q4	1423	1805	1378	1378
q5	4318	4726	4614	4614
q6	161	155	124	124
q7	2073	1969	1799	1799
q8	2645	2831	2663	2663
q9	7273	7167	7219	7167
q10	3016	3294	2798	2798
q11	563	498	481	481
q12	635	719	568	568
q13	3524	3945	3289	3289
q14	278	294	274	274
q15	535	469	486	469
q16	653	715	652	652
q17	1245	1757	1281	1281
q18	7648	7431	7340	7340
q19	800	1179	1023	1023
q20	2000	2021	1901	1901
q21	5804	5143	4928	4928
q22	631	645	600	600
Total cold run time: 53249 ms
Total hot run time: 51344 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191734 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 94be24d16de3a258ef44b3005653bbcb321ded35, data reload: false

query1	1296	962	923	923
query2	6214	2119	2079	2079
query3	10962	4477	4466	4466
query4	60893	28796	23320	23320
query5	5443	590	439	439
query6	427	190	195	190
query7	5561	516	297	297
query8	311	228	214	214
query9	8610	2597	2573	2573
query10	464	303	249	249
query11	17677	15100	15736	15100
query12	158	108	111	108
query13	1423	560	427	427
query14	10987	7389	6457	6457
query15	227	219	192	192
query16	7346	635	494	494
query17	1125	720	589	589
query18	1902	416	327	327
query19	210	193	166	166
query20	120	117	113	113
query21	215	120	108	108
query22	4454	5037	4601	4601
query23	33927	33287	33781	33287
query24	5773	2337	2315	2315
query25	463	449	406	406
query26	682	280	153	153
query27	1803	475	336	336
query28	4668	2467	2434	2434
query29	522	549	446	446
query30	205	187	156	156
query31	949	882	830	830
query32	65	58	60	58
query33	435	387	301	301
query34	762	866	541	541
query35	819	854	761	761
query36	1006	1054	975	975
query37	121	104	80	80
query38	4367	4320	4242	4242
query39	1536	1427	1472	1427
query40	205	113	101	101
query41	50	50	51	50
query42	118	107	109	107
query43	529	530	517	517
query44	1322	832	821	821
query45	188	178	170	170
query46	905	1049	663	663
query47	1939	1913	1873	1873
query48	384	408	324	324
query49	717	492	411	411
query50	661	659	416	416
query51	4322	4243	4209	4209
query52	113	110	98	98
query53	239	267	182	182
query54	486	499	442	442
query55	81	83	80	80
query56	262	270	271	270
query57	1221	1223	1148	1148
query58	234	241	254	241
query59	3147	3476	3301	3301
query60	277	283	248	248
query61	120	116	112	112
query62	707	716	651	651
query63	220	194	188	188
query64	1340	998	664	664
query65	3240	3143	3202	3143
query66	757	395	299	299
query67	16107	15853	15631	15631
query68	4652	838	531	531
query69	481	284	255	255
query70	1195	1163	1150	1150
query71	407	281	255	255
query72	6195	3897	3821	3821
query73	756	754	364	364
query74	10348	9115	8690	8690
query75	3246	3173	2648	2648
query76	3509	1155	773	773
query77	518	363	345	345
query78	10053	10095	9299	9299
query79	2558	790	602	602
query80	1557	523	439	439
query81	537	266	241	241
query82	350	147	120	120
query83	262	171	154	154
query84	290	92	72	72
query85	772	372	300	300
query86	409	361	279	279
query87	4562	4536	4444	4444
query88	3511	2177	2150	2150
query89	390	324	292	292
query90	1558	189	192	189
query91	134	143	105	105
query92	59	60	54	54
query93	1802	866	537	537
query94	687	408	306	306
query95	330	270	255	255
query96	489	600	286	286
query97	2814	2898	2796	2796
query98	219	197	200	197
query99	1292	1401	1245	1245
Total cold run time: 309961 ms
Total hot run time: 191734 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.22 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 94be24d16de3a258ef44b3005653bbcb321ded35, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.03
query3	0.24	0.06	0.07
query4	1.61	0.11	0.11
query5	0.41	0.41	0.42
query6	1.16	0.66	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.59	0.52	0.50
query10	0.56	0.55	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	2.82	2.74	2.73
query15	0.91	0.83	0.84
query16	0.38	0.37	0.38
query17	1.02	1.06	1.06
query18	0.23	0.20	0.21
query19	1.97	1.85	2.00
query20	0.01	0.01	0.01
query21	15.36	0.87	0.59
query22	0.77	0.81	0.63
query23	15.24	1.45	0.59
query24	3.02	1.54	1.89
query25	0.22	0.12	0.17
query26	0.29	0.15	0.15
query27	0.06	0.05	0.06
query28	14.17	1.01	0.43
query29	12.60	3.96	3.27
query30	0.25	0.09	0.07
query31	2.83	0.60	0.39
query32	3.23	0.56	0.45
query33	2.98	2.98	2.99
query34	16.46	5.14	4.49
query35	4.50	4.48	4.46
query36	0.82	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.26 s
Total hot run time: 31.22 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.06% (10970/26083)
Line Coverage: 32.33% (92706/286779)
Region Coverage: 31.48% (47546/151018)
Branch Coverage: 27.52% (24080/87486)
Coverage Report: http://coverage.selectdb-in.cc/coverage/94be24d16de3a258ef44b3005653bbcb321ded35_94be24d16de3a258ef44b3005653bbcb321ded35/report/index.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants