Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](new_json_reader)fix core for new_json_reader #45905

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

amorynan
Copy link
Contributor

@amorynan amorynan commented Dec 25, 2024

What problem does this PR solve?

fix coredump with new_json_reader if we read invalid data will core which source from #43469
image

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@amorynan
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.89% (10121/26022)
Line Coverage: 29.89% (85525/286103)
Region Coverage: 29.03% (43712/150563)
Branch Coverage: 25.57% (22302/87218)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2d330f960282d37d961f07d6c4757c4fcc98156d_2d330f960282d37d961f07d6c4757c4fcc98156d/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 32669 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2d330f960282d37d961f07d6c4757c4fcc98156d, data reload: false

------ Round 1 ----------------------------------
q1	17576	6196	6039	6039
q2	2041	302	168	168
q3	10477	1204	746	746
q4	10201	851	430	430
q5	7525	2183	1966	1966
q6	204	179	144	144
q7	900	765	597	597
q8	9221	1341	1191	1191
q9	5174	4886	4859	4859
q10	6802	2313	1843	1843
q11	491	284	255	255
q12	347	371	227	227
q13	18361	3710	3129	3129
q14	256	241	225	225
q15	572	511	504	504
q16	636	621	595	595
q17	579	850	324	324
q18	6801	6394	6388	6388
q19	1725	951	532	532
q20	316	333	192	192
q21	2849	2237	1997	1997
q22	368	337	318	318
Total cold run time: 103422 ms
Total hot run time: 32669 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6208	6266	6223	6223
q2	245	321	244	244
q3	2231	2621	2319	2319
q4	1353	1822	1399	1399
q5	4351	4795	4802	4795
q6	185	174	140	140
q7	2064	2002	1844	1844
q8	2652	2762	2697	2697
q9	7368	7230	7317	7230
q10	3096	3376	2807	2807
q11	576	508	487	487
q12	639	777	658	658
q13	3403	3768	3166	3166
q14	277	306	278	278
q15	568	540	500	500
q16	651	697	672	672
q17	1263	1709	1263	1263
q18	7647	7428	7360	7360
q19	835	1166	1076	1076
q20	2045	2013	1907	1907
q21	5597	5375	4927	4927
q22	602	637	612	612
Total cold run time: 53856 ms
Total hot run time: 52604 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197527 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2d330f960282d37d961f07d6c4757c4fcc98156d, data reload: false

query1	1291	982	930	930
query2	6491	2488	2483	2483
query3	11028	4748	4656	4656
query4	33326	23733	23346	23346
query5	4505	617	467	467
query6	282	194	203	194
query7	4629	502	315	315
query8	296	260	234	234
query9	9365	2755	2752	2752
query10	462	314	243	243
query11	17706	15255	15780	15255
query12	172	105	103	103
query13	1647	539	435	435
query14	10520	7134	7764	7134
query15	255	200	198	198
query16	8457	637	491	491
query17	1574	790	602	602
query18	2103	421	333	333
query19	213	195	161	161
query20	120	118	123	118
query21	209	125	107	107
query22	4463	4590	4415	4415
query23	34730	33676	33505	33505
query24	6366	2363	2342	2342
query25	474	484	401	401
query26	796	270	156	156
query27	2050	464	333	333
query28	5678	2508	2455	2455
query29	628	562	438	438
query30	207	184	153	153
query31	1010	918	828	828
query32	75	60	59	59
query33	484	373	294	294
query34	776	867	539	539
query35	790	850	793	793
query36	1044	1056	985	985
query37	115	98	73	73
query38	4406	4306	4297	4297
query39	1590	1503	1486	1486
query40	222	116	106	106
query41	45	68	43	43
query42	122	105	106	105
query43	541	554	519	519
query44	1406	833	819	819
query45	187	175	169	169
query46	899	1052	677	677
query47	1992	1997	1930	1930
query48	388	439	326	326
query49	725	471	392	392
query50	631	671	416	416
query51	7319	7334	7343	7334
query52	107	113	95	95
query53	230	253	186	186
query54	483	508	415	415
query55	82	77	79	77
query56	279	263	255	255
query57	1218	1234	1175	1175
query58	239	225	235	225
query59	3314	3493	3455	3455
query60	279	267	254	254
query61	142	122	108	108
query62	867	834	749	749
query63	224	198	197	197
query64	3120	1028	666	666
query65	3313	3337	3310	3310
query66	792	405	311	311
query67	16619	15824	15501	15501
query68	9543	737	507	507
query69	481	293	250	250
query70	1259	1165	1125	1125
query71	436	280	250	250
query72	6011	3851	3947	3851
query73	674	746	355	355
query74	9897	9150	8986	8986
query75	4644	3158	2681	2681
query76	5115	1172	762	762
query77	962	381	301	301
query78	10128	10504	9434	9434
query79	2839	883	597	597
query80	863	518	446	446
query81	494	260	224	224
query82	661	146	123	123
query83	189	159	147	147
query84	279	82	72	72
query85	799	367	308	308
query86	356	318	279	279
query87	4725	4467	4380	4380
query88	3349	2244	2234	2234
query89	418	331	298	298
query90	1878	190	189	189
query91	135	142	105	105
query92	61	58	50	50
query93	1652	875	515	515
query94	647	399	287	287
query95	380	260	246	246
query96	494	605	282	282
query97	2786	2850	2706	2706
query98	225	201	196	196
query99	1740	1560	1417	1417
Total cold run time: 300238 ms
Total hot run time: 197527 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.77 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2d330f960282d37d961f07d6c4757c4fcc98156d, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.61	0.10	0.10
query5	0.42	0.41	0.43
query6	1.16	0.66	0.65
query7	0.02	0.02	0.01
query8	0.03	0.03	0.03
query9	0.60	0.51	0.52
query10	0.56	0.59	0.56
query11	0.14	0.10	0.09
query12	0.13	0.11	0.11
query13	0.61	0.62	0.59
query14	2.75	2.84	2.73
query15	0.88	0.82	0.82
query16	0.40	0.40	0.38
query17	1.07	1.05	1.07
query18	0.22	0.21	0.21
query19	1.95	1.85	1.97
query20	0.01	0.00	0.01
query21	15.35	0.93	0.58
query22	0.74	0.91	0.56
query23	15.32	1.40	0.59
query24	3.25	0.47	1.24
query25	0.19	0.19	0.22
query26	0.26	0.15	0.13
query27	0.06	0.05	0.04
query28	13.56	1.52	1.04
query29	12.58	3.97	3.26
query30	0.26	0.09	0.06
query31	2.82	0.61	0.38
query32	3.24	0.54	0.46
query33	3.05	3.12	3.12
query34	16.69	5.05	4.48
query35	4.54	4.43	4.44
query36	0.65	0.48	0.46
query37	0.10	0.06	0.05
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 105.98 s
Total hot run time: 30.77 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants