Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Profile](scan) Add profile to detail the scan #46110

Open
wants to merge 1 commit into
base: branch-2.0
Choose a base branch
from

Conversation

HappenLee
Copy link
Contributor

What problem does this PR solve?

Add some profile to detail the scan cherry pick some code #45556

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@HappenLee HappenLee marked this pull request as ready for review December 27, 2024 09:30
@HappenLee
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49099 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4d0bd4ddfff995ea3baedbc18d9fe72c78603e37, data reload: false

------ Round 1 ----------------------------------
q1	17678	4446	4391	4391
q2	2065	153	158	153
q3	10482	1943	1944	1943
q4	10260	1267	1322	1267
q5	8425	3883	3950	3883
q6	238	124	121	121
q7	2056	1624	1592	1592
q8	9536	2786	2741	2741
q9	10302	9932	9755	9755
q10	8671	3606	3526	3526
q11	423	256	256	256
q12	477	302	299	299
q13	18371	3961	4036	3961
q14	362	327	329	327
q15	501	452	465	452
q16	553	454	469	454
q17	1143	938	918	918
q18	7278	6958	6785	6785
q19	1716	1592	1515	1515
q20	543	294	303	294
q21	4386	4149	4076	4076
q22	498	415	390	390
Total cold run time: 115964 ms
Total hot run time: 49099 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4312	4340	4354	4340
q2	324	232	223	223
q3	4177	4124	4154	4124
q4	2783	2751	2762	2751
q5	7216	7075	7156	7075
q6	240	117	118	117
q7	3254	2838	2844	2838
q8	4449	4471	4467	4467
q9	13738	13594	13688	13594
q10	4225	4292	4251	4251
q11	738	694	682	682
q12	1020	858	853	853
q13	6780	3735	3729	3729
q14	468	421	429	421
q15	492	457	459	457
q16	628	587	608	587
q17	3893	3883	3827	3827
q18	8761	8761	8808	8761
q19	1733	1735	1678	1678
q20	2342	2165	2122	2122
q21	8617	8485	8418	8418
q22	984	905	960	905
Total cold run time: 81174 ms
Total hot run time: 76220 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.54% (8308/21555)
Line Coverage: 30.23% (68693/227205)
Region Coverage: 29.67% (35383/119273)
Branch Coverage: 25.43% (18186/71526)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4d0bd4ddfff995ea3baedbc18d9fe72c78603e37_4d0bd4ddfff995ea3baedbc18d9fe72c78603e37/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 212591 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4d0bd4ddfff995ea3baedbc18d9fe72c78603e37, data reload: false

query1	918	435	381	381
query2	6550	2309	2091	2091
query3	6923	199	204	199
query4	23137	21712	21909	21712
query5	19732	6530	6513	6513
query6	284	222	237	222
query7	4331	301	310	301
query8	267	258	229	229
query9	3112	2635	2564	2564
query10	471	318	304	304
query11	15792	15141	15022	15022
query12	136	78	76	76
query13	1040	455	466	455
query14	17550	13304	13217	13217
query15	375	220	230	220
query16	6490	287	260	260
query17	1759	936	916	916
query18	900	315	309	309
query19	214	153	159	153
query20	99	93	100	93
query21	193	98	96	96
query22	5140	5197	5048	5048
query23	34350	33584	33592	33584
query24	7901	6337	6333	6333
query25	538	433	429	429
query26	1270	162	159	159
query27	2472	290	286	286
query28	6113	2265	2262	2262
query29	2942	2727	2697	2697
query30	246	169	167	167
query31	960	738	756	738
query32	73	61	61	61
query33	458	270	261	261
query34	859	490	477	477
query35	1128	902	928	902
query36	1418	1213	1186	1186
query37	173	61	62	61
query38	3070	2959	2918	2918
query39	1365	1338	1326	1326
query40	321	102	91	91
query41	40	36	36	36
query42	85	80	90	80
query43	621	636	666	636
query44	1159	723	725	723
query45	244	227	232	227
query46	1229	970	959	959
query47	1917	1657	1872	1657
query48	509	416	405	405
query49	658	388	384	384
query50	873	608	574	574
query51	4759	4668	4715	4668
query52	100	82	83	82
query53	233	186	186	186
query54	2664	2453	2476	2453
query55	98	80	93	80
query56	229	222	210	210
query57	1320	1149	1096	1096
query58	219	227	212	212
query59	3480	3194	3462	3194
query60	218	215	201	201
query61	99	105	98	98
query62	819	476	497	476
query63	209	176	172	172
query64	3510	1578	1511	1511
query65	3619	3543	3576	3543
query66	771	459	434	434
query67	15686	15180	15931	15180
query68	9258	656	667	656
query69	505	255	272	255
query70	1736	1368	1520	1368
query71	403	314	322	314
query72	6855	4677	4776	4677
query73	764	324	325	324
query74	6317	5829	5805	5805
query75	5040	3713	3706	3706
query76	5061	1179	1223	1179
query77	779	251	263	251
query78	12556	11719	11829	11719
query79	7882	656	658	656
query80	1717	399	381	381
query81	495	236	241	236
query82	1640	101	97	97
query83	176	132	133	132
query84	268	73	69	69
query85	884	309	312	309
query86	332	325	294	294
query87	3252	3063	3052	3052
query88	4635	2322	2303	2303
query89	409	292	292	292
query90	1920	216	216	216
query91	167	126	128	126
query92	61	53	56	53
query93	6130	586	554	554
query94	738	213	208	208
query95	2014	1903	2006	1903
query96	657	329	327	327
query97	6468	6295	6376	6295
query98	224	203	194	194
query99	2918	965	844	844
Total cold run time: 318699 ms
Total hot run time: 212591 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.41 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4d0bd4ddfff995ea3baedbc18d9fe72c78603e37, data reload: false

query1	0.03	0.02	0.02
query2	0.08	0.03	0.03
query3	0.25	0.05	0.06
query4	1.75	0.08	0.08
query5	0.54	0.53	0.52
query6	1.27	0.61	0.66
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.50	0.49
query10	0.56	0.53	0.54
query11	0.13	0.09	0.09
query12	0.12	0.09	0.10
query13	0.62	0.62	0.61
query14	0.79	0.78	0.77
query15	0.78	0.76	0.76
query16	0.39	0.38	0.37
query17	1.00	1.02	1.02
query18	0.22	0.26	0.25
query19	1.87	1.88	1.88
query20	0.02	0.01	0.01
query21	15.47	0.56	0.55
query22	2.15	2.35	1.98
query23	17.33	0.98	0.97
query24	3.76	1.07	1.50
query25	0.35	0.07	0.05
query26	0.55	0.16	0.16
query27	0.05	0.04	0.05
query28	8.75	0.71	0.71
query29	12.64	2.43	2.35
query30	0.56	0.50	0.51
query31	2.78	0.38	0.36
query32	3.41	0.49	0.50
query33	3.09	3.12	3.10
query34	15.27	4.84	4.78
query35	4.87	4.84	4.84
query36	1.06	1.00	1.02
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.15	0.14
query41	0.07	0.02	0.01
query42	0.03	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.48 s
Total hot run time: 31.41 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4d0bd4ddfff995ea3baedbc18d9fe72c78603e37 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       21.7 seconds inserted 10000000 Rows, about 460K ops/s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants