Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement](tools) Introduce new BE benchmark and remove old one #44181

Merged
merged 2 commits into from
Nov 20, 2024

Conversation

zclllyybb
Copy link
Contributor

@zclllyybb zclllyybb commented Nov 18, 2024

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

the old be-benchmark has been not maintained for long time. now it's time to drop it. in this pr we split google-benchmark as a individual component rather than in be-ut.
replace #41180

To get closest to the actual test results, we compile the executable with the entire BE binary (so that binary bloat, etc. is consistent with the actual BE) and start it with start_be.sh, so that jemalloc, etc. conditions are also consistent.

usage:

./build.sh --benchmark
......
be/bin/start_be.sh --benchmark

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/benchmark/benchmark_main.cpp Show resolved Hide resolved
@zclllyybb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.01% (9898/26042)
Line Coverage: 29.19% (82764/283548)
Region Coverage: 28.32% (42496/150078)
Branch Coverage: 24.89% (21551/86600)
Coverage Report: http://coverage.selectdb-in.cc/coverage/69af298c2897a96d3d4af80ae8bb4a2df68e1129_69af298c2897a96d3d4af80ae8bb4a2df68e1129/report/index.html

BiteTheDDDDt
BiteTheDDDDt previously approved these changes Nov 19, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 19, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@zclllyybb
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 19, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 45331 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5fde914f366733c1301d139c77561371e622d76f, data reload: false

------ Round 1 ----------------------------------
q1	17630	7454	7300	7300
q2	2204	1164	1171	1164
q3	9982	1144	1157	1144
q4	10224	783	715	715
q5	7628	2749	2724	2724
q6	238	150	149	149
q7	990	631	617	617
q8	9352	2356	2358	2356
q9	6602	6426	6422	6422
q10	6960	2256	2309	2256
q11	465	261	275	261
q12	408	213	208	208
q13	17765	3065	3041	3041
q14	248	209	211	209
q15	565	536	515	515
q16	656	593	577	577
q17	1003	556	546	546
q18	7336	6873	6734	6734
q19	1339	1001	937	937
q20	2920	2714	2704	2704
q21	4196	3416	3432	3416
q22	1450	1346	1336	1336
Total cold run time: 110161 ms
Total hot run time: 45331 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7296	7325	7269	7269
q2	326	224	232	224
q3	2906	2809	2776	2776
q4	1975	1737	1694	1694
q5	5404	5480	5512	5480
q6	226	139	138	138
q7	2168	1778	1783	1778
q8	3331	3416	3435	3416
q9	8579	8581	8565	8565
q10	3480	3430	3402	3402
q11	587	499	493	493
q12	763	561	578	561
q13	16244	3074	3075	3074
q14	283	258	256	256
q15	560	517	507	507
q16	654	645	638	638
q17	1832	1594	1561	1561
q18	7803	7304	7310	7304
q19	1663	1536	1485	1485
q20	2052	1825	1836	1825
q21	5433	5153	5085	5085
q22	633	533	584	533
Total cold run time: 74198 ms
Total hot run time: 58064 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.02% (9898/26035)
Line Coverage: 29.19% (82779/283540)
Region Coverage: 28.33% (42504/150055)
Branch Coverage: 24.90% (21556/86584)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5fde914f366733c1301d139c77561371e622d76f_5fde914f366733c1301d139c77561371e622d76f/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 32.32 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5fde914f366733c1301d139c77561371e622d76f, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.05
query3	0.23	0.07	0.06
query4	1.63	0.10	0.10
query5	0.42	0.42	0.40
query6	1.18	0.66	0.67
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.56	0.51	0.51
query10	0.56	0.54	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.60	0.60	0.60
query14	2.73	2.73	2.72
query15	0.91	0.82	0.82
query16	0.38	0.40	0.37
query17	1.02	1.06	0.98
query18	0.23	0.23	0.22
query19	1.95	1.84	1.99
query20	0.02	0.01	0.01
query21	15.37	0.58	0.58
query22	2.32	2.13	1.35
query23	17.38	0.86	0.82
query24	3.25	1.29	1.72
query25	0.39	0.10	0.08
query26	0.53	0.14	0.14
query27	0.05	0.05	0.04
query28	9.92	1.10	1.06
query29	12.57	3.23	3.20
query30	0.24	0.06	0.06
query31	2.87	0.38	0.37
query32	3.28	0.47	0.47
query33	2.95	2.99	3.03
query34	17.05	4.46	4.50
query35	4.50	4.50	4.47
query36	0.66	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.68 s
Total hot run time: 32.32 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 20, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@zhangstar333 zhangstar333 merged commit 5260a98 into apache:master Nov 20, 2024
27 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants