Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix benchmark behavior #631

Closed
wants to merge 3 commits into from
Closed

Conversation

Rigidity
Copy link
Contributor

@Rigidity Rigidity commented Jul 30, 2024

Results compared to prior to interior mutability:

Gnuplot not found, using plotters backend
Benchmarking bls_cache.aggregate_verify, 0% cache hits: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 51.1s, or reduce sample count to 10.
bls_cache.aggregate_verify, 0% cache hits
                        time:   [512.40 ms 516.72 ms 522.79 ms]
                        change: [-0.1042% +0.7965% +2.0013%] (p = 0.16 > 0.05)
                        No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe

Benchmarking bls_cache.aggregate_verify, 10% cache hits: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 47.0s, or reduce sample count to 10.
bls_cache.aggregate_verify, 10% cache hits
                        time:   [464.36 ms 465.94 ms 467.72 ms]
                        change: [-8.6168% -8.2999% -7.9271%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
  2 (2.00%) high mild
  9 (9.00%) high severe

Benchmarking bls_cache.aggregate_verify, 20% cache hits: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 40.8s, or reduce sample count to 10.
bls_cache.aggregate_verify, 20% cache hits
                        time:   [407.63 ms 408.87 ms 410.40 ms]
                        change: [-19.863% -19.596% -19.288%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe

Benchmarking bls_cache.aggregate_verify, 50% cache hits: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 25.6s, or reduce sample count to 10.
bls_cache.aggregate_verify, 50% cache hits
                        time:   [258.91 ms 259.95 ms 261.08 ms]
                        change: [-49.998% -49.686% -49.347%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild

bls_cache.aggregate_verify, 100% cache hits
                        time:   [2.1386 ms 2.1594 ms 2.1925 ms]
                        change: [+3.8251% +5.2293% +7.0844%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

Benchmarking bls_cache.aggregate_verify, no cache: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 26.6s, or reduce sample count to 10.
bls_cache.aggregate_verify, no cache
                        time:   [266.31 ms 267.23 ms 268.25 ms]
                        change: [-1.3208% -0.5508% +0.1486%] (p = 0.14 > 0.05)
                        No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe

@Rigidity Rigidity requested review from arvidn and matt-o-how July 30, 2024 12:44
@arvidn
Copy link
Contributor

arvidn commented Aug 5, 2024

I think we're abandoning the target branch and picking the parts we want to keep into separate PRs. I don't believe this issue of the benchmark exists in main

@Rigidity Rigidity closed this Aug 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants