Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmarks without stats because the samples are not normally distributed #54

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

benchmarks without stats because the samples are not normally distributed #54

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
aea16b8
benchmarks without stats because the samples are not normally distrib…
fonsp Jun 8, 2023
c482ce6
compare median not mean
fonsp Jun 8, 2023
330c705
Merge branch 'main' into no-stats-for-benchmark
fonsp Jun 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions test/benchmark.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,25 +89,33 @@ const TEST_BENCHMARK = true
b1 = run(bench1)
b2 = run(bench2)

t1 = mean(b1).time
t2 = mean(b2).time
mean1 = mean(b1).time
mean2 = mean(b2).time

median1 = BenchmarkTools.median(b1).time
median2 = BenchmarkTools.median(b2).time

σ1 = BenchmarkTools.std(b1).time
σ2 = BenchmarkTools.std(b2).time

tdiff = t1 - t2
tdiff = mean1 - mean2
σdiff = sqrt(σ1^2 + σ2^2)

ratio = t1 / t2
ratio_mean = mean1 / mean2
ratio_median = median1 / median2

@info "Expr $i" t1 t2 ratio diff=Text("$(round(Int64, tdiff)) ± $(round(Int64, σdiff)))") b1 b2
@info "Expr $i" mean1 mean2 ratio_mean ratio_median diff=Text("$(round(Int64, tdiff)) ± $(round(Int64, σdiff))") b1 b2

if TEST_BENCHMARK
# we should be faster, i.e.
# @test tdiff < 0

# and we have an admissible error of 2.5%
@test tdiff < 2*σdiff

# NO because the samples are not normally distributed
# so let's just do a percentage again
@test ratio_median < 1.2
end
end

Expand Down