This package provides micro-benchmarks to measure and compare the performance of various streaming implementations in Haskell.
We have taken due to care to make sure that we are benchmarking correctly and fairly. See the notes on correct benchmarking.
DISCLAIMER: This package is a result of benchmarking effort done during the development of streamly by the authors of streamly.
The benchmark names are obvious, some of them are described below. Single operation benchmarks:
| Name | Description |
|---|---|
drain |
Just discards all the elements in the stream |
drop-all |
drops all element using the drop operation |
last |
extract the last element of the stream |
fold |
sum all the numbers in the stream |
map |
increments each number in the stream by 1 |
take-all |
Use take to retain all the elements in the stream |
filter-even |
Keep even numbers, discard odd |
scan |
scan the stream using + operation |
mapM |
transform the stream using a monadic action |
zip |
combines corresponding elements of the two streams together |
Composite operation benchmarks:
| Name | Description |
|---|---|
map x 4 |
perform map operation 4 times |
take-map |
take followed by a map |
For more details on how each benchmark is implemented see this benchmark file.
Each benchmark is run in a separate process to avoid any effects of GC interference and sharing across benchmarks.
Below we present some results comparing streamly with other streaming implementations. Due care has been taken to keep the comparisons fair. We have optimized each library's code to the best of our knowledge, please point out if you find any measurement issues.
You can run individual benchmarks using cabal bench <target>. You may
need to specify a build flag to include a particular streaming library
e.g. --flag pipes to benchmark the pipes library. Please consult the cabal
file to find the exact flag names.
You can generate the comparison reports presented in this page yourself. To do so you need to run the benchmarks using the reporting tool, first build the reporting tool using the following command:
$ cd bench-runner
$ cabal install --project-file cabal.project.user --installdir ../bin
If you want to create a report for benchmarks showing a 10% or greater improvement with Streamly over Lists, use:
$ bin/bench-runner --package-name streaming-benchmarks --package-version 0.4.0 --compare --diff-cutoff-percent 10 --diff-style absolute --targets "StreamlyPure List"
After running once, you can add --no-measure option to use the same benchmark
measurements for different reports. For example, use:
--diff-cutoff-percent -10to know where lists are better than streamly--diff-style multiplesto generate ratios instead of absolute values
Streamly, when used with Identity monad, is almost the same as Haskell lists
(in the base package).
See this
for more details.
The following table compares the timing of several operations for streamly with lists using a one million element stream. For brevity only those operations where the performance of the two packages differ by more than 10% are shown in the table below. The last column shows how many times slower list is compared to streamly.
| Benchmark | streamly(μs) | list(μs) | list/streamly |
|---|---|---|---|
| drop-map x 4 | 375.09 | 76925.32 | 205.08 |
| filter-drop x 4 | 382.03 | 54848.54 | 143.57 |
| drop-scan x 4 | 795.81 | 76716.79 | 96.40 |
| filter-scan x 4 | 795.60 | 44559.15 | 56.01 |
| scan-map x 4 | 1192.19 | 48838.22 | 40.97 |
| take-map x 4 | 1500.99 | 60126.58 | 40.06 |
| filter-take x 4 | 1502.01 | 48766.87 | 32.47 |
| take-drop x 4 | 1499.62 | 41720.03 | 27.82 |
| take-scan x 4 | 1874.94 | 51283.30 | 27.35 |
| drop-one x 4 | 375.33 | 8993.87 | 23.96 |
| dropWhile-false x 4 | 374.61 | 8957.79 | 23.91 |
| dropWhile-false | 374.83 | 8670.05 | 23.13 |
| drop-one | 390.77 | 8681.85 | 22.22 |
| dropWhile-true | 571.60 | 12237.48 | 21.41 |
| drop-all | 562.94 | 8262.38 | 14.68 |
| take-all | 624.83 | 564.34 | 1/1.11 |
| scan x 4 | 795.83 | 385.85 | 1/2.06 |
| appendR[10000] | 360.75 | 126.95 | 1/2.84 |
| concatMap | 34957.71 | 1124.85 | 1/31.08 |
- streamly-0.8.0, base-4.14.1.0, ghc-8.10.4, Linux
To generate these reports run bench-runner with:
--targets "StreamlyPure List"
The following table compares the timing of several operations for streamly with streaming using a million element stream.
| Benchmark | streamly(μs) | streaming(μs) | streaming/streamly |
|---|---|---|---|
| appendR[10000] | 326.56 | 1301176.69 | 3984.54 |
| mapM x 4 | 374.42 | 223591.08 | 597.17 |
| filter-map x 4 | 381.07 | 194903.88 | 511.47 |
| filter-scan x 4 | 795.66 | 233527.90 | 293.50 |
| filter-all-in x 4 | 375.40 | 102629.64 | 273.38 |
| filter-drop x 4 | 387.15 | 99096.98 | 255.96 |
| map x 4 | 386.49 | 94944.87 | 245.66 |
| drop-map x 4 | 375.62 | 89669.37 | 238.73 |
| scan x 4 | 797.00 | 166332.40 | 208.70 |
| scan-map x 4 | 1194.30 | 238804.48 | 199.95 |
| filter-even x 4 | 396.37 | 77865.47 | 196.45 |
| drop-scan x 4 | 796.98 | 156063.52 | 195.82 |
| takeWhile-true x 4 | 562.49 | 90183.53 | 160.33 |
| scan | 375.24 | 47520.57 | 126.64 |
| filter-take x 4 | 1498.55 | 189635.34 | 126.55 |
| mapM | 388.10 | 46689.61 | 120.30 |
| take-map x 4 | 1500.71 | 178954.50 | 119.25 |
| zip | 656.65 | 66689.73 | 101.56 |
| take-scan x 4 | 2380.35 | 241675.75 | 101.53 |
| filter-all-in | 375.97 | 33590.14 | 89.34 |
| map | 375.02 | 33081.13 | 88.21 |
| filter-even | 393.26 | 30458.46 | 77.45 |
| filter-all-out | 382.87 | 26826.21 | 70.07 |
| take-all x 4 | 1499.71 | 101332.53 | 67.57 |
| take-drop x 4 | 1498.53 | 98281.99 | 65.59 |
| takeWhile-true | 562.62 | 31863.25 | 56.63 |
| foldl' | 388.22 | 18503.15 | 47.66 |
| drop-all | 562.08 | 25200.32 | 44.83 |
| take-all | 768.65 | 33247.97 | 43.26 |
| dropWhile-true | 564.87 | 24431.50 | 43.25 |
| last | 385.53 | 15240.85 | 39.53 |
| dropWhile-false | 374.83 | 14566.70 | 38.86 |
| drop-one | 374.80 | 14565.01 | 38.86 |
| drop-one x 4 | 375.88 | 14448.67 | 38.44 |
| dropWhile-false x 4 | 390.12 | 14619.42 | 37.47 |
| drain | 375.06 | 13702.29 | 36.53 |
| toList | 117708.83 | 201444.81 | 1.71 |
- streamly-0.8.0, streaming-0.2.3.0, ghc-8.10.4, Linux
To generate these reports run bench-runner with:
--targets "Streamly Streaming" --cabal-build-options "--flag streaming"
The following table compares the timing of several operations for streamly with pipes using a million element stream.
| Benchmark | streamly(μs) | pipes(μs) | pipes/streamly |
|---|---|---|---|
| appendR[10000] | 327.90 | 901135.92 | 2748.21 |
| mapM x 4 | 375.20 | 407184.39 | 1085.23 |
| filter-map x 4 | 381.52 | 366759.70 | 961.31 |
| drop-map x 4 | 375.48 | 281296.82 | 749.16 |
| filter-all-in x 4 | 375.60 | 222331.68 | 591.93 |
| filter-drop x 4 | 387.44 | 222830.71 | 575.14 |
| drop-scan x 4 | 797.23 | 336737.89 | 422.39 |
| filter-even x 4 | 389.87 | 152688.91 | 391.64 |
| filter-scan x 4 | 797.38 | 309733.91 | 388.44 |
| drop-one x 4 | 375.48 | 139851.13 | 372.46 |
| map x 4 | 386.56 | 136289.32 | 352.57 |
| dropWhile-false x 4 | 390.72 | 137395.44 | 351.65 |
| scan-map x 4 | 1194.38 | 381286.88 | 319.23 |
| takeWhile-true x 4 | 562.86 | 165143.23 | 293.40 |
| scan x 4 | 796.68 | 222986.17 | 279.90 |
| mapM | 388.19 | 95576.97 | 246.21 |
| filter-all-in | 375.21 | 71297.42 | 190.02 |
| take-map x 4 | 1502.76 | 275887.24 | 183.59 |
| scan | 374.81 | 65549.13 | 174.89 |
| take-drop x 4 | 1503.43 | 256448.45 | 170.58 |
| filter-even | 390.29 | 66183.72 | 169.57 |
| filter-all-out | 376.99 | 59074.54 | 156.70 |
| drop-one | 375.19 | 58395.24 | 155.64 |
| dropWhile-false | 375.35 | 58223.03 | 155.12 |
| map | 375.05 | 57736.43 | 153.94 |
| filter-take x 4 | 1503.00 | 227925.71 | 151.65 |
| take-scan x 4 | 2455.91 | 354284.33 | 144.26 |
| zip | 657.07 | 86011.93 | 130.90 |
| takeWhile-true | 564.14 | 61390.21 | 108.82 |
| take-all x 4 | 1502.32 | 139730.70 | 93.01 |
| dropWhile-true | 564.03 | 49227.19 | 87.28 |
| drop-all | 562.05 | 46505.37 | 82.74 |
| take-all | 824.09 | 60511.34 | 73.43 |
| drain | 375.29 | 26390.59 | 70.32 |
| foldl' | 397.34 | 19064.05 | 47.98 |
| last | 387.11 | 17364.44 | 44.86 |
| toList | 117257.09 | 207405.94 | 1.77 |
- streamly-0.8.0, pipes-4.3.16, ghc-8.10.4, Linux
To generate these reports run bench-runner with:
--targets "Streamly Pipes" --cabal-build-options "--flag pipes"
The following table compares the timing of several operations for streamly with conduit using a million element stream.
| Benchmark | streamly(μs) | conduit(μs) | conduit/streamly |
|---|---|---|---|
| mapM x 4 | 375.46 | 297002.31 | 791.04 |
| filter-map x 4 | 380.79 | 267543.81 | 702.60 |
| drop-map x 4 | 375.66 | 232307.84 | 618.39 |
| filter-drop x 4 | 386.05 | 235029.15 | 608.81 |
| filter-scan x 4 | 796.56 | 306556.67 | 384.85 |
| drop-scan x 4 | 797.19 | 300789.06 | 377.31 |
| zip | 657.29 | 210069.05 | 319.60 |
| filter-all-in x 4 | 375.24 | 118506.68 | 315.82 |
| scan-map x 4 | 1194.67 | 360671.18 | 301.90 |
| map x 4 | 387.00 | 113497.14 | 293.27 |
| drop-one x 4 | 375.49 | 101842.95 | 271.23 |
| dropWhile-false x 4 | 389.44 | 102051.22 | 262.04 |
| scan x 4 | 796.72 | 190479.35 | 239.08 |
| takeWhile-true x 4 | 564.58 | 114459.57 | 202.73 |
| filter-even x 4 | 391.76 | 72369.30 | 184.73 |
| filter-take x 4 | 1502.04 | 267921.27 | 178.37 |
| take-map x 4 | 1502.88 | 238875.95 | 158.95 |
| take-drop x 4 | 1500.34 | 232606.19 | 155.04 |
| take-scan x 4 | 2443.83 | 309738.86 | 126.74 |
| mapM | 389.15 | 41897.48 | 107.66 |
| scan | 375.40 | 38137.85 | 101.59 |
| take-all x 4 | 1502.32 | 110682.74 | 73.67 |
| filter-all-in | 375.31 | 26024.21 | 69.34 |
| dropWhile-false | 375.10 | 25307.13 | 67.47 |
| map | 375.18 | 23088.09 | 61.54 |
| drop-one | 375.43 | 22020.65 | 58.65 |
| filter-even | 392.28 | 21504.28 | 54.82 |
| takeWhile-true | 562.79 | 29012.68 | 51.55 |
| filter-all-out | 378.76 | 15736.05 | 41.55 |
| drop-all | 562.89 | 19916.48 | 35.38 |
| foldl' | 388.88 | 12499.03 | 32.14 |
| dropWhile-true | 564.43 | 17983.35 | 31.86 |
| take-all | 784.67 | 24425.36 | 31.13 |
| last | 385.75 | 10974.84 | 28.45 |
| drain | 375.18 | 4272.15 | 11.39 |
| appendR[10000] | 326.93 | 1207.88 | 3.69 |
| toList | 116441.26 | 199138.09 | 1.71 |
- streamly-0.8.0, conduit-1.3.4.1, ghc-8.10.4, Linux
To generate these reports run bench-runner with:
--targets "Streamly Conduit" --cabal-build-options "--flag conduit"
bench-runner also generates a maxrss comparison report, displaying the
maximum resident memory for each benchmark.
This package supports many streaming libraries, bytestring, text, vector etc.
You can run the benchmarks directly or use bench-runner to create comparison
reports. Check out the targets in the cabal file.
It is trivial to add a new package. This is how a benchmark file for a streaming package looks like. Pull requests are welcome, we will be happy to help, just join the gitter chat and ask!