Throughput benchmark question

[Yiling-J]

@maypok86 I'm invisting the Theine's read75/write25 performance in this benchmark, but the code seems distribute read/write based on goroutine? I add some counters to show the ratio:

	var rcc atomic.Uint64
	var wcc atomic.Uint64

	runParallelBenchmark(b, func(pb *testing.PB) {
		index := int(fastrand() & uint32(mask))
		mc := atomic.AddUint64(&rc, 1)
		if benchCase.setPercentage*mc/100 != benchCase.setPercentage*(mc-1)/100 {
			for pb.Next() {
				wcc.Add(1)
				c.Set(keys[index&mask], values[index&mask])
				index++
			}
		} else {
			for pb.Next() {
				rcc.Add(1)
				c.Get(keys[index&mask])
				index++
			}
		}
	})
	fmt.Println("== read/total", float32(rcc.Load())/float32(rcc.Load()+wcc.Load()))

Result:

go test -run='^$' -cpu=8 -bench . -timeout=0 -benchtime=100000000x

cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkCache/zipf_otter_reads=75%,writes=25%-8         	== read/total 0.93773663
100000000	        36.23 ns/op	  27600574 ops/s
== read/total 1
BenchmarkCache/zipf_theine_reads=75%,writes=25%-8        	== read/total 0.86596227
100000000	       169.9 ns/op	   5886497 ops/s
== read/total 1
BenchmarkCache/zipf_ristretto_reads=75%,writes=25%-8     	== read/total 0.9112825
100000000	       102.8 ns/op	   9727388 ops/s
== read/total 1
BenchmarkCache/zipf_ccache_reads=75%,writes=25%-8        	== read/total 0.99518824
100000000	       153.7 ns/op	   6504329 ops/s
== read/total 1
BenchmarkCache/zipf_gcache_reads=75%,writes=25%-8        	== read/total 0.7469865
100000000	       359.8 ns/op	   2779626 ops/s
== read/total 1
BenchmarkCache/zipf_ttlcache_reads=75%,writes=25%-8      	== read/total 0.4470781
100000000	       553.6 ns/op	   1806246 ops/s
== read/total 1
BenchmarkCache/zipf_golang-lru_reads=75%,writes=25%-8    	== read/total 0.5607554
100000000	       269.5 ns/op	   3710782 ops/s
PASS

Because Otter's read speed is extremely fast, the read/total ratio exceeds 0.75 largely. I think 'read75/write25' indicates that clients send 75% read requests, so distributing traffic in that manner seems more appropriate?

[maypok86]

Hi, it seems you are repeating my path a few months ago :).

I also tried to go this way, trying to write a common benchmark for the main workload, but it all ended up with a huge headache due to changing the results when changing the parameters. The decisive factors for me were the excellent results of the cache with a random-based policy on such benchmarks and Ben's answers to my questions about caffeine benchmarks.

And now more specifically about benchmarks.

1. These benchmarks are a very close port of the caffeine benchmarks. Ristretto also used something very similar.
2. The description of the charts with the results describes exactly what is meant by them.
3. It can be perceived that benchmarks (reads=100%,writes=0%, reads=75%,writes=25%, reads=50%,writes=50% and reads=25%,writes=75%) simply show whether the cache hits are a bottleneck of the program or not.
4. The last (reads=0%,writes=100%) benchmark shows whether updates are a bottleneck of the program or not. For otter, this is actually a terrible workload, since updating is creating a new node and most of the optimizations just don't work. For example, otter has to put up with a queue that is slower than channels.
5. Usually, the problem with caches is precisely the speed of cache hits. The authors of dgraph, pebble (cockroachdb) and so on complained about this.

[Yiling-J]

@maypok86 Ah I see. That's reasonable. Thanks for the explanation.