any benchmark against cloudflare quiche / lsquic and https://github.com/mozilla/neqo?

[hiqsociety]

any benchmark against cloudflare quiche / lsquic and https://github.com/mozilla/neqo?

[djc]

We have some benchmarks in the repository but I'm not aware of any in-depth comparisons to other implementations. It's possible the interop runner has some basic benchmarks though.

[dzvon]

You may want to see this, https://interop.seemann.io/

[nmittler]

Just came across this (@dzvon thanks for the link). At first glance, it appears that many implementations (e.g. quic-go, quiche) show significantly higher throughput.

@djc any thoughts?

[Ralith]

What numbers are you looking at specifically? We haven't had the resources to keep that image very up to date, but at a glance it looks like even the old build that's up there is reasonably competitive.

[djc]

I don't have any comments, no. However, right now @aochagavia is doing a bunch of performance-oriented work. Quinn likely just needs more investment on the performance front, which is unlikely to come from @Ralith or me in the current situation. Obviously we're always happy to provide guidance to folks interested in benchmarking and improving performance.

Don't forget, too, that Quinn is mainly maintained by two volunteers whereas some of these other implementations have had much more time spent on them.

[nmittler]

@Ralith

What numbers are you looking at specifically?

Under the measurement section, just picking one out of a hat:

quic-go->quiche ~ 9443 kbps
quic-go->quinn ~ 7943 kbps

So around 16% lower throughput for quinn.

For the same row, all of the following servers show > 9000 kbps: quic-go, ngtcp2, quiche, picoquic, aioquic, nginx, xquic, lsquic, haproxy, s2n-quic (so 10 out of 16, with 2 that failed that particular test)

[nmittler]

@djc understood, thanks. Maybe we can at least update the image so we know the current state of things?

[Ralith]

quic-go->quiche ~ 9443 kbps
quic-go->quinn ~ 7943 kbps

Ah, I see. If I understand the test, it looks like that benchmark is showing quinn performing competitively as a client (receiver), but a bit slow as a server (sender). If the issue still exists with current versions, #1543 might be a significant help.

[Ralith]

Actually, PMTUD (#1510) might already fully account for this discrepancy, since it's a sender-side optimization that should give 1452/1200 = ~21% more bandwidth at a given packet rate (with per-packet costs likely dominating).

[nmittler]

Ah nice! @Ralith would it be possible to update the image used by https://interop.seemann.io/ to see the new results?

[moranno]

Ah nice! @Ralith would it be possible to update the image used by https://interop.seemann.io/ to see the new results?

It seems that the current performance of quinn is still at the bottom. Did you know that their latest tests use the latest version of quinn?

edit:

Just found the rusult, the test is still using the older version of quinn which define here:
https://github.com/marten-seemann/quic-interop-runner/blob/master/implementations.json

https://hub.docker.com/r/stammw/quinn-interop/tags

[djc]

A while back I worked on an upgrade to the quinn-interop repo, just turned it into a PR:

quinn-rs/quinn-interop#3

If anyone was able to test that this stuff actually works, we can maybe update it upstream?

[Ralith]

Yeah, we should update this. I've never managed to get docker to work right on my machine, unfortunately.