Interested in experiments with Mellanox cards

[stackedsax]

From a conversation with Martin:

we are going to experiment with Mellanox cards to see if we can similar results to what we get with Solarflare for reducing latency. We understand you use Mellanox cards on your Windows machines so this could be a nice perf boost if it works out.

I think we're actually more interested in this functionality on Linux boxes, since our future state will be Linux. A number of folks were interested in the subject, so adding them here so we can discuss: @UncleMattHope, @StephanDollberg, @chrisejones, @kastel.

[mjpt777]

We had previous been told that you are using Solarflare on Linux and Mellanox on Windows. If our information is out of date then it is great to know this. We would be happy to explore options that work for you in this space. @mikeb01 has done our DPDK implementation and is getting a Mellanox setup to try this.

[StephanDollberg]

Hi Martin,

Yes, we are currently in progress of migrating some of our low latency Solarflare+Linux estate to Mellanox so are definitely interested in this.

Windows is also generally on the way out so there is probably more Linux+Mellanox to come.

Stephan

[mikeb01]

Hi Stephan,

I've managed to make some progress with integrating Mellanox's VMA with Aeron. It did require a couple of small tricks to get it to work as a non-root user. This doesn't yet make use of the zero copy or socket xtreme polling (that can come later) and it will be Linux only at this point. I could put together a build for testing (not production) early in the New Year if you are interested?

Mike.

[StephanDollberg]

That would be cool yeah. I don't think we are in a particular rush on our side so don't worry if it's a bit later as well.

Would also be interesting to know what you are doing for the non-root workarounds. I think we were planning to use something based on ambient capabilities in the end. @chrisejones was working on that.

[chrisejones]

Yeah, that’s right. We’re running the Java media driver from a wrapper that has the required capabilities and use ambient capabilities, so that Java inherits them.

[mikeb01]

I've implemented this as a bindings plugin for the C driver and avoid requiring the LD_PRELOAD by loading libvma.so directly. Mellanox have some recommendations around this that suggest looking how it is done in sockperf (https://github.com/Mellanox/sockperf/blob/sockperf_v2/src/vma-redirect.cpp). The VMA bindings plugins has a copy of our existing plain UDP implementation tweaked to use the functions loaded from the VMA library. I haven't looked at the custom VMA APIs yet (e.g. zero copy receives and Socket Xtreme polling).

I ran a quick ping/pong benchmark, with the ping host running a Mellanox ConnectX-5 and the pong host with an Intel 82598EB, so I was only able to get the VMA optimisations on the ping host. This seems to improve RTT by around 5µs. I haven't tuned the OS on these boxes so there is some noise in the long tail.

[mikeb01]

A quick update on the VMA bindings. I've added support for zero copy recv now. I've looked into the socket xtreme polling, but I've put that on hold for the moment as the implementation will be more complicated. I've run into one issue with timestamps (Mellanox/libvma#976) which has been acknowledged.

Would you be interested in me putting together a binary for testing (with the caveat around the timestamps not working)? I would also be happy to do a quick call to go over any questions that you may have around set up.

[chrisejones]

Great thanks. We’re not currently using the C media driver. We’re hoping to find time to try it later in the year. We’ll contact you then to ask for binaries.

[mikeb01]

Another quick update, we've resolved the issue with the VMA binding and timestamping, so all is working correctly now. Let us know when you have when you are ready to start experimenting with the C driver and VMA support and we can make binaries available.