VMA over RHEL 7.x with inbox driver

This page will help you get VMA installed and running on a clean RHEL with the inbox drivers coming from Redhat.
Meaning, you will not need any additional MLNX_OFED driver besides what you get from Redhat and the latest VMA copy.
NOTE: Only versions 7.2 and up are supported.

1. Install RHEL-7.x

Follow the installation guide from Redhat

2. Verify a ConnectX-3 Pro or ConnectX-4 NIC is connected properly

$ lspci | grep Mellanox
02:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
03:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

3. Install libvma

RHEL-7.3 and above:

 yum install libvma

RHEL-7.2

1. ## Install Prerequisites:

Basic drivers for user space access for Ethernet and Infiniband:

yum groupinstall -y Infiniband

Additional devel packages for libvma compilation:

yum install libibverbs-devel.x86_64 librdmacm-devel.x86_64

Important! Don’t use libnl-devel, use libnl3-devel:

yum --setopt=group_package_types=optional install libnl3-devel  

NOTE: You may need to edit "/etc/yum.repos.d/redhat.repo" and enable "rhel-7-server-optional-rpms" if the above command does not work. Do not be tempted to use libnl-devel !

2. Download libvma source:

git clone https://github.com/Mellanox/libvma.git
cd libvma

3. Build libvma and install:

./autogen.sh
./configure --prefix=/usr --libdir=/usr/lib64
make -j install

4. Configure and load driver modules for ConnectX3:

echo options mlx4_core log_num_mgm_entry_size=-1 >> /etc/modprobe.d/mlnx.conf
dracut -f
modprobe -rv mlx4_en mlx4_ib mlx4_core
modprobe -v mlx4_en mlx4_ib mlx4_core rdma_ucm

NOTES: ConnectX4/LX inbox drivers coming in RHEL-7.3. You may need to put this in a boot script, as some kernels do not load the mlx4 driver with the correct parameters at boot

Common Configurations:

1. Memory settings for high performance and RDMA

Configure huge-pages for better libvma performance:

echo 1000000000 > /proc/sys/kernel/shmmax
echo 800 > /proc/sys/vm/nr_hugepages

Relax memlock restrictions for users allowing more direct memory mapping for HW.
Create file /etc/security/limits.d/rdma.conf with the following:

 # configuration for rdma tuning
 *       soft    memlock         unlimited
 *       hard    memlock         unlimited
 # rdma tuning end

Enable new limit for the user.
You will need to logout and back in and then run:

ulimit -l unlimited

2. Disable firewall service:

systemctl disable firewalld

3. Load libvma and run the app (as root):

LD_PRELOAD=libvma.so <test_app>

4. For running as user (as non root user):

Set cap_net_raw for the executable. For example to use sockperf:

setcap cap_net_raw=ep /usr/bin/sockperf

Set special permission for library (SET_UID) and place in standard location (which libvma is now already). These steps are required when using LD_PRELOAD with capabilities set on executable.

chmod u+s /usr/lib64/libvma.so.8*