Skip to content

VMA over RHEL 7.x with inbox driver

Jump to bottom
FerasHamad edited this page Jul 4, 2017 · 24 revisions

This page will help you get VMA installed and running on a clean RHEL with the inbox drivers coming from Redhat.
Meaning, you will not need any additional MLNX_OFED driver besides what you get from Redhat and the latest VMA copy.
NOTE: Only versions 7.2 and up are supported.

1. Install RHEL-7.x

Follow the installation guide from Redhat

2. Verify a ConnectX-3 Pro or ConnectX-4 NIC is connected properly

$ lspci | grep Mellanox
02:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
03:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

3. Install libvma

RHEL-7.3 and above:

 yum install libvma

RHEL-7.2

  1. Install Prerequisites:
    Basic drivers for user space access for Ethernet and Infiniband:

    yum groupinstall -y Infiniband

    Additional devel packages for libvma compilation:

    yum install libibverbs-devel.x86_64 librdmacm-devel.x86_64

    Important! Don’t use libnl-devel, use libnl3-devel:

    yum --setopt=group_package_types=optional install libnl3-devel

    NOTE: You may need to edit "/etc/yum.repos.d/redhat.repo" and enable "rhel-7-server-optional-rpms" if the above command does not work. Do not be tempted to use libnl-devel !

  2. Download libvma source: git clone https://github.com/Mellanox/libvma.git cd libvma

  3. Build libvma and install: ./autogen.sh ./configure --prefix=/usr --libdir=/usr/lib64 make -j install

  4. Configure and load driver modules for ConnectX3: echo options mlx4_core log_num_mgm_entry_size=-1 >> /etc/modprobe.d/mlnx.conf dracut -f modprobe -rv mlx4_en mlx4_ib mlx4_core modprobe -v mlx4_en mlx4_ib mlx4_core rdma_ucm

    NOTES: ConnectX4/LX inbox drivers coming in RHEL-7.3. You may need to put this in a boot script, as some kernels do not load the mlx4 driver with the correct parameters at boot

Common Configurations:

  1. Memory settings for high performance and RDMA Configure huge-pages for better libvma performance:

    echo 1000000000 > /proc/sys/kernel/shmmax echo 800 > /proc/sys/vm/nr_hugepages

Relax memlock restrictions for users allowing more direct memory mapping for HW.
Create file /etc/security/limits.d/rdma.conf with the following:

 # configuration for rdma tuning
 *       soft    memlock         unlimited
 *       hard    memlock         unlimited
 # rdma tuning end

Enable new limit for the user.
You will need to logout and back in and then run:

ulimit -l unlimited

  1. Disable firewall service: systemctl disable firewalld

  2. Load libvma and run the app (as root): LD_PRELOAD=libvma.so <test_app>

  3. For running as user (as non root user): Set cap_net_raw for the executable. For example to use sockperf:

    setcap cap_net_raw=ep /usr/bin/sockperf

    Set special permission for library (SET_UID) and place in standard location (which libvma is now already). These steps are required when using LD_PRELOAD with capabilities set on executable.

    chmod u+s /usr/lib64/libvma.so.8*

Clone this wiki locally