diff --git a/README.md b/README.md index 66e03fd..d48c072 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,13 @@ InfiniBand ========== -This role installs and configure InfiniBand interfaces. +This role installs and configures InfiniBand interfaces. Role Variables -------------- +### Installation + Define the APT or YUM/DNF repositories, with the path to the GPG key. # Default apt repository @@ -17,9 +19,26 @@ Define the APT or YUM/DNF repositories, with the path to the GPG key. # Mellanox GPG Key infiniband_gpg_key: 'http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox' +The name of the kernel headers package might change across distros. This can be +defined with: + + infiniband_kernel_headers_package: 'linux-headers' + +Some appliances like the NVIDIA DGX run their own software stack. When an +appliance already manages the installation of the Mellanox OFED drivers, there +is no need to configure additional repositories or to install the drivers. In +such cases, the parameters below can be set to False in the host inventory. +This is also true when using the kernel modules from the distro. + + infiniband_configure_repos: True + infiniband_install_kernel_modules: True + +### IPoIB + Define the list of interfaces to configure with IPoIB. Each item of the list -must define the interface name (iface), the offset from the default ipv4 -address and the CIDR prefix. +must define the interface name (iface), the offset from the default ipv4 address +and the CIDR prefix. If the list is not defined, the role will not configure any +IPoIB interface. infiniband_ipoib_interfaces: - iface: 'ib0' @@ -36,19 +55,37 @@ To compute the offset, use: print(f"offset: -{offset}") EOF -The name of the kernel headers package might change across distros. This can be -defined with: +### Virtualization - SR-IOV - infiniband_kernel_headers_package: 'linux-headers' +Configure the list of Mellanox HCAs with the parameters to apply. The pci_bus +must match the value in /sys/bus/pci/devices/ (e.g. +/sys/bus/pci/devices/0000:41:00.0/) -Some appliances like the NVIDIA DGX run their own software stack. When an -appliance already manages the installation of the Mellanox OFED drivers, there -is no need to configure additional repositories or to install the drivers. In -such cases, the parameters below can be set to False in the host inventory. -This is also true when using the kernel modules from the distro. + infiniband_hca_devices: + - device: mlx5_0 + pci_bus: '0000:41:00.0' + sriov_en: True + num_of_vfs: 8 + +Only parameters `SRIOV_EN` and `NUM_OF_VFS` are supported for now. + +Define the prefix to use for the 64-bits IB GUID of VFs. The role will define +the GUID with: + + - prefix (40 bits) + - "00" (8 bits) + - device ID (8 bits): item index in list infiniband_hca_devices above + - VF ID (8 bits): index of VF in range(infiniband_hca_devices[*].num_of_vfs) + +It is highly recommended to define a value different than the default if you +plan to configure more than one host with SR-IOV in your IB fabric. + +To allow failover in high availability configuration, make sure to use the same +infiniband_hca_devices and infiniband_guid_prefix for all hosts where a given VM +could run. + + infiniband_guid_prefix: "4d:69:6c:61:00" - infiniband_configure_repos: True - infiniband_install_kernel_modules: True Example Playbook ----------------