feat: Add new parameter infiniband_ipoib_enhanced

The role was enforcing Connected mode on RHEL-like, without disabling
ipoib_enhanced, which led to errors.

Introduce a new parameter `infiniband_ipoib_enhanced` which only has
effect on RHEl-like OS. If set to False, the IPoIB interfaces will
default to Connected mode (previous behavior). If set to True (default),
the IPoIB interfaces will default to Datagram mode and Enhanced IPoIB.

For ConnectX-4 and later, Mellanox recommends to keep Enhanced IPoIB
enabled.

defaults/main.yml

@@ -69,3 +69,9 @@ infiniband_install_kernel_modules: True
 # given VM could run.
 #
 infiniband_guid_prefix: "4d:69:6c:61:00"
+
+# Configure Enhanced IPoIB
+# For ConnectX-4 and later, Mellanox recommends to keep Enhanced IPoIB enabled.
+# This parameter only has effect on RHEl-like OS. If set to False, the IPoIB
+# interfaces will default to Connected mode.
+infiniband_ipoib_enhanced: True

handlers/main.yml

@@ -11,6 +11,11 @@
     name: openibd.service
     state: started
 
+- name: Restart openibd
+  ansible.builtin.systemd:
+    name: openibd.service
+    state: restarted
+
 - name: Reboot to load new configurations
   throttle: 1
   ansible.builtin.reboot:

tasks/configure-RedHat.yml

@@ -9,3 +9,9 @@
     label: "Configure {{ item.iface }}"
   notify: Restart interface
   register: configure_ipoib
+
+- name: Configure ipoib_enhanced
+  ansible.builtin.lineinfile:
+    path: /etc/modprobe.d/ib_ipoib_enhanced.conf
+    line: "options ib_ipoib ipoib_enhanced={{ infiniband_ipoib_enhanced | bool }}"
+  notify: Restart openibd

templates/ifcfg-ib.j2

@@ -18,5 +18,7 @@ BOOTPROTO=static
 ONBOOT=yes
 IPV4_FAILURE_FATAL=yes
 IPV6INIT=no
+{% if not infiniband_ipoib_enhanced %}
 MTU=65520
 CONNECTED_MODE=yes
+{% endif %}