[smartswitch] Update IP assignment flow (#1586)

* Update ip assignment flow Signed-off-by: Ze Gan <[email protected]> * Refactor hwsku.json to platform.json Signed-off-by: Ze Gan <[email protected]> * Remove interface-config.service and refactor midplane-netowrk.service for NPU Signed-off-by: Ze Gan <[email protected]> * Change interface name Signed-off-by: Ze Gan <[email protected]> * Change interface name Signed-off-by: Ze Gan <[email protected]> * Revert "Remove interface-config.service and refactor midplane-netowrk.service for NPU" This reverts commit b4d7707. * Leverage systemd-networkd to configure the midplane network Signed-off-by: Ze Gan <[email protected]> * Polish systemd configuration according to the comment Signed-off-by: Ze Gan <[email protected]> * Rename midplane-network.network Signed-off-by: Ze Gan <[email protected]> --------- Signed-off-by: Ze Gan <[email protected]>
sonic-net · Feb 28, 2024 · 11f42fa · 11f42fa
1 parent ed202c7
commit 11f42fa
Show file tree

Hide file tree

Showing 3 changed files with 369 additions and 47 deletions.
diff --git a/doc/smart-switch/ip-address-assigment/smart-switch-ip-address-assignment.md b/doc/smart-switch/ip-address-assigment/smart-switch-ip-address-assignment.md
@@ -11,14 +11,17 @@
   - [Requirements](#requirements)
     - [IP address assignment requirements](#ip-address-assignment-requirements)
   - [Architecture Design](#architecture-design)
-    - [Device data and HWSKU](#device-data-and-hwsku)
+    - [Device data and PLATFORM](#device-data-and-platform)
+      - [NPU platform.json](#npu-platformjson)
+      - [DPU platform.json](#dpu-platformjson)
     - [DPU and PCIe interfaces naming convention](#dpu-and-pcie-interfaces-naming-convention)
     - [Configuration generation for IP address assignment](#configuration-generation-for-ip-address-assignment)
       - [DPU IP address allocation](#dpu-ip-address-allocation)
       - [Smart Switch configuration](#smart-switch-configuration)
-    - [Midplane bridge configuration flow](#midplane-bridge-configuration-flow)
+    - [Midplane network configuration flow](#midplane-network-configuration-flow)
+      - [NPU](#npu)
+      - [DPU](#dpu)
     - [IP assignment flow](#ip-assignment-flow)
-    - [DPU](#dpu)
     - [High-Level Design](#high-level-design)
     - [SAI API](#sai-api)
     - [Configuration and management](#configuration-and-management)
@@ -39,6 +42,7 @@
 |  Rev  | Date  |       Author       | Change Description                     |
 | :---: | :---: | :----------------: | -------------------------------------- |
 |  0.1  |       | Oleksandr Ivantsiv | Initial version. IP address assignment |
+|  0.2  |       |       Ze Gan       | Update the services flow               |
 
 ## Scope ##
 
@@ -88,22 +92,27 @@ To implement the switch side functionality the following changes shall be done:
 - Config DB schema should be extended to include the following information:
   - Support of the new `SmartSwitch` subtype in DEVICE_METADATA config DB table. The new subtype shall allow to identify when the SONiC is running on the Smart Switch.
   - Support of the new `MID_PLANE_BRIDGE` table with the following information
-    - Midplane bridge configuration with the bridge IP address and list of the DPU PCIe interfaces that should be added to the bridge for the Smart Switch case.
+    - Midplane bridge configuration with the list of the DPU PCIe interfaces that should be added to the bridge for the Smart Switch case.
 - sonic-cfggen utility shall be extended to generate the following sample configuration based on the `t1-smartswitch` topology:
-  - Midplane bridge and DHCP server configuration based on the DPU information provided in the HWSKU file.
-- interface-config.service shall be extended to create a midplane bridge with the corresponding configuration in the Linux Kernel.
+  - Midplane bridge and DHCP server configuration based on the DPU information provided in the PLATFORM file.
+- systemd-networkd.service shall configure the midplane network of NPU and DPUs
+  - NPU side: create a midplane bridge with the corresponding configuration in the Linux Kernel.
+  - DPU side: start a DHCP client on the PCIe interface
+- midplane-network-(npu/dpu).service is a oneshot service that waits for the midplane network to be initialized.
 - DHCP server container should be included into the switch image. DHCP server feature should be enabled by default
 
-The midplane bridge configuration and DHCP server configuration shall be generated by sonic-cfggen utility. No user involvement is required.
+The midplane bridge configuration shall includes two stages: 1. Create the Linux Kernel bridge and assign a fixed IP address to it in the midplane-network-(npu/dpu).service 2. Add the PCIe interfaces to the midplane bridge. The DHCP server configuration shall be generated by sonic-cfggen utility. No user involvement is required.
 
-### Device data and HWSKU ##
+### Device data and PLATFORM ##
 
 The Smart Switch platform by default shall use the `t1-smartswitch`. The topology shall be used together with the DPU information for the sample configuration generation.
 
-The HWSKU configuration shall be extended to provide the information about the DPU available in the system.
+The PLATFORM configuration shall be extended to provide the information about the DPU available in the system.
 
 **Smart Switch**
 
+#### NPU platform.json
+
 ```json
 {
     "DPUS": {
@@ -113,6 +122,21 @@ The HWSKU configuration shall be extended to provide the information about the D
         "dpu1": {
             "midplane_interface":  "dpu1",
         }
+    },
+    "midplane_network": {
+        "bridge_name": "bridge-midplane",
+        "bridge_address": "169.254.200.254/24"
+    }
+}
+```
+
+#### DPU platform.json
+
+```json
+{
+    "DPU":{},
+    "midplane_network": {
+        "bridge_address": "169.254.200.254/24"
     }
 }
 ```
@@ -151,12 +175,11 @@ Based on the preset `t1-smartswitch` default topology the configuration generate
     },
     "MID_PLANE_BRIDGE" {
         "GLOBAL": {
-            "bridge" : "bridge_midplane",
-            "address": "169.254.200.254/24"
+            "bridge" : "bridge-midplane"
         }
     },
     "DHCP_SERVER_IPV4": {
-        "bridge_midplane": {
+        "bridge-midplane": {
             "gateway": "169.254.200.254",
             "lease_time": "infinite",
             "mode": "PORT",
@@ -165,12 +188,12 @@ Based on the preset `t1-smartswitch` default topology the configuration generate
         }
     },
     "DHCP_SERVER_IPV4_PORT": {
-        "bridge_midplane|dpu0": {
+        "bridge-midplane|dpu0": {
             "ips": [
                 "169.254.200.1"
             ]
         },
-        "bridge_midplane|dpu1": {
+        "bridge-midplane|dpu1": {
             "ips": [
                 "169.254.200.2"
             ]
@@ -199,53 +222,103 @@ The DHCP_SERVER_IPV4_PORT table includes the following:
 
 - For each DPU available in the system the IP address that should be assigned to the DPU.
 
-### Midplane bridge configuration flow ###
+### Midplane network configuration flow ###
 
-![image](./smart-switch-ip-assignment-midplane-bridge-configuration-flow.png)
+![image](./smart-switch-ip-assignment-midplane-network-configuration-flow.svg)
 
-1. Interface-config service when started consumes configuration from the config DB.
-2. Interface-config service renders interfaces configuration file that includes configuration for midplane bridge.
-3. Networking service consumes the interface configuration and configures the networking subsystem
+#### NPU ####
 
-After the transformations performed by the interface-config service the interfaces configuration will have the following entries:
+1. systemd-sonic-generator renders configuration file of systemd-networkd and midplane-network-npu.service according to the platform.json.
 
+``` text
+# bridge-midplane.netdev
+[NetDev]
+Name=bridge-midplane
+Kind=bridge
 ```
-auto bridge_midplane
-iface bridge_midplane inet static
-    bridge_ports Dpu0 Dpu1
-    address 169.254.200.254
-    netmask 255.255.255.0
-    network 169.254.200.0
-    broadcast 169.254.200.255
+
+``` text
+# bridge-midplane.network
+[Match]
+Name=bridge-midplane
+
+[Network]
+Address=169.254.200.254/24
 ```
 
-### IP assignment flow ###
+``` text
+# midplane-network-npu.network
+[Match]
+Name=dpu*
 
-![image](./smart-switch-ip-assignment-ip-assignment-flow.png)
+[Network]
+Bridge=bridge-midplane
+```
 
-- (1) sonic-cfggen based on hwsku.json renders midplane bridge and DHCP server configuration.
-- (2) sonic-cfggen pushes configuration into the Config DB.
-- (3-4) DHCP server and DHCP relay containers upon start consume configuration from the config DB and start listening for a requests from DHCP clients.
-- (5) DHCP client on the DPU sends a request over the eth0 interfaces. The request comes to the DHCP relay through the midplane bridge. DHCP relay inserts option 82 into the request with the information about the interface from which the request came. DHCP relay forwards the packet to the DHCP server.
-- (6) The DHCP server sends a reply with the IP configuration to the DHCP client on the DPU.
+2. systemd-networkd helps to create the bridge-midplane interface and assign the specific IP address according to above configuration. Meanwhile, systemd-networkd will monitor the DPU PCIe interface. Once the PCIe interface is created, it will automatically add it into the bridge-midplane.
+3. midplane-network-npu.service will be used to wait for the midplane bridge configured.
 
-### DPU ###
+``` text
+# midplane-network-npu.service
 
-In the Smart Switch DPU, the PCIe interface should be used as a default management interface for communication with the switch and the outside world.
+[Unit]
+Description=Midplane network service
+Requires=systemd-networkd.service
+After=systemd-networkd.service
+Before=database.service
 
-It is the vendors' responsibility to make sure that the PCIe interface that is connected to the switch has given the `eth0` name.
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/lib/systemd/systemd-networkd-wait-online -i bridge-midplane
 
-The IP address assignment for the default `eth0` management interface in the existing SONiC implementation is the following:
+[Install]
+WantedBy=multi-user.target
+```
+
+#### DPU ####
+
+In the DPU side, the steps are almost similar to NPU. But the generated configuration is different but simpler.
 
-- If the `MGMT_INTERFACE` table from Config DB is empty
-  - Query the IP address for the `eth0` interface from the DHCP server
-- Else
-  - For each entry
-    - Assign static IP configuration for the interface
+``` text
+# midplane-network-dpu.network
+[Match]
+Name=eth0-midplane
+
+[Network]
+DHCP=yes
+```
 
-To trigger the DPU SONiC to query the IP address configuration from the DHCP server the `MGMT_INTERFACE` in the config DB MUST remain empty. This also ensures that the DPU SONiC image can boot and bring the management interface up automatically with the default configuration.
+``` text
+# midplane-network-dpu.service
+
+[Unit]
+Description=Midplane network service
+Requires=systemd-networkd.service
+After=systemd-networkd.service
+Before=database.service
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/lib/systemd/systemd-networkd-wait-online -i eth0-midplane --timeout=600
+
+[Install]
+WantedBy=multi-user.target
+```
 
-It is the vendors' responsibility to ensure that the PCIe interfaces on the both switch and DPU sides are initialized with the same speed, duplex, and MTU and can communicate with no additional configuration.
+Step 2: systemd-networkd helps to start the dhcp client for eth0-midplane
+Setp 3: midplane-network-dpu.service will be used to wait for IP assigned for eth0-midplane from DHCP server. We expect the eth0-midplane can be configured within 10 mins.
+
+### IP assignment flow ###
+
+![image](./smart-switch-ip-assignment-ip-assignment-flow.png)
+
+- (1) sonic-cfggen based on platform.json renders midplane network and DHCP server configuration.
+- (2) sonic-cfggen pushes configuration into the Config DB.
+- (3-4) DHCP server and DHCP relay containers upon start consume configuration from the config DB and start listening for a requests from DHCP clients.
+- (5) DHCP client on the DPU sends a request over the eth0-midplane interfaces. The request comes to the DHCP relay through the midplane bridge. DHCP relay inserts option 82 into the request with the information about the interface from which the request came. DHCP relay forwards the packet to the DHCP server.
+- (6) The DHCP server sends a reply with the IP configuration to the DHCP client on the DPU.
 
 ### High-Level Design ###
 
@@ -295,7 +368,6 @@ The YANG model shown in this section is provided as a reference. The complete mo
                     }
                     description "Name of the midplane bridge";
 
-                    must "(current()/../ip_prefix)";
                 }
 
                 leaf ip_prefix {
@@ -353,7 +425,7 @@ The feature has minimal impact on the memory consumption. Overall it requires le
 2. Extend existing YANG model tests to cover MGMT_INTERFACE table changes.
 3. Add new tests to cover the handling of MID_PLANE_BRIDGE and DPUS tables in sonic-cfggen utility.
 4. Add new tests to cover DHCP server configuration generation for the Smart Switch in sonic-cfggen utility.
-5. Extend interface-config.service tests to verify midplane bridge configuration generation.
+5. Test to verify the midplane network configuration generation in NPU and DPU side.
 
 ### System Test cases ###
 

diff --git a/...ess-assigment/smart-switch-ip-assignment-midplane-bridge-configuration-flow.png b/...ess-assigment/smart-switch-ip-assignment-midplane-bridge-configuration-flow.png