After enabling SmartSwitch, when VM sends out traffic, it will be encap’ed with the destination IP set to the DP VIP. This makes all the VM traffic coming to our SmartSwitch for packet processing and transformation. | +| Upstream Service | The services that programs ENIs to smart switches, e.g., SDN controller. | +| UD / Update domain | It defines the groups of virtual machines and underlying physical hardware that can be rebooted at the same time. | +| FD / Fault domain | It defines the group of virtual machines that share a common power source and network switch. | +| SPOF | Single point of failure. | +| FP port | Front panel port. The ports on the front panel of the switch. | +| BP port | Back panel port. The ports that connects the NPU and DPU. | + +For more terminology, please refer to
However, the peer can be still connected, because in certain failure cases, we have to drive the DPU to standalone mode for mitigation. | +| SwitchingToActive | Connected and preparing to switch over to active. | No | Yes | Yes | Yes | Yes | No | SwitchingToActive state is a transient state to help old active moving to standby, hence it accepts flow sync from old active, as well as making decision and sync flow back, when old active moved to standby.
Bulk sync is not used in SwitchOver at all. | +| SwitchingToStandby | Connected, leaving active state and preparing to become standby. | Yes | Tunneled to pair | Tunneled to pair | Yes | No | No | SwitchingToStandby is a transient state to help new active moving from SwitchingToActive state to active state. It is identical to standby except responding NPU probe to tunneling traffic. This is needed to make sure we always only have 1 decider at any moment during the transition.
Bulk sync is not used in SwitchOver at all. | + +Here are the definitions of the columns above: + +* **Receive traffic from NPU**: ENI level NPU-to-DPU traffic control state. If yes, we will set this DPU as the next hop for the ENI, so NPU will forwarding traffic to it. +* **Make decision**: Will this DPU create new flows for new connections (not found in flow table, including both syn/non-syn cases, see "[Flow creation and initial sync](#113-flow-creation-and-initial-sync)" section) +* **Old flow handling**: When packets of existing flow arrive to this DPU, what will the DPU do? +* **Respond flow sync**: Will this DPU accept flow replication request from its paired DPU? +* **Init flow sync**: Will this DPU send replicate flow request to its paired DPU when new flow is created? +* **Init bulk sync**: Will this DPU start bulk sync in this state? + +### 7.2. State transition + +The state transition graph is shown as below: + +
(Desired Active) + participant S0N as Switch 0 NPU + participant S1N as Switch 1 NPU + participant S1D as Switch 1 DPU
(Desired Standby) + participant SA as All Switches + + S0N->>S0N: Move to Connecting state + S1N->>S1N: Move to Connecting state + + S0N->>S1N: Connect to peer and
send RequestVote with its
term and desired state + S1N->>S0N: Connect to peer and
send RequestVote with its
term and desired state + S0N->>S1N: Respond BecomeStandby + S1N->>S0N: Respond BecomeActive + + S0N->>S0N: Move to InitializingToActive state + S1N->>S1N: Move to InitializingToStandby state + S1N->>S1D: Activate standby role + S1N->>S0N: Send HAStateChanged event + + S0N->>S1N: Send BulkSyncDone event + S0N->>S0N: Move to Active state
and move to next term (Term 1) + S0N->>S0D: Activate active role with term 1 + S0N->>SA: Update ENI traffic forwarding
rule next hop to DPU0 + Note over S0N,SA: From now on, traffic will be forwarded
from all NPUs to DPU0 + S0N->>S1N: Send HAStateChanged event + + S1N->>S1N: Move to Standby state and update its term +``` + +#### 8.1.2. Launch with standalone peer + +While ENI is created, its peer might be already created and running. This can happen in multiple cases, such as recovering from planned/unplanned failure or ENI being moved from 1 switch to another. + +Once created, the new ENI will go through similar steps as above and join the HA set/pair. Say, DPU1 is one trying to start: + +
(Initial Standalone) + participant S0N as Switch 0 NPU + participant S1N as Switch 1 NPU + participant S1D as Switch 1 DPU
(Initial Dead) + participant SA as All Switches + + S1N->>S1N: Move to Connecting state + + S1N->>S0N: Connect to peer + S1N->>S1N: Move to Connected state + + S1N->>S0N: Send RequestVote with its
term and desired state + S0N->>S1N: Respond BecomeStandby + + S1N->>S1N: Move to InitializingToStandby state + S1N->>S1D: Activate standby role + S1N->>S0N: Send HAStateChanged event + + S0N->>S0N: Move to Active state and move to next term + S0N->>S0D: Activate active role with new term + S0N->>S0D: Start bulk sync + S0N->>S1N: Send HAStateChanged event + + S0N->>S1N: Send BulkSyncDone event + S1N->>S1N: Move to Standby state and update its term to match its peer +``` + +#### 8.1.3. Launch with no peer + +However, when the DPU is launching or ENI being created, its peer might not exist due to not being created yet or crashed not recovered yet (If peer ENI is running, we will be able to connect to it). + +In this case, we will go through the same checks as we handle the unplanned events and drive the new ENI to standalone state. + +The detailed steps are shown below: + +
(Dead) + participant S0N as Switch 0 NPU + participant S1N as Switch 1 NPU + participant S1D as Switch 1 DPU
(Initial Dead) + participant SA as All Switches + + S1N->>S1N: Move to Connecting state + + S1N->>S0N: Connect to peer and
send RequestVote with its
term and desired state + S0N->>S1N: Respond BecomeStandalone
with Reason PeerDead or PeerNotFound + + S1N->>S1N: Move to Standalone state and move to next term + S1N->>S1D: Activate Standalone role
with the new term + S1N->>SA: Update ENI traffic forwarding
rule next hop to DPU1 + Note over S1N,SA: From now on, traffic will be forwarded
from all NPUs to DPU1 +``` + +Afterwards, when DPU0 comes back, it will launch again, following the sequence of "[Launch with standalone peer](#812-launch-with-standalone-peer)" as above. + +### 8.2. Planned switchover + +Planned switchovers are usually done for reasons like maintenance. + +With the planned switchover, we are trying to achieve the following goals: + +1. Close to zero loss and to coordinate between the active and standby to achieve this goal (see "[Requirements, Assumption and SLA](#3-requirements-assumptions-and-sla)" section). +2. Avoid extra flow merge during and after switchover, which includes: + 1. Only 1 instance takes traffic and makes flow decisions at any moment during the entire switchover. + 1. Avoid using bulk sync for any flow replication. During the transition, all flow should be replicated inline. + +#### 8.2.1. Workflow + +Planned switchover starts from a standby node, because in order to avoid flow loss, we need to make sure we have 1 valid standby node that works. And here are the main steps: + +1. Upstream service set standby (DPU1) desired state to `Active` and current Active (DPU0) to `None` to initiate switchover. + +
(Initial Active) + participant S0N as Switch 0 NPU + participant S1N as Switch 1 NPU + participant S1D as Switch 1 DPU
(Initial Standby) + participant SA as All Switches + + U->>S1N: Set desired state to Active + U->>S0N: Set desired state to None + + S1N->>U: Send request switchover notification + U->>S1N: Set approved switchover id + + S1N->>S1N: Move to SwitchingToActive state + S1N->>S1D: Activate SwitchingToActive role + S1N->>S0N: Send SwitchOver to active ENI + + S0N->>S0N: Move to SwitchingToStandby state + S0N->>S0D: Activate Standby role + Note over S0N,S0D: From now on, traffic will be
tunneled from old active ENI to new + S0N->>S1N: Send HAStateChanged to standby ENI + + S1N->>S1N: Move to Active state + S1N->>S1D: Active Active role + S1N->>SA: Update ENI next hop to DPU1 + Note over S1N,SA: From now on, traffic will be forwarded
from all NPUs to DPU1 + S1N->>S0N: Send HAStateChanged to old active ENI + + S0N->>S0N: Move to Standby state + S0N->>S1N: Send HAStateChanged to new active ENI +``` + +#### 8.2.2. Packet racing during switchover + +Although we have ensure that there will be only 1 flow decider at all time, from state machine point of view. In practice, there still could be some packet racing during the switchover, because flow replication packet delay caused by flow insertion latency and network latency. + +```mermaid +sequenceDiagram + autonumber + + participant C as Customer + participant S0D as DPU0
(Initial Active) + participant S1D as DPU1
(Initial Standby) + + C->>S0D: First packet lands on DPU0 + S0D->>S0D: First packet creates flow on DPU0
and start replication + note over S0D,S1D: NPU shifted traffic to DPU1 + C->>S1D: Second packet lands on DPU1
before flow replication packet arrives + S1D->>S1D: Second packet creates another flow on DPU1 + S0D->>S1D: Flow replication packet arrives on DPU1 + note over S0D,S1D: Flow conflict!!! +``` + +This case will mostly happen on UDP traffic or other connection-less traffic, because TCP traffic will go through handshake which usually won't run this into problem. + +To avoid this issue causing problems, during switchover we will do 2 things below: + +1. In the case above, the flow replication packet will be dropped, because the flow is already created on DPU1. This prevents the flow being created on DPU0 and solves this problem. +2. In switchover step 2 above, we will request approval from our upstream service, which will ensure both DPU runs same set of policy and stop future policy programming until switchover is finished. This helps us ensure the flows that created on both sides are identical, if it happens due to bugs and etc. (see [stable decision assumption](#33-assumptions)) + +This helps us to avoid conflicting / stale flows being created during the switchover. + +#### 8.2.3. Failure handling during switchover + +During switchover, both NPU and DPU liveness detection are still running in the background. If any node died in the middle, instead of rolling states back, we would start unplanned event workflow right away by driving the HA pair into Standalone setup. + +See discussions for [unplanned events](#9-unplanned-events) below. + +### 8.3. Planned shutdown + +Planned shutdown is usually used for maintenance purposes, like upgrade. Same as planned switchover, we aim to have close to zero traffic loss (see "[Requirements, Assumption and SLA](#3-requirements-assumptions-and-sla)" section). + +#### 8.3.1. Planned shutdown standby node + +Because each ENI is programmed independently with our northbound interface, the request will start from the standby side. However, when planned shutdown standby node, we need to make sure the current active works, so we need to ask the active node to initiate the shutdown process. + +1. Upstream service update standby (DPU1) desired state to `Dead`. + +
(Initial Active) + participant S0N as Switch 0 NPU + participant S1N as Switch 1 NPU + participant S1D as Switch 1 DPU
(Initial Standby) + + U->>S1N: Set desired state to Dead + + S1N->>S0N: Send RequestShutdown to active ENI + + S0N->>S0N: Move to Standalone state + S0N->>S0D: Activate Standalone role + S0N->>S1N: Send ShutdownStandby to standby ENI + + S1N->>S1N: Move to Destroying state + S1N->>S1D: Shutdown ENI + S1N->>S0N: Send HAStateChanged to active ENI + S1N->>S1D: Pull DPU counters and wait for traffic to drain + + S1N->>S1D: Move to Dead state + S1N->>S0N: Send HAStateChanged to active ENI +``` + +#### 8.3.2. Planned shutdown active node + +To shutdown active node, the most important thing is to make sure the new active node will be running properly. Hence, this can be split into 2 steps: + +1. Switchover the current active to the desired active node. (See "[Planned switchover](#82-planned-switchover)" for detailed workflow) +2. Shutdown the old active node, which is running as standby now. (See "[Planned shutdown standby node](#831-planned-shutdown-standby-node)" for detailed workflow). + +### 8.4. ENI-level DPU isolation / Standalone pinning + +If we suspect an ENI pipeline is not working right and it is caused by unknown DPU hardware/software problem, we could try to isolate this DPU for this ENI. This can be achieved by 2 steps: + +1. Use [Planned Switchover](#82-planned-switchover) to move the active to the other DPU. +2. Set active node desired state to `Standalone` to drive this ENI into standalone setup. + +
**Peer**: Peer state update to Dead by hamgrd | **Local**: DPU state change by pmon
**Peer**: Peer state update to Connected | [Drive to standalone setup](#101-working-as-standalone-setup) | +| NPU failure | | | | | +| | hamgrd crash | ha container supervisor | hamgrd recovers | Withdraw SmartSwitch BGP routes, until hamgrd recovers | +| | Switch power down or kernel crash | PeerLost SAI notification on the peer switch | PeerConnected SAI notification | [Drive to standalone setup](#101-working-as-standalone-setup) | +| | Back panel port failure | PeerLost SAI notification on the peer switch | PeerConnected SAI notification | [Drive to standalone setup](#101-working-as-standalone-setup) | +| | PCIe failure | pmon | PCIe recovers | Treat as hard down and power cycle the DPU from NPU | + +## 10. Unplanned operations + +### 10.1. Working as standalone setup + +The standalone setup is used whenever we only have one side of DPU running or we detect large data path impact. Once we move a DPU into standalone state, it will stop flow replication, which reduces the chance the packet lands on bad links and getting dropped. + +#### 10.1.1. Design considerations + +The standalone setup is designed with the following considerations and principles: + +1. A ENI ***MUST*** only be served by a single DPU, just like Active-Standby setup. + * If both DPUs run as Standalone-Standalone pair, the incoming and outgoing traffic might land on 2 different DPUs. Since both DPU can make flow decisions, we will create unexpected flows and causing packet to be dropped. +2. Standalone setup is a best-effort mitigation. When network is dropping packet, no matter what we do, we will never be able to avoid the impact perfectly. We can only try to reduce the network hops to reduce the chance of packet drop. +3. Automated actions ***MUST*** be safe, otherwise it will cause more damage than good. If something we cannot handle automatically, we will fire alerts and wait for manual mitigation. +4. If we are running in standalone setup for a long time, we shall raise alerts, because it means something is going wrong and we cannot recover. +5. Driving into standalone setup will require 2 DPUs to work together. This communication is still done via HA control channel, because [as long as the peer DPU is running fine, we can always get a working control channel within a bounded time](#5223-ha-control-plane-channel-data-path-ha). + +Because of these, the standalone setup is designed to be Standalone-Standby pair. + +##### 10.1.1.1. Standalone setup vs bulk sync + +Once the HA pair starts to run as standalone setup, the inline sync will stop working, and their saved flows will start to diverge. + +1. New flows can be created on one side, but not the other. +2. Existing flows can be terminated on one side, but not the other. +3. Existing flows can be aged out on one side, but not the other, depending on how we manage the lifetime of the lows. +4. Due to policy updates, the same flow might get different packet transformations now, e.g., flow resimulation or flow recreation after policy update. + +And during recovery, we need to merge these 2 sets of flows back to one using "[bulk sync](#115-bulk-sync)". + +##### 10.1.1.2. Card-level vs ENI-level standalone setup + +There are 2 ways to implement standalone setup: Card-level and ENI-level. + +* Card-level standalone setup: All ENIs on the paired cards will be failover to one side and go into the `Standalone` state, so only one card can have ENIs running in `Standalone` state. +* ENI-level standalone setup: Each ENI can failover by itself and go into the `Standalone` state, so both cards can have ENIs running in `Standalone` state. + +Obviously, card-level standalone setup is easier to implement, but any ENI-level operations that involves standalone state, such as planned shutdown, will cause all ENIs failover to one side of the card, even if only one ENI is having problem. On the other hand, ENI-level standalone setup will be much more flexible, safe, but also harder to implement. + +However, which one is supported depends on the bulk sync ability. If ENI level bulk sync is supported, we can drive the HA pair into standalone setup at ENI level. Otherwise, we can only do it at card level. + +More details on bulk sync implementation can be found later in "[Bulk sync](#115-bulk-sync)" section. + +#### 10.1.2. Workflow triggers + +The standalone setup can be triggered by following types of signals. Each signal will work as a different type of pinning. Upon request (or any valid signal we consider), we start to drive the HA pair into standalone setup. And we will not come out of standalone setup, unless all signals being reset. + +| Problem | Trigger | Resolve signal | +| --- | --- | --- | +| Peer shutdown | Planned shutdown request | HAStateChanged with Connected state | +| Peer DPU lost | Peer lost SAI notification | Peer connected SAI notification | +| Peer DPU dead | HAStateChanged with dead peer | HAStateChanged with non-dead peer | +| High data plane packet drop rate | ENI-level data plane counters | ENI-level data plane counters | +| Manual Pinned | ENI-level DPU isolation | Isolation removed | +| Card pinned to standalone | Card pinned to standalone | Pinning removed | + +#### 10.1.3. Determine desired standalone setup + +The key of driving the HA pair into standalone setup is to determine which side should be the standalone. The steps are different for card-level and ENI-level standalone setup. + +##### 10.1.3.1. Determine card-level standalone setup + +Since many signals we are detecting are on ENI level, we need to merge them into card level first. If any trigger is hit, we will start the process as below: + +First, we need to check the DPU health signals: + +1. If the card is already pinned to standalone, we drive ourselves to standalone (no-op). +2. If the signals have "Peer DPU lost" or "Peer DPU dead", we drive ourselves to standalone. + * This covers the cases where the paired card or entire switch is dead, as well as manual operations. + +At this moment, both DPU should be running fine, so we start to check the ENI status and data path status. To ensure we have the latest state, we will send the `DPURequestEnterStandalone` message with aggregated signals and ENI states to the peer DPU. And upon receiving the message, we will run the following checks: + +1. If the signals from local DPU have "Card pinned to standalone", we return `Deny` to the peer DPU. +2. Check "Manual Pinned" for manual opertaions: + 1. If the signals from local DPU have "Manual Pinned", which the peer DPU doesn't have, we return `Deny` to the peer DPU. + 2. If the signals from peer DPU have "Manual Pinned", which the local DPU doesn't have, we return `Allow` to the peer DPU. + 3. If both sides have "Manual Pinned", we return `Deny` and raise alert after retrying certain amount of times. +3. Check "Peer shutdown" for planned shutdown: + 1. If the signals from local DPU have "Peer shutdown", we return `Deny` to the peer DPU. + 2. If the signals from peer DPU have "Peer shutdown", we return `Allow` to the peer DPU. + 3. If both sides have "Peer shutdown", we return `Deny` and raise alert after retrying certain amount of times. +4. If PreferredStandalone is set to local DPU, we return `Deny` to the peer DPU, otherwise, we return `Allow`. + * This covers the last case - high data plane packet drop rate. + * The reason we use predefined side here is that - there is no perfect solution in this case, and both side can send request at the same time. If we use information such as packet drop rate, we might get inconsistent data and making conflict decisions. So, instead of being too smart, we prefer to be stable. + +Once the we determined which DPU should be the standalone, we notify all ENIs in the DPU with "Card pinned to standalone" signal, so they can start to drive themselves to standalone. + +##### 10.1.3.2. Determine ENI-level standalone setup + +If ENI-level standalone is supported, each ENI decide on its own with the following steps: + +1. If the ENI is already pinned to standalone, we drive ourselves to standalone (no-op). +2. If the signals have "Manual Pinned", we drive ourselves to standalone. +3. If the signals have "Peer DPU lost" or "Peer DPU dead", we drive ourselves to standalone. + * This should cover the DPU or switch level failure. +4. Use primary election (maybe w/ local cached data) and use primary as the preferred standalone. + * This covers cases where both DPU and services are running fine, but network is having gray failure and killing flow replication packets. + +When doing these tests, we won’t change the HA state. Remaining stable and avoiding making bad moves are very important when we don’t have enough confidence. The worse case than not doing anything is breaking the current steady state and causing more damage. + +#### 10.1.4. Entering standalone setup + +Once we have decided who to be the standalone, we will drive the HA pair into Standalone-Standby pair. And no matter we are using card level or ENI level standalone setup, each ENI will drive itself to standalone state independently. + +##### 10.1.4.1. When peer is up + +This is usually caused by data plane gray failure. The detailed steps are listed as below: + +1. Firstly, packet loss will be detected by our counters in active (DPU0). + +
