[202405][DualToR] : FRR 8.5.4 regression in Dual ToR.

[vivekverma-arista]

Description

Steps to reproduce the issue:

1. Setup active-active DualTor

admin@gd426:~$ show mux config
SWITCH_NAME    PEER_TOR
-------------  ----------
gd377          10.1.0.32
port         state    ipv4             ipv6               cable_type     soc_ipv4
-----------  -------  ---------------  -----------------  -------------  ---------------
Ethernet0    auto     192.168.0.2/32   fc02:1000::2/128   active-active  192.168.0.3/32

admin@gd426:~$ show mux status                                                                                                                                               
PORT         STATUS    SERVER_STATUS    HEALTH    HWSTATUS    LAST_SWITCHOVER_TIME                                                                                           
-----------  --------  ---------------  --------  ----------  ---------------------------                                                                                    
Ethernet0    active    active           healthy   consistent  2024-Dec-10 07:33:58.400547          

admin@gd377:~$ show mux config
SWITCH_NAME    PEER_TOR
-------------  ----------
gd426          10.1.0.33
port         state    ipv4             ipv6               cable_type     soc_ipv4
-----------  -------  ---------------  -----------------  -------------  ---------------
Ethernet0    auto     192.168.0.2/32   fc02:1000::2/128   active-active  192.168.0.3/32


admin@gd377:~$ show mux status
PORT         STATUS    SERVER_STATUS    HEALTH    HWSTATUS    LAST_SWITCHOVER_TIME
-----------  --------  ---------------  --------  ----------  ---------------------------
Ethernet0    active    active           healthy   consistent  2024-Dec-10 07:37:50.616041

3. Assign IP address on the PTF side ( so that the ARP is answered )

root@23ead2109901:~# ifconfig eth0 192.168.0.2/21
root@23ead2109901:~# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9216
        inet 192.168.0.2  netmask 255.255.248.0  broadcast 192.168.7.255
        inet6 fe80::bc68:6eff:fef7:b000  prefixlen 64  scopeid 0x20<link>
        ether be:68:6e:f7:b0:00  txqueuelen 1000  (Ethernet)
        RX packets 3275174  bytes 234532539 (223.6 MiB)
        RX errors 0  dropped 37665  overruns 0  frame 0
        TX packets 2827103  bytes 195135239 (186.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

4. Configure static route

admin@gd426:~$ sudo config route add prefix 2.0.0.0/24 nexthop 192.168.0.2
admin@gd426:~$ show ip route static
:
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

S>* 2.0.0.0/24 [1/0] via 192.168.0.2, Vlan1000, weight 1, 00:00:07


admin@gd426:~$ ip -4 route | grep 2.0.0.0
2.0.0.0/24 nhid 245 via 192.168.0.2 dev Vlan1000 proto 196 metric 20 


admin@gd426:~$ show ip route 2.0.0.0 json
{
    "2.0.0.0/24": [
        {
            "destSelected": true,
            "distance": 1,
            "installed": true,
            "installedNexthopGroupId": 245,
            "internalFlags": 329,
            "internalNextHopActiveNum": 1,
            "internalNextHopNum": 1,
            "internalStatus": 16,
            "metric": 0,
            "nexthopGroupId": 245,
            "nexthops": [
                {
                    "active": true,
                    "afi": "ipv4",
                    "fib": true,
                    "flags": 3,
                    "interfaceIndex": 147,
                    "interfaceName": "Vlan1000",
                    "ip": "192.168.0.2",
                    "weight": 1
                }
            ],
            "offloaded": true,
            "prefix": "2.0.0.0/24",
            "prefixLen": 24,
            "protocol": "static",
            "selected": true,
            "table": 254,
            "tag": 1,
            "uptime": "00:01:26",
            "vrfId": 0,
            "vrfName": "default"
        }
    ]
}

5. Issue a config reload on one ToR and wait for it to become active again.

admin@gd426:~$ sudo config reload -y
Acquired lock on /etc/sonic/reload.lock
Disabling container monitoring ...
Stopping SONiC target ...
The config file - doesn't exist
Running command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment
Restarting SONiC target ...
Enabling container monitoring ...
Reloading Monit configuration ...
Reinitializing monit daemon
Released lock on /etc/sonic/reload.lock


admin@gd426:~$ date
Tue Dec 10 07:47:38 AM UTC 2024

admin@gd426:~$ date
Tue Dec 10 07:48:44 AM UTC 2024

admin@gd426:~$ show mux status
PORT         STATUS    SERVER_STATUS    HEALTH    HWSTATUS    LAST_SWITCHOVER_TIME
-----------  --------  ---------------  --------  ----------  ---------------------------
Ethernet0    active    active           healthy   consistent  2024-Dec-10 07:48:07.059682

6. Do show ip route static

admin@gd426:~$ show ip route static
:
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

S>r 2.0.0.0/24 [1/0] via 192.168.0.2, Vlan1000, weight 1, 00:02:10

Describe the results you received:

After config reload in this case the route is in rejected state and neither it is programmed in the hardware.

This can also be reproduced by simply running the sonic-mgmt test route/test_static_route.py::test_static_route_ecmp.

Please note that this issue is not seen with this test in active-standby, becasue we do not run ICMP responder and there is no switchover during config reload, however we have manually tested by turning on ICMP responder, that the static route goes into rejected state when standby ToR becomes active after config reload.

Describe the results you expected:

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[vivekverma-arista]

Underlying issue in 8.5.4: #17345
During the LAG member flap the route is not reinstated when the member
has come up. This was a regression in 8.5.4. For which a temporary
workaround was added
#17344.

Later fixes were backported from later releases of FRR to 8.5.4 in the
form of patches in
#18953. This fixes the
case for LAG.

Looks like the problem is not specific to LAG but applicable to any link
flap. Config reload also leads to link flap.

The transition from active to standby and then standby to active is
similar to this case. The only difference in this case is that when the
MUX link goes down, a tunnel route gets installed in it's place rather
than simply the deletion of route. This case needs some additonal
handling in newer version of FRR ( or maybe in SONiC given something
different happens in FRR 8.5.4 for this case).

[tjchadaga]

@sudhanshukumar22 to update findings on current testing

[vkjammala-arista]

One interesting observation is, In the issue state (i.e when static route is in rejected state after config reload), if we remove that rejected static route and wait for atleast 3 mins, then re-adding of the static route is going through successfully.

This 3 mins comes from zebra nexthop-group keep.

[sudhanshukumar22]

sonic# show ip route static
Codes: K - kernel route, C - connected, L - local, S - static,
R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric, t - Table-Direct,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

S>* 2.0.0.0/24 [1/0] via 46.1.1.3, Vlan100, weight 1, 00:01:05

•              via 47.1.1.3, Vlan101, weight 1, 00:01:05
  

sonic# exit
root@sonic:# config save -y
Running command: /usr/local/bin/sonic-cfggen -d --print-data > /etc/sonic/config_db.json
root@sonic:# show ip route 2.0.0.0 json
{
"2.0.0.0/24": [
{
"destSelected": true,
"distance": 1,
"installed": true,
"installedNexthopGroupId": 337,
"internalFlags": 329,
"internalNextHopActiveNum": 2,
"internalNextHopNum": 2,
"internalStatus": 16,
"metric": 0,
"nexthopGroupId": 337,
"nexthops": [
{
"active": true,
"afi": "ipv4",
"fib": true,
"flags": 3,
"interfaceIndex": 125,
"interfaceName": "Vlan100",
"ip": "46.1.1.3",
"weight": 1
},
{
"active": true,
"afi": "ipv4",
"fib": true,
"flags": 3,
"interfaceIndex": 126,
"interfaceName": "Vlan101",
"ip": "47.1.1.3",
"weight": 1
}
],
"offloaded": true,
"prefix": "2.0.0.0/24",
"prefixLen": 24,
"protocol": "static",
"selected": true,
"table": 254,
"tag": 1,
"uptime": "00:01:34",
"vrfId": 0,
"vrfName": "default"
}
]
}
root@sonic:# config reload -y
Acquired lock on /etc/sonic/reload.lock
Disabling container monitoring ...
Stopping SONiC target ...
Running command: /usr/local/bin/sonic-cfggen -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db.json --write-to-db
Running command: /usr/local/bin/db_migrator.py -o migrate
Running command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment
Restarting SONiC target ...
Enabling container monitoring ...
Reloading Monit configuration ...
Reinitializing monit daemon
Released lock on /etc/sonic/reload.lock
root@sonic:# vtysh

Hello, this is FRRouting (version 10.0.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

2024/10/16 08:48:48 [YDG3W-JND95] FD Limit set: 1048576 is stupidly large. Is this what you intended? Consider using --limit-fds also limiting size to 100000
sonic# show ip route static
Codes: K - kernel route, C - connected, L - local, S - static,
R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric, t - Table-Direct,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

S>* 2.0.0.0/24 [1/0] via 46.1.1.3, Vlan100, weight 1, 00:00:48

•              via 47.1.1.3, Vlan101, weight 1, 00:00:48
  

sonic# exit
root@sonic:~# config route ?
Usage: config route [OPTIONS] COMMAND [ARGS]...

route-related configuration tasks

Options:
-?, -h, --help Show this message and exit.

Commands:
add Add route command
del Del route command
root@sonic:~# config route del ?
Usage: config route del [OPTIONS] prefix [vrf <vrf_name>] <A.B.C.D/M> nexthop
<[vrf <vrf_name>] <A.B.C.D>>|<dev <dev_name>>
Try "config route del -h" for help.

Error: argument is not in pattern prefix [vrf <vrf_name>] <A.B.C.D/M> nexthop <[vrf <vrf_name>] <A.B.C.D>>|<dev <dev_name>>!
root@sonic:# config route del prefix 2.0.0.0/24 nexthop 47.1.1.3
root@sonic:# vtysh

Hello, this is FRRouting (version 10.0.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

2024/10/16 08:50:11 [YDG3W-JND95] FD Limit set: 1048576 is stupidly large. Is this what you intended? Consider using --limit-fds also limiting size to 100000
sonic# show ip route static
Codes: K - kernel route, C - connected, L - local, S - static,
R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric, t - Table-Direct,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

S>* 2.0.0.0/24 [1/0] via 46.1.1.3, Vlan100, weight 1, 00:00:09
sonic# exit
root@sonic:# config save -y
Running command: /usr/local/bin/sonic-cfggen -d --print-data > /etc/sonic/config_db.json
root@sonic:# show ip route 2.0.0.0 json
{
"2.0.0.0/24": [
{
"destSelected": true,
"distance": 1,
"installed": true,
"installedNexthopGroupId": 484,
"internalFlags": 329,
"internalNextHopActiveNum": 1,
"internalNextHopNum": 1,
"internalStatus": 16,
"metric": 0,
"nexthopGroupId": 484,
"nexthops": [
{
"active": true,
"afi": "ipv4",
"fib": true,
"flags": 3,
"interfaceIndex": 180,
"interfaceName": "Vlan100",
"ip": "46.1.1.3",
"weight": 1
}
],
"offloaded": true,
"prefix": "2.0.0.0/24",
"prefixLen": 24,
"protocol": "static",
"selected": true,
"table": 254,
"tag": 1,
"uptime": "00:00:42",
"vrfId": 0,
"vrfName": "default"
}
]
}
root@sonic:# show ip route 2.0.0.0 json
{
"2.0.0.0/24": [
{
"destSelected": true,
"distance": 1,
"installed": true,
"installedNexthopGroupId": 484,
"internalFlags": 329,
"internalNextHopActiveNum": 1,
"internalNextHopNum": 1,
"internalStatus": 16,
"metric": 0,
"nexthopGroupId": 484,
"nexthops": [
{
"active": true,
"afi": "ipv4",
"fib": true,
"flags": 3,
"interfaceIndex": 180,
"interfaceName": "Vlan100",
"ip": "46.1.1.3",
"weight": 1
}
],
"offloaded": true,
"prefix": "2.0.0.0/24",
"prefixLen": 24,
"protocol": "static",
"selected": true,
"table": 254,
"tag": 1,
"uptime": "00:01:38",
"vrfId": 0,
"vrfName": "default"
}
]
}
root@sonic:# vtysh

Hello, this is FRRouting (version 10.0.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

2024/10/16 08:51:57 [YDG3W-JND95] FD Limit set: 1048576 is stupidly large. Is this what you intended? Consider using --limit-fds also limiting size to 100000
sonic# show ip route static
Codes: K - kernel route, C - connected, L - local, S - static,
R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric, t - Table-Direct,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

S>* 2.0.0.0/24 [1/0] via 46.1.1.3, Vlan100, weight 1, 00:01:55
sonic# exit
root@sonic:# config reload -y
Acquired lock on /etc/sonic/reload.lock
Disabling container monitoring ...
Stopping SONiC target ...
Running command: /usr/local/bin/sonic-cfggen -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db.json --write-to-db
Running command: /usr/local/bin/db_migrator.py -o migrate
Running command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment
Restarting SONiC target ...
Enabling container monitoring ...
Reloading Monit configuration ...
Reinitializing monit daemon
Released lock on /etc/sonic/reload.lock
root@sonic:# vtysh
Exiting: failed to connect to any daemons.
root@sonic:# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a5653393b1e3 docker-fpm-frr:latest "/usr/bin/docker_ini…" 3 days ago Up 8 seconds bgp
d219f8061157 docker-nat:latest "/usr/local/bin/supe…" 3 days ago Exited (137) 3 days ago nat
cf05e67072b8 e01552ea504a "/usr/bin/docker_ini…" 3 days ago Exited (0) 3 days ago dhcp_relay
24890567fafc docker-snmp:latest "/usr/local/bin/supe…" 5 days ago Exited (0) 46 seconds ago snmp
6710d9b18534 docker-platform-monitor:latest "/usr/bin/docker_ini…" 5 days ago Exited (0) 48 seconds ago pmon
810fb67731fc docker-sonic-mgmt-framework:latest "/usr/local/bin/supe…" 5 days ago Exited (0) 50 seconds ago mgmt-framework
a2ba7cb2afab docker-lldp:latest "/usr/bin/docker-lld…" 5 days ago Exited (137) 41 seconds ago lldp
e18500079639 docker-sonic-gnmi:latest "/usr/local/bin/supe…" 5 days ago Exited (0) 51 seconds ago gnmi
a8dba1c05920 docker-router-advertiser:latest "/usr/bin/docker-ini…" 5 days ago Up 13 seconds radv
ab69de19a50e docker-syncd-brcm:latest "/usr/local/bin/supe…" 5 days ago Up 14 seconds syncd
48b65a20dbbc docker-teamd:latest "/usr/local/bin/supe…" 5 days ago Up 14 seconds teamd
a9e3ce8e62b7 docker-orchagent:latest "/usr/bin/docker-ini…" 5 days ago Up 17 seconds swss
8cc2372d4c65 docker-eventd:latest "/usr/local/bin/supe…" 5 days ago Up 17 seconds eventd
ea72b56e6110 docker-database:latest "/usr/local/bin/dock…" 5 days ago Up About an hour database
root@sonic:# vtysh

Hello, this is FRRouting (version 10.0.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

2024/10/16 08:53:10 [YDG3W-JND95] FD Limit set: 1048576 is stupidly large. Is this what you intended? Consider using --limit-fds also limiting size to 100000
sonic# show ip route static
Codes: K - kernel route, C - connected, L - local, S - static,
R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric, t - Table-Direct,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

S>* 2.0.0.0/24 [1/0] via 46.1.1.3, Vlan100, weight 1, 00:00:14
sonic# show ip route static

[sudhanshukumar22]

I tried in a 2 node setup with one or two nexthops and issuing config relaod, the issue was not happening(see logs above). How can I create a DuelTOR setup ?