Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

inconsistent route propagation #351

Open
rcgoodfellow opened this issue Aug 27, 2024 · 0 comments
Open

inconsistent route propagation #351

rcgoodfellow opened this issue Aug 27, 2024 · 0 comments
Assignees
@rcgoodfellow
Labels
Bug
Milestone
13

Comments

@rcgoodfellow
Copy link
Collaborator

An a4x2 testbed run came up in a state where not all advertised prefixes had been propagated to all server routers.

On the node g3 we see

root@g3:~# /opt/oxide/mg-ddm/ddmadm get-prefixes
Destination                Next Hop                   Path
fdb0:a840:2500:5::/64      fe80::aa40:25ff:fe08:422c  oxz_switch g2
fd00:1122:3344:102::/64    fe80::aa40:25ff:fe08:422c  oxz_switch g1
fd4f:f6e0:7a9a:e0df::1/64  fe80::aa40:25ff:fe08:422c  oxz_switch
fdb0:a840:2500:3::/64      fe80::aa40:25ff:fe08:422c  oxz_switch g1
fd00:1122:3344:2::/64      fe80::aa40:25ff:fe08:422c  oxz_switch g1
fd00:1122:3344:1::/64      fe80::aa40:25ff:fe14:d250  oxz_switch g0
fdb0:a840:2500:1::/64      fe80::aa40:25ff:fe14:d250  oxz_switch g0
fd00:1122:3344:2::/64      fe80::aa40:25ff:fe14:d250  oxz_switch g1
fd00:1122:3344:102::/64    fe80::aa40:25ff:fe14:d250  oxz_switch g1
fd00:1122:3344:101::/64    fe80::aa40:25ff:fe14:d250  oxz_switch g0
fdb0:a840:2500:3::/64      fe80::aa40:25ff:fe14:d250  oxz_switch g1
fd8e:6d1b:ddb4:7f4b::1/64  fe80::aa40:25ff:fe14:d250  oxz_switch
fdb0:a840:2500:5::/64      fe80::aa40:25ff:fe14:d250  oxz_switch g2

root@g3:~# /opt/oxide/mg-ddm/ddmadm get-peers
Interface  Host        Address                    Kind     Status
3          oxz_switch  fe80::aa40:25ff:fe08:422c  Transit  Active
4          oxz_switch  fe80::aa40:25ff:fe14:d250  Transit  Active

So we are missing a route to fd00:1122:3344:10::/64viafe80::aa40:25ff:fe08:422c`. Let's look at that transit routers tables.

This is switch0

root@oxz_switch:~# ipadm | grep 422c
tfportrear3_0/ll  addrconf ok           fe80::aa40:25ff:fe08:422c%tfportrear3_0/10

we have a route to fd00:1122:3344:101::/64, so that should be getting redistributed to g3`.

root@oxz_switch:~# ddmadm get-prefixes
Destination              Next Hop                Path
fd00:1122:3344:1::/64    fe80::aa40:25ff:fe00:1  g0
fdb0:a840:2500:1::/64    fe80::aa40:25ff:fe00:1  g0
fd00:1122:3344:101::/64  fe80::aa40:25ff:fe00:1  g0
fdb0:a840:2500:5::/64    fe80::aa40:25ff:fe00:5  g2
fd00:1122:3344:2::/64    fe80::aa40:25ff:fe00:3  g1
fd00:1122:3344:102::/64  fe80::aa40:25ff:fe00:3  g1
fdb0:a840:2500:3::/64    fe80::aa40:25ff:fe00:3  g1
fd00:1122:3344:3::/64    fe80::aa40:25ff:fe00:7  g3
fdb0:a840:2500:7::/64    fe80::aa40:25ff:fe00:7  g3
fd00:1122:3344:103::/64  fe80::aa40:25ff:fe00:7  g3

Here is the log from g3 for interactions with switch0

root@g3:~# cat $(svcs -L mg-ddm) | grep vioif1 | looker
17:05:04.095Z INFO slog-rs: [vioif1] sm initialized with addr fe80::aa40:25ff:fe00:7 on if vioif1 index 3
17:05:04.095Z DEBG slog-rs: [vioif1] starting discovery handler
17:05:04.542Z WARN slog-rs: [vioif1] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fdb0:a840:2500:7::, width: 64 }}))
17:07:03.679Z WARN slog-rs: [vioif1] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd00:1122:3344:103::, width: 64 }}))
17:07:12.317Z INFO slog-rs: [vioif1] nbr is fe80::aa40:25ff:fe08:422c@oxz_switch transit
17:07:12.461Z DEBG slog-rs: [vioif1] transition solicit -> exchange
17:07:12.461Z INFO slog-rs: [vioif1] exchange: listening on [fe80::aa40:25ff:fe00:7]:56797
17:07:12.461Z INFO slog-rs: [vioif1] waiting for exchange server to start
17:07:14.438Z WARN slog-rs: [vioif1] exchange pull: hyper error: error trying to connect: tcp connect error: Connection refused (os error 146)
17:07:14.462Z INFO slog-rs: [vioif1] sending 5 routes to illumos
17:07:14.462Z INFO slog-rs: [vioif1] adding route fdb0:a840:2500:1:: -> fe80::aa40:25ff:fe08:422c
17:07:14.465Z INFO slog-rs: [vioif1] adding route fd00:1122:3344:101:: -> fe80::aa40:25ff:fe08:422c
17:07:14.465Z INFO slog-rs: [vioif1] adding route fdb0:a840:2500:5:: -> fe80::aa40:25ff:fe08:422c
17:07:14.468Z INFO slog-rs: [vioif1] adding route fd00:1122:3344:102:: -> fe80::aa40:25ff:fe08:422c
17:07:14.468Z INFO slog-rs: [vioif1] adding route fdb0:a840:2500:3:: -> fe80::aa40:25ff:fe08:422c
17:07:14.476Z INFO slog-rs: [vioif1] removing 0 routes from illumos
17:07:14.508Z INFO slog-rs: [vioif1] sending 2 routes to illumos
17:07:14.508Z INFO slog-rs: [vioif1] adding route fd00:1122:3344:102:: -> fe80::aa40:25ff:fe08:422c
17:07:14.509Z INFO slog-rs: [vioif1] adding route fdb0:a840:2500:3:: -> fe80::aa40:25ff:fe08:422c
17:07:14.511Z INFO slog-rs: [vioif1] removing 0 routes from illumos
17:07:14.518Z INFO slog-rs: [vioif1] sending 1 routes to illumos
17:07:14.518Z INFO slog-rs: [vioif1] adding route fdb0:a840:2500:5:: -> fe80::aa40:25ff:fe08:422c
17:07:14.518Z INFO slog-rs: [vioif1] removing 0 routes from illumos
17:07:14.768Z INFO slog-rs: [vioif1] sending 2 routes to illumos
17:07:14.993Z INFO slog-rs: [vioif1] adding route fd00:1122:3344:101:: -> fe80::aa40:25ff:fe08:422c
17:07:14.993Z INFO slog-rs: [vioif1] adding route fdb0:a840:2500:1:: -> fe80::aa40:25ff:fe08:422c
17:07:14.993Z INFO slog-rs: [vioif1] removing 0 routes from illumos
17:07:18.566Z INFO slog-rs: [vioif1] sending 1 routes to illumos
17:07:18.566Z INFO slog-rs: [vioif1] adding route fd4f:f6e0:7a9a:e0df::1 -> fe80::aa40:25ff:fe08:422c
17:07:18.567Z INFO slog-rs: [vioif1] removing 0 routes from illumos
17:07:18.576Z INFO slog-rs: [vioif1] adding tunnel route 0.0.0.0/0 -[99]-> fd4f:f6e0:7a9a:e0df::1
17:07:18.749Z INFO slog-rs: [vioif1] removing tunnel route 0.0.0.0/0 -[99]-> fd4f:f6e0:7a9a:e0df::1
17:07:19.426Z INFO slog-rs: [vioif1] sending 1 routes to illumos
17:07:19.426Z INFO slog-rs: [vioif1] adding route fd00:1122:3344:1:: -> fe80::aa40:25ff:fe08:422c
17:07:19.427Z INFO slog-rs: [vioif1] removing 0 routes from illumos
17:07:20.359Z INFO slog-rs: [vioif1] sending 1 routes to illumos
17:07:20.359Z INFO slog-rs: [vioif1] adding route fd00:1122:3344:2:: -> fe80::aa40:25ff:fe08:422c
17:07:20.360Z INFO slog-rs: [vioif1] removing 0 routes from illumos
17:07:20.897Z INFO slog-rs: [vioif1] adding tunnel route 0.0.0.0/0 -[99]-> fd4f:f6e0:7a9a:e0df::1
17:15:49.795Z WARN slog-rs: [vioif1] neighbor oxz_switch@fe80::aa40:25ff:fe08:422c missed solicit interval
17:15:53.805Z WARN slog-rs: [vioif1] neighbor oxz_switch@fe80::aa40:25ff:fe08:422c missed solicit interval
17:15:59.284Z WARN slog-rs: [vioif1] neighbor oxz_switch@fe80::aa40:25ff:fe08:422c missed solicit interval
17:16:19.215Z WARN slog-rs: [vioif1] neighbor oxz_switch@fe80::aa40:25ff:fe08:422c missed solicit interval
17:17:12.812Z WARN slog-rs: [vioif1] neighbor oxz_switch@fe80::aa40:25ff:fe08:422c missed solicit interval
17:17:31.422Z INFO slog-rs: [vioif1] sending 0 routes to illumos
17:17:33.900Z INFO slog-rs: [vioif1] removing 3 routes from illumos
17:17:33.900Z INFO slog-rs: [vioif1] removing route fdb0:a840:2500:1:: -> fe80::aa40:25ff:fe08:422c
17:17:33.900Z INFO slog-rs: [vioif1] removing route fd00:1122:3344:101:: -> fe80::aa40:25ff:fe08:422c
17:17:33.900Z INFO slog-rs: [vioif1] removing route fd00:1122:3344:1:: -> fe80::aa40:25ff:fe08:422c
17:17:49.783Z INFO slog-rs: [vioif1] sending 0 routes to illumos
17:17:50.127Z INFO slog-rs: [vioif1] removing 3 routes from illumos
17:17:50.127Z INFO slog-rs: [vioif1] removing route fdb0:a840:2500:3:: -> fe80::aa40:25ff:fe08:422c
17:17:50.127Z INFO slog-rs: [vioif1] removing route fd00:1122:3344:2:: -> fe80::aa40:25ff:fe08:422c
17:17:50.127Z INFO slog-rs: [vioif1] removing route fd00:1122:3344:102:: -> fe80::aa40:25ff:fe08:422c
17:18:11.154Z INFO slog-rs: [vioif1] sending 3 routes to illumos
17:18:11.154Z INFO slog-rs: [vioif1] adding route fdb0:a840:2500:3:: -> fe80::aa40:25ff:fe08:422c
17:18:12.033Z INFO slog-rs: [vioif1] adding route fd00:1122:3344:2:: -> fe80::aa40:25ff:fe08:422c
17:18:12.033Z INFO slog-rs: [vioif1] adding route fd00:1122:3344:102:: -> fe80::aa40:25ff:fe08:422c
17:18:12.034Z INFO slog-rs: [vioif1] removing 0 routes from illumos

Here are the logs from switch0 for interactions with g3

root@oxz_switch:~# cat $(svcs -L mg-ddm) | grep 'tfportrear3_0' | looker
17:15:02.996Z WARN slog-rs: [tfportrear3_0] neighbor g3@fe80::aa40:25ff:fe00:7 missed solicit interval
17:15:13.012Z WARN slog-rs: [tfportrear3_0] neighbor g3@fe80::aa40:25ff:fe00:7 missed solicit interval
17:15:19.080Z WARN slog-rs: [tfportrear3_0] neighbor g3@fe80::aa40:25ff:fe00:7 missed solicit interval
17:15:43.246Z WARN slog-rs: [tfportrear3_0] neighbor g3@fe80::aa40:25ff:fe00:7 missed solicit interval
17:15:49.505Z WARN slog-rs: [tfportrear3_0] neighbor g3@fe80::aa40:25ff:fe00:7 missed solicit interval
17:15:55.547Z WARN slog-rs: [tfportrear3_0] neighbor g3@fe80::aa40:25ff:fe00:7 missed solicit interval
17:15:57.548Z WARN slog-rs: [tfportrear3_0] neighbor g3@fe80::aa40:25ff:fe00:7 missed solicit interval
17:16:03.696Z WARN slog-rs: [tfportrear3_0] neighbor g3@fe80::aa40:25ff:fe00:7 missed solicit interval
17:16:20.539Z WARN slog-rs: [tfportrear3_0] neighbor g3@fe80::aa40:25ff:fe00:7 missed solicit interval
17:17:19.847Z INFO slog-rs: [tfportrear3_0] push from fe80::aa40:25ff:fe00:7: Update {
        underlay: Some(
            UnderlayUpdate {
                announce: {},
                withdraw: {
                    PathVector {
                        destination: Ipv6Net {
                            addr: fd00:1122:3344:102::,
                            width: 64,
                        },
                        path: [
                            "g1",
                            "oxz_switch",
                        ],
                    },
                    PathVector {
                        destination: Ipv6Net {
                            addr: fdb0:a840:2500:3::,
                            width: 64,
                        },
                        path: [
                            "g1",
                            "oxz_switch",
                        ],
                    },
                    PathVector {
                        destination: Ipv6Net {
                            addr: fd00:1122:3344:2::,
                            width: 64,
                        },
                        path: [
                            "g1",
                            "oxz_switch",
                        ],
                    },
                },
            },
        ),
        tunnel: None,
    }
17:17:23.216Z ERRO slog-rs: [tfportrear3_0] peer request timeout to http://[fe80::aa40:25ff:fe00:7%8]:56797/v2/push: deadline has elapsed
17:17:23.216Z ERRO slog-rs: [tfportrear3_0] withdraw: timeout error: deadline has elapsed
17:17:23.216Z WARN slog-rs: [tfportrear3_0] expiring peer fe80::aa40:25ff:fe00:7 due to failed withdraw
17:17:23.338Z INFO slog-rs: [tfportrear3_0] removing routes 3 from dendrite
17:17:26.144Z ERRO slog-rs: [tfportrear3_0] failed to remove tunnel routes: {} OPTE driver is not attached
17:17:26.144Z DEBG slog-rs: [tfportrear3_0] redistributing expire to 3 peers
17:17:26.144Z WARN slog-rs: [tfportrear3_0] peer event in solicit state: Push(Update { underlay: None, tunnel: None })
17:17:26.144Z DEBG slog-rs: [tfportrear3_0] transition solicit -> exchange
17:17:26.184Z INFO slog-rs: [tfportrear3_0] exchange: listening on [fe80::aa40:25ff:fe08:422c]:56797
17:17:26.185Z INFO slog-rs: [tfportrear3_0] waiting for exchange server to start
17:17:26.471Z INFO slog-rs: [tfportrear3_0] push from fe80::aa40:25ff:fe00:7: Update {
        underlay: Some(
            UnderlayUpdate {
                announce: {},
                withdraw: {
                    PathVector {
                        destination: Ipv6Net {
                            addr: fd00:1122:3344:101::,
                            width: 64,
                        },
                        path: [
                            "g0",
                            "oxz_switch",
                        ],
                    },
                    PathVector {
                        destination: Ipv6Net {
                            addr: fdb0:a840:2500:1::,
                            width: 64,
                        },
                        path: [
                            "g0",
                            "oxz_switch",
                        ],
                    },
                    PathVector {
                        destination: Ipv6Net {
                            addr: fd00:1122:3344:1::,
                            width: 64,
                        },
                        path: [
                            "g0",
                            "oxz_switch",
                        ],
                    },
                },
            },
        ),
        tunnel: None,
    }
17:17:28.571Z WARN slog-rs: [tfportrear3_0] exchange pull: timeout error: deadline has elapsed
17:17:29.615Z ERRO slog-rs: [tfportrear3_0] peer request timeout to http://[fe80::aa40:25ff:fe00:7%8]:56797/v2/push: deadline has elapsed
17:17:29.615Z ERRO slog-rs: [tfportrear3_0] withdraw: timeout error: deadline has elapsed
17:17:29.615Z WARN slog-rs: [tfportrear3_0] expiring peer fe80::aa40:25ff:fe00:7 due to failed withdraw
17:17:29.615Z INFO slog-rs: [tfportrear3_0] removing routes 0 from dendrite
17:17:29.859Z ERRO slog-rs: [tfportrear3_0] failed to remove tunnel routes: {} OPTE driver is not attached
17:17:29.859Z DEBG slog-rs: [tfportrear3_0] redistributing expire to 3 peers
17:17:29.859Z DEBG slog-rs: [tfportrear3_0] transition solicit -> exchange
17:17:29.859Z INFO slog-rs: [tfportrear3_0] exchange: listening on [fe80::aa40:25ff:fe08:422c]:56797
17:17:29.859Z INFO slog-rs: [tfportrear3_0] waiting for exchange server to start
17:17:29.859Z INFO slog-rs: [tfportrear3_0] push from fe80::aa40:25ff:fe00:7: Update {
        underlay: Some(
            UnderlayUpdate {
                announce: {
                    PathVector {
                        destination: Ipv6Net {
                            addr: fd00:1122:3344:101::,
                            width: 64,
                        },
                        path: [
                            "g0",
                            "oxz_switch",
                        ],
                    },
                    PathVector {
                        destination: Ipv6Net {
                            addr: fd00:1122:3344:1::,
                            width: 64,
                        },
                        path: [
                            "g0",
                            "oxz_switch",
                        ],
                    },
                    PathVector {
                        destination: Ipv6Net {
                            addr: fdb0:a840:2500:1::,
                            width: 64,
                        },
                        path: [
                            "g0",
                            "oxz_switch",
                        ],
                    },
                },
                withdraw: {},
            },
        ),
        tunnel: None,
    }
17:17:30.828Z WARN slog-rs: [tfportrear3_0] exchange pull: timeout error: deadline has elapsed
17:17:32.144Z WARN slog-rs: [tfportrear3_0] exchange pull: timeout error: deadline has elapsed
17:17:32.834Z ERRO slog-rs: [tfportrear3_0] peer request timeout to http://[fe80::aa40:25ff:fe00:7%8]:56797/v2/push: deadline has elapsed
17:17:32.834Z ERRO slog-rs: [tfportrear3_0] announce: timeout error: deadline has elapsed
17:17:32.834Z WARN slog-rs: [tfportrear3_0] expiring peer fe80::aa40:25ff:fe00:7 due to failed announce
17:17:32.834Z INFO slog-rs: [tfportrear3_0] removing routes 0 from dendrite
17:17:33.702Z ERRO slog-rs: [tfportrear3_0] failed to remove tunnel routes: {} OPTE driver is not attached
17:17:33.928Z DEBG slog-rs: [tfportrear3_0] redistributing expire to 3 peers
17:17:33.928Z WARN slog-rs: [tfportrear3_0] peer event in solicit state: Push(Update { underlay: Some(UnderlayUpdate { announce: {PathVector { destination: Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }, path: ["g1", "oxz_switch"] }, PathVector { destination: Ipv6Net { addr: fdb0:a840:2500:3::, width: 64 }, path: ["g1", "oxz_switch"] }, PathVector { destination: Ipv6Net { addr: fd00:1122:3344:102::, width: 64 }, path: ["g1", "oxz_switch"] }}, withdraw: {} }), tunnel: None })
17:17:33.928Z DEBG slog-rs: [tfportrear3_0] transition solicit -> exchange
17:17:34.409Z WARN slog-rs: [tfportrear3_0] exchange pull: timeout error: deadline has elapsed
17:17:35.009Z INFO slog-rs: [tfportrear3_0] exchange: listening on [fe80::aa40:25ff:fe08:422c]:56797
17:17:35.010Z INFO slog-rs: [tfportrear3_0] waiting for exchange server to start
17:17:37.486Z WARN slog-rs: [tfportrear3_0] exchange pull: timeout error: deadline has elapsed
17:17:39.899Z WARN slog-rs: [tfportrear3_0] exchange pull: timeout error: deadline has elapsed
17:17:42.162Z WARN slog-rs: [tfportrear3_0] exchange pull: timeout error: deadline has elapsed
17:17:42.333Z INFO slog-rs: [tfportrear3_0] sending 3 routes to dendrite
17:17:42.333Z DEBG slog-rs: [tfportrear3_0] sending to dpd host=localhost port=12224
17:17:42.635Z INFO slog-rs: [tfportrear3_0] adding route fd00:1122:3344:103:: -> fe80::aa40:25ff:fe00:7 on port Rear(RearPort(3))/0
17:17:44.396Z INFO slog-rs: [tfportrear3_0] adding route fdb0:a840:2500:7:: -> fe80::aa40:25ff:fe00:7 on port Rear(RearPort(3))/0
17:17:44.501Z INFO slog-rs: [tfportrear3_0] adding route fd00:1122:3344:3:: -> fe80::aa40:25ff:fe00:7 on port Rear(RearPort(3))/0
17:17:44.738Z INFO slog-rs: [tfportrear3_0] removing routes 0 from dendrite
17:17:44.831Z DEBG slog-rs: [tfportrear3_0] redistributing update to 3 peers
17:17:48.797Z INFO slog-rs: [tfportrear3_0] push from fe80::aa40:25ff:fe00:7: Update {
        underlay: Some(
            UnderlayUpdate {
                announce: {},
                withdraw: {
                    PathVector {
                        destination: Ipv6Net {
                            addr: fd00:1122:3344:2::,
                            width: 64,
                        },
                        path: [
                            "g1",
                            "oxz_switch",
                        ],
                    },
                    PathVector {
                        destination: Ipv6Net {
                            addr: fdb0:a840:2500:3::,
                            width: 64,
                        },
                        path: [
                            "g1",
                            "oxz_switch",
                        ],
                    },
                    PathVector {
                        destination: Ipv6Net {
                            addr: fd00:1122:3344:102::,
                            width: 64,
                        },
                        path: [
                            "g1",
                            "oxz_switch",
                        ],
                    },
                },
            },
        ),
        tunnel: None,
    }
17:18:10.865Z INFO slog-rs: [tfportrear3_0] push from fe80::aa40:25ff:fe00:7: Update {
        underlay: Some(
            UnderlayUpdate {
                announce: {
                    PathVector {
                        destination: Ipv6Net {
                            addr: fd00:1122:3344:102::,
                            width: 64,
                        },
                        path: [
                            "g1",
                            "oxz_switch",
                        ],
                    },
                    PathVector {
                        destination: Ipv6Net {
                            addr: fd00:1122:3344:2::,
                            width: 64,
                        },
                        path: [
                            "g1",
                            "oxz_switch",
                        ],
                    },
                    PathVector {
                        destination: Ipv6Net {
                            addr: fdb0:a840:2500:3::,
                            width: 64,
                        },
                        path: [
                            "g1",
                            "oxz_switch",
                        ],
                    },
                },
                withdraw: {},
            },
        ),
        tunnel: None,
    }
18:24:30.812Z WARN slog-rs: [tfportrear3_0] neighbor g3@fe80::aa40:25ff:fe00:7 missed solicit interval
@rcgoodfellow rcgoodfellow added this to the 11 milestone Aug 27, 2024
@rcgoodfellow rcgoodfellow self-assigned this Aug 27, 2024
@morlandi7 morlandi7 modified the milestones: 11, 12 Oct 11, 2024
@morlandi7 morlandi7 modified the milestones: 12, 13 Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rcgoodfellow rcgoodfellow

Labels
Bug
Projects
None yet
Milestone
13
Development

No branches or pull requests
2 participants
@rcgoodfellow @morlandi7