Transparent management interface

README.md

@@ -5,6 +5,7 @@ This is a fork of the original [plajjan/vrnetlab](https://github.com/plajjan/vrn
 The documentation provided in this fork only explains the parts that have been changed in any way from the upstream project. To get a general overview of the vrnetlab project itself, consider reading the docs of the upstream repo.
 
 ## What is this fork about?
+
 At [containerlab](https://containerlab.srlinux.dev) we needed to have [a way to run virtual routers](https://containerlab.srlinux.dev/manual/vrnetlab/) alongside the containerized Network Operating Systems.
 
 Vrnetlab provides a perfect machinery to package most-common routing VMs in the container packaging. What upstream vrnetlab doesn't do, though, is creating datapath between the VMs in a "container-native" way.  
@@ -15,6 +16,7 @@ This fork adds additional option for `launch.py` script of the supported VMs cal
 By adding a few options a `connection-mode` value can be set to, we made it possible to run vrnetlab containers with the networking that doesn't require a separate container and is native to the tools like docker.
 
 ### Container-native networking?
+
 Yes, the term is bloated, what it actually means is that with the changes we made in this fork it is possible to add interfaces to a container that hosts a qemu VM and vrnetlab will recognize those interfaces and stitch them with the VM interfaces.
 
 With this you can just add, say, veth pairs between the containers as you would do normally, and vrnetlab will make sure that these ports get mapped to your router' ports. In essence, that allows you to work with your vrnetlab containers like with a normal container and get the datapath working in the same "native" way.
@@ -23,6 +25,7 @@ With this you can just add, say, veth pairs between the containers as you would
 > With this being said, we recommend the readers to start their journey from this [documentation entry](https://containerlab.srlinux.dev/manual/vrnetlab/) which will show you how easy it is to run routers in a containerized setting.
 
 ## Connection modes
+
 As mentioned above, the major change this fork brings is the ability to run vrnetlab containers without requiring vr-xcon and by using container-native networking.
 
 The default option that we use in containerlab for this setting is `connection-mode=tc`. With this particular mode we use tc-mirred redirects to stitch container's interfaces `eth1+` with the ports of the qemu VM running inside.
@@ -39,7 +42,26 @@ Other connection mode values are:
 * ovs-bridge - same as a regular bridge, but uses OvS. Can pass LACP traffic.
 * macvtap
 
+## Management interface
+
+There are two types of management connectivity for NOS VMs: _pass-through_ and _host-forwarded_ (legacy) management interfaces.
+
+_Pass-through management_ interfaces allows the use of the assigned management IP within the NOS VM, management traffic is transparently passed through to the VM, and the NOS configuration can accurately reflect the management IP. However, it is no longer possible to send or receive traffic directly in the vrnetlab container (e.g. for installing additional packages within the container), other than to pre-defined exceptions, such as the QEMU serial port on TCP port 5000.
+
+NOSes defaulting to _pass-through_ management interfaces are:
+
+* All vJunos routers
+
+In case of _host-forwarded_ management interfaces, certain ports are forwarded to the NOS VM IP, which is always 10.0.0.15/24. The management gateway in this case is 10.0.0.2/24, and outgoing traffic is NATed to the container management IP. This management interface connection mode does not allow for traffic such as LLDP to pass through the management interface.
+
+NOSes defaulting to _host-forwarded_ management interfaces are:
+
+* None so far, we are gathering feedback on this, and will update this list as feedback is received. Please contact us in [Discord](https://discord.gg/vAyddtaEV9) or open up an issue here if you have found any issues when trying the passthrough mode.
+
+It is possible to change from the default management interface mode by setting the `CLAB_MGMT_PASSTHROUGH` environment variable to 'true' or 'false', however, it is left up to the user to provide a startup configuration compatible with the requested mode.
+
 ## Which vrnetlab routers are supported?
+
 Since the changes we made in this fork are VM specific, we added a few popular routing products:
 
 * Arista vEOS
@@ -59,4 +81,5 @@ Since the changes we made in this fork are VM specific, we added a few popular r
 The rest are left untouched and can be contributed back by the community.
 
 ## Does the build process change?
+
 No. You build the images exactly as before.

common/vrnetlab.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python3
 
 import datetime
+import ipaddress
 import json
 import logging
 import math
@@ -10,7 +11,6 @@
 import subprocess
 import telnetlib
 import time
-import ipaddress
 from pathlib import Path
 
 MAX_RETRIES = 60
@@ -78,6 +78,7 @@ def __init__(
         provision_pci_bus=True,
         cpu="host",
         smp="1",
+        mgmt_passthrough=False,
         min_dp_nics=0,
     ):
         self.logger = logging.getLogger()
@@ -98,7 +99,7 @@ def __init__(
         self._cpu = cpu
         self._smp = smp
 
-        #  various settings
+        # various settings
         self.uuid = None
         self.fake_start_date = None
         self.nic_type = "e1000"
@@ -109,16 +110,36 @@ def __init__(
         # to have them allocated sequential from eth1
         self.highest_provisioned_nic_num = 0
 
+        # Whether the management interface is pass-through or host-forwarded.
+        # Host-forwarded is the original vrnetlab mode where a VM gets a static IP for its management address,
+        # which **does not** match the eth0 interface of a container.
+        # In pass-through mode the VM container uses the same IP as the container's eth0 interface and transparently forwards traffic between the two interfaces.
+        # See https://github.com/hellt/vrnetlab/issues/286
+        self.mgmt_passthrough = mgmt_passthrough
+        mgmt_passthrough_override = os.environ.get("CLAB_MGMT_PASSTHROUGH", "")
+        if mgmt_passthrough_override:
+            self.mgmt_passthrough = mgmt_passthrough_override.lower() == "true"
+
+        # Populate management IP and gateway
+        if self.mgmt_passthrough:
+            self.mgmt_address_ipv4, self.mgmt_address_ipv6 = self.get_mgmt_address()
+            self.mgmt_gw_ipv4, self.mgmt_gw_ipv6 = self.get_mgmt_gw()
+        else:
+            self.mgmt_address_ipv4 = "10.0.0.15/24"
+            self.mgmt_address_ipv6 = "2001:db8::2/64"
+            self.mgmt_gw_ipv4 = "10.0.0.2"
+            self.mgmt_gw_ipv6 = "2001:db8::1"
+
         self.insuffucient_nics = False
         self.min_nics = 0
         # if an image needs minimum amount of dataplane nics to bootup, specify
         if min_dp_nics:
             self.min_nics = min_dp_nics
 
         # management subnet properties, defaults
-        self.mgmt_subnet    = "10.0.0.0/24"
-        self.mgmt_host_ip   = 2
-        self.mgmt_guest_ip  = 15
+        self.mgmt_subnet = "10.0.0.0/24"
+        self.mgmt_host_ip = 2
+        self.mgmt_guest_ip = 15
 
         #  Default TCP ports forwarded (TODO tune per platform):
         #  80    - http
@@ -129,7 +150,7 @@ def __init__(
         #  9339  - iana gnmi/gnoi
         #  32767 - gnmi/gnoi juniper
         #  57400 - nokia gnmi/gnoi
-        self.mgmt_tcp_ports = [80,443,830,6030,8080,9339,32767,57400]
+        self.mgmt_tcp_ports = [80, 443, 830, 6030, 8080, 9339, 32767, 57400]
 
         # we setup pci bus by default
         self.provision_pci_bus = provision_pci_bus
@@ -299,20 +320,51 @@ def create_tc_tap_ifup(self):
         ip link set $TAP_IF mtu 65000
 
         # create tc eth<->tap redirect rules
-        tc qdisc add dev eth$INDEX ingress
-        tc filter add dev eth$INDEX parent ffff: protocol all u32 match u8 0 0 action mirred egress redirect dev tap$INDEX
+        tc qdisc add dev eth$INDEX clsact
+        tc filter add dev eth$INDEX ingress flower action mirred egress redirect dev tap$INDEX
 
-        tc qdisc add dev $TAP_IF ingress
-        tc filter add dev $TAP_IF parent ffff: protocol all u32 match u8 0 0 action mirred egress redirect dev eth$INDEX
+        tc qdisc add dev $TAP_IF clsact
+        tc filter add dev $TAP_IF ingress flower action mirred egress redirect dev eth$INDEX
         """
 
         with open("/etc/tc-tap-ifup", "w") as f:
             f.write(ifup_script)
         os.chmod("/etc/tc-tap-ifup", 0o777)
 
+    def create_tc_tap_mgmt_ifup(self):
+        """Create tap ifup script that is used in tc datapath mode, specifically for the management interface"""
+        ifup_script = """#!/bin/bash
+
+        ip link set tap0 up
+        ip link set tap0 mtu 65000
+
+        # create tc eth<->tap redirect rules
+
+        tc qdisc add dev eth0 clsact
+        # exception for TCP ports 5000-5007
+        tc filter add dev eth0 ingress prio 1 protocol ip flower ip_proto tcp dst_port 5000-5007 action pass
+        # mirror ARP traffic to container
+        tc filter add dev eth0 ingress prio 2 protocol arp flower action mirred egress mirror dev tap0
+        # redirect rest of ingress traffic of eth0 to egress of tap0
+        tc filter add dev eth0 ingress prio 3 flower action mirred egress redirect dev tap0
+
+        tc qdisc add dev tap0 clsact
+        # redirect all ingress traffic of tap0 to egress of eth0
+        tc filter add dev tap0 ingress flower action mirred egress redirect dev eth0
+
+        # clone management MAC of the VM
+        ip link set dev eth0 address {MGMT_MAC}
+        """
+
+        ifup_script = ifup_script.replace("{MGMT_MAC}", self.mgmt_mac)
+
+        with open("/etc/tc-tap-mgmt-ifup", "w") as f:
+            f.write(ifup_script)
+        os.chmod("/etc/tc-tap-mgmt-ifup", 0o777)
+
     def gen_mgmt(self):
         """Generate qemu args for the mgmt interface(s)
-        
+
         Default TCP ports forwarded:
           80    - http
           443   - https
@@ -323,34 +375,97 @@ def gen_mgmt(self):
           32767 - gnmi/gnoi juniper
           57400 - nokia gnmi/gnoi
         """
-        if self.mgmt_host_ip+1>=self.mgmt_guest_ip:
-            self.logger.error("Guest IP (%s) must be at least 2 higher than host IP(%s)", 
-                              self.mgmt_guest_ip, self.mgmt_host_ip)
+        if self.mgmt_host_ip + 1 >= self.mgmt_guest_ip:
+            self.logger.error(
+                "Guest IP (%s) must be at least 2 higher than host IP(%s)",
+                self.mgmt_guest_ip,
+                self.mgmt_host_ip,
+            )
 
         network = ipaddress.ip_network(self.mgmt_subnet)
         host = str(network[self.mgmt_host_ip])
-        dns = str(network[self.mgmt_host_ip+1])
+        dns = str(network[self.mgmt_host_ip + 1])
         guest = str(network[self.mgmt_guest_ip])
 
         res = []
-        # mgmt interface is special - we use qemu user mode network with DHCP
         res.append("-device")
-        mac = (
+        self.mgmt_mac = (
             "c0:00:01:00:ca:fe"
             if getattr(self, "_static_mgmt_mac", False)
             else gen_mac(0)
         )
-        res.append(self.nic_type + f",netdev=p00,mac={mac}")
+
+        res.append(self.nic_type + f",netdev=p00,mac={self.mgmt_mac}")
         res.append("-netdev")
-        res.append(
-            f"user,id=p00,net={self.mgmt_subnet},host={host},dns={dns},dhcpstart={guest}," +
-            f"hostfwd=tcp:0.0.0.0:22-{guest}:22," +   # ssh
-            f"hostfwd=udp:0.0.0.0:161-{guest}:161," +  # snmp
-            (",".join([ f"hostfwd=tcp:0.0.0.0:{p}-{guest}:{p}" for p in self.mgmt_tcp_ports ])) +
-            ",tftp=/tftpboot"
-        )
+
+        if self.mgmt_passthrough:
+            # mgmt interface is passthrough - we just create a normal mirred tap interface
+            res.append(
+                "tap,id=p00,ifname=tap0,script=/etc/tc-tap-mgmt-ifup,downscript=no"
+            )
+            self.create_tc_tap_mgmt_ifup()
+        else:
+            # mgmt interface is host-forwarded - we use qemu user mode network
+            # with hostfwd rules to forward ports from the host to the guest
+            res.append(
+                f"user,id=p00,net={self.mgmt_subnet},host={host},dns={dns},dhcpstart={guest},"
+                + f"hostfwd=tcp:0.0.0.0:22-{guest}:22,"  # ssh
+                + f"hostfwd=udp:0.0.0.0:161-{guest}:161,"  # snmp
+                + (
+                    ",".join(
+                        [
+                            f"hostfwd=tcp:0.0.0.0:{p}-{guest}:{p}"
+                            for p in self.mgmt_tcp_ports
+                        ]
+                    )
+                )
+                + ",tftp=/tftpboot"
+            )
+
         return res
 
+    def get_mgmt_address(self):
+        """Returns the IPv4 and IPv6 address of the eth0 interface of the container"""
+        stdout, _ = run_command(["ip", "--json", "address", "show", "dev", "eth0"])
+        command_json = json.loads(stdout.decode("utf-8"))
+        intf_addrinfos = command_json[0]["addr_info"]
+        mgmt_cidr_v4 = None
+        mgmt_cidr_v6 = None
+        for addrinfo in intf_addrinfos:
+            if addrinfo["family"] == "inet" and addrinfo["scope"] == "global":
+                mgmt_address_v4 = addrinfo["local"]
+                mgmt_prefixlen_v4 = addrinfo["prefixlen"]
+                mgmt_cidr_v4 = mgmt_address_v4 + "/" + str(mgmt_prefixlen_v4)
+            if addrinfo["family"] == "inet6" and addrinfo["scope"] == "global":
+                mgmt_address_v6 = addrinfo["local"]
+                mgmt_prefixlen_v6 = addrinfo["prefixlen"]
+                mgmt_cidr_v6 = mgmt_address_v6 + "/" + str(mgmt_prefixlen_v6)
+
+        if not mgmt_cidr_v4:
+            raise ValueError("No IPv4 address set on management interface eth0!")
+
+        return mgmt_cidr_v4, mgmt_cidr_v6
+
+    def get_mgmt_gw(self):
+        """Returns the IPv4 and IPv6 default gateways of the container, used for generating the management default route"""
+        stdout_v4, _ = run_command(["ip", "--json", "-4", "route", "show", "default"])
+        command_json_v4 = json.loads(stdout_v4.decode("utf-8"))
+        try:
+            mgmt_gw_v4 = command_json_v4[0]["gateway"]
+        except IndexError as e:
+            raise IndexError(
+                "No default gateway route on management interface eth0!"
+            ) from e
+
+        stdout_v6, _ = run_command(["ip", "--json", "-6", "route", "show", "default"])
+        command_json_v6 = json.loads(stdout_v6.decode("utf-8"))
+        try:
+            mgmt_gw_v6 = command_json_v6[0]["gateway"]
+        except IndexError:
+            mgmt_gw_v6 = None
+
+        return mgmt_gw_v4, mgmt_gw_v6
+
     def nic_provision_delay(self) -> None:
         self.logger.debug(
             f"number of provisioned data plane interfaces is {self.num_provisioned_nics}"
@@ -526,7 +641,9 @@ def restart(self):
         self.stop()
         self.start()
 
-    def wait_write(self, cmd, wait="__defaultpattern__", con=None, clean_buffer=False, hold=""):
+    def wait_write(
+        self, cmd, wait="__defaultpattern__", con=None, clean_buffer=False, hold=""
+    ):
         """Wait for something on the serial port and then send command
 
         Defaults to using self.tn as connection but this can be overridden
@@ -548,8 +665,10 @@ def wait_write(self, cmd, wait="__defaultpattern__", con=None, clean_buffer=Fals
             self.logger.trace(f"waiting for '{wait}' on {con_name}")
             res = con.read_until(wait.encode())
 
-            while (hold and (hold in res.decode())):
-                self.logger.trace(f"Holding pattern '{hold}' detected: {res.decode()}, retrying in 10s...")
+            while hold and (hold in res.decode()):
+                self.logger.trace(
+                    f"Holding pattern '{hold}' detected: {res.decode()}, retrying in 10s..."
+                )
                 con.write("\r".encode())
                 time.sleep(10)
                 res = con.read_until(wait.encode())
@@ -661,9 +780,19 @@ def qemu_additional_args(self):
 
 
 class VR:
-    def __init__(self, username, password):
+    def __init__(self, username, password, mgmt_passthrough: bool = False):
         self.logger = logging.getLogger()
 
+        # Whether the management interface is pass-through or host-forwarded.
+        # Host-forwarded is the original vrnetlab mode where a VM gets a static IP for its management address,
+        # which **does not** match the eth0 interface of a container.
+        # In pass-through mode the VM container uses the same IP as the container's eth0 interface and transparently forwards traffic between the two interfaces.
+        # See https://github.com/hellt/vrnetlab/issues/286
+        self.mgmt_passthrough = mgmt_passthrough
+        mgmt_passthrough_override = os.environ.get("CLAB_MGMT_PASSTHROUGH", "")
+        if mgmt_passthrough_override:
+            self.mgmt_passthrough = mgmt_passthrough_override.lower() == "true"
+
         try:
             os.mkdir("/tftpboot")
         except: