Docker container using bridge networking hosted in jail using macvlan networking

[RyanMelena]

I've run into a networking issue with my jail (based on the jailmaker Docker template) and am wondering if it is expected behavior.

Setup:

Jailmaker jail created from jailmaker Docker template, containing the following network binds:

--network-macvlan=br30
--network-macvlan=br40
--network-macvlan=br70

The idea is to allow my Docker containers to connect to the three vlans represented by those bridges without having access to my management vlan (br10).

Inside the docker jail I have the following interfaces:

root@docker:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: mv-br30@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ce:11:e3:f1:8a:73 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.80.30.205/24 metric 1024 brd 10.80.30.255 scope global dynamic mv-br30
       valid_lft 4294sec preferred_lft 4294sec
    inet6 fe80::cc11:e3ff:fef1:8a73/64 scope link
       valid_lft forever preferred_lft forever
3: mv-br40@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 2e:34:8a:a9:16:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.80.40.100/24 metric 1024 brd 10.80.40.255 scope global dynamic mv-br40
       valid_lft 4293sec preferred_lft 4293sec
    inet6 fe80::2c34:8aff:fea9:16e1/64 scope link
       valid_lft forever preferred_lft forever
4: mv-br70@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 72:66:e6:23:38:00 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.80.70.147/24 metric 1024 brd 10.80.70.255 scope global dynamic mv-br70
       valid_lft 4342sec preferred_lft 4342sec
    inet6 fe80::7066:e6ff:fe23:3800/64 scope link
       valid_lft forever preferred_lft forever
5: br-5aa4412a8ee3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:8c:bf:fb:29 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-5aa4412a8ee3
       valid_lft forever preferred_lft forever
    inet6 fe80::42:8cff:febf:fb29/64 scope link
       valid_lft forever preferred_lft forever
6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:39:7c:44:66 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:39ff:fe7c:4466/64 scope link
       valid_lft forever preferred_lft forever
43: veth4581b65@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-5aa4412a8ee3 state UP group default
    link/ether 06:cf:95:d2:99:45 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::4cf:95ff:fed2:9945/64 scope link
       valid_lft forever preferred_lft forever

With this configuration the jail gets one IP on each vlan (each on its respective interface) as shown above and network connectivity seems perfect (DNS working, can access each VLAN, internet access works).

When I create docker containers in this jail and specify macvlan networking (with the parent interface set to one of the mv-br* interfaces listed above) everything also works perfectly.

However, when I create a docker container in this jail and specify bridge networking, the resultant container has no ability to access the outside world.

If anyone has any tips or suggestions for troubleshooting they would be very much appreciated!

[mrstux]

have you tried "network-bridge" instead of "network-macvlan"

[RyanMelena]

I have, and in that case both bridge mode and macvlan mode docker containers seemed to work as expected. However, nspawn only appears to allow a single network-bridge binding which precludes my desired config with the three different vlans. I also prefer the segregation from the host system provided by macvlan on the jail although I could live without that.

[Jip-Hop]

I don't think the macvlan flag in nspawn is supposed to be used like that. For your usecase you're supposed to use bridge networking for the first bridge interface and then use --network-veth-extra= for each additional interface. https://manpages.debian.org/bookworm/systemd-container/systemd-nspawn.1.en.html

I wrote about it on the old forum: https://www.truenas.com/community/threads/linux-jails-experimental-script.106926/page-6#post-805938

As far as I know none of the jailmaker users went through this trouble. So if you do please post back and consider updating the documentation.

[RyanMelena]

Thank you for the additional suggestion. I read through the linked discussions and unless I'm misunderstanding the nspawn docs, it seems like --network-veth-extra will create an additional link but won't add the host side to the existing bridge (which I believe is what --network-bridge accomplishes). It seems like the suggested solution is to use nspawn's ExecStartPost as you mentioned in your forum post but from what I can tell JailMaker doesn't currently offer a post_start_hook equivalent to the pre_start_hook that would allow assigning the veth-extra interfaces to their respective host bridges after startup. Am I missing something perhaps?

[Jip-Hop]

You're right. It currently doesn't have the post_start_hook. I'm working on adding that as we speak.

[Jip-Hop]

Please try jailmaker from the veth-extra branch. This config template should get you started.

startup=0
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1

systemd_nspawn_user_args=--network-bridge=br30
    --network-veth-extra=ve-docker-1:vee-1
    --network-veth-extra=ve-docker-2:vee-2
    --resolv-conf=bind-host
    --system-call-filter='add_key keyctl bpf'

# Script to run on the HOST before starting the jail
# Load kernel module and config kernel settings required for docker
pre_start_hook=#!/usr/bin/bash
    set -euo pipefail
    echo 'PRE_START_HOOK'
    echo 1 > /proc/sys/net/ipv4/ip_forward
    modprobe br_netfilter
    echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
    echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

# Specify command/script to run on the HOST after starting the jail
# For example to attach to multiple bridge interfaces
post_start_hook=#!/usr/bin/bash
    set -euo pipefail
    echo 'POST_START_HOOK'
    ip link set dev ve-docker-1 master br40
    ip link set dev ve-docker-1 up
    ip link set dev ve-docker-2 master br70
    ip link set dev ve-docker-2 up

# Only used while creating the jail
distro=debian
release=bookworm

# Install docker inside the jail:
# https://docs.docker.com/engine/install/debian/#install-using-the-repository
# Will also install the NVIDIA Container Toolkit if gpu_passthrough_nvidia=1 during initial setup
# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
initial_setup=#!/usr/bin/bash
    set -euo pipefail

    apt-get update && apt-get -y install ca-certificates curl
    install -m 0755 -d /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
    chmod a+r /etc/apt/keyrings/docker.asc

    echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
    $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
    tee /etc/apt/sources.list.d/docker.list > /dev/null
    
    apt-get update
    apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    
    # The /usr/bin/nvidia-smi will be present when gpu_passthrough_nvidia=1
    if [ -f /usr/bin/nvidia-smi ]; then
        curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey -o /etc/apt/keyrings/nvidia.asc
        chmod a+r /etc/apt/keyrings/nvidia.asc
        curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
        sed 's#deb https://#deb [signed-by=/etc/apt/keyrings/nvidia.asc] https://#g' | \
        tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

        apt-get update
        apt-get install -y nvidia-container-toolkit

        nvidia-ctk runtime configure --runtime=docker
        systemctl restart docker
    fi

    docker info

# Usually no need to change systemd_run_default_args
systemd_run_default_args=--collect
    --property=Delegate=yes
    --property=RestartForceExitStatus=133
    --property=SuccessExitStatus=133
    --property=TasksMax=infinity
    --property=Type=notify
    --setenv=SYSTEMD_NSPAWN_LOCK=0
    --property=KillMode=mixed

# Usually no need to change systemd_nspawn_default_args
systemd_nspawn_default_args=--bind-ro=/sys/module
    --boot
    --inaccessible=/sys/module/apparmor
    --quiet
    --keep-unit

Let me know if it works. I'd appreciate it if you could update the networking docs if this works for you.

[RyanMelena]

@Jip-Hop Thank you so much for adding this functionality. I was experimenting with --property and adding my own /etc/systemd/network/*.network file after reading your forum post again but this branch makes everything much simpler! I tried out the new branch with a new jail and it worked perfectly right off the bat.

I've now switched my existing docker jail to the ----network-veth-extra binds, added the post_start_hook, created the /etc/systemd/vee-dhcp.network file and updated my docker networks to reflect the new vee-* interface naming scheme. After doing so everything is back up and running and I'm very happy to report that both Docker macvlan and bridge networks inside the jail are performing as expected.

I truly appreciate all the effort that must have gone into the creation of jailmaker and your exceptional support has been above and beyond. Jailmaker has already greatly enhanced my TrueNAS Scale experience and will allow me to disable TrueNAS Scale apps altogether which (given general k3s and especially TrueCharts trouble) has been my biggest pain point.

[Jip-Hop]

If you'd be able to contribute some documentation then I can create a new release. 🙂

[RyanMelena]

Happy to do it. Are you just thinking some additional details around implementing multiple bridge networks under the Bridge Networking section of the networking doc?

[Jip-Hop]

Yes that would be fine. Perhaps add a link to my post above which contains a complete example config template and include the relevant code snippet (post start hook and the relevant systemd_nspawn_user_args) in the networking doc itself. And some explanation about the naming scheme and the .network file would be great! Ah and let's not forget a couple of links to additional resources such as https://wiki.csclub.uwaterloo.ca/Systemd-nspawn#Multiple_network_interfaces ^^ Thanks!