Docker jail gets assigned wrong ip

[PandiO]

Hello. I am fairly new to Truenas Scale and subsequently Jip Hops Jailmaker. A while back I followed Stux' tutorial to set up jailmaker, docker and dockge to run docker compose containers. This all worked out great, untill I wanted to set up the same system on a different pool.

Again, I followed the tutorial of found on the main github page of Jailmaker.

The problem is that for some reason when I start or restart the docker jail (I suspect Truenas system reboot is the trigger) it gets assigned an ip starting with 172. I am not able to enter the shell of the jail and when it gets this ip range assigned to it it wont go back to the regular ip as I intended it to.

The config file of the docker jail is as follows:

gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1

# Use bridge networking to provide an isolated network namespace,
# so docker can manage firewall rules
# Alternatively use --network-macvlan=eno1 instead of --network-bridge
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
systemd_nspawn_user_args=--network-macvlan=enp6s0
	--resolv-conf=bind-host
	--system-call-filter='add_key keyctl bpf'
#	--bind='/mnt/Main/NVR-recordings:/mnt/NVR-recordings'
	--bind='/mnt/FastStorage/docker/data:/mnt/data'
	--bind='/mnt/FastStorage/docker/stacks:/opt/stacks'

# Script to run on the HOST before starting the jail
# Load kernel module and config kernel settings required for docker
pre_start_hook=#!/usr/bin/bash
	set -euo pipefail
	echo 'PRE_START_HOOK'
	echo 1 > /proc/sys/net/ipv4/ip_forward
	modprobe br_netfilter
	echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
	echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

# Only used while creating the jail
distro=debian
release=bookworm

# Install docker inside the jail:
# https://docs.docker.com/engine/install/debian/#install-using-the-repository
# Will also install the NVIDIA Container Toolkit if gpu_passthrough_nvidia=1 during initial setup
# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
initial_setup=#!/usr/bin/bash
	set -euo pipefail
	
	apt-get update && apt-get -y install ca-certificates curl
	install -m 0755 -d /etc/apt/keyrings
	curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
	chmod a+r /etc/apt/keyrings/docker.asc
	
	echo \
	"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
	$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
	tee /etc/apt/sources.list.d/docker.list > /dev/null
	
	apt-get update
	apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
	
	# The /usr/bin/nvidia-smi will be present when gpu_passthrough_nvidia=1
	if [ -f /usr/bin/nvidia-smi ]; then
	curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey -o /etc/apt/keyrings/nvidia.asc
	chmod a+r /etc/apt/keyrings/nvidia.asc
	curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
	sed 's#deb https://#deb [signed-by=/etc/apt/keyrings/nvidia.asc] https://#g' | \
	tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
	
	apt-get update
	apt-get install -y nvidia-container-toolkit
	
	nvidia-ctk runtime configure --runtime=docker
	systemctl restart docker
	fi
	
	docker info

# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed
	--property=Type=notify
	--property=RestartForceExitStatus=133
	--property=SuccessExitStatus=133
	--property=Delegate=yes
	--property=TasksMax=infinity
	--collect
	--setenv=SYSTEMD_NSPAWN_LOCK=0

systemd_nspawn_default_args=--keep-unit
	--quiet
	--boot
	--bind-ro=/sys/module
	--inaccessible=/sys/module/apparmor

My network interface in Truenas is as follows:

ip a command in the shell returns the following, containing a lot of interfaces I don't recognize and suspect might be part of the problem:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 58:11:22:c5:c8:59 brd ff:ff:ff:ff:ff:ff
    inet 192.168.50.119/24 brd 192.168.50.255 scope global dynamic enp6s0
       valid_lft 74348sec preferred_lft 74348sec
    inet6 fe80::5a11:22ff:fec5:c859/64 scope link 
       valid_lft forever preferred_lft forever
3: kube-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fa:90:8c:e3:87:bc brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.1/16 brd 172.16.255.255 scope global kube-bridge
       valid_lft forever preferred_lft forever
    inet6 fe80::4011:faff:fe78:c4c4/64 scope link 
       valid_lft forever preferred_lft forever
4: kube-dummy-if: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether 9e:9d:35:5d:dd:a4 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/32 scope link kube-dummy-if
       valid_lft forever preferred_lft forever
    inet 172.17.0.10/32 scope link kube-dummy-if
       valid_lft forever preferred_lft forever
    inet6 fe80::e0ea:4cff:fe78:469a/64 scope link 
       valid_lft forever preferred_lft forever
5: veth066cfbff@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master kube-bridge state UP group default 
    link/ether a2:28:2a:de:5c:9a brd ff:ff:ff:ff:ff:ff link-netns cni-ddbeed77-bbb2-5c82-3036-40853cf2ea03
    inet6 fe80::949d:1dff:fe14:4e9a/64 scope link 
       valid_lft forever preferred_lft forever
6: veth43313790@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master kube-bridge state UP group default 
    link/ether ca:7a:41:f5:4a:ec brd ff:ff:ff:ff:ff:ff link-netns cni-1a859141-9f2d-117f-938e-8224c8b2025a
    inet6 fe80::c87a:41ff:fef5:4aec/64 scope link 
       valid_lft forever preferred_lft forever
7: veth8afceccd@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master kube-bridge state UP group default 
    link/ether 5e:a8:b6:b4:02:d1 brd ff:ff:ff:ff:ff:ff link-netns cni-bcd21a0e-7a34-7ac2-b40d-e7ae2cc30c6c
    inet6 fe80::5ca8:b6ff:feb4:2d1/64 scope link 
       valid_lft forever preferred_lft forever
8: vethdbddaea4@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master kube-bridge state UP group default 
    link/ether 76:9e:95:fd:14:96 brd ff:ff:ff:ff:ff:ff link-netns cni-a564d92d-da3c-e676-3a55-8b89e9e2dbc2
    inet6 fe80::749e:95ff:fefd:1496/64 scope link 
       valid_lft forever preferred_lft forever

One other note is that I tried following the static ip tutorial of Stux, but I couldn't get it working, due to some errors popping up when I tried removing the alias from the enp6s0 interface. At this point I thought assigning a static ip through my router would be fine.

I am really out of options and do not know what causes this problem. I hope someone might have some insight or tips to resolve this. Thanks in advance.

[Lockszmith-GH]

I'll try and unpack your description:

The problem is that for some reason when I start or restart the docker jail (I suspect Truenas system reboot is the trigger) it gets assigned an ip starting with 172.

172.x.x.x is usually assigned by the k3s internal dhcp server.
I see you're using macvlan, but did you set static IP on it?

I am not able to enter the shell of the jail

What is the output of jlmker shell <jailname> ?
Networking shouldn't have anything to do with your ability to shell into it.
Also, check the output of journalctl (see #135 for more troubleshooting details)

... and when it gets this ip range assigned to it it wont go back to the regular ip as I intended it to.

What is the intended IP? Where did you set it up?

[PandiO]

Hi. My apologies for my slow response.

I tried setting up de static ip as described by Stux’ tutorial, but this did not work and the jail would not start.
I assumed if I do not set this up it would get an ip assigned by my router.

If I try to enter the shell of the jail, it takes long while for I get a response (around a minute) saying it cannot connect.

I will check the logs.