Macvlan interface can't be created when jailmaker dataset type is "apps"

[rowanmoul]

I've been trying to setup a macvlan network for my jail, but it just doesn't want to work and I can't figure out why.
Lets start with the basics:
Am I correct in understanding that passing the --network-macvlan=eno1 argument will cause systemd-nspawn to automatically setup a macvlan interface for the container without any other arguments?
Do I need to do some manual work too (eg, setting eno1 to promiscuous mode)?
Given that TrueNAS SCALE isn't running systemd-networkd itself, perhaps there is some missing auto-configuration that can't be done without that service?

Should I see said interface listed when running ip link on the host (TrueNAS shell)?

I DO see the mv-eno1 interface inside the container, but it doesn't seem to matter what I try. Even if I set the interface to "up" systemd-networkd (inside the container) won't configure it, and dhcpcd (not needed if using networkd, I know, but trying different things) claims there aren't any valid interfaces at all.
I have also tried using eno2 just in case TrueNAS is somehow locking down eno1 for it's own use (that's where the web interface is listening).

I previously tried to use macvlan in an ubuntu vm (on the same TrueNAS SCALE server) with docker and also struggled to get it to work. I ended up giving up and using ipvlan on dedicated physical NICs (so only one IP per MAC) but I'm fairly certain my Unifi switch can handle multiple MACs per physical NIC, and it would allow me to support a lot more containers without resorting to full ipvlan (multiple IPs per MAC) and basically turning my NAS into a router, which is NOT it's job.

[Jip-Hop]

I don't think you need to do anything special other than creating the jail with the --network-macvlan=eno1 --resolv-conf=bind-host. Perhaps you have to wait a few minutes for DHCP to assign the jail an IP address. Also I recommend looking in the jlmkr.py code and search for macvlan, to understand how jlmkr prepares the jail with a networking config file file macvlan networking. And perhaps have a look at bridge networking in the advanced networking docs. Good luck!

[rowanmoul]

I did a fresh install of TrueNAS Scale on my server (Dell R630), and after manually reconfiguring it (I did not restore my saved config) to be the same as before, it is still failing to work. In fact, systemd-nspawn won't even work with a straight NIC pass through (--network-interface=eno2), which I might not have tried before.
I also booted the same server with a fresh install of Arch Linux, and that worked perfectly (still using the jlmkr script). Macvlan "just worked".
I also installed both Arch Linux and TrueNAS Scale on a different machine that I have kicking around, and in BOTH cases it also worked perfectly - even after I configured TrueNAS to be as close as I could make it to the server, macvlan still worked.
I am down to two theories:

1. It could be NIC driver/hardware issue. The Dell server has Broadcom BMC5720 NICs, which use the tg3 linux driver. I was able to find a number of references to various issues with this NIC and/or this driver. My other machine has a venerable Intel Pro/1000 NIC, using the e1000e linux driver. I've ordered an Intel I350 based network daughter card for the R630. It was only $23 USD including shipping, so not a big loss if that's not the issue, but Intel's drivers for Linux have always been regarded as the best. I may also try an ASIX USB network adapter that I have in the meantime.
   I've looked at the changes to the tg3 driver between Linux 6.1 and Linux 6.6, and there's nothing obvious that might have been fixed related to this, but Arch Linux definitely worked with the BMC5720 NICs.
2. It could be one of the few differences in configuration between the test machine and the server. I'm going to try another fresh install on the server hardware, and not touch anything before I test jailmaker with macvlan, just to see if a stock install works, but I don't expect it to, as I think the first theory is much more likely.

[Jip-Hop]

You could also try to setup a bridge network instead of macvlan. Could perhaps serve a similar use case. Maybe that works with your current hardware.

[Jip-Hop]

Latest version of SCALE has some issues editing bridge interfaces though: https://www.truenas.com/docs/scale/23.10/gettingstarted/scalereleasenotes/#23101-ongoing-issues

[rowanmoul]

I bridge would work fine in principle, but at least for the unifi controller, it needs to be directly L2 discoverable, and for other things, I don't intend to run all of them in the same VLAN as TrueNAS, so I'd prefer stronger network segregation.

As for my theories, I just tried a completely stock install this morning (2), and once again, macvlan "just works" so it's looking like it's a weird configuration issue rather than a hardware/driver issue... but I'd still rather have an intel NIC so I'll probably keep the order. Now time to slowly configure this fresh install to look like the previous install until something stops working...

[rowanmoul]

no matter what I did, I could not find a configuration change that would trigger the issue on the stock install, even after configuring everything short of a few replication tasks to be the same, macvlan continued to work, but my original install still won't work. I even tried resetting the network settings to defaults on my original install, and I also exported the config db for both installs and cannot find any relevant differences in them, forcing me to assume it's something that isn't stored in the config db. I'm just going to wait for my intel network card to arrive and hope for the best...

[rowanmoul]

Solved!
It wasn't the NIC, or the drivers, or any of the configuration settings. It turns out that when you create a dataset and choose "apps" as the type, this somehow prevents systemd-nspawn from properly creating/enabling a macvlan interface. Without knowing what exactly is different about "apps" vs "generic", I couldn't possibly speculate further about what is actually causing the issue, but using "generic" seems to allow it to work as expected.

[mcmehrtens]

Hi, I'm having a very similar problem as you described in this discussion, but I haven't manually created any datasets with the Share Type "Apps." Do you have any other thoughts as to what it may be? I'll post my system specifications in a thread below.

[mcmehrtens]

I'm having a very similar problem, but I haven't created any datasets with the "Apps" Share Type. I've defined my jail to use macvlan networking, but it will not configure the interface properly. While creating the jail, it hangs for a while waiting for the network to come up, and then continues, but all the apt calls fail since the interface is down.

NIC: QNAP QXG-2G2T-I225

The interface I'm trying to use macvlan with has its IP address set via DHCP and I'd like my jail's IP address to be set via DHCP as well. I'm running a DHCP server on my OPNsense router.

The only way I'm able to get internet connection within the jail is by running the following three commands inside the jail:

ip a add 10.0.0.22/24
ifconfig mv-enp3s0 up
ip r add default via 10.0.0.1 dev mv-enp3s0

But this is obviously not being configured via DHCP like it should.

I've attached a number of outputs and configs to this post. @Jip-Hop does anything obvious jump out at you in my logs? I'm happy to dig up anything else that might be useful.

TrueNAS Scale Network Settings:

docker_jail_config.txt

startup=0
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1

# Use macvlan networking to provide an isolated network namespace,
# so docker can manage firewall rules
# Alternatively use --network-macvlan=eno1 instead of --network-bridge
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
systemd_nspawn_user_args=--network-macvlan=enp3s0
    --resolv-conf=bind-host
    --system-call-filter='add_key keyctl bpf'

# Script to run on the HOST before starting the jail
# Load kernel module and config kernel settings required for docker
pre_start_hook=#!/usr/bin/bash
    set -euo pipefail
    echo 'PRE_START_HOOK'
    echo 1 > /proc/sys/net/ipv4/ip_forward
    modprobe br_netfilter
    echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
    echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

# Only used while creating the jail
distro=debian
release=bookworm

# Install docker inside the jail:
# https://docs.docker.com/engine/install/debian/#install-using-the-repository
# Will also install the NVIDIA Container Toolkit if gpu_passthrough_nvidia=1 during initial setup
# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
initial_setup=#!/usr/bin/bash
    set -euo pipefail

    apt-get update && apt-get -y install ca-certificates curl
    install -m 0755 -d /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
    chmod a+r /etc/apt/keyrings/docker.asc

    echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
    $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
    tee /etc/apt/sources.list.d/docker.list > /dev/null

    apt-get update
    apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

    # The /usr/bin/nvidia-smi will be present when gpu_passthrough_nvidia=1
    if [ -f /usr/bin/nvidia-smi ]; then
        curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey -o /etc/apt/keyrings/nvidia.asc
        chmod a+r /etc/apt/keyrings/nvidia.asc
        curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
        sed 's#deb https://#deb [signed-by=/etc/apt/keyrings/nvidia.asc] https://#g' | \
        tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

        apt-get update
        apt-get install -y nvidia-container-toolkit

        nvidia-ctk runtime configure --runtime=docker
        systemctl restart docker
    fi

    docker info

# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed
    --property=Type=notify
    --property=RestartForceExitStatus=133
    --property=SuccessExitStatus=133
    --property=Delegate=yes
    --property=TasksMax=infinity
    --collect
    --setenv=SYSTEMD_NSPAWN_LOCK=0

systemd_nspawn_default_args=--keep-unit
    --quiet
    --boot
    --bind-ro=/sys/module
    --inaccessible=/sys/module/apparmor

host_ip_a_output.txt

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 3c:ec:ef:d7:ba:3a brd ff:ff:ff:ff:ff:ff
    altname enp11s0
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 24:5e:be:83:e8:76 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.21/24 brd 10.0.0.255 scope global dynamic enp3s0
       valid_lft 3215sec preferred_lft 3215sec
    inet6 fdce:8e53:629a:a34e:265e:beff:fe83:e876/64 scope global dynamic mngtmpaddr
       valid_lft 1767sec preferred_lft 1767sec
    inet6 fe80::265e:beff:fe83:e876/64 scope link
       valid_lft forever preferred_lft forever
4: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 3c:ec:ef:d7:ba:3b brd ff:ff:ff:ff:ff:ff
    altname enp12s0
5: enp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 24:5e:be:83:e8:75 brd ff:ff:ff:ff:ff:ff
6: enxbe3af2b6059f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether be:3a:f2:b6:05:9f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::bc3a:f2ff:feb6:59f/64 scope link
       valid_lft forever preferred_lft forever

jail_ip_a_output.txt

root@docker:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: mv-enp3s0@if3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 4a:1d:2a:dd:80:b2 brd ff:ff:ff:ff:ff:ff link-netnsid 0

jlmkr_create_output.txt

root@nas:~# jlmkr create --config /mnt/tank/zroot0/jailmaker/jail_configs/docker docker
USE THIS SCRIPT AT YOUR OWN RISK!
IT COMES WITHOUT WARRANTY AND IS NOT SUPPORTED BY IXSYSTEMS.
Creating jail docker from config template /mnt/tank/zroot0/jailmaker/jail_configs/docker.

TIP: Run `jlmkr create` without any arguments for interactive config.
Or use CLI args to override the default options.
For more info, run: `jlmkr create --help`

Using image from local cache
Unpacking the rootfs
tar: setxattrat: Cannot set 'system.posix_acl_access' extended attribute for file './var/log/journal': Operation not supported
tar: setxattrat: Cannot set 'system.posix_acl_default' extended attribute for file './var/log/journal': Operation not supported

---
You just created a Debian bookworm amd64 (20240527_05:24) container.

To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.

jlmkr_list_output.txt

root@nas:~# jlmkr list
NAME   RUNNING STARTUP GPU_INTEL GPU_NVIDIA OS     VERSION ADDRESSES
docker True    False   False     False      debian 12      -

jlmkr_start_output.txt

root@nas:~# jlmkr start docker

Starting jail docker with the following command:

systemd-run --property=KillMode=mixed --property=Type=notify --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=Delegate=yes --property=TasksMax=infinity --collect --setenv=SYSTEMD_NSPAWN_LOCK=0 --unit=jlmkr-docker --working-directory=./jails/docker '--description=My nspawn jail docker [created with jailmaker]' --property=ExecStartPre=/mnt/tank/zroot0/jailmaker/jailmaker/jails/docker/.ExecStartPre -- systemd-nspawn --keep-unit --quiet --boot --bind-ro=/sys/module --inaccessible=/sys/module/apparmor --machine=docker --directory=rootfs --notify-ready=yes --network-macvlan=enp3s0 --resolv-conf=bind-host '--system-call-filter=add_key keyctl bpf'

Running as unit: jlmkr-docker.service
About to run the initial setup.
Waiting for networking in the jail to be ready.
Please wait (this may take 90s in case of bridge networking)...
Ign:1 http://deb.debian.org/debian bookworm InRelease
Ign:2 http://deb.debian.org/debian bookworm-updates InRelease
Ign:3 http://deb.debian.org/debian-security bookworm-security InRelease
Ign:1 http://deb.debian.org/debian bookworm InRelease
Ign:2 http://deb.debian.org/debian bookworm-updates InRelease
Ign:3 http://deb.debian.org/debian-security bookworm-security InRelease
Ign:1 http://deb.debian.org/debian bookworm InRelease
Ign:2 http://deb.debian.org/debian bookworm-updates InRelease
Ign:3 http://deb.debian.org/debian-security bookworm-security InRelease
Err:1 http://deb.debian.org/debian bookworm InRelease
  Temporary failure resolving 'deb.debian.org'
Err:2 http://deb.debian.org/debian bookworm-updates InRelease
  Temporary failure resolving 'deb.debian.org'
Err:3 http://deb.debian.org/debian-security bookworm-security InRelease
  Temporary failure resolving 'deb.debian.org'
Reading package lists... Done
W: Failed to fetch http://deb.debian.org/debian/dists/bookworm/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://deb.debian.org/debian/dists/bookworm-updates/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://deb.debian.org/debian-security/dists/bookworm-security/InRelease  Temporary failure resolving 'deb.debian.org'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libbrotli1 libcurl4 libldap-2.5-0 libldap-common libnghttp2-14 libpsl5 librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libssh2-1 openssl publicsuffix
Suggested packages:
  libsasl2-modules-gssapi-mit | libsasl2-modules-gssapi-heimdal libsasl2-modules-ldap libsasl2-modules-otp libsasl2-modules-sql
The following NEW packages will be installed:
  ca-certificates curl libbrotli1 libcurl4 libldap-2.5-0 libldap-common libnghttp2-14 libpsl5 librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libssh2-1 openssl
  publicsuffix
0 upgraded, 15 newly installed, 0 to remove and 0 not upgraded.
Need to get 3,409 kB of archives.
After this operation, 7,291 kB of additional disk space will be used.
Ign:1 http://deb.debian.org/debian bookworm/main amd64 openssl amd64 3.0.11-1~deb12u2
Ign:2 http://deb.debian.org/debian bookworm/main amd64 ca-certificates all 20230311
Ign:3 http://deb.debian.org/debian bookworm/main amd64 libbrotli1 amd64 1.0.9-2+b6
Ign:4 http://deb.debian.org/debian bookworm/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg-10
Ign:5 http://deb.debian.org/debian bookworm/main amd64 libsasl2-2 amd64 2.1.28+dfsg-10
Ign:6 http://deb.debian.org/debian bookworm/main amd64 libldap-2.5-0 amd64 2.5.13+dfsg-5
Ign:7 http://deb.debian.org/debian bookworm/main amd64 libnghttp2-14 amd64 1.52.0-1+deb12u1
Ign:8 http://deb.debian.org/debian bookworm/main amd64 libpsl5 amd64 0.21.2-1
Ign:9 http://deb.debian.org/debian bookworm/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2+b2
Ign:10 http://deb.debian.org/debian bookworm/main amd64 libssh2-1 amd64 1.10.0-3+b1
Ign:11 http://deb.debian.org/debian bookworm/main amd64 libcurl4 amd64 7.88.1-10+deb12u5
Ign:12 http://deb.debian.org/debian bookworm/main amd64 curl amd64 7.88.1-10+deb12u5
Ign:13 http://deb.debian.org/debian bookworm/main amd64 libldap-common all 2.5.13+dfsg-5
Ign:14 http://deb.debian.org/debian bookworm/main amd64 libsasl2-modules amd64 2.1.28+dfsg-10
Ign:15 http://deb.debian.org/debian bookworm/main amd64 publicsuffix all 20230209.2326-1
Ign:1 http://deb.debian.org/debian bookworm/main amd64 openssl amd64 3.0.11-1~deb12u2
Ign:2 http://deb.debian.org/debian bookworm/main amd64 ca-certificates all 20230311
Ign:3 http://deb.debian.org/debian bookworm/main amd64 libbrotli1 amd64 1.0.9-2+b6
Ign:4 http://deb.debian.org/debian bookworm/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg-10
Ign:5 http://deb.debian.org/debian bookworm/main amd64 libsasl2-2 amd64 2.1.28+dfsg-10
Ign:6 http://deb.debian.org/debian bookworm/main amd64 libldap-2.5-0 amd64 2.5.13+dfsg-5
Ign:7 http://deb.debian.org/debian bookworm/main amd64 libnghttp2-14 amd64 1.52.0-1+deb12u1
Ign:8 http://deb.debian.org/debian bookworm/main amd64 libpsl5 amd64 0.21.2-1
Ign:9 http://deb.debian.org/debian bookworm/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2+b2
Ign:10 http://deb.debian.org/debian bookworm/main amd64 libssh2-1 amd64 1.10.0-3+b1
Ign:11 http://deb.debian.org/debian bookworm/main amd64 libcurl4 amd64 7.88.1-10+deb12u5
Ign:12 http://deb.debian.org/debian bookworm/main amd64 curl amd64 7.88.1-10+deb12u5
Ign:13 http://deb.debian.org/debian bookworm/main amd64 libldap-common all 2.5.13+dfsg-5
Ign:14 http://deb.debian.org/debian bookworm/main amd64 libsasl2-modules amd64 2.1.28+dfsg-10
Ign:15 http://deb.debian.org/debian bookworm/main amd64 publicsuffix all 20230209.2326-1
Ign:1 http://deb.debian.org/debian bookworm/main amd64 openssl amd64 3.0.11-1~deb12u2
Ign:2 http://deb.debian.org/debian bookworm/main amd64 ca-certificates all 20230311
Ign:3 http://deb.debian.org/debian bookworm/main amd64 libbrotli1 amd64 1.0.9-2+b6
Ign:4 http://deb.debian.org/debian bookworm/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg-10
Ign:5 http://deb.debian.org/debian bookworm/main amd64 libsasl2-2 amd64 2.1.28+dfsg-10
Ign:6 http://deb.debian.org/debian bookworm/main amd64 libldap-2.5-0 amd64 2.5.13+dfsg-5
Ign:7 http://deb.debian.org/debian bookworm/main amd64 libnghttp2-14 amd64 1.52.0-1+deb12u1
Ign:8 http://deb.debian.org/debian bookworm/main amd64 libpsl5 amd64 0.21.2-1
Ign:9 http://deb.debian.org/debian bookworm/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2+b2
Ign:10 http://deb.debian.org/debian bookworm/main amd64 libssh2-1 amd64 1.10.0-3+b1
Ign:11 http://deb.debian.org/debian bookworm/main amd64 libcurl4 amd64 7.88.1-10+deb12u5
Ign:12 http://deb.debian.org/debian bookworm/main amd64 curl amd64 7.88.1-10+deb12u5
Ign:13 http://deb.debian.org/debian bookworm/main amd64 libldap-common all 2.5.13+dfsg-5
Ign:14 http://deb.debian.org/debian bookworm/main amd64 libsasl2-modules amd64 2.1.28+dfsg-10
Ign:15 http://deb.debian.org/debian bookworm/main amd64 publicsuffix all 20230209.2326-1
Ign:1 http://deb.debian.org/debian bookworm/main amd64 openssl amd64 3.0.11-1~deb12u2
Err:2 http://deb.debian.org/debian bookworm/main amd64 ca-certificates all 20230311
  Temporary failure resolving 'deb.debian.org'
Err:3 http://deb.debian.org/debian bookworm/main amd64 libbrotli1 amd64 1.0.9-2+b6
  Temporary failure resolving 'deb.debian.org'
Err:4 http://deb.debian.org/debian bookworm/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg-10
  Temporary failure resolving 'deb.debian.org'
Err:5 http://deb.debian.org/debian bookworm/main amd64 libsasl2-2 amd64 2.1.28+dfsg-10
  Temporary failure resolving 'deb.debian.org'
Err:6 http://deb.debian.org/debian bookworm/main amd64 libldap-2.5-0 amd64 2.5.13+dfsg-5
  Temporary failure resolving 'deb.debian.org'
Ign:7 http://deb.debian.org/debian bookworm/main amd64 libnghttp2-14 amd64 1.52.0-1+deb12u1
Err:8 http://deb.debian.org/debian bookworm/main amd64 libpsl5 amd64 0.21.2-1
  Temporary failure resolving 'deb.debian.org'
Err:9 http://deb.debian.org/debian bookworm/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2+b2
  Temporary failure resolving 'deb.debian.org'
Err:10 http://deb.debian.org/debian bookworm/main amd64 libssh2-1 amd64 1.10.0-3+b1
  Temporary failure resolving 'deb.debian.org'
Err:1 http://deb.debian.org/debian bookworm/main amd64 openssl amd64 3.0.11-1~deb12u2
  Temporary failure resolving 'deb.debian.org'
Ign:11 http://deb.debian.org/debian bookworm/main amd64 libcurl4 amd64 7.88.1-10+deb12u5
Ign:12 http://deb.debian.org/debian bookworm/main amd64 curl amd64 7.88.1-10+deb12u5
Err:13 http://deb.debian.org/debian bookworm/main amd64 libldap-common all 2.5.13+dfsg-5
  Temporary failure resolving 'deb.debian.org'
Err:14 http://deb.debian.org/debian bookworm/main amd64 libsasl2-modules amd64 2.1.28+dfsg-10
  Temporary failure resolving 'deb.debian.org'
Err:15 http://deb.debian.org/debian bookworm/main amd64 publicsuffix all 20230209.2326-1
  Temporary failure resolving 'deb.debian.org'
Err:7 http://deb.debian.org/debian bookworm/main amd64 libnghttp2-14 amd64 1.52.0-1+deb12u1
  Temporary failure resolving 'deb.debian.org'
Err:11 http://deb.debian.org/debian bookworm/main amd64 libcurl4 amd64 7.88.1-10+deb12u5
  Temporary failure resolving 'deb.debian.org'
Err:12 http://deb.debian.org/debian bookworm/main amd64 curl amd64 7.88.1-10+deb12u5
  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/o/openssl/openssl_3.0.11-1%7edeb12u2_amd64.deb  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian/pool/main/c/ca-certificates/ca-certificates_20230311_all.deb  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian/pool/main/b/brotli/libbrotli1_1.0.9-2%2bb6_amd64.deb  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian/pool/main/c/cyrus-sasl2/libsasl2-modules-db_2.1.28%2bdfsg-10_amd64.deb  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian/pool/main/c/cyrus-sasl2/libsasl2-2_2.1.28%2bdfsg-10_amd64.deb  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian/pool/main/o/openldap/libldap-2.5-0_2.5.13%2bdfsg-5_amd64.deb  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/n/nghttp2/libnghttp2-14_1.52.0-1%2bdeb12u1_amd64.deb  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian/pool/main/libp/libpsl/libpsl5_0.21.2-1_amd64.deb  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian/pool/main/r/rtmpdump/librtmp1_2.4%2b20151223.gitfa8646d.1-2%2bb2_amd64.deb  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian/pool/main/libs/libssh2/libssh2-1_1.10.0-3%2bb1_amd64.deb  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/c/curl/libcurl4_7.88.1-10%2bdeb12u5_amd64.deb  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/c/curl/curl_7.88.1-10%2bdeb12u5_amd64.deb  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian/pool/main/o/openldap/libldap-common_2.5.13%2bdfsg-5_all.deb  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian/pool/main/c/cyrus-sasl2/libsasl2-modules_2.1.28%2bdfsg-10_amd64.deb  Temporary failure resolving 'deb.debian.org'
E: Failed to fetch http://deb.debian.org/debian/pool/main/p/publicsuffix/publicsuffix_20230209.2326-1_all.deb  Temporary failure resolving 'deb.debian.org'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Tried to run the following commands inside the jail:
#!/usr/bin/bash
set -euo pipefail

apt-get update && apt-get -y install ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc

echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
tee /etc/apt/sources.list.d/docker.list > /dev/null

apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# The /usr/bin/nvidia-smi will be present when gpu_passthrough_nvidia=1
if [ -f /usr/bin/nvidia-smi ]; then
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey -o /etc/apt/keyrings/nvidia.asc
chmod a+r /etc/apt/keyrings/nvidia.asc
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/etc/apt/keyrings/nvidia.asc] https://#g' | \
tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

apt-get update
apt-get install -y nvidia-container-toolkit

nvidia-ctk runtime configure --runtime=docker
systemctl restart docker
fi

docker info

Failed to run initial setup... you may want to stop and remove the jail and try again.

jlmkr_log_output.txt

May 28 18:56:56 nas systemd[1]: Starting jlmkr-docker.service - My nspawn jail docker [created with jailmaker]...
May 28 18:56:56 nas .ExecStartPre[2021245]: PRE_START_HOOK
May 28 18:56:56 nas systemd-nspawn[2021247]: systemd 252.22-1~deb12u1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +B>
May 28 18:56:56 nas systemd-nspawn[2021247]: Detected virtualization systemd-nspawn.
May 28 18:56:56 nas systemd-nspawn[2021247]: Detected architecture x86-64.
May 28 18:56:56 nas systemd-nspawn[2021247]: Detected first boot.
May 28 18:56:56 nas systemd-nspawn[2021247]:
May 28 18:56:56 nas systemd-nspawn[2021247]: Welcome to Debian GNU/Linux 12 (bookworm)!
May 28 18:56:56 nas systemd-nspawn[2021247]:
May 28 18:56:56 nas systemd-nspawn[2021247]: Initializing machine ID from container UUID.
May 28 18:56:56 nas systemd-nspawn[2021247]: Populated /etc with preset unit settings.
May 28 18:56:56 nas systemd-nspawn[2021247]: Queued start job for default target graphical.target.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Created slice system-getty.slice - Slice /system/getty.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Created slice system-modprobe.slice - Slice /system/modprobe.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Created slice user.slice - User and Session Slice.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Started systemd-ask-password-consol…quests to Console Directory Watch.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Started systemd-ask-password-wall.p… Requests to Wall Directory Watch.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target cryptsetup.target - Local Encrypted Volumes.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target integritysetup.targe…Local Integrity Protected Volumes.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target paths.target - Path Units.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target remote-cryptsetup.target - Remote Encrypted Volumes.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target remote-fs.target - Remote File Systems.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target remote-veritysetup.t…- Remote Verity Protected Volumes.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target slices.target - Slice Units.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target swap.target - Swaps.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target veritysetup.target - Local Verity Protected Volumes.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Listening on systemd-initctl.socket… initctl Compatibility Named Pipe.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Listening on systemd-journald-dev-l…ocket - Journal Socket (/dev/log).
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Listening on systemd-journald.socket - Journal Socket.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Listening on systemd-networkd.socket - Network Service Netlink Socket.
May 28 18:56:56 nas systemd-nspawn[2021247]:          Mounting dev-hugepages.mount - Huge Pages File System...
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-journald.service - Journal Service...
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-network-generator.… units from Kernel command line...
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-remount-fs.service…nt Root and Kernel File Systems...
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-sysctl.service - Apply Kernel Variables...
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Mounted dev-hugepages.mount - Huge Pages File System.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Finished systemd-remount-fs.service…ount Root and Kernel File Systems.
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-firstboot.service - First Boot Wizard...
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Finished systemd-network-generator.…rk units from Kernel command line.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target network-pre.target - Preparation for Network.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Finished systemd-sysctl.service - Apply Kernel Variables.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Started systemd-journald.service - Journal Service.
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-journal-flush.serv…h Journal to Persistent Storage...
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Finished systemd-firstboot.service - First Boot Wizard.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target first-boot-complete.target - First Boot Complete.
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-sysusers.service - Create System Users...
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Finished systemd-sysusers.service - Create System Users.
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-tmpfiles-setup-dev…ate Static Device Nodes in /dev...
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Finished systemd-tmpfiles-setup-dev…reate Static Device Nodes in /dev.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target local-fs-pre.target …reparation for Local File Systems.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target local-fs.target - Local File Systems.
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-machine-id-commit.… a transient machine-id on disk...
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-networkd.service - Network Configuration...
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Started systemd-networkd.service - Network Configuration.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Finished systemd-journal-flush.serv…ush Journal to Persistent Storage.
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-tmpfiles-setup.ser… Volatile Files and Directories...
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Finished systemd-tmpfiles-setup.ser…te Volatile Files and Directories.
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-resolved.service - Network Name Resolution...
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-update-utmp.servic…rd System Boot/Shutdown in UTMP...
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Finished systemd-update-utmp.servic…cord System Boot/Shutdown in UTMP.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Started systemd-resolved.service - Network Name Resolution.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target network.target - Network.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target nss-lookup.target - Host and Network Name Lookups.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target sysinit.target - System Initialization.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Started apt-daily.timer - Daily apt download activities.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Started apt-daily-upgrade.timer - D… apt upgrade and clean activities.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Started dpkg-db-backup.timer - Daily dpkg database backup timer.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Started e2scrub_all.timer - Periodi…etadata Check for All Filesystems.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Started systemd-tmpfiles-clean.time… Cleanup of Temporary Directories.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target timers.target - Timer Units.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Listening on dbus.socket - D-Bus System Message Bus Socket.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target sockets.target - Socket Units.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target basic.target - Basic System.
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting dbus.service - D-Bus System Message Bus...
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-logind.service - User Login Management...
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-user-sessions.service - Permit User Sessions...
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Started dbus.service - D-Bus System Message Bus.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Finished systemd-user-sessions.service - Permit User Sessions.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Started console-getty.service - Console Getty.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target getty.target - Login Prompts.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Started systemd-logind.service - User Login Management.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target multi-user.target - Multi-User System.
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Reached target graphical.target - Graphical Interface.
May 28 18:56:56 nas systemd-nspawn[2021247]:          Starting systemd-update-utmp-runlev… Record Runlevel Change in UTMP...
May 28 18:56:56 nas systemd-nspawn[2021247]: [  OK  ] Finished systemd-update-utmp-runlev… - Record Runlevel Change in UTMP.
May 28 18:56:59 nas systemd-nspawn[2021247]: [  OK  ] Finished systemd-machine-id-commit.…it a transient machine-id on disk.
May 28 18:56:59 nas systemd[1]: Started jlmkr-docker.service - My nspawn jail docker [created with jailmaker].
May 28 18:57:00 nas systemd-nspawn[2021247]:
May 28 18:57:00 nas systemd-nspawn[2021247]: Debian GNU/Linux 12 docker pts/0
May 28 18:57:00 nas systemd-nspawn[2021247]:

docker_jail_config.txt
host_ip_a_output.txt
jail_ip_a_output.txt
jlmkr_create_output.txt
jlmkr_list_output.txt
jlmkr_start_output.txt
jlmkr_log_output.txt

[mcmehrtens]

Nevermind, it seems to a be a path issue. I'm doing some further investigating, but when my executable was in /mnt/tank/zroot0/jailmaker/jailmaker/jlmkr.py, it failed with the aforementioned issue, but when it was in /mnt/tank/jailmaker/jlmkr.py or /mnt/tank/zroot0/jailmaker/jlmkr.py, it worked.

[mcmehrtens]

@rowanmoul Did you have a similar configuration with jlmkr.py? What was the file path to your executable when it was failing versus when it was succeeding?

[rowanmoul]

My jlmkr.py is at /mnt/myrootdataset/jailmaker/jlmkr.py and jailmaker is definitely a dataset, not just a subfolder on the root dataset. Owner and group of the jailmaker dataset is root, and it is encrypted, while the root dataset is not (probably irrelevant, but thought I'd mention it). I don't think that was ever different during my testing. I might have found that a non-dataset jailmaker subfolder caused problems, but frankly I can't remember.

Probably irrelevant network configuration details

I have a static DHCP mapping on my PFSense router for the interface that TrueNAS is using, but I am not actually using it. TrueNAS is configured with a static IP ("aliases" in the network interface menu), and I also manually set the nameserver and default route under "Global Configuration".
Each of my jails IS using DHCP successfully via static mappings to the virtual interface mac addresses (which thankfully don't change between boots). Not all of my jails are using the interface that TrueNAS is configured to use, but one of them does share that physical NIC.

I'm not sure if any of those networking details are helpful, but given what you have already discovered about the path, I would try creating a completely new dataset with default settings off the root dataset and testing to see if that works for you. Similar to my case, it sounds like it's a path/folder/dataset issue rather than anything more "obvious".

[mcmehrtens]

Hmm, I think I'm going to start a new discussion about this. I think my bug definitely seems to be related to the folder containing jlmkr.py being a folder vs. a dataset. Thank you for your feedback :)