Jailmaker architecture

[Jip-Hop]

For anyone interested in discussing the overall architecture of jailmaker, this is the topic!

Let me start by mentioning some key concepts:

• Single Python file. Easy to deploy, nothing to install. Like a single binary but not compiled.
• Transparency. You can read exactly what it does (no need to trust me).
• Minimal external dependencies. Obviously it requires systemd-nspawn (and tar, xz-utils, sh and coreutils to run the lxc image download script).
• Portability. Store everything inside a single directory. Easy to backup/migrate. Don't modify the host rootfs.
• No lock in. Once you've created your jail with jailmaker, you could throw away the jailmaker script and run your jail directly with the underlying tools (systemd-nspawn etc.). So you're not even dependent on jailmaker for the future of your jail.
• Versatility. Choose from various Linux distros. Make a single jail, or many. Run your apps directly in the jail, or nested inside another container. It's up to the user!
• Guided interactive wizard and full CLI support. Easy to use manually while also supporting scripted usage.
• Full control. The config file offers some convenient options for frequently used features. But it also offers the possibility to add any flags to the underlying tools (systemd-nspawn and systemd-run) as to not limit the user.

[Jip-Hop]

@jonct

[jonct]

Hey! Thanks again for entertaining discussion. I know that's not always the first impulse, and I appreciate and salute you for it.

IMHO that's a pretty good list. I'll add Security, and then treat it as organizing principles for further discussion.

@Lockszmith-GH noted that Discussions are hierarchical. I'll start by posting category banners and then toss in my own 2¢ below each, as just some dude with his own perspectives and a hunch.

[jonct]

I'm starting a second pass. I really can't thank you enough for indulging my long-winded opinion pieces. You're a mensch.

[jonct]

Principle: Single Python file

Easy to deploy, nothing to install. Like a single binary but not compiled.

[jonct]

Hmm. For me, this wouldn't be a principle in itself, but rather an emerging side-effect of your principle of Portability.

Pro: easy to install and easy to relocate.
Con: large monolithic scripts foster spaghetti code, lax testing and difficult/ineffective auditing.

It seems to me that you can preserve all of the Pro while dropping all of the Con by compiling down a modular src/ codebase to a single executable using the .pyz format (but without the unnecessary suffix).

There are also direct security implications, which I'll address under Security.

[Jip-Hop]

I could let go of this principle if someone were to organize my spaghetti into modules. I agree it's the right thing to do. I'd probably instruct users to clone the repo instead of curling a single python file. Bonus of using git is that users can easily upgrade/downgrade by checking out release tags. Curious about the security implications of the current single-file solution though.

[jonct]

Totally happy to adapt and modularize. I see this happening in three waves:

1. I'd start by just moving existing functions as-is into a small number of topic-minded files and importing them from the main script. No actual changes. Rinse and repeat to get the groupings right. That's one PR.
2. Then we lock in some test coverage based on that breakdown. I'll need help there.
3. Then over time we can reason about individual modules: one at a time, with their own discussions and PRs. Backstopped by these regression tests.

IMHO regular users shouldn't need to clone the repo or learn git. They should grab a .tar.gz or a .zip from the Releases page; you have automation now.

Inside that archive I'd only expect to see the single built executable file and a README. The latter of which guides users to put the executable anywhere on their system that suits them, just like any other tool or script.

[jonct]

Principle: Transparency

You can read exactly what it does (no need to trust me).

[jonct]

Bravo! Yes.

It seems to me there's overwhelming consensus on the expected means to achieve this, and that's your use of GitHub.

As briefly mentioned in Single file and soon in Security, the ability to understand and to audit a codebase largely depends on its modular design. Some additional benefits can be found in organizing code modules according to well-known, proven conventions.

All of this applies to robust test coverage, as well. Especially given the challenge of keeping tests up-to-date through ongoing development and maintenance.

[jonct]

Principle: Minimal external dependencies

Obviously it requires systemd-nspawn (and tar, xz-utils, sh and coreutils to run the lxc image download script).

[jonct]

This strikes me as another emerging property of Single file and thus Portability, rather than a guiding principle in and of itself.

I'd offer that the word "external" is key here. For example: PEP 441 would bundle and deliver its dependencies within the compiled .pyz executable. The result is a single file tool, free of external dependencies.

[Jip-Hop]

I didn't know about PEP 441. It's actually more interesting than I first thought, since you mentioned compiled. But it seems the source code is just zipped (not compiled) into a single file, thus still easily readable when extracted. Cool.

[jonct]

Principle: Portability

Store everything inside a single directory. Easy to backup/migrate. Don't modify the host rootfs.

[jonct]

First, just to state up front: none of us means "portability" in the sense of platform-independence. Maybe the word is relocatability, filesystem path agnosticism, or good namespace neighborliness.

My thoughts on this go back to early impressions, and I'll address them separately between the CLI tool and individual jail datasets.

Portability of the executable

For me it's not a single-file install unless I can put it alongside other tools and custom scripts in my $PATH. Currently for me that would mean in ~/.bin. I might choose to add symlinks, or aliases, but by definition those can't be necessary to a single-file install.

You're overloading the parent directory of the tool as a stand-in shorthand for the configuration of a path to the common parent dataset of jails. I didn't expect this, and I'm not fully on board. (Nor do I want any ix_applications dataset at the root of my pool, nor a Microsoft Word settings file with a galling ugly 8.3 filename constantly bubbling up as the most recently modified file in my personal Documents folder. My working namespace is mine alone.)

The tool could easily take that path from a documented sequence of [e.g.]

• ~/.config/jailmaker
• ~/.jailmaker
• $JAILMAKER_PATH
• --dir/-D
• . (but there only if certain conditions are met)

We briefly considered loading dependencies into the jails' parent dataset, alongside the script. Of course that would compromise this principle, and is thus not ideal.

As I've argued in Single file and Minimal external dependencies, a compiled .ptz executable would be more free to roam with other tools and scripts.

Portability of jail root datasets

Similarly, I was somewhat put off being required to name the parent dataset jailmaker in lowercase. These are more unnecessary expressions of opinionated design, which we'd all do well to avoid.

I'll talk about this more in No lock in, but in short: consider that a managed parent dataset may not otherwise need to be any special structured repository of its own.

[Jip-Hop]

Yes I mean something along the lines of 'good namespace neighborliness'.

You're overloading the parent directory of the tool as a stand-in shorthand for the configuration of a path to the common parent dataset of jails.

You describe it quite well. I think it's important to keep in mind the need to create a 'safe space' within the context of SCALE, where changes to the boot pool would be gone after an upgrade (until iX decided to make large parts of the rootfs read-only which prevents changes it in the first places) with further choices by iX remaining unpredictable. Besides I wanted some assurance (for both myself and users) that jailmaker wouldn't end up modifying or removing files or create/destroy zfs datasets in (existing) locations the user didn't know about. Although perhaps not best practice or common, this decision serves its purpose IMHO.

Nor do I want any ix_applications dataset at the root of my pool

If it helps the jailmaker dataset doesn't have to be at the root of your pool. You can put it wherever you like. Configurability of the location where jails reside (and the option to choose the dataset name) will probably come when Sandboxes is properly integrated in SCALE. I think iX was planning a "Choose Sandboxes dataset" setting (akin to setting the Apps Pool).

I don't think there's the need to support storing jails and/or config in so many different places. I feel like your suggestion would complicate things. For example the root user has a different home folder than the admin user (which has root privileges when running commands with sudo). And the root user has a hardcoded home directory of /root... what happens to these files when migrating your data disks to a new NAS? I think currently it's clear where the boundaries of jailmaker are and it makes for a quick getting started (without having to memorize/look up a documented sequence of paths).

I stand by my design decision to put everything jailmaker related in a central place.

[jonct]

changes to the boot pool would be gone after an upgrade

This isn't unique to Jailmaker. I'd mention it in the docs, but not build up any special processes or restrictions around it.

Certainly not any off-the-beaten-path adventurism that goads OCD admins into counteraction with their own workarounds.

I wanted some assurance (for both myself and users) that jailmaker wouldn't end up modifying or removing files or create/destroy zfs datasets in (existing) locations the user didn't know about.

That's where I think an invisible sigil file can help validate expectations and avoid mistakes. Consider the .git/ directory at the root of a source project. It serves multiple purposes, but the silent foundational one is that it marks/anchors the scope of the project's root directory.

macOS uses this pattern extensively for its "bundle" folders that look and behave like self-contained app or document icons.

I think iX was planning a "Choose Sandboxes dataset" setting

In TrueNAS Core they have you select which pool to use, and then park a rigid iocage dataset at the root of that pool. It has a whole slew of contents for managing "jail stuff" and "plugin stuff" and templates and caches and bears.

The admin UI expresses jails in a flat list, so those are all kept as named datasets directly under iocage/jails. Each one consists of at least a config.json, an fstab, and a root/.

Whereas if I take your "No lock in" principle at face value, all of this custom metadata becomes inapplicable noise and detritus. Use the tool and the template to generate a jail, and then the jail is primarily a standalone service object. Fully native to the system.

Sensible defaults and best practice might steer toward creating any and all jails directly under a common parent dataset. Sure; cool by me. But if I take your "Full control" principle at face value, there's no earthly reason to enforce such a convention.

In fact there could be very good reasons to put different jails in different pools, e.g. for different resource and performance characteristics.

Of the dozen or so long-running jails between us collectively, I might be running all but one. 🤪

I think currently it's clear where the boundaries of jailmaker are and it makes for a quick getting started (without having to memorize/look up a documented sequence of paths).

Hey, now. Maybe don't tell a room full of sysadmins that you know better than they do, exactly where and how to keep their tools? 🤓

Like many (most? nearly all?) command-line users I come to the table with reasonable confidence in how and why to keep git(1) in one directory, yet apply its work to my own source-managed project in another. I'm missing why this should be any different.

[Lockszmith-GH]

Hey, now. Maybe don't tell a room full of sysadmins that you know better than they do, exactly where and how to keep their tools? 🤓

Like many (most? nearly all?) command-line users I come to the table with reasonable confidence in how and why to keep git(1) in one directory, yet apply its work to my own source-managed project in another. I'm missing why this should be any different.

I agree. And I understand Jip-Hop's intentions full well, but I think part of the goals should be to steer whoever comes to use this tool - whether experienced or not - towards good practices, let's use the PATH for invocation, and let's make sure the userbase understands the user/root environment barrier properly.

[jonct]

I think part of the goals should be to steer whoever comes to use this tool - whether experienced or not - towards good practices, let's use the PATH for invocation, and let's make sure the userbase understands the user/root environment barrier properly.

TIL that systemd itself wants locally-installed user-owned tools to live in ~/.local/bin by default.

You can confirm this with systemd-path user-binaries or with man 7 file-hierarchy.

Debian and TrueNAS SCALE don't currently include this in a user's $PATH by default — but Ubuntu does.

I came across it because pipx creates and uses that directory if it doesn't already exist. It also appends the necessary $PATH redeclaration to the current shell's startup resource file.

[jonct]

Principle: No lock in

Once you've created your jail with jailmaker, you could throw away the jailmaker script and run your jail directly with the underlying tools (systemd-nspawn etc.). So you're not even dependent on jailmaker for the future of your jail.

[jonct]

it would have to be compiled and then it's definitely no longer a script. 😆

Heh. ⭐️ True, and the reason for that goes point-by-point with my concern over reflexive sudo at a large interpreted script.

I do think the benefits of readable interpreter scripts are vanishingly few today. At least for code hosted in public, and built and distributed right in view. I'd feel a bit safer if more of this privileged execution took place in a fully compiled language.

As a meaningless aside: by nature and custom, I'd have to admit the main tool should probably be written in Go or in Rust. I don't know either of those two, and am in no rush to learn them. C or whatever will do for a stub. Python is nifty, and aligns well with TrueNAS culture.

[Jip-Hop]

$JAIL/.ExecStartPre/Post — I like how you're thinking; I'll follow up in Security

What about this? ^^

[jonct]

What about this? ^^

Yes, I'll absolutely double down on "I like how you're thinking". 🥹

Honestly I haven't yet done a deep dive on which content can or should live inside machine specs. I left myself the escape valve of a start/stop stub script in case the tool has gone away.

FWIW I'm also not yet sure of when a host-side pre or post script is truly necessary: I'm concerned over the inherent dangling side-effects and how they interact given multiple jails triggered out-of-sequence.

I'm agnostic on deep details right now, but I see them being driven by whatever best integrates with the system's expectations.

[Jip-Hop]

If I'd have to integrate systemd-nspawn in SCALE with the reassurance I could put stuff in /var I'd probably symlink /var/lib/machines to a dataset chosen by the user and let machines live there and create/manage them as intended by systemd. Then configure the machines using the native .nspawn config file syntax. With drop-in overrides to configure stuff like seccomp and pre/post start/stop scripts. Then there's no need for my config file format nor yaml.

My research + working POC on this can be found in my TrueNAS SCALE Sandboxes repo. Turns out it's already feasible, even with a readonly rootfs, when using bind mounts.

[jonct]

Very, very, very cool. Thanks so much for the invaluable peak brain dump.

Honestly, for my own use I'd be satisfied if all we could do is create and destroy Debian sandboxes using command-line flags alone. No config, no yaml, no mandatory parent. Folks would share their configs via one-liner script invocations; that's healthy.

The primary use for accepting YAML (if any) would be to facilitate middlewared passing GUI settings as JSON.

Let iX figure out if they want to buy into any value-add package ecosystem at all — and if so, whether that should start with Incus.

[jonct]

Principle: Versatility

Choose from various Linux distros. Make a single jail, or many. Run your apps directly in the jail, or nested inside another container. It's up to the user!

[jonct]

I'm open to "where possible" being answered with "never". But that's an inauspicious goal to set in advance.

I think e.g. modifying iptables from jailmaker code is taking it too far.

(It's nftables; I think iptables commands at this point are all back-compatibility shims.)

I'll venture that we occupy different extremes of the anticipated range of network options — and that this is healthy for the project.

Given Captain_Morgan's mention of their plans…

They can be connected via networking, but it will be like connecting to another machine.

… I reserve the right to help make various options beyond host networking safe, convenient, and scalable.

Actually, on the basis of secure-by-default I'd hope that the default is for no networking at all. In the absence of any declared network settings, maybe the user wants to noodle with ffmpeg or something. The jail should use host networking during the install, but then nothing as it starts up.

Of course activating host networking should be the most trivial enhancement possible. The embodiment of approachability. If in YAML: simply network: host as shown right up at the top of most examples. Maybe even just network: would do it.

[Lockszmith-GH]

I'll venture that we occupy different extremes of the anticipated range of network options — and that this is healthy for the project.

Discussion is always healthy. ALWAYS.
Setting priorities, and taking action are something else. I've never seen Jip-Hop against good contributions - just not keen on venturing into areas that might bite us in the behind.

This is one aspect where testing (which has just taken-off) and good contributions will dictate how this looks like in the future.
While I doubt this would be the case, I really hope iX treats this project as an upstream one, and contribute back. But until then, we're on our own.

Up until now, every accepted contribution to this project literally made it better and brought in more and more users.
Following the rate of contribution and response from Jip-Hop, which I assume is without compensation for the time devoted to this awesome project, I know I wouldn't be able to lead such a thing myself, ever.

This is all to say - offering a contribution, with good testing, would probably a better way to drive this point.

[Jip-Hop]

Discussion is good. But so is action! I've prepared a new v3.0.0 branch from which I have already removed the interactive install process (wizard). Let's see if we can make jailmaker easy to use, even without it.

Totally happy to adapt and modularize. I see this happening in three waves:

1. I'd start by just moving existing functions **as-is** into a small number of topic-minded files and importing them from the main script. No actual changes. Rinse and repeat to get the groupings right. That's one PR.

2. Then we lock in some test coverage based on that breakdown. I'll need help there.

3. Then over time we can reason about individual modules: one at a time, with their own discussions and PRs. Backstopped by these regression tests.

Looking forward to this 3 step process! Please submit PRs containing drastic changes against the v3.0.0 base branch.

[jonct]

On it. 🤝 This is moving nimbly and I commend you for it.

[jonct]

Please submit PRs containing drastic changes against the v3.0.0 base branch.

That's about as drastic as I dare take stage one. I hope it helps.

[jonct]

Principle: Guided interactive wizard and full CLI support

Easy to use manually while also supporting scripted usage.

[jonct]

OK… on this one I'd widen out to "Approachability" and seriously question whether a wizard is a positive way to achieve it.

Pro

• It already exists!
• It's kind of a natural extension of guided multipartite installation.

Con

• It's one-way; no going back to revisit choices.
• The user may not be prepared to answer every question up front. See previous con.
• Separate tracks for beginners vs. beyond pile onto an existing experience gulf.
• And a maintenance gulf.
• Eventually a feature gulf.
• All trending toward toxic support patterns.

With all due caveats about presuming to speak for them: I think beginners would be far better welcomed if we start from the assumption that they can do this, and design accordingly. Lean into progressive disclosure: start with simple, clean, relatable YAML examples. Backstopped by thoughtful defaults. Offering self-evident customization points. And the ability to iterate in place.

On iteration: I'll expand on my premise from No lock in. Assume that the tool is a generator…

The first run of jlmkr create would take a (possibly null) YAML file and produce the jail dataset. The resulting jail could be started and stopped, with or without the tool.

Any subsequent jlmkr update would look at the current state of the YAML file and reconstruct only the necessary metadata and surface area. Restarting the jail as necessary. Oodles of pre-flight safety checks, but baking in unrestricted iteration as its own core principle for the sake of approachability, self-guidance, and adventure.

The user might choose to move the input YAML into the dataset, if so desired. It would not be endangered. My proposed hidden sigil file would be rewritten during any update, as would the other surrounding metadata. But the YAML could live there, or elsewhere, or in source control, or online or whatever. And could be edited and used to update the jail's environment and behavior.

[Jip-Hop]

Hmm not fully convinced yet but I'm not too fond of the wizard myself. So if there's a nice way to offer Approachability without it I'd be interested.

[Lockszmith-GH]

@jonct has point here. Structured interactivity is usually a dumbing down that in my experience is better achieved with good documentation and training material.

Hand holding is a limiting factor.

When you introduced the config templates, that's when this tool really started to click for me, because it allowed me to NOT use the wizard - ever.

[jonct]

Principle: Full control

The config file offers some convenient options for frequently used features. But it also offers the possibility to add any flags to the underlying tools (systemd-nspawn and systemd-run) as to not limit the user.

[jonct]

Yup yup. And at any point if they choose to go it alone, each generated system-standard jail is left as its own starting point.

[jonct]

Principle: Security

Apply best practices throughout internal code. Model and facilitate safe practices for users of every background.

[jonct]

OK. Woof. There's a lot to cover here, and I may have to come back at it a few times. Perhaps after some sleep.

My first stab at modeling recommended safe practices would be to stomp out reflexive sudo, everywhere possible. No sudo for help/version/status, no sudo for dryrun/configtest, absolutely positively no incentive nor encouragement nor even passive endorsement of alias jlmkr='sudo /path/to/jlmkr.py' at any point. You'll need sudo to create, update, or remove a jail. You may or may not need it to start/stop/restart a jail. (I'll double back to how that informs the bundle structure I described in No lock in.)

And that brings us to readable scripts. Below a certain size, it can be helpful to read along. Above a certain size and past a few strategic inflection points, a monolithic script in its direct interpreted form is more pernicious. It starts to serve more practically and directly as a place to hide malware or surprises. Bear in mind that a great many users grab or clone your source file and never revoke its write access — so encouraging them to casually hit it with sudo and leave it sitting around in between sessions is a capital-B bad idea. (I'll double back to this in Single file.)

Even the choice of template structure has safety implications. Right now there are very sharp edges on common network settings: it's a puzzling brew spanning both declarative and imperative stanzas. Even to simply customize an IP address. That could instead be a simple YAML network: host or network: none (the default) or networks: […] and a few lines of network settings per interface. Let the tool figure it out, apply whatever systemd syntax is needed, and report any conflicts (e.g. sorry, you can't join two zones; try changing one network to an existing bridge.) This is where the principles of Approachability and Security reinforce each other in a virtuous cycle. They still generate concrete standard nspawn incantations on disk, but at this level the tool is adding value by helping the user to express clear intent, to receive helpful pre-flight feedback, and to translate to system lingo.

At the development level: I claimed in Single file that modularity can improve auditability, and especially unit testing in lockstep while further development progresses. The key is local reasoning. And this also helps with collaboration — there might be real advantages to aping some cues from iX's in-house Python style and discipline.

Maybe that's a good start for now.

And completes my first pass from top to bottom of this book, so I'm really jazzed to go back and engage with your back-and-forth. Cheers!

[Jip-Hop]

Bear in mind that a great many users grab or clone your source file and never revoke its write access

I mitigated this by automatically revoking this when running jlmkr.py the first time:

jailmaker/jlmkr.py

Lines 1988 to 1989 in b221bf0

	# Set appropriate permissions (if not already set) for this file, since it's executed as root
	stat_chmod(SCRIPT_PATH, 0o700)

I also revoke read and exec permission for anyone except root... And as a consequence sudo is required for any subcommand. 😅

[jonct]

I may have miscommunicated here. Please forgive me if I've transgressed. 🙊

I did not intend to suggest that you've made a string of mistakes that are due fixing in the current code.

My clumsy intent was to acknowledge the daunting scope of planning and refinement, appropriate to this discussion. That this subheading is more complex and more dense than most of the others, and that I can't claim to have encircled it on my first try.