Add a template for building images from Ubuntu cloud images. (#58)

It's faster than using the other Ubuntu templates, and allows to create images
that don't have a kernel in them, like the images from images.maas.io.

It also allows building for arm64, as well as amd64.

Makefile

@@ -5,6 +5,12 @@ FULL_NAME := packer-maas-$(shell git describe --dirty)
 default:
 	$(error Change your working directory to the image name you want to build!)
 
+lint:
+	make -C ubuntu lint
+
+format:
+	make -C ubuntu format
+
 tar:
 	git ls-files --recurse-submodules LICENSE vmware-esxi centos* rhel* | \
 	tar -cJf $(FULL_NAME).tar.xz --transform="s,^,$(FULL_NAME)/," -T -

README.md

@@ -20,21 +20,3 @@ If you wish to use QEMU's UI also remove "headless": true
 
 If you keep "headless": true you can connect using VNC. Packer will output the
 IP and port to connect to when run.
-
-## Makefiles
-
-Each directory contains a Makefile to help build the image with the correct
-arguments. The default make target will remove the output-qemu directory and
-previously generated image before building the new image.
-
-The path to the Packer binary can be overridden with the `PACKER` variable:
-
-```
-$ make PACKER=/home/user/go/bin/packer
-```
-
-Images which require a user specified ISO can be set with the `ISO` variable:
-
-```
-$ make ISO=/path/to/iso
-```

ubuntu/Makefile

@@ -4,6 +4,16 @@ PACKER ?= packer
 
 all: custom-ubuntu-lvm.dd.gz
 
+lint:
+	packer validate -var customize_script=scripts/cloudimg/setup-boot.sh ubuntu-cloudimg.pkr.hcl
+	packer fmt -check -diff .
+
+format:
+	packer fmt .
+
+seeds-cloudimg.iso: user-data-cloudimg meta-data
+	cloud-localds $@ $^
+
 seeds-lvm.iso: user-data-lvm meta-data
 	cloud-localds $@ $^
 

ubuntu/README.md

@@ -1,8 +1,8 @@
-# Ubuntu Packer Template for MAAS
+# Ubuntu Packer Templates for MAAS
 
 ## Introduction
 
-The Packer template in this directory creates a Ubuntu AMD64 image for use with MAAS.
+The Packer templates in this directory creates Ubuntu images for use with MAAS.
 
 ## Prerequisites (to create the image)
 
@@ -18,15 +18,95 @@ The Packer template in this directory creates a Ubuntu AMD64 image for use with
 * [MAAS](https://maas.io) 3.0+
 * [Curtin](https://launchpad.net/curtin) 21.0+
 
-## Customizing the Image
+## ubuntu-cloudimg.pkr.hcl
+
+This template builds a tgz image from the official Ubuntu cloud images. This
+results in an image that is very close to the ones that are on
+https://images.maas.io/.
+
+### Prerequisites
+
+Before the template can be used, `seeds-cloudimg.iso` needs to be created:
+
+```shell
+$ make seeds-cloudimg.iso
+```
+
+This combines user-data-cloudimg and meta-data into a seeds image that sets up
+the VM for image building. Most importantly it sets up ssh access for the root user. A the end of the image build process, this is change is reverted.
+
+You shouldn't modify any of those files.
+
+### Building the image
+
+The build the image you give the template a script which has all the
+customizations:
+
+```shell
+$ sudo packer -var customize_script=my-changes.sh -var ubuntu_series=jammy \
+    ubuntu-cloudimg.pkr.hcl
+```
+
+`my-changes.sh` is a script you write which customizes the image from within
+the VM. For example, you can install packages using `apt-get`, call out to
+ansible, or whatever you want.
+
+#### Accessing external files from you script
+
+If you want to put or use some files in the image, you can put those in the `http` directory.
+
+Whatever file you put there, you can access from within your script like this:
+
+```shell
+$ wget http://${PACKER_HTTP_IP}:${PACKER_HTTP_PORT}:/my-file
+```
+
+### Installing a kernel
+
+Usually, images used by MAAS don't include a kernel. When a machine is deployed
+in MAAS, the approriate kernel is chosen for that machine and installed on top
+of the chosen image.
+
+If you do want to force an image to always use a specific kernel, you can
+include it in the image.
+
+The easiest way of doing this is to use the `kernel` parameter:
+
+```shell
+$ sudo packer build -var kernel=linux-lowlatency -var customize_script=my-changes.sh \
+    ubuntu-cloudimg.pkr.hcl
+```
+
+You can also install the kernel manually in your `my-changes.sh` script, but in
+that case you also need to write the name of the kernel package to
+`/curtin/CUSTOM_KERNEL`. This is to ensure that MAAS won't install another
+kernel on deploy.
+
+
+### Building different architectures
+
+By default, images are produces for amd64. You can build for arm64 as well if
+you specify the `architecture` parameter:
+
+```shell
+$ sudo packer build -var architecture=arm64 -var customize_script=my-changes.sh \
+    ubuntu-cloudimg.pkr.hcl
+```
+
+## ubuntu-flat.json and ubuntu-lvm.json
+
+These templates use an Ubuntu server image to install the image to the VM. It
+takes longer than using a cloud image, but can be useful for certain use cases.
+
+### Customizing the Image
 
 It is possible to customize the image either during the Ubuntu installation or afterwards, before packing the final image. The former is done by providing [autoinstall config](https://ubuntu.com/server/docs/install/autoinstall), editing the _user-data-flat_ and _user-data-lvm_ files. The latter is performed by the _install-custom-packages_ script.
 
-## Building the image using a proxy
+### Building the image using a proxy
 
 The Packer template downloads the Ubuntu net installer from the Internet. To tell Packer to use a proxy set the HTTP_PROXY environment variable to your proxy server. Alternatively you may redefine iso_url to a local file, set iso_checksum_type to none to disable checksuming, and remove iso_checksum_url.
 
-## Building an image
+### Building an image
 
 You can easily build the image using the Makefile:
 
@@ -54,18 +134,12 @@ Note: ubuntu-lvm.json and ubuntu-flat.json are configured to run Packer in headl
 
 Installation is non-interactive.  Note that the installation will attempt an SSH connection to the QEMU VM where the newly-built image is being booted.  This is the final provisioning step in the process.  Packer uses SSH to discover that the image has, in fact, booted, so there may be a number of failed tries -- over 3-5 minutes -- until the connection is successful.  This is normal behaviour for packer.
 
-## Uploading images to MAAS
 
-LVM raw image
+###  Default Username
 
-```shell
-$ maas admin boot-resources create \
-    name='custom/ubuntu-raw' \
-    title='Ubuntu Custom RAW' \
-    architecture='amd64/generic' \
-    filetype='ddgz' \
-    content@=custom-ubuntu-lvm.dd.gz
-```
+The default username is ```ubuntu```
+
+## Uploading images to MAAS
 
 TGZ image
 
@@ -78,6 +152,13 @@ $ maas admin boot-resources create \
     content@=custom-ubuntu.tar.gz
 ```
 
-## Default Username
+LVM raw image
 
-The default username is ```ubuntu```
+```shell
+$ maas admin boot-resources create \
+    name='custom/ubuntu-raw' \
+    title='Ubuntu Custom RAW' \
+    architecture='amd64/generic' \
+    filetype='ddgz' \
+    content@=custom-ubuntu-lvm.dd.gz
+```

ubuntu/meta-data

@@ -0,0 +1,2 @@
+instance-id: iid-local01
+local-hostname: packer-ubuntu

ubuntu/scripts/cloudimg/cleanup.sh

@@ -0,0 +1,34 @@
+#!/bin/bash -e
+#
+# cleanup.sh - Clean up what we did to be able to build the image.
+#
+# Copyright (C) 2022 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+# cloud-init put networking in place on initial boot. Let's remove that, to
+# allow MAAS to configure the networking on deploy.
+rm /etc/netplan/50-cloud-init.yaml
+
+# Everything in /run/packer_backup should be restored.
+find /run/packer_backup
+cp --preserve -r /run/packer_backup/ /
+rm -rf /run/packer_backup
+
+# We had to allow root to ssh for the image setup. Let's try to revert that.
+sed -i s/^root:[^:]*/root:*/ /etc/shadow
+rm -r /root/.ssh
+rm -r /root/.cache
+rm -r /etc/ssh/ssh_host_*

ubuntu/scripts/cloudimg/curtin-hooks

@@ -0,0 +1,59 @@
+#!/usr/bin/env python3
+# curtin-hooks - Curtin installation hooks for Ubuntu
+#
+# Copyright (C) 2022 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import shutil
+
+from curtin.commands.curthooks import builtin_curthooks
+from curtin.config import load_command_config
+from curtin.util import load_command_environment
+
+
+def configure_custom_kernel(config):
+    """Amend the curtin config to explicity specify the kernel to install.
+
+    The name of the kernel to install should already have been written to the
+    CUSTOM_KERNEL file in the same directory as this file.
+    """
+    custom_kernel_path = os.path.join(
+        os.path.dirname(__file__), "CUSTOM_KERNEL")
+    with open(custom_kernel_path, "r") as custom_kernel_file:
+        custom_kernel_package = custom_kernel_file.read().strip()
+    kernel_config = config.setdefault("kernel", {})
+    kernel_config["package"] = custom_kernel_package
+    return config
+
+
+def cleanup():
+    """Remove curtin-hooks so its as if we were never here."""
+    curtin_dir = os.path.dirname(__file__)
+    shutil.rmtree(curtin_dir)
+
+
+def main():
+    state = load_command_environment()
+    config = load_command_config(None, state)
+    target = state['target']
+
+    config = configure_custom_kernel(config)
+    builtin_curthooks(config, target, state)
+    cleanup()
+
+
+if __name__ == "__main__":
+    main()

ubuntu/scripts/cloudimg/install-custom-kernel.sh

@@ -0,0 +1,31 @@
+#!/bin/bash -e
+#
+# install-custom-kernel.sh - Install custom kernel, if specified
+#
+# Copyright (C) 2021 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+export DEBIAN_FRONTEND=noninteractive
+
+if  [ -z  "${CLOUDIMG_CUSTOM_KERNEL}" ]; then
+  echo "Not installing custom kernel, since none was specified."
+  exit 0
+fi
+
+echo "Installing custom kernel ${CLOUDIMG_CUSTOM_KERNEL}"
+apt-get install -y ${CLOUDIMG_CUSTOM_KERNEL}
+
+# Record the installed kernel version, so that the curtin hook knows about it.
+mkdir -p /curtin
+echo -n "${CLOUDIMG_CUSTOM_KERNEL}" > /curtin/CUSTOM_KERNEL

ubuntu/scripts/cloudimg/setup-boot.sh

@@ -0,0 +1,38 @@
+#!/bin/bash -e
+#
+# setup-boot.sh - Set up the image after initial boot
+#
+# Copyright (C) 2022 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+# Configure apt proxy if needed.
+packer_apt_proxy_config="/etc/apt/apt.conf.d/packer-proxy.conf"
+if  [ ! -z  "${http_proxy}" ]; then
+  echo "Acquire::http::Proxy \"${http_proxy}\";" >> ${packer_apt_proxy_config}
+fi
+if  [ ! -z  "${https_proxy}" ]; then
+  echo "Acquire::https::Proxy \"${https_proxy}\";" >> ${packer_apt_proxy_config}
+fi
+
+# Reset cloud-init, so that it can run again when MAAS deploy the image.
+cloud-init clean --logs
+
+# The cloud image for qemu has a kernel already. Remove it, since the user
+# should either install a kernel in the customize script, or let MAAS install
+# the right kernel when deploying.
+apt-get remove --purge -y linux-virtual 'linux-image-*'
+apt-get autoremove --purge -yq
+apt-get clean -yq