LX-1550 create migration image

This change creates a new hook for live-build that creates a tarball
for illumos to Linux migrations. The tarball contains our normal
Linux root filesystem, packaged in a way that it can be understood
by our current upgrade logic (dx_unpack specifically).

The signed hashes file is not generated by default. If you need that
file please specify the following environment variables:
- DLPX_SIGNING_LOGIN
- DLPX_KEY_URL

In addition, I've added the pigz package to the environment of the
docker container to speed up the compression of the tarball.
Using the single-threaded compression from built into tar it would
take ~12m 40s to archive and compress 21GB of data. Pigz in the
container with 2 CPUs brings that to ~9m 41s.

Finally, I've created a migration-scripts directory similar to the
upgrade one. It may seem useless in this PR as I only have dx_prepare
which is basically a no-op, but I actually do have some WIP scripts
there in my local repo for the next steps of the migration.

After running appliance build with the two new environment variables
set I scp'd the compressed migration tar artifact of the interna-qa
variant to an illumos 5.3 VM and called dx_unpack:
```
$ sudo /opt/delphix/server/bin/upgrade/dx_unpack internal-qa.migration.tar.gz
Progress increment: 18:55:58:872276013+0000, 10, Extracting upgrade image
18:55:58:875080352:+0000: Unpacking internal-qa.migration.tar.gz ...
19:05:24:436752844:+0000: done.
Progress increment: 19:05:24:440796731+0000, 40,  verifying format
Progress increment: 19:05:24:507477818+0000, 50,  verifying signature
19:05:24:510590314:+0000: Verify signature ... Verified OK
Progress increment: 19:05:25:072324099+0000, 70, Verifying integrity of upgrade image
19:05:25:075460462:+0000: Verifying contents of dx_apply ... dx_apply: OK
19:05:26:518439350:+0000: Verifying contents of dx_execute ... dx_execute: OK
19:05:26:534407155:+0000: Verifying contents of dx_prepare ... dx_prepare: OK
19:05:26:547907130:+0000: Verifying contents of dx_verify ... dx_verify: OK
19:05:26:558378011:+0000: Verifying contents of os-root.cpio ... os-root.cpio: OK
19:07:03:668049935:+0000: Verifying contents of os-root.hashes ... os-root.hashes: OK
19:07:03:795593227:+0000: Verifying contents of version.info ... version.info: OK
Progress increment: 19:07:03:809621540+0000, 80, Verifying integrity of upgrade image done
Progress increment: 19:07:03:936732121+0000, 90, preparing upgrade image
Progress increment: 19:07:03:944176874+0000, 100, unpacking successful
```

There reason why I tried the internal-qa variant and not the
internal-dev was that the qa variant does not include the source
code of the app-gate and masking and thus takes less space.
(/var/dlpx-update which is the place where images are unpacked has
a reservation of 12G so internal-dev doesn't fit).

Unfortunately we can't really boot into Linux from 5.3 as the rpool
has features like the Log Spacemap that are enabled and don't exist
in ZoL. For that reason I ensured that I could boot into the Linux
image by untaring the internal-qa variant in a 5.2 VM and doing the
following things (some of them will end up in the migration version
of dx_apply which is currently a WIP):

```
TMPDIR=$(mktemp -d -p "/tmp" -t delphix.XXXXXXX)
FSNAME=$(basename "$TMPDIR")
zfs create -o canmount=off -o mountpoint=none "$RPOOL/ROOT"
zfs create -o canmount=noauto -o mountpoint=/ "$RPOOL/ROOT/$FSNAME"

TMP_ROOT="$TMPDIR/root"
zfs set mountpoint="$TMP_ROOT" "$RPOOL/ROOT/$FSNAME"
zfs mount "$RPOOL/ROOT/$FSNAME"
(cd $TMP_ROOT; cpio -idmu) < $ARCHIVE_DIR/os-root.cpio

cp $(ls $TMP_ROOT/boot/initrd.img-* | sort -r | head -n 1) /boot
cp $(ls $TMP_ROOT/boot/vmlinuz-* | sort -r | head -n 1) /boot

zfs umount "$RPOOL/ROOT/$FSNAME"
zfs set mountpoint=/ "$RPOOL/ROOT/$FSNAME"
```

Then rebooting the machine and getting in the bootloader's prompt:
```
> set console=ttya
> load load /boot/vmlinuz-4.15.0-38-generic root=ZFS=rpool/ROOT/delphix.HIFKlSm console=tty0 console=ttyS0,115200n8 zfsforce=1
> load -t rootfs /boot/initrd.img-4.15.0-38-generic
> boot
```

Then wait for the new system to boot up and login (includes interesting bits from console):
```
...
Begin: Sleeping for ... done.
[    6.247809] spl: loading out-of-tree module taints kernel.
[    6.250949] spl: module verification failed: signature and/or required key missing - tainting kernel
[    6.257025] znvpair: module license 'CDDL' taints kernel.
[    6.259017] Disabling lock debugging due to kernel taint
[    7.939793] ZFS: Loaded module v1.0.0-20181101T233420-aa2d882, ZFS pool version 5000, ZFS filesystem version 5
Begin: Sleeping for ... done.
Begin: Importing ZFS root pool 'rpool' ... Begin: Importing pool 'rpool' using defaults ... [    8.136087] print_req_error: I/O error, dev fd0, sector 0
[    8.138347] floppy: error 10 while reading block 0
[    8.249208] random: crng init done
[    8.250434] random: 7 urandom warning(s) missed due to ratelimiting
done.
done.
Begin: Mounting 'rpool/ROOT/delphix.HIFKlSm' on '/root//' ... done.
Begin: Running /scripts/local-bottom ... done.
...
[  OK  ] Mounted Kernel Debug File System.
...
[  OK  ] Started udev Wait for Complete Device Initialization.
[  OK  ] Reached target ZFS pool import target.
         Starting Mount ZFS filesystems...
[  OK  ] Started Mount ZFS filesystems.
...
[  OK  ] Started Initial cloud-init job (metadata service crawler).
[  OK  ] Reached target Cloud-config availability.
...
[  OK  ] Started ZFS Event Daemon (zed).
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Daily Cleanup of Temporary Directories.
         Starting ZFS file system shares...
...
[  OK  ] Started Delphix Perfomance Statistics.
[  OK  ] Started Delphix default PostgreSQL database server.
[  OK  ] Started Delphix Appliance Platform Service
...
localhost login: delphix
password:
```

And then within the VM:
```
delphix@localhost:~$ uname -a
Linux localhost 4.15.0-38-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
```

Other notes:
Postgress is up but the app-stack fails with failures like the following in the
error log:
```
Context initialization failed java.lang.NullPointerException ...
        at java.lang.NullPointerException.<init>(NullPointerException.java:60)
        at com.delphix.appliance.server.upg.impl.UpgradeManagerImpl.setVersionStatus(UpgradeManagerImpl.java:791)
        at com.delphix.appliance.server.upg.impl.UpgradeManagerImpl.versionFromDirectoryLinux(UpgradeManagerImpl.java:850)
        at com.delphix.appliance.server.upg.impl.UpgradeManagerImpl.addNewVersions(UpgradeManagerImpl.java:1022)
        at com.delphix.appliance.server.upg.impl.UpgradeManagerImpl.update(UpgradeManagerImpl.java:907)
        at com.delphix.appliance.server.upg.impl.UpgradeManagerImpl.start(UpgradeManagerImpl.java:463)
```

Will look into it but it can be part of a separate issue.

Makefile

@@ -57,7 +57,8 @@ shellcheck:
 	shellcheck --exclude=SC1090,SC1091 \
 		$$(find scripts -type f $(FINDEXEC)) \
 		$$(find live-build/misc/live-build-hooks -type f $(FINDEXEC)) \
-		$$(find live-build/misc/upgrade-scripts -type f)
+		$$(find live-build/misc/upgrade-scripts -type f) \
+		$$(find live-build/misc/migration-scripts -type f)
 
 #
 # There doesn't appear to be a way to have "shfmt" return non-zero when
@@ -82,7 +83,8 @@ shellcheck:
 shfmtcheck:
 	! shfmt -d $$(find scripts -type f $(FINDEXEC)) \
 		$$(find live-build/misc/live-build-hooks -type f $(FINDEXEC)) \
-		$$(find live-build/misc/upgrade-scripts -type f) | grep .
+		$$(find live-build/misc/upgrade-scripts -type f) \
+		$$(find live-build/misc/migration-scripts -type f) | grep .
 
 ansiblecheck:
 	ansible-lint $$(find bootstrap live-build/variants -name playbook.yml)

docker/Dockerfile

@@ -44,6 +44,7 @@ RUN \
       livecd-rootfs \
       make \
       man \
+      pigz \
       rename \
       shellcheck \
       vim \

live-build/misc/ansible-roles/appliance-build.minimal-common/tasks/main.yml

@@ -44,6 +44,21 @@
     password:
       "{{ lookup('env', 'APPLIANCE_PASSWORD') | password_hash('sha512') }}"
 
+#
+# In a similar fashion with other critical data that are required
+# we specify "no_log" for any information related to signing images.
+# Currently: DLPX_KEY_URL, DLPX_SIGNING_LOGIN.
+#
+- fail:
+    msg: "Required environment variable 'DLPX_KEY_URL' is empty."
+  when: lookup('env', 'DLPX_KEY_URL') == ''
+  no_log: true
+
+- fail:
+    msg: "Required environment variable 'DLPX_SIGNING_LOGIN' is empty."
+  when: lookup('env', 'DLPX_SIGNING_LOGIN') == ''
+  no_log: true
+
 #
 # The virtualization package uses the /etc/issue file to store a
 # customer-supplied banner that is displayed prior to login. By default,

live-build/misc/live-build-hooks/90-linux-migration-artifact.binary

@@ -0,0 +1,165 @@
+#!/bin/bash -eux
+#
+# Copyright 2018 Delphix
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+die() {
+	echo "$*" 1>&2
+	exit 1
+}
+
+#
+# This script is intended to be used as part of Delphix's build process.
+# It's role is to convert the "binary" directory generated by live-build,
+# into a cpio archive that will be wrapper in a tarball. That tarball is
+# the migration artifact that illumos-based engines use to migrate to
+# Linux.
+#
+MIGRATION_ARTIFACT="$APPLIANCE_VARIANT.migration.tar.gz"
+
+#
+# At the time of this writing dx_unpack.sh expects a tarball of the
+# following structure:
+#
+# depot/                       the top level container
+# depot/dx_prepare             top-level script to prepare the archive
+#                              for upgrade
+# depot/version.info           version and date information about the
+#                              upgrade image
+# depot/hashes                 the sha256sum of any upgrade artifacts
+#                              (see below)
+# depot/hashes.sig[.<release>] one cryptographic signature of depot/hashes
+#                              per Delphix release
+# depot/<upgrade artifacts>    generally cpio archives of OS and appliance
+#
+# As dx_upack.sh is the only piece of code that runs from the old
+# world, we treat it as a contract that lists the bare minimum
+# requirements for our artifact.
+#
+DEPOT_DIRECTORY=depot
+
+#
+# Delete any leftover files from previous runs.
+#
+rm -rf "$MIGRATION_ARTIFACT" "$DEPOT_DIRECTORY"
+
+if [ ! -d binary/opt/delphix ]; then
+	echo "This artifact does not contain the appliance." \
+		" Skipping migration image..."
+	exit 0
+fi
+
+mkdir $DEPOT_DIRECTORY
+
+#
+# Generate archive for os root directory.
+#
+# Note: We temporarily change directory in the subshell (we move to the
+# "binary" directory) because we find prints evenrything as a relative
+# path to the current directory and we don't want the generated archive
+# to have "binary" as its top-level directory.
+#
+(
+	cd binary
+	find ./* -depth -print | cpio -ocB
+) >$DEPOT_DIRECTORY/os-root.cpio
+
+#
+# Copy all migration scripts including dx_prepare which is needed by
+# dx_unpack.
+#
+cp migration-scripts/* $DEPOT_DIRECTORY
+
+#
+# There may be a version.info file in the current directory already.
+# That version.info is Linux-specific and does not necessarily have
+# all the fields that the illumos upgrade process requires. Thus, we
+# create our own special version info directly in the depot directory
+# with all the expected fields plus a special variable that indicates
+# that this is a migration image, so dx_prepare knows how to tell the
+# difference between normal illumos upgrade and illumos-to-Linux
+# migrations.
+#
+KERNEL_VERSION=$(basename "$(find binary/boot/vmlinuz-* | sort -r | head -n 1)")
+{
+	echo "DLPX_DATE=$(date '+%Y.%m.%d.%H.%M.%S')"
+	echo "DLPX_OS_VERSION=$KERNEL_VERSION"
+	echo "DLPX_VERSION=$(date '+%Y.%m.%d.%H')"
+	echo "DLPX_MIN_VERSION=5.3.0.0"
+	echo "MIGRATION_IMAGE=y"
+} >$DEPOT_DIRECTORY/version.info
+
+#
+# Generate hashes file.
+#
+(
+	cd binary
+	find ./* -type f -print0 | xargs -0 sha256sum
+) >$DEPOT_DIRECTORY/os-root.hashes || die "generating os-root.hashes failed"
+(
+	cd $DEPOT_DIRECTORY
+	sha256sum ./* >hashes
+)
+
+#
+# DLPX_SIGNING_LOGIN is used for signing our migration image.
+# The signature is later unpacked and verified on the VM
+# performing the migration. If DLPX_SIGNING_LOGIN has not been
+# specified by the user when running this hook we skip generating
+# the signature for this image.
+#
+# Note that DLPX_KEY_URL should also be specified for this to
+# work, as the default is really a placeholder.
+#
+if [ "$DLPX_SIGNING_LOGIN" != "delphix" ]; then
+	#
+	# Generate depot/hashes.sig[.<release>]
+	#
+	# Assumption: we always migrate from version 5.3
+	#
+	UPGRADE_VERSION="5.3"
+	DLPX_SIGNING_URL="$DLPX_KEY_URL/upgrade/keyVersion/$UPGRADE_VERSION/sign"
+
+	hashes_data=$(base64 -w 0 $DEPOT_DIRECTORY/hashes)
+	json_payload="{\"data\": \"${hashes_data}\"}"
+	request_url="$DLPX_SIGNING_URL"
+
+	#
+	# Compute signature
+	#
+	result=$(
+		set -o pipefail
+		echo "$json_payload" | curl -s -f -H \
+			"Content-Type: application/json" \
+			-u "$DLPX_SIGNING_LOGIN" "$request_url" -d @-
+	) || die "Failure signing hashes with $request_url"
+
+	#
+	# Extract signature from JSON response, ensure it is not empty
+	# and save it.
+	#
+	signature=$(echo "$result" | grep signature |
+		sed 's/.*"signature": "\([^"]*\)".*/\1/')
+	[ -n "$signature" ] || die "Empty signature obtained with $request_url"
+	echo "$signature" | base64 -d >$DEPOT_DIRECTORY/hashes.sig.$UPGRADE_VERSION
+fi
+
+#
+# Tape-ARchive & Compress Zee File!
+#
+# We enable verbose output in the hope that all the files that are part of
+# the artifact are logged somewhere which could aid debugging in the future.
+#
+tar -cvf - $DEPOT_DIRECTORY | pigz >"$MIGRATION_ARTIFACT"

live-build/misc/migration-scripts/dx_prepare

@@ -0,0 +1,23 @@
+#!/bin/bash
+#
+# Copyright 2018 Delphix
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+# This script is a no-op and only exist to satisfy dx_unpack.sh
+# from illumos.
+#
+
+exit 0

live-build/variants/external-standard/.gitignore

@@ -19,3 +19,4 @@
 !ansible/**
 !config/hooks
 !upgrade-scripts
+!migration-scripts

live-build/variants/external-standard/migration-scripts

@@ -0,0 +1 @@
+../../misc/migration-scripts

live-build/variants/internal-dcenter/migration-scripts

@@ -0,0 +1 @@
+../../misc/migration-scripts

live-build/variants/internal-dev/.gitignore

@@ -19,3 +19,4 @@
 !ansible/**
 !config/hooks
 !upgrade-scripts
+!migration-scripts

live-build/variants/internal-dev/migration-scripts

@@ -0,0 +1 @@
+../../misc/migration-scripts

live-build/variants/internal-minimal/.gitignore

@@ -19,3 +19,4 @@
 !ansible/**
 !config/hooks
 !upgrade-scripts
+!migration-scripts

live-build/variants/internal-minimal/migration-scripts

@@ -0,0 +1 @@
+../../misc/migration-scripts

live-build/variants/internal-qa/.gitignore

@@ -19,3 +19,4 @@
 !ansible/**
 !config/hooks
 !upgrade-scripts
+!migration-scripts

live-build/variants/internal-qa/migration-scripts

@@ -0,0 +1 @@
+../../misc/migration-scripts

scripts/create-variant.sh

@@ -63,3 +63,6 @@ ln -s ../../../misc/live-build-hooks \
 
 ln -s ../../misc/upgrade-scripts \
 	"$TOP/live-build/variants/$1/upgrade-scripts"
+
+ln -s ../../misc/migration-scripts \
+	"$TOP/live-build/variants/$1/migration-scripts"

scripts/docker-run.sh

@@ -54,6 +54,8 @@ $DOCKER_RUN --rm \
 	--env APPLIANCE_PASSWORD \
 	--env AWS_ACCESS_KEY_ID \
 	--env AWS_SECRET_ACCESS_KEY \
+	--env DLPX_KEY_URL \
+	--env DLPX_SIGNING_LOGIN \
 	--volume "$TOP:/opt/appliance-build" \
 	--workdir "/opt/appliance-build" \
 	appliance-build "$@"

scripts/run-live-build.sh

@@ -46,6 +46,13 @@ set -o pipefail
 #
 export APPLIANCE_PASSWORD="${APPLIANCE_PASSWORD:-delphix}"
 
+#
+# Similarly to the appliance credentials above do the same for info
+# needed to sign artifacts.
+#
+export DLPX_KEY_URL="${DLPX_KEY_URL:-https://example.com}"
+export DLPX_SIGNING_LOGIN="${DLPX_SIGNING_LOGIN:-delphix}"
+
 #
 # We need to be careful to set xtrace after we set the USERNAME and
 # PASSWORD variables above; otherwise, we could leak the values of the
@@ -122,7 +129,7 @@ fi
 # user (e.g. other software); this is most useful when multiple variants
 # are built via a single call to "make" (e.g. using the "all" target).
 #
-for ext in ova qcow2 upgrade.tar.gz gcp.tar.gz vhdx vmdk; do
+for ext in ova qcow2 upgrade.tar.gz migration.tar.gz gcp.tar.gz vhdx vmdk; do
 	if [[ -f "$APPLIANCE_VARIANT.$ext" ]]; then
 		mv "$APPLIANCE_VARIANT.$ext" "$TOP/live-build/artifacts"
 	fi