run-vmtest: action to test bpf using danobi/vmtest

Add a new action to mimick what used to be done by `run-qemu`.

This action is roughly a `cp run-qemu run-vmtest` in term of functionalities.
See end of this commit message for a rationale of this change.

Just like `run-qemu` assumes the presence of a rootfs which is provisioned with
a copy of the script from `ci/vmtest/run_selftests.sh`, `run-vmtest` assumes the
presence of `ci/vmtest/vmtest_selftests.sh` and will run it.

`vmtest_selftests.sh` functionally does the same as `run_selftests.sh` with a
few adjustments to make it work in the `vmtest` world` and leave in the "callee"
repository, not libbpf-ci.

`print_test_summary.py` was copied over unchanged. Later diff will remove the
version from `run-qemu` and point to this one instead.

`action.yml` needs to install a few tools that were historically baked in the rootfs.
A bunch of parameters are now historical.... this diff does not attempt to remove them
yet. This will be address later too and will probably change as libbpf/libbpf use case
is merged in.

`run.sh` gets rid of the logic to create the `qemu` one-liner as well as the
downloading of files from within the rootfs, and adjust it to using files on
disk that were left over by the test run.

Full end-to-end testing will be done through a PR in https://github.com/kernel-patches/vmtest

== Main functional difference between `run-qemu` and `run-vmtest`.

`run-qemu` runs the kernel inside a rootfs which is isolated from the host FS,
meaning that we potentially have a host in which we build the kernel/selftests
which is different than the host we run in. Causing depedency issues, see libbpf#83 and
to some extends libbpf#103 .
We could work around this with statically built binaries, but because of the different
rootfs, we end up having to maintain rootfs *and* doing a dance of copying files
in and out of the rootfs (through the prepare-rootfs action), which is slow, pulls
a fair amount of bytes through the network...

`run-vmtest` on the other hand does mount the host rootfs (read-only) and shares the
current directory (read-write) under /mnt/vmtest.
This resolves 2 things for us.
1. We don't have libraries incompatibilitie issues anymore as the building OS
and the OS running the tests will be the same.
1. We don't need to copy files in and out of the root fs. The files are already
on the FS seen by the VM, and files dumped by the tests are directly accessible
outside the VM.

On top of this, `vmtest` also handles the peculiarities of crafting the right
qemu one-liner.

Signed-off-by: Manu Bretelle <[email protected]>

run-vmtest/action.yml

@@ -0,0 +1,64 @@
+name: 'run vmtest'
+description: 'Run vmtest and print summary'
+inputs:
+  arch:
+    description: 'arch'
+    required: true
+  img:
+    description: 'img path'
+    required: true
+  vmlinuz:
+    description: 'vmlinuz path'
+    required: true
+  kernel-root:
+    description: 'kernel source dir'
+    default: '.'
+  max-cpu:
+    description: 'Maximum number of CPU allocated to a VM (regardless of number of CPUs available on the host). Default is unset, e.g it will default to the number of CPU on the host.'
+    default: ''
+  kernel-test:
+    description: 'Test to run'
+    default: ''
+  output-dir:
+    description: |
+      Some sub-commands produce output dir within VM file system (/command_output/).
+      If this option is set that dir's content would be copied to corresponding location.
+    default: ''
+runs:
+  using: "composite"
+  steps:
+    - name: Find kernel
+      id: find-kernel
+      shell: bash
+      run: |
+        BUILDDIR=$(realpath kbuild-output)
+        vmlinuz="$BUILDDIR/$(KBUILD_OUTPUT="${BUILDDIR}" make -C "${{ inputs.kernel-root }}" -s image_name)"
+        cp "$vmlinuz" ${{ inputs.vmlinuz }}
+    - name: Download vmtest
+      shell: bash
+      # FIXME: move to proper release
+      run: |
+        curl -L https://github.com/chantra/danobi-vmtest/releases/download/v2.1.0/vmtest-$(uname -m) > /usr/bin/vmtest && chmod 755 /usr/bin/vmtest
+    - name: install qemu tools and selftest dependencies
+      shell: bash
+      run: |
+        source "${GITHUB_ACTION_PATH}/../helpers.sh"
+        foldable start install_qemu "Installing QEMU tools"
+        # need gawk to support `--field-separator`
+        sudo apt-get update && sudo apt-get install -y cpu-checker qemu-kvm qemu-utils qemu-system-x86 qemu-system-s390x qemu-system-arm qemu-guest-agent \
+          ethtool keyutils iptables \
+          gawk
+        foldable end install_qemu
+    - name: Run vmtest
+      shell: bash
+      env:
+        VMLINUZ: ${{ inputs.vmlinuz }}
+        IMG: ${{ inputs.img }}
+        KERNEL_ROOT: ${{ inputs.kernel-root }}
+        MAX_CPU: ${{ inputs.max-cpu }}
+        KERNEL_TEST: ${{ inputs.kernel-test }}
+        OUTPUT_DIR: ${{ inputs.output-dir }}
+        PROJECT_NAME: "/mnt/vmtest"
+      run: |
+        ${GITHUB_ACTION_PATH}/run.sh
+

run-vmtest/print_test_summary.py

@@ -0,0 +1,105 @@
+#!/bin/python3
+# prints a summary of the tests to both the console and the job summary:
+# https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#adding-a-job-summary
+#
+# To test the output of the GH test summary:
+# python3 run-vmtest/print_test_summary.py  -j run-vmtest/fixtures/test_progs.json -s /dev/stderr  > /dev/null
+# To test the output of the console:
+# python3 run-vmtest/print_test_summary.py  -j run-vmtest/fixtures/test_progs.json -s /dev/stderr  2> /dev/null
+
+import argparse
+import json
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "-j",
+        "--json-summary",
+        required=True,
+        metavar="FILE",
+        help="test_progs's json summary file",
+    )
+    parser.add_argument(
+        "-s",
+        "--step-summary",
+        required=True,
+        metavar="FILE",
+        help="Github step summary file",
+    )
+    parser.add_argument(
+        "-a", "--append", action="store_true", help="Append to github step summary file"
+    )
+    return parser.parse_args()
+
+
+def notice(text: str) -> str:
+    return f"::notice::{text}"
+
+
+def error(text: str) -> str:
+    return f"::error::{text}"
+
+
+def markdown_summary(json_summary: json):
+    return f"""- :heavy_check_mark: Success: {json_summary['success']}/{json_summary['success_subtest']}
+- :next_track_button: Skipped: ${json_summary['skipped']}
+- :x: Failed: {json_summary['failed']}"""
+
+
+def console_summary(json_summary: json):
+    return f"Success: {json_summary['success']}/{json_summary['success_subtest']}, Skipped: {json_summary['skipped']}, Failed: {json_summary['failed']}"
+
+
+def log_gh_summary(file, text: str):
+    print(text, file=file)
+
+
+def log_console(text: str):
+    print(text)
+
+
+def group(text: str, title: str = "", error: bool = False) -> str:
+    if error and title:
+        title = f"\033[1;31mError:\033[0m {title}"
+    return f"""::group::{title}
+{text}
+::endgroup::"""
+
+
+def test_error_console_log(test_error: str, test_message: str) -> str:
+    error_msg = error(test_error)
+    if test_message:
+        error_msg += "\n" + test_message.strip()
+        return group(error_msg, title=test_error, error=True)
+    else:
+        return error_msg
+
+
+if __name__ == "__main__":
+    args = parse_args()
+    step_open_mode = "a" if args.append else "w"
+    json_summary = None
+
+    with open(args.json_summary, "r") as f:
+        json_summary = json.load(f)
+
+    with open(args.step_summary, step_open_mode) as f:
+        log_gh_summary(f, "# Tests summary")
+        log_gh_summary(f, markdown_summary(json_summary))
+
+        log_console(notice(console_summary(json_summary)))
+
+        for test in json_summary["results"]:
+            test_name = test["name"]
+            test_number = test["number"]
+            if test["failed"]:
+                test_log = f"#{test_number} {test_name}"
+                log_gh_summary(f, test_log)
+                log_console(test_error_console_log(test_log, test["message"]))
+
+            for subtest in test["subtests"]:
+                if subtest["failed"]:
+                    subtest_log = f"#{test_number}/{subtest['number']} {test_name}/{subtest['name']}"
+                    log_gh_summary(f, subtest_log)
+                    log_console(test_error_console_log(subtest_log, subtest["message"]))

run-vmtest/run.sh

@@ -0,0 +1,79 @@
+#!/bin/bash
+
+set -euo pipefail
+trap 'exit 2' ERR
+
+source $(cd $(dirname $0) && pwd)/../helpers.sh
+
+foldable start bpftool_checks "Running bpftool checks..."
+bpftool_exitstatus=0
+
+# bpftool checks are aimed at checking type names, documentation, shell
+# completion etc. against the current kernel, so only run on LATEST.
+if [[ "${KERNEL}" = 'LATEST' ]]; then
+	# "&& true" does not change the return code (it is not executed if the
+	# Python script fails), but it prevents the trap on ERR set at the top
+	# of this file to trigger on failure.
+	"${REPO_ROOT}/${KERNEL_ROOT}/tools/testing/selftests/bpf/test_bpftool_synctypes.py" && true
+	bpftool_exitstatus=$?
+	if [[ $bpftool_exitstatus -eq 0 ]]; then
+		echo "bpftool checks passed successfully."
+	else
+		echo "bpftool checks returned ${bpftool_exitstatus}."
+	fi
+else
+	echo "bpftool checks skipped."
+fi
+
+bpftool_exitstatus="bpftool:${bpftool_exitstatus}"
+foldable end bpftool_checks
+
+foldable start vmtest "Starting virtual machine..."
+
+# HACK: We need to unmount /tmp to access /tmp from the container....
+
+vmtest -k "${VMLINUZ}" --kargs "panic=-1 sysctl.vm.panic_on_oom=1" "umount /tmp && \
+        	/bin/mount bpffs /sys/fs/bpf -t bpf && \
+            ip link set lo up && \
+            cd '${GITHUB_WORKSPACE}' && \
+            ./ci/vmtest/vmtest_selftests.sh ${KERNEL_TEST}"
+
+foldable end vmtest
+
+foldable start collect_status "Collecting exit status"
+
+exitfile="${bpftool_exitstatus}\n"
+exitfile+="$(cat exitstatus 2>/dev/null)"
+exitstatus="$(echo -e "$exitfile" | awk --field-separator ':' \
+  'BEGIN { s=0 } { if ($2) {s=1} } END { print s }')"
+
+if [[ "$exitstatus" =~ ^[0-9]+$ ]]; then
+  printf '\nTests exit status: %s\n' "$exitstatus" >&2
+else
+  printf '\nCould not read tests exit status ("%s")\n' "$exitstatus" >&2
+  exitstatus=1
+fi
+
+foldable end collect_status
+
+# Try to collect json summary from VM
+if [[ -n ${KERNEL_TEST} && ${KERNEL_TEST} =~ test_progs* ]]
+then
+	## Job summary
+	"${GITHUB_ACTION_PATH}/print_test_summary.py" -s "${GITHUB_STEP_SUMMARY}" -j "${KERNEL_TEST}.json"
+fi
+
+# Final summary - Don't use a fold, keep it visible
+echo -e "\033[1;33mTest Results:\033[0m"
+echo -e "$exitfile" | while read result; do
+  testgroup=${result%:*}
+  status=${result#*:}
+  # Print final result for each group of tests
+  if [[ "$status" -eq 0 ]]; then
+    printf "%20s: \033[1;32mPASS\033[0m\n" "$testgroup"
+  else
+    printf "%20s: \033[1;31mFAIL\033[0m (returned %s)\n" "$testgroup" "$status"
+  fi
+done
+
+exit "$exitstatus"