zsys-commit.service will fail when zfs tools don't match DKMS version

[Lockszmith-GH]

Describe the bug
This isn't a bug in zsys so much as a bug in the way Ubuntu manages ZFS packages which zsys relies on.

The root cause is already reported at:

https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1939210

When using HWE based kernel, the zfs-kmod is updated (currently to 2.x) while the LTS zfs packages are at 0.8.3
Currently the solution is to update the tools, the bug linked above points to a 3rd party ppa with updated tools.

The reason for reporting the bug here

• it affects the zfs promote command, which is used after a restore from a previous state occurs. So I thought this would be a good place to document this issue.
• I also suggest that zsysctl should test for this discrepancy and warn about it.
• Hoping an Ubuntu insider would help addressing this issue with the original bug maintainers for next LTS version (coming up in a couple of months)

To Reproduce
Steps to reproduce the behavior:

1. Make sure your system uses HWE kernel
2. Run zfs version and get a discrepancy in versions:

zfs-0.8.3-1ubuntu12.13
zfs-kmod-2.0.6-1ubuntu2

2. Make sure you have a saved zsys state in your GRUB menu.
3. Reboot to GRUB and select a saved state
4. Run systemctl status zsys-commit to see the error.
   zsys-commit service runs the equivalent of sudo zsysctl boot commit which marks the boot as successful.

● zsys-commit.service - Mark current ZSYS boot as successful
    Loaded: loaded (/lib/systemd/system/zsys-commit.service; enabled; vendor preset: enabled)
    Active: failed (Result: exit-code) since Sun 2022-01-23 23:21:41 EST; 10h ago
  Main PID: 12287 (code=exited, status=1/FAILURE)

Jan 23 23:21:40 hostname systemd[1]: Starting Mark current ZSYS boot as successful...
Jan 23 23:21:41 hostname zsysctl[12287]: level=error msg="couldn't commit: couldn't promote dataset \"rpool/ROOT/ubuntu_ssfirw\": couldn't promote \"rpool/ROOT/ubuntu_ssfirw\": not a cloned filesystem"
Jan 23 23:21:41 hostname systemd[1]: zsys-commit.service: Main process exited, code=exited, status=1/FAILURE
Jan 23 23:21:41 hostname systemd[1]: zsys-commit.service: Failed with result 'exit-code'.
Jan 23 23:21:41 hostname systemd[1]: Failed to start Mark current ZSYS boot as successful.

Expected behavior

● zsys-commit.service - Mark current ZSYS boot as successful
     Loaded: loaded (/lib/systemd/system/zsys-commit.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2022-01-24 18:51:11 EST; 15h ago
   Main PID: 297120 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 18951)
     Memory: 0B
     CGroup: /system.slice/zsys-commit.service

Jan 24 18:51:10 Sygin systemd[1]: Starting Mark current ZSYS boot as successful...
Jan 24 18:51:11 Sygin systemd[1]: Finished Mark current ZSYS boot as successful.

ubuntu-bug output
I'm no longer in the broken state, and didn't run ubuntu-bug before.
Bug report (mentioned above) contains much more info.

Installed versions:

• OS: Zorin OS 16 (based on Focal Fossa - Ubuntu 20.04 LTS)
• Zsysd running version: 0.4.8

Additional context
I've documented my ordeal as an askubuntu question which I ended up answering (link).