Fedora hangs immediately prior to Gnome login when booting from Btrfs Assistant read only snapshot

[thecodygriffin]

Summary
Upon attempting to boot a read only snapshot created by Btrfs Assistant, Fedora hangs immediately prior to Gnome login.

References
Thus far, I have thoroughly reviewed and followed the following information related to the topic:

• https://github.com/Antynea/grub-btrfs/tree/master/initramfs
• Services fail when booting into snapshot #92
• b2a3a23

Installation Steps

1. Clone the grub-btrfs repository
   git clone https://github.com/Antynea/grub-btrfs
2. Navigate into the grub-btrfs directory
   cd grub-btrfs
3. Edit the config file
   nano config
4. Set the following parameters in the config file
   GRUB_BTRFS_SYSTEMD_VOLATILE="true"
GRUB_BTRFS_GRUB_DIRNAME="/boot/grub2"
GRUB_BTRFS_MKCONFIG=/sbin/grub2-mkconfig
GRUB_BTRFS_SCRIPT_CHECK=grub2-script-check
5. Save and exit the config file
6. Build and install grub-btrfs
   sudo make install
7. Create and edit the 02_tpm file
   sudo nano /etc/grub.d/02_tpm
8. Add the following lines to the 02_tpm file

   #!/usr/bin/sh -e
   #
   # A grub configuration file to avoid the following error when booting snapshots
   # error: ../../grub-core/commands/efi/tpm.c:150:unknown TPM error
   
   echo "rmmod tpm"

9. Save and exit the 02_tpm file
10. Make the 02_tpm file executable
    sudo chmod +x /etc/grub.d/02_tpm
11. Update the grub configuration
    sudo grub2-mkconfig -o /boot/grub2/grub.cfg
12. Enable and start the grub-btrfs systemd service
    sudo systemctl enable --now grub-btrfsd.service
13. Leave the grub-btrfs directory
    cd ..
14. Remove the grub-btrfs directory
    rm -rf grub-btrfs

What I Have Attempted
Thus far I have attempted the following:

• declared GRUB_BTRFS_SNAPSHOT_KERNEL_PARAMETERS="rd.live.overlay.overlayfs=1" in the /etc/default/grub-btrfs/config file
• declared GRUB_BTRFS_SNAPSHOT_KERNEL_PARAMETERS="rd.live.overlay.readonly=1" in the /etc/default/grub-btrfs/config file
• declared GRUB_BTRFS_SYSTEMD_VOLATILE="true" in the /etc/default/grub-btrfs/config file

After each update to the config file, I do the following:

1. run sudo /etc/grub.d/41_snapshots-btrfs to detect the snapshots for the grub menu
2. run sudo grub2-mkconfig -o /boot/grub2/grub.cfg to rebuild the grub menu (which I believe is redundant, but harmless)
3. create a new snapshot manually via Btrfs Assistant for testing to rule out that a snapshot prior to the change is causing the issue

In each case, the result is the same.

Work-Around
Once a week, I am manually creating a snapshot in Btrfs Assistant and changing it to read-write. Although less than ideal, these snapshots boot without fail and would allow me to recover my system in an absolute worst case scenario.

Goal
I am hoping that I am overlooking something and that a fresh set of eyes can easily identify it and point me in the correct direction.

My preference is to declare and use GRUB_BTRFS_SYSTEMD_VOLATILE="true" in the /etc/default/grub-btrfs/config file since it seems to be the simplest solution given my installation and snapshots satisfy the criteria for it to be used.

I do not want to create subvolumes for /var and or any of its subdirectories. My goal with snapshots and grub-btrfs is to simplify recovery without adding to system complexity.

System Details

• Secure Boot: Enabled
• TPM v2.0: Enabled
• Operating System: Fedora Linux 39 (Workstation Edition)
• Kernel: 6.7.9-200.fc39.x86_64
• Desktop Environment: Gnome 45.4 on Wayland
• Btrfs Assistant: 1.9-1.fc39
• grub: grub2-efi-x64-2.06-118.fc39.x86_64
• grub-btrfs: 4.3
• Device Partition |Mount Point | Subvolume | Options
  • /dev/nvme0n1p1 | /boot/efi | umask=0007,shortname=winnt
  • /dev/nvme0n1p2 | /boot | defaults
  • /dev/nvme0n1p3 | / | root | subvol=root,compress=zstd:1
  • /dev/nvme0n1p3 | /home | home | subvol=home,compress=zstd:1

Latest Config

#!/usr/bin/env bash


GRUB_BTRFS_VERSION=4.13-yabsnap_info_support-2024-03-06T13:43:57+00:00

# Disable grub-btrfs.
# Default: "false"
#GRUB_BTRFS_DISABLE="true"

# Name appearing in the Grub menu.
# Default: "Use distribution information from /etc/os-release."
#GRUB_BTRFS_SUBMENUNAME="Arch Linux snapshots"

# Custom title.
# Shows/Hides "date" "snapshot" "type" "description" in the Grub menu, custom order available.
# Default: ("date" "snapshot" "type" "description")
#GRUB_BTRFS_TITLE_FORMAT=("date" "snapshot" "type" "description")

# Limit the number of snapshots populated in the GRUB menu.
# Default: "50"
#GRUB_BTRFS_LIMIT="50"

# Sort the found subvolumes by "ogeneration" or "generation" or "path" or "rootid".
# # See Sorting section to https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-subvolume#SUBCOMMAND
# "-rootid" means list snapshot by new ones first.
# Default: "-rootid"
#GRUB_BTRFS_SUBVOLUME_SORT="+ogen,-gen,path,rootid"

# Show snapshots found during run "grub-mkconfig"
# Default: "true"
#GRUB_BTRFS_SHOW_SNAPSHOTS_FOUND="false"

# Show Total of snapshots found during run "grub-mkconfig"
# Default: "true"
#GRUB_BTRFS_SHOW_TOTAL_SNAPSHOTS_FOUND="true"

# By default, "grub-btrfs" automatically detects most existing kernels.
# If you have one or more custom kernels, you can add them here.
# Default: ("")
#GRUB_BTRFS_NKERNEL=("kernel-custom" "vmlinux-custom")

# By default, "grub-btrfs" automatically detects most existing initramfs.
# If you have one or more custom initramfs, you can add them here.
# Default: ("")
#GRUB_BTRFS_NINIT=("initramfs-custom.img" "initrd-custom.img" "otherinit-custom.gz")

# By default, "grub-btrfs" automatically detects most existing microcodes.
# If you have one or more custom microcodes, you can add them here.
# Default: ("")
#GRUB_BTRFS_CUSTOM_MICROCODE=("custom-ucode.img" "custom-uc.img "custom_ucode.cpio")

# Additional kernel command line parameters that should be passed to the kernel
# when booting a snapshot.
# For dracut based distros this could be useful to pass "rd.live.overlay.overlayfs=1"
# or "rd.live.overlay.readonly=1" to the Kernel for booting snapshots read only.
# Default: ""
#GRUB_BTRFS_SNAPSHOT_KERNEL_PARAMETERS="rd.live.overlay.overlayfs=1"
GRUB_BTRFS_SYSTEMD_VOLATILE="true"

# Comma separated mount options to be used when booting a snapshot.
# They can be defined here as well as in the "/" line inside the respective snapshots'
# "/etc/fstab" files.  Mount options found in both places are combined, and this variable
# takes priority over `fstab` entries.
# NB: Do NOT include "subvol=..." or "subvolid=..." here.
# Default: ""
#GRUB_BTRFS_ROOTFLAGS="space_cache,commit=10,norecovery"

# Ignore specific path during run "grub-mkconfig".
# Only exact paths are ignored.
# e.g : if `specific path` = @, only `@` snapshot will be ignored.
# Default: ("@")
GRUB_BTRFS_IGNORE_SPECIFIC_PATH=("@")

# Ignore prefix path during run "grub-mkconfig".
# Any path starting with the specified string will be ignored.
# e.g : if `prefix path` = @, all snapshots beginning with "@/..." will be ignored.
# Default: ("var/lib/docker" "@var/lib/docker" "@/var/lib/docker")
GRUB_BTRFS_IGNORE_PREFIX_PATH=("var/lib/docker" "@var/lib/docker" "@/var/lib/docker")

# Ignore specific type/tag of snapshot during run "grub-mkconfig".
# For snapper:
# Type = single, pre, post.
# For Timeshift:
# Tag = boot, ondemand, hourly, daily, weekly, monthly.
# For yabsnap:
# Trigger = S, I, U.
# Default: ("")
#GRUB_BTRFS_IGNORE_SNAPSHOT_TYPE=("")

# Ignore specific description of snapshot during run "grub-mkconfig".
# e.g: timeline
# Default: ("")
#GRUB_BTRFS_IGNORE_SNAPSHOT_DESCRIPTION=("")

# By default "grub-btrfs" automatically detects your boot partition,
# either located at the system root or on a separate partition or in a subvolume,
# Change to "true" if your boot partition isn't detected as separate.
# Default: "false"
#GRUB_BTRFS_OVERRIDE_BOOT_PARTITION_DETECTION="true"

# Location of the folder containing the "grub.cfg" file.
# Might be grub2 on some systems.
# Default: "/boot/grub"
GRUB_BTRFS_GRUB_DIRNAME="/boot/grub2"

# Location of kernels/initramfs/microcode.
# Use by "grub-btrfs" to detect the boot partition and the location of kernels/initrafms/microcodes.
# Default: "/boot"
#GRUB_BTRFS_BOOT_DIRNAME="/boot"

# Location where grub-btrfs.cfg should be saved.
# Some distributions (like OpenSuSE) store those files at the snapshot directory
# instead of boot. Be aware that this directory must be available for grub during
# startup of the system.
# Default: $GRUB_BTRFS_GRUB_DIRNAME
#GRUB_BTRFS_GBTRFS_DIRNAME="/boot/grub"

# Location of the directory where Grub searches for the grub-btrfs.cfg file.
# Some distributions (like OpenSuSE) store those file at the snapshot directory
# instead of boot. Be aware that this directory must be available for grub during
# startup of the system.
# Default: "\${prefix}" # This is a grub variable that resolves to where grub is
# installed. (like /boot/grub, /boot/efi/grub)
# NOTE: If variables of grub are used here (like ${prefix}) they need to be escaped
# with `\` before the `$`
#GRUB_BTRFS_GBTRFS_SEARCH_DIRNAME="\${prefix}"


# Name/path of grub-mkconfig command, use by "grub-btrfs.service"
# Might be 'grub2-mkconfig' on some systems (Fedora ...)
# Default paths are /sbin:/bin:/usr/sbin:/usr/bin,
# if your path is missing, report it on the upstream project.
# For example, on Fedora : "/sbin/grub2-mkconfig"
# You can use only name or full path.
# Default: grub-mkconfig
GRUB_BTRFS_MKCONFIG=/sbin/grub2-mkconfig

# Name of grub-script-check command, use by "grub-btrfs"
# Might be 'grub2-script-check' on some systems (Fedora ...)
# For example, on Fedora : "grub2-script-check"
# Default: grub-script-check
GRUB_BTRFS_SCRIPT_CHECK=grub2-script-check

# Path of grub-mkconfig_lib file, use by "grub-btrfs"
# Might be '/usr/share/grub2/grub-mkconfig_lib' on some systems (Opensuse ...)
# Default: /usr/share/grub/grub-mkconfig_lib
#GRUB_BTRFS_MKCONFIG_LIB=/usr/share/grub2/grub-mkconfig_lib

# Password protection management for submenu,snapshots
# Refer to the Grub documentation https://www.gnu.org/software/grub/manual/grub/grub.html#Authentication-and-authorisation
# and this comment https://github.com/Antynea/grub-btrfs/issues/95#issuecomment-682295660
#
# Add authorized usernames separate by comma (foo,bar)
# When Grub's password protection is enabled, the superuser is authorized by default, it isn't necessary to add it
# Default: ""
#GRUB_BTRFS_PROTECTION_AUTHORIZED_USERS="foo,bar"
#
# Disable authentication support for submenu of Grub-btrfs only (--unrestricted)
# doesn't work if GRUB_BTRFS_PROTECTION_AUTHORIZED_USERS isn't empty
# Default: "false"
#GRUB_BTRFS_DISABLE_PROTECTION_SUBMENU="true"

[smathev]

Hi.

Thanks for documenting so thouroughly your tests and tries. I have the same issue as you do, and I can't seem to find an answer neither.
For now, when I choose to boot into a snapshot I get stuck in terminal right when it prints:
[Ok] Reached target graphical interface.

But, I noticed that I can make a snapshot boot, if I make it read-write enabled or rather read-only=false.

I've chosen to make a script that finds all snapshots in the snapshots dir and make them read-write, so I can boot them.
I have no idea if that's dangerous in the long run, but I seem able to boot of my snapshots and restore them - which is what I would want.

I'm in the process of creating a path_unit and a service that would execute whenever it detects changes in my snapshots-dir, so all snapshots that are created are then set to read-only=false.

I'm hoping that the proper way, the overlayfs way, is implemented or someone else can discover what's going wrong - so I don't have to keep my somewhat hacky solution around.

[HanM23]

Apparently, i have the same issue, i created it here Unable to boot into a snapshot since kernel 6.8 (read-only), but i can with 6.6, it's probably the same but with mkinitcpio.

I am on Manjaro/KDE, and i cannot boot any snapshot created with kernel 6.8. I do not have any issue with kernel 6.6.
When i boot up into a snapshot, my system hangs, i can login into TTY, and there i clearly see my system is read-only although the grub-btrfs-overlayfs hook is well implemented and no issue if i choose kernel 6.6 when i boot my snapshot.

Like you if i set the 'ro' property of the snapshot to false, i can boot normally, witout issue, but not ideal solution.

it seems something has changed with kernel 6.8 (i cannot say how it is with 6.7 i did not install it) and the hook does not work anymore.

[HanM23]

I've chosen to make a script that finds all snapshots in the snapshots dir and make them read-write, so I can boot them.

Would it be possible for you to share this script ?

At first i wanted to change existing snapper or btrfs-assistant internal scripts because we can create a R/W snapshot with snapper with --read-write option but there is apparently no possibility of customization.

Thank you