Unable to boot into a snapshot since kernel 6.8 (read-only), but i can with 6.6

[HanM23]

Hi,

I cannot boot into a snapshot since kernel 6.8. I have two kernels installed in my system 6.6 lts and 6.8.
Presets for both are exactly the same, and i can boot into any snapshot if i choose kernel 6.6 in the grub menu after i have chosen the snapshot itself.

If i boot on a snapshot with kernel 6.8, it does not want to start KDE, i can switch with ctrl+alt F to tty, i can then login in, and when i want to start KDE with startx i have the following message :

Can't move /var/log/Xorg.0.log to /var/log/Xorg.0.log.old. This is surprising because /var/log is a subvolume, and i do not make any snapshot of it, only the subvolume /.
By checking the logs, yes, my snapshot's filesystem is read-only, therefore i cannot boot to kde. But i really do not understand why,

Once again i do not have this issue at all with 6.6, i can boot my snapshots normally with it.

I do not know really how to investigate more. Do you have any nice idea ? I have Manjaro/KDE with Snapper and Btrfs-Assistant.

Thank you

*Here is the hook line from mkinitcpio.conf :

HOOKS=(base udev autodetect microcode kms modconf block keyboard keymap consolefont plymouth encrypt filesystems grub-btrfs-overlayfs)

*Here is my config

#!/usr/bin/env bash


GRUB_BTRFS_VERSION=4.12-master-2023-04-28T16:26:00+00:00

# Disable grub-btrfs.
# Default: "false"
#GRUB_BTRFS_DISABLE="true"

# Name appearing in the Grub menu.
# Default: "Use distribution information from /etc/os-release."
#GRUB_BTRFS_SUBMENUNAME="Arch Linux snapshots"

# Custom title.
# Shows/Hides "date" "snapshot" "type" "description" in the Grub menu, custom order available.
# Default: ("date" "snapshot" "type" "description")
#GRUB_BTRFS_TITLE_FORMAT=("date" "snapshot" "type" "description")

# Limit the number of snapshots populated in the GRUB menu.
# Default: "50"
#GRUB_BTRFS_LIMIT="50"

# Sort the found subvolumes by "ogeneration" or "generation" or "path" or "rootid".
# # See Sorting section to https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-subvolume#SUBCOMMAND
# "-rootid" means list snapshot by new ones first.
# Default: "-rootid"
#GRUB_BTRFS_SUBVOLUME_SORT="+ogen,-gen,path,rootid"

# Show snapshots found during run "grub-mkconfig"
# Default: "true"
#GRUB_BTRFS_SHOW_SNAPSHOTS_FOUND="false"

# Show Total of snapshots found during run "grub-mkconfig"
# Default: "true"
#GRUB_BTRFS_SHOW_TOTAL_SNAPSHOTS_FOUND="true"

# By default, "grub-btrfs" automatically detects most existing kernels.
# If you have one or more custom kernels, you can add them here.
# Default: ("")
#GRUB_BTRFS_NKERNEL=("kernel-custom" "vmlinux-custom")

# By default, "grub-btrfs" automatically detects most existing initramfs.
# If you have one or more custom initramfs, you can add them here.
# Default: ("")
#GRUB_BTRFS_NINIT=("initramfs-custom.img" "initrd-custom.img" "otherinit-custom.gz")

# By default, "grub-btrfs" automatically detects most existing microcodes.
# If you have one or more custom microcodes, you can add them here.
# Default: ("")
#GRUB_BTRFS_CUSTOM_MICROCODE=("custom-ucode.img" "custom-uc.img "custom_ucode.cpio")

# Additonal kernel command line parameters that should be passed to the kernel
# when booting a snapshot.
# For dracut based distros this could be useful to pass "rd.live.overlay.overlayfs=1"
# or "rd.live.overlay.readonly=1" to the Kernel for booting snapshots read only.
# Default: ""
#GRUB_BTRFS_SNAPSHOT_KERNEL_PARAMETERS="rd.live.overlay.overlayfs=1"

# Comma seperated mount options to be used when booting a snapshot.
# They can be defined here as well as in the "/" line inside the respective snapshots'
# "/etc/fstab" files.  Mount options found in both places are combined, and this variable
# takes priority over `fstab` entries.
# NB: Do NOT include "subvol=..." or "subvolid=..." here.
# Default: ""
#GRUB_BTRFS_ROOTFLAGS="space_cache,commit=10,norecovery"

# Ignore specific path during run "grub-mkconfig".
# Only exact paths are ignored.
# e.g : if `specific path` = @, only `@` snapshot will be ignored.
# Default: ("@")
GRUB_BTRFS_IGNORE_SPECIFIC_PATH=("@")

# Ignore prefix path during run "grub-mkconfig".
# Any path starting with the specified string will be ignored.
# e.g : if `prefix path` = @, all snapshots beginning with "@/..." will be ignored.
# Default: ("var/lib/docker" "@var/lib/docker" "@/var/lib/docker")
GRUB_BTRFS_IGNORE_PREFIX_PATH=("var/lib/docker" "@var/lib/docker" "@/var/lib/docker")

# Ignore specific type/tag of snapshot during run "grub-mkconfig".
# For snapper:
# Type = single, pre, post.
# For Timeshift:
# Tag = boot, ondemand, hourly, daily, weekly, monthly.
# Default: ("")
#GRUB_BTRFS_IGNORE_SNAPSHOT_TYPE=("")

# Ignore specific description of snapshot during run "grub-mkconfig".
# e.g: timeline
# Default: ("")
#GRUB_BTRFS_IGNORE_SNAPSHOT_DESCRIPTION=("")

# By default "grub-btrfs" automatically detects your boot partition,
# either located at the system root or on a separate partition or in a subvolume,
# Change to "true" if your boot partition isn't detected as separate.
# Default: "false"
#GRUB_BTRFS_OVERRIDE_BOOT_PARTITION_DETECTION="true"

# Location of the folder containing the "grub.cfg" file.
# Might be grub2 on some systems.
# Default: "/boot/grub"
#GRUB_BTRFS_GRUB_DIRNAME="/boot/grub2"

# Location of kernels/initramfs/microcode.
# Use by "grub-btrfs" to detect the boot partition and the location of kernels/initrafms/microcodes.
# Default: "/boot"
#GRUB_BTRFS_BOOT_DIRNAME="/boot"

# Location where grub-btrfs.cfg should be saved.
# Some distributions (like OpenSuSE) store those files at the snapshot directory
# instead of boot. Be aware that this direcory must be available for grub during
# startup of the system.
# Default: $GRUB_BTRFS_GRUB_DIRNAME
#GRUB_BTRFS_GBTRFS_DIRNAME="/boot/grub"

# Location of the directory where Grub searches for the grub-btrfs.cfg file.
# Some distributions (like OpenSuSE) store those file at the snapshot directory
# instead of boot. Be aware that this direcory must be available for grub during
# startup of the system.
# Default: "\${prefix}" # This is a grub variable that resolves to where grub is
# installed. (like /boot/grub, /boot/efi/grub)
# NOTE: If variables of grub are used here (like ${prefix}) they need to be escaped
# with `\` before the `$`
#GRUB_BTRFS_GBTRFS_SEARCH_DIRNAME="\${prefix}"


# Name/path of grub-mkconfig command, use by "grub-btrfs.service"
# Might be 'grub2-mkconfig' on some systems (Fedora ...)
# Default paths are /sbin:/bin:/usr/sbin:/usr/bin,
# if your path is missing, report it on the upstream project.
# For example, on Fedora : "/sbin/grub2-mkconfig"
# You can use only name or full path.
# Default: grub-mkconfig
#GRUB_BTRFS_MKCONFIG=/usr/bin/grub2-mkconfig

# Name of grub-script-check command, use by "grub-btrfs"
# Might be 'grub2-script-check' on some systems (Fedora ...)
# For example, on Fedora : "grub2-script-check"
# Default: grub-script-check
#GRUB_BTRFS_SCRIPT_CHECK=grub2-script-check

# Path of grub-mkconfig_lib file, use by "grub-btrfs"
# Might be '/usr/share/grub2/grub-mkconfig_lib' on some systems (Opensuse ...)
# Default: /usr/share/grub/grub-mkconfig_lib
#GRUB_BTRFS_MKCONFIG_LIB=/usr/share/grub2/grub-mkconfig_lib

# Password protection management for submenu,snapshots
# Refer to the Grub documentation https://www.gnu.org/software/grub/manual/grub/grub.html#Authentication-and-authorisation
# and this comment https://github.com/Antynea/grub-btrfs/issues/95#issuecomment-682295660
#
# Add authorized usernames separate by comma (foo,bar)
# When Grub's password protection is enabled, the superuser is authorized by default, it isn't necessary to add it
# Default: ""
#GRUB_BTRFS_PROTECTION_AUTHORIZED_USERS="foo,bar"
#
# Disable authentication support for submenu of Grub-btrfs only (--unrestricted)
# doesn't work if GRUB_BTRFS_PROTECTION_AUTHORIZED_USERS isn't empty
# Default: "false"
#GRUB_BTRFS_DISABLE_PROTECTION_SUBMENU="true"

[HanM23]

I get some more information once i boot into my snapshot in tty :

lsinitcpio -a /boot/initramfs-6.8-x64_64.img

==> Image: /boot/initramfs-6.8-x86_64.img
==> Created with mkinitcpio 38.1
==> Kernel: 6.8.3-1-MANJARO
==> Early CPIO: 220 KiB
==> Size: 62.51 MiB
==> Compressed with: gzip
  -> Uncompressed size: 125 MiB (.500 ratio)
  -> Estimated decompression time: 0,659s

==> Included modules:
  842				  cast6-avx-x86_64		  echainiv			  nhpoly1305			  sm3-avx-x86_64
  842_compress			  cast6_generic			  ecrdsa_generic		  nhpoly1305-avx2		  sm3_generic
  842_decompress		  cast_common			  encrypted-keys		  nhpoly1305-sse2		  sm4
  adiantum			  cbc				  essiv				  nouveau			  sm4-aesni-avx-x86_64
  aegis128			  ccm				  fcrypt			  nvme				  sm4-aesni-avx2-x86_64
  aegis128-aesni		  ccp				  gcm				  nvme-auth			  sm4_generic
  aes_ti			  ccp-crypto			  geniv				  nvme-core			  spi-intel
  aesni-intel			  cec				  gf128mul			  overlay			  spi-intel-pci
  af_alg			  chacha-x86_64			  ghash-clmulni-intel		  padlock-aes			  streebog_generic
  algif_aead			  chacha20poly1305		  ghash-generic			  padlock-sha			  tcrypt
  algif_hash			  chacha_generic		  gpu-sched			  pcbc				  tee
  algif_rng			  chcr				  hctr2				  pcrypt			  tls
  algif_skcipher		  cmac				  hid-generic			  pkcs8_key_parser		  trusted
  amlogic-gxl-crypto		  crc16				  i2c-algo-bit			  poly1305-x86_64		  ttm
  ansi_cprng			  crc32-pclmul			  i8042				  poly1305_generic		  twofish-avx-x86_64
  aria-aesni-avx-x86_64		  crc32_generic			  iaa_crypto			  polyval-clmulni		  twofish-x86_64
  aria-aesni-avx2-x86_64	  crc32c-intel [explicit]	  idxd				  polyval-generic		  twofish-x86_64-3way
  aria-gfni-avx512-x86_64	  crc32c_generic		  idxd_bus			  pps_core			  twofish_common
  aria_generic			  crc8				  intel_qat			  ptp				  twofish_generic
  asn1_encoder			  crct10dif-pclmul		  keywrap			  qat_420xx			  uas
  async_memcpy			  cryptd			  libarc4			  qat_4xxx			  usb-storage
  async_pq			  crypto_engine			  libchacha			  qat_c3xxx			  usbhid
  async_raid6_recov		  crypto_null			  libchacha20poly1305		  qat_c3xxxvf			  video
  async_tx			  crypto_safexcel		  libcrc32c			  qat_c62x			  virtio_blk
  async_xor			  crypto_simd			  libcurve25519			  qat_c62xvf			  virtio_crypto
  atkbd				  crypto_user			  libcurve25519-generic		  qat_dh895xcc			  virtio_scsi
  atmel-ecc			  cts				  libdes			  qat_dh895xccvf		  vivaldi-fmap
  atmel-i2c			  curve25519-generic		  libpoly1305			  raid6_pq			  vmac
  atmel-sha204a			  curve25519-x86_64		  libps2			  raid6test			  vmd
  authenc			  cxgb4				  lrw				  rmd160			  wmi
  authencesn			  deflate			  lz4				  seqiv				  wp512
  blake2b_generic		  des3_ede-x86_64		  lz4_compress			  serio				  xcbc
  blowfish-x86_64		  des_generic			  lz4hc				  serio_raw			  xctr
  blowfish_common		  dm-bufio			  lz4hc_compress		  serpent-avx-x86_64		  xhci-pci
  blowfish_generic		  dm-crypt			  lzo				  serpent-avx2			  xhci-pci-renesas
  btrfs				  dm-integrity			  lzo-rle			  serpent-sse2-x86_64		  xor
  camellia-aesni-avx-x86_64	  dm-mod			  md4				  serpent_generic		  xts
  camellia-aesni-avx2		  drm_display_helper		  michael_mic			  sha1-ssse3			  xxhash_generic
  camellia-x86_64		  drm_exec			  mmc_block			  sha256-ssse3
  camellia_generic		  drm_gpuvm			  mmc_core			  sha512-ssse3
  cast5-avx-x86_64		  drm_ttm_helper		  mxm-wmi			  sm2_generic
  cast5_generic			  ecdh_generic			  n5pf				  sm3

==> Included binaries:
  blkid			  cryptsetup		  loginctl		  plymouthd		  systemd-tmpfiles
  btrfs			  dmsetup		  mount			  setfont		  udevadm
  busybox		  kmod			  plymouth		  switch_root

==> Early hook run order:
  udev

==> Hook run order:
  udev keymap plymouth encrypt

==> Late hook run order:
  plymouth grub-btrfs-overlayfs

==> Cleanup hook run order:
  udev

==> Emergency hook run order:
  plymouth

dmesg | grep overlay
[    5.316566] overlayfs: "xino" feature enabled using 2 upper inode bits.

findmnt

TARGET                        SOURCE                                                             FSTYPE     OPTIONS
/                             rootfs                                                             overlay    rw,relatime,lowerdir=/tmp.JoTBFm,upperdir=/tmp.vQzsno/upper,workdir=/tmp.vQzsno/work,uuid=on
├─/proc                       proc                                                               proc       rw,nosuid,nodev,noexec,relatime
│ └─/proc/sys/fs/binfmt_misc  systemd-1                                                          autofs     rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3856
├─/sys                        sys                                                                sysfs      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/firmware/efi/efivars efivarfs                                                           efivarfs   rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security      securityfs                                                         securityfs rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup            cgroup2                                                            cgroup2    rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
│ ├─/sys/fs/pstore            pstore                                                             pstore     rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf               bpf                                                                bpf        rw,nosuid,nodev,noexec,relatime,mode=700
│ ├─/sys/kernel/debug         debugfs                                                            debugfs    rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/tracing       tracefs                                                            tracefs    rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/config        configfs                                                           configfs   rw,nosuid,nodev,noexec,relatime
│ └─/sys/fs/fuse/connections  fusectl                                                            fusectl    rw,nosuid,nodev,noexec,relatime
├─/dev                        dev                                                                devtmpfs   rw,nosuid,relatime,size=8068464k,nr_inodes=2017116,mode=755,inode64
│ ├─/dev/pts                  devpts                                                             devpts     rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ ├─/dev/shm                  tmpfs                                                              tmpfs      rw,nosuid,nodev,inode64
│ ├─/dev/hugepages            hugetlbfs                                                          hugetlbfs  rw,nosuid,nodev,relatime,pagesize=2M
│ └─/dev/mqueue               mqueue                                                             mqueue     rw,nosuid,nodev,noexec,relatime
├─/run                        run                                                                tmpfs      rw,nosuid,nodev,relatime,mode=755,inode64
│ └─/run/user/1000            tmpfs                                                              tmpfs      rw,nosuid,nodev,relatime,size=1621584k,nr_inodes=405396,mode=700,uid=1000,gid=1000,inode64
├─/.snapshots                 /dev/mapper/luks-d916a464-4abf-46bd-b36a-e81a6381e0fc[/@snapshots] btrfs      ro,noatime,compress=zstd:1,ssd,space_cache=v2,subvolid=265,subvol=/@snapshots
├─/swap                       /dev/mapper/luks-d916a464-4abf-46bd-b36a-e81a6381e0fc[/@swap]      btrfs      ro,noatime,compress=zstd:1,ssd,space_cache=v2,subvolid=259,subvol=/@swap
├─/var/cache                  /dev/mapper/luks-d916a464-4abf-46bd-b36a-e81a6381e0fc[/@cache]     btrfs      ro,relatime,compress=zstd:1,ssd,space_cache=v2,subvolid=257,subvol=/@cache
├─/var/log                    /dev/mapper/luks-d916a464-4abf-46bd-b36a-e81a6381e0fc[/@log]       btrfs      ro,relatime,compress=zstd:1,ssd,space_cache=v2,subvolid=258,subvol=/@log
├─/tmp                        tmpfs                                                              tmpfs      rw,noatime,inode64
├─/boot/efi                   /dev/nvme1n1p1                                                     vfat       rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro
├─/home                       /dev/mapper/luks-c3e0e757-8f71-4642-bce0-fa52246af02a              ext4       rw,relatime
├─/mnt/storage                /dev/mapper/luks-2aafa376-c7c9-4928-b649-a853af9ea859              ext4       rw,relatime

Does anyone also have some issues with grub-btrfs-overlayfs and kernel 6.8 ?

Thanks

[Zesko]

I can confirm the issue with Dracut and kernel 6.8. Kernel 6.6 has no issue.
I tried to disable grub-btrfs-overlayfs for Dracut, kernel 6.8 can boot into a read-only snapshot when using SDDM.

[HanM23]

I can confirm the issue with Dracut and kernel 6.8. Kernel 6.6 has no issue. I tried to disable grub-btrfs-overlayfs for Dracut, kernel 6.8 can boot into a read-only snapshot when using SDDM.

Did you also have this issue in your VM with mkinitcpio eventually ?

[Zesko]

Did you also have this issue in your VM with mkinitcpio eventually ?

Yes

[kevinchevreuil]

Hello,

You should add the rw option in the snapshot CMDLINE to resolve this issue.

[Zesko]

This issue would be obsolete.

It was fixed for me using kernel 6.10 on Arch Linux.
That works fine with booting into a snapshot with overlayfs hook.

I already tested SDDM and GDM on overlayfs on VM, a result: no issue.

[kevinchevreuil]

Hello @Zesko,

I use this fix on a Kernel 6.9 and 6.10 with my own initramfs script to boot into an overlay fs. That works.

I was developed an initramfs script for Kaisen Linux foundable here: https://gitlab.com/kaisenlinux/packages/grub-btrfs/-/blob/main/debian/initramfs/local-bottom/kaisen-grub-btrfs?ref_type=heads
I hope that help you.