Filesystem doesn't resize on RPI4 and alpine flavor #2863

Ludea · 2024-09-17T07:50:21Z

Kairos version:

tested version : 3.1.2 and 3.1.3
flavor: alpine

CPU architecture, OS, and Version:

Rp4 aarch64

Describe the bug

Filesystem is not resizing
Partition resizing works as expected

To Reproduce

Do a fresh install install of Kairos on sd card
Boot Rpi
run df -h to check the COS_PERSISTENT disk

Expected behavior

Filesystem expand all available free space

Logs

df -h:

Filesystem                         Size  Used Avail Use% Mounted on
 /dev/disk/by-label/COS_ACTIVE      2.0G  697M  1.2G  38% /
 /dev/disk/by-label/COS_OEM          55M   32K   52M   1% /oem
 /dev/disk/by-label/COS_PERSISTENT   55M  1.9M   50M   4% /usr/local
 overlay                            2.0G  180K  2.0G   1% /var
 overlay                            2.0G  180K  2.0G   1% /etc
 overlay                            2.0G  180K  2.0G   1% /srv
 devtmpfs                            10M     0   10M   0% /dev
 tmpfs                              3.9G  544K  3.9G   1% /run
 shm                                3.9G     0  3.9G   0% /dev/shm
 tmpfs                              3.9G     0  3.9G   0% /tmp
 /dev/mmcblk0p2                     5.9G  1.4G  4.3G  25% /run/initramfs/cos-state
 tmpfs                              2.0G     0  2.0G   0% /run/overlay

dmesg:

[    4.651890] udevd[995]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:1': No such file or directory
 [    4.652275] udevd[997]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 179:0': No such file or directory
 [    4.652795] udevd[996]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:10': No such file or directory
 [    4.653498] udevd[998]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:0': No such file or directory
 [    4.653902] udevd[999]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:11': No such file or directory
 [    4.654311] udevd[1000]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:12': No such file or directory
 [    4.654535] udevd[1001]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:13': No such file or directory
 [    4.677703] NET: Registered PF_PACKET protocol family
 [    4.683017] udevd[1007]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:7': No such file or directory
 [    4.686944] udevd[1009]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 7:0': No such file or directory

immucore.log:

1970-01-01T00:00:11Z INF Immucore commit=none compiled with=go1.22.5 version=v0.4.2
 1970-01-01T00:00:11Z INF creating a runtime
 1970-01-01T00:00:11Z INF detecting boot state
 1970-01-01T00:00:11Z INF Booting on active/passive/recovery.
 1970-01-01T00:00:11Z INF 1.
  <init> (background: false) (weak: false) (run: false)
 2.
  <lvm-activation> (background: false) (weak: false) (run: false)
  <mount-state> (background: false) (weak: false) (run: false)
  <mount-tmpfs> (background: false) (weak: false) (run: false)
  <create-sentinel> (background: false) (weak: false) (run: false)
 3.
  <discover-state> (background: false) (weak: false) (run: false)
  <upgrade-kcrypt> (background: false) (weak: false) (run: false)
 4.
  <mount-root> (background: false) (weak: false) (run: false)
 5.
  <mount-oem> (background: false) (weak: false) (run: false)
 6.
  <rootfs-hook> (background: false) (weak: false) (run: false)
  <unlock-all> (background: false) (weak: false) (run: false)
 7.
  <load-config> (background: false) (weak: false) (run: false)
 8.
  <mount-base-overlay> (background: false) (weak: false) (run: false)
  <custom-mount> (background: false) (weak: false) (run: false)
 9.
  <overlay-mount> (background: false) (weak: false) (run: false)
 10.
  <mount-bind> (background: false) (weak: false) (run: false)
 11.
  <write-fstab> (background: false) (weak: false) (run: false)
 12.
  <initramfs-hook> (background: false) (weak: false) (run: false)
 
 1970-01-01T00:00:11Z INF creating a runtime
 1970-01-01T00:00:11Z INF detecting boot state
 1970-01-01T00:00:11Z INF creating a runtime
 1970-01-01T00:00:11Z INF detecting boot state
 1970-01-01T00:00:11Z INF creating a runtime
 1970-01-01T00:00:11Z INF Setting sentinel file to=active_mode
 1970-01-01T00:00:11Z INF detecting boot state
 1970-01-01T00:00:12Z INF mount done options=["rw"] type=tmpfs what=tmpfs where=/tmp
 1970-01-01T00:00:12Z INF mount done options=["ro"] type=ext3 what=/dev/disk/by-label/COS_STATE where=/sysroot/run/initramfs/cos-state
 1970-01-01T00:00:13Z INF mount done options=["ro","suid","dev","exec","async"] type=ext4 what=/dev/disk/by-label/COS_ACTIVE where=/sysroot
 1970-01-01T00:00:13Z INF creating a runtime
 1970-01-01T00:00:13Z INF detecting boot state
 1970-01-01T00:00:14Z INF mount done options=["rw","suid","dev","exec","async"] type=ext3 what=/dev/disk/by-label/COS_OEM where=/sysroot/oem
 1970-01-01T00:00:14Z INF Running rootfs stage
 1970-01-01T00:00:14Z INF triggering udev to populate disk info
 1970-01-01T00:00:21Z INF mount done options=["rw"] type=ext4 what=/dev/disk/by-label/COS_PERSISTENT where=/sysroot/usr/local
 1970-01-01T00:00:21Z WRN
 1970-01-01T00:00:21Z WRN executing mount callback error="mkdir /sysroot/snap: read-only file system" options=["bind"] type=overlay what=/sysroot/usr/local/.state/snap.bind where=/sysroot/snap
 1970-01-01T00:00:21Z ERR error="mkdir /sysroot/snap: read-only file system"
 1970-01-01T00:00:24Z WRN executing mount callback error="mkdir /sysroot/usr/share/pki: read-only file system" options=["bind"] type=overlay what=/sysroot/usr/local/.state/usr-share-pki-trust.bind where=/sysroot/usr/share/pki/trust
 1970-01-01T00:00:24Z ERR error="mkdir /sysroot/usr/share/pki: read-only file system"
 1970-01-01T00:00:24Z WRN executing mount callback error="mkdir /sysroot/usr/share/pki: read-only file system" options=["bind"] type=overlay what=/sysroot/usr/local/.state/usr-share-pki-trust-anchors.bind where=/sysroot/usr/share/pki/trust/anchors
 1970-01-01T00:00:24Z ERR error="mkdir /sysroot/usr/share/pki: read-only file system"
 1970-01-01T00:00:24Z WRN error="3 errors occurred:\n\t* mkdir /sysroot/snap: read-only file system\n\t* mkdir /sysroot/usr/share/pki: read-only file system\n\t* mkdir /sysroot/usr/share/pki: read-only file system\n\n"
 1970-01-01T00:00:24Z INF Running initramfs stage
 2006-08-14T08:35:02Z INF 1.
  <init> (background: false) (weak: false) (run: false)
 2.
  <create-sentinel> (background: false) (weak: false) (run: true)
  <mount-tmpfs> (background: false) (weak: false) (run: true)
  <lvm-activation> (background: false) (weak: false) (run: true)
  <mount-state> (background: false) (weak: false) (run: true)
 3.
  <upgrade-kcrypt> (background: false) (weak: false) (run: true)
  <discover-state> (background: false) (weak: false) (run: true)
 4.
  <mount-root> (background: false) (weak: false) (run: true)
 5.
  <mount-oem> (background: false) (weak: false) (run: true)
 6.
  <rootfs-hook> (background: false) (weak: false) (run: true)
  <unlock-all> (background: false) (weak: false) (run: true)
 7.
  <load-config> (background: false) (weak: false) (run: true)
 8.
  <custom-mount> (background: false) (weak: false) (run: true)
  <mount-base-overlay> (background: false) (weak: false) (run: true)
 9.
  <overlay-mount> (background: false) (weak: false) (run: true)
 10.
  <mount-bind> (error: 3 errors occurred:
         * mkdir /sysroot/snap: read-only file system
         * mkdir /sysroot/usr/share/pki: read-only file system
         * mkdir /sysroot/usr/share/pki: read-only file system
 
 ) (background: false) (weak: false) (run: true)
 11.
  <write-fstab> (background: false) (weak: false) (run: true)
 12.
  <initramfs-hook> (background: false) (weak: false) (run: true)

Additional context

Itxaka try to reproduce issue on his Rpi, but FS resize as expected.
With Tumbleweed flavor, no issue, partition and fs resizing work as expected

The text was updated successfully, but these errors were encountered:

Itxaka · 2024-10-14T12:09:03Z

going to retry and put the logs here to see.

Itxaka · 2024-10-14T13:26:20Z

just intalled from scratch and cant reproduce:

Disk /dev/sda: 115.1 GiB, 123589361664 bytes, 241385472 sectors
Disk model: 3.2 Gen 1       
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: 85800D3E-DE8D-4BE1-8D6A-FDE3987F8CD8

Device        Start       End   Sectors   Size Type
/dev/sda1      2048    198655    196608    96M EFI System
/dev/sda2    198656  12896255  12697600   6.1G Linux filesystem
/dev/sda3  12896256  21497855   8601600   4.1G Linux filesystem
/dev/sda4  21497856  21628927    131072    64M Linux filesystem
/dev/sda5  21628928 241385436 219756509 104.8G Linux filesystem
localhost:~$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/disk/by-label/COS_ACTIVE      2.0G  697M  1.2G  38% /
/dev/disk/by-label/COS_OEM          55M   32K   52M   1% /oem
/dev/disk/by-label/COS_PERSISTENT   99G  1.8M   93G   1% /usr/local
overlay                            2.0G  176K  2.0G   1% /var
overlay                            2.0G  176K  2.0G   1% /etc
overlay                            2.0G  176K  2.0G   1% /srv
devtmpfs                            10M     0   10M   0% /dev
tmpfs                              3.9G  608K  3.9G   1% /run
efivarfs                           128K   920  128K   1% /sys/firmware/efi/efivars
shm                                3.9G     0  3.9G   0% /dev/shm
tmpfs                              3.9G     0  3.9G   0% /tmp
/dev/sda2                          5.9G  1.4G  4.3G  25% /run/initramfs/cos-state
tmpfs                              2.0G     0  2.0G   0% /run/overlay

localhost:~$ cat /etc/kairos-release 
KAIROS_PRETTY_NAME="kairos-standard-alpine-3.19 v3.2.1-7-ga068314-v1.31.1-k3s1"
KAIROS_IMAGE_REPO="quay.io/kairos/alpine:3.19-standard-arm64-rpi4-v3.2.1-7-ga068314-k3sv1.31.1-k3s1"
KAIROS_IMAGE_LABEL="3.19-standard-arm64-rpi4-v3.2.1-7-ga068314-k3sv1.31.1-k3s1"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_SOFTWARE_VERSION="v1.31.1+k3s1"
KAIROS_ID="kairos"
KAIROS_VERSION="v3.2.1-7-ga068314-v1.31.1-k3s1"
KAIROS_FLAVOR="alpine"
KAIROS_FAMILY="alpine"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_NAME="kairos-standard-alpine-3.19"
KAIROS_VERSION_ID="v3.2.1-7-ga068314-v1.31.1-k3s1"
KAIROS_MODEL="rpi4"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_ID_LIKE="kairos-standard-alpine-3.19"
KAIROS_FLAVOR_RELEASE="3.19"
KAIROS_VARIANT="standard"
KAIROS_TARGETARCH="arm64"
KAIROS_RELEASE="v3.2.1-7-ga068314"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_ARTIFACT="kairos-alpine-3.19-standard-arm64-rpi4-v3.2.1-7-ga068314-k3sv1.31.1+k3s1"

Stage is correctly executed

1970-01-01T00:00:20Z INF Processing stage step 'RPI configs.Grow persistent'. ( commands: 0, files: 0, ... )
1970-01-01T00:00:20Z DBG Stage: schema.Stage{
  If: "[ ! -f /run/cos/recovery_mode ] && [ ! -f /run/cos/live_mode ] && [ -f \"/sys/firmware/devicetree/base/model\" ] && grep -i \"Raspberry Pi 4\" \"/sys/firmware/devicetree/base/model\"",
  Name: "Grow persistent",
  Layout: schema.Layout{
    Device: &schema.Device{
      Label: "COS_PERSISTENT",
    },
    Expand: &schema.Expand{},
  },
}
1970-01-01T00:00:20Z DBG Using label COS_PERSISTENT for layout expansion
1970-01-01T00:00:20Z DBG Output of udevadm settle: 
1970-01-01T00:00:20Z DBG Got device /dev/sda for label COS_PERSISTENT
1970-01-01T00:00:21Z INF Extending last partition to max space
1970-01-01T00:00:21Z DBG Expanding partition 5 up to 0 sectors
1970-01-01T00:00:22Z DBG Output of growpart: CHANGED: partition=5 start=21628928 old: size=131072 end=21759999 new: size=219756510 end=241385437

1970-01-01T00:00:22Z DBG Trying to reread the partition table of /dev/sda (try number 1)
1970-01-01T00:00:22Z DBG output of partprobe: 
1970-01-01T00:00:22Z DBG Output of sync: 
1970-01-01T00:00:22Z DBG Trying to resize filesystem for device /dev/sda5
1970-01-01T00:00:22Z DBG Found filesystem ext3
 for device /dev/sda5
1970-01-01T00:00:22Z DBG Output from running e2fsck e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
COS_PERSISTENT: 11/16384 files (0.0% non-contiguous), 9525/65536 blocks

1970-01-01T00:01:57Z DBG Output from running resize2fs resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/sda5 to 109878252 (1k) blocks.
The filesystem on /dev/sda5 is now 109878252 (1k) blocks long.


1970-01-01T00:01:57Z DBG Trying to reread the partition table of /dev/sda (try number 1)
1970-01-01T00:01:57Z DBG output of partprobe: 
1970-01-01T00:01:57Z DBG Output of sync:

I have no idea whats going on @Ludea
Please retry from scratch and run the Kairos entry adding rd.immucore.debug to the cmdline and then paste the /run/immucore/immucore.log and /run/immucore/rootfs_stage.log full logs to see whats going on.

Ludea · 2024-10-22T08:53:32Z

I attach some logs from immucore.log and rootfs_stage.log

Ludea · 2024-10-22T09:02:52Z

Partition is resizing (check with lsblk)
Fs is not resizing (check with df - h)
I see #2128
May I wait this release?

jimmykarily · 2024-11-06T06:54:40Z

@Ludea 's output:

Got device /dev/mmcblk0 for label COS_PERSISTENT

@Itxaka 's :

Got device /dev/sda for label COS_PERSISTENT

Maybe relevant to this? kairos-io/kairos-agent@84d87b3

Itxaka · 2024-11-06T08:34:08Z

Difference between SD and USB boot device I guess. I also tried with SD card

jimmykarily · 2024-11-11T09:12:21Z

Let's wait for the upcoming release 3.2.2 (which includes the linked fix) so @Ludea can give it another try.

Ludea · 2024-11-12T07:18:02Z

I cannot test the fix :s
I resize myself the fs and deploy all my infra.
But thanks for fixing it ! In case I re flash kairos, will happy to try.

jimmykarily · 2024-11-15T13:37:34Z

All good then. I'll close this but let us know if you face any issues in the future.

Ludea added bug Something isn't working triage Add this label to issues that should be triaged and prioretized in the next planning call unconfirmed labels Sep 17, 2024

github-project-automation bot added this to 🧙Issue tracking board Sep 17, 2024

jimmykarily moved this to In Progress 🏃 in 🧙Issue tracking board Oct 14, 2024

Itxaka self-assigned this Oct 14, 2024

Itxaka added the waiting for reporter label Oct 14, 2024

Itxaka moved this from In Progress 🏃 to Todo 🖊 in 🧙Issue tracking board Oct 14, 2024

Itxaka removed their assignment Oct 14, 2024

Itxaka removed the triage Add this label to issues that should be triaged and prioretized in the next planning call label Oct 14, 2024

jimmykarily closed this as completed Nov 15, 2024

github-project-automation bot moved this from Under review 🔍 to Done ✅ in 🧙Issue tracking board Nov 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filesystem doesn't resize on RPI4 and alpine flavor #2863

Filesystem doesn't resize on RPI4 and alpine flavor #2863

Ludea commented Sep 17, 2024

Itxaka commented Oct 14, 2024

Itxaka commented Oct 14, 2024

Ludea commented Oct 22, 2024

Ludea commented Oct 22, 2024 •

edited

Loading

jimmykarily commented Nov 6, 2024 •

edited

Loading

Itxaka commented Nov 6, 2024

jimmykarily commented Nov 11, 2024

Ludea commented Nov 12, 2024

jimmykarily commented Nov 15, 2024

Filesystem doesn't resize on RPI4 and alpine flavor #2863

Filesystem doesn't resize on RPI4 and alpine flavor #2863

Comments

Ludea commented Sep 17, 2024

Itxaka commented Oct 14, 2024

Itxaka commented Oct 14, 2024

Ludea commented Oct 22, 2024

Ludea commented Oct 22, 2024 • edited Loading

jimmykarily commented Nov 6, 2024 • edited Loading

Itxaka commented Nov 6, 2024

jimmykarily commented Nov 11, 2024

Ludea commented Nov 12, 2024

jimmykarily commented Nov 15, 2024

Ludea commented Oct 22, 2024 •

edited

Loading

jimmykarily commented Nov 6, 2024 •

edited

Loading