Catastrophic failure error after adding 'memory' to cgroup_no_v1 kernel command line with systemd enabled

[mbuella]

Version

Microsoft Windows [Version 10.0.22621.755]

WSL Version

• WSL 2
• WSL 1

Kernel Version

5.15.68

Distro Version

Ubuntu 22.04.1

Other Software

• Podman

  podman version 3.4.4
  

Repro Steps

1. Enabling systemd and Cgroups v2
   1. Enable systemd in WSL:

   $ cat /etc/wsl.conf
   [boot]
   systemd=true
   

   2. Enable Cgroup v2 via fstab:

   $ cat /etc/fstab
   cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
   

2. Enabling cgroups V2 without memory added to cgroup_no_v1 kernel commandline
   1. Add cgroup_no_v1 cmdline to kernel without memory:

   # .wslconfig
   [wsl2]
   kernelCommandLine = systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=cpuset,cpu,cpuacct,io,devices,freezer,net_cls,perf_event,net_prio,hugetlb,pids,rdma
   

   2. Restart distro:

   wsl -t Ubuntu
   wsl --shutdown
   wsl --distribution Ubuntu
   

   3. Distro starts successfully
   4. Podman displays cgroup as V2 but cgroupControllers lists only cpuset, cpu, io and pids. Same with cgroup.controllers:

   $ cat /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers
   cpuset cpu io pids
   

3. Enabling cgroups V2 with memory added to cgroup_no_v1 kernel commandline
   1. Add cgroup_no_v1 cmdline to kernel without memory (or setting all):

   # .wslconfig
   [wsl2]
   kernelCommandLine = systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=memory,cpuset,cpu,cpuacct,io,devices,freezer,net_cls,perf_event,net_prio,hugetlb,pids,rdma
   # or
   # kernelCommandLine = systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all
   

   2. Restart distro:

   wsl -t Ubuntu
   wsl --shutdown
   wsl --distribution Ubuntu
   

   3. Distro start fails, with the following message:

    Catastrophic failure
    Error code: Wsl/Service/CreateInstance/CreateVm/ConfigureNetworking/0x8000ffff 

Expected Behavior

Cgroup v2 is enabled with memory included in controllers. This will be useful for setting up rootless podman and even Kubernetes.

Actual Behavior

If memory is included, distro start will fail with the following error message:

Catastrophic failure
Error code: Wsl/Service/CreateInstance/CreateVm/ConfigureNetworking/0x8000ffff 

Diagnostic Logs

WSL Logs:
WslLogs-2022-11-05_12-39-48.zip

Podman info (rootful):

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 12
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: file
  hostname: alonpc
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.68.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 7195217920
  memTotal: 7987146752
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 2147483648
  swapTotal: 2147483648
  uptime: 29.83s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 0
  BuiltTime: Thu Jan  1 08:00:00 1970
  GitCommit: ""
  GoVersion: go1.17.3
  OsArch: linux/amd64
  Version: 3.4.4

podman info (rootless):

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 12
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: file
  hostname: alonpc
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.68.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 7192166400
  memTotal: 7987146752
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 2147483648
  swapTotal: 2147483648
  uptime: 1m 21.06s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/alonbuella/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
  graphRoot: /home/alonbuella/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /run/user/1000/containers
  volumePath: /home/alonbuella/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 0
  BuiltTime: Thu Jan  1 08:00:00 1970
  GitCommit: ""
  GoVersion: go1.17.3
  OsArch: linux/amd64
  Version: 3.4.4

[cerebrate]

At the moment, unfortunately, it appears that the Microsoft init requires v1 memory cgroups to work. Sadly, no workaround at present.

(There's an extended discussion of this in #6662, in particular here: #6662 (comment) and onward.)

[cerebrate]

Seems to work for me (or rather, cgroup_no_v1=all does) in WSL 1.0.1. Give that one a try, if you can.

[microsoft-github-policy-service]

This issue has been automatically closed since it has not had any activity for the past year. If you're still experiencing this issue please re-file this as a new issue or feature request.

Thank you!