Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

blkio stats empty for zfs on kernel 5.x #78

Open
Rid opened this issue Mar 21, 2019 · 0 comments
Open

blkio stats empty for zfs on kernel 5.x #78

Rid opened this issue Mar 21, 2019 · 0 comments

Comments

@Rid
Copy link

Rid commented Mar 21, 2019

I have two docker instances on the same server, both using:
Docker version 18.09.3, build 774a1f4

and

containerd github.com/containerd/containerd 1.2.4 e6b3f5632f50dbc4e9cb6288d911bf4f5e95b18e

Kernel:
5.0.1-050001-generic

One is using overlay2, the other is using zfs as the graph driver. The instance on overlay2 is running fine, however the zfs instance is not reporting blkio stats:

  "blkio_stats": {
    "io_service_bytes_recursive": [],
    "io_serviced_recursive": [],
    "io_queue_recursive": [],
    "io_service_time_recursive": [],
    "io_wait_time_recursive": [],
    "io_merged_recursive": [],
    "io_time_recursive": [],
    "sectors_recursive": []
  },

I am trying to debug this further, the blkio controller for cgroups seems to have changed, I am only seeing the following stats inside container cgroups:

# ls -la /sys/fs/cgroup/blkio/
total 0
drwxr-xr-x    2 root     root             0 Mar 21 07:48 .
dr-xr-xr-x   14 root     root           360 Mar 21 07:48 ..
-r--r--r--    1 root     root             0 Mar 21 12:29 blkio.bfq.io_service_bytes
-r--r--r--    1 root     root             0 Mar 21 12:29 blkio.bfq.io_service_bytes_recursive
-r--r--r--    1 root     root             0 Mar 21 12:29 blkio.bfq.io_serviced
-r--r--r--    1 root     root             0 Mar 21 12:29 blkio.bfq.io_serviced_recursive
-rw-r--r--    1 root     root             0 Mar 21 12:29 blkio.bfq.weight
--w-------    1 root     root             0 Mar 21 12:29 blkio.reset_stats
-r--r--r--    1 root     root             0 Mar 21 07:48 blkio.throttle.io_service_bytes
-r--r--r--    1 root     root             0 Mar 21 12:29 blkio.throttle.io_service_bytes_recursive
-r--r--r--    1 root     root             0 Mar 21 07:48 blkio.throttle.io_serviced
-r--r--r--    1 root     root             0 Mar 21 12:29 blkio.throttle.io_serviced_recursive
-rw-r--r--    1 root     root             0 Mar 21 12:29 blkio.throttle.read_bps_device
-rw-r--r--    1 root     root             0 Mar 21 12:29 blkio.throttle.read_iops_device
-rw-r--r--    1 root     root             0 Mar 21 12:29 blkio.throttle.write_bps_device
-rw-r--r--    1 root     root             0 Mar 21 12:29 blkio.throttle.write_iops_device
-rw-r--r--    1 root     root             0 Mar 21 12:29 cgroup.clone_children
-rw-r--r--    1 root     root             0 Mar 21 12:29 cgroup.procs
-rw-r--r--    1 root     root             0 Mar 21 12:29 notify_on_release
-rw-r--r--    1 root     root             0 Mar 21 12:29 tasks

So none of the blkio.io* files mentioned here exist:
https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-controller.txt

These are the default kernel config options in 5.0.1 (we're using a default mainline kernel):

CONFIG_BLK_CGROUP=y
CONFIG_CGROUP_SCHED=y
# CONFIG_DEBUG_BLK_CGROUP is not set
# CONFIG_BLK_CGROUP_IOLATENCY is not set

I have tried with both BFQ enabled and disabled (mq-deadline), also tried zfs with and without the zfs_vdev_scheduler=bfq option, but both made no difference.

I believe the relevant code is:
https://github.com/containerd/cgroups/blob/master/blkio.go

However I haven't had a chance to properly dive into the code yet, perhaps I'm missing something obvious, can anyone point me in the right direction? I've also posted in the relevant docker issue for checking kernel 5.x compatibility here moby/moby#38887, however this seems to be a containerd issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant