Enforcer Memory Stats File Not Found After Upgrade to 2.6.1

[another-novelty]

After upgrading from version 2.4.5 to version 2.6.1, our enforcers are constantly generating the following error message:

2023-08-03T10:04:47.145|ERRO|AGT|system.(*SystemTools).getMemoryStats: Could not find memory stats file - error=open /host/proc/38729/root/sys/fs/cgroup/memory/memory.stat: no such file or directory filePath=/host/proc/38729/root/sys/fs/cgroup/memory/memory.stat systemtools={bEnable:true info:{Meta:{Version:0.9.2 Timestamp:{wall:13919140360197750914 ext:10046709 loc:0x2fc2400}} Node:{Hostname:neuvector-enforcer-pod-2g2pw MachineID: Hypervisor:vmware Timezone:Europe/Berlin} OS:{Name:SUSE Linux Enterprise Server 15 SP4 Vendor:sles Version:15.4 Release: Architecture:amd64} Kernel:{Release:5.14.21-150400.24.69-default Version:#1 SMP PREEMPT_DYNAMIC Tue Jul 4 13:36:11 UTC 2023 (28b65ec) Architecture:x86_64} Product:{Name:VMware Virtual Platform Vendor:VMware, Inc. Version:None Serial:VMware-### UUID:###} Board:{Name:440BX Desktop Reference Platform Vendor:Intel Corporation Version:None Serial:None AssetTag:} Chassis:{Type:1 Vendor:No Enclosure Version:N/A Serial:None AssetTag:No Asset Tag} BIOS:{Vendor:Phoenix Technologies LTD Version:6.00 Date:11/12/2020} CPU:{Vendor:GenuineIntel Model:Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz Speed:3000 Cache:25344 Cpus:0 Cores:0 Threads:16} Memory:{Type:DRAM Speed:0 Size:65536} Storage:[{Name:sda Driver:sd Vendor:VMware Model:Virtual disk Serial: Size:107}] Network:[{Name:eth0 Driver:vmxnet3 MACAddress:### Port: Speed:0}]} procDir:/host/proc/ cgroupDir:/host/cgroup/ clockTicksPerSecond:100 cgroupVersion:1 cgroupMemoryDir:/sys/fs/cgroup/memory}

Environment:

• Kubernetes version: 1.25.11+rke2r1 (on RKE2)
• OS of VMs: SUSE Linux Enterprise Server 15 SP4
• The federation master is still on the older version (2.4.5).

The error message indicates that the enforcers are looking for the memory stats file in the path /host/proc/38729/root/sys/fs/cgroup/memory/memory.stat, which results in a "no such file or directory" error.

[0Styless]

These messages constantly spam our logging stack with various procs. I already checked the volume mount inside the enforcer pods and in my opinion this is mounted as expected. These issues didn't appear with NeuVector running on version 2.4.5.

For example:

/ # hostname
neuvector-enforcer-pod-2g2pw
/ # ls -l /host/proc/
total 0
dr-xr-xr-x    9 root     root             0 Aug  1 01:50 1
dr-xr-xr-x    9 root     root             0 Aug  1 01:50 10
dr-xr-xr-x    9 80       www-data         0 Aug  4 06:42 100034
dr-xr-xr-x    9 root     root             0 Aug  1 01:50 1006
dr-xr-xr-x    9 root     root             0 Aug  1 01:50 1007
dr-xr-xr-x    9 root     root             0 Aug  4 06:55 10086
dr-xr-xr-x    9 root     root             0 Aug  1 01:50 101
dr-xr-xr-x    9 root     root             0 Aug  1 01:50 1012
dr-xr-xr-x    9 root     root             0 Aug  4 06:42 101279
dr-xr-xr-x    9 root     root             0 Aug  1 01:54 10179
dr-xr-xr-x    9 65535    65535            0 Aug  1 01:54 10198
dr-xr-xr-x    9 root     root             0 Aug  1 01:50 102
dr-xr-xr-x    9 root     root             0 Aug  1 01:54 10217
dr-xr-xr-x    9 root     root             0 Aug  1 01:50 1024
dr-xr-xr-x    9 65535    65535            0 Aug  1 01:54 10250
dr-xr-xr-x    9 root     root             0 Aug  1 01:50 103
dr-xr-xr-x    9 root     root             0 Aug  1 01:50 1036
dr-xr-xr-x    9 root     root             0 Aug  1 01:50 104
dr-xr-xr-x    9 root     root             0 Aug  1 01:54 10421
dr-xr-xr-x    9 65535    65535            0 Aug  1 01:54 10441
dr-xr-xr-x    9 root     root             0 Aug  1 01:50 105
dr-xr-xr-x    9 root     root             0 Aug  1 01:54 10513
dr-xr-xr-x    9 65535    65535            0 Aug  1 01:54 10533
dr-xr-xr-x    9 185      root             0 Aug  1 01:54 10615
dr-xr-xr-x    9 root     root             0 Aug  1 01:50 107
dr-xr-xr-x    9 root     root             0 Aug  4 06:44 108409
dr-xr-xr-x    9 root     root             0 Aug  1 01:50 11
dr-xr-xr-x    9 root     root             0 Aug  1 01:54 11107
dr-xr-xr-x    9 65535    65535            0 Aug  1 01:54 11131

[0Styless]

Detailed example:

/ # ls -l /host/proc/2661/root/sys/fs/
total 0
dr-xr-xr-x    2 root     root             0 Aug  1 02:00 bpf
dr-xr-xr-x    2 root     root             0 Aug  1 02:00 cgroup
drwxr-xr-x    4 root     root             0 Aug  1 02:00 ext4
drwxr-xr-x    3 root     root             0 Aug  1 02:00 fuse
drwxr-xr-x    3 root     root             0 Aug  1 03:00 nfs
dr-xr-xr-x    2 root     root             0 Aug  1 02:00 pstore
/ # ls -l /host/proc/2661/root/sys/fs/cgroup/
total 0
/ #

But its also empty on the node itself:

worker-03:~ # ll /proc/2661/root/sys/fs/
total 0
dr-xr-xr-x 2 root root 0 Aug  1 04:00 bpf
dr-xr-xr-x 2 root root 0 Aug  1 04:00 cgroup
drwxr-xr-x 4 root root 0 Aug  1 04:00 ext4
drwxr-xr-x 3 root root 0 Aug  1 04:00 fuse
drwxr-xr-x 3 root root 0 Aug  1 05:00 nfs
dr-xr-xr-x 2 root root 0 Aug  1 04:00 pstore
worker-03:~ # ll /proc/2661/root/sys/fs/cgroup/
total 0

[jaimeyu-suse]

Hi, did you make an issue in the main neuvector issues (https://github.com/neuvector/neuvector/issues)? I often get this type of issues sent to me. There is a PR that will potentially fix this issue that is in review. I only found this ticket because by chance as this repo is specifically for the helm charts.

[0Styless]

@jaimeyu-suse no we didn't - if there is already a PR we still should open this issue in neuvector/neuvector?

[jaimeyu-suse]

Please do and close this ticket.

You could just watch the PR and follow it but we do check the issues list in the main repo for future reference if you want quicker responses/tracking.

Thanks

[another-novelty]

Do you mean this PR?

[another-novelty]

I opened the issue there. I'd like to leave this issue here open until the fixed version is used in the helm chart.