Fix mem usage reporting when using docker limits #5011
Conversation
✅ Deploy Preview for frigate-docs canceled.
|
frigate/util.py
Outdated
| # devcontainer seems to use this path | ||
| memlimit_command = ["cat", "/sys/fs/cgroup/memory/memory.max_usage_in_bytes"] | ||
|
|
||
| p = sp.run( | ||
| memlimit_command, | ||
| encoding="ascii", | ||
| capture_output=True, | ||
| ) | ||
|
|
||
| if p.returncode != 0: | ||
| logger.debug(f"Unable to get docker memlimit: {p.stderr}") | ||
| return -1 | ||
| else: | ||
| value: str = p.stdout | ||
|
|
||
| if value.strip().isnumeric(): | ||
| return int(value) | ||
| elif value.strip().lower() == "max": | ||
| return -1 |
There was a problem hiding this comment.
I don't think it makes sense to handle this in the dev container. I don't expect our config to be setting memory limits and in any case I don't really see the benefit
There was a problem hiding this comment.
Well, apart from anything else it's far easier to test there. I still haven't got a "proper" docker build working locally, so had to get the path by attaching to a container and looking up the path manually.
I also tend to avoid different code in dev/release environments as far as possible to catch bugs earlier.
There was a problem hiding this comment.
The way I see it having code to specifically handle the dev container is different code. There's also already a number of differences in the way the dev container runs vs the release container as the dev container doesn't use S6, frigate is started manually, etc. so to me that is moot.
Just my 2 cents happy to hear what others think
There was a problem hiding this comment.
Can you give any pointers to producing a release docker image locally? I tried docker build . -t mybuild but it just produces another dev container from what I can see
There was a problem hiding this comment.
Can you give any pointers to producing a release docker image locally? I tried docker build . -t mybuild but it just produces another dev container from what I can see
make build
There was a problem hiding this comment.
Can you give any pointers to producing a release docker image locally? I tried docker build . -t mybuild but it just produces another dev container from what I can see
make build
Thanks! I'm used to Java tooling in Eclipse/maven so VS Code, make, docker and devcontainers have been a bit of a learning curve
|
It turns out this isn't so much a case of devcontainer vs release build, but dependent on the version of Docker engine in use. The This should just be a case of handling both paths, but I'll rework this to do it in a more deliberate manner with an explicit check on which cgroups version we expect. The cgroups v1 path also needs correcting to use a different metric. |
|
So, it seems the official release builds of frigate are using a sufficiently new distro/docker version to default to cgroups v2. However my local devcontainer builds (build on Windows using WSL2) only show cgroups v1 enabled, which apparently may change in future but is not there yet (see microsoft/WSL#6662). I don't have other dev envs to test on so not clear what they do. In theory both cgroups1 and 2 support checking memory limits - but on cgroups1 it appears to be problematic in some containers (just reporting 9223372036854771712 for example). On that basis, proceeding with a solution for cgroups v2 only. Docker does support updates to memory limits on running containers, and the command to check memory is lightweight so running it each time should not impact stats load times. |
Fixes #5004
This identifies if we are running in a container with a memory limit set. If so, rather than taking the %MEM from
topdirectly, it calculates it as (resident memory / docker limit).It seems the base container is different to the production builds - the path changes slightly. I've handled both cases for now.
[edit]
After some local testing, my expectation is that this will work with Docker engine 20.10 (released December 2020) on suitably modern hosts, but not WSL2 on Windows without extra configuraiton.