-
Notifications
You must be signed in to change notification settings - Fork 156
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
reload hitch cause memory big increase: read all certificates into memory and then never released it? #374
Comments
It seems reload hitch will somehow read all certificates into memory and then never released it. ~ 6k bytes per certificate * 6400 (certificate) = 380M So every reload will add 380M to memory. Will you check it? |
Hi. Do you have plots of memory consumption over time? On a reload, Hitch will launch a new set of worker processes while draining the old generation. So it is expected that there will be a period with significantly increased memory consumption - this should however go back down after the previous gen worker processes are drained of traffic and retire. Could I ask you to monitor the number of total hitch processes running when you see this, and also see if the usage drops after they are cleaned up? |
With htop, I can only see only one hitch process, nearly 99.9% during first few seconds, then it drops. The memory also increase a lot in first a few seconds, then stable at 0.3G increase. And first reload will always doubles the memory, then next reload will increase 0.3-0.4G increase. I have seen it in many ubuntu. |
Perhaps it always exists, you can see it when you have thousands of certs. I bear it for years. :) |
Same machine, with wildcard certs (acme.sh), the first reload will double memory. Interesting ... |
And certbot ssl certs are bigger than wildcard ssl in file size. |
It is caused by Automatic OCSP staple retrieval. I have to switch off Automatic OCSP staple retrieval by:
|
Will you update hitch version to fix this bug? |
Hello,
Ubuntu 22.04/20.04
hitch 1.73/1.71
12 workers
6500+ LETS ssl certs
service hitch reload:
First reload will always double hitch memory used: from 3.1 to 6.5G
next reload will increase ~ 400M each
I have tried several adjustments in hitch: with or without session, with or without ocsp.
The text was updated successfully, but these errors were encountered: