Spring Boot app failed to start because of OOM (Metaspace OOM) #208
Comments
|
Although I produced the same Memory Calculator values on startup using your sample app, I did not hit the OOM error: The Metaspace size is automatically calculated using a formula, by counting the number of classes loaded, i.e. the number of class files in the jar, multiplied by a fixed class size value + a fixed class overhead value. This calculation usually works well for Spring Boot apps. We recently added support for counting classes specified within Unless there is another difference here, it is possible that the calculated value for Metaspace is just below the amount required. Adding the Eureka jar will increase the class count and therefore increase the calculated Metaspace size (by around 30M here) which seems to be just enough to start successfully. One thing you can try is adjusting the class count - a runtime variable exists to set a class count adjustment factor. |
|
@pivotal-david-osullivan I keep my JAVA_OPTS as same as you, but it still OOM, the only difference is Double confirm with you: Did the app keep running successfully after starting for several minutes? Because my app will start to run first then a few seconds later, the OOM occur. The error logs: |
|
The Following your comment I left the app running for several hours but it never hits an OOM. The application seems to start much more quickly for me: Have you tried setting the variable to allow for an increase in Metaspace? |
//===================================================== Hi @pivotal-david-osullivan Sorry for not following this question for a long time, Now we have upgraded the java-azure buildpack version to 8.8.0 and the kpack version to 0.7.2. the issue still exists I'm not sure how you deploy an app with dynamic, from the previous logs, it seems that the app doesn't run with AppDynamics APM(I didn't see the logs about AppDynamics) , I use pack CLI to build the app with AppDynamic and then run it locally, the metaspace OOM happened. see |
|
Hi @zhoufenqin I did build & run with AppDynamics but I cut out the logs for readability, you can see the helper log in my output above Here is the full output from a new build, also left running for some time: Have you tested setting the class count adjustment parameter I mentioned above to see if this makes a difference? |
|
Hi @pivotal-david-osullivan thank you, I tried your above suggestion to increase metadata space(--env JAVA_OPTS=-XX:MaxMetaspaceSize=159425K), and it can solve the But I want to know why the metadata space value calculated by buildpack will lead to OOM because if the buildpack can calculate it correctly, we don't want to let customers config the java_opts manually. I notice that only a few differences from your logs and my logs, but the your log shows: my log shows: I upload my builder images here and use the oom-app to test, can you help to try it again. Thanks so much :) //======================== I also want to know some detail logs |
|
I could not reproduce even with your builder @zhoufenqin (I was already using your sample app). My suggestion was not to adjust the Metaspace directly but instead to adjust the class count variable at runtime which should achieve an increase also: It may be helpful to share your container/runtime configuration also, this may be where the difference lies. The extra debug output can be seen by setting the env var |
|
cglib is widely used in spring and it will create the classes during runtime, how can we handle this case? Normaly I saw the final metaspace size is less than 100m, I think it is too smaller according to me experience for spring project |
|
@showpune It's a possibility but I don't think that we've observed this making a material difference in most apps. The formula we use counts the classes and then multiplies by a factor of 0.35. It then multiplies by 5,800 bytes per class. So if cglib generated 1000 extra classes that's only roughly a 2M difference in the calculation. There's overhead included in the calculation by default to account for things like this. If you are doing something that is causing an extremely large number of proxy classes to be calculated, then you'd need to adjust the setting @pivotal-david-osullivan mentioned to account for what your app is doing. If you have a concern I would suggest you monitor your metaspace usage and see what is consuming the space. You can then tune accordingly. @zhoufenqin @pivotal-david-osullivan
It is showing only 147 agent classes for AppDynamics. That seems low to me. I wonder if the agent changed something, some of the agents do things to hide the classes. We are looking at these extensions for class file counting now,
Maybe there's an extension we need to add? |
|
@showpune I think you should be able to use AppDynamics too, like if you're already using that. There's a lot of overlap between the two tools. Just not both at the same time. It appears like there's an issue between AppD & JFR there. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I create a spring boot app with Dynamic buildpack, the app failed to start with
Terminating due to java.lang.OutOfMemoryError: Metaspaceerror.The OOM app sample respo: https://github.com/zhoufenqin/java-sample/tree/main/echo-app-oom
The buildpack i'm using:
tanzu-java-azure:7.28.0
libjvm v1.37.0
The errorlog show:
Then I add some other dependency in
pom.xml, likeThe app can be started, the metaspace is larger than before
The correct app repo: https://github.com/zhoufenqin/java-sample/tree/main/echo-app
The log shows:
These two app are only different in
pom.xml, the correct one have eureka dependency but the oom one hasn'tHow the
Metaspace sizecalculate?Expected Behavior
The calculate Metaspace size should large enough
Current Behavior
The app failed to start because of OOM metaspace
Possible Solution
Steps to Reproduce
Motivations
The text was updated successfully, but these errors were encountered: