Cannot reserve bytes of direct buffer memory

[jd1289]

Hi All,

Having some issue with running image build with spring boot buildpacks via bootBuildImage. I've run the image locally and in a dummy rancher k8s env, and it runs fine.

The app runs a Netty Server (due to SMPP library cloudhopper)

When I deploy the app to the test k8s env. I ran into the
Failed to instantiate [com.cloudhopper.smpp.impl.DefaultSmppServer]: Factory method 'defaultSmppServer' threw exception with message: Cannot reserve 65536 bytes of direct buffer memory (allocated: 20931426, limit: 20971520)

The start up logs had the following log:
Setting Active Processor Count to 8 Adding $JAVA_OPTS to $JAVA_TOOL_OPTIONS Calculated JVM Memory Configuration: -Xmx611568K -XX:MaxMetaspaceSize=119567K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 1G, Thread Count: 50, Loaded Class Count: 18696, Headroom: 0%) Enabling Java Native Memory Tracking Adding 137 container CA certificates to JVM truststore Spring Cloud Bindings Enabled Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=8 -XX:MaxDirectMemorySize=20M -Xmx611568K -XX:MaxMetaspaceSize=119567K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

the k8s resources are:
requests: cpu: 400m memory: 500Mi limits: cpu: 800m memory: 1Gi

I thought maybe it was the MaxDirectMemorySize so I upped it to 100m, but I think it misunderstood the issue.

Can someone shed some light on this?

snippet of stack trace:
org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)\n\t... 41 more\nCaused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.cloudhopper.smpp.impl.DefaultSmppServer]: Factory method 'defaultSmppServer' threw exception with message: Cannot reserve 65536 bytes of direct buffer memory (allocated: 10445666, limit: 10485760)\n\tat org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:171)\n\tat org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:655)\n\t... 55 more\nCaused by: java.lang.OutOfMemoryError: Cannot reserve 65536 bytes of direct buffer memory (allocated: 10445666, limit: 10485760)\n\tat java.base/java.nio.Bits.reserveMemory(Unknown Source)\n\tat java.base/java.nio.DirectByteBuffer.<init>(Unknown Source)\n\tat java.base/java.nio.ByteBuffer.allocateDirect(Unknown Source)\n\tat org.jboss.netty.channel.socket.nio.SocketSendBufferPool$Preallocation.<init>(SocketSendBufferPool.java:156)\n\tat org.jboss.netty.channel.socket.nio.SocketSendBufferPool.<init>(SocketSendBufferPool.java:42)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:45)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)\n\tat org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)\n\tat org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:142)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)\n\tat org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)\n\tat org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)\n\tat org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)\n\tat org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)\n\tat com.cloudhopper.smpp.impl.DefaultSmppServer.<init>(DefaultSmppServer.java:142)\n\tat config.SmppServerDefaultConfiguration.defaultSmppServer(SmppServerDefaultConfiguration.java:23)\n\tat smpp.config.SmppServerDefaultConfiguration$$SpringCGLIB$$0.CGLIB$defaultSmppServer$0(<generated>)\n\tat smpp.config.SmppServerDefaultConfiguration$$SpringCGLIB$$2.invoke(<generated>)\n\tat org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:258)\n\tat org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)\n\tat smpp.config.SmppServerDefaultConfiguration$$SpringCGLIB$$0.defaultSmppServer(<generated>)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.base/java.lang.reflect.Method.invoke(Unknown Source)\n\tat org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:139)\n\t... 56 more\n","exception":"org.springframework.beans.factory.UnsatisfiedDependencyException"}

thanks

[dmikusa]

Your initial request has -XX:MaxDirectMemorySize=20M and in the error you can see the limit is set to 20M

Failed to instantiate [com.cloudhopper.smpp.impl.DefaultSmppServer]: Factory method 'defaultSmppServer' threw exception with message: Cannot reserve 65536 bytes of direct buffer memory (allocated: 20931426, limit: 20971520)

The second time, you said you set it to 100M, but in the error it is reporting 10M.

Cannot reserve 65536 bytes of direct buffer memory (allocated: 10445666, limit: 10485760)

You might want to double-check that the setting got applied as 100M. This is exactly the type of error you'd see when the direct memory limit is just too low, so increasing the direct memory limit is what I would suggest here.