Docker image from build-pack Gradle get trouble with volume mount AWS EFS!

[kenwiths]

Hi, my project get troubled when uses image from build-pack with volume mount EFS in AWS EKS! The bug is that my container not boot up, and it just stuck with log "Setting Active Processor Count to 4", this issue occurs when volume is heavier (may over 20GB)

But after I remove volume mount from k8s yaml file, the container run normally. I think this container go scan the folder mount before boot application, and when I join to stuck's container, i do these command ls -la or du -sh and it's load very long, about 2 - 3m will return result. The heavier volume will make the container load longer. I see that container starting boot after 7 - 10m (time when created container) while volume over 20BG, another container starting boot after 14 - 18m while volume over 30G.

Then I do another try, I use same k8s yaml file (with volume mount), use all same config. I just build project image through Dockerfile, not build-pack ./gradlew bootBuildImage. And the container running well with same volume, I also re-check between both image, Dockerfile tag 1.2 run ok, but Gradle tag 1.1 is stucked.

Here is my Dockerfile - This build is not stuck, loading folder mount very well

FROM openjdk:17-jdk-alpine
WORKDIR /workspace

ARG JAR_FILE=./build/libs/*.jar
COPY ${JAR_FILE} app.jar

EXPOSE 8080

ENTRYPOINT ["java", "-jar", "/workspace/app.jar"]

Here is my setup with Gradle build-pack - This build make container got stuck when boot application, folder mount still need time when i ls -la file

• File build.gradle

    bootBuildImage {
        builder = "paketobuildpacks/builder:tiny"
        pullPolicy = "IF_NOT_PRESENT"
        bindings = [rootDir.absolutePath + '/paketo-buildpacks/new-relic:/platform/bindings/new-relic']
        buildpacks = [
                "gcr.io/paketo-buildpacks/ca-certificates:3.6.1",
                "gcr.io/paketo-buildpacks/amazon-corretto:8.1.1",
                "gcr.io/paketo-buildpacks/syft:1.29.0",
                "gcr.io/paketo-buildpacks/executable-jar:6.7.0",
                "gcr.io/paketo-buildpacks/dist-zip:5.6.0",
                "gcr.io/paketo-buildpacks/spring-boot:5.24.0",
                "gcr.io/paketo-buildpacks/new-relic:7.0.0"
        ]
        environment = [
                "BP_JVM_VERSION" : "17.*",
                "BPL_SPRING_CLOUD_BINDINGS_DISABLED": "true"
        ]
        runImage = "public.ecr.aws/************/runimage:2.0"
    }

• Dockerfile (Here is runImage for setup tool or copy some needed file)

FROM ubuntu:bionic as base

ENV CNB_STACK_ID="io.paketo.stacks.tiny"
LABEL io.buildpacks.stack.id="io.paketo.stacks.tiny"

USER root

WORKDIR /app

# Install packages that we want to make available at both build and run time  
RUN echo "-----> INSTALL xz-utils && ca-certificates" && \
    apt-get update >> /dev/null && \
    apt-get install -y xz-utils ca-certificates >> /dev/null && \
    rm -rf /var/lib/apt/lists/*

# apt-get install
RUN echo "-----> INSTALL google-library, heif" && \
    apt-get update && \
    apt-get install -y libheif-examples gconf-service libasound2 libatk1.0-0 libatk-bridge2.0-0 libc6 libcairo2 libcups2 libdbus-1-3 \
    libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 \
    libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 \
    libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release \
    xdg-utils
    
# COPY TOOL
COPY ./tool .
  
# Move file into /usr/local/bin
RUN	cp -r ./node16/bin/ ./node16/include ./node16/lib ./node16/share /usr/local/ && \
    # remove un-use folder
    rm -rf node16

RUN apt install ffmpeg -y

## ---Run image---
FROM base as run

LABEL io.buildpacks.stack.mixins="[\"run:heif-convert\", \"run:node\", \"run:npm\", \"run:npx\", \"run:chrome\", \"run:ffmpeg\", \"run:ffprobe\"]"

RUN groupadd -g 1000 eks && \
    useradd -u 1000 -g 1000 -s /bin/bash -m eks
USER eks

• This is my command to build image ./gradlew -p apilayer bootBuildImage --imageName=***:***

Here is Success Image when container booted

• Build from Dockerfile
  

• Build from Gradle build-pack
  

Do you guys found any trouble with my build-pack setup ?

[dmikusa]

build-pack with volume mount EFS in AWS EKS

Where are you mounting this in the container? What exists on your EKS mount?

[kenwiths]

I mount to folder /workspace/efs in container. The mount folder contain upload file like jpg, mp3, mp4, pdf, ... and many mail file for RabbitMQ

[dmikusa]

Ok, so the buildpack has a helper that runs before your app starts which dynamically adjusts the memory configuration of the JVM based on the memory available to your container. To do this, it needs to know how many class files are in your app so it can size the metaspace.

To do this, it walks the app directory and counts up the class files. This should generally be fast, where you don't even notice it, but you're binding in remote storage which probably has tons of files and making tons of files, and that is going to be noticeable.

There are two things you can do about this now:

1. Mount your files somewhere else. Don't put them under /workspace & point your application to the new location.
2. You can manually input a class count by setting BPL_JVM_LOADED_CLASS_COUNT at launch time (i.e. when you run your container). The memory calculator will see this and skip doing a count in favor of the value you entered. The downfall of this approach is that if you size the count wrong, you'll have metaspace issues (not enough/too much).

Because it's hard to plan in terms of class count, what I would suggest if you go with option 2.) is that you set some arbitrary value for class count like 10. This will bypass the look-up that is taking a lot of time in your environment, then manually add -XX:MaxMetaspaceSize= to JAVA_TOOL_OPTIONS. This will allow you to set the metaspace in terms of memory size, which is probably easier to do. This works because you can override any of the memory calculator settings by adding JVM memory flags to JAVA_TOOL_OPTIONS.

Lastly, feel free to open an issue against https://github.com/paketo-buildpacks/libjvm/issues. We could probably add an env variable to allow you to skip directories or something when the memory calculator is doing its class counting. That way, if you know a large sub directory won't contain Java classes, you can pass over that directory and speed things up.

[kenwiths]

Thanks you, I move folder mount to a new folder not in "/workspace", then it work!