Critical issue on nginx worker exited on signal 4

[hieuhtr]

Issue: nginx worker continuously exited on signal 4

My brotli configuration

brotli on;
brotli_comp_level 9;
brotli_static on;
brotli_min_length 1024;
brotli_types    application/atom+xml application/javascript application/json application/rss+xml
                    application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype
                    application/x-font-ttf application/x-javascript application/xhtml+xml application/xml
                    font/eot font/opentype font/otf font/truetype image/svg+xml image/vnd.microsoft.icon
                    image/x-icon image/x-win-bitmap text/css text/javascript text/plain text/xml application/x-ndjson application/vnd.spring-boot.actuator.v2+json;

Nginx error logs

2024/02/29 08:06:22 [notice] 1#1: signal 29 (SIGIO) received
2024/02/29 08:06:27 [notice] 1#1: signal 17 (SIGCHLD) received from 11110
2024/02/29 08:06:27 [alert] 1#1: worker process 11110 exited on signal 4
2024/02/29 08:06:27 [notice] 1#1: start worker process 11112
2024/02/29 08:06:27 [notice] 1#1: signal 29 (SIGIO) received
2024/02/29 08:06:27 [notice] 1#1: signal 17 (SIGCHLD) received from 11111
2024/02/29 08:06:27 [alert] 1#1: worker process 11111 exited on signal 4
2024/02/29 08:06:27 [notice] 1#1: start worker process 11113
2024/02/29 08:06:27 [notice] 1#1: signal 29 (SIGIO) received
2024/02/29 08:06:32 [info] 11112#11112: *64972 client 127.0.0.1 closed keepalive connection
2024/02/29 08:06:37 [notice] 1#1: signal 17 (SIGCHLD) received from 11112
2024/02/29 08:06:37 [alert] 1#1: worker process 11112 exited on signal 4
2024/02/29 08:06:37 [notice] 1#1: start worker process 11123
2024/02/29 08:06:37 [notice] 1#1: signal 29 (SIGIO) received

• I'm trying nginx:1.23.3 and nginx:1.24.0, using dynamic load brotli but it persist.
• content-length below 1024 B is fine as well (non compress)

Try to switch back gzip and everything is OK

As I remember, my previous nginx & brotli compiling on 2023-08-30 still fine, I just updated recently on Feb 2024

[simneu]

hi @hieuhtr,

I'm facing similar issues with a recent (but new, we didn't use brotli before) release. Did you find a resolution yet?

On my side:

• exit on signal 4 is happening every time we have brotli on in our configuration and nginx tries to serve a br compressed page. Removing brotli on from config fixes the issue.
• setup is using nginx:1.18.0 and ngx_brotli module was compiled and then packed into an AWS AMI, than we then are trying to use on our final instances. The instance building the AWS AMI and the actual end instance using the AMI are both amd64 (if that can hint to a particular direction, cause I thought this could be a module compilation issue.
• I tried debugging (with limited knowledge) what causes the signal 4, but couldn't find any hints yet.

Thanks.

[NicolasCARPi]

Had the same issue (elabftw/elabftw#4600). Had to revert to using commit 6e975bc because newer version cause this crash. Hope it can be adressed, or we'll all be stuck to this working commit ;)

It's with nginx 1.24. Full Dockerfile is available here: https://github.com/elabftw/elabimg/blob/master/Dockerfile

[simneu]

Awesome @NicolasCARPi, thanks for taking the time to help. 👍

I'll try commit you suggested when I get few minutes, and will update here with news.

[hieuhtr]

Hi guys, sorry for late update. I tried several ways but there is no hope on latest codebase.

I reverted build step by this commit hash 6e975bcb015f62e1f303054897783355e2a877dc thanks @NicolasCARPi for recommendation.

Here is my simple solution.

FROM debian:bullseye-slim AS builder

ARG NGINX_VERSION=1.24.0
ARG MOD_SECURITY_VERSION=3.0.8

# https://github.com/google/ngx_brotli/issues/166
ENV NGX_BROTLI_COMMIT_HASH=6e975bcb015f62e1f303054897783355e2a877dc

WORKDIR /root/

# Download nginx source
RUN wget http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz \
    && tar zxf nginx-${NGINX_VERSION}.tar.gz

# Install & Compile dynamic module Brotli
RUN git clone https://github.com/google/ngx_brotli \
    && cd ngx_brotli && git reset --hard $NGX_BROTLI_COMMIT_HASH \
    && git submodule update --init \
    && cd ../nginx-${NGINX_VERSION} \
    && ./configure --with-compat --add-dynamic-module=../ngx_brotli \
    && make modules

...

[b-brefeld]

After many hours of pain and misery I noticed the following. The current version works perfectly fine when running the container on the same machine where it was built, and even works on my laptop when I pull the Docker image.

However, as soon as I try running it on my VPS or a Windows 10 machine, I encounter the same "exited on signal 4 (core dumped)" error. Interestingly, building the image directly on the Windows 10 machine allows it to run on that specific machine.

This strongly suggests that the issue is related to how the Brotli module is compiled. I haven't been able to pinpoint the exact cause yet, but I hope this information can help narrow down the problem.

Based on the help in this thread I was able to get a working Dockerfile with the commit specified by @NicolasCARPi

Here is my current Dockerfile for those who need a reference:

FROM alpine:3.20.2 AS nginx_build
# NGINX stable version
ENV NGINX_VERSION=1.26.1

# Latest PCRE, ZLIB versions
ENV PCRE_VERSION=10.44
ENV ZLIB_VERSION=1.3.1

# https://github.com/google/ngx_brotli/issues/166
ENV NGX_BROTLI_COMMIT_HASH=6e975bcb015f62e1f303054897783355e2a877dc

# Install dependencies
RUN apk add --no-cache build-base git cmake

# Install PCRE2 dependency for NGINX (Required for regex support)
RUN wget https://github.com/PCRE2Project/pcre2/releases/download/pcre2-${PCRE_VERSION}/pcre2-${PCRE_VERSION}.tar.gz && \
    tar -zxf pcre2-${PCRE_VERSION}.tar.gz && \
    cd pcre2-${PCRE_VERSION} && \
    ./configure && \
    make && \
    make install

# Install ZLIB dependency for NGINX (Required for gzip compression)
RUN wget http://zlib.net/zlib-${ZLIB_VERSION}.tar.gz && \
    tar -zxf zlib-${ZLIB_VERSION}.tar.gz && \
    cd zlib-${ZLIB_VERSION} && \
    ./configure && \
    make && \
    make install

# Download NGINX
RUN wget https://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz && \
    tar zxf nginx-${NGINX_VERSION}.tar.gz

# Clone specific Brotli commit
RUN git clone https://github.com/google/ngx_brotli && \
    cd ngx_brotli && git reset --hard $NGX_BROTLI_COMMIT_HASH && \
    git submodule update --init

# Configure & Build NGINX with Brotli module
RUN cd nginx-${NGINX_VERSION} && ./configure \
    --without-http_access_module \
    --without-http_auth_basic_module \
    --without-http_autoindex_module \
    --without-http_browser_module \
    --without-http_charset_module \
    --without-http_empty_gif_module \
    --without-http_fastcgi_module \
    --without-http_geo_module \
    --without-http_limit_conn_module \
    --without-http_limit_req_module \
    --without-http_memcached_module \
    --without-http_referer_module \
    --without-http_scgi_module \
    --without-http_ssi_module \
    --without-http_split_clients_module \
    --without-http_upstream_hash_module \
    --without-http_upstream_ip_hash_module \
    --without-http_upstream_keepalive_module \
    --without-http_upstream_least_conn_module \
    --without-http_upstream_zone_module \
    --without-http_userid_module \
    --without-http_uwsgi_module \
    --with-http_sub_module \
    --with-pcre=../pcre2-${PCRE_VERSION} \
    --with-zlib=../zlib-${ZLIB_VERSION} \
    --with-compat --add-dynamic-module=../ngx_brotli \
    && make && make install