Database connection closes unexpectedly

[Ka0o0]

Hi,

I'm running acme2certifier with Django and uwsgi behind a NGINX reverse proxy. Here are my server configuration files:

Django Settings

"""
Django settings for acme2certifier project.

Generated by 'django-admin startproject' using Django 1.11.15.

For more information on this file, see
https://docs.djangoproject.com/en/1.11/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.11/ref/settings/
"""

import os
import sys

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

def get_env_value_or_exit(var):
    if var not in os.environ:
        print(f"Error: The following environment variable is missing: {var}")
        sys.exit(1)
    else:
        return os.environ[var]


# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.11/howto/deployment/checklist/


# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = False

ALLOWED_HOSTS = ['127.0.0.1','*']


# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'acme_srv'
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    # 'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'acme2certifier.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'acme2certifier.wsgi.application'


# Database
# https://docs.djangoproject.com/en/1.11/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': get_env_value_or_exit('DB_NAME'),
        'USER': get_env_value_or_exit('DB_USER'),
        'PASSWORD': get_env_value_or_exit('DB_PASSWORD'),
        'HOST': get_env_value_or_exit('DB_HOST'),
        'PORT': get_env_value_or_exit('DB_PORT'),
    }
}


# Password validation
# https://docs.djangoproject.com/en/1.11/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]


# Internationalization
# https://docs.djangoproject.com/en/1.11/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = False

USE_L10N = True

USE_TZ = False


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.11/howto/static-files/

STATIC_URL = '/static/'
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = get_env_value_or_exit('SECRET_KEY')

DEFAULT_AUTO_FIELD='django.db.models.AutoField'

UWSGI Configuration File

[uwsgi]
strict = true
module = acme2certifier_wsgi:application
master = true
enable-threads = true
vacuum = true
single-interpreter = true
need-app = true
processes = 1
threads = 5
uid = www-data
socket = :9090
die-on-term = true

NGINX Sever Section

        server {
            listen 443 default_server ssl;
            listen [::]:443 default_server ssl;

            server_name acme01.company.com;

            ssl_certificate /data/tls_certificates/tls.crt;
            ssl_certificate_key /data/tls_certificates/tls.key;

            # first 5 requests go trough instantly 5more requests evey 100ms
            limit_req zone=ip burst=10; # delay=5;

            server_name _;
            location = favicon.ico { access_log off; log_not_found off; }
            location / {
                access_log off;
                include uwsgi_params;
                uwsgi_pass "acme2certifier:9090";
                if ($request_method = "HEAD" ) {
                  add_header Content-length 0;
                  # add_header Transfer-Encoding identity;
                }
          }
        }

My problem is that after running for a couple of hours, acme2certifier seems to loose connection to the PostgreSQL database. Here is the error message:

acme2certifier database error in Nonce.generate_and_add(): server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

A restart of the service solves the problem so I'm pretty sure that this is not a problem with the DB itself.
I'm a bit unsure if there is some sort of keep-alive mechanism not working properly and which component might be configured wrongly.
I've activated threading support on UWSGI and limited the number of processes to 1, so might this be a problem?
Anyway, any help is highly appreciated.

BR Kai

[grindsa]

Hi,

What are you trying to achieve by enabling threading support/limiting the uwsgi process? (sorry I am not really an expert in this topic). Can you please share the UWSGI configuration that i can try to replicate this issue on side?

BR G.

[Ka0o0]

The communication with our PKI takes some time when issuing certificates. I wanted to serve multiple requests in parallel so I've enabled threading here. I've read, however, that uWSGI threading might create some issues with database connections. I'll test if disabling threading fixes the problem.

[Ka0o0]

It seems that the error occurs regardless of the thread and process configuration. I've tried to reduce it to 1 thread and 1 process but the error persists. Whenever I start the service and let it run for a couple of hours then the error will occur.

[grindsa]

can you please describe your environment a bid more in detail? I am especially interested in:

• HostOS of the server runing a2c
• acme2certifier version
• django version
• version of the psycopg2 module
• deployment type (manual, rpm, deb)
• are a2c and your database server co-located or deployed on different systems? If they are deployed on different systems, is there a firewall in between?

[Ka0o0]

Hey,

I'm running acme2certifier in a Kubernetes environment. acme2certifier is deployed as deployment and I use a NGINX reverse proxy.

Here is the Docker build file for the acme2certifier container

FROM debian:12.5-slim as build

# python3 general requirement for acme2certifier
# uwsgi WSGI runtime server for acme2certifier
# uwsgi-plugin-python3 plugin for uwsgi for python support
# python3-impacket required for CA handler for Microsoft Windows Client Certificate Enrollment Protocol
RUN apt-get update &&\
  apt-get upgrade -y &&\
  apt-get install -y --no-install-recommends \
    wget \
    ca-certificates \
    python3 \
    uwsgi \
    uwsgi-plugin-python3 \
    python3-django \
    python3-psycopg2 \
    python3-impacket &&\
  apt-get clean

RUN wget -O /acme2certifier.deb https://github.com/grindsa/acme2certifier/releases/download/0.34/acme2certifier_0.34-1_all.deb &&\
  apt-get --fix-broken -y install /acme2certifier.deb &&\
  rm /acme2certifier.deb

RUN apt-get -y remove wget

COPY build/root /

RUN mkdir /var/www/acme2certifier/volume/

# Copy some django stuff into acme2certifier root folder
RUN cp -R /var/www/acme2certifier/examples/django/* /var/www/acme2certifier/
# Overwrite the DB handler script to be a django db handler
RUN cp /var/www/acme2certifier/examples/db_handler/django_handler.py /var/www/acme2certifier/acme_srv/db_handler.py

COPY ./build/misc/django_settings.py /var/www/acme2certifier/acme2certifier/settings.py

RUN chown www-data:www-data -R /var/www/acme2certifier

FROM build as live

# First, harden the container by specifying a non root user
USER www-data

WORKDIR /var/www/acme2certifier

ENTRYPOINT [ "/docker-entrypoint.sh" ]

CMD [ "uwsgi", "--plugin", "python3", "acme2certifier.uswgi.ini" ]

The entrypoint of the Docker container looks like this:

#!/bin/bash

set -e

echo "apply migrations"  >> /proc/1/fd/1
python3 /var/www/acme2certifier/tools/django_update.py
python3 manage.py loaddata acme_srv/fixture/status.yaml

chown -R www-data /var/www/acme2certifier/volume
chmod u+s /var/www/acme2certifier/volume/

exec "$@"

The database is located on another server in the same network. But even if there were network outages, I would expect a2c to recover itself, e.g. try to reconnect to the server, which does not happen. Instead I have to manually restart the application.

[grindsa]

Hi, i am sorry but still not able to replicate your issue. Can you try to replicate the issue with one of the a2c containers available at docker-hub