Traefik generating massive log files on error logging #189

plagueofgophers · 2021-10-19T01:50:43Z

plagueofgophers
Oct 19, 2021

Hello.

I've been following the guide- which is fantastic, by the way! - and have Traefik up and running, accessible through my domain and with Authelia 2FA functioning using Duo push notifications. However, I've experienced an issue where Traefik is generating thousands of log entries per second, and it's rapidly bloating my VM's storage. The containers have been up for just over an hour and the log file is just over 1GB. It's constant activity with the static IP I have set for the Docker t2 network. Interestingly, I had moved to Authelia as I had the same issue with OAuth.

I've checked my Cloudflare dashboard to see if I was being hit by some kind of attack and it's only registering ~1k requests over the last 24 hours.

Log extract below:

192.168.90.254 - - [18/Oct/2021:23:23:49 +0000] "POST /boaform/admin/formLogin HTTP/1.1" 499 21 "-" "-" 26920 "http-catchall@docker" "http://192.168.90.254:80" 9939ms
192.168.90.254 - - [18/Oct/2021:23:23:49 +0000] "POST /boaform/admin/formLogin HTTP/1.1" 499 21 "-" "-" 26921 "http-catchall@docker" "http://192.168.90.254:80" 9939ms
192.168.90.254 - - [18/Oct/2021:23:23:49 +0000] "POST /boaform/admin/formLogin HTTP/1.1" 499 21 "-" "-" 26922 "http-catchall@docker" "http://192.168.90.254:80" 9939ms
192.168.90.254 - - [18/Oct/2021:23:23:49 +0000] "POST /boaform/admin/formLogin HTTP/1.1" 499 21 "-" "-" 26923 "http-catchall@docker" "http://192.168.90.254:80" 9939ms
192.168.90.254 - - [18/Oct/2021:23:23:49 +0000] "POST /boaform/admin/formLogin HTTP/1.1" 499 21 "-" "-" 26924 "http-catchall@docker" "http://192.168.90.254:80" 9938ms
192.168.90.254 - - [18/Oct/2021:23:23:49 +0000] "POST /boaform/admin/formLogin HTTP/1.1" 499 21 "-" "-" 26925 "http-catchall@docker" "http://192.168.90.254:80" 9938ms
192.168.90.254 - - [18/Oct/2021:23:23:49 +0000] "POST /boaform/admin/formLogin HTTP/1.1" 499 21 "-" "-" 26926 "http-catchall@docker" "http://192.168.90.254:80" 9938ms
192.168.90.254 - - [18/Oct/2021:23:23:49 +0000] "POST /boaform/admin/formLogin HTTP/1.1" 499 21 "-" "-" 26927 "http-catchall@docker" "http://192.168.90.254:80" 9938ms
192.168.90.254 - - [18/Oct/2021:23:23:49 +0000] "POST /boaform/admin/formLogin HTTP/1.1" 499 21 "-" "-" 26928 "http-catchall@docker" "http://192.168.90.254:80" 9938ms
.........
Skipped some 6.4m lines here
.........
192.168.90.254 - - [19/Oct/2021:00:48:23 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 6461802 "http-catchall@docker" "http://192.168.90.254:80" 1ms
192.168.90.254 - - [19/Oct/2021:00:48:23 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 6461803 "http-catchall@docker" "http://192.168.90.254:80" 1ms
192.168.90.254 - - [19/Oct/2021:00:48:23 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 6461804 "http-catchall@docker" "http://192.168.90.254:80" 1ms
192.168.90.254 - - [19/Oct/2021:00:48:23 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 6461805 "http-catchall@docker" "http://192.168.90.254:80" 1ms
192.168.90.254 - - [19/Oct/2021:00:48:23 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 6461806 "http-catchall@docker" "http://192.168.90.254:80" 1ms
192.168.90.254 - - [19/Oct/2021:00:48:23 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 6461807 "http-catchall@docker" "http://192.168.90.254:80" 1ms
192.168.90.254 - - [19/Oct/2021:00:48:23 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 6461808 "http-catchall@docker" "http://192.168.90.254:80" 0ms
192.168.90.254 - - [19/Oct/2021:00:48:23 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 6461809 "http-catchall@docker" "http://192.168.90.254:80" 0ms
192.168.90.254 - - [19/Oct/2021:00:48:23 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 6461810 "http-catchall@docker" "http://192.168.90.254:80" 0ms
192.168.90.254 - - [19/Oct/2021:00:48:23 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 6461811 "http-catchall@docker" "http://192.168.90.254:80" 0ms
192.168.90.254 - - [19/Oct/2021:00:48:23 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 6461813 "http-catchall@docker" "http://192.168.90.254:80" 0ms
192.168.90.254 - - [19/Oct/2021:00:48:23 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 6461812 "http-catchall@docker" "http://192.168.90.254:80" 0ms
192.168.90.254 - - [19/Oct/2021:00:48:23 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 6461814 "http-catchall@docker" "http://192.168.90.254:80" 0ms

Authelia configuration.yml:

###############################################################
#                   Authelia configuration                    #
###############################################################

host: 0.0.0.0
port: 9091
log_level: error

# This secret can also be set using the env variables AUTHELIA_JWT_SECRET_FILE
# I used this site to generate the secret: https://www.grc.com/passwords.htm
jwt_secret: TEXT_OF_SECRET_HERE
 # use docker secret file instead AUTHELIA_JWT_SECRET_FILE

# https://docs.authelia.com/configuration/miscellaneous.html#default-redirection-url
default_redirection_url: https://authelia.mydomain.com

totp:
  issuer: authelia.com
  period: 30
  skew: 1

# Enable the following for Duo Push Notification support
# https://www.authelia.com/docs/features/2fa/push-notifications.html
duo_api:
  hostname: DUO_API_URL
  integration_key: DUO_INT_KEY
  # This secret can also be set using the env variables AUTHELIA_DUO_API_SECRET_KEY_FILE
  secret_key: DUO_SECRET

authentication_backend:
  file:
    path: /config/users_database.yml
    # customize passwords based on https://docs.authelia.com/configuration/authentication/file.html
    password:
      algorithm: argon2id
      iterations: 1
      salt_length: 16
      parallelism: 8
      memory: 1024 # blocks this much of the RAM. Tune this.

# https://docs.authelia.com/configuration/access-control.html
access_control:
  default_policy: deny
  rules:
    # Rules applied to everyone
    - domain: authelia.mydomain.com
      policy: bypass
#    - domain: radarr.mydomain.com
#      policy: bypass
#      subject: "group:admins"
#      resources:
#      - "^/api/.*$"      
#    - domain: "*.mydomain.com"
#      policy: one_factor
    - domain: "*.mydomain.com"
      policy: two_factor
    - domain: "mydomain.com"
      policy: two_factor
      networks:
        - 192.168.1.0/24
#         - HOME-WAN-IP-ADDRESS

session:
  name: authelia_session
  # This secret can also be set using the env variables AUTHELIA_SESSION_SECRET_FILE
  # Used a different secret, but the same site as jwt_secret above.
  # secret: SECRET_GOES_HERE # use docker secret file instead AUTHELIA_SESSION_SECRET_FILE
  expiration: 3600 # 1 hour
  inactivity: 300 # 5 minutes
  domain: mydomain.com # Should match whatever your root protected domain is

# Optional. Can improve performance on a busy system. If not enabled, session info is stored in memory.
#  redis:
#    host: redis
#    port: 6379
    # This secret can also be set using the env variables AUTHELIA_SESSION_REDIS_PASSWORD_FILE
    # password: authelia

regulation:
  max_retries: 3
  find_time: 120
  ban_time: 300

storage:
# For local storage, uncomment lines below and comment out mysql. https://docs.authelia.com/configuration/storage/sqlite.html
  local:
    path: /config/db.sqlite3
#  mysql:
  # MySQL allows running multiple authelia instances. Create database and enter details below.
#    host: MYSQL_HOST
#    port: 3306
#    database: authelia
#    username: DBUSERNAME
    # Password can also be set using a secret: https://docs.authelia.com/configuration/secrets.html
    # password: use docker secret file instead AUTHELIA_STORAGE_MYSQL_PASSWORD_FILE

notifier:
#  smtp:
#    username: SMTP_USERNAME
    # This secret can also be set using the env variables AUTHELIA_NOTIFIER_SMTP_PASSWORD_FILE
    # password: # use docker secret file instead AUTHELIA_NOTIFIER_SMTP_PASSWORD_FILE
#    host: SMTP_HOST
#    port: 587 #465
#    sender: SENDER_EMAIL
  # For testing purpose, notifications can be sent in a file. Be sure map the volume in docker-compose.
  filesystem:
    filename: /config/notification.txt

Docker-compose:

version: "3.7"

####### NETWORKS

networks:
  t2_proxy:
    external:
      name: t2_proxy
  default:
    driver: bridge


####### SERVICES
services:
  traefik:
    container_name: traefik
    image: traefik:v2.5
    restart: always
    command: # CLI arguments
      - --global.checkNewVersion=true
      - --global.sendAnonymousUsage=true
      - --entryPoints.http.address=:80
      - --entryPoints.https.address=:443
        # Allow these IPs to set the X-Forwarded-* headers - Cloudflare IPs: https://www.cloudflare.com/ips/
      - --entrypoints.https.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22
      - --entryPoints.traefik.address=:8080
      - --api=true
#      - --api.insecure=true
#      - --serversTransport.insecureSkipVerify=true
      - --log=true
      - --log.level=ERROR # (Default: error) DEBUG, INFO, WARN, ERROR, FATAL, PANIC
      - --accessLog=true
      - --accessLog.filePath=/traefik.log
      - --accessLog.bufferingSize=100 # Configuring a buffer of 100 lines
      - --accessLog.filters.statusCodes=400-499
      - --providers.docker=true
      - --providers.docker.endpoint=unix:///var/run/docker.sock
      - --providers.docker.defaultrule=Host(`{{ index .Labels "com.docker.compose.service" }}.$DOMAINNAME`)
      - --providers.docker.exposedByDefault=false
      - --providers.docker.network=t2_proxy
      - --providers.docker.swarmMode=false
      - --providers.file.directory=/rules # Load dynamic configuration from one or more .toml or .yml files in a directory.
#      - --providers.file.filename=/path/to/file # Load dynamic configuration from a file.
      - --providers.file.watch=true # Only works on top level files in the rules folder
#       - --certificatesResolvers.dns-cloudflare.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory # LetsEncrypt Staging Server - uncomment when testing
      - --certificatesResolvers.dns-cloudflare.acme.email=$CLOUDFLARE_EMAIL
      - --certificatesResolvers.dns-cloudflare.acme.storage=/acme.json
      - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.provider=cloudflare
      - --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.resolvers=1.1.1.1:53,1.0.0.1:53
    networks:
      t2_proxy:
        ipv4_address: 192.168.90.254 # You can specify a static IP
      # networks:
      #   - t2_proxy
#       socket_proxy:
    #depends_on:
    #  - socket-proxy
    security_opt:
      - no-new-privileges:true
    #healthcheck:
    #  test: ["CMD", "traefik", "healthcheck", "--ping"]
    #  interval: 5s
    #  retries: 3
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 8080
        published: 8080
        protocol: tcp
        mode: host
      #- target: 8081
      #  published: 8082
      #  protocol: tcp
      #  mode: host
    volumes:
      - $DOCKERDIR/traefik2/rules:/rules # file provider directory
      - /var/run/docker.sock:/var/run/docker.sock:ro # Use Docker Socket Proxy instead for improved security
      - $DOCKERDIR/traefik2/acme/acme.json:/acme.json # cert location - you must touch this file and change permissions to 600
      - $DOCKERDIR/traefik2/traefik.log:/traefik.log # for fail2ban - make sure to touch file before starting container
      - $DOCKERDIR/shared:/shared
    environment:
      - CF_API_EMAIL=$CLOUDFLARE_EMAIL
      - CF_API_KEY=$CLOUDFLARE_API_KEY
      # - CF_API_EMAIL_FILE=/run/secrets/cloudflare_email
      # - CF_API_KEY_FILE=/run/secrets/cloudflare_api_key
#     secrets:
#       - cloudflare_email
#       - cloudflare_api_key
    labels:
      #- "autoheal=true"
      - "traefik.enable=true"
      # HTTP-to-HTTPS Redirect
      - "traefik.http.routers.http-catchall.entrypoints=http"
      - "traefik.http.routers.http-catchall.rule=HostRegexp(`{host:.+}`)"
#       - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
#       - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
#       - "traefik.http.middlewares.redirect-to-https.redirectscheme.permanent=true"
      # HTTP Routers
      - "traefik.http.routers.traefik-rtr.entrypoints=https"
      - "traefik.http.routers.traefik-rtr.rule=Host(`traefik.$DOMAINNAME`)"
      - "traefik.http.routers.traefik-rtr.tls=true"
#       - "traefik.http.routers.traefik-rtr.tls.certresolver=dns-cloudflare" # Comment out this line after first run of traefik to force the use of wildcard certs
      - "traefik.http.routers.traefik-rtr.tls.domains[0].main=$DOMAINNAME"
      - "traefik.http.routers.traefik-rtr.tls.domains[0].sans=*.$DOMAINNAME"
      ## Services - API
      - "traefik.http.routers.traefik-rtr.service=api@internal"
      ## Healthcheck/ping
      #- "traefik.http.routers.ping.rule=Host(`traefik.$DOMAINNAME`) && Path(`/ping`)"
      #- "traefik.http.routers.ping.tls=true"
      #- "traefik.http.routers.ping.service=ping@internal"
      ## Middlewares
      - "traefik.http.routers.traefik-rtr.middlewares=chain-authelia@file"

  # Google OAuth - Single Sign On using OAuth 2.0
  # https://www.smarthomebeginner.com/google-oauth-with-traefik-docker/
#   oauth:
#     container_name: oauth
#     image: thomseddon/traefik-forward-auth:2
    # image: thomseddon/traefik-forward-auth:2.1-arm # Use this image with Raspberry Pi
#     restart: always
    # profiles:
    # - core
#     networks:
#       - t2_proxy
#     security_opt:
#       - no-new-privileges:true
    # Allow apps to bypass OAuth. Radarr example below will bypass OAuth if API key is present in the request (eg. from NZB360 mobile app).
    # While this is one way, the recommended way is to bypass authentication using Traefik labels shown in some of the apps later.
    # command: --rule.radarr.action=allow --rule.radarr.rule="Headers(`X-Api-Key`, `$RADARR_API_KEY`)"
    # command: --rule.sabnzbd.action=allow --rule.sabnzbd.rule="HeadersRegexp(`X-Forwarded-Uri`, `$SABNZBD_API_KEY`)"
#     environment:
#       - PROVIDERS_GOOGLE_CLIENT_ID=$GOOGLE_CLIENT_ID
#       - PROVIDERS_GOOGLE_CLIENT_SECRET=$GOOGLE_CLIENT_SECRET
#       - SECRET=$OAUTH_SECRET
#       - COOKIE_DOMAIN=$DOMAINNAME
#       - INSECURE_COOKIE=false
#       - AUTH_HOST=oauth.$DOMAINNAME
#       - URL_PATH=/_oauth
#       - WHITELIST=$MY_EMAIL
#       - LOG_LEVEL=warn
#       - LOG_FORMAT=text
#       - LIFETIME=2592000 # 30 days
#       - DEFAULT_ACTION=auth
#       - DEFAULT_PROVIDER=google
#     secrets: # had trouble getting secrets to work: https://github.com/thomseddon/traefik-forward-auth/issues/155#issuecomment-664630985
#       - google_client_id
#       - google_client_secret
#       - oauth_secret
#       - my_email
#     labels:
#       - "traefik.enable=true"
#       ## HTTP Routers
#       - "traefik.http.routers.oauth-rtr.tls=true"
#       - "traefik.http.routers.oauth-rtr.entrypoints=https"
#       - "traefik.http.routers.oauth-rtr.rule=Host(`oauth.$DOMAINNAME`)"
#       ## Middlewares
#       - "traefik.http.routers.oauth-rtr.middlewares=chain-oauth@file"
#       ## HTTP Services
#       - "traefik.http.routers.oauth-rtr.service=oauth-svc"
#       - "traefik.http.services.oauth-svc.loadbalancer.server.port=4181"

  # Authelia (Lite) - Self-Hosted Single Sign-On and Two-Factor Authentication
  authelia:
    container_name: authelia
#     image: authelia/authelia:latest
    image: authelia/authelia:4.31.0
    restart: always
    networks:
      - t2_proxy
#     depends_on:
#       - mariadb
#       - redis
    volumes:
      - $DOCKERDIR/authelia:/config
    environment:
      - TZ=$TZ
#       - AUTHELIA_JWT_SECRET_FILE=/run/secrets/authelia_jwt_secret
#       - AUTHELIA_SESSION_SECRET_FILE=/run/secrets/authelia_session_secret
#       - AUTHELIA_STORAGE_MYSQL_PASSWORD_FILE=/run/secrets/authelia_storage_mysql_password
#       - AUTHELIA_NOTIFIER_SMTP_PASSWORD_FILE=/run/secrets/authelia_notifier_smtp_password
#       - AUTHELIA_DUO_API_SECRET_KEY_FILE=/run/secrets/authelia_duo_api_secret_key
#     secrets:
#       - authelia_jwt_secret
#       - authelia_session_secret
#       - authelia_storage_mysql_password
#       - authelia_notifier_smtp_password
#       - authelia_duo_api_secret_key
    labels:
      - "traefik.enable=true"
      ## HTTP Routers
      - "traefik.http.routers.authelia-rtr.tls=true"
      - "traefik.http.routers.authelia-rtr.entrypoints=https"
      - "traefik.http.routers.authelia-rtr.rule=Host(`authelia.$DOMAINNAME`)"
      - "traefik.http.routers.authelia-rtr.rule=HostHeader(`authelia.$DOMAINNAME`)"
      ## Middlewares
      - "traefik.http.routers.authelia-rtr.middlewares=chain-authelia@file"
      ## HTTP Services
      - "traefik.http.routers.authelia-rtr.service=authelia-svc"
      - "traefik.http.services.authelia-svc.loadbalancer.server.port=9091"

Can anyone shed some light on what's going on here?

Thanks all.

plagueofgophers · 2021-10-19T08:27:15Z

plagueofgophers
Oct 19, 2021
Author

Bloody hell, I'm an idiot. I've been looking at this problem for two weeks and never spotted the obvious until now.

I'm going to leave this here as an instructive lesson, that this is what happens when you comment out most of your https redirect.

#       - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
#       - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
#       - "traefik.http.middlewares.redirect-to-https.redirectscheme.permanent=true"

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Traefik generating massive log files on error logging #189

{{title}}

Replies: 1 comment

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Traefik generating massive log files on error logging #189

plagueofgophers Oct 19, 2021

Replies: 1 comment

plagueofgophers Oct 19, 2021 Author

plagueofgophers
Oct 19, 2021

plagueofgophers
Oct 19, 2021
Author