Connection Between Portus and Registry Fails Using docker-compose in Internal Network

[klmmr]

Description

I try to setup an internal docker registry using portus. In my case I cannot add the registry in Portus due to some connection problems (see below). This works fine on another test server (VPS which is publicly available to the internet). However, this fails in a restricted environment:

• Internal network, no public access from the internet
• split-DNS (FQDN is only resolved by own nameservers in the internal network)
• HTTP proxies are used for outbound traffic

As it works on a VPS but not on a machine within the restricted network, it seems that one of the above points may cause the problem. Especially proxies and DNS seem to be causing problems sometimes (possibly related issues: #2312, #2050, #1660) but there seems to be no real solution at least for proxies (issue #1683).

I appreciate any hints for further debugging. Please let me know, when you have a possible solution.

Steps to reproduce

1. Cloned this repo.
2. Change into
3. Set the machines FQDN in .env at MACHINE_FQDN.
4. Create TLS key and certificate and save as secrets/portus.{crt|key}
5. Use docker-compose.yml based on the examples (see actual file below).
6. docker-compose up -d
7. Visit Portus with a browser and create an admin user.
8. Try to setup the registry in Portus.

• Expected behavior: Set hostname to FQDN and enable SSL. Then Portus connection check should work so the registry could be added.
• Actual behavior: I tried multiple combinations of hostname and SSL on/off but none of them work.
  • Hostname: FQDN + Use SSL false: Errno::EHOSTUNREACH: connection refused You can skip this check by clicking on the "Skip remote checks" checkbox.
  • Hostname: FQDN + Use SSL true: Errno::EHOSTUNREACH: connection refused You can skip this check by clicking on the "Skip remote checks" checkbox.
  • Hostname: registry:5000 + Use SSL false: Net::HTTPBadResponse: could not stablish connection: SSL error You can skip this check by clicking on the "Skip remote checks" checkbox.
  • Hostname: registry:5000 + Use SSL true: OpenSSL::SSL::SSLError: could not stablish connection: SSL error You can skip this check by clicking on the "Skip remote checks" checkbox.

I already checked whether this is some general SSL/TLS issue. When checking connectivity via openssl s_client -connect <fqdn>:443 (Nginx) and the openssl s_client -connect <fqdn>:5000 (registry container) everything works, so it does not seem to be a SSL problem. Also no problems when using my browser.

Deployment information

Deployment method: I deploy using docker-compose on Ubuntu 20.04 LTS based on the docker-compose.yml from the examples. I only changed the volume locations on the host. This results in the following

version: "2"

services:
  portus:
    image: opensuse/portus:head
    environment:
      - PORTUS_MACHINE_FQDN_VALUE=${MACHINE_FQDN}

      # DB. The password for the database should definitely not be here. You are
      # probably better off with Docker Swarm secrets.
      - PORTUS_DB_HOST=db
      - PORTUS_DB_DATABASE=portus_production
      - PORTUS_DB_PASSWORD=${DATABASE_PASSWORD}
      - PORTUS_DB_POOL=5

      # Secrets. It can possibly be handled better with Swarm's secrets.
      - PORTUS_SECRET_KEY_BASE=${SECRET_KEY_BASE}
      - PORTUS_KEY_PATH=/certificates/portus.key
      - PORTUS_PASSWORD=${PORTUS_PASSWORD}

      # SSL
      - PORTUS_PUMA_TLS_KEY=/certificates/portus.key
      - PORTUS_PUMA_TLS_CERT=/certificates/portus.crt

      # NGinx is serving the assets instead of Puma. If you want to change this,
      # uncomment this line.
      #- RAILS_SERVE_STATIC_FILES='true'
    ports:
      - 3000:3000
    links:
      - db
    volumes:
      - ./secrets:/certificates:ro
      - static:/srv/Portus/public

  background:
    image: opensuse/portus:head
    depends_on:
      - portus
      - db
    environment:
      # Theoretically not needed, but cconfig's been buggy on this...
      - CCONFIG_PREFIX=PORTUS
      - PORTUS_MACHINE_FQDN_VALUE=${MACHINE_FQDN}

      # DB. The password for the database should definitely not be here. You are
      # probably better off with Docker Swarm secrets.
      - PORTUS_DB_HOST=db
      - PORTUS_DB_DATABASE=portus_production
      - PORTUS_DB_PASSWORD=${DATABASE_PASSWORD}
      - PORTUS_DB_POOL=5

      # Secrets. It can possibly be handled better with Swarm's secrets.
      - PORTUS_SECRET_KEY_BASE=${SECRET_KEY_BASE}
      - PORTUS_KEY_PATH=/certificates/portus.key
      - PORTUS_PASSWORD=${PORTUS_PASSWORD}

      - PORTUS_BACKGROUND=true
    links:
      - db
    volumes:
      - ./secrets:/certificates:ro

  db:
    image: library/mariadb:10.0.23
    command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci --init-connect='SET NAMES UTF8;' --innodb-flush-log-at-trx-commit=0
    environment:
      - MYSQL_DATABASE=portus_production

      # Again, the password shouldn't be handled like this.
      - MYSQL_ROOT_PASSWORD=${DATABASE_PASSWORD}
    volumes:
      - ./var/lib/portus/mariadb:/var/lib/mysql

  registry:
    image: library/registry:2.6
    command: ["/bin/sh", "/etc/docker/registry/init"]
    environment:
      # Authentication
      REGISTRY_AUTH_TOKEN_REALM: https://${MACHINE_FQDN}/v2/token
      REGISTRY_AUTH_TOKEN_SERVICE: ${MACHINE_FQDN}
      REGISTRY_AUTH_TOKEN_ISSUER: ${MACHINE_FQDN}
      REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /secrets/portus.crt

      # SSL
      REGISTRY_HTTP_TLS_CERTIFICATE: /secrets/portus.crt
      REGISTRY_HTTP_TLS_KEY: /secrets/portus.key

      # Portus endpoint
      REGISTRY_NOTIFICATIONS_ENDPOINTS: >
        - name: portus
          url: https://${MACHINE_FQDN}/v2/webhooks/events
          timeout: 2000ms
          threshold: 5
          backoff: 1s
    volumes:
      - ./var/lib/portus/registry:/var/lib/registry
      - ./secrets:/secrets:ro
      - ./registry/config.yml:/etc/docker/registry/config.yml:ro
      - ./registry/init:/etc/docker/registry/init:ro
    ports:
      - 5000:5000
      - 5001:5001 # required to access debug service
    links:
      - portus:portus

  nginx:
    image: library/nginx:alpine
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./secrets:/secrets:ro
      - static:/srv/Portus/public:ro
    ports:
      - 80:80
      - 443:443
    links:
      - registry:registry
      - portus:portus

volumes:
  static:
    driver: local

Configuration:

schema] Selected the schema for mysql
[Mailer config] Host:     portus.test.lan
[Mailer config] Protocol: https://
Evaluated configuration:
---
email:
  from: [email protected]
  name: Portus
  reply_to: ''
  smtp:
    enabled: false
    address: smtp.example.com
    port: 587
    domain: example.com
    ssl_tls: ''
    enable_starttls_auto: false
    openssl_verify_mode: none
    ca_path: ''
    ca_file: ''
    user_name: ''
    password: "****"
    authentication: login
gravatar:
  enabled: true
delete:
  enabled: true
  contributors: false
  garbage_collector:
    enabled: false
    older_than: 30
    keep_latest: 5
    tag: ''
ldap:
  enabled: false
  hostname: ldap_hostname
  port: 389
  timeout: 5
  encryption:
    method: ''
    options:
      ca_file: ''
      ssl_version: TLSv1_2
  base: ''
  admin_base: ''
  group_base: ''
  filter: ''
  uid: uid
  authentication:
    enabled: false
    bind_dn: ''
    password: "****"
  group_sync:
    enabled: true
    default_role: viewer
  guess_email:
    enabled: false
    attr: ''
oauth:
  local_login:
    enabled: true
  google_oauth2:
    enabled: false
    id: ''
    secret: ''
    domain: ''
    options:
      hd: ''
  open_id:
    enabled: false
    identifier: ''
    domain: ''
  openid_connect:
    enabled: false
    issuer: ''
    identifier: ''
    secret: ''
  github:
    enabled: false
    client_id: ''
    client_secret: ''
    organization: ''
    team: ''
    domain: ''
  gitlab:
    enabled: false
    application_id: ''
    secret: ''
    group: ''
    domain: ''
    server: ''
  bitbucket:
    enabled: false
    key: ''
    secret: ''
    domain: ''
    options:
      team: ''
first_user_admin:
  enabled: true
signup:
  enabled: true
check_ssl_usage:
  enabled: true
registry:
  jwt_expiration_time:
    value: 15
  catalog_page:
    value: 100
  timeout:
    value: 2
  read_timeout:
    value: 120
machine_fqdn:
  value: my.fqdn.org
display_name:
  enabled: false
user_permission:
  change_visibility:
    enabled: true
  create_team:
    enabled: true
  manage_team:
    enabled: true
  create_namespace:
    enabled: true
  manage_namespace:
    enabled: true
  create_webhook:
    enabled: true
  manage_webhook:
    enabled: true
  push_images:
    policy: allow-teams
security:
  clair:
    server: ''
    health_port: 6061
    timeout: 900
  zypper:
    server: ''
  dummy:
    server: ''
anonymous_browsing:
  enabled: true
background:
  registry:
    enabled: true
  sync:
    enabled: true
    strategy: initial
pagination:
  per_page: 10
  before_after: 2

Portus version: 2.5.0-dev@a1b9f2ebfeb84680a9dcd5629195e4c52815735c

(I replaced my real FQDN with a generic one.)

[stale]

Thanks for all your contributions!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.