Garbage collection running wild

[danielbecker]

Description

Since old tags for our images were piling up, I enabled garbage collection to run with keep_latest: 10 and older_than: 90. From the documentation, I assume this means remove all tags but the last 10 and don't consider any tag younger than 90 days.

However, what happened was a ton of old, admin and template images being removed entirely. The images weren't updated recently, but frequently pulled, as we need them in our daily deployment process.

Steps to reproduce

1. Enable garbage collection:

• PORTUS_DELETE_GARBAGE_COLLECTOR_ENABLED=true
• PORTUS_DELETE_GARBAGE_COLLECTOR_OLDER_THAN=90
• PORTUS_DELETE_GARBAGE_COLLECTOR_KEEP_LATEST=10

2. Restart portus
   docker-compose up -d
3. Watch your images diappear

• Expected behavior: Remove tags older than 90 days and keep at least 10 around
• Actual behavior: Entire images were removed

Providing logs of the moment when the issue has happened would also be
useful. If you are in production, you might want to set the PORTUS_LOG_LEVEL
to debug to get a more verbose log.

Deployment information

Deployment method: docker-compose.yml

version: "3.7"

services:
portus:
image: opensuse/portus:2.4
restart: unless-stopped
environment:
- PORTUS_MACHINE_FQDN_VALUE=xxx

  # DB. The password for the database should definitely not be here. You are
  # probably better off with Docker Swarm secrets.
  - PORTUS_DB_HOST=db
  - PORTUS_DB_DATABASE=xxx
  - PORTUS_DB_PASSWORD=xxx
  - PORTUS_DB_POOL=5

  # Secrets. It can possibly be handled better with Swarm's secrets.
  - PORTUS_SECRET_KEY_BASE=xxx
  - PORTUS_KEY_PATH=/certificates/2019_key.pem
  - PORTUS_PASSWORD=xxx

  # SSL
  - PORTUS_PUMA_TLS_KEY=/certificates/2019_key.pem
  - PORTUS_PUMA_TLS_CERT=/certificates/2019_cert.pem

  - RAILS_SERVE_STATIC_FILES=false
ports:
  - 127.0.0.1:8063:3000
links:
  - db
volumes:
  - /srv/xxx/portus/certs:/certificates:ro
  - /srv/xxx/portus/assets:/srv/Portus/public
extra_hosts:
  - "xxx:192.168.252.113"
logging:
  driver: "gelf"
  options:
    gelf-address: "tcp://xxx:5555"
    tag: "portus-frontend"
    mode: non-blocking
    max-buffer-size: 1g

background:
image: opensuse/portus:2.4
restart: unless-stopped
environment:
- CCONFIG_PREFIX=PORTUS
- PORTUS_MACHINE_FQDN_VALUE=xxx

  # DB. The password for the database should definitely not be here. You are
  # probably better off with Docker Swarm secrets.
  - PORTUS_DB_HOST=db
  - PORTUS_DB_DATABASE=xxx
  - PORTUS_DB_PASSWORD=xxx
  - PORTUS_DB_POOL=5

  # Secrets. It can possibly be handled better with Swarm's secrets.
  - PORTUS_SECRET_KEY_BASE=xxx
  - PORTUS_KEY_PATH=/certificates/2019_key.pem
  - PORTUS_PASSWORD=xxx

  # Enable garbage collection
  - PORTUS_DELETE_GARBAGE_COLLECTOR_ENABLED=true
  - PORTUS_DELETE_GARBAGE_COLLECTOR_OLDER_THAN=90
  - PORTUS_DELETE_GARBAGE_COLLECTOR_KEEP_LATEST=10

  - PORTUS_BACKGROUND=true
links:
  - db
volumes:
  - /srv/xxx/portus/certs:/certificates:ro
extra_hosts:
  - "xxx:192.168.252.113"
logging:
  driver: "gelf"
  options:
    gelf-address: "tcp://xxx:5555"
    tag: "portus-backend"
    mode: non-blocking
    max-buffer-size: 1g

db:
image: library/mariadb:10.0.23
restart: unless-stopped
command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci --init-connect='SET NAMES UTF8;' --innodb-flush-log-at-trx-commit=0
environment:
- MYSQL_DATABASE=xxx
- MYSQL_ROOT_PASSWORD=xxx
volumes:
- /srv/xxx/portus/database:/var/lib/mysql

registry:
image: library/registry:2.6
restart: unless-stopped
command: ["/bin/sh", "/etc/docker/registry/init"]
environment:
# Authentication
REGISTRY_AUTH_TOKEN_REALM: https://xxx/v2/token
REGISTRY_AUTH_TOKEN_SERVICE: xxx
REGISTRY_AUTH_TOKEN_ISSUER: xxx
REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /secrets/2019_bundle.pem

  # SSL
  REGISTRY_HTTP_TLS_CERTIFICATE: /secrets/2019_cert.pem
  REGISTRY_HTTP_TLS_KEY: /secrets/2019_key.pem

  # Portus endpoint
  REGISTRY_NOTIFICATIONS_ENDPOINTS: >
    - name: portus
      url: https://xxx/v2/webhooks/events
      timeout: 2000ms
      threshold: 5
      backoff: 1s
volumes:
  - /srv/xxx/portus/registry/storage:/var/lib/registry
  - /srv/xxx/portus/certs:/secrets:ro
  - /srv/xxx/portus/registry/config/config.yml:/etc/docker/registry/config.yml:ro
  - /srv/xxx/portus/registry/init/init:/etc/docker/registry/init:ro
ports:
  - 127.0.0.1:xxx:5000
  - 127.0.0.1:xxx:5001 # required to access debug service
links:
  - portus:portus
logging:
  driver: "gelf"
  options:
    gelf-address: "tcp://xxx:5555"
    tag: "portus-registry"
    mode: non-blocking
    max-buffer-size: 1g

Configuration:

Please note that I had to turn off garbage collection for now to stop portus from causing more damage.

[Mailer config] Host: portus.test.lan
[Mailer config] Protocol: https://
Evaluated configuration:

email:
from: [email protected]
name: Portus
reply_to: ''
smtp:
enabled: false
address: smtp.example.com
port: 587
domain: example.com
ssl_tls: ''
enable_starttls_auto: false
openssl_verify_mode: none
ca_path: ''
ca_file: ''
user_name: ''
password: ""
authentication: login
gravatar:
enabled: true
delete:
enabled: true
contributors: false
garbage_collector:
enabled: false
older_than: 30
tag: ''
ldap:
enabled: false
hostname: ldap_hostname
port: 389
timeout: 5
encryption:
method: ''
options:
ca_file: ''
ssl_version: TLSv1_2
base: ''
filter: ''
uid: uid
authentication:
enabled: false
bind_dn: ''
password: ""
guess_email:
enabled: false
attr: ''
oauth:
local_login:
enabled: true
google_oauth2:
enabled: false
id: ''
secret: ''
domain: ''
options:
hd: ''
open_id:
enabled: false
identifier: ''
domain: ''
github:
enabled: false
client_id: ''
client_secret: ''
organization: ''
team: ''
domain: ''
gitlab:
enabled: false
application_id: ''
secret: ''
group: ''
domain: ''
server: ''
bitbucket:
enabled: false
key: ''
secret: ''
domain: ''
options:
team: ''
first_user_admin:
enabled: true
signup:
enabled: true
check_ssl_usage:
enabled: true
registry:
jwt_expiration_time:
value: 15
catalog_page:
value: 100
timeout:
value: 2
read_timeout:
value: 120
machine_fqdn:
value: portus.easybell.de
display_name:
enabled: false
user_permission:
change_visibility:
enabled: true
create_team:
enabled: true
manage_team:
enabled: true
create_namespace:
enabled: true
manage_namespace:
enabled: true
create_webhook:
enabled: true
manage_webhook:
enabled: true
push_images:
policy: allow-teams
security:
clair:
server: ''
health_port: 6061
timeout: 900
zypper:
server: ''
dummy:
server: ''
anonymous_browsing:
enabled: true
background:
registry:
enabled: true
sync:
enabled: true
strategy: initial
pagination:
limit: 10
per_page: 10
before_after: 2

Portus version: 2.4.3@5a616c0ef860567df5700708256f42505cdb9952

version: 0.1
storage:
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true
http:
  addr: 0.0.0.0:5000
  debug:
    addr: 0.0.0.0:5001

Portus version: 2.4.3@5a616c0ef860567df5700708256f42505cdb9952

[Bonko]

As far as i know this feature was planned for Portus 2.5: #2051

Sadly the documentation refers to master instead of current stable

[stale]

Thanks for all your contributions!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.