Skip to content

Latest commit

 

History

History
562 lines (425 loc) · 15.1 KB

README.md

File metadata and controls

562 lines (425 loc) · 15.1 KB
Raw

Artifactory cleanup

artifactory-cleanup is an extended and flexible cleanup tool for JFrog Artifactory.

The tool has simple YAML-defined cleanup configuration and can be extended with your own rules on Python. Everything must be as a code, even cleanup policies!

Tables of Contents

Installation

As simple as one command!

# docker
docker pull devopshq/artifactory-cleanup
docker run --rm devopshq/artifactory-cleanup artifactory-cleanup --help

# python (later we call it 'cli')
python3 -mpip install artifactory-cleanup
artifactory-cleanup --help

Usage

Suppose you want to remove all artifacts older than N days from reponame repository. You should take the following steps:

  1. Install artifactory-cleanup (see above)
  2. Create a configuration file artifactory-cleanup.yaml. variables.
# artifactory-cleanup.yaml
artifactory-cleanup:
  server: https://repo.example.com/artifactory
  # $VAR is auto populated from environment variables
  user: $ARTIFACTORY_USERNAME
  password: $ARTIFACTORY_PASSWORD

  policies:
    - name: Remove all files from repo-name-here older than 7 days
      rules:
        - rule: Repo
          name: "reponame"
        - rule: DeleteOlderThan
          days: 7
  1. Run the command TO SHOW (not remove) artifacts that will be deleted. By default artifactory-cleanup uses "dry mode".
# Set the credentials with delete permissions
export ARTIFACTORY_USERNAME=usernamehere
export ARTIFACTORY_PASSWORD=password

# docker
docker run --rm -v "$(pwd)":/app -e ARTIFACTORY_USERNAME -e ARTIFACTORY_PASSWORD devopshq/artifactory-cleanup artifactory-cleanup

# cli
artifactory-cleanup
  1. Verify that right artifacts will be removed and add --destroy flag TO REMOVE artifacts:
# docker
docker run --rm -v "$(pwd)":/app -e ARTIFACTORY_USERNAME -e ARTIFACTORY_PASSWORD devopshq/artifactory-cleanup artifactory-cleanup --destroy

# cli
artifactory-cleanup --destroy

Looking for more examples? Check examples folder!

Notes

# docker
docker pull devopshq/artifactory-cleanup:1.0.0
docker run --rm devopshq/artifactory-cleanup:1.0.0 artifactory-cleanup --version

# python (later we call it 'cli')
python3 -mpip install artifactory-cleanup==1.0.0
artifactory-cleanup --help
  • Use CI servers or cron-like utilities to run artifactory-cleanup every day (or every hour). TeamCity and GitHub have built-in support and show additional logs format
  • Do not save credentials in the configuration file, use environment variables.
  • Use --ignore-not-found flag to ignore errors when the repository is not found. It's useful when you have a configuration for multiple repositories and some of them are not found.
  • use --worker-count=<WORKER_NUM> to increase the number of workers. By default, it's 1. It's useful when you have a lot of artifacts and you want to speed up the process.

Commands

# Debug - "dry run" mode by default
# debug run - only print artifacts. it does not delete any artifacts
artifactory-cleanup

# Debug run only for policytestname.
artifactory-cleanup --policy-name policytestname

# REMOVE
# For remove artifacts use --destroy
artifactory-cleanup --destroy

# For remove artifacts use environment variable
export ARTIFACTORY_CLEANUP_DESTROY=True
artifactory-cleanup

# Specify config filename
artifactory-cleanup --config artifactory-cleanup.yaml

# Specify config filename using environment variable
export ARTIFACTORY_CLEANUP_CONFIG_FILE=artifactory-cleanup.yaml
artifactory-cleanup --config artifactory-cleanup.yaml

# Look in the future - shows what the tool WILL remove after 10 days
artifactory-cleanup --days-in-future=10

# Not satisfied with built-in rules? Write your own rules in python and connect them!
artifactory-cleanup --load-rules=myrule.py
docker run -v "$(pwd)":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py

# Save the table summary in a file
artifactory-cleanup --output=myfile.txt

# Save the summary in a Json file
artifactory-cleanup --output=myfile.txt --output-format=json

Rules

Common

  • Repo - Apply the rule to one repository. If no name is specified, it is taken from the rule name (in CleanupPolicy definition)
- rule: Repo
  name: reponame
# OR - if you have a single policy for the repo - you can name the policy as reponame
# Both configurations are equal
policies:
  - name: reponame
    rules:
      - rule: Repo
  • RepoList - Apply the policy to list of repositories.
- rule: RepoList
  repos:
    - repo1
    - repo2
    - repo3
  • RepoByMask - Apply rule to repositories matching by mask
- rule: RepoByMask
  mask: "*.banned"
  • PropertyEq- Delete repository artifacts only with a specific property value (property_key is the name of the parameter, property_value is the value)
- rule: PropertyEq
  property_key: key-name
  property_value: 1
  • PropertyNeq- Delete repository artifacts only if the value != specified. If there is no value, delete it anyway. Allows you to specify the deletion flag do_not_delete = 1
- rule: PropertyNeq
  property_key: key-name
  property_value: 1

Delete

  • DeleteOlderThan - deletes artifacts that are older than N days
- rule: DeleteOlderThan
  days: 1
  • DeleteWithoutDownloads - deletes artifacts that have never been downloaded (DownloadCount=0). Better to use with DeleteOlderThan rule
- rule: DeleteWithoutDownloads
  • DeleteOlderThanNDaysWithoutDownloads - deletes artifacts that are older than N days and have not been downloaded
- rule: DeleteOlderThanNDaysWithoutDownloads
  days: 1
  • DeleteNotUsedSince - delete artifacts that were downloaded, but for a long time. N days passed. Or not downloaded at all from the moment of creation and it's been N days
- rule: DeleteNotUsedSince
  days: 1
  • DeleteEmptyFolders - Clean up empty folders in given repository list
- rule: DeleteEmptyFolders
  • DeleteByRegexpName - delete artifacts whose name matches the specified regexp
- rule: DeleteByRegexpName
  regex_pattern: "\d"

Keep

  • KeepLatestNFiles - Leaves the last (by creation time) files in the amount of N pieces. WITHOUT accounting subfolders
- rule: KeepLatestNFiles
  count: 1
  • KeepLatestNFilesInFolder - Leaves the last (by creation time) files in the number of N pieces in each folder
- rule: KeepLatestNFilesInFolder
  count: 1
  • KeepLatestVersionNFilesInFolder - Leaves the latest N (by version) files in each folder. The definition of the version is using regexp. By default it parses semver using the regex - ([\d]+\.[\d]+\.[\d]+)")
- rule: KeepLatestVersionNFilesInFolder
  count: 1
  custom_regexp: "[^\\d][\\._]((\\d+\\.)+\\d+)"
  • KeepLatestNupkgNVersions - Leaves N nupkg (adds *.nupkg filter) in release feature builds
- rule: KeepLatestNupkgNVersions
  count: 1

Docker

  • DeleteDockerImagesOlderThan - Delete docker images that are older than N days
- rule: DeleteDockerImagesOlderThan
  days: 1
  • DeleteDockerImagesOlderThanNDaysWithoutDownloads - Deletes docker images that are older than N days and have not been downloaded
- rule: DeleteDockerImagesOlderThanNDaysWithoutDownloads
  days: 1
  • DeleteDockerImagesNotUsed - Removes Docker image not downloaded since N days
- rule: DeleteDockerImagesNotUsed
  days: 1
  • IncludeDockerImages - Apply to docker images with the specified names and tags
- rule: IncludeDockerImages
  masks: "*singlemask*"
- rule: IncludeDockerImages
  masks:
    - "*production*"
    - "*release*"
  • ExcludeDockerImages - Exclude Docker images by name and tags.
- rule: ExcludeDockerImages
  masks:
    - "*production*"
    - "*release*"
  • KeepLatestNVersionImagesByProperty(count=N, custom_regexp='some-regexp', number_of_digits_in_version=X) - Leaves N Docker images with the same major. (^\d+\.\d+\.\d+$) is the default regexp how to determine version which matches semver 1.1.1. If you need to add minor then set number_of_digits_in_version to 2 or if patch then set to 3 (by default we match major, which 1). Semver tags prefixed with v are supported by updating the regexp to include (an optional) v in the expression (e.g., (^v?\d+\.\d+\.\d+$)).
- rule: KeepLatestNVersionImagesByProperty
  count: 1
  custom_regexp: "[^\\d][\\._]((\\d+\\.)+\\d+)"

  • DeleteDockerImageIfNotContainedInProperties(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None) - Remove Docker image, if it is not found in the properties of the artifact repository.

  • DeleteDockerImageIfNotContainedInPropertiesValue(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None) - Remove Docker image, if it is not found in the properties of the artifact repository.

Filters

  • IncludePath - Apply to artifacts by path / mask.
- rule: IncludePath
  masks: "*production*"
- rule: IncludePath
  masks:
   - "*production*"
   - "*develop*"
  • IncludeFilename - Apply to artifacts by name/mask
- rule: IncludeFilename
  masks:
   - "*production*"
   - "*develop*"
  • ExcludePath - Exclude artifacts by path/mask
- rule: ExcludePath
  masks:
   - "*production*"
   - "*develop*"
  • ExcludeFilename - Exclude artifacts by name/mask
- rule: ExcludeFilename
  masks:
    - "*.tag.gz"
    - "*.zip"

Create your own rule

If you want to create your own rule, you can do it!

The basic flow how the tool calls Rules:

  1. Rule.check(*args, **kwargs) - verify that the Rule configured right. Call other services to get more information.
  2. Rule.aql_add_filter(filters) - add Artifactory Query Language expressions
  3. Rule.aql_add_text(aql) - add text to the result aql query
  4. artifactory-cleanup calls Artifactory with AQL and pass the result to the next step
  5. Rule.filter(artifacts) - filter out artifacts. The method returns artifacts that will be removed!.
    • To keep artifacts use artifacts.keep(artifact) method

Create myrule.py file at the same folder as artifactory-cleanup.yaml:

# myrule.py
from typing import List

from artifactory_cleanup import register
from artifactory_cleanup.rules import Rule, ArtifactsList


class MySimpleRule(Rule):
    """
    This doc string is used as rule title

    For more methods look at Rule source code
    """

    def __init__(self, my_param: str, value: int):
        self.my_param = my_param
        self.value = value

    def aql_add_filter(self, filters: List) -> List:
        print(f"Today is {self.today}")
        print(self.my_param)
        print(self.value)
        return filters

    def filter(self, artifacts: ArtifactsList) -> ArtifactsList:
        """I'm here just to print the list"""
        print(self.my_param)
        print(self.value)
        # You can make requests to artifactory by using self.session:
        # url = f"/api/storage/{self.repo}"
        # r = self.session.get(url)
        # r.raise_for_status()
        return artifacts


# Register your rule in the system
register(MySimpleRule)

Use rule: MySimpleRule in configuration:

# artifactory-cleanup.yaml
- rule: MySimpleRule
  my_param: "Hello, world!"
  value: 42

Specify --load-rules to the command:

# docker
docker run -v "$(pwd)":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py

# cli
artifactory-cleanup --load-rules=myrule.py

How to

How to connect self-signed certificates for docker?

In case you have set up your Artifactory self-signed certificates, place all certificates of the chain of trust into the certificates folder and add additional argument to the command:

docker run -v "$(pwd)":/app -v "$(pwd)/certificates":/mnt/self-signed-certs/ devopshq/artifactory-cleanup artifactory-cleanup

How to clean up Conan repository?

We can handle conan's metadata by creating two policies:

  1. First one removes files but keep all metadata.
  2. Second one look at folders and if it contains only medata files - removes it (because there's no associated with metadata files)

The idea came from devopshq#47

# artifactory-cleanup.yaml
artifactory-cleanup:
  server: https://repo.example.com/artifactory
  user: $ARTIFACTORY_USERNAME
  password: $ARTIFACTORY_PASSWORD

  policies:
    - name: Conan - delete files older than 60 days
      rules:
        - rule: Repo
          name: "conan-testing"
        - rule: DeleteNotUsedSince
          days: 60
        - rule: ExcludeFilename
          masks:
            - ".timestamp"
            - "index.json"
    - name: Conan - delete empty folders (to fix the index)
      rules:
        - rule: Repo
          name: "conan-testing"
        - rule: DeleteEmptyFolders
        - rule: ExcludeFilename
          masks:
            - ".timestamp"
            - "index.json"

How to keep latest N docker images?

We can combine docker rules with usual "files" rules!

The idea came from devopshq#61

# artifactory-cleanup.yaml
artifactory-cleanup:
  server: https://repo.example.com/artifactory
  user: $ARTIFACTORY_USERNAME
  password: $ARTIFACTORY_PASSWORD

  policies:
    - name: Remove docker images, but keep last 3
      rules:
        # Select repo
        - rule: Repo
          name: docker-demo
        # Delete docker images older than 30 days
        - rule: DeleteDockerImagesOlderThan
          days: 30
        # Keep these tags for all images
        - rule: ExcludeDockerImages
          masks:
            - "*:latest"
            - "*:release*"
        # Exclude these docker tags
        - rule: ExcludePath
          masks: "*base-tools*"
        # Keep 3 docker tags for all images
        - rule: KeepLatestNFilesInFolder
          count: 3

Release

In order to provide a new release of artifactory-cleanup, there are two steps involved.

  1. Bump the version in the setup.py
  2. Bump the version in the init.py
  3. Create a Git release tag (in format 1.0.1) by creating a release on GitHub