Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added documentation for DeletionPropagationPolicy for cleanupPolicy a… #1426

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions content/en/docs/writing-policies/cleanup.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,50 @@ rules:
- delete
```

### Deletion Propagation Policy

The `deletionPropagationPolicy` field is an optional setting in `CleanupPolicy` and `ClusterCleanupPolicy` that specifies how Kubernetes should handle the deletion of dependent resources. The available options are:

- **Foreground**: Deletes the resource and waits until all of its dependent resources are also deleted.
- **Background**: Deletes the resource immediately, and its dependents are deleted asynchronously.
- **Orphan**: Deletes the resource without deleting its dependents, leaving them orphaned.

> **Note**: If `deletionPropagationPolicy` is not set, Kyverno defaults to the API server’s behavior. This default allows Kyverno to delete the primary resource asynchronously, giving the API server the flexibility to manage the deletion of dependents as per cluster settings.

An example `ClusterCleanupPolicy` with `deletionPropagationPolicy` is shown below. This cleanup policy removes Deployments with the label `canremove: "true"` if they have fewer than two replicas, on a schedule of every 5 minutes, and deletes dependents in the **Foreground** mode.

```yaml
# ClusterCleanupPolicy with deletionPropagationPolicy
apiVersion: kyverno.io/v2
kind: ClusterCleanupPolicy
metadata:
name: cleandeploy
spec:
match:
any:
- resources:
kinds:
- Deployment
selector:
matchLabels:
canremove: "true"
conditions:
any:
- key: "{{ target.spec.replicas }}"
operator: LessThan
value: 2
schedule: "*/5 * * * *"
deletionPropagationPolicy: "Foreground"
```

{{% alert title="Note" color="info" %}} Since cleanup policies always operate against existing resources in a cluster, policies created with subjects, Roles, or ClusterRoles in the match/exclude block are not allowed since this information is only known at admission time. Additionally, operations[], while permitted, are ignored as the only trigger is schedule based. {{% /alert %}}

Values from resources to be evaluated during a policy may be referenced with target.* similar to mutate existing rules.

Because Kyverno follows the principle of least privilege, depending on the resources you wish to remove it may be necessary to grant additional permissions to the cleanup controller. Kyverno will assist in informing you if additional permissions are required by validating them at the time a new cleanup policy is installed. See the Customizing Permissions section for more details.

An example ClusterRole which allows Kyverno to cleanup Pods is shown below. This may need to be customized based on the values used to deploy Kyverno.

## Cleanup Label

In addition to policies which can declaratively define what resources to remove and when to remove them, the second option for cleanup involves assignment of a reserved label called `cleanup.kyverno.io/ttl` to the exact resource(s) which should be removed. The value of this label can be one of two supported formats. Any unrecognized formats will trigger a warning.
Expand Down Expand Up @@ -96,3 +140,35 @@ spec:
Although labeled resources are watched by Kyverno, the cleanup interval (the time resolution at which any cleanup can be performed) is controlled by a flag passed to the cleanup controller called `ttlReconciliationInterval`. This value is set to `1m` by default and can be changed if a longer resolution is required.

Because this is a label, there is opportunity to chain other Kyverno functionality around it. For example, it is possible to use a Kyverno mutate rule to assign this label to matching resources. A validate rule could be written prohibiting, for example, users from the `infra-ops` group from assigning the label to resources in certain Namespaces. Or, Kyverno could generate a new resource with this label as part of the resource definition.

### Deletion Propagation Policy in TTL-based Cleanup

The deletionPropagationPolicy can also be specified for resources with a TTL-based cleanup label. This field provides flexibility in handling dependent resources when a resource reaches its expiration. For example:

- **Foreground**: Ensures that all dependent resources are deleted before the resource itself is removed.
- **Background**: Deletes the resource first, while dependents are removed asynchronously.
- **Orphan**: Deletes the resource but leaves its dependents in place.

For example, consider a Pod with the TTL label cleanup.kyverno.io/ttl: 2m. After two minutes, the Pod will be deleted, but the dependents will be handled according to the specified deletionPropagationPolicy. If the policy is set to Orphan, the Pod will be deleted, but its dependent resources will remain in the cluster.

```yaml
# TTL-based cleanup with deletionPropagationPolicy
apiVersion: v1
kind: Pod
metadata:
labels:
cleanup.kyverno.io/ttl: 2m
name: foo
spec:
containers:
- args:
- sleep
- 1d
image: busybox:1.35
name: foo
deletionPropagationPolicy: "Orphan"
```

In this example, the TTL is set to 2m, which removes the Pod after two minutes but leaves any dependent resources because of the Orphan policy.

Although labeled resources are watched by Kyverno, the cleanup interval (the time resolution at which any cleanup can be performed) is controlled by a flag passed to the cleanup controller called ttlReconciliationInterval. This value is set to 1m by default and can be changed if a longer resolution is required.