diff --git a/diagrams/backlog-quota-new.graffle b/diagrams/backlog-quota-new.graffle new file mode 100644 index 000000000000..ca7c967399be Binary files /dev/null and b/diagrams/backlog-quota-new.graffle differ diff --git a/diagrams/retention-new.graffle b/diagrams/retention-new.graffle new file mode 100644 index 000000000000..2193c0ba99e6 Binary files /dev/null and b/diagrams/retention-new.graffle differ diff --git a/docs/cookbooks-retention-expiry.md b/docs/cookbooks-retention-expiry.md index 67062b72582b..8d7a3ee96477 100644 --- a/docs/cookbooks-retention-expiry.md +++ b/docs/cookbooks-retention-expiry.md @@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*. The diagram below illustrates the concept of message retention. -![Concept of message retention in Pulsar.](/assets/retention.svg) +![Concept of message retention in Pulsar.](/assets/retention-new.svg) Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgments, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases. @@ -189,7 +189,7 @@ admin.namespaces().getRetention(namespace); You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Pulsar uses a quota to enforce a hard limit on the logical size of the backlogs in a topic. Backlog quota triggers an alert policy (for example, producer exception) once the quota limit is reached. The diagram below illustrates the concept of backlog quota. -![Concept of backlog quota in Pulsar](/assets/backlog-quota.svg) +![Concept of backlog quota in Pulsar](/assets/backlog-quota-new.svg) Setting a backlog quota involves setting: @@ -204,9 +204,9 @@ The following retention policies are available: | `producer_exception` | The producer throws an exception when trying to send a message | | `consumer_backlog_eviction` | The broker will begin discarding backlog messages | -> #### Beware the distinction between retention policy types -> As you may have noticed, there are two definitions of the term "retention policy" in Pulsar, one that applies to persistent storage of messages not in backlogs, and one that applies to messages within backlogs. - +:::note +The scope of retention includes the backlog, so retention must be greater than the backlog quota. Otherwise, Pulsar will throw an error when both are set. +::: Backlog quotas are handled at the namespace level. They can be managed via: diff --git a/static/assets/backlog-quota-new.svg b/static/assets/backlog-quota-new.svg new file mode 100644 index 000000000000..aac00dc87b62 --- /dev/null +++ b/static/assets/backlog-quota-new.svg @@ -0,0 +1,262 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Canvas 9 + + + Layer 1 + + + + + 0 + + + + + + + 1 + + + + + + + 2 + + + + + + + 3 + + + + + + + 4 + + + + + + + 5 + + + + + + + 6 + + + + + + + 7 + + + + + + + 8 + + + + + + + 9 + + + + + + + 10 + + + + + + + 11 + + + + + + + 12 + + + + + + + 13 + + + + + + + 14 + + + + + + + + + Backlog + (Unacked messages) + + + + + + + + + Ackd message + + + + + + + + + + Subsctiption + + + + + + + + + Retained policy + + + + + + + + + Will be deleted message + + + + + + + Producer + + + + + + Cursor + + + + + + + + + + + + + Acked messages within retention limit (not ready for deletion) + + + + + Acked messages over retention limit (ready for deletion) + + + + + + + A new + message + + + + + + + + + + + Assume a message is 1MB + Retention Policy: {size: 10MB Time: 0} + + + + + + + + + The new message is over the backlog quota limit. + Pulsar applies backlog retention policy here. + + + + + + + + + Unacked messages within + the backlog quota limit are + kept in the backlog + + + + + diff --git a/static/assets/retention-new.svg b/static/assets/retention-new.svg new file mode 100644 index 000000000000..d0fc70f96961 --- /dev/null +++ b/static/assets/retention-new.svg @@ -0,0 +1,233 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Canvas 8 + + + Layer 1 + + + + + 0 + + + + + + + 1 + + + + + + + 2 + + + + + + + 3 + + + + + + + 4 + + + + + + + 5 + + + + + + + 6 + + + + + + + 7 + + + + + + + 8 + + + + + + + 9 + + + + + + + 10 + + + + + + + 11 + + + + + + + 12 + + + + + + + 13 + + + + + + + 14 + + + + + + + + + Backlog + (Unacked messages) + + + + + + + + + Ackd message + + + + + + + + + + Subsctiption + + + + + + + + + Retained policy + + + + + + + + + Will be deleted message + + + + + + + Producer + + + + + + Cursor + + + + + + + + + + + + + Acked messages within retention limit (not ready for deletion) + + + + + Acked messages over retention limit (ready for deletion) + + + + + + + + + + + Assume a message is 1MB + Retention Policy: {size: 10MB Time: 0} + + + + + diff --git a/versioned_docs/version-3.0.x/cookbooks-retention-expiry.md b/versioned_docs/version-3.0.x/cookbooks-retention-expiry.md index 7c8a0216e3d7..48cc110af6d2 100644 --- a/versioned_docs/version-3.0.x/cookbooks-retention-expiry.md +++ b/versioned_docs/version-3.0.x/cookbooks-retention-expiry.md @@ -36,7 +36,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*. The diagram below illustrates the concept of message retention. -![](/assets/retention.svg) +![](/assets/retention-new.svg) Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgments, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases. @@ -188,7 +188,7 @@ admin.namespaces().getRetention(namespace); You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Pulsar uses a quota to enforce a hard limit on the logical size of the backlogs in a topic. Backlog quota triggers an alert policy (for example, producer exception) once the quota limit is reached. The diagram below illustrates the concept of backlog quota. -![](/assets/backlog-quota.svg) +![](/assets/backlog-quota-new.svg) Setting a backlog quota involves setting: @@ -203,10 +203,9 @@ Policy | Action `producer_exception` | The producer throws an exception when trying to send a message `consumer_backlog_eviction` | The broker will begin discarding backlog messages - -> #### Beware the distinction between retention policy types -> As you may have noticed, there are two definitions of the term "retention policy" in Pulsar, one that applies to persistent storage of messages not in backlogs, and one that applies to messages within backlogs. - +:::note +The scope of retention includes the backlog, so retention must be greater than the backlog quota. Otherwise, Pulsar will throw an error when both are set. +::: Backlog quotas are handled at the namespace level. They can be managed via: diff --git a/versioned_docs/version-3.3.x/cookbooks-retention-expiry.md b/versioned_docs/version-3.3.x/cookbooks-retention-expiry.md index df46ab3e05fd..b0ad2535aec5 100644 --- a/versioned_docs/version-3.3.x/cookbooks-retention-expiry.md +++ b/versioned_docs/version-3.3.x/cookbooks-retention-expiry.md @@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*. The diagram below illustrates the concept of message retention. -![Concept of message retention in Pulsar.](/assets/retention.svg) +![Concept of message retention in Pulsar.](/assets/retention-new.svg) Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgments, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases. @@ -189,7 +189,7 @@ admin.namespaces().getRetention(namespace); You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Pulsar uses a quota to enforce a hard limit on the logical size of the backlogs in a topic. Backlog quota triggers an alert policy (for example, producer exception) once the quota limit is reached. The diagram below illustrates the concept of backlog quota. -![Concept of backlog quota in Pulsar](/assets/backlog-quota.svg) +![Concept of backlog quota in Pulsar](/assets/backlog-quota-new.svg) Setting a backlog quota involves setting: @@ -204,9 +204,9 @@ The following retention policies are available: | `producer_exception` | The producer throws an exception when trying to send a message | | `consumer_backlog_eviction` | The broker will begin discarding backlog messages | -> #### Beware the distinction between retention policy types -> As you may have noticed, there are two definitions of the term "retention policy" in Pulsar, one that applies to persistent storage of messages not in backlogs, and one that applies to messages within backlogs. - +:::note +The scope of retention includes the backlog, so retention must be greater than the backlog quota. Otherwise, Pulsar will throw an error when both are set. +::: Backlog quotas are handled at the namespace level. They can be managed via: diff --git a/versioned_docs/version-4.0.x/cookbooks-retention-expiry.md b/versioned_docs/version-4.0.x/cookbooks-retention-expiry.md index 67062b72582b..8d7a3ee96477 100644 --- a/versioned_docs/version-4.0.x/cookbooks-retention-expiry.md +++ b/versioned_docs/version-4.0.x/cookbooks-retention-expiry.md @@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*. The diagram below illustrates the concept of message retention. -![Concept of message retention in Pulsar.](/assets/retention.svg) +![Concept of message retention in Pulsar.](/assets/retention-new.svg) Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgments, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases. @@ -189,7 +189,7 @@ admin.namespaces().getRetention(namespace); You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Pulsar uses a quota to enforce a hard limit on the logical size of the backlogs in a topic. Backlog quota triggers an alert policy (for example, producer exception) once the quota limit is reached. The diagram below illustrates the concept of backlog quota. -![Concept of backlog quota in Pulsar](/assets/backlog-quota.svg) +![Concept of backlog quota in Pulsar](/assets/backlog-quota-new.svg) Setting a backlog quota involves setting: @@ -204,9 +204,9 @@ The following retention policies are available: | `producer_exception` | The producer throws an exception when trying to send a message | | `consumer_backlog_eviction` | The broker will begin discarding backlog messages | -> #### Beware the distinction between retention policy types -> As you may have noticed, there are two definitions of the term "retention policy" in Pulsar, one that applies to persistent storage of messages not in backlogs, and one that applies to messages within backlogs. - +:::note +The scope of retention includes the backlog, so retention must be greater than the backlog quota. Otherwise, Pulsar will throw an error when both are set. +::: Backlog quotas are handled at the namespace level. They can be managed via: