Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix retention and backlog quota diagram #973

Merged
merged 3 commits into from
Nov 12, 2024
Merged

Fix retention and backlog quota diagram #973

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
57b1c6e
Fix retention and backlog quota diagram
shibd Nov 6, 2024
11ab53c
Address code review
shibd Nov 6, 2024
811f573
Add source diagrams file
shibd Nov 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added diagrams/backlog-quota-new.graffle
View file
Open in desktop
Binary file not shown.
Binary file added diagrams/retention-new.graffle
View file
Open in desktop
Binary file not shown.
10 changes: 5 additions & 5 deletions docs/cookbooks-retention-expiry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.

The diagram below illustrates the concept of message retention.
![Concept of message retention in Pulsar.](/assets/retention.svg)
![Concept of message retention in Pulsar.](/assets/retention-new.svg)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just wondering why isn't the file retention.svg simply updated? If this diagram update applies to all versions of Pulsar, there shouldn't be a need to use a new file name.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since I haven't modified the references for older versions, like 2.10, I've kept the original files.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lhotari WHYT?

Do we need to update all versions of the documentation to point to the new diagrams? This way, I can remove the original diagrams.


Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgments, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.

Expand Down Expand Up @@ -189,7 +189,7 @@ admin.namespaces().getRetention(namespace);
You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Pulsar uses a quota to enforce a hard limit on the logical size of the backlogs in a topic. Backlog quota triggers an alert policy (for example, producer exception) once the quota limit is reached.

The diagram below illustrates the concept of backlog quota.
![Concept of backlog quota in Pulsar](/assets/backlog-quota.svg)
![Concept of backlog quota in Pulsar](/assets/backlog-quota-new.svg)

Setting a backlog quota involves setting:

Expand All @@ -204,9 +204,9 @@ The following retention policies are available:
| `producer_exception` | The producer throws an exception when trying to send a message |
| `consumer_backlog_eviction` | The broker will begin discarding backlog messages |

> #### Beware the distinction between retention policy types
> As you may have noticed, there are two definitions of the term "retention policy" in Pulsar, one that applies to persistent storage of messages not in backlogs, and one that applies to messages within backlogs.

:::note
The scope of retention includes the backlog, so retention must be greater than the backlog quota. Otherwise, Pulsar will throw an error when both are set.
:::

Backlog quotas are handled at the namespace level. They can be managed via:

Expand Down
262 changes: 262 additions & 0 deletions static/assets/backlog-quota-new.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
233 changes: 233 additions & 0 deletions static/assets/retention-new.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
11 changes: 5 additions & 6 deletions versioned_docs/version-3.0.x/cookbooks-retention-expiry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.

The diagram below illustrates the concept of message retention.
![](/assets/retention.svg)
![](/assets/retention-new.svg)

Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgments, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.

Expand Down Expand Up @@ -188,7 +188,7 @@ admin.namespaces().getRetention(namespace);
You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Pulsar uses a quota to enforce a hard limit on the logical size of the backlogs in a topic. Backlog quota triggers an alert policy (for example, producer exception) once the quota limit is reached.

The diagram below illustrates the concept of backlog quota.
![](/assets/backlog-quota.svg)
![](/assets/backlog-quota-new.svg)

Setting a backlog quota involves setting:

Expand All @@ -203,10 +203,9 @@ Policy | Action
`producer_exception` | The producer throws an exception when trying to send a message
`consumer_backlog_eviction` | The broker will begin discarding backlog messages


> #### Beware the distinction between retention policy types
> As you may have noticed, there are two definitions of the term "retention policy" in Pulsar, one that applies to persistent storage of messages not in backlogs, and one that applies to messages within backlogs.

:::note
The scope of retention includes the backlog, so retention must be greater than the backlog quota. Otherwise, Pulsar will throw an error when both are set.
:::

Backlog quotas are handled at the namespace level. They can be managed via:

Expand Down
10 changes: 5 additions & 5 deletions versioned_docs/version-3.3.x/cookbooks-retention-expiry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.

The diagram below illustrates the concept of message retention.
![Concept of message retention in Pulsar.](/assets/retention.svg)
![Concept of message retention in Pulsar.](/assets/retention-new.svg)

Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgments, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.

Expand Down Expand Up @@ -189,7 +189,7 @@ admin.namespaces().getRetention(namespace);
You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Pulsar uses a quota to enforce a hard limit on the logical size of the backlogs in a topic. Backlog quota triggers an alert policy (for example, producer exception) once the quota limit is reached.

The diagram below illustrates the concept of backlog quota.
![Concept of backlog quota in Pulsar](/assets/backlog-quota.svg)
![Concept of backlog quota in Pulsar](/assets/backlog-quota-new.svg)

Setting a backlog quota involves setting:

Expand All @@ -204,9 +204,9 @@ The following retention policies are available:
| `producer_exception` | The producer throws an exception when trying to send a message |
| `consumer_backlog_eviction` | The broker will begin discarding backlog messages |

> #### Beware the distinction between retention policy types
> As you may have noticed, there are two definitions of the term "retention policy" in Pulsar, one that applies to persistent storage of messages not in backlogs, and one that applies to messages within backlogs.

:::note
The scope of retention includes the backlog, so retention must be greater than the backlog quota. Otherwise, Pulsar will throw an error when both are set.
:::

Backlog quotas are handled at the namespace level. They can be managed via:

Expand Down
10 changes: 5 additions & 5 deletions versioned_docs/version-4.0.x/cookbooks-retention-expiry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.

The diagram below illustrates the concept of message retention.
![Concept of message retention in Pulsar.](/assets/retention.svg)
![Concept of message retention in Pulsar.](/assets/retention-new.svg)

Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgments, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.

Expand Down Expand Up @@ -189,7 +189,7 @@ admin.namespaces().getRetention(namespace);
You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Pulsar uses a quota to enforce a hard limit on the logical size of the backlogs in a topic. Backlog quota triggers an alert policy (for example, producer exception) once the quota limit is reached.

The diagram below illustrates the concept of backlog quota.
![Concept of backlog quota in Pulsar](/assets/backlog-quota.svg)
![Concept of backlog quota in Pulsar](/assets/backlog-quota-new.svg)

Setting a backlog quota involves setting:

Expand All @@ -204,9 +204,9 @@ The following retention policies are available:
| `producer_exception` | The producer throws an exception when trying to send a message |
| `consumer_backlog_eviction` | The broker will begin discarding backlog messages |

> #### Beware the distinction between retention policy types
> As you may have noticed, there are two definitions of the term "retention policy" in Pulsar, one that applies to persistent storage of messages not in backlogs, and one that applies to messages within backlogs.

:::note
The scope of retention includes the backlog, so retention must be greater than the backlog quota. Otherwise, Pulsar will throw an error when both are set.
:::

Backlog quotas are handled at the namespace level. They can be managed via:

Expand Down