You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is it possible to set up a consumer that fetches messages from a set of topics using a wildcard (for example public/default/jobs-.*), where the messages are pulled in a round-robin manner from all matching topics?
The problem I'm trying to solve is that we have a bunch of customers who periodically queue up jobs on their own customer-specific topic (public/default/jobs-customer1, public/default/jobs-customer2 etc), and our servers processes them. Sometime a customer can queue up a couple 100k jobs, which should not block all other customers who queue up jobs after. The number of active customers vary a lot, so a hard-coded list in the consumers would not be feasible. We also require low response times, so periodically re-loading the consumer to include new topics would not work well, as we would need to do that at least once every second.
Is there a pattern we can use that solves this, or is this simply not possible to do with Pulsar?
Is there for example a way of creating a pulsar function that makes sure there are at least x messages in a topic, populating it in a round-robin manner from all topics matching a regex? There is no need for a completely fair system, just one that doesn't let one (or a few) users block the rest
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Is it possible to set up a consumer that fetches messages from a set of topics using a wildcard (for example
public/default/jobs-.*
), where the messages are pulled in a round-robin manner from all matching topics?The problem I'm trying to solve is that we have a bunch of customers who periodically queue up jobs on their own customer-specific topic (
public/default/jobs-customer1
,public/default/jobs-customer2
etc), and our servers processes them. Sometime a customer can queue up a couple 100k jobs, which should not block all other customers who queue up jobs after. The number of active customers vary a lot, so a hard-coded list in the consumers would not be feasible. We also require low response times, so periodically re-loading the consumer to include new topics would not work well, as we would need to do that at least once every second.Is there a pattern we can use that solves this, or is this simply not possible to do with Pulsar?
Is there for example a way of creating a pulsar function that makes sure there are at least x messages in a topic, populating it in a round-robin manner from all topics matching a regex? There is no need for a completely fair system, just one that doesn't let one (or a few) users block the rest
Beta Was this translation helpful? Give feedback.
All reactions