Using retry_topic-backoff-timestamp as a timestamp of retry publishing and not as a "retryAt" timestamp

[ro0sterjam]

Hi, if we have multiple instances of a consumer live, where some may (temporarily) have a different retry configuration from one another during traffic shifting (e.g. old consumers have exponential backoff of 1.5x, and new have 2.5x), then for that time window, the configured retry topics will have a mix of both the old and new messages.

From my understanding, this breaks the existing retry delay mechanism because from what I can tell, retry_topic-backoff-timestamp is the timestamp at which the next retry should be performed. If the old messages are scheduled, let's say, to be processed in 1 day, but the new messages (due to the new configurations) are scheduled to be processed in the next hour, then the new messages will not be able to be processed until the next day, due to the ordered nature of the kafka topics.

I wonder then, why not, instead of using retry_topic-backoff-timestamp to indicate when the next retry should be performed, the framework would instead use retry_topic-backoff-timestamp (or some other header) to indicate the timestamp in which the retry message was published, and have the active consumer (that is processing the given message) calculate the delay on the fly, using the active configuration for that consumer? In my previous example, what that would mean is that any new consumers in the wild processing the old messages would process them in the next hour rather (minus already elapsed time), rather than the originally scheduled next day. I feel as though this would be a more desirable outcome, especially if the developer's intent is to either slow down or speed up the existing messages on the retry topic.

[garyrussell]

Interesting idea; of course it would only work if the retry index naming strategy is used (rather than the delay time naming strategy, in which case, the records would be in different topics anyway.

Contributions are welcome!

[ro0sterjam]

Interesting, so if I understand it, if the delay time naming strategy is used, then the new messages with the updated configurations would essentially be published to a different retry topic anyway, so the concerns I laid out would not actually be applicable?

In that case, I do have a side question which is, if a configuration update is made, and a specific "delay time naming strategy" topic is no longer a valid retry topic to be published to, does the consumer still listen to those old topics? Or will the unprocessed messages remaining on those topics be essentially abandoned?

[garyrussell]

The framework only knows about the current configuration so, no, it won't listen to the old topics.