Non Blocking Retry Support for KafkaStreams

[vararo27]

Is there support for non blocking retry functionality for KafkaStreams in Spring Kafka? I see details mentioned in this link related to setting it up for Kafka Listener.

In case non blocking retry works with Kafka Stream, kindly guide with documentation or example.

[sobychacko]

Hi, currently, we do not have any out-of-the-box support for non-blocking retryable topics as available in the regular Kafka clients. The closest that comes to my mind is a recently introduced DltAwareProcessor in the Kafka Streams binder for Spring Cloud Stream. See more details in this section: https://docs.spring.io/spring-cloud-stream/reference/kafka/kafka-streams-binder/error-handling.html#runtime-error-handling

Here is a sample app for this feature: https://github.com/spring-cloud/spring-cloud-stream-samples/tree/main/kafka-streams-samples/kafka-streams-recoverable

When you get an error, you can decide to publish to a DLT. The processor code will be invoked asynchronously.
We can consider moving these components out of the binder and migrating to Spring for Apache Kafka.

[StevenPG]

I agree with @sobychacko regarding usage of the DltAwareProcessor/RecordRecoverableProcessor.

They (DltAwareProcessor or RecordRecoverableProcessor) can be used to recreate the retry topic workflow that is available in Spring Kafka.

It doesn't come with all of the additional features, but you lose Kafka's ordering guarantee with both workflows either way.

A retry topic stream might have it's own stream configuration. In that stream is where components like Backoff delay, specific DLT strategies, etc. can be implemented.

[sobychacko]

It might be a reasonable idea to migrate this as a Spring Kafka-level feature so that both the Spring Kafka and Spring Cloud Stream users can benefit from this. Currently, only the Spring Cloud Stream users can use this feature.

[StevenPG]

If it was migrated to Spring Kafka, I wonder if it would need to be altered to exist within an existing hierarchy and integrate with the optional configurations. E.g.FailedRecordProcessor or if it could exist on its own.

[sobychacko]

I am sure some re-works will be needed, but we will only move something away from the binder side once we have a new major version. I suggest migrating the core API to Spring Kafka while leaving the binder-specific components in SCSt. I know we need to think through this further, though.