Spring MANUAL_IMMEDIATE/MANUAL Acknolwedgement with retry

[vineela-gampa]

Hi

our app listens to message from an upstream order provider. With manula acknowledgement we are loosing messages incase of a DB Failure . Manual/Manual_IMMEDIATE will help with this. When we ack on a give message it acks that offset and all the messages till then. Is there a possibility of loosing messages with offset committed till then. Kakfa has ordering in a given partition but no ordering exists across partitions. If a commit happens on partition with offset 10 and other parition is processing 5th offset. will we loose 5 messages?

[garyrussell]

Committing an offset on one partition has no bearing on the offsets in other partitions.

You should generally not processes (and acknowledge) records out of order within a partitions, since Kafka only maintains a committed offsets.

The 2.8 release optionally supports out of order commits within a partition by deferring the later commits until the gaps are filled.

[vineela-gampa]

So a client needs to maintain the offsets of the message its processing for manual processing. how would messages with a larger offset be processed before a shorter offset ? For ex how can a. consumer process a message with offset 10 before processing offset 5 within a partion ? Kakfa sends 5 before sending 10 , is that right ?

[garyrussell]

You will receive records in order, as long as you don't perform asynchronous processing on the listener, they will be processed in order - 5 will always be processed before 10.

It is generally best practice to only process records within a partition in order. If you must process them out of order, for some reason, you need the new 2.8 feature, which defers the commits and ensures they are committed in order (with the increased risk of duplicates after a failure).

[vineela-gampa]

@garyrussell can you also share best practices for retry processing with a manual ack. Is there a way we can do the retry with a backoff/delay without creating several topics for increments?

[garyrussell]

Yes; the default error handler can be configured with a BackOff that determines delays between retries and the number of retries.

https://docs.spring.io/spring-kafka/docs/current/reference/html/#default-eh (2.8 and later)

https://docs.spring.io/spring-kafka/docs/2.7.x/reference/html/#seek-to-current (earlier supported versions)

[vineela-gampa]

Thanks @garyrussell . Will check and reach out for more questions on retry/error handling.