Diff offset in rollback transactions and other cases

[alograg]

Hi everyoune!!!

After the report, review of possible fixes, and interaction with the Kafka community, this lag is documented in other processes as an expected behavior. The issue was reported in version 2.8.11 with any version of Kafka. I haven't tested the latest versions of Spring-Kafka yet, but I'd like to hear your opinion on this matter regarding the latest versions.

I have the following questions:

• Is this a documented case within Kafka, as mentioned in the Kafka ticket? It seems more like a flawed implementation. Should we provide more documentation that this lag is reproduced 100% of the time and list the scenarios?
• Is it worth attempting a fix since it is already a known issue, and there are plans for a Consumer reconstruction?
• If we decide to pursue a fix, how can we listen for all cases where a transaction triggers a rollback to immediately correct the offset? Do you think this is feasible?

In any case, I would like to know your point of view and opinion on this situation.

Looking forward to your feedback and insights.

Best regards,

[garyrussell]

I don't understand why this is so hard to fix on the broker.

The broker clearly knows what the latest stable offset is in the topic, because it doesn't send any records (including transaction control records) to the consumer, yet the consumer metrics reports this as a lag. It shouldn't be difficult to change the lag calculation to use the latest stable offset, instead of the end offset.