Why Kafka Idempotency is not enabled in Firehose

[mittulmadaan]

Currently, our Firehose Kafka integration based on KafkaJS does not provide guarantees about idempotency.

On further discussion, we identified certain challenges (or rare cases) for us in supporting idempotency in Kafka publishing:

• While Kafka provides idempotency, there should be a maximum of one publish operation in-flight to Kafka, imposing a rate limit on publishing into Kafka.
• More fragility, as specific circumstances may arise. For instance, if a message is published, it must have a strictly and uniformly increasing identifier. Consequently, if one message fails to be delivered for any reason, further messages cannot be published because Kafka only accepts the next message if it bears the subsequent identifier.
• Rarely there could be scenarios where disruptions occur. If one of our instances is terminated due to system fault tolerance and recovery measures are initiated for retrying the message with the correct unique serial identifier for Kafka – this task essentially becomes quite challenging for us to handle.

@paddybyers - could you please help us prepare a public-facing document on this subject, so we can add it in our docs?

┆Issue is synchronized with this Jira Task by Unito