transactional.id per topic/partition

[garyrussell]

https://www.confluent.io/blog/transactions-apache-kafka/

The key to fencing out zombies properly is to ensure that the input topics and partitions in the read-process-write cycle is always the same for a given transactional.id. If this isn’t true, then it is possible for some messages to leak through the fencing provided by transactions.

For instance, in a distributed stream processing application, suppose topic-partition tp0 was originally processed by transactional.id T0. If, at some point later, it could be mapped to another producer with transactional.id T1, there would be no fencing between T0 and T1. So it is possible for messages from tp0 to be reprocessed, violating the exactly once processing guarantee.

Practically, one would either have to store the mapping between input partitions and transactional.ids in an external store, or have some static encoding of it. Kafka Streams opts for the latter approach to solve this problem.

[Rajh]

Hi,

I was just going to write an issue about that here :)
I'm using kafka 1.3.6.RELEASE and I'm having a headache trying to understand and implement transaction in a read-process-write cycle.

Having several instance of my application running, when a consumer partition rebalance occur, the transaction are not preventing me from getting duplicates.

The transaction id should be computed with topic-name.group-id.partition-id when using it in a listener container or something.

With the actuel implementation, the id being generated on producer creation, I can't achieve this.

Thanks for opening this.

[garyrussell]

@Rajh 2.1.10 and 1.3.7 containing this fix is now in maven central.