Message duplication after ConsumerRebalancing

[stepio]

It's not a bug report, more like a consultancy request... maybe an idea for the feature.

While playing with kafka-console-consumer I've spotted that sometimes after consumer rebalancing same message is received the second time. Checked google, read the article:
https://cwiki.apache.org/confluence/display/KAFKA/Idempotent+Producer
As I understand, it's not a bug, but kind of limitation.

But we all know that in many cases message uniqueness is important & same message should not be processed second time. So is there any already implemented approach to achieve such uniqueness?

As of now, I see the next quite easy mechanism, which may help:

1. On producer side: each message should be wrapped into a "holder" object with unique id (UUID).
2. On consumer side: each message Id should be compared with the previous one (or ones) - if such an id is already known, the received message should be logged & discarded.

What do you think?

P.S.: This may be related to #66 & #79

[garyrussell]

Since this is a "higher" level problem that affects many messaging providers, our general solution for this is in the "higher level" Spring Integration abstraction and particularly the Idempotent Receiver pattern. This project (sping-kafka) has a sister project spring-integration-kafka which adds Spring Integration components on top of the functionality provided here.

For that reason, I am not keen on baking in such functionality in this lower-level project.

[stepio]

Thanks @garyrussell, that explains much - I'll study the provided links and will consider this option.

But let me just share my thoughts. I try using Kafka as a little less developed, but a very promising replacement for traditional JMS brokers. My colleagues love spring-amqp, but they are not happy to depend on something huge, like apache-camel, or spring-integration. So spring-kafka project was exactly what I needed.

As of now, duplicated messages are the only significant weakness in comparison to Rabbit MQ for our needs - I've never encountered such issues with traditional JMS, so I assume that they are at least much less common for Rabbit MQ (if not impossible). So some basic "quick win" implementation may be a strong reason to stay with Kafka - not just for me, but also for future users of this project.

[garyrussell]

Duplicate messages are certainly possible (although perhaps less often) with both JMS and RabbitMQ.

One thing we could consider here is to introduce an advice chain into the container (similar to Spring AMQP) whereby you could insert an Advice object around the listener. I don't mind introducing a general purpose feature like that which allows easy extension by the user. With that in place, we could consider adding a library of Advices that perform useful features.

[garyrussell]

Here's another possibility: garyrussell@0cd1a5c - a simple wrapper where you can provide your own deduplication logic via a strategy interface.

[stepio]

That's what I've needed, thanks!