Skip to content

Commit 48caf23

Browse files
garyrussellartembilan
authored andcommitted
Doc: Partition Assignment
See: https://stackoverflow.com/questions/52077027/kafkalistener-concurrency-multiple-topics/52078148#52078148 **cherry-pick to 2.1.x, 1.3.x** (cherry picked from commit 9f828a8)
1 parent f99c74a commit 48caf23

File tree

1 file changed

+19
-1
lines changed

1 file changed

+19
-1
lines changed

src/reference/asciidoc/kafka.adoc

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -405,7 +405,25 @@ public ConcurrentMessageListenerContainer(ConsumerFactory<K, V> consumerFactory,
405405

406406
It also has a property `concurrency`, e.g. `container.setConcurrency(3)` will create 3 `KafkaMessageListenerContainer` s.
407407

408-
For the first constructor, kafka will distribute the partitions across the consumers.
408+
For the first constructor, kafka will distribute the partitions across the consumers using its group management capabilities.
409+
410+
[IMPORTANT]
411+
====
412+
When listening to multiple topics, the default partition distribution may not be what you expect.
413+
For example, if you have 3 topics with 5 partitions each and you want to use `concurrency=15` you will only see 5 active consumers, each assigned one partition from each topic, with the other 10 consumers being idle.
414+
This is because the default Kafka `PartitionAssignor` is the `RangeAssignor` (see its javadocs).
415+
For this scenario, you may want to consider using the `RoundRobinAssignor` instead, which will distribute the partitions across all of the consumers.
416+
Then, each consumer will be assigned one topic/partition.
417+
To change the `PartitionAssignor`, set the `partition.assignment.strategy` consumer property (`ConsumerConfigs.PARTITION_ASSIGNMENT_STRATEGY_CONFIG`) in the properties provided to the `DefaultKafkaConsumerFactory`.
418+
419+
When using Spring Boot:
420+
421+
[source]
422+
----
423+
spring.kafka.consumer.properties.partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
424+
----
425+
====
426+
409427
For the second constructor, the `ConcurrentMessageListenerContainer` distributes the `TopicPartition` s across the
410428
delegate `KafkaMessageListenerContainer` s.
411429

0 commit comments

Comments
 (0)