Re: Scaling Apache Kafka Producers & Consumers

Very good description with pictures in the book Kafka: The Definitive Guide

https://www.oreilly.com/library/view/kafka-the-definitive/9781491936153/ch04.html

-hans

> On Mar 26, 2020, at 12:00 PM, sunil chaudhari <sunilmchaudhari05@gmail.com> wrote:
>
> Again....
> A consumer can have one or more consumer thread.
> The analogy of 12 partitions and 4 consumer is true when each consumer has
> 3 consumer threads.
> Please don't skip the important factor "consumer thread" in this matter.
>
> If you run each consumer with threads then you may need max 3 consumers for
> that consumer group.
>
> If you have 12 partitions and you run 4 consumers with 4 consumer threads
> then 4 threads will be idle at any time T1.
>
> I hope this is clear.
>
> Thanks,
> Sunil.
>
> On Thu, 26 Mar 2020 at 7:52 PM, Hans Jespersen <hans@confluent.io> wrote:
>
>>> As per my understanding, in Apache Kafka a single consumer from a
>> consumer
>>> group can consume messages from one partition only.
>>
>> Not correct. A single consumer from a consumer group can consume from many
>> partitions. For example if you had a topic with 12 partitions and 4
>> consumers in a consumer group, each consumer in the group would consume
>> from 3 partitions.
>>
>> -hans