No, it's not bad. Kafka is designed to serve data to many consumers at the same time, whether they are independent of each other or in the same consumer group.
I would encourage you to play with different partition counts and use kafka's performance testing tools (kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh) to test throughput in different scenarios and see the results for yourself.
—
Peter
> On Feb 27, 2020, at 1:28 AM, 张祥 <xiangzhang1128@gmail.com> wrote:
>
> I believe no matter the partition count exceeds the broker count, we can
> always have the same number of consumer instances as the partition count.
>
> So what I want to know is when two partition exists on the same broker, two
> consumer instances will be talking to same broker, is that bad ?
>
> 张祥 <xiangzhang1128@gmail.com> 于2020年2月27日周四 下午2:20写道：
>
>> Thanks. What influence does it have for consumers and producers when
>> partition number is more than broker number, which means at least one
>> broker serves two partitions for one topic ? performance wise.
>>
>> Peter Bukowinski <pmbuko@gmail.com> 于2020年2月26日周三 下午11:02写道：
>>
>>> Disk usage is one reason to expand. Another reason is if you need more
>>> ingest or output throughout for your topic data. If your producers aren't
>>> able to send data to kafka fast enough or your consumers are lagging, you
>>> might benefit from more brokers and more partitions.
>>>
>>> -- Peter
>>>
>>>> On Feb 26, 2020, at 12:56 AM, 张祥 <xiangzhang1128@gmail.com> wrote:
>>>>
>>>> In documentation, it is described how to expand cluster:
>>>>
>>> https://kafka.apache.org/20/documentation.html#basic_ops_cluster_expansion
>>> .
>>>> But I am wondering what the criteria for expand is. I can only think of
>>>> disk usage threshold. For example, suppose several disk usage exceed
>>> 80%.
>>>> Is this correct and is there more ?
>>>
>>
I would encourage you to play with different partition counts and use kafka's performance testing tools (kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh) to test throughput in different scenarios and see the results for yourself.
—
Peter
> On Feb 27, 2020, at 1:28 AM, 张祥 <xiangzhang1128@gmail.com> wrote:
>
> I believe no matter the partition count exceeds the broker count, we can
> always have the same number of consumer instances as the partition count.
>
> So what I want to know is when two partition exists on the same broker, two
> consumer instances will be talking to same broker, is that bad ?
>
> 张祥 <xiangzhang1128@gmail.com> 于2020年2月27日周四 下午2:20写道：
>
>> Thanks. What influence does it have for consumers and producers when
>> partition number is more than broker number, which means at least one
>> broker serves two partitions for one topic ? performance wise.
>>
>> Peter Bukowinski <pmbuko@gmail.com> 于2020年2月26日周三 下午11:02写道：
>>
>>> Disk usage is one reason to expand. Another reason is if you need more
>>> ingest or output throughout for your topic data. If your producers aren't
>>> able to send data to kafka fast enough or your consumers are lagging, you
>>> might benefit from more brokers and more partitions.
>>>
>>> -- Peter
>>>
>>>> On Feb 26, 2020, at 12:56 AM, 张祥 <xiangzhang1128@gmail.com> wrote:
>>>>
>>>> In documentation, it is described how to expand cluster:
>>>>
>>> https://kafka.apache.org/20/documentation.html#basic_ops_cluster_expansion
>>> .
>>>> But I am wondering what the criteria for expand is. I can only think of
>>>> disk usage threshold. For example, suppose several disk usage exceed
>>> 80%.
>>>> Is this correct and is there more ?
>>>
>>
Comments
Post a Comment