On 2023-04-24 11:19, Margaret Figura wrote:
> Hi Arvid,
>
> Just as another user of Kafka, I suspect the claim has to do with the
> producer's ability to perform batching and compression across produced
> data, so it should be more efficient in some cases, especially with
> smaller volumes of data and larger linger.ms settings.
>
> However, like you, I've found that a single KafkaProducer instance can
> sometimes become a bottleneck. For instance, threads can be blocked in
> certain cases when they are producing to the same partition. In our
> application, we found a producer per thread wasn't necessary, but
> adding a small pool of producers (e.g. 4 producers shared by 32
> threads) helped to improve performance. It probably varies quite a bit
> on your use-case and configuration.
>
> I do wonder if the Kafka team would consider it a bug in the cases
> where a single producer isn't faster, or if they'd consider it a doc
> bug where the recommendation should be a bit more nuanced.
>
> Hope that helps,
> Meg
>
> -----Original Message-----
> From: Arvid Sundbom <arvid.sundbom98@gmail.com>
> Sent: Monday, April 24, 2023 9:01 AM
> To: users@kafka.apache.org
> Subject: Producer Application Utilizing Multiple Threads
>
> CAUTION: External Email : Be wary of clicking links or if this claims
> to be internal.
>
> Hi!
> I have a question about executing a Kafka producer application,
> utilizing multiple threads.
> In the documentation for Kafka producers (
> https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html)
> it says that "...sharing a single producer instance across threads
> will generally be faster than having multiple instances.".
>
> I am mainly wondering if there is any data to verify this claim?
> The reason for this is that I have carried out rather extensive
> performance tests, with varying levels of computational load per
> message produced, using between 1 and 24 threads, and it seems that
> there is really no situation in which a single, shared producer
> achieves a higher performance than if each executing thread is
> assigned its own producer instance.
>
> Kind regards,
> Arvid
Hi,
New information.
Kind regards,
> Hi Arvid,
>
> Just as another user of Kafka, I suspect the claim has to do with the
> producer's ability to perform batching and compression across produced
> data, so it should be more efficient in some cases, especially with
> smaller volumes of data and larger linger.ms settings.
>
> However, like you, I've found that a single KafkaProducer instance can
> sometimes become a bottleneck. For instance, threads can be blocked in
> certain cases when they are producing to the same partition. In our
> application, we found a producer per thread wasn't necessary, but
> adding a small pool of producers (e.g. 4 producers shared by 32
> threads) helped to improve performance. It probably varies quite a bit
> on your use-case and configuration.
>
> I do wonder if the Kafka team would consider it a bug in the cases
> where a single producer isn't faster, or if they'd consider it a doc
> bug where the recommendation should be a bit more nuanced.
>
> Hope that helps,
> Meg
>
> -----Original Message-----
> From: Arvid Sundbom <arvid.sundbom98@gmail.com>
> Sent: Monday, April 24, 2023 9:01 AM
> To: users@kafka.apache.org
> Subject: Producer Application Utilizing Multiple Threads
>
> CAUTION: External Email : Be wary of clicking links or if this claims
> to be internal.
>
> Hi!
> I have a question about executing a Kafka producer application,
> utilizing multiple threads.
> In the documentation for Kafka producers (
> https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html)
> it says that "...sharing a single producer instance across threads
> will generally be faster than having multiple instances.".
>
> I am mainly wondering if there is any data to verify this claim?
> The reason for this is that I have carried out rather extensive
> performance tests, with varying levels of computational load per
> message produced, using between 1 and 24 threads, and it seems that
> there is really no situation in which a single, shared producer
> achieves a higher performance than if each executing thread is
> assigned its own producer instance.
>
> Kind regards,
> Arvid
Hi,
New information.
Kind regards,
Comments
Post a Comment