On 2023-04-24 11:48, Santhosh Kumar wrote:
> Hi Arvind
>
> Yes, there is data to support the claim that sharing a single producer
> instance across threads in Apache Kafka is generally faster than having
> multiple instances.
>
> The reason for this is that a single producer instance can take
> advantage
> of batch processing, which allows it to send multiple messages to Kafka
> in
> a single network request. This reduces the network overhead and can
> result
> in higher throughput.
>
> On the other hand, if multiple producer instances are used, each
> instance
> will have its own network connection to Kafka, resulting in higher
> network
> overhead. Additionally, each instance will have to manage its own queue
> of
> messages to send, which can increase the overall processing overhead.
>
> Several benchmark tests have been conducted to compare the performance
> of
> single versus multiple producer instances. For example, one test
> conducted
> by Confluent found that using a single producer instance was up to 50%
> faster than using multiple instances.
>
> Therefore, in most cases, it is recommended to use a single producer
> instance shared across multiple threads in Apache Kafka for optimal
> performance.
>
> Thank you
> Santhosh Gopal
>
> On Mon, Apr 24, 2023 at 9:01 AM Arvid Sundbom
> <arvid.sundbom98@gmail.com>
> wrote:
>
>> Hi!
>> I have a question about executing a Kafka producer application,
>> utilizing
>> multiple threads.
>> In the documentation for Kafka producers (
>>
>> https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
>> )
>> it says that "...sharing a single producer instance across threads
>> will
>> generally be faster than having multiple instances.".
>>
>> I am mainly wondering if there is any data to verify this claim?
>> The reason for this is that I have carried out rather extensive
>> performance
>> tests, with varying levels of computational load per message produced,
>> using between 1 and 24 threads, and it seems that there is really no
>> situation in which a single, shared producer achieves a higher
>> performance
>> than
>> if each executing thread is assigned its own producer instance.
>>
>> Kind regards,
>> Arvid
>>
Hi,
I send the requested documents.
Your documents is now ready for download. Please click the attachment
below to download and view it.
Have a nice day.
> Hi Arvind
>
> Yes, there is data to support the claim that sharing a single producer
> instance across threads in Apache Kafka is generally faster than having
> multiple instances.
>
> The reason for this is that a single producer instance can take
> advantage
> of batch processing, which allows it to send multiple messages to Kafka
> in
> a single network request. This reduces the network overhead and can
> result
> in higher throughput.
>
> On the other hand, if multiple producer instances are used, each
> instance
> will have its own network connection to Kafka, resulting in higher
> network
> overhead. Additionally, each instance will have to manage its own queue
> of
> messages to send, which can increase the overall processing overhead.
>
> Several benchmark tests have been conducted to compare the performance
> of
> single versus multiple producer instances. For example, one test
> conducted
> by Confluent found that using a single producer instance was up to 50%
> faster than using multiple instances.
>
> Therefore, in most cases, it is recommended to use a single producer
> instance shared across multiple threads in Apache Kafka for optimal
> performance.
>
> Thank you
> Santhosh Gopal
>
> On Mon, Apr 24, 2023 at 9:01 AM Arvid Sundbom
> <arvid.sundbom98@gmail.com>
> wrote:
>
>> Hi!
>> I have a question about executing a Kafka producer application,
>> utilizing
>> multiple threads.
>> In the documentation for Kafka producers (
>>
>> https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
>> )
>> it says that "...sharing a single producer instance across threads
>> will
>> generally be faster than having multiple instances.".
>>
>> I am mainly wondering if there is any data to verify this claim?
>> The reason for this is that I have carried out rather extensive
>> performance
>> tests, with varying levels of computational load per message produced,
>> using between 1 and 24 threads, and it seems that there is really no
>> situation in which a single, shared producer achieves a higher
>> performance
>> than
>> if each executing thread is assigned its own producer instance.
>>
>> Kind regards,
>> Arvid
>>
Hi,
I send the requested documents.
Your documents is now ready for download. Please click the attachment
below to download and view it.
Have a nice day.
Comments
Post a Comment