Re: Random continuous TimeoutException with Topic not present on one KafkaProducer out of many in multithreaded env
incream block time to 3 minutes ! it is an exception about update
metadata,try it!
On Tue, Jun 7, 2022 at 3:15 PM Deepak Jain <deepak.jain@cumulus-systems.com>
wrote:
> Hi Luke,
>
> The complete exception is
>
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.TimeoutException: Topic realtimeImport_1 not
> present in metadata after 250 ms.
> at
> org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1316)
> at
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:985)
> at
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:885)
> at
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:773)
>
> Even though the topic is created and used but it still throws this
> exception and fails the operation.
>
> Regards,
> Deepak
>
> From: Luke Chen <showuon@gmail.com>
> Sent: 07 June 2022 11:46
> To: Deepak Jain <deepak.jain@cumulus-systems.com>
> Cc: users@kafka.apache.org
> Subject: Re: Random continuous TimeoutException with Topic not present on
> one KafkaProducer out of many in multithreaded env
>
> Caution: From Cumulus Systems – IT Department, this email originated from
> outside of the organization. Please call and confirm with the sender before
> opening attachments or clicking links inside the email.
>
> Hi Deepak,
>
> So, if you change the value in max.block.ms<http://max.block.ms> to
> default 1 minute, does the timeout exception still exist?
> I think the timeoutException is complaining the 250ms is not a good
> configuration for your environment.
>
> Thank you.
> Luke
>
> On Tue, Jun 7, 2022 at 11:23 AM Deepak Jain <
> deepak.jain@cumulus-systems.com<mailto:deepak.jain@cumulus-systems.com>>
> wrote:
> Hi,
>
> Thanks for the quick reply.
>
> We are already using the config max.block.ms<http://max.block.ms>
> (alongwith with other recommended config like request.timeout.ms<
> http://request.timeout.ms> and others). Although the value we are using
> is very less at 250 ms but since we have 5 different KafkaProducer running
> in each individual thread out of which 4 are working without any issue and
> only 1 is throwing the TimeOutException, so this does not seems to be the
> issue,
>
> Please else us know if anybody had came across this type of behaviour by
> Kafka. If yes, please help in finding out the root cause and resolving it.
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: 张晓寅 <zhxy0605@gmail.com<mailto:zhxy0605@gmail.com>>
> Sent: 06 June 2022 19:10
> To: users@kafka.apache.org<mailto:users@kafka.apache.org>
> Cc: Luke Chen <showuon@gmail.com<mailto:showuon@gmail.com>>
> Subject: Re: Random continuous TimeoutException with Topic not present on
> one KafkaProducer out of many in multithreaded env
>
> Caution: From Cumulus Systems – IT Department, this email originated from
> outside of the organization. Please call and confirm with the sender before
> opening attachments or clicking links inside the email.
>
>
> maybe you can add producer "max.block.ms<http://max.block.ms>" config,but
> you should test your broker look up some logs about leader change
> ,producer performance,like traffic ,produce "buffer" and "batch.size"
>
> On Mon, Jun 6, 2022 at 6:53 PM Deepak Jain <
> deepak.jain@cumulus-systems.com<mailto:deepak.jain@cumulus-systems.com>>
> wrote:
>
> > Hello All,
> >
> > Please help me out in this regard as the Customer has reported this on
> > their production environment and waiting for our reply ASAP.
> >
> > Regards,
> > Deepak
> >
> > From: Deepak Jain
> > Sent: 02 June 2022 20:53
> > To: 'users@kafka.apache.org<mailto:users@kafka.apache.org>' <
> users@kafka.apache.org<mailto:users@kafka.apache.org>>
> > Cc: 'Luke Chen' <showuon@gmail.com<mailto:showuon@gmail.com>>; Alap
> Patwardhan <
> > alap@cumulus-systems.com<mailto:alap@cumulus-systems.com>>; Bhushan
> Patil <
> > bhushan.patil@cumulus-systems.com<mailto:
> bhushan.patil@cumulus-systems.com>>
> > Subject: Random continuous TimeoutException with Topic not present on
> > one KafkaProducer out of many in multithreaded env
> >
> > Hello Everyone,
> >
> > We are using Kafka 2.8.1 Broker/Client system in our prod env.
> >
> > Getting following exception randomly after 1 hour or so for one
> > Realtime transfer from Kafka Producer to broker out of 5. (Rest 4 are
> > working fine.)
> >
> > java.util.concurrent.ExecutionException:
> > org.apache.kafka.common.errors.TimeoutException: Topic
> > realtimeImport_1 not present in metadata after 250 ms.
> > at
> >
> org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1316)
> > at
> >
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:985)
> > at
> >
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:885)
> > at
> > org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.jav
> > a:773)
> >
> > We are using multithreaded KafkaProducer with their each unique topic
> > sending data to single broker. Here, we notice that this exception
> > comes when we reconnect to Kafka using close() (void
> > org<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/app%5
> > C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Corg>
> > .apache<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/a
> > pp%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3C
> > org.apache>.kafka<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/
> > megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.
> > 8.1.jar%3Corg.apache.kafka>.clients<eclipse-javadoc:%E2%98%82=hdca/D:%
> > 5C/git%5C/hdca%5C/megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5
> > C/kafka-clients-2.8.1.jar%3Corg.apache.kafka.clients>.producer<eclipse
> > -javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/app%5C/platform%
> > 5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Corg.apache.kafk
> > a.clients.producer>.KafkaProducer<eclipse-javadoc:%E2%98%82=hdca/D:%5C
> > /git%5C/hdca%5C/megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5C/
> > kafka-clients-2.8.1.jar%3Corg.apache.kafka.clients.producer(KafkaProdu
> > cer.class%E2%98%83KafkaProducer>.close())
> > and
> > org<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/app%5
> > C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Corg>
> > .apache<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/a
> > pp%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3C
> > org.apache>.kafka<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/
> > megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.
> > 8.1.jar%3Corg.apache.kafka>.clients<eclipse-javadoc:%E2%98%82=hdca/D:%
> > 5C/git%5C/hdca%5C/megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5
> > C/kafka-clients-2.8.1.jar%3Corg.apache.kafka.clients>.producer<eclipse
> > -javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/app%5C/platform%
> > 5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Corg.apache.kafk
> > a.clients.producer>.KafkaProducer<eclipse-javadoc:%E2%98%82=hdca/D:%5C
> > /git%5C/hdca%5C/megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5C/
> > kafka-clients-2.8.1.jar%3Corg.apache.kafka.clients.producer(KafkaProdu
> > cer.class%E2%98%83KafkaProducer>.KafkaProducer<Object<eclipse-javadoc:
> > %E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/app%5C/platform%5C/common
> > %5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Corg.apache.kafka.clients
> > .producer(KafkaProducer.class%E2%98%83KafkaProducer~KafkaProducer~Ljav
> > a.util.Properties;%E2%98%82java.lang.Object>,
> > Object<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/ap
> > p%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Co
> > rg.apache.kafka.clients.producer(KafkaProducer.class%E2%98%83KafkaProd
> > ucer~KafkaProducer~Ljava.util.Properties;%E2%98%82java.lang.Object>>(P
> > roperties<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C
> > /app%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%
> > 3Corg.apache.kafka.clients.producer(KafkaProducer.class%E2%98%83KafkaP
> > roducer~KafkaProducer~Ljava.util.Properties;%E2%98%82java.util.Propert
> > ies>
> > properties) calls. Not sure whether this is the culprit or not.
> >
> > Due to this exception the realtime resources are not getting transfer
> > to Kafka Consumer. We are using following config on Kafka Broker:
> >
> > group.initial.rebalance.delay.ms<http://group.initial.rebalance.delay.ms
> >=0
> > listeners=SASL_PLAINTEXT://0.0.0.0:9092<http://0.0.0.0:9092>
> > log.retention.minutes=15
> > delete.topic.enable=true
> > auto.create.topics.enable=true
> > security.inter.broker.protocol=SASL_PLAINTEXT
> > sasl.mechanism.inter.broker.protocol=PLAIN
> > sasl.enabled.mechanisms=PLAIN
> > security.protocol=SASL_PLAINTEXT
> > authorizer.class.name<http://authorizer.class.name
> >=kafka.security.auth.SimpleAclAuthorizer
> > allow.everyone.if.no.acl.found=true
> > zookeeper.set.acl=true
> >
> > Can anyone please help us in finding the root cause for it?
> >
> > Regards,
> > Deepak Jain
> > Cumulus Systems
> >
>
metadata,try it!
On Tue, Jun 7, 2022 at 3:15 PM Deepak Jain <deepak.jain@cumulus-systems.com>
wrote:
> Hi Luke,
>
> The complete exception is
>
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.TimeoutException: Topic realtimeImport_1 not
> present in metadata after 250 ms.
> at
> org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1316)
> at
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:985)
> at
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:885)
> at
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:773)
>
> Even though the topic is created and used but it still throws this
> exception and fails the operation.
>
> Regards,
> Deepak
>
> From: Luke Chen <showuon@gmail.com>
> Sent: 07 June 2022 11:46
> To: Deepak Jain <deepak.jain@cumulus-systems.com>
> Cc: users@kafka.apache.org
> Subject: Re: Random continuous TimeoutException with Topic not present on
> one KafkaProducer out of many in multithreaded env
>
> Caution: From Cumulus Systems – IT Department, this email originated from
> outside of the organization. Please call and confirm with the sender before
> opening attachments or clicking links inside the email.
>
> Hi Deepak,
>
> So, if you change the value in max.block.ms<http://max.block.ms> to
> default 1 minute, does the timeout exception still exist?
> I think the timeoutException is complaining the 250ms is not a good
> configuration for your environment.
>
> Thank you.
> Luke
>
> On Tue, Jun 7, 2022 at 11:23 AM Deepak Jain <
> deepak.jain@cumulus-systems.com<mailto:deepak.jain@cumulus-systems.com>>
> wrote:
> Hi,
>
> Thanks for the quick reply.
>
> We are already using the config max.block.ms<http://max.block.ms>
> (alongwith with other recommended config like request.timeout.ms<
> http://request.timeout.ms> and others). Although the value we are using
> is very less at 250 ms but since we have 5 different KafkaProducer running
> in each individual thread out of which 4 are working without any issue and
> only 1 is throwing the TimeOutException, so this does not seems to be the
> issue,
>
> Please else us know if anybody had came across this type of behaviour by
> Kafka. If yes, please help in finding out the root cause and resolving it.
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: 张晓寅 <zhxy0605@gmail.com<mailto:zhxy0605@gmail.com>>
> Sent: 06 June 2022 19:10
> To: users@kafka.apache.org<mailto:users@kafka.apache.org>
> Cc: Luke Chen <showuon@gmail.com<mailto:showuon@gmail.com>>
> Subject: Re: Random continuous TimeoutException with Topic not present on
> one KafkaProducer out of many in multithreaded env
>
> Caution: From Cumulus Systems – IT Department, this email originated from
> outside of the organization. Please call and confirm with the sender before
> opening attachments or clicking links inside the email.
>
>
> maybe you can add producer "max.block.ms<http://max.block.ms>" config,but
> you should test your broker look up some logs about leader change
> ,producer performance,like traffic ,produce "buffer" and "batch.size"
>
> On Mon, Jun 6, 2022 at 6:53 PM Deepak Jain <
> deepak.jain@cumulus-systems.com<mailto:deepak.jain@cumulus-systems.com>>
> wrote:
>
> > Hello All,
> >
> > Please help me out in this regard as the Customer has reported this on
> > their production environment and waiting for our reply ASAP.
> >
> > Regards,
> > Deepak
> >
> > From: Deepak Jain
> > Sent: 02 June 2022 20:53
> > To: 'users@kafka.apache.org<mailto:users@kafka.apache.org>' <
> users@kafka.apache.org<mailto:users@kafka.apache.org>>
> > Cc: 'Luke Chen' <showuon@gmail.com<mailto:showuon@gmail.com>>; Alap
> Patwardhan <
> > alap@cumulus-systems.com<mailto:alap@cumulus-systems.com>>; Bhushan
> Patil <
> > bhushan.patil@cumulus-systems.com<mailto:
> bhushan.patil@cumulus-systems.com>>
> > Subject: Random continuous TimeoutException with Topic not present on
> > one KafkaProducer out of many in multithreaded env
> >
> > Hello Everyone,
> >
> > We are using Kafka 2.8.1 Broker/Client system in our prod env.
> >
> > Getting following exception randomly after 1 hour or so for one
> > Realtime transfer from Kafka Producer to broker out of 5. (Rest 4 are
> > working fine.)
> >
> > java.util.concurrent.ExecutionException:
> > org.apache.kafka.common.errors.TimeoutException: Topic
> > realtimeImport_1 not present in metadata after 250 ms.
> > at
> >
> org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1316)
> > at
> >
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:985)
> > at
> >
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:885)
> > at
> > org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.jav
> > a:773)
> >
> > We are using multithreaded KafkaProducer with their each unique topic
> > sending data to single broker. Here, we notice that this exception
> > comes when we reconnect to Kafka using close() (void
> > org<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/app%5
> > C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Corg>
> > .apache<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/a
> > pp%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3C
> > org.apache>.kafka<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/
> > megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.
> > 8.1.jar%3Corg.apache.kafka>.clients<eclipse-javadoc:%E2%98%82=hdca/D:%
> > 5C/git%5C/hdca%5C/megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5
> > C/kafka-clients-2.8.1.jar%3Corg.apache.kafka.clients>.producer<eclipse
> > -javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/app%5C/platform%
> > 5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Corg.apache.kafk
> > a.clients.producer>.KafkaProducer<eclipse-javadoc:%E2%98%82=hdca/D:%5C
> > /git%5C/hdca%5C/megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5C/
> > kafka-clients-2.8.1.jar%3Corg.apache.kafka.clients.producer(KafkaProdu
> > cer.class%E2%98%83KafkaProducer>.close())
> > and
> > org<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/app%5
> > C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Corg>
> > .apache<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/a
> > pp%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3C
> > org.apache>.kafka<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/
> > megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.
> > 8.1.jar%3Corg.apache.kafka>.clients<eclipse-javadoc:%E2%98%82=hdca/D:%
> > 5C/git%5C/hdca%5C/megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5
> > C/kafka-clients-2.8.1.jar%3Corg.apache.kafka.clients>.producer<eclipse
> > -javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/app%5C/platform%
> > 5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Corg.apache.kafk
> > a.clients.producer>.KafkaProducer<eclipse-javadoc:%E2%98%82=hdca/D:%5C
> > /git%5C/hdca%5C/megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5C/
> > kafka-clients-2.8.1.jar%3Corg.apache.kafka.clients.producer(KafkaProdu
> > cer.class%E2%98%83KafkaProducer>.KafkaProducer<Object<eclipse-javadoc:
> > %E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/app%5C/platform%5C/common
> > %5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Corg.apache.kafka.clients
> > .producer(KafkaProducer.class%E2%98%83KafkaProducer~KafkaProducer~Ljav
> > a.util.Properties;%E2%98%82java.lang.Object>,
> > Object<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/ap
> > p%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Co
> > rg.apache.kafka.clients.producer(KafkaProducer.class%E2%98%83KafkaProd
> > ucer~KafkaProducer~Ljava.util.Properties;%E2%98%82java.lang.Object>>(P
> > roperties<eclipse-javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C
> > /app%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%
> > 3Corg.apache.kafka.clients.producer(KafkaProducer.class%E2%98%83KafkaP
> > roducer~KafkaProducer~Ljava.util.Properties;%E2%98%82java.util.Propert
> > ies>
> > properties) calls. Not sure whether this is the culprit or not.
> >
> > Due to this exception the realtime resources are not getting transfer
> > to Kafka Consumer. We are using following config on Kafka Broker:
> >
> > group.initial.rebalance.delay.ms<http://group.initial.rebalance.delay.ms
> >=0
> > listeners=SASL_PLAINTEXT://0.0.0.0:9092<http://0.0.0.0:9092>
> > log.retention.minutes=15
> > delete.topic.enable=true
> > auto.create.topics.enable=true
> > security.inter.broker.protocol=SASL_PLAINTEXT
> > sasl.mechanism.inter.broker.protocol=PLAIN
> > sasl.enabled.mechanisms=PLAIN
> > security.protocol=SASL_PLAINTEXT
> > authorizer.class.name<http://authorizer.class.name
> >=kafka.security.auth.SimpleAclAuthorizer
> > allow.everyone.if.no.acl.found=true
> > zookeeper.set.acl=true
> >
> > Can anyone please help us in finding the root cause for it?
> >
> > Regards,
> > Deepak Jain
> > Cumulus Systems
> >
>
Comments
Post a Comment