Kafka

Thanks, for following up on this, Tony. It's always super helpful to hear how things get resolved. -D On Fri, Jul 16, 2021 at 12:14 PM Tony John < tonyjohnantony@gmail.com > wrote: > Hi All, > > An update on this. Finally I could figure out the cause for this. I have a > consumer with *MAX_POLL_INTERVAL_MS_CONFIG* set to *Integer.MAX_VALUE*, > which was causing the problem. Looks like its a combination of > * group.initial.rebalance.delay.ms > < http://group.initial.rebalance.delay.ms >* in kafka + the * > max.poll.interval.ms > < http://max.poll.interval.ms >* causing the *Rebalance failed. > org.apache.kafka.common.errors.DisconnectException*. After debugging I > could see the below line from AbstractCoordinator class (line #337), which > leads integer overflow if * max.poll.interval.ms > < http://max.poll.interval.ms >* is greater than (Integer.MAX_VALUE - 5000) > and thus *joinGroupTimeout...

Perfect. Thank you. ________________________________ From: Luke Chen < showuon@gmail.com > Sent: Monday, July 19, 2021 5:37 AM To: Kafka Users < users@kafka.apache.org > Subject: Re: Kafka incremental sticky rebalancing. Hi Mazen, > can it be because the rebalancing is taking place before the offset commit request has reached the coordinator (but this is a blocking commit so can not happen…) --> Yes, this error is thrown before sending offset commit request to the coordinator, and check if consumer group is undergoing rebalance. And yes, while rebalancing, the commit is blocked, that's why you got the RebalanceInProgressException. As discussed, the RebalanceInProgressException is expected result, so, back to your problem: > Given that I am running on Kubernetes, the consumer pods are considered in error state and Killed/restarted by Kubernetes --> I think Kubernetes can configure when the pod should be killed/restarted and what ...

Hi Mazen, > can it be because the rebalancing is taking place before the offset commit request has reached the coordinator (but this is a blocking commit so can not happen…) --> Yes, this error is thrown before sending offset commit request to the coordinator, and check if consumer group is undergoing rebalance. And yes, while rebalancing, the commit is blocked, that's why you got the RebalanceInProgressException. As discussed, the RebalanceInProgressException is expected result, so, back to your problem: > Given that I am running on Kubernetes, the consumer pods are considered in error state and Killed/restarted by Kubernetes --> I think Kubernetes can configure when the pod should be killed/restarted and what state is considered "error". Maybe that's the place you can check. Thank you. Luke On Sat, Jul 17, 2021 at 8:59 PM Mazen Ezzeddine < mazen.ezzeddine@etu.univ-cotedazur.fr > wrote: > Hello Luke, > > Pleas...

Yep that's the one. You can see here for a great example on the typical flow https://www.confluent.io/blog/transactions-apache-kafka/ . On Sat, Jul 17, 2021 at 3:24 AM Pushkar Deole < pdeole2015@gmail.com > wrote: > Hi Lerh Chuan Low, > > MAny thanks for your response. I get it now, that it provides exactly-once > semantics i.e it looks to user that it is processed exactly once. > Also, i am clear on the aspect about read_committed level so the > uncommitted transaction and hence uncommitted send won't be visible to > consumers. > > However one last query i have is how to make sure that as part of the same > transaction, i am also sending and also committing offsets. Which API > should i look at: is this correct API : > KafkaProducer. > sendOffsetsToTransaction > > On Fri, Jul 16, 2021 at 9:57 PM Lerh Chuan Low < lerhchuan@gmail.com > > wrote: > > > Pushkar, > > > > My und...

Hello Luke, Please note that recently I was working on tasks other than incremental rebalancing, but now I have returned to experimenting with incremental rebalancing. Please note that I am running kafka broker clients version 2.7 on Kubernetes GKE. My setup consist of dynamic autoscaling of consumers configured with the CooperativeStickyAssignor As recently discussed, when a scale up is performed and a new consumer joins the group the below exception is raised. "Exception in thread "main" org.apache.kafka.common.errors.RebalanceInProgressException: Offset commit cannot be completed since the consumer is undergoing a rebalance for auto partition assignment. You can try completing the rebalance by calling poll() and then retry the operation." You suggested to just ignore the exception as the offset commit can be completed in the next poll round. Please note that as I scale the consumer group, all the members of the group are raising this except...

Please note that recently I was working on tasks other than incremental rebalancing, but now I have returned to experimenting with incremental rebalancing. Please note that I am running kafka broker clients version 2.7 on Kubernetes GKE. My setup consist of dynamic autoscaling of consumers configured with the CooperativeStickyAssignor As recently discussed, when a scale up is performed and a new consumer joins the group the below exception is raised. "Exception in thread "main" org.apache.kafka.common.errors.RebalanceInProgressException: Offset commit cannot be completed since the consumer is undergoing a rebalance for auto partition assignment. You can try completing the rebalance by calling poll() and then retry the operation." You suggested to just ignore the exception as the offset commit can be completed in the next poll round. Please note that as I scale the consumer group, all the members of the group are raising this exception. Given that...

Hi Lerh Chuan Low, MAny thanks for your response. I get it now, that it provides exactly-once semantics i.e it looks to user that it is processed exactly once. Also, i am clear on the aspect about read_committed level so the uncommitted transaction and hence uncommitted send won't be visible to consumers. However one last query i have is how to make sure that as part of the same transaction, i am also sending and also committing offsets. Which API should i look at: is this correct API : KafkaProducer. sendOffsetsToTransaction On Fri, Jul 16, 2021 at 9:57 PM Lerh Chuan Low < lerhchuan@gmail.com > wrote: > Pushkar, > > My understanding is you can easily turn it on by using Kafka streams as > Chris mentioned. Otherwise you'd have to do it yourself - I don't think you > can get exactly once processing, but what you can do (which is also what > Kafka streams does) is exactly once schematics (You won't be able to get > ever...