Kafka

Hi, i have a question about  kafka transaction hanging  issue: when i restart flink job,  something wrong with kafka transaction.  can't consumer in read_commit status. 1. find-hanging ./bin/kafka-transactions.sh  --command-config ./config/clent_jaas.conf --bootstrap-server 172.18.8.18:9092 , 172.18.8.19:9092 , 172.18.8.20:9092 find-hanging --topic ods_csys_20220805_prd_01 --partition 0   Topic                           Partition       ProducerId      ProducerEpoch   CoordinatorEpoch        StartOffset     LastTimestamp   Duration(min) ods_csys_20220805_prd_01        0    ...

Tomasz you'll need to upgrade the kafka Streams dependency to 3.0 (or above) to get the fix that John mentioned before -- this behavior is known/expected on earlier versions such as 2.8.1 as you are using On Tue, Nov 15, 2022 at 2:21 AM Tomasz Gac <tomasz.gac@empirica.io.invalid> wrote: > Hi John, > > I've reviewed the test you sent and it seems to be correct, but it may not > reproduce our setup. > > We are using Java 8, kafka-client 2.8.1 with kafka streams version 2.8.1 > against the kafka broker version 2.1.1. We are running it as an OSGI bundle > with dependencies packaged within the bundle. > > Thank you, > Tomasz > > > pt., 30 wrz 2022 o 17:57 John Roesler < vvcephei@apache.org > napisał(a): > > > Hi again, Tomasz, > > > > Your issue is really bugging me, since I'm pretty sure it shouldn't be > > happening. > > > > I went ahead and added an i...

Interesting, this does look like it could be a bug in Streams and I'm not aware of any known or already-fixed issues resembling this. Could you file a bug report over here < https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA > and include as much context/information as possible? Providing logs from around the time leading up to this exception in particular would greatly help in debugging this On Tue, Nov 15, 2022 at 2:15 AM Tomasz Gac <tomasz.gac@empirica.io.invalid> wrote: > Hi group, > > We've encountered a problem during regular operation of the kafka-streams > application. While processing a record we received the following error. > There's very little documentation on this kind of problem but I've gathered > that it's a synchronization issue between kafka consumer and producer. Have > you ever encountered it before? > > My questions are: is this a matter of misconfiguration, or rather ...

Hi Pierre, You may try to use cruise control: https://github.com/linkedin/cruise-control I didn't try it yet but it has task which may help you to auto-balance partitions in the cluster. BR, Mikhail On Tue, Nov 15, 2022 at 3:17 PM sunil chaudhari < sunilmchaudhari05@gmail.com > wrote: > Hi, > Use confluent. It has auto balancing feature. > You dont need to do these manual things. > > > On Tue, 15 Nov 2022 at 7:22 PM, Pierre Coquentin < > pierre.coquentin@gmail.com > > wrote: > > > Hello Luke, and thank you for your answer. > > What I would have hoped for is something more automatic, something that > > will spread the load when a Kafka broker goes down without any human > > intervention. The reassign script is a bit complicated, you need to > > generate the topics and partitions list, then get the current assignment > > and rework it to force a new leader. > > > ...

Hi, Use confluent. It has auto balancing feature. You dont need to do these manual things. On Tue, 15 Nov 2022 at 7:22 PM, Pierre Coquentin < pierre.coquentin@gmail.com > wrote: > Hello Luke, and thank you for your answer. > What I would have hoped for is something more automatic, something that > will spread the load when a Kafka broker goes down without any human > intervention. The reassign script is a bit complicated, you need to > generate the topics and partitions list, then get the current assignment > and rework it to force a new leader. > > On Tue, Nov 15, 2022 at 5:18 AM Luke Chen < showuon@gmail.com > wrote: > > > Hi Pierre, > > > > Try using kafka-reassign-partitions.sh to reassign partitions to > different > > replicas you like. > > ref: https://kafka.apache.org/documentation/#basic_ops_automigrate > > > > Luke > > > > On Mon, Nov 14, 2022 at 3:55 ...

Hello Luke, and thank you for your answer. What I would have hoped for is something more automatic, something that will spread the load when a Kafka broker goes down without any human intervention. The reassign script is a bit complicated, you need to generate the topics and partitions list, then get the current assignment and rework it to force a new leader. On Tue, Nov 15, 2022 at 5:18 AM Luke Chen < showuon@gmail.com > wrote: > Hi Pierre, > > Try using kafka-reassign-partitions.sh to reassign partitions to different > replicas you like. > ref: https://kafka.apache.org/documentation/#basic_ops_automigrate > > Luke > > On Mon, Nov 14, 2022 at 3:55 PM Pierre Coquentin < > pierre.coquentin@gmail.com > > wrote: > > > Hello, > > We have a Kafka cluster (2.4.1) with a replication factor of 3. I notice > > when we stop a broker that only one broker takes all the load from the > > missing brok...

Hi John, I've reviewed the test you sent and it seems to be correct, but it may not reproduce our setup. We are using Java 8, kafka-client 2.8.1 with kafka streams version 2.8.1 against the kafka broker version 2.1.1. We are running it as an OSGI bundle with dependencies packaged within the bundle. Thank you, Tomasz pt., 30 wrz 2022 o 17:57 John Roesler < vvcephei@apache.org > napisał(a): > Hi again, Tomasz, > > Your issue is really bugging me, since I'm pretty sure it shouldn't be > happening. > > I went ahead and added an integration test with your exact scenario, as I > understand it: https://github.com/apache/kafka/pull/12706 > > The test passes for me. > > Do you think you can check it out and try adjusting the test setup until > you're able to reproduce the behavior you're seeing? If you can do that, I > think we will get to the bottom of it. > > Thanks, > -John > ...