Kafka

Hi Jorn, I was talking with the context of Hyperledger Fabric Blockchain where the cluster of kafka zookeeper is used where there might be multiple orgs taking part in the network and transactions where a single system getting failed or a malicious node might disrupt the whole network which would cost a lot. So on that aspect I was asking if its possible to have a pluggable algorithm for zookeeper ? Regards, Soumya -----Original Message----- From: Jörn Franke < jornfranke@gmail.com > Sent: Tuesday, August 27, 2019 11:03 AM To: users@kafka.apache.org Subject: Re: Byzantine Fault Tolerance Implementation What kind of stability problems do you have? It is surprising to me that you have them and it is unlikely that you have them due to a specific consensus algorithm. If you have stability issues then I would look at your architecture for weak spots. Btw. Paxos is a consensus mechanism. Bft just describes a specific type of failures in distributed systems, so ...

What kind of stability problems do you have? It is surprising to me that you have them and it is unlikely that you have them due to a specific consensus algorithm. If you have stability issues then I would look at your architecture for weak spots. Btw. Paxos is a consensus mechanism. Bft just describes a specific type of failures in distributed systems, so implement BFT does probably not make sense. > Am 27.08.2019 um 06:36 schrieb Nayak, Soumya R. < snayak@firstam.com >: > > Hi Team, > > Currently Zookeeper and Kafka cluster are Crash Fault Tolerant. > Zookeeper uses a version of Paxos - Zookeeper atomic broadcast. Is there any plan in future or current in progress where zookeeper will be implemented with a BFT algorithm. This might help to have a more stable distributed environment when we have the cluster across different machines. > > Regards, > Soumya > > ********************************************************************...

Hi, We have kafka brokers with kafka version 0.10.0.1. And brooklin( https://github.com/linkedin/brooklin ) is using kafka client - 2.0.1. I'm testing brooklin for kafka mirroring, Due to kafka version mismatch, Do you see any issue while using brooklin kafka-mirror with our kafka brokers? Or in general: Do you see any issue while using kafka client 2.0.1 with kafka broker 0.10.0.1? I went through below link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy but I'm not sure : is brooklin using such any new APIs that is not supported in kafka broker 0.10.0.1. If someone have time then please look into this and let me know. Thank you so much...

Hi Team, Currently Zookeeper and Kafka cluster are Crash Fault Tolerant. Zookeeper uses a version of Paxos - Zookeeper atomic broadcast. Is there any plan in future or current in progress where zookeeper will be implemented with a BFT algorithm. This might help to have a more stable distributed environment when we have the cluster across different machines. Regards, Soumya ****************************************************************************************** This message may contain confidential or proprietary information intended only for the use of the addressee(s) named above or may contain information that is legally privileged. If you are not the intended addressee, or the person responsible for delivering it to the intended addressee, you are hereby notified that reading, disseminating, distributing or copying this message is strictly prohibited. If you have received this message by mistake, please immediately notify us by replying to the message and delete t...

Awesome, thanks for clarifying :) On Tue, Aug 27, 2019 at 1:08 PM Guozhang Wang < wangguoz@gmail.com > wrote: > Right, the fix itself actually add more headers even if there were none > from the source topics, and hence cause old versioned brokers to fail. But > theoretically speaking, as long as the streams clients are version 0.11.0+ > the broker version should be 0.11.0+ for various features that may require > higher message format (eos, suppression, etc). > > On Mon, Aug 26, 2019 at 5:42 PM Sophie Blee-Goldman < sophie@confluent.io > > wrote: > > > I'm pretty sure one of the Suppress bug fixes that went into 2.2.1 > involved > > adding headers. Updating the compatibility matrix must have just slipped > > when that bugfix was merged -- thanks for bringing this up! > > > > On Mon, Aug 26, 2019 at 5:37 PM Alisson Sales < alisson.sales@gmail.com > > > wrote: > > > ...

Right, the fix itself actually add more headers even if there were none from the source topics, and hence cause old versioned brokers to fail. But theoretically speaking, as long as the streams clients are version 0.11.0+ the broker version should be 0.11.0+ for various features that may require higher message format (eos, suppression, etc). On Mon, Aug 26, 2019 at 5:42 PM Sophie Blee-Goldman < sophie@confluent.io > wrote: > I'm pretty sure one of the Suppress bug fixes that went into 2.2.1 involved > adding headers. Updating the compatibility matrix must have just slipped > when that bugfix was merged -- thanks for bringing this up! > > On Mon, Aug 26, 2019 at 5:37 PM Alisson Sales < alisson.sales@gmail.com > > wrote: > > > Hi Guozhang, thanks for your reply. > > > > I suspect the "problem" has to do with the fixes released on 2.2.1. I'm > > upgrading to this version mostly because we were ...

I'm pretty sure one of the Suppress bug fixes that went into 2.2.1 involved adding headers. Updating the compatibility matrix must have just slipped when that bugfix was merged -- thanks for bringing this up! On Mon, Aug 26, 2019 at 5:37 PM Alisson Sales < alisson.sales@gmail.com > wrote: > Hi Guozhang, thanks for your reply. > > I suspect the "problem" has to do with the fixes released on 2.2.1. I'm > upgrading to this version mostly because we were facing problems with > KTable suppress. > > I was experiencing this exact same problem: > > https://stackoverflow.com/questions/54145281/why-do-the-offsets-of-the-consumer-group-app-id-of-my-kafka-streams-applicatio/54227156 > This was the fix: https://issues.apache.org/jira/browse/KAFKA-7895 . > > When trying to confirm the fix worked for my topology/app I encountered the > issue: java.lang.IllegalArgumentException: Magic v1 does not support > record....