Kafka

-----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIzBAEBCgAdFiEE8osu2CcCCF5douGQu8PBaGu5w1EFAl1qBbQACgkQu8PBaGu5 w1GEqA//V4RPxW+3kWPDLFBFpZknXbjleNXmOxJs1Qh+/yxRmMeJMHcO5SYCSsAa Seans1WEtCQAgiiWbC3sXGfrQLmQ30UxSYq7gK7RwEZCRlHC2axIP/FVk/DMGup9 WDzeix/DrQMvuTFmhjHU/aweRhTtZoqp5JWYqTMFeW78AJOINpfnIBxBl9hkHcon zHmc4CZH5qfzCP2VXmbztK7n/jccX2qB5mIQAHe2vdy0U2ZygiXd09mTZKxulCyQ FJyUjODmZrHpRvLgI0eFtw9sLmAfcurS8ywWBNL5wdr5DtnbQlnHVxJoaennPcq3 ZWFj+4api3ljABXhN9jZh7aV6XbYnHOcmD0CBnENYw6DgDH5f9HaS8CjjjEggIi0 F0MAMlP5EzduMjwEahAi1jSPoe9RVcTziIeSp7b1cSqj1d34fWR+uTRryhurhWc7 m4cZyusp1HjcDSqnEq3QGckRakN7yg16snNHLBMP50BNRynDYXxhrH4vk+89FhcU WgArlnrcl9L6GPcx08s3tE5kNgaweJz+e00O2YgjfH1ohKlopbi0VfDHaIx4InQ8 MFysnDT/clo6IseIIpGZi5H1BAJT6LbiVMgtQB5Czpx289XPtU1NXdbhjDKn83sC IFCaZ+p1xZ1oRDzMGKrriEkMmJqDaSsgUWoEH8OvYQv7ugCgn34= =dKRp -----END PGP SIGNATURE----- Exactly-once on the producer will only ensure that no duplicate writes happen. If a downstream consumer fai...

For a producer that emits messages to a single topic (i.e. no single message is sent to multiple topics), will enabling idempotency but not transactions provide exactly once guarantees for downstream consumers of said topic? Ordering is not important I just want to make sure consumers only consumer messages sent once.

Hi, I am trying to authenticate with "super" user - admin as per the above configuration. Is this possible with only "SASL_SSL" listeners? Or should I have a "PLAIN" listener as well to authenticate with super user account? Thanks On Thu, Aug 29, 2019 at 8:43 PM Antony A < antonyaugustus@gmail.com > wrote: > Hi, > > I have configured the brokers and zookeepers as below to enable SSL and > authentication with SASL/Kerberos. I have tried with and without > advertised.listeners, advertised.host.name , host.name , port. > > *server.properties* > > listeners=SASL_SSL://<hostname>:9092 > advertised.listeners=SASL_SSL://<hostname>:9092 > > advertised.host.name =<hostname> > host.name =<hostname> > port=9092 > > ssl.truststore.location=kafka.server.truststore.jks > ssl.truststore.password=password > ssl.keystore.location=kafka.server.keystore.jks ...

Hi, I have configured the brokers and zookeepers as below to enable SSL and authentication with SASL/Kerberos. I have tried with and without advertised.listeners, advertised.host.name , host.name , port. *server.properties* listeners=SASL_SSL://<hostname>:9092 advertised.listeners=SASL_SSL://<hostname>:9092 advertised.host.name =<hostname> host.name =<hostname> port=9092 ssl.truststore.location=kafka.server.truststore.jks ssl.truststore.password=password ssl.keystore.location=kafka.server.keystore.jks ssl.keystore.password=password ssl.key.password=password security.inter.broker.protocol=SASL_SSL sasl.mechanism.inter.broker.protocol=GSSAPI sasl.enabled.mechanisms=GSSAPI sasl.kerberos.service.name =HTTP allow.everyone.if.no.acl.found=true zookeeper.set.acl=true authorizer.class.name =kafka.security.auth.SimpleAclAuthorizer super.users=User:kafka;User:admin *zookeeper.properties* authProvider.1=org.apache.zookeeper.serve...

Will, the idea is to have a DNS name or VIP which resolves to IP addresses from one cluster or the other depending on health checks. You can have your producers load-balanced 50/50 between two clusters, or could have a backup cluster that is only used when the primary cluster is unreachable, etc. When a producer times-out trying to send to the current set of IP address, it will try to resolve the DNS name again and receive a new set of IP addresses. For this to work, you may need to tweak various producer properties, e.g. retries, timeouts, refresh intervals. > Or is it more that the load balancer acts as more of a service discovery point for clients? Yes, that's an apt description. Ryanne On Thu, Aug 29, 2019 at 7:47 AM Will Weber < rwaweber@gmail.com > wrote: > Hey Ryanne, thanks for the feedback. > > Would you mind going into more detail with how that would work? > > My understanding is that the Kafka wire protocol isn...

Hello, is it possible to revert the log.message.format.version when we find out that a consumer doesn't work with the new version? I read in the documentation that setting this in the config would tell Kafka that all messages in the logs are not higher than that number, so I think we would have to clean all logs/topics when reverting this? Thanks Sebastian -- DISCLAIMER This email contains information that is confidential and which may be legally privileged. If you have received this email in error please notify the sender immediately and delete the email. This email is intended solely for the use of the intended recipient and you may not use or disclose this email in any way.

What exactly is the problem? Kafka states it in documentation and it is not new. Quote from docs: max.in.flight.requests.per.connection – The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled). The possible problem is only with some people not able to read documentation. 29 авг. 2019 г., в 17:11, Jose Manuel Vega Monroy < jose.monroy@williamhill.com <mailto: jose.monroy@williamhill.com >> написал(а): Hi there, Recently we found this blog entry about a possible problem with messages order: https://blog.softwaremill.com/does-kafka-really-guarantee-the-order-of-messages-3ca849fd19d2 < https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.softwaremill.com%2Fdoes-kafka-really-guarantee-the-order-of-messages-3ca849fd19d2&da...

Hi the description of that blog is correct. as the default value of retries has been change to max_int from 0, then the order is not guaranteed in single partition any more by default even if records was send from same producer. the following description come from "Kafka: The Definitive Guide" Apache Kafka preserves the order of messages within a partition. This means that if messages were sent from the producer in a spe‐ cific order, the broker will write them to a partition in that order and all consumers will read them in that order. For some use cases, order is very important. There is a big difference between deposit‐ ing $100 in an account and later withdrawing it, and the other way around! However, some use cases are less sensitive. Setting the retries parameter to nonzero and the max.in.flights.requests.per.session to more than one means that it is possible that the broker will fail to write the first batch of messages, succeed to write the sec...

Hi there,   Recently we found this blog entry about a possible problem with messages order: https://blog.softwaremill.com/does-kafka-really-guarantee-the-order-of-messages-3ca849fd19d2   Please could you confirm about? And if so, how to fix? Blog entry suggesting how to.   Thanks   Jose Manuel Vega Monroy  Java Developer / Software Developer Engineer in Test Direct: + 0035 0 2008038 (Ext. 8038) Email:  jose.monroy@williamhill.com William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA      

Hey Ryanne, thanks for the feedback. Would you mind going into more detail with how that would work? My understanding is that the Kafka wire protocol isn't really intended to be loadbalanced. Or is it more that the load balancer acts as more of a service discovery point for clients? I've seen in other threads that LinkedIn typically fronts their clusters with a load balancer, but I haven't found much information on that strategy and the associated tradeoffs. Best, On Thu, Aug 29, 2019 at 12:36 AM Ryanne Dolan < ryannedolan@gmail.com > wrote: > Will and Garvit, you can use a load balancer with health checks for this > purpose. > > Ryanne > > On Wed, Aug 28, 2019, 6:09 PM Will Weber < rwaweber@gmail.com > wrote: > > > Apologies for piggybacking on a thread, figured the discussion was pretty > > relevant to a thought I had kicking around my brain. > > > > In the event of complete failure o...

Hello all, we have a question about librdkafka - Kafka version : 2.0.1 - librdkafka version : 0.11.4 We have an application connecting to kafka using librdkafka. It needs to consume a topic starting at a specific offset value stored in the application. It does so by calling rd_kafka_seek(). In standard cases, everything is fine; however in some specific use cases, the value stored in our application is greater that the latest offset in kafka. But we are not able to detect these cases as rd_kafka_seek() is not returning any error when it is passed out-of-range offsets. Does anyone have experience with librdkafka and could tell how to either : - get an error when we seek() to an out-of-range offset, - or retrieve the max offset in kafka before performing seek()? Thanks Aurelien

Did you see the warning "Error connecting to node" on consumer log? Best, Lisheng Hrishikesh Mishra < sd.hrishi@gmail.com > 于2019年8月29日周四 下午2:45写道： > Please find my reply in blue colour: > > > > On Thu, Aug 29, 2019 at 11:32 AM Lisheng Wang < wanglisheng81@gmail.com > > wrote: > > > Hi > > > > about question 1, it's dosen't matter that how many consumers in same > > consumer group. > > > > So you means the broker which is coordinator did not crashed at all > before? > > > > We didn't see any shutdown error on Brokers & we faced similar problem > with multiple coordinators. > > > > > May i know if only exact one broker(coordinator) is unavailable or many > > are? if only exact one, you can try to transfer leader of > _consumer_offset > > which on that broker to another broker to see if it's no problem any ...

Why is this group being targeted by the false political propaganda? The sender did not even use a valid email account. On Thu, Aug 29, 2019 at 7:05 AM ant_fighter <ant_fighter@protonmail.com.invalid> wrote: > Hi all, > Sorry for disturbing you guys. Though I don't think here as a proper place > to do this, I need your help, your vote, your holy vote, for us Chinese, > for conscience and justice, for better world. > > In the over 70 years of ruling over China, the Chinese Communist Party has > done many horrible things humans can think of. These malicious and evil > deeds include but are not limited to: falsifying national history, > suppression of freedom of speech and press, money laundering in the scale > of trillions, live organ harvesting, sexual harassment and assault to > underaged females, slaughtering innocent citizens with > counter-revolutionary excuses, etc. > > In light of the recent violent actions to...

Please find my reply in blue colour: On Thu, Aug 29, 2019 at 11:32 AM Lisheng Wang < wanglisheng81@gmail.com > wrote: > Hi > > about question 1, it's dosen't matter that how many consumers in same > consumer group. > > So you means the broker which is coordinator did not crashed at all before? > We didn't see any shutdown error on Brokers & we faced similar problem with multiple coordinators. > May i know if only exact one broker(coordinator) is unavailable or many > are? if only exact one, you can try to transfer leader of _consumer_offset > which on that broker to another broker to see if it's no problem any more? > > It happened with multiple consumer groups. > i found the following issue seems similar with yours, FYR: > > > https://stackoverflow.com/questions/51952398/kafka-connect-distributed-mode-the-group-coordinator-is-not-available > We have gone thro...

Hi None of your image can be shown, actually, mail server aggressively strips attachment. Best, Lisheng 荣益丰 < ryf841203739@163.com > 于2019年8月29日周四 下午12:47写道： > > > > > > > -------- 转发邮件信息 -------- > 发件人："荣益丰" < ryf841203739@163.com > > 发送日期：2019-08-29 08:31:40 > 收件人： users-subscribe@kafka.apache.org > 主题：kafka poll阻塞怎么解决 > > > > > > >

Hi about question 1, it's dosen't matter that how many consumers in same consumer group. So you means the broker which is coordinator did not crashed at all before? May i know if only exact one broker(coordinator) is unavailable or many are? if only exact one, you can try to transfer leader of _consumer_offset which on that broker to another broker to see if it's no problem any more? i found the following issue seems similar with yours, FYR: https://stackoverflow.com/questions/51952398/kafka-connect-distributed-mode-the-group-coordinator-is-not-available Best, Lisheng Hrishikesh Mishra < sd.hrishi@gmail.com > 于2019年8月29日周四 下午12:19写道： > Hi, > > We are facing following issues with Kafka cluster. > > - Kafka Version: 2.0.0 > - We following cluster configuration: > - Number of Broker: 14 > - Per Broker: 37GB Memory and 14 Cores. > - Topics: 40 - 50 > - Partitions per topic: 32 > ...

Hi all, Sorry for disturbing you guys. Though I don't think here as a proper place to do this, I need your help, your vote, your holy vote, for us Chinese, for conscience and justice, for better world. In the over 70 years of ruling over China, the Chinese Communist Party has done many horrible things humans can think of. These malicious and evil deeds include but are not limited to: falsifying national history, suppression of freedom of speech and press, money laundering in the scale of trillions, live organ harvesting, sexual harassment and assault to underaged females, slaughtering innocent citizens with counter-revolutionary excuses, etc. In light of the recent violent actions to Hong Kongers by the People's Liberation Army (PLA) disguised as Hong Kong Police Force, we the people petition to officially recognize the Chinese Communist Party as a terrorist organization. PLEASE SIGNUP and VOTE for us： https://petitions.whitehouse.gov/petition/call-official-recognition-chinese-...

Will and Garvit, you can use a load balancer with health checks for this purpose. Ryanne On Wed, Aug 28, 2019, 6:09 PM Will Weber < rwaweber@gmail.com > wrote: > Apologies for piggybacking on a thread, figured the discussion was pretty > relevant to a thought I had kicking around my brain. > > In the event of complete failure or sustained loss of connectivity of the > first cluster, could the secondary cluster act as a failover for a given > configuration? > > Assuming the respective clusters are set up like the following: > > Cluster A > brokerA01:9092 > brokerA02:9092 > brokerA03:9092 > > Cluster B > brokerB01:9092 > brokerB02:9092 > brokerB03:9092 > > And the corresponding producer's properties line contained something like: > > > bootstrap.servers="brokerA01:9092,brokerA02:9092,brokerA03:9092,brokerB01:9092,brokerB02:9092,brokerB03:9092" > > My initial th...

Hi, We are facing following issues with Kafka cluster. - Kafka Version: 2.0.0 - We following cluster configuration: - Number of Broker: 14 - Per Broker: 37GB Memory and 14 Cores. - Topics: 40 - 50 - Partitions per topic: 32 - Replicas: 3 - Min In Sync Replica: 2 - __consumer_topic partition: 50 - offsets.topic.replication.factor=3 - default.replication.factor=3 - Consumers#: ~4000 (will grow to ~7K) - Consumer Groups#: ~4000 (will grow to ~7K) Imp: Here one consumer is consuming from one topic and one consumer group has only one consumer due to some architectural constraints. Two major problems we are facing with consumer group: - First time when we are starting consumer with new group name it working very well. But subsequent restart (with previous / older group name) is causing problems from some consumers. We are getting following errors: INFO [2019-08-28 19:05:34,481] [main] [AbstractCoord...

Hi Koushik Seems there is something lead to didn't replicated in time, so the follower was kicked out from ISR. May i know if you have any chance can lead to that issue, e.g. the throughput is too high to can not complete replication in time or there is record can not replicated to follower as some configs fo max size not proper? Best, Lisheng Koushik Chitta <kchitta@microsoft.com.invalid> 于2019年8月29日周四 上午2:03写道： > Hi All, > > We had a topic partition(with 5 replication) going offline when leader of > the partition was down. Below is some analysis > > Kafka server - 1.1 , relevant config ( replica.fetch.wait.max.ms = 500, > replica.fetch.min.bytes = 50000, replica.lag.time.max.ms =10000) > Topic partition (Test.Request-3) - replication 5 , Replica List [17, > 425222741, 425222681, 423809494,425222740] , unclean leader election = > false > > Sequence of events. > > > 1. Leader(425222740) o...

-------- 转发邮件信息 -------- 发件人："荣益丰" <ryf841203739@163.com> 发送日期：2019-08-29 08:31:40 收件人：users-subscribe@kafka.apache.org 主题：kafka poll阻塞怎么解决    

Apologies for piggybacking on a thread, figured the discussion was pretty relevant to a thought I had kicking around my brain. In the event of complete failure or sustained loss of connectivity of the first cluster, could the secondary cluster act as a failover for a given configuration? Assuming the respective clusters are set up like the following: Cluster A brokerA01:9092 brokerA02:9092 brokerA03:9092 Cluster B brokerB01:9092 brokerB02:9092 brokerB03:9092 And the corresponding producer's properties line contained something like: bootstrap.servers="brokerA01:9092,brokerA02:9092,brokerA03:9092,brokerB01:9092,brokerB02:9092,brokerB03:9092" My initial theory is that a failover action would respond like: 1. Attempting to reach brokerA01:9092 with a metadata fetch, cannot reach, attempt to reach next broker 2. Attempting to reach brokerA02:9092 with a metadata fetch, cannot reach, attempt to reach next broker 3. Attempting to reach brokerA0...

Hi All, We had a topic partition(with 5 replication) going offline when leader of the partition was down. Below is some analysis Kafka server - 1.1 , relevant config ( replica.fetch.wait.max.ms = 500, replica.fetch.min.bytes = 50000, replica.lag.time.max.ms =10000) Topic partition (Test.Request-3) - replication 5 , Replica List [17, 425222741, 425222681, 423809494,425222740] , unclean leader election = false Sequence of events. 1. Leader(425222740) of the partition is down. 1. Controller detects the offline broker. [2019-08-26 13:00:22,037] INFO [Controller id=423809469] Newly added brokers: , deleted brokers: 425222740, all live brokers: 15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,423809443,423809444,423809450,423809458,423809463,423809464,423809469,423809474,423809494,425218574,425222675,425222681,425222741,425222745 (kafka.controller.KafkaController) 1. Controller sends update metadata request , but observes only leader in the isr. Plea...

Hello, I just had a weird problem in one of my topics. It accepted messages from producers but consumers couldn't see those messages. The producers didn't get any errors when sending messages to the topic, but even my test-console-consumer didn't see anything in the topic. Also checking the topic with the kafka-consumer-groups.sh using the option of --describe returned a TimeoutException on failing to get offsets in 30001ms. My quick and dirty solution was to delete the topic and with auto-creation enabled is was recreated and messages started showing up to consumersand --describe working again. As I have no idea where to start to find the issue of this, and then of course open an issue on the website, I'd need some hints on what to look for.... Thank you Sebastian -- DISCLAIMER This email contains information that is confidential and which may be legally privileged. If you have received this email in error please not...

Hi Community, I am accessing events from the enterorise Kafka topics via Windows command line via keytab successfully. However, I have transitioned to macbook this week and the same set of commands are not working in Mac. Any documentation on how to connect to kafka topic via macbook and using keytab file? In windows, I first use the below environment variables for setup and then proceed with Kafka consuming 0) Specify where jaas.conf file is located. Need to set an environment variable. KAFKA_OPTS= -Djava.security.auth.login.config=C:\myfolder\Kafka_Configs\jaas.conf KAFKA_REALM= -Djava.security.krb5.conf=krb5.conf 1) Authenticate app id to Kerberos with ticket cached on server kinit -kt C:\myfolder\Kafka_Configs\keytab_file.keytab your_kerberos_principal_name@domain.COM<mailto: your_kerberos_principal_name@domain.COM > 2) Listen to a message on a topic <path>/kafka-console-consumer.<sh/bat> --bootstrap-server <server1:6668, server2:666...

Hi all, I put together a draft release plan with Oct 2019 as the release month and a list of KIPs that have already been voted: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=125307901 Here are the dates: 1) KIP Freeze: Sep 25, 2019 (A KIP must be accepted by this date in order to be considered for this release) 2) Feature Freeze: Oct 2, 2019 (Major features merged & working on stabilization, minor features have PR, release branch cut; anything not in this state will be automatically moved to the next release in JIRA. 3) Code Freeze: Oct 16, 2019 4) Release Date: Oct 30, 2019 (tentative) Please plan accordingly for the features you want push into Apache Kafka 2.4.0 release. Regards, Manikumar On Mon, Aug 12, 2019 at 9:08 PM Ismael Juma < ismael@juma.me.uk > wrote: > Thanks for volunteering Manikumar. +1 > > Ismael > > On Mon, Aug 12, 2019 at 7:54 AM Manikumar < manikumar.reddy@gmail.com > ...

FYI: I changed the dependencies of brooklin to use kafka 0.10.0.1. But compilation is getting failed. 1st failure related test case where it trying to create KafkaServer, where constructor args mismatch. I have changed KafkaServer creation API to old one, but another set of issues: brooklin having LIKafkaVersion - 1.0.4, and due to this it is downloading kafka-client 2.0.1. so I have changed LIKafkaVersion to 0.0.6(where dependency kafka-client 0.10.1.1). once again, while preparing build test cases are failing: some where related to kafka topic creation test cases. So now I have 2 options: 1. move my kafka brokers with 2.0.1. not easy because have multiple dependencies. 2. or develop my own kafka-mirror or try some other open-source projects. If you know any good kafka mirror opensource projects then please let me know. Thank you. On Tue, Aug 27, 2019 at 10:11 AM Upendra Yadav < upendra1024@gmail.com > wrote: > Hi, > > We have kafka...

Hi Eliza, I used the wrong word stability. Rather it should be a byzantine fault tolerant network. Regards, Soumya -----Original Message----- From: Eliza <eli@ChinaBuckets.com> Sent: Tuesday, August 27, 2019 12:55 PM To: users@kafka.apache.org Subject: Re: Byzantine Fault Tolerance Implementation Hi on 2019/8/27 12:36, Nayak, Soumya R. wrote: > Zookeeper uses a version of Paxos - Zookeeper atomic broadcast. Is there any plan in future or current in progress where zookeeper will be implemented with a BFT algorithm. This might help to have a more stable distributed environment when we have the cluster across different machines. Why do you think BFT algorithm will improve the stability of the cluster? :) regards. ****************************************************************************************** This message may contain confidential or proprietary information intended only for the use of the addressee(s) named above or may contain information ...

Hi Liam, Thank you for you kind reply. I will check it out to see if this helps. There are amost 5K topics in our cluster with 6 nodes. We collect the metrics by collectd ( https://collectd.org/ ) and visualize with Grafana. If all these per topic metrics are collected, will it affect the performance of the cluster? Haijun Liam Clarke < liam.clarke@adscale.co.nz > 于2019年8月27日周二 下午4:15写道： > Hi, > > The broker exposes per topic error rates, that might help? > > https://docs.confluent.io/current/kafka/monitoring.html#per-topic-metrics > > Kind regards, > > Liam Clarke > > > On Tue, 27 Aug. 2019, 8:07 pm 李海军, < lihaijun0629@gmail.com > wrote: > > > Hi all, > > > > We have a kafka cluster consisted of 6 brokers with the version of > > kafka_2.11-0.10.2.1. These days we saw a continuous increasing on the > > metrics of "BytesRejectedPerSec" on broker 1. I already kne...

Hi, The broker exposes per topic error rates, that might help? https://docs.confluent.io/current/kafka/monitoring.html#per-topic-metrics Kind regards, Liam Clarke On Tue, 27 Aug. 2019, 8:07 pm 李海军, < lihaijun0629@gmail.com > wrote: > Hi all, > > We have a kafka cluster consisted of 6 brokers with the version of > kafka_2.11-0.10.2.1. These days we saw a continuous increasing on the > metrics of "BytesRejectedPerSec" on broker 1. I already knew that this > could happen when the message records were too large and raise a > "RecordTooLargeException" on the client side, which could be seen in the > client's log by using "Callback()" in the send method or just adapting the > log level to debug. > > However, from the perspective of the server side, there are no logs about > the exceptions at all. Since there are too many producers involved in the > clusters and checking all the cli...

Hi all, We have a kafka cluster consisted of 6 brokers with the version of kafka_2.11-0.10.2.1. These days we saw a continuous increasing on the metrics of "BytesRejectedPerSec" on broker 1. I already knew that this could happen when the message records were too large and raise a "RecordTooLargeException" on the client side, which could be seen in the client's log by using "Callback()" in the send method or just adapting the log level to debug. However, from the perspective of the server side, there are no logs about the exceptions at all. Since there are too many producers involved in the clusters and checking all the client logs is out of the question, how could I find these certain topics causing "BytesRejectedPerSec" in the server logs? Thanks in advance. Sinsecrely, Haijun

Hi on 2019/8/27 12:36, Nayak, Soumya R. wrote: > Zookeeper uses a version of Paxos - Zookeeper atomic broadcast. Is there any plan in future or current in progress where zookeeper will be implemented with a BFT algorithm. This might help to have a more stable distributed environment when we have the cluster across different machines. Why do you think BFT algorithm will improve the stability of the cluster? :) regards.

Thanks Boyang. Will check that link and update there. Regards, Soumya -----Original Message----- From: Boyang Chen < reluctanthero104@gmail.com > Sent: Tuesday, August 27, 2019 11:50 AM To: users@kafka.apache.org Subject: Re: Byzantine Fault Tolerance Implementation Hey Nayak, there is an on-going KIP in the community about deprecating zookeeper: https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum It should be a good place to raise your question about making consensus algorithm pluggable in the future. Boyang On Mon, Aug 26, 2019 at 10:41 PM Nayak, Soumya R. < snayak@firstam.com > wrote: > Hi Jorn, > > I was talking with the context of Hyperledger Fabric Blockchain where > the cluster of kafka zookeeper is used where there might be multiple > orgs taking part in the network and transactions where a single system > getting failed or a malicious node mig...

Hey Nayak, there is an on-going KIP in the community about deprecating zookeeper: https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum It should be a good place to raise your question about making consensus algorithm pluggable in the future. Boyang On Mon, Aug 26, 2019 at 10:41 PM Nayak, Soumya R. < snayak@firstam.com > wrote: > Hi Jorn, > > I was talking with the context of Hyperledger Fabric Blockchain where the > cluster of kafka zookeeper is used where there might be multiple orgs > taking part in the network and transactions where a single system getting > failed or a malicious node might disrupt the whole network which would cost > a lot. > So on that aspect I was asking if its possible to have a pluggable > algorithm for zookeeper ? > > Regards, > Soumya > -----Original Message----- > From: Jörn Franke < jornfranke@gmail.com > ...