Kafka

Hi Anup, I agree that the documentation could be improved. Are you able to provide the steps you did try and the specific error message you're seeing? Thanks, Kirk On Thu, Mar 27, 2025, at 10:02 AM, Shirolkar, Anup wrote: > Hi Team, > > I am trying to setup Kafka authentication using OAuth with Okta as token provider. > I am following a few online blogs but it is not completely working. > > There is no comprehensive documentation about this that I can use. > Can someone please list down the steps needed for this. > > Thanks, > Anup Shirolkar. >

Thanks Maciej for the prompt response. But we tried the Percent encoding and unicode encoding but it did not work either. Please note that we are able to pass the special character via server.properties but not with the zookeeper config in the shell script. We tried putting environment variable in the server.properties and setting it's value with the decrypted password (containing special characters) from the shell script but it also didn't work. Due to the security limitations, we cannot pass the password in the plaintext in the server.properties file and we found no way to pass encrypted password via it. We uses our internal encryption/decryption mechanism which is running fine but unable to pass the decrypted password with special characters via the shell script. Please help us in this regards. Thanks, Deepak ________________________________ From: Maciej Malecki < emaciejm@gmail.com > Sent: Tuesday, March 25, 2025 9:51 PM To: users@kafka.apache.or...

Hi Team, I am trying to setup Kafka authentication using OAuth with Okta as token provider. I am following a few online blogs but it is not completely working. There is no comprehensive documentation about this that I can use. Can someone please list down the steps needed for this. Thanks, Anup Shirolkar.

Hej, Look here: https://en.m.wikipedia.org/wiki/Percent-encoding *__________________* *Regards,* *Maciej Małecki* wt., 25 mar 2025, 11:30 użytkownik Deepak Jain < deepak.jain@cumulus-systems.com > napisał: > Hi Luke, > > > > We are using Kafka 3.7.0 Broker/Client system in our prod environment with > SASL_SSL communication between Kafka Clients and Broker. We are starting > the Kafka process from the shell using the below command. > > > `nohup $EXEC_KAFKA_CONFIG --zookeeper 127.0.0.1:2181 --entity-type > brokers --entity-name 0 --alter --add-config $zooKeeperConfig >> > $KAFKA_HOME/logs/nohup_z.out 2>&1 &` > `nohup $KAFKA_HOME/bin/kafka-server-start.sh > $KAFKA_HOME/config/server.properties >> $KAFKA_HOME/logs/nohup_b.out 2>&1 &` > > > Here, we are passing the SSL Keystore and truststore password details in > $zooKeeperConfig as shown below: > > ...

Hi Luke, We are using Kafka 3.7.0 Broker/Client system in our prod environment with SASL_SSL communication between Kafka Clients and Broker. We are starting the Kafka process from the shell using the below command. `nohup $EXEC_KAFKA_CONFIG --zookeeper 127.0.0.1:2181 --entity-type brokers --entity-name 0 --alter --add-config $zooKeeperConfig >> $KAFKA_HOME/logs/nohup_z.out 2>&1 &` `nohup $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties >> $KAFKA_HOME/logs/nohup_b.out 2>&1 &` Here, we are passing the SSL Keystore and truststore password details in $zooKeeperConfig as shown below: zooKeeperConfig="listener.name.sasl_ssl.ssl.truststore.password=$KAFKA_SSL_KEYSTORE_AND_TRUSTSTORE_PWD,listener.name.sasl_ssl.ssl.keystore.password=$KAFKA_SSL_KEYSTORE_AND_TRUSTSTORE_PWD,listener.name.sasl_ssl.ssl.key.password=$KAFKA_SSL_KEYSTORE_AND_TRUSTSTORE_PWD,$KAFKA_SSL_PASSWORD_ENCODER_SECRET_PROP" Due to ...

Hello everyone, Can someone suggest here please. Regards, Sisindri M. On Fri, Mar 7, 2025 at 7:33 AM Manabolu Sisindri < manabolusisi@gmail.com > wrote: > > Hi Team, > > We're evaluating whether to migrate to *KRaft mode* (Kafka 3.9.0) from > our current Zookeeper-based setup. Given the stability of our current > system, do you recommend migrating now, or should we continue with > Zookeeper for the time being and plan the migration later? > > If we do migrate, can we consider *Kafka 3.9.0* as stable for production > workloads, or are there any known limitations or issues in KRaft at this > moment? > > Looking forward to your thoughts. > -- > Regards, > Sisindri, > 8317502751. >

Hi, Like Strimzi, Red Hat folks has its schema registry offering called Apicurio Registry which plays nicely with k8s through its operator. Currently Apicurio suit is getting regular updates. Good luck, OSB On Thu, Mar 20, 2025, 22:23 karan alang < karan.alang@gmail.com > wrote: > Hello - > > We have Strimzi Kafka installed on kubernetes (on GCP, as well as onPrem). > I'm looking for feedback on the Schema Registry options - > > 1. Confluent Schema Registry > 2. Karapace > 3. any others > > There is a strimzi registry operator as well, but that has not been updated > in the last 3 years > https://github.com/lsst-sqre/strimzi-registry-operator > > Has anyone used this ? > > Pls let me know. > > thanks! >

Kafka 4.0 here. I thought that KIP-687 < https://cwiki.apache.org/confluence/display/KAFKA/KIP-687%3A+Automatic+Reloading+of+Security+Store > was available but " ssl.keystore.location.refresh.interval.ms " seems to be silently ignored and not certificate reloading is done. KIP-687 is marked as "accepted" in 2020, but I haven't found any reference to it in the release notes of any Kafka release. Is it actually available in Kafka 4.0? If not, what is people doing to refresh certificates under Kraft, beside manualy updating the certificate files and rebooting the brokers/controllers? Thanks. -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea@jcea.es - https://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" ...

Hello - We have Strimzi Kafka installed on kubernetes (on GCP, as well as onPrem). I'm looking for feedback on the Schema Registry options - 1. Confluent Schema Registry 2. Karapace 3. any others There is a strimzi registry operator as well, but that has not been updated in the last 3 years https://github.com/lsst-sqre/strimzi-registry-operator Has anyone used this ? Pls let me know. thanks!

Hello I had a Kafka v3.8 running in Kubernetes with 3 replicas each of broker and zookeeper. After upgrading to v3.9, I am attempting to migrate away from zookeeper to kraft. So,       broker.zookeeperMigrationMode: true       broker.minId: 0       controller.zookeeperMigrationMode: true       controller.minId: 100       kraft.isenabled: true       zookeeper.isenabled: true. I see in the log error messages "INCONSISTENT_CLUSTER_ID in FETCH response". Migration is never started or completed (Zk_MigrationState value is 2). I have checked contents of meta.properties and I see that the " cluster.id " is same in all PODs. The value matches what exists in the zookeeper. I have also deleted meta.properties file from all PODs and restarted them and I still see the error. 1. Where is the discrepancy? ClusterId was never changed. 2. Why is the error message not printing the clusterId's being compared? https://github.com/apache/kafka/blo...

Congrats!!! Thanks David for the release and big thanks to all contributors!! Best regards Dave Canton On Wed, 19 Mar 2025 at 09:14, andreasvdberghe@avathar.be < andreasvdberghe@avathar.be > wrote: > Well done! I'm looking forward to learning what's changed. > > On 19 Mar 2025, at 9:03 AM, Satish Duggana < satish.duggana@gmail.com > > wrote: > > Amazing release!! Thanks to all the contributors, and to David(as RM) > for running the release. > > On Tue, 18 Mar 2025 at 21:13, David Jacot < dajac@apache.org > wrote: > > > > The Apache Kafka community is pleased to announce the release for > > Apache Kafka 4.0.0 > > > > - This is the first release without Apache Zookeeper > > - The Next Generation of the Consumer Rebalance Protocol is Generally > Available > > - The Transactions Server-Side Defense (Phase 2) is Generally Available > > - Kafka Clients and Kaf...

Congrats to the entire community! Really excited to try out all these new features. --- Regards, Steve gradient fox - Web UI for Apache Kafka www.gradientfox.io On 2025-03-18 11:43, David Jacot wrote: > The Apache Kafka community is pleased to announce the release for > Apache Kafka 4.0.0 > > - This is the first release without Apache Zookeeper > - The Next Generation of the Consumer Rebalance Protocol is Generally > Available > - The Transactions Server-Side Defense (Phase 2) is Generally Available > - Kafka Clients and Kafka Streams require Java 11, while Kafka > Brokers, Connect, and Tools, now require Java 17 > - Old client protocol API versions have been removed > - Queues for Kafka is in Early Access > - Kafka uses log4j2 > > All of the changes in this release can be found in the release notes: > https://www.apache.org/dist/kafka/4.0.0/RELEASE_NOTES.html > > An overview of the release can be f...

Oh also the other update, this happened with a cluster that is on 3.8.1. I still suspect a client bug rather than broker bug though. On Wed, Mar 19, 2025 at 9:01 AM Donny Nadolny < donny@stripe.com > wrote: > Here's the jira: https://issues.apache.org/jira/browse/KAFKA-19012 > > I have a bit of new information as well: > - to your comment Alex, in our case the original message is not published > to the intended topic as it happened for you. Instead, the message only > ends up on the wrong topic > - Ismael: the client had permission to publish to any topic. It first > started happening around 4 months ago, because of the low frequency it's > tough to point to what changed that it's now triggered. The biggest recent > changes (upgrading broker version, upgrading kafka client version) all > happened > 6 months before it occurred. We did have other smaller changes > of varying closeness in time, but none of them ...

Here's the jira: https://issues.apache.org/jira/browse/KAFKA-19012 I have a bit of new information as well: - to your comment Alex, in our case the original message is not published to the intended topic as it happened for you. Instead, the message only ends up on the wrong topic - Ismael: the client had permission to publish to any topic. It first started happening around 4 months ago, because of the low frequency it's tough to point to what changed that it's now triggered. The biggest recent changes (upgrading broker version, upgrading kafka client version) all happened > 6 months before it occurred. We did have other smaller changes of varying closeness in time, but none of them seem to be the culprit (code changes that increased the load average on the client machine -> we've scaled out to lower the load and had no chance in frequency, various code changes we temporarily reverted or deactivated but still saw occurrences, etc). Thanks, Do...

Well done! I'm looking forward to learning what's changed. On 19 Mar 2025, at 9:03 AM, Satish Duggana < satish.duggana@gmail.com > wrote: Amazing release!! Thanks to all the contributors, and to David(as RM) for running the release. On Tue, 18 Mar 2025 at 21:13, David Jacot < dajac@apache.org > wrote: > > The Apache Kafka community is pleased to announce the release for > Apache Kafka 4.0.0 > > - This is the first release without Apache Zookeeper > - The Next Generation of the Consumer Rebalance Protocol is Generally Available > - The Transactions Server-Side Defense (Phase 2) is Generally Available > - Kafka Clients and Kafka Streams require Java 11, while Kafka > Brokers, Connect, and Tools, now require Java 17 > - Old client protocol API versions have been removed > - Queues for Kafka is in Early Access > - Kafka uses log4j2 > > All of the changes in this release can be found in the release not...

Amazing release!! Thanks to all the contributors, and to David(as RM) for running the release. On Tue, 18 Mar 2025 at 21:13, David Jacot < dajac@apache.org > wrote: > > The Apache Kafka community is pleased to announce the release for > Apache Kafka 4.0.0 > > - This is the first release without Apache Zookeeper > - The Next Generation of the Consumer Rebalance Protocol is Generally Available > - The Transactions Server-Side Defense (Phase 2) is Generally Available > - Kafka Clients and Kafka Streams require Java 11, while Kafka > Brokers, Connect, and Tools, now require Java 17 > - Old client protocol API versions have been removed > - Queues for Kafka is in Early Access > - Kafka uses log4j2 > > All of the changes in this release can be found in the release notes: > https://www.apache.org/dist/kafka/4.0.0/RELEASE_NOTES.html > > An overview of the release can be found in our announcement blog post: > htt...

On 2025-03-18 23:43, David Jacot wrote: > The Apache Kafka community is pleased to announce the release for > Apache Kafka 4.0.0 > > - This is the first release without Apache Zookeeper Now kafka can work without zookeeper? or we need to install zookeeper separately? > - Kafka Clients and Kafka Streams require Java 11, while Kafka > Brokers, Connect, and Tools, now require Java 17 if the system has only one java installed, so java 17 is the must? Thank you for your work. regards.

Well done to the community and David (as RM) for this release! It's a great milestone on the journey to a simpler and better Apache Kafka. For our users and the developers too. Ismael On Tue, Mar 18, 2025 at 8:44 AM David Jacot < dajac@apache.org > wrote: > The Apache Kafka community is pleased to announce the release for > Apache Kafka 4.0.0 > > - This is the first release without Apache Zookeeper > - The Next Generation of the Consumer Rebalance Protocol is Generally > Available > - The Transactions Server-Side Defense (Phase 2) is Generally Available > - Kafka Clients and Kafka Streams require Java 11, while Kafka > Brokers, Connect, and Tools, now require Java 17 > - Old client protocol API versions have been removed > - Queues for Kafka is in Early Access > - Kafka uses log4j2 > > All of the changes in this release can be found in the release notes: > https://www.apache.org/dist/kafka/4.0.0/RELEASE_NOT...

The simultaneous launch of Java 24 and Kafka 4.0 is a landmark event for engineers. This day should be recognized as a national holiday for our profession :) thanks to David for his outstanding work! Best, Chia-Ping Bruno Cadonna < cadonna@apache.org > 於 2025年3月18日 週二 下午11:48寫道： > Thank you, David! Well done! > > Best, > Bruno > > On 18.03.25 16:43, David Jacot wrote: > > The Apache Kafka community is pleased to announce the release for > > Apache Kafka 4.0.0 > > > > - This is the first release without Apache Zookeeper > > - The Next Generation of the Consumer Rebalance Protocol is Generally > Available > > - The Transactions Server-Side Defense (Phase 2) is Generally Available > > - Kafka Clients and Kafka Streams require Java 11, while Kafka > > Brokers, Connect, and Tools, now require Java 17 > > - Old client protocol API versions have been removed > > - Queues for K...

Thank you, David! Well done! Best, Bruno On 18.03.25 16:43, David Jacot wrote: > The Apache Kafka community is pleased to announce the release for > Apache Kafka 4.0.0 > > - This is the first release without Apache Zookeeper > - The Next Generation of the Consumer Rebalance Protocol is Generally Available > - The Transactions Server-Side Defense (Phase 2) is Generally Available > - Kafka Clients and Kafka Streams require Java 11, while Kafka > Brokers, Connect, and Tools, now require Java 17 > - Old client protocol API versions have been removed > - Queues for Kafka is in Early Access > - Kafka uses log4j2 > > All of the changes in this release can be found in the release notes: > https://www.apache.org/dist/kafka/4.0.0/RELEASE_NOTES.html > > An overview of the release can be found in our announcement blog post: > https://kafka.apache.org/blog#apache_kafka_400_release_announcement > > You can downl...

The Apache Kafka community is pleased to announce the release for Apache Kafka 4.0.0 - This is the first release without Apache Zookeeper - The Next Generation of the Consumer Rebalance Protocol is Generally Available - The Transactions Server-Side Defense (Phase 2) is Generally Available - Kafka Clients and Kafka Streams require Java 11, while Kafka Brokers, Connect, and Tools, now require Java 17 - Old client protocol API versions have been removed - Queues for Kafka is in Early Access - Kafka uses log4j2 All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/4.0.0/RELEASE_NOTES.html An overview of the release can be found in our announcement blog post: https://kafka.apache.org/blog#apache_kafka_400_release_announcement You can download the source and binary release (Scala 2.13) from: https://kafka.apache.org/downloads#4.0.0 -----------------------------------------------------------------------------------...

This vote passes with 10 +1 votes (4 bindings) and no 0 or -1 votes. +1 votes PMC Members: * David Arthur * Bill Bejeck * Justine Olshan * Luke Chen Committers: * Lianet Magrans * Andrew Schofield Community: * Federico Valeri * TengYao Chi * Jiunn-Yang * Paolo Patierno 0 votes * No votes -1 votes * No votes Vote thread: https://lists.apache.org/thread/czxrsdsk9nw03rp7bzqhdjbt2wzhtjll I'll continue with the release process and the release announcement will follow. David Jacot

Hi David, we ran all the automated regression pipelines by using the Kafka 4.0.0 RC4 binaries within the next planned Strimzi 0.46.0 release (Kafka on Kubernetes). Everything seems to be fine so this RC4 looks good to me. +1 non binding Thanks for running the release. Cheers, Paolo On Fri, 14 Mar 2025 at 11:09, David Jacot < dajac@apache.org > wrote: > Hello Kafka users, developers and client-developers, > > This is the fourth candidate for release of Apache Kafka 4.0.0. > > - This is the first release without Apache Zookeeper > - The Next Generation of the Consumer Rebalance Protocol is Generally > Available > - The Transactions Server-Side Defense (Phase 2) is Generally Available > - Queues for Kafka is in Early Access > - Kafka uses log4j2 > - Drop broker and tools support for Java 11 > - Remove old client protocol API versions > > Release notes for the 4.0.0 release: > https://dist.apache.org/repos...

Hi David, I verified the new RC with the following steps: - Checked system tests run: there were 2 new failures in connect that we didn't have in the previous RC (test_broker_compatibility and test_pause_state_persistent). I ran them locally and they both pass (so based on this and the error msg seems like just flakiness) - Build from src with java 17, followed the quickstart and played with consume/produce & topic management with the command line tools - Run client examples from the examples module (java 17) - Tested consuming in groups with both classic and new consumer protocol - Tested the consumer group online migration & the command line new tools to visualize/track the live migration - Build clients with java 11 , run unit and integration tests. - Run client examples from the examples module (java 11) +1 (non-binding) Thanks! Lianet On Fri, Mar 14, 2025 at 9:28 AM David Arthur < mumrah@gmail.com > wrote: > David, thanks for running...