Kafka

Hello Kafka users, developers and client-developers, This is the third candidate for release of Apache Kafka 4.1.0. The release brings many new features and improvements: - Mechanism for plugin to register metrics - Allow running multiple versions of connector plugins - Queue are now in preview access - New Streams rebalance protocol in early access - Support for OAuth jwt-bearer grant type - Improved metadata replication Release notes for the 4.1.0 release: https://dist.apache.org/repos/dist/dev/kafka/4.1.0-rc2/RELEASE_NOTES.html *** Please download, test and vote by Friday August 8. Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://dist.apache.org/repos/dist/dev/kafka/4.1.0-rc2/ * Docker release artifacts to be voted upon: apache/kafka:4.1.0-rc2 apache/kafka-native:4.1.0-rc2 * Maven artifacts to be voted upon: https://repository.ap...

Hi Luke and community! Luke, thank you for being proactive and caring about this topic! In the meantime we have been keeping ourselves busy pushing our implementation of KIP-1150 to production to validate our assumptions and confirm its strengths while discovering its weaknesses. Now, after gathering some experience running it, we are (as I'm writing this, gathered in the same room) working on an improved proposal for KIP-1150 that also addresses the concerns from the community. We expect to share the updated KIP in the next couple of weeks. We apologize for the recent period of silence and are committed to more regular communication as we move forward. Best, On Tue, Aug 5, 2025 at 10:31 AM Luke Chen < showuon@gmail.com > wrote: > Hi all, > > The Kafka community is currently seeing an unprecedented situation with > three KIPs (KIP-1150, IP-1176, KIP-1183) simultaneously addressing the same > challenge of high replication costs whe...

Hi all, The Kafka community is currently seeing an unprecedented situation with three KIPs (KIP-1150, IP-1176, KIP-1183) simultaneously addressing the same challenge of high replication costs when running Kafka across multiple cloud availability zones. Each KIP offers a different solution to this issue. While diversity of innovative ideas is a key strength of open-source projects, it creates a burden for reviewers and users who must compare and comment on multiple proposals simultaneously. Furthermore, discussion around the three KIPs has stalled for over two months now. This could be due to the authors being hesitant to proceed due to the existence of alternative, potentially conflicting, solutions. Addressing replication cost is a key concern of Kafka's userbase and we should try to move the conversation forward if we can. From what I understand, these three KIPs are not mutually exclusive. But adopting all three KIPs in the community might not be what we expect...

Hi all, Here's the update to the decisions: > 2. Could we merge this PR < https://github.com/apache/kafka/pull/20269 > even though this will change the user interface? I think this is definitely a bug fix, not a public API change. So no KIP needed. Any objections? Will treat it as a bug, no KIP needed. 3. We should include this patch in kafka v4.0.1. Will be included in v4.0.1. 4. Should we include this patch in kafka v4.1.0, even though the v4.1.0 RC1 is out for vote now? Will be included in v4.1.0. Thank you. Luke On Fri, Aug 1, 2025 at 11:07 AM Chia-Ping Tsai < chia7712@gmail.com > wrote: > hi Luke > > Thanks for raising this discussion > > > Luke Chen < showuon@gmail.com > 於 2025年8月1日 上午9:42 寫道： > > > > 1. Notify users using kafka v4.0.0 about this typo bug, in case you are > > trying to grep state logs using regex or something. > > IIRC, we had a similar discussion about l...

Dear Apache Kafka Users, I hope you're doing well. I'm encountering an issue with our Kafka setup where the consumer stops receiving messages after running for a period of time, even though: - The Kafka producer continues to send payloads successfully - The consumer remains connected and shows no visible errors - Restarting the consumer causes it to start receiving messages again temporarily - The messages are confirmed to be in the Kafka topic (e.g., via kafka-console-consumer) - There are no signs of consumer crashes or disconnections in the logs Our Kafka version is ruby-kafka (1.5.0) as we have integrated it into the Ruby on Rails application. This behavior appears intermittent, and we're unsure whether it's related to offset management, consumer group behavior, or something else. We would greatly appreciate your help in identifying what could be causing this issue, or any suggestions on ad...

Greetings, TLDR: are there any known problems with replacing reload4j with log4j2 directly on the CLASSPATH for Apache Kafka 3.9.1? Our application runs Apache Kafka 3.9.1 and there is a thought to replace the jar of reload4j with the jar of log4j2 on the CLASSPATH when starting the Apache Kafka cluster, could anyone, please share their experience on the matter and what problems such a replacement might lead to? I perused the source code of Apache Kafka 3.9.1 and I found many `import`s of `org.apache.log4j.*` in Kafka Connect, does it mean that Mirror Maker 2's logger can malfunction? Main main concern with log4j2 is that such a migration only happened in Apache Kafka 4.0 almost 10 years after log4j1.2 was declared End of Life (August 2015). Does anyone remember if there have been any obstacles to migrate to log4j2 earlier? I know of the Log4Shell (CVE-2021-44228) vulnerability in log4j2, but it was only reported in 2021, almost 6 years after log4j1.2 was decla...

hi Luke Thanks for raising this discussion > Luke Chen < showuon@gmail.com > 於 2025年8月1日 上午9:42 寫道： > > 1. Notify users using kafka v4.0.0 about this typo bug, in case you are > trying to grep state logs using regex or something. IIRC, we had a similar discussion about log prefix. "Should we keep the log output format, such as the prefix, for users who might grep logs using regex" That is a gray area, but I agree that is not a part of public APIs. Best, Chia-Ping