Kafka

To all Kafka users, There is a discussion thread [0] in the dev channel about `streams-scala`. Could you please share your feedback there? The main topic is to deprecate the module now and remove it in 5.0. [0] https://lists.apache.org/thread/jpqpoxq3jr22hjpwrskzvwcjfm0j0z22 Best, Chia-Ping

Hello Kafka users, developers, and client-developers, This is the third candidate for the release of Apache Kafka 4.1.1. This is a bug-fix release, including several critical ones. All listed in the release notes. Release notes for the 4.1.1 release: https://dist.apache.org/repos/dist/dev/kafka/4.1.1-rc2/RELEASE_NOTES.html *** Please download, test, and vote by Friday, Oct 31th, 9:00 AM EDT Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://dist.apache.org/repos/dist/dev/kafka/4.1.1-rc2/ * Docker release artifacts to be voted upon: apache/kafka:4.1.1-rc2 https://hub.docker.com/layers/apache/kafka/4.1.1-rc2/images/sha256-7240ff4534bd23dac2f215ba03a2d0aa9d041b45b830804bbdec3b81c2bdf479 apache/kafka-native:4.1.1-rc2 https://hub.docker.com/layers/apache/kafka-native/4.1.1-rc2/images/sha256-017871b0ef2ad665030fc66596b4b9ce583c7faee530829d17d02...

Yes, Java 21 support started with Kafka 4.0. See KAFKA-17086  ---- On Mon, 27 Oct 2025 07:08:12 -0400 srebecchi@kameleoon.com.INVALID wrote ---- Hello Can we use Java 21 to run Apache Kafka 4.1.0 on production environment? Thanks, Sébastien.

Hello Can we use Java 21 to run Apache Kafka 4.1.0 on production environment? Thanks, Sébastien.

Hi Chia. The gap seems sensitive enough, so I think it makes sense to include the minor fix if we can stabilize it shortly. Follow-up on the PR. Thanks! Lianet On Thu, Oct 23, 2025 at 4:10 PM Chia-Ping Tsai < chia7712@gmail.com > wrote: > hi Lianet > > Would you mind taking a look at https://github.com/apache/kafka/pull/20759 > ? > > Best, > Chia-Ping > > Lianet M. < lianetmr@gmail.com > 於 2025年10月24日 週五 上午2:43寫道： > > > Hello Kafka users, developers, and client-developers, > > > > This is the second candidate for the release of Apache Kafka 4.1.1. > > > > This is a bug-fix release, including several critical ones. All listed > > in the release notes. > > > > Release notes for the 4.1.1 release: > > > https://dist.apache.org/repos/dist/dev/kafka/4.1.1-rc1/RELEASE_NOTES.html > > > > *** Please download, test, and vote by Wednesday, Oct 29th, 9:0...

hi Lianet Would you mind taking a look at https://github.com/apache/kafka/pull/20759 ? Best, Chia-Ping Lianet M. < lianetmr@gmail.com > 於 2025年10月24日 週五 上午2:43寫道： > Hello Kafka users, developers, and client-developers, > > This is the second candidate for the release of Apache Kafka 4.1.1. > > This is a bug-fix release, including several critical ones. All listed > in the release notes. > > Release notes for the 4.1.1 release: > https://dist.apache.org/repos/dist/dev/kafka/4.1.1-rc1/RELEASE_NOTES.html > > *** Please download, test, and vote by Wednesday, Oct 29th, 9:00 AM EDT > > Kafka's KEYS file containing PGP keys we use to sign the release: > https://kafka.apache.org/KEYS > > * Release artifacts to be voted upon (source and binary): > https://dist.apache.org/repos/dist/dev/kafka/4.1.1-rc1/ > > * Docker release artifacts to be voted upon: > apache/kafka:4.1.1-rc1 > > http...

Hello Kafka users, developers, and client-developers, This is the second candidate for the release of Apache Kafka 4.1.1. This is a bug-fix release, including several critical ones. All listed in the release notes. Release notes for the 4.1.1 release: https://dist.apache.org/repos/dist/dev/kafka/4.1.1-rc1/RELEASE_NOTES.html *** Please download, test, and vote by Wednesday, Oct 29th, 9:00 AM EDT Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://dist.apache.org/repos/dist/dev/kafka/4.1.1-rc1/ * Docker release artifacts to be voted upon: apache/kafka:4.1.1-rc1 https://hub.docker.com/layers/apache/kafka/4.1.1-rc1/images/sha256-262d880caa26f510bdbc890b9a2d03a7c4f24f22194e6e19fe35637a3073663b apache/kafka-native:4.1.1-rc1 https://hub.docker.com/layers/apache/kafka-native/4.1.1-rc1/images/sha256-3a094cb1f462c23ee56e887ba410441956537ab2a6c7d4ae1...

Agreed Chia. I checked and this is indeed a noisy regression. I will prepare a new RC. Thanks! Lianet On Wed, Oct 22, 2025 at 8:50 PM Chia-Ping Tsai < chia7712@gmail.com > wrote: > > hi Lianet > > Thanks for the quick and effective RC preparation! > > Sorry for the bad news, but could you please include commit ` > https://github.com/apache/kafka/commit/4099f75e559f4581302a0de0a26cf0fd4e371eda` > in 4.1.1? It is a small but critical patch that removes a warning message > that might alarm users > Best, > Chia-Ping > > Lianet M. < lianetmr@gmail.com > 於 2025年10月23日 週四 上午3:43寫道： > > > Hello Kafka users, developers, and client-developers, > > > > This is the first candidate for the release of Apache Kafka 4.1.1. > > > > This is a bug-fix release, including several critical ones. All listed > > in the release notes. > > > > Release notes for the 4.1.1 release: ...

hi Lianet Thanks for the quick and effective RC preparation! Sorry for the bad news, but could you please include commit ` https://github.com/apache/kafka/commit/4099f75e559f4581302a0de0a26cf0fd4e371eda` in 4.1.1? It is a small but critical patch that removes a warning message that might alarm users Best, Chia-Ping Lianet M. < lianetmr@gmail.com > 於 2025年10月23日 週四 上午3:43寫道： > Hello Kafka users, developers, and client-developers, > > This is the first candidate for the release of Apache Kafka 4.1.1. > > This is a bug-fix release, including several critical ones. All listed > in the release notes. > > Release notes for the 4.1.1 release: > https://dist.apache.org/repos/dist/dev/kafka/4.1.1-rc0/RELEASE_NOTES.html > > *** Please download, test, and vote by Tuesday, Oct 28th, 9:00 AM EDT > > Kafka's KEYS file containing PGP keys we use to sign the release: > https://kafka.apache.org/KEYS > > * Relea...

Hello Kafka users, developers, and client-developers, This is the first candidate for the release of Apache Kafka 4.1.1. This is a bug-fix release, including several critical ones. All listed in the release notes. Release notes for the 4.1.1 release: https://dist.apache.org/repos/dist/dev/kafka/4.1.1-rc0/RELEASE_NOTES.html *** Please download, test, and vote by Tuesday, Oct 28th, 9:00 AM EDT Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://dist.apache.org/repos/dist/dev/kafka/4.1.1-rc0/ * Docker release artifacts to be voted upon: apache/kafka:4.1.1-rc0 https://hub.docker.com/layers/apache/kafka/4.1.1-rc0/images/sha256-f6c7291be9e6942713ba47a51d8201d60dcd315d2db8063e8f807f21f1184d16 apache/kafka-native:4.1.1-rc0 https://hub.docker.com/layers/apache/kafka-native/4.1.1-rc0/images/sha256-2a9866a9eb1845c399e6e6344abe1bf3114b030e05de7a9f1760...

Hi, In our cluster (across many), rarely I see a deleted key reappearing during consume. This happens in kafka 3.2 and also in 3.9 (2 versions used in various cluster) Example: Key:abc (offset 100) - sent at T1 Tombstone for key:abc - sent at T1 + 2 days (Tombstone delete.retention.ms is 1 day) Consume at T1 + 4 days (rehydration - consume from beginning) - No key Consume at T1 + 6 days (rehydration - consume from beginning) - key:abc appears with same offset(100) These number of days varies but I get this issue approximately 1% of time (not consistent) Anybody faced this issue and any solution recommended. My topic config is below Partition=10 Replication-factor=5 min.insync.replicas=2 cleanup.policy=compact segment.bytes=15728640 min.compaction.lag.ms =900000 max.compaction.lag.ms =1800000 delete.retention.ms =86400000 retention.ms =259200000 message.timestamp.type=CreateTime Regards, Nanda

Hello. We're building KafkaPilot < https://softwaremill.github.io/kafkapilot/ > – a tool that proactively detects and resolves common issues in Apache Kafka before they become production headaches. Right now, KafkaPilot handles 17 real-world fault scenarios related to configuration, availability, and performance. Now, we'd love your input to make it truly useful for Kafka practitioners like you: - What kinds of Kafka incidents tend to hit you the hardest? - What do you want a tool to auto-fix or explain when things break? - Anything else Kafka-related that drives you crazy? Help us create a tool that will make your life much easier in the future: https://softwaremill.github.io/kafkapilot/ -- Szymon Dyrlaga Senior Content Marketer ------------------------------ szymon.dyrlaga@softwaremill.com < https://softwaremill.com/portfolio-clients/ > DELIVERING SOFTWARE REMOTELY SINCE 2009 < https://clutch.co/profile...

Hi, I was able to reproduce an issue with different source connector plugins, so the issue seems to be in a Connect itself. Got the issue with the next versions: 1. Connector plugin: com.nordstrom.kafka.connect.sqs.SqsSourceConnector 1.5.0 Kafka Connect cluster version: 3.9.0 2. Connector plugin: Debezium MySQL source 1.9.7 Kafka Connect cluster version: 3.9.0 When producer client for some particular task is overloaded, changing connector config leads to missing MBeans with metrics for particular client-id. How I reproduced it: 1. Created a SQS source connector configured to have a high throughput, so it's able to generate SourceRecord faster than Connect Worker can put them to Kafka. This leads to producer buffer to run out of free space, produce average request latency to increase significantly (30ms -> 3 seconds), average record time in a queue to also increase significantly (50ms -> 1.5 minutes). 2. Using jconsole to check producer MBeans st...

The Apache Kafka community is pleased to announce the release for Apache Kafka 4.0.1 This bug-fix has over 40 improvements since 4.0.0! All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/4.0.1/RELEASE_NOTES.html An overview of the release can be found in our announcement blog post: https://kafka.apache.org/blog#apache_kafka_401_release_announcement You can download the source and binary release (Scala 2.13) from: https://kafka.apache.org/downloads#4.0.1 --------------------------------------------------------------------------------------------------- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream of records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them. ** The Streams API allows an application to ac...

Hello, I've noticed that the `org.apache.kafka.clients.producer.internals.TransactionManager` class has generated ~30k log lines in the last hour and that means ~100mb (structured logging with json). I can of course simply disable the logs of this class or set the log level to be minimum warning, but I don't know if I could miss important information if/when things go wrong. Most of the log lines look like this: [ProducerId set to 250075021 with epoch 0 [Discovered transaction coordinator xyz.amazonaws.com:9098 (id: 2 rack: null) Invoking InitProducerId for the first time in order to acquire a producer ID So, I'd like to ask: 1. Is it ok to lower the log level of this class to warning? 2. Is there the possibility that I'm doing something wrong and that's why I see so many of those logs? If yes, what could that be? Thank you Best regards, Renann

Hi Is there a good method to collect kafka metrics via command line instead of using grafana or console etc .. ? Thanks Rajesh Rajesh G Naicken - Senior Software Engineer - Appliance Security and DR Architect IBM Data & AI www: www.ibm.com Email: rnaicken@in.ibm.com , Ph: +91 9177753780 "Do your best to make sure your actions affect others in a positive way" - Tri Hita Karana

> As far as running, as opposed to building, we run Kafka Client and Streams > on JDK 25 in testing so far and haven't noticed any significant regressions. thanks to the great JDK team :smile

Hi Paweł, Here's a blog I wrote a while back about Kafka Queues in 4.0, hope that helps: https://www.instaclustr.com/blog/apache-kafka-4-0-share-groups-what-you-need-to-know-about-queues-for-kafka/ Regards, Paul Brebner From: Paweł Szymczyk < pawel.szymczyk90@gmail.com > Date: Monday, 6 October 2025 at 5:08 am To: users@kafka.apache.org < users@kafka.apache.org > Subject: Queues for Kafka Early Access Release Notes confusion EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Hi guys I am playing right now with Shared Consumer and I am following the Queues for Kafka (KIP-932) - Early Access Release Notes < https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/Queues*for*Kafka*(KIP-932)*-*Early*Access*Release*Notes__;KysrKysrKys!!Nhn8V6BzJA!XtokZwuvZKeX19tspnPXo1gWhFAoTUOK9855rNz3351xr_jL1SbBedJqxgXIIVu9Epsaw6kxK787UHKrxXvTN5bsuSegBXA6$ > manual. To make it fly I had to add group.share.enable=t...

Hi guys I am playing right now with Shared Consumer and I am following the Queues for Kafka (KIP-932) - Early Access Release Notes < https://cwiki.apache.org/confluence/display/KAFKA/Queues+for+Kafka+(KIP-932)+-+Early+Access+Release+Notes > manual. To make it fly I had to add group.share.enable=true configuration parameter to the server.properties file, this is not mentioned in the manual. Without this configuration kafka-console-share-consumer.sh is throwing and exception: pawel@DESKTOP-3T909GL:~/kafka_2.13-4.1.0$ bin/kafka-console-share-consumer.sh --bootstrap-server localhost:9092 --topic quickstart-events [2025-10-05 20:02:22,464] WARN Share groups and KafkaShareConsumer are part of a preview feature introduced by KIP-932, and are not recommended for use in production. (org.apache.kafka.clients.consumer.internals.ShareConsumerDelegateCreator) [2025-10-05 20:02:22,863] ERROR [ShareConsumer clientId=console-share-consumer, groupId=console-share-consumer] Sh...

Hello Edgar, The comparability matrix was only added to the docs for 4.0 release (something way overdue...), as 4.0 contains major changes. I assume you are talking about broker version 3.9.1. For this case, all clients, even back to 0.x, and also all newer clients are supported. There might be some broker features, that older clients cannot use of course, as older client got developed before these newer broker features got added, but 3.9.1 brokers are fully backward compatible to older clients. Apache Kafka 4.0.0 is the first release, which drop supports for older clients, and the minimum supported client version is 2.1 now. This KIP might help: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1124%3A+Providing+a+clear+Kafka+Client+upgrade+path+for+4.x Please let us know if you have any more specific follow up questions. -Matthias On 10/2/25 11:23 PM, Zubel, Edgar wrote: > Hello! > > I was looking for kafka-client and kafka-b...

> On Oct 2, 2025, at 7:26 PM, Yeikel Santana < email@yeikel.com > wrote: > > Awesome, thank you for the update! I'll track https://github.com/apache/kafka/pull/19513 in the meantime > > ---- On Thu, 02 Oct 2025 01:46:54 -0400 Chia-Ping Tsai < chia7712@apache.org > wrote --- > > > > hi Yeikel > > We are upgrading the build tool to JDK 25 [0]. and we will then upgrade the CI accordingly. I expect Kafka 4.2 to support the Java 25 runtime. > > [0] https://github.com/apache/kafka/pull/19513 > > Best, > Chia-Ping > > On 2025/09/19 17:20:33 Yeikel Santana wrote: >> Hi team, >> >> >> What's the current status of the Java 25 support for the project? >> >> >> Is it safe to deploy Kafka and Connect using a Java 25 runtime? >> As far as running, as opposed to building, we run Kafka Client and Streams on JDK 25 in testing s...

Hello! I was looking for kafka-client and kafka-broker API version compatibility matrix and unfortunately didn't find that information anywhere. Kafka 4.0.0 has some sort of compatibility matrix provided in the documentation, but I'm interested in version 3.9.1. Is there a way to find that out? My assumption here would be, that broker provided API version support – SupportedMinVersion: 3.0-IV1 SupportedMaxVersion: 3.9-IV0 , would mean, that clients from version 3.0.X to 3.9.X are supported? Please help me out here. Thanks! Best regards, Edgar Zubel DevOps Engineer This email may contain information which is privileged or protected against unauthorized disclosure or communication. If you are not the intended recipient, please notify the sender and delete this message and any attachments from your system without producing, distributing or retaining copies thereof or disclosing its contents to any other person. Telia Company processes emails and other files ...

Awesome, thank you for the update! I'll track  https://github.com/apache/kafka/pull/19513  in the meantime ---- On Thu, 02 Oct 2025 01:46:54 -0400 Chia-Ping Tsai < chia7712@apache.org > wrote --- hi Yeikel We are upgrading the build tool to JDK 25 [0]. and we will then upgrade the CI accordingly. I expect Kafka 4.2 to support the Java 25 runtime. [0] https://github.com/apache/kafka/pull/19513 Best, Chia-Ping On 2025/09/19 17:20:33 Yeikel Santana wrote: > Hi team, > > > What's the current status of the Java 25 support for the project? > > > Is it safe to deploy Kafka and Connect using a Java 25 runtime? > > > Thank you

hi Nirjas Kafka 3.9.1 supports JDK 21[0]. Sorry about the outdated website. I have filed KAFKA-19751 to trace it. Thanks for the report!! [0] https://github.com/apache/kafka/commit/99e6f12dd09952fccde8f4826b65cadab332c2dc Best, Chia-Ping On 2025/09/08 06:11:07 Nirjas Jakilim wrote: > Hi the doc here https://kafka.apache.org/39/documentation.html#java didn't > mention jdk21 but 3.9.1 does support jdk21. Also, the official docker image > here uses jdk21 for kafka 3.9.1 > https://hub.docker.com/layers/apache/kafka/3.9.1/images/sha256-5862db4a63a6dd7d46fd14771b10a1b39e069c2c47f17d8e4640f960720a0ead > > so can you please verify if jdk21 is supported on kafka 3.9.1 or not? If > yes is it possible to update the doc mentioning jdk21? > Thanks > Regards, > Nirjas >

hi Yeikel We are upgrading the build tool to JDK 25 [0]. and we will then upgrade the CI accordingly. I expect Kafka 4.2 to support the Java 25 runtime. [0] https://github.com/apache/kafka/pull/19513 Best, Chia-Ping On 2025/09/19 17:20:33 Yeikel Santana wrote: > Hi team, > > > What's the current status of the Java 25 support for the project? > > > Is it safe to deploy Kafka and Connect using a Java 25 runtime? > > > Thank you

On 2025/09/19 17:20:33 Yeikel Santana wrote: > Hi team, > > > What's the current status of the Java 25 support for the project? > > > Is it safe to deploy Kafka and Connect using a Java 25 runtime? > > > Thank you