Kafka

Hi Mickael, I did the following: - Built from source with Java 17 - Validated all checksums and signatures - Ran unit tests - Spot checked the docs and java doc - Ran the broker quickstarts including the docker images - Ran the Kafka Streams quickstart - Used classic and the new streams mode - Ran through the instructions for the Early access of the Streams Rebalance Protocol +1(binding) Thanks, Bill On Fri, Aug 29, 2025 at 12:36 PM Mickael Maison < mickael.maison@gmail.com > wrote: > Hi, > > The docs PR has been merged so the documentation and protocol for 4.1 are > up: > - https://kafka.apache.org/41/documentation.html > - https://kafka.apache.org/41/protocol.html > > The system tests have completed. > I still got the network_degrade_test failure (tracked by > https://issues.apache.org/jira/browse/KAFKA-19549 ). > Also I reverted by mistake the temporary fix [0] I had made to ...

Hi, The docs PR has been merged so the documentation and protocol for 4.1 are up: - https://kafka.apache.org/41/documentation.html - https://kafka.apache.org/41/protocol.html The system tests have completed. I still got the network_degrade_test failure (tracked by https://issues.apache.org/jira/browse/KAFKA-19549 ). Also I reverted by mistake the temporary fix [0] I had made to the 4.1 branch for https://issues.apache.org/jira/browse/KAFKA-19581 , so the Streams upgrade tests failed in the results, but they pass locally. - Core: https://drive.google.com/file/d/1-XdhozO1t78vWW4z127lweNa45DiYbmd/view?usp=drive_link - Non core: https://drive.google.com/file/d/1BLydIgKGBsV6B0UtnKJMNKNwAgz0ABGy/view?usp=drive_link 0: https://github.com/apache/kafka/commit/de16dd103af93bb68a329987ff19469941f85cbc Thanks, Mickael On Fri, Aug 29, 2025 at 12:54 PM Paolo Patierno < paolo.patierno@gmail.com > wrote: > > We ran all regression tests with Strimzi and ever...

We ran all regression tests with Strimzi and everything looks good. +1 (non binding) Thanks, Paolo On Wed, 27 Aug 2025 at 14:39, Mickael Maison < mickael.maison@gmail.com > wrote: > Hello Kafka users, developers and client-developers, > > This is the 4th candidate for release of Apache Kafka 4.1.0. > > The release brings many new features and improvements: > - Mechanism for plugin to register metrics > - Allow running multiple versions of connector plugins > - Queue are now in preview access > - New Streams rebalance protocol in early access > - Support for OAuth jwt-bearer grant type > - Improved metadata replication > > Release notes for the 4.1.0 release: > https://dist.apache.org/repos/dist/dev/kafka/4.1.0-rc3/RELEASE_NOTES.html > > *** Please download, test and vote by Monday September 1 > > Kafka's KEYS file containing PGP keys we use to sign the release: > https://kafka.apache.org...

Hello Kafka users, developers and client-developers, This is the 4th candidate for release of Apache Kafka 4.1.0. The release brings many new features and improvements: - Mechanism for plugin to register metrics - Allow running multiple versions of connector plugins - Queue are now in preview access - New Streams rebalance protocol in early access - Support for OAuth jwt-bearer grant type - Improved metadata replication Release notes for the 4.1.0 release: https://dist.apache.org/repos/dist/dev/kafka/4.1.0-rc3/RELEASE_NOTES.html *** Please download, test and vote by Monday September 1 Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://dist.apache.org/repos/dist/dev/kafka/4.1.0-rc3/ * Docker release artifacts to be voted upon: apache/kafka:4.1.0-rc3 apache/kafka-native:4.1.0-rc3 * Maven artifacts to be voted upon: https://repository.ap...

This was interesting - some lessons (re)learned about using micro benchmarks and how producer linger and batching work - I wrote a quick and nasty article here https://www.linkedin.com/pulse/increased-lingering-apache-kafka-40-can-increase-latency-paul-brebner-z7qlc/ Regards, Paul From: Daniel Germon <daniel.germon@lmax.com.INVALID> Date: Saturday, 23 August 2025 at 2:27 am To: users@kafka.apache.org < users@kafka.apache.org > Subject: Re: Re: Increased latency when upgrading brokers to V4.0.0? [You don't often get email from daniel.germon@lmax.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Hi Sean, Thank you for the suggestion, reducing the linger to zero has indeed removed all the differences in latencies between the two versions! I managed to skim over that bit of the release notes so will have a deeper look into the KIP to se...

Hi everyone, I'd like to reinitiate the conversation about KIP-977. This KIP was approved and scheduled for release 3.8.0 but appears not to have been shipped. At Salesforce, we're exploring implementing this KIP (draft PR: [ https://github.com/apache/kafka/pull/20397](https://github.com/apache/kafka/pull/20397) ) and are particularly interested in its performance implications. I envision two possible implementations: 1. The approach in our draft PR checks verbosity rules for every request to determine which metrics to emit. This could lead to performance degradation if the rules are complex or if there's a large cluster with many topics, especially when wildcards are involved. Each topic in a request would need to be checked against the rules every time. 2. Alternatively, we could build and maintain a map of topics to metrics in memory for quicker lookups. This map would need to be updated when verbosity configurations change (which is straightforw...

Hi Sean, Thank you for the suggestion, reducing the linger to zero has indeed removed all the differences in latencies between the two versions! I managed to skim over that bit of the release notes so will have a deeper look into the KIP to see if there's anything else that could be tuned. Thanks again, Dan This message and its attachments are confidential, may not be disclosed or used by any person other than the addressee and are intended only for the named recipient(s). If you are not the intended recipient, please notify the sender immediately and delete any copies of this message. LMAX Group is the holding company of LMAX Exchange, LMAX Global and LMAX Digital. Our registered address is Yellow Building, 1A Nicholas Road, London W11 4AN.

Hi Daniel, Kafka 4.0 includes a rewrite of the group coordinator with a new batching mechanism. The batching interval defaults to 5 ms and is controlled by the ` group.coordinator.append.linger.ms ` broker config option. Could you try lowering this config and rerunning the benchmarks? Thanks, Sean On 2025/08/21 15:50:58 Daniel Germon wrote: > Hi Greg, > > I have changed the linger.ms configuration on the producer and yes it explains the difference in client version latencies, but I am still seeing the broker latency differences with this configuration. > > Kafka broker 3.9.1, client 4.0.0, linger.ms = 0 > > 50th percentile: 310us > 90th percentile: 414us > 99th percentile: 825us > 99.9th percentile: 9813us > > Kafka broker 4.0.0, client 4.0.0, linger.ms = 0 > > 50th percentile: 8193us > 90th percentile: 10373us > 99th percentile: 11109us > 99.9th percentile: 13812us > > > Regards, > ...

Hi Daniel, I don't have any changes in mind that might be affecting the latency increase from a broker upgrade. Since you're using replication factor 1, it shouldn't be related to the replication protocol or inter-broker network latency. Are you able to take a profile [1] of the leader broker and make some flame graphs to compare 3.9.1 and 4.0.0? I hope that a 20x difference would be visible on the Wall profile. Thanks, Greg [1] https://github.com/async-profiler/async-profiler On Thu, Aug 21, 2025 at 8:53 AM Daniel Germon <daniel.germon@lmax.com.invalid> wrote: > Hi Greg, > > I have changed the linger.ms configuration on the producer and yes it > explains the difference in client version latencies, but I am still seeing > the broker latency differences with this configuration. > > Kafka broker 3.9.1, client 4.0.0, linger.ms = 0 > > 50th percentile: 310us > 90th percentile: 414us > 99th percentile: ...

Hi Greg, I have changed the linger.ms configuration on the producer and yes it explains the difference in client version latencies, but I am still seeing the broker latency differences with this configuration. Kafka broker 3.9.1, client 4.0.0, linger.ms = 0 50th percentile: 310us 90th percentile: 414us 99th percentile: 825us 99.9th percentile: 9813us Kafka broker 4.0.0, client 4.0.0, linger.ms = 0 50th percentile: 8193us 90th percentile: 10373us 99th percentile: 11109us 99.9th percentile: 13812us Regards, Dan This message and its attachments are confidential, may not be disclosed or used by any person other than the addressee and are intended only for the named recipient(s). If you are not the intended recipient, please notify the sender immediately and delete any copies of this message. LMAX Group is the holding company of LMAX Exchange, LMAX Global and LMAX Digital. Our registered address is Yellow Building, 1A Nicholas Road, London W11 4AN.

Hi all, There were several configuration defaults that were changed with the release of 4.0.0 [1], I wonder if one or more of them could be affecting your observed end-to-end latency. Notably, the linger.ms on the producer side has increased from 0ms to 5ms. You may consider running 4.0.0 clients but with linger.ms set to 0 to see if that resolves this latency regression. Thanks, Greg [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations On Thu, Aug 21, 2025 at 3:40 AM Daniel Germon <daniel.germon@lmax.com.invalid> wrote: > Hi Paul, > > Thank you for taking the time to reply! I have just ran the benchmarks > using the v4.0.0 clients and if anything the latencies are actually worse > than on client v3.9.1: > > > Kafka broker 3.9.1 client 3.9.1 > > 50th percentile: 297 us > 90th percentile: 366 us > 99th percentile: 528 us...

Hi Paul, Thank you for taking the time to reply! I have just ran the benchmarks using the v4.0.0 clients and if anything the latencies are actually worse than on client v3.9.1: Kafka broker 3.9.1 client 3.9.1 50th percentile: 297 us 90th percentile: 366 us 99th percentile: 528 us 99.9th percentile: 1,460 us Kafka broker 3.9.1 client 4.0.0 50th percentile: 3,295 us 90th percentile: 5,418 us 99th percentile: 5,712 us 99.9th percentile: 6,602 us Kafka broker 4.0.0 client 3.9.1 50th percentile: 8,178 us 90th percentile: 10,355 us 99th percentile: 11,042 us 99.9th percentile: 13,638 us Kafka broker 4.0.0 client 4.0.0 50th percentile: 10,889 us 90th percentile: 13,914 us 99th percentile: 15,750 us 99.9th percentile: 18,167 us I did run the internal kafka-e2e-latency tool however did not see any noticeable difference, I believe this is because the tool is waiting for a record to be received before sending anoth...

For our combined kraft setup, We recently migrated the kafka broker and controller authentication mechanisms as follow: Broker: from sasl.mechanism.inter.broker.protocol="PLAIN" to "SCRAM-SHA-512" Controller: from SASL_SSL to SSL. After applying these changes, we restarted the Kafka pods in Kubernetes. However, once the pods came up, the brokers were unable to authenticate with each other. When we deleted the Kafka PVC and restarted the pods, they ran successfully — suggesting that the old inter-broker credentials were still being persisted in the PVC. Is this an expected behaviour? or are we missing something while implementing new auth mechanism

Hi Daniel, I wonder if using client version 3.9.1 with 4.0.0 could be the problem? I recall that to get all the benefits of the new 4.0 protocols (e.g. the next gen consumer rebalancing protocol) you need to use 4.0 clients - e.g. see this blog https://www.instaclustr.com/blog/rebalance-your-apache-kafka-partitions-with-the-next-generation-consumer-rebalance-protocol/ I will also check our internal benchmarking to see if there's anything odd ... Regards, Paul Brebner From: Daniel Germon <daniel.germon@lmax.com.INVALID> Date: Wednesday, 20 August 2025 at 8:22 pm To: users@kafka.apache.org < users@kafka.apache.org > Subject: Increased latency when upgrading brokers to V4.0.0? [You don't often get email from daniel.germon@lmax.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Hello, We have recently upgraded our Kafka brokers to...

Hello, We have recently upgraded our Kafka brokers to version 4.0.0 and have noticed that our latencies are substantially worse than on 3.9.1 (both versions running with KRaft). We have created a small project that demonstrates the issue. It has a single producer and consumer and measures how long it takes between the producer sending the record and the consumer pulling it off (both clients are using version 3.9.1 in the experiments). A record is sent every millisecond. Below are the results: Kafka 3.9.1 50th percentile: 297 us 90th percentile: 366 us 99th percentile: 528 us 99.9th percentile: 1,460 us Kafka 4.0.0 50th percentile: 8,178 us 90th percentile: 10,355 us 99th percentile: 11,042 us 99.9th percentile: 13,638 us Kafka has been started locally in a docker container using `docker run -p 9092:9092 apache/kafka:3.9.1` and then `docker run -p 9092:9092 apache/kafka:4.0.0`. As shown, upgrading versions appears to cause ~20x increase ...

Hello Kafka users, developers and client-developers, This is the second candidate for release of Apache Kafka 4.0.1. Since this is a patch version it contains the bug fixes outlined in the release notes below! Release notes for the 4.0.1 release: https://dist.apache.org/repos/dist/dev/kafka/4.0.1-rc1/RELEASE_NOTES.html *** Please download, test and vote by Tuesday August 26. Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://dist.apache.org/repos/dist/dev/kafka/4.0.1-rc1/ * Docker release artifacts to be voted upon: apache/kafka:4.0.1-rc1 apache/kafka-native:4.0.1-rc1 * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://dist.apache.org/repos/dist/dev/kafka/4.0.1-rc1/javadoc/ * Tag to be voted upon (off 4.0 branch) is the 4.0.1 tag: https://github.com/apache/...

Hi Mickael, Thanks for running this release. I've completed the following tasks: 1. Verify the signature and checksum 2. Run the quickstart 3. Upgrade from v4.0.0, make sure ELR features works, then downgrade the ELR to disable the feature, and make sure it still works well. 4. Browse the doc PR < https://github.com/apache/kafka-site/pull/702 >, it looks like the content is still in v4.0.0. It's not a blocker for sure. We can discuss that in PR. +1 (binding) from me. Thank you. Luke On Thu, Aug 7, 2025 at 2:36 PM Paolo Patierno < paolo.patierno@gmail.com > wrote: > We ran all regression tests with Strimzi and everything looks good. > > +1 (non binding) > > Thanks, > Paolo > > On Tue, 5 Aug 2025 at 19:57, Mickael Maison < mickael.maison@gmail.com > > wrote: > > > Hello Kafka users, developers and client-developers, > > > > This is the third candidate for release of Apache Kaf...

> > Hello, >> >> We've identified a few Apache/Strimzi/Flink vulnerabilities. I was >> hoping to see if there is a plan to address these vulnerabilities? >> >> CVE-2021-3572 (Kafka, Strimzi-Operator) >> CVE-2023-52428 (Kafka) >> CVE-2024-47554 (Kafka, Flink-operator, Strimzi-operator) >> CVE-2024-6827 (superset) >> CVE-2024-7254 (Kafka) >> CVE-2024-7592 (Kafka, Strimzi-Operator) >> CVE-2025-24970 (Kafka, Strimzi-Operator) >> CVE-2025-24528 (Kafka, Superset, Strimzi-operator) >> >> Respectfully, >> >> -- >> Branden J. Carlson >> Information Security Compliance Manager >> >> P: +1 515-709-9711 >> E: bcarlson@synack.com >> < https://www.facebook.com/synack/ > < https://twitter.com/synack > >> < https://www.linkedin.com/company/synack-inc- > >> linkedin.com/in/bcarlson94 < https://www...

Hi Imcom Jin, Thanks for your question! It is expected behavior that Connect's internal topics are read completely from the beginning each time the worker starts, regardless of the auto.offset.reset configuration [1]. This is because they are compacted topics, and the first message in the topic may be necessary for correctness reasons. For example, if a worker only reads from the latest offset of the status topic, it may not know the status of long-running stable tasks. If you want to reduce the startup time, I suggest reducing the segment rolling configurations [2,3] for the internal topics. This will permit Kafka to compact away the duplicate status messages sooner, preventing them from being read on a future startup. This was previously reported [4] but we have not yet changed the default. I hope this helps, Greg [1] https://github.com/apache/kafka/blob/c4fb1008c4856c8cf9594269c86323753e6860ce/connect/runtime/src/main/java/org/apache/kafka/connect/util...

Hi dear Kafka team, I see that no matter what properties I give to the connector, the offset reset config for internal topics, especially the offset storage topic, say my-connect-offsets always use "earliest" which leads to very long bootstrap time during restart or stuck workers Log sample and config sample print in the log 2025-08-12 10:10:45,531 INFO [Consumer clientId=cbdhk04-data-cluster-offsets, groupId=cbdhk04-data-cluster] Seeking to earliest offset of partition root@cbd:/usr/local/nxg/docker/kafka-connect# docker logs connect-replication-8085 | grep "auto.offset.reset = earliest" -C2 auto.commit.interval.ms = 5000 auto.include.jmx.reporter = true auto.offset.reset = earliest My connect-districuted.properties contains the following config producer.override.auto.offset.reset=latest consumer.override.auto.offset.reset=latest producer.auto.offset.reset=latest consumer.auto.offset.reset=latest auto.offset.reset=latest connector.c...

Oops sorry looks like graph didn't work? Raw result for 1GB/s in, and 1GB/s out ($/month for inter AZ costs on AWS with AZ=RF=3) are: default replication no follower fetching $/month 175334.4 default replication follower fetching $/month 140267.52 simple diskless KIP-1176 without follower fetching $/month 70133.76 simple diskless KIP-1176 with follower fetching $/month 35066.88 complex diskless KIP-1150 $/month 0 Paul From: Brebner, Paul <Paul.Brebner@netapp.com.INVALID> Date: Friday, 8 August 2025 at 10:04 pm To: users@kafka.apache.org < users@kafka.apache.org >, dev < dev@kafka.apache.org > Subject: Re: [DISCUSS] The Path Forward for Saving Cross-AZ Costs KIPs EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Initial POC cost modelling (just for inter AZ costs, not overall costs) results are in: [image.png] From: Brebner, Paul <Paul.Brebner@netapp.com.INVALID> Date: Friday, 8 August 2025 at 12:37 pm To: ...

Initial POC cost modelling (just for inter AZ costs, not overall costs) results are in: From: Brebner, Paul <Paul.Brebner@netapp.com.INVALID> Date: Friday, 8 August 2025 at 12:37 pm To: users@kafka.apache.org <users@kafka.apache.org>, dev <dev@kafka.apache.org> Subject: Re: [DISCUSS] The Path Forward for Saving Cross-AZ Costs KIPs EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Thanks Luke, I'm taking a closer look at these now … Paul From: Luke Chen <showuon@gmail.com> Date: Tuesday, 5 August 2025 at 6:32 pm To: dev <dev@kafka.apache.org>, Kafka Users <users@kafka.apache.org> Subject: [DISCUSS] The Path Forward for Saving Cross-AZ Costs KIPs EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Hi all, The Kafka community is currently seeing an unprecedented situation with three KIPs (KIP-1150, IP-1176, KIP-1183) simultaneously addressing the same ...

Thanks Luke, I'm taking a closer look at these now … Paul From: Luke Chen < showuon@gmail.com > Date: Tuesday, 5 August 2025 at 6:32 pm To: dev < dev@kafka.apache.org >, Kafka Users < users@kafka.apache.org > Subject: [DISCUSS] The Path Forward for Saving Cross-AZ Costs KIPs EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Hi all, The Kafka community is currently seeing an unprecedented situation with three KIPs (KIP-1150, IP-1176, KIP-1183) simultaneously addressing the same challenge of high replication costs when running Kafka across multiple cloud availability zones. Each KIP offers a different solution to this issue. While diversity of innovative ideas is a key strength of open-source projects, it creates a burden for reviewers and users who must compare and comment on multiple proposals simultaneously. Furthermore, discussion around the three KIPs has stalled for over two months now. This could be due to the author...

Yes, sounds familiar (from a while back!) - I encountered similar issues with hash collisions (a version of Knuth's key parking problem), rebalancing storms, and slow consumers etc - see this blog: https://www.instaclustr.com/blog/apache-kafka-kongo-6-3-production-kafka-application-scalng-on-instaclustr/ Good luck! Paul Brebner From: Prasad Sawool < thc.prasads@gmail.com > Date: Thursday, 7 August 2025 at 2:14 pm To: users@kafka.apache.org < users@kafka.apache.org > Subject: Re: Kafka Consumer Not Receiving Messages Until Restart [You don't often get email from thc.prasads@gmail.com . Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Hi Waseem, We had faced a similar issue before.. Have you monitored your application logs as to much time does it take your application to process one record(kafka event). In our case, this was the bottleneck. Th...

We ran all regression tests with Strimzi and everything looks good. +1 (non binding) Thanks, Paolo On Tue, 5 Aug 2025 at 19:57, Mickael Maison < mickael.maison@gmail.com > wrote: > Hello Kafka users, developers and client-developers, > > This is the third candidate for release of Apache Kafka 4.1.0. > > The release brings many new features and improvements: > - Mechanism for plugin to register metrics > - Allow running multiple versions of connector plugins > - Queue are now in preview access > - New Streams rebalance protocol in early access > - Support for OAuth jwt-bearer grant type > - Improved metadata replication > > Release notes for the 4.1.0 release: > https://dist.apache.org/repos/dist/dev/kafka/4.1.0-rc2/RELEASE_NOTES.html > > *** Please download, test and vote by Friday August 8. > > Kafka's KEYS file containing PGP keys we use to sign the release: > https://kafka.apache.org/...

Hi Waseem, We had faced a similar issue before.. Have you monitored your application logs as to much time does it take your application to process one record(kafka event). In our case, this was the bottleneck. The consumer used to take too much time, so the consumer heatbeats stopped and next messages were not received. Do try and monitor your processing loop. By default kafka consumer has sane default configuration, so you need not tweak it but you can try and adapt it as per your use case. Regards, Prasad Sawool On Thu, 7 Aug 2025 at 05:06, Waseem Anwar < waseem@evolverstech.com > wrote: > Dear Apache Kafka Users, > > I hope you're doing well. > > I'm encountering an issue with our Kafka setup where the consumer stops > receiving messages after running for a period of time, even though: > > - > > The Kafka producer continues to send payloads successfully > - > > The consumer remains...

Hi, Here are the system test results. To speed it up I ran kafkatest/tests/core and all the other ones in parallel: - core: https://drive.google.com/file/d/1nvFq2L35FgFRF9PL6Zv7iRK56V2-ZBft/view?usp=drive_link - non-core: https://drive.google.com/file/d/1vzdVxEBmaJYa32PWuobu_UvgKIrovrjZ/view?usp=drive_link The only failure is still kafkatest.tests.core.network_degrade_test.test_rate_device_name=eth0_latency_ms=50_metadata_quorum=COMBINED_KRAFT_rate_limit_kbit=1000000_task_name=rate-1000-latency-50 This is tracked in https://issues.apache.org/jira/browse/KAFKA-19549 Thanks, Mickael On Tue, Aug 5, 2025 at 7:56 PM Mickael Maison < mickael.maison@gmail.com > wrote: > > Hello Kafka users, developers and client-developers, > > This is the third candidate for release of Apache Kafka 4.1.0. > > The release brings many new features and improvements: > - Mechanism for plugin to register metrics > - Allow running multiple versions of co...