Kafka

Hi, The follower is not able to sync-up with the leader due to epochs diverged between leader and follower. To confirm this, you can enable request logger and check the diverging-epoch field in the fetch-response: https://sourcegraph.com/github.com/apache/kafka@a640a81040f6ef6f85819b60194f0394f5f2194e/-/blob/clients/src/main/resources/common/message/FetchResponse.json?L76 This issue can happen when the leader-epoch-checkpoint file is corrupted in the leader node. To mitigate the issue, you have to: 1. Stop the leader broker 2. Remove the `leader-epoch-checkpoint` file for that affected partition 3. Recover the partition by deleting the partition entry from the checkpoint files: `log-start-offset-checkpoint`, `replication-offset-checkpoint`, `recovery-point-offset-checkpoint`, and `cleaner-offset-checkpoint`. Note that when removing the entry, you also have to update the number of entries in those files in Line 2. 4. Remove the `.kafka_cleanshutdown` marker file....

Hi All, System test runs are green. There were 13 test failures in the first run. All the failed tests passed in the second run. System test results: https://gist.github.com/omkreddy/17d23d3eb36ef840011f2494d65bbd4f Thanks, On Thu, Mar 28, 2024 at 3:21 PM Manikumar < manikumar@apache.org > wrote: > Hello Kafka users, developers and client-developers, > > This is the second candidate we are considering for the release of Apache > Kafka 3.6.2. > > This is a bugfix release with several fixes, including dependency > version bumps for CVEs. > > Release notes for the 3.6.2 release: > https://home.apache.org/~manikumar/kafka-3.6.2-rc2/RELEASE_NOTES.html > > *** Please download, test and vote by by Wednesday, April 3rd > > Kafka's KEYS file containing PGP keys we use to sign the release: > https://kafka.apache.org/KEYS > > * Release artifacts to be voted upon (source and binary): > https://home...

Hello Kafka users, developers and client-developers, This is the second candidate we are considering for the release of Apache Kafka 3.6.2. This is a bugfix release with several fixes, including dependency version bumps for CVEs. Release notes for the 3.6.2 release: https://home.apache.org/~manikumar/kafka-3.6.2-rc2/RELEASE_NOTES.html *** Please download, test and vote by by Wednesday, April 3rd Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://home.apache.org/~manikumar/kafka-3.6.2-rc2/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~manikumar/kafka-3.6.2-rc2/javadoc/ * Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag: https://github.com/apache/kafka/releases/tag/3.6.2-rc2 * Documentation: https://kafka.apache.org/36/document...

Congratulations! On Wed, 27 Mar 2024 at 21:10, Matthias J. Sax < mjsax@apache.org > wrote: > Congrats! > > On 3/26/24 9:39 PM, Christo Lolov wrote: > > Thank you everyone! > > > > It wouldn't have been possible without quite a lot of reviews and > extremely > > helpful inputs from you and the rest of the community! I am looking > forward > > to working more closely with you going forward :) > > > > On Tue, 26 Mar 2024 at 14:31, Kirk True < kirk@kirktrue.pro > wrote: > > > >> Congratulations Christo! > >> > >>> On Mar 26, 2024, at 7:27 AM, Satish Duggana < satish.duggana@gmail.com > > >> wrote: > >>> > >>> Congratulations Christo! > >>> > >>> On Tue, 26 Mar 2024 at 19:20, Ivan Yurchenko < ivan@ivanyu.me > wrote: > >>>> > >>>> Congrats! > >>>...

Hello to all users, contributors and Committers! [ You are receiving this email as a subscriber to one or more ASF project dev or user mailing lists and is not being sent to you directly. It is important that we reach all of our users and contributors/committers so that they may get a chance to benefit from this. We apologise in advance if this doesn't interest you but it is on topic for the mailing lists of the Apache Software Foundation; and it is important please that you do not mark this as spam in your email client. Thank You! ] The Travel Assistance Committee (TAC) are pleased to announce that travel assistance applications for Community over Code NA 2024 are now open! We will be supporting Community over Code NA, Denver Colorado in October 7th to the 10th 2024. TAC exists to help those that would like to attend Community over Code events, but are unable to do so for financial reasons. For more info on this years applications and qualifyin...

Congrats! On 3/26/24 9:39 PM, Christo Lolov wrote: > Thank you everyone! > > It wouldn't have been possible without quite a lot of reviews and extremely > helpful inputs from you and the rest of the community! I am looking forward > to working more closely with you going forward :) > > On Tue, 26 Mar 2024 at 14:31, Kirk True < kirk@kirktrue.pro > wrote: > >> Congratulations Christo! >> >>> On Mar 26, 2024, at 7:27 AM, Satish Duggana < satish.duggana@gmail.com > >> wrote: >>> >>> Congratulations Christo! >>> >>> On Tue, 26 Mar 2024 at 19:20, Ivan Yurchenko < ivan@ivanyu.me > wrote: >>>> >>>> Congrats! >>>> >>>> On Tue, Mar 26, 2024, at 14:48, Lucas Brutschy wrote: >>>>> Congrats! >>>>> >>>>> On Tue, Mar 26, 2024 at 2:44 PM Federico Valeri < fedevaleri@gmail...

Hi, Thanks for letting me know. Pls let me know after merging the PR. I will generate RC2. Thanks On Sat, Mar 23, 2024 at 1:58 AM Colin McCabe < cmccabe@apache.org > wrote: > Sorry but I have to vote -1 > > I tried verifying that the migration quotas bug described in > https://issues.apache.org/jira/browse/KAFKA-16222 was fixed, and it > appears to still be an issue with 3.6.2 RC1. The quota on the default > resource is still getting translated improperly. > > I am looking into what the issue is here. > > best, > Colin > > > On Thu, Mar 21, 2024, at 19:32, Chia-Ping Tsai wrote: > > hi Manikumar > > > >> Pls let me know after merging the PR. I will generate RC2 later today. > > > > Sure. We will complete it ASAP > > > > > >> Manikumar < manikumar.reddy@gmail.com > 於 2024年3月22日 上午9:26 寫道： > >> > >> Hi, > >> > ...

Hi Venkatesh, The 1 core 1 stream thread recommendation is just s starting point. You need to set the number of stream thread as it fits you by monitoring the app. Maybe this blog post might be interesting for you: https://www.responsive.dev/blog/a-size-for-every-stream Best, Bruno On 3/19/24 4:14 AM, Venkatesh Nagarajan wrote: > Thanks very much for sharing the links and for your important inputs, Bruno! > >> We recommend to use as many stream threads as cores on the compute node where the Kafka Streams client is run. How many Kafka Streams tasks do you have to distribute over the clients? > > We use 1vCPU (probably 1 core) per Kafka Streams Client (ECS Task). Each client/ECS Task runs 10 streaming threads and the CPU utilisation is just 4% on an average. It increases when transient errors occur as they require retries and threads to be replaced. > > We run a maximum of 6 clients/ECS Tasks when the offset lags are high. The inp...

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.6.2. This is a bugfix release with several fixes, including dependency version bumps for CVEs. Release notes for the 3.6.2 release: https://home.apache.org/~manikumar/kafka-3.6.2-rc1/RELEASE_NOTES.html *** Please download, test and vote by Tuesday, March 26th Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://home.apache.org/~manikumar/kafka-3.6.2-rc1/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~manikumar/kafka-3.6.2-rc1/javadoc/ * Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag: https://github.com/apache/kafka/releases/tag/3.6.2-rc1 * Documentation: https://kafka.apache.org/36/documentation.html * Protocol: h...

Hi Sanaa, I actually ran a migration twice. First locally just following the procedure described by the official Kafka documentation https://kafka.apache.org/documentation/#kraft_zk_migration and then on Kubernetes, because I notice you are talking about StatefulSet. But in this case I used the Strimzi operator https://strimzi.io/ to do so (disclaimer, I am one of the maintainers and we recently added the automatic migration feature in latest 0.40.0 release). But referring to your problem, you are also mentioning a dashboard which is not clear from where it's coming and which kind of metrics are you looking at to say that you have two controllers at the same time (a broker and a KRaft controller) which is weird because the source of truth is the /controller znode on ZooKeeper. I think you should look at brokers and controllers logs and investigate if anything is going bad which doesn't allow a KRaft node to take over as controller. Thanks Paolo On Tue,...

Hi all, Community Over Code < https://communityovercode.org/ >, the ASF conference, will be held in Denver, Colorado, October 7-10, 2024. The call for presentations < https://news.apache.org/foundation/entry/apache-software-foundation-opens-cfp-for-community-over-code-north-america-2024 > is open now through April 15, 2024. (This is two months earlier than last year!) I am one of the co-chairs for the stream processing track, and we would love to see you there and hope that you will consider submitting a talk. About the Streaming track: There are many top-level ASF projects which focus on and push the envelope for stream and event processing. ActiveMQ, Beam, Bookkeeper, Camel, Flink, Kafka, Pulsar, RocketMQ, and Spark are all house-hold names in the stream processing and analytics world at this point. These projects show that stream processing has unique characteristics requiring deep expertise. On the other hand, users need easy to apply soluti...

Hi Paolo, Thank you for your response! I tested out a different theory today where I deployed the kraft controller statefulset and waited to see which brokers would be elected as controllers. Here is an example of my migration right after I have provisioned the kraft controller brokers/statefulset. At this point, the brokers haven't been restarted. get /controller {"version":2,"brokerid":1,"timestamp":"1710876891432","kraftControllerEpoch":-1} get /migration {"version":0,"kraft_metadata_offset":-1,"kraft_controller_id":-1,"kraft_metadata_epoch":-1,"kraft_controller_epoch":-1} At this point, on a dashboard I have I see that a kafka broker is a controller and a kraft controller broker is also a controller (although it's not what I see in zookeeper as shown above). One thing to note is I am doing this migration on a stretched cluster so this may alter the way...

Hi, I have an unusual situation where I have a cluster running Kafka 3.5.1 in strimzi where 4 of the __consumer_offset partitions have dropped under min isr. Everything else appears to be working fine. Upon investigating, i've found that the partition followers appear to be out of sync with the leader in terms of leader epoch For example the leader-epoch-checkpoint file on the leader partition is 0 4 0 0 1 4 4 6 27 10 while the followers are 0 5 0 0 1 4 4 6 5 7 6 9 which appears to me like the followers are 2 elections ahead of the leader and i'm not sure how they got to this situation. I've attempted to force a new leader election via kafka-leader-elections but it refused for both PREFERRED and UNCLEAN. I've also tried a manual partition assignment to move the leader to another broker but it wont do it. What is even more strange is that if i watch the leader-epoch-checkpoint file on one of the followers I can see it constantly c...

Hi Sanaa, from my experiece about running migration it never happened to me and it should not happen anyway. When a (ZooKeeper-based) broker registers to be the controller at the beginning, you can see that the corresponding /controller znode will have an -1 as epoch. Something like: {"version":2,"brokerid":0,"timestamp":"1710845218527","kraftControllerEpoch":-1} When you deploy the KRaft quorum controller and roll the brokers to register and start the migration, the controller role is got by one of the KRaft controller and its epoch will be for sure greater than -1. Something like: {"version":2,"brokerid":4,"timestamp":"1710844690234","kraftControllerEpoch":10} A KRaft controller is able to "steal" the controller role even because its epoch will be for sure greater than -1. So during or after a migration, a broker could not get the controller role because onl...

Thanks very much for sharing the links and for your important inputs, Bruno! > We recommend to use as many stream threads as cores on the compute node where the Kafka Streams client is run. How many Kafka Streams tasks do you have to distribute over the clients? We use 1vCPU (probably 1 core) per Kafka Streams Client (ECS Task). Each client/ECS Task runs 10 streaming threads and the CPU utilisation is just 4% on an average. It increases when transient errors occur as they require retries and threads to be replaced. We run a maximum of 6 clients/ECS Tasks when the offset lags are high. The input topics have 60 partitions each and this matches (total number of clients/ECS Tasks i.e. 6) * ( Streaming threads per client/ECS task i.e.10). With the 1 streaming thread per core approach, we will need 60 vCPUs/cores. As I mentioned above, we have observed 10 threads using just 4% of 1 vCPU/core on an average. It may be difficult to justify provisioning more cores as it will be ...

Hello, I've begun migrating some of my Zookeeper Kafka clusters to KRaft. A behaviour I've noticed twice across two different kafka cluster environments is after provisioning a kraft controller quorum in migration mode, it is possible for a kafka broker to become an active controller alongside a kraft controller broker. For example, here are the steps I follow and the behaviour I notice (I'm currently using Kafka v3.6): 1. Enable the KRaft migration on the existing Kafka brokers (set the `controller.quorum.voter`, `controller.listener.names` and `zookeeper.metadata.migration.enable` configs in the server.properties file). 2. Deploy a kraft controller statefulset and service with the migration enabled so that data is copied over from Zookeeper and we enter a dual-write mode. 3. After a few minutes, I see that the migration has completed (it's a pretty small cluster). At this point, the kraft controller pod has been elected to be the controller (and I...

Hi Vansh, Great that you want to join our community! Subscription to the mailing list is self-serve. See details to subscribe under the following link: https://kafka.apache.org/contact Thank you for your interest in Apache Kafka! Best, Bruno On 3/15/24 1:59 PM, Vansh Kabra wrote: > Dear Kafka Users Community, > > > My name is Vansh Kabra, and I'm reaching out to express my interest in > joining the Kafka Users mailing list ( users@kafka.apache.org ). > > > I have been actively working with Kafka in my projects and have found it to > be an invaluable tool for building scalable and reliable real-time data > pipelines. I believe that being a part of the Kafka Users mailing list will > provide me with valuable insights, allow me to learn from the experiences > of other community members, and contribute to discussions on Kafka-related > topics. > > > I am eager to engage with the vibrant Kafka c...

Dear Kafka Users Community, My name is Vansh Kabra, and I'm reaching out to express my interest in joining the Kafka Users mailing list ( users@kafka.apache.org ). I have been actively working with Kafka in my projects and have found it to be an invaluable tool for building scalable and reliable real-time data pipelines. I believe that being a part of the Kafka Users mailing list will provide me with valuable insights, allow me to learn from the experiences of other community members, and contribute to discussions on Kafka-related topics. I am eager to engage with the vibrant Kafka community, share my knowledge, seek assistance when needed, and collaborate on solving challenges together. Could you please add my email address ( vanshkabra05@gmail.com ) to the users@kafka.apache.org mailing list? Thank you for considering my request. I look forward to being an active member of the Kafka Users community. Best regards, Vansh Kabra