Kafka

unsubscribe On Thu, Jan 29, 2026 at 5:37 PM xander12926 < xander12926@proton.me > wrote: > Hello, > > I'm looking for guidance on how to properly configure MirrorMaker to > ensure that no data is lost during normal replication, as well as during > planned maintenance windows. > > I recently encountered an issue where not all records were replicated to > the target cluster, as only 974,345 out of 1 million records were present, > which was verified using the kafka-get-offsets script. (only reproduced > once) > > The environment consists of two Kubernetes clusters configured in an > active/standby topology, where the Strimzi Operator is used to deploy Kafka > with three replicas and MirrorMaker2. > > Before performing a switchover, I scale down MirrorMaker 2 and delete the > heartbeats topic, as otherwise it gets replicated under different names, > such as source.heartbeats, source.source.heartbeats, a...

Hello, I'm looking for guidance on how to properly configure MirrorMaker to ensure that no data is lost during normal replication, as well as during planned maintenance windows. I recently encountered an issue where not all records were replicated to the target cluster, as only 974,345 out of 1 million records were present, which was verified using the kafka-get-offsets script. (only reproduced once) The environment consists of two Kubernetes clusters configured in an active/standby topology, where the Strimzi Operator is used to deploy Kafka with three replicas and MirrorMaker2. Before performing a switchover, I scale down MirrorMaker 2 and delete the heartbeats topic, as otherwise it gets replicated under different names, such as source.heartbeats, source.source.heartbeats, and so on. The configuration currently in use is the following: > apiVersion: kafka.strimzi.io/v1beta2 > kind: KafkaMirrorMaker2 > metadata: > name: kafka-main > spec: > clusters: > - alia...

Hi Chia-Ping, Yes, you are correct, my mistake. We should get this in. Thanks, Bill On Thu, Jan 29, 2026 at 11:39 AM Chia-Ping Tsai < chia7712@gmail.com > wrote: > Thanks for the feedback, Bill. > > Actually, I double-checked KAFKA-16505, and it appears this feature > (KIP-1034) is new in 4.2.0. > > Please correct me if I misunderstood. > > Best, > > Chia-Ping > > Bill Bejeck < bbejeck@apache.org > 於 2026年1月30日週五 上午12:19寫道： > > > Hi Chia-Ping, > > > > Thanks for mentioning this. Since this commit was in the previous > release > > and not a regression, I'm not sure that it meets the bar for a blocker. > > But I don't have a strong opinion, so I'll leave it to Christo to decide. > > > > Regards, > > Bill > > > > On Thu, Jan 29, 2026 at 7:57 AM Chia-Ping Tsai < chia7712@apache.org > > > wrote: > > > > ...

Thanks for the feedback, Bill. Actually, I double-checked KAFKA-16505, and it appears this feature (KIP-1034) is new in 4.2.0. Please correct me if I misunderstood. Best, Chia-Ping Bill Bejeck < bbejeck@apache.org > 於 2026年1月30日週五 上午12:19寫道： > Hi Chia-Ping, > > Thanks for mentioning this. Since this commit was in the previous release > and not a regression, I'm not sure that it meets the bar for a blocker. > But I don't have a strong opinion, so I'll leave it to Christo to decide. > > Regards, > Bill > > On Thu, Jan 29, 2026 at 7:57 AM Chia-Ping Tsai < chia7712@apache.org > > wrote: > > > hi Christo > > > > I have spotted a potential blocker, and a patch will be ready soon > > > > https://github.com/apache/kafka/pull/17942#discussion_r2737868443 > > > > Best, > > Chia-Ping > > > > On 2026/01/27 15:50:25 Christo Lolov wrote: ...

Hi Chia-Ping, Thanks for mentioning this. Since this commit was in the previous release and not a regression, I'm not sure that it meets the bar for a blocker. But I don't have a strong opinion, so I'll leave it to Christo to decide. Regards, Bill On Thu, Jan 29, 2026 at 7:57 AM Chia-Ping Tsai < chia7712@apache.org > wrote: > hi Christo > > I have spotted a potential blocker, and a patch will be ready soon > > https://github.com/apache/kafka/pull/17942#discussion_r2737868443 > > Best, > Chia-Ping > > On 2026/01/27 15:50:25 Christo Lolov wrote: > > Hello Kafka users, developers and client-developers, > > > > This is the RC2 candidate for release of Apache Kafka 4.2.0. > > > > This release has many exciting changes: > > * Kafka Queues (Share Groups) is now production-ready with new > > features like the RENEW acknowledgement type for extended processing > > ti...

hi Christo I have spotted a potential blocker, and a patch will be ready soon https://github.com/apache/kafka/pull/17942#discussion_r2737868443 Best, Chia-Ping On 2026/01/27 15:50:25 Christo Lolov wrote: > Hello Kafka users, developers and client-developers, > > This is the RC2 candidate for release of Apache Kafka 4.2.0. > > This release has many exciting changes: > * Kafka Queues (Share Groups) is now production-ready with new > features like the RENEW acknowledgement type for extended processing > times, adaptive batching for coordinators, and comprehensive lag > metrics. > * Kafka Streams brings the server-side rebalance protocol to GA with a > limited feature set, adds dead letter queue support in exception > handlers, introduces anchored wall-clock punctuation for deterministic > scheduling, and gives users full control over whether to send a leave > group request on closing. > * This release also delive...

Hi, I downloaded the binaries and check the signature. We ran all tests within the Strimzi project (Apache Kafka on Kubernetes) by using such binaries to build the container images. From regression to upgrade testing everything looks fine now. +1 (non binding) Thanks, Paolo. Paolo Patierno *Senior Principal Software Engineer @ IBM**CNCF Ambassador* Twitter : @ppatierno < http://twitter.com/ppatierno > Linkedin : paolopatierno < http://it.linkedin.com/in/paolopatierno > GitHub : ppatierno < https://github.com/ppatierno > On Tue, 27 Jan 2026, 16:52 Christo Lolov, < christololov@gmail.com > wrote: > Hello Kafka users, developers and client-developers, > > This is the RC2 candidate for release of Apache Kafka 4.2.0. > > This release has many exciting changes: > * Kafka Queues (Share Groups) is now production-ready with new > features like the RENEW acknowledgement type for extended processing > times, adapti...

Hi Jianbin, It'd be great if you could try it first in another environment with the in-place migration. I've never done this before. But please remember, even if it _might_ work, it is not under tested and might not be supported. I had a quick look at the config: > process.roles=broker,controller Why do you set the broker config as dual role? It is not supported for KRaft migration. Thank you, Luke On Thu, Jan 29, 2026 at 10:20 AM Jianbin Chen < jianbin@apache.org > wrote: > Could someone please help answer this question? I would really appreciate > it—thank you! > > Jianbin Chen < jianbin@apache.org > 于2026年1月28日周三 16:19写道： > > > > > > > ---------- Forwarded message --------- > > 发件人： Jianbin Chen < jianbin@apache.org > > > Date: 2026年1月28日周三 09:49 > > Subject: Inquiry about migrating from ZooKeeper to KRaft > > To: < users@kafka.apache.org > > > >...

Could someone please help answer this question? I would really appreciate it—thank you! Jianbin Chen < jianbin@apache.org > 于2026年1月28日周三 16:19写道： > > > ---------- Forwarded message --------- > 发件人： Jianbin Chen < jianbin@apache.org > > Date: 2026年1月28日周三 09:49 > Subject: Inquiry about migrating from ZooKeeper to KRaft > To: < users@kafka.apache.org > > > > Hi everyone, > > I'm trying to migrate a test cluster from ZooKeeper to KRaft in-place > (i.e., not provisioning three new controller-only nodes first and then > pointing the existing brokers to them). I hit a problem and would > appreciate any pointers. > > What I did > - Enabled zookeeper.metadata.migration.enable on each existing broker and > set the controller quorum settings so each broker acts as a > controller+broker (process.roles=broker,controller). > - Rolled the three nodes. > > Relevant broker configura...

Hi everyone, I'm trying to migrate a test cluster from ZooKeeper to KRaft in-place (i.e., not provisioning three new controller-only nodes first and then pointing the existing brokers to them). I hit a problem and would appreciate any pointers. What I did - Enabled zookeeper.metadata.migration.enable on each existing broker and set the controller quorum settings so each broker acts as a controller+broker (process.roles=broker,controller). - Rolled the three nodes. Relevant broker configuration (each broker has similar config; example shown): ``` process.roles=broker,controller node.id =7 broker.id =7 zookeeper.metadata.migration.enable=true controller.quorum.voters=7@broker1:9093,6@broker2:9093,4@broker3:9093 controller.quorum.bootstrap.servers=broker1:9093,broker2:9093,broker3:9093 zookeeper.connect=zk1:2181,zk2:2181,zk3:2181 controller.listener.names=CONTROLLER group.initial.rebalance.delay.ms =0 listeners=SSL://ip:9092,PLAINTEXT://ip:9192,CONTROLLER...

Hello Kafka users, developers and client-developers, This is the RC2 candidate for release of Apache Kafka 4.2.0. This release has many exciting changes: * Kafka Queues (Share Groups) is now production-ready with new features like the RENEW acknowledgement type for extended processing times, adaptive batching for coordinators, and comprehensive lag metrics. * Kafka Streams brings the server-side rebalance protocol to GA with a limited feature set, adds dead letter queue support in exception handlers, introduces anchored wall-clock punctuation for deterministic scheduling, and gives users full control over whether to send a leave group request on closing. * This release also delivers significant improvements to consistency and observability: CLI tools now feature standardized arguments like --bootstrap-server across all tools, metric naming has been corrected to follow the kafka.COMPONENT convention, and new idle ratio metrics provide better visibility into controlle...

Hello! I'm struggling with a RecordTooLargeException using MirrorMaker 2 on Kafka 3.8.1 and I'd like to sanity‑check my config with the community. Context: - Kafka version: 3.8.1 - Component: MirrorSourceConnector (MirrorMaker 2) - Error (target side): org.apache.kafka.common.errors.RecordTooLargeException: The message is 1049405 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration. What I'm trying to do I want MM2 to replicate messages a bit larger than 1 MB, so I added configs along these lines: text# In the connector / MM2 config source->target.enabled = true producer.max.request.size=2049405 max.request.size=2049405 target.max.request.size=2049405 At startup, I can see in the logs that the parameters are picked up with my custom value, but after some time (under load) it behaves like it's using the default again and I hit the error above, whic...

I was curious about this and did a quick search. It appears that this approach is generally considered an anti pattern [1]. Please do not take this as an authoritative opinion as that's my only source. I am simply trying to understand why such a large header is required and if the problem could be solved using the request body or any other way like a proxy. Jetty does have a rationale for these limits, as they are based on the HTTP specification. On that note, something you should consider is that If you have other systems in between your client and Kafka Connect (such as load balancers, and friends), you may face the same issue there as well In any case, if we decide to take the KIP approach to enable this, I assume we will need to provide a justification that goes beyond citing "business requirements" to explain the effort involved in building it and the overall benefits it'll bring to the framework Thanks!  [1]  https://gi...

Hi, I have run our Strimzi test container tests on AArch64, along with unit and integration tests, and verified checksums. All tests seemed to pass and without any further issues. +1 (non-binding). Cheers, Maros

Ah, I see! Sorry about that, been a sec since I've gotten my hands dirty with the server logic for Connect. I can't think of any other workarounds, unfortunately. I'd be in favor of a small KIP to plug the gap though! On Thu, Jan 22, 2026, 15:19 Chitra Elangovan < chitrae92@gmail.com > wrote: > Hi Chris, > > We looked at the ConnectRestExtension.html > < > https://kafka.apache.org/39/javadoc/org/apache/kafka/connect/rest/ConnectRestExtension.html > >, > but since the 8KB limit is enforced at the Jetty Server level, the > extension isn't invoked until after the 431 error is already triggered. Are > there any other workarounds that are currently available? > > Thanks, > Chitra Elangovan > > > On Wed, Jan 21, 2026 at 5:23 PM Chris Egerton < fearthecellos@gmail.com > > wrote: > > > Hi Chitra, > > > > Unsure about whether this will help with your specific req...

hi Anand thanks for this report. I have opened https://issues.apache.org/jira/browse/KAFKA-20089 to track it. Please feel free to leave your comment on the ticket. Best, Chia-Ping On 2026/01/20 13:02:34 Subra I wrote: > Hello All, > > I am using Kafka-clients-4.1.1.jar with a kafka broker that is on version > 4.0.0. From my Java code, when I call the KafkaConsumer partitionsFor API, > I see that it works when the topic is already created in kafka. But if it > is not yet created, the above API call returns empty - no information. > > I noticed that just before I call partitionsFor API, the topic is not > created. But after I call this API, the topic gets created, but this API > still returns no information. Of course, if I call the same API again, I do > get valid information for this topic. > > I used to have kafka-clients-3.9.0.jar earlier. There both these cases were > working fine. Is this a new change in behavior...

Hi Chris, We looked at the ConnectRestExtension.html < https://kafka.apache.org/39/javadoc/org/apache/kafka/connect/rest/ConnectRestExtension.html >, but since the 8KB limit is enforced at the Jetty Server level, the extension isn't invoked until after the 431 error is already triggered. Are there any other workarounds that are currently available? Thanks, Chitra Elangovan On Wed, Jan 21, 2026 at 5:23 PM Chris Egerton < fearthecellos@gmail.com > wrote: > Hi Chitra, > > Unsure about whether this will help with your specific requirements or not > but have you looked into the REST extension API? Linking to the 3.9.x docs > since that's the version you're running but AFAIK the feature hasn't > changed much or even at all in a few years: > > https://kafka.apache.org/39/javadoc/org/apache/kafka/connect/rest/ConnectRestExtension.html > > If this doesn't help, I think a KIP could make sense. > >...

Hi Chitra, Unsure about whether this will help with your specific requirements or not but have you looked into the REST extension API? Linking to the 3.9.x docs since that's the version you're running but AFAIK the feature hasn't changed much or even at all in a few years: https://kafka.apache.org/39/javadoc/org/apache/kafka/connect/rest/ConnectRestExtension.html If this doesn't help, I think a KIP could make sense. Cheers, Chris On Wed, Jan 21, 2026, 19:57 Chitra Elangovan < chitrae92@gmail.com > wrote: > Hello Kafka Community, > > I am reaching out to discuss a limitation regarding the Kafka Connect REST > API's embedded Jetty server (we are using kafka 3.9.0 version). > > Currently, the default header size limit is 8KB. In our environment, we are > exceeding this limit due to our business needs. When this limit is > exceeded, Jetty returns a 431 Request Header Fields Too Large error. > > I'...

Hello Kafka Community, I am reaching out to discuss a limitation regarding the Kafka Connect REST API's embedded Jetty server (we are using kafka 3.9.0 version). Currently, the default header size limit is 8KB. In our environment, we are exceeding this limit due to our business needs. When this limit is exceeded, Jetty returns a 431 Request Header Fields Too Large error. I've noted that while listeners and a few other properties are configurable via the Worker properties, there doesn't seem to be a straightforward way to increase the requestHeaderSize for the internal HttpConfiguration. Questions for the group: - Is there an existing JVM system property or undocumented config that can override this in the current version? - If not, would the community be open to a KIP (Kafka Improvement Proposal) to expose this as a worker configuration (e.g., rest.request.header.size.default)? - Are there any known workarounds for Kafka Connect distributed work...

Thanks for the information. Regards, James -----Original Message----- From: Ivan Yurchenko < ivan@ivanyu.me > Sent: 21 January 2026 13:24 To: users@kafka.apache.org Subject: [EXTERNAL] Re: Question about Kafka 3.9.2 version Hi James, 3.9.2 is about to be released soon. "RC0" means "release candidate", a build on which final pre-release checks and tests are performed. If nothing major is discovered, it'll become the 3.9.2 release. Best, Ivan On Wed, Jan 21, 2026, at 04:31, JAMES JOSE wrote: > Hello, > > The latest download available is 3.9.1 version as per https://kafka.apache.org/community/downloads/ . However, the source repository shows a Tag named 3.9.2-rc0. Does it mean 3.9.2 is released, if so, is there a plan to add the downloadable binaries for 3.9.2 ? > > Regards, > James >

Hi James, 3.9.2 is about to be released soon. "RC0" means "release candidate", a build on which final pre-release checks and tests are performed. If nothing major is discovered, it'll become the 3.9.2 release. Best, Ivan On Wed, Jan 21, 2026, at 04:31, JAMES JOSE wrote: > Hello, > > The latest download available is 3.9.1 version as per https://kafka.apache.org/community/downloads/ . However, the source repository shows a Tag named 3.9.2-rc0. Does it mean 3.9.2 is released, if so, is there a plan to add the downloadable binaries for 3.9.2 ? > > Regards, > James >

Hello, The latest download available is 3.9.1 version as per https://kafka.apache.org/community/downloads/ . However, the source repository shows a Tag named 3.9.2-rc0. Does it mean 3.9.2 is released, if so, is there a plan to add the downloadable binaries for 3.9.2 ? Regards, James

Hello All, I am using Kafka-clients-4.1.1.jar with a kafka broker that is on version 4.0.0. From my Java code, when I call the KafkaConsumer partitionsFor API, I see that it works when the topic is already created in kafka. But if it is not yet created, the above API call returns empty - no information. I noticed that just before I call partitionsFor API, the topic is not created. But after I call this API, the topic gets created, but this API still returns no information. Of course, if I call the same API again, I do get valid information for this topic. I used to have kafka-clients-3.9.0.jar earlier. There both these cases were working fine. Is this a new change in behavior in kafka-clients-4.1.1.jar? Thanks, Anand

Hi Christo, It looks like you did not close the RC1 Maven artifacts in Nexus. All the 4.2.0 artifacts are from Jan 13 so presumably it's still RC0. For example, see https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.13/4.2.0/ Thanks, Mickael On Mon, Jan 19, 2026 at 5:28 PM Christo Lolov < christololov@gmail.com > wrote: > > Hello Kafka users, developers and client-developers, > > This is the RC1 candidate for release of Apache Kafka 4.2.0. > > This release has many exciting changes: > * Kafka Queues (Share Groups) is now production-ready with new > features like the RENEW acknowledgement type for extended processing > times, adaptive batching for coordinators, and comprehensive lag > metrics. > * Kafka Streams brings the server-side rebalance protocol to GA with a > limited feature set, adds dead letter queue support in exception > handlers, introduces anchored wall-clock punctuatio...

Hello Lucas! I already created a new RC and was just about to write the vote email - are these critical? Is it possible that you cherry-pick them on 4.2 with the goal that if 4.2.0-rc2 is cut they make it in there, but for now we exclude them from 4.2.0-rc1? Or are they critical enough that they would justify creating 4.2.0-rc2 on their own? Best, Christo On Mon, 19 Jan 2026 at 14:28, Lucas Brutschy < lbrutschy@confluent.io > wrote: > > Hey Christo, > > could we include fixes for two usability issues in the new RC? They > are minor low-risk changes, but it would be good to include them. Let > me know, so I can cherry-pick them today. > > - https://github.com/apache/kafka/commit/7fa5121a172e4ac0248c15e73d5833dcec9bb7d8 > - https://github.com/apache/kafka/commit/c552631801f709ff7d1090b99dd5f0053d5c08ce > > Cheers, > Lucas > > On Mon, Jan 19, 2026 at 11:34 AM Christo Lolov < christololov@gmail.com > wrote:...