Kafka

Hello Kafka users, developers and client-developers, This is the RC3 candidate for release of Apache Kafka 4.2.0. This release has many exciting changes: * Kafka Queues (Share Groups) is now production-ready with new features like the RENEW acknowledgement type for extended processing times, adaptive batching for coordinators, and comprehensive lag metrics. * Kafka Streams brings the server-side rebalance protocol to GA with a limited feature set, adds dead letter queue support in exception handlers, introduces anchored wall-clock punctuation for deterministic scheduling, and gives users full control over whether to send a leave group request on closing. * This release also delivers significant improvements to consistency and observability: CLI tools now feature standardized arguments like --bootstrap-server across all tools, metric naming has been corrected to follow the kafka.COMPONENT convention, and new idle ratio metrics provide better visibility into controlle...

Hi everyone, I'm implementing active-passive HA across two sites for a Flink streaming job with large RocksDB state. I sync checkpoint directories via rsync library from primary → secondary continuously. On primary failure, I want to auto-start the job on secondary using the latest complete checkpoint. For that, I need a reliable way to check if a checkpoint is fully complete before using it for job recovery. *My Understanding & Concern* As I understand, the _metadata file is created last by JobManager after all TaskManagers acknowledge. But even if _metadata exists, there's a chance it was partially written (crash mid-write/rsync copied an intermittent half file). *Questions* 1. Is there a definitive way to verify checkpoint completeness? Something beyond just checking if _metadata file exists? 2. If I start a job with incomplete _metadata: - Does Flink fail immediately during startup? - Or does it retry multiple checkpoints befo...

unsubscribe On Thu, Jan 29, 2026 at 5:37 PM xander12926 < xander12926@proton.me > wrote: > Hello, > > I'm looking for guidance on how to properly configure MirrorMaker to > ensure that no data is lost during normal replication, as well as during > planned maintenance windows. > > I recently encountered an issue where not all records were replicated to > the target cluster, as only 974,345 out of 1 million records were present, > which was verified using the kafka-get-offsets script. (only reproduced > once) > > The environment consists of two Kubernetes clusters configured in an > active/standby topology, where the Strimzi Operator is used to deploy Kafka > with three replicas and MirrorMaker2. > > Before performing a switchover, I scale down MirrorMaker 2 and delete the > heartbeats topic, as otherwise it gets replicated under different names, > such as source.heartbeats, source.source.heartbeats, a...

Hello, I'm looking for guidance on how to properly configure MirrorMaker to ensure that no data is lost during normal replication, as well as during planned maintenance windows. I recently encountered an issue where not all records were replicated to the target cluster, as only 974,345 out of 1 million records were present, which was verified using the kafka-get-offsets script. (only reproduced once) The environment consists of two Kubernetes clusters configured in an active/standby topology, where the Strimzi Operator is used to deploy Kafka with three replicas and MirrorMaker2. Before performing a switchover, I scale down MirrorMaker 2 and delete the heartbeats topic, as otherwise it gets replicated under different names, such as source.heartbeats, source.source.heartbeats, and so on. The configuration currently in use is the following: > apiVersion: kafka.strimzi.io/v1beta2 > kind: KafkaMirrorMaker2 > metadata: > name: kafka-main > spec: > clusters: > - alia...

Hi Chia-Ping, Yes, you are correct, my mistake. We should get this in. Thanks, Bill On Thu, Jan 29, 2026 at 11:39 AM Chia-Ping Tsai < chia7712@gmail.com > wrote: > Thanks for the feedback, Bill. > > Actually, I double-checked KAFKA-16505, and it appears this feature > (KIP-1034) is new in 4.2.0. > > Please correct me if I misunderstood. > > Best, > > Chia-Ping > > Bill Bejeck < bbejeck@apache.org > 於 2026年1月30日週五 上午12:19寫道： > > > Hi Chia-Ping, > > > > Thanks for mentioning this. Since this commit was in the previous > release > > and not a regression, I'm not sure that it meets the bar for a blocker. > > But I don't have a strong opinion, so I'll leave it to Christo to decide. > > > > Regards, > > Bill > > > > On Thu, Jan 29, 2026 at 7:57 AM Chia-Ping Tsai < chia7712@apache.org > > > wrote: > > > > ...

Thanks for the feedback, Bill. Actually, I double-checked KAFKA-16505, and it appears this feature (KIP-1034) is new in 4.2.0. Please correct me if I misunderstood. Best, Chia-Ping Bill Bejeck < bbejeck@apache.org > 於 2026年1月30日週五 上午12:19寫道： > Hi Chia-Ping, > > Thanks for mentioning this. Since this commit was in the previous release > and not a regression, I'm not sure that it meets the bar for a blocker. > But I don't have a strong opinion, so I'll leave it to Christo to decide. > > Regards, > Bill > > On Thu, Jan 29, 2026 at 7:57 AM Chia-Ping Tsai < chia7712@apache.org > > wrote: > > > hi Christo > > > > I have spotted a potential blocker, and a patch will be ready soon > > > > https://github.com/apache/kafka/pull/17942#discussion_r2737868443 > > > > Best, > > Chia-Ping > > > > On 2026/01/27 15:50:25 Christo Lolov wrote: ...

Hi Chia-Ping, Thanks for mentioning this. Since this commit was in the previous release and not a regression, I'm not sure that it meets the bar for a blocker. But I don't have a strong opinion, so I'll leave it to Christo to decide. Regards, Bill On Thu, Jan 29, 2026 at 7:57 AM Chia-Ping Tsai < chia7712@apache.org > wrote: > hi Christo > > I have spotted a potential blocker, and a patch will be ready soon > > https://github.com/apache/kafka/pull/17942#discussion_r2737868443 > > Best, > Chia-Ping > > On 2026/01/27 15:50:25 Christo Lolov wrote: > > Hello Kafka users, developers and client-developers, > > > > This is the RC2 candidate for release of Apache Kafka 4.2.0. > > > > This release has many exciting changes: > > * Kafka Queues (Share Groups) is now production-ready with new > > features like the RENEW acknowledgement type for extended processing > > ti...