Kafka

Hi TengYao, I have done the following tests: - Built from 3.9.1-rc1 with OpenJDK-17 and OpenJDK-21. - Ran all unit and integration tests. - Verified basic produce/consume. - Validated the checksums and keys. - Ran KRaft with kafka_2.13-3.9.1.tgz Thanks, PoAn > On Apr 23, 2025, at 5:00 PM, TengYao Chi < frankvicky@apache.org > wrote: > > Hello Kafka users, developers, and client-developers, > > This is the second candidate for the release of Apache Kafka 3.9.1. > This RC fixed the release tag off to a temporary branch issue (KAFKA-19166 > < https://issues.apache.org/jira/browse/KAFKA-19166 >) compared with RC0. > > This is a bug-fix release with several fixes, and most importantly, it adds > Java 23 support for 3.9 > > Release notes for the 3.9.1 release: > https://dist.apache.org/repos/dist/dev/kafka/3.9.1-rc1/RELEASE_NOTES.html > > Please download, test, and vote by *Wednesday, April 30, 9:00...

Hello Kafka users, developers, and client-developers, This is the second candidate for the release of Apache Kafka 3.9.1. This RC fixed the release tag off to a temporary branch issue (KAFKA-19166 < https://issues.apache.org/jira/browse/KAFKA-19166 >) compared with RC0. This is a bug-fix release with several fixes, and most importantly, it adds Java 23 support for 3.9 Release notes for the 3.9.1 release: https://dist.apache.org/repos/dist/dev/kafka/3.9.1-rc1/RELEASE_NOTES.html Please download, test, and vote by *Wednesday, April 30, 9:00 AM PT* Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://dist.apache.org/repos/dist/dev/kafka/3.9.1-rc1/ * Docker release artifacts to be voted upon: apache/kafka:3.9.1-rc1 https://hub.docker.com/layers/apache/kafka/3.9.1-rc1/images/sha256-459c4e12260a1426a4f53c357d6e0c52b58ed70ff4118de1ffceb3f131d64a...

Hi Jamie, We had success implanting basic authentication using out of the box constructs. Is this what you tried? https://docs.confluent.io/platform/current/security/authentication/http-basic-auth/overview.html We also implemented JWT but we did not have any success doing that out of the box. For that, we used a "proxy" to intercept and forward the requests from the proxy to the workers ---- On Tue, 22 Apr 2025 05:56:12 -0400 jamiedd13@aol.co.uk.INVALID wrote ---- Hi All,  Has anyone managed to get a REST extension working with multiple connect instances / workers? Many Thanks,  Jamie On Wednesday 16 April 2025 at 10:57:01 BST, Jamie <mailto: jamiedd13@aol.co.uk > wrote: Hi All,  I'm trying to implement a custom REST extension for Kafka Connect that allows a token to be used for authentication.  When I have a single instance of Kafka connect running in distributed mode ...

Hi All,  Has anyone managed to get a REST extension working with multiple connect instances / workers? Many Thanks,  Jamie On Wednesday 16 April 2025 at 10:57:01 BST, Jamie < jamiedd13@aol.co.uk > wrote: Hi All,  I'm trying to implement a custom REST extension for Kafka Connect that allows a token to be used for authentication.  When I have a single instance of Kafka connect running in distributed mode (i.e. 1 worker) this works as expected. However, when I add another instance (another worker) I get an error when the worker that received the request has to forward it on to the group leader: Error IO error forwarding REST request: (org.apache.kafka.connect.runtime.rest.RestClient:143) java.util.concurrent.ExecutionException: org.eclipse.jetty.client.HttpResponseException: HTTP protocol violation: Authentication challenge without WWW-Authenticate header  I noticed that someone also had the same issue with the basic authentication...

Yes, since Kafka 3.8, there is a so called "state updated thread". https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/DefaultStateUpdater.java#L183 -Matthias On 4/17/25 6:58 AM, Chad Preisler wrote: > In the latest version of Kafka Streams does KTable restoration happen in > its own thread? Can you point me to where that happens in the code base? > > Thanks, > Chad >

Hey folks, Just wanted to share something I thought might be useful to others in the community who are experimenting with or building on GenAI, LLMs, or RAG systems. 📅 *AI Learning Week* (April 28 – May 1) is a free 4-day webinar series organized by Xebia Academy. It covers both strategy and engineering sides of AI—but if you're more on the *builder* side like me, the *Tech Track* is where it gets really interesting. 🔧 Here's what's on the Tech Track: - *Building LLM-Powered Apps (Python)* – learn how to interface with LLMs, and spin up your first RAG + Agent - *Prompt Engineering Tips* – techniques to get clearer, more consistent results from tools like ChatGPT or Claude - *PDFs in RAG* – how to use multimodal models to parse complex PDF content (think tables, figures) and feed cleaner chunks into vector databases All sessions are *1 hour long*, with *zero marketing fluff*—just focused, practical insights. S...

Hi all, We have a kafka cluster with 10 servers. First 3 of them are set as controller and broker. All 10 servers are brokers. Rest seven servers are set as brokers. v3.7.0 In the controller.log file we always see that leader is changing. [2025-04-16 16:02:46,023] WARN [QuorumController id=1] *Renouncing the leadership due to a metadata log event. We were the leader at epoch 7614, but in the new epoch 7615, the leader is (none). Reverting to last stable offset 12960831. (org.apache.kafka.controller.QuorumController)* [2025-04-16 16:02:46,295] INFO [QuorumController id=1] processBrokerHeartbeat: event failed with NotControllerException in 90 microseconds. (org.apache.kafka.controller.QuorumController) [2025-04-16 16:02:46,515] INFO [QuorumController id=1] In the new epoch 7616, the leader is (none). (org.apache.kafka.controller.QuorumController) [2025-04-16 16:02:47,192] INFO [QuorumController id=1] processBrokerHeartbeat: event failed with NotControllerExcep...

In the latest version of Kafka Streams does KTable restoration happen in its own thread? Can you point me to where that happens in the code base? Thanks, Chad

Hi Team, Regarding the Kafka cluster upgrade from VersionA to VersionB, I have a query about message format compatibility. Let's assume we have a topic "topicA" containing messages written with the message log format version "v1-a". After the Kafka cluster migration is complete and the broker's log.message.format is set to "v1-b": 1. *Will Kafka automatically migrate the existing messages in "topicA" (written with "v1-a") to the new format "v1-b"?* 2. *Will new messages written to "topicA" after the upgrade be in the "v1-b" format?* 3. *Is it possible for a single topic to contain messages with different log format versions (e.g., both "v1-a" and "v1-b") after an upgrade?* 4. *What meta-data gets changed when a upgrade happens and what factors does it depends on?* I understand that newly created topics after the Kafka upgrade will ...

Hi All,  I'm trying to implement a custom REST extension for Kafka Connect that allows a token to be used for authentication.  When I have a single instance of Kafka connect running in distributed mode (i.e. 1 worker) this works as expected. However, when I add another instance (another worker) I get an error when the worker that received the request has to forward it on to the group leader: Error IO error forwarding REST request: (org.apache.kafka.connect.runtime.rest.RestClient:143) java.util.concurrent.ExecutionException: org.eclipse.jetty.client.HttpResponseException: HTTP protocol violation: Authentication challenge without WWW-Authenticate header  I noticed that someone also had the same issue with the basic authentication extension that comes as part of connect:  https://stackoverflow.com/questions/60977591/kafkaconnect-rest-api-basic-authentication-issue . Has anyone been able to get either the basic authentication or another custom authentication...

Hello Kafka users, developers, and client-developers, This is the first candidate for the release of Apache Kafka 3.9.1. This is a bug-fix release with several fixes, and most importantly, it adds Java 23 support for 3.9 Release notes for the 3.9.1 release: https://dist.apache.org/repos/dist/dev/kafka/3.9.1-rc0/RELEASE_NOTES.html Please download, test, and vote by *Wednesday, April 23, 9:00 AM PT* Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://dist.apache.org/repos/dist/dev/kafka/3.9.1-rc0/ * Docker release artifacts to be voted upon: apache/kafka:3.9.1-rc0 https://hub.docker.com/layers/apache/kafka/3.9.1-rc0/images/sha256-3f7d1298c0ff2cdfac2a65e36c2515ca9288d89c64e071300008bc61843b648e apache/kafka-native:3.9.1-rc0 https://hub.docker.com/layers/apache/kafka-native/3.9.1-rc0/images/sha256-2a7f7f178d862b7f0b41b007732edb1c5882f2f6062c47dd...

Hello everyone, We're experiencing an issue where Kafka clients are significantly delayed in rediscovering the GroupCoordinator after the broker originally assigned as the GroupCoordinator becomes unreachable. In this scenario, while most clients are able to quickly locate a new GroupCoordinator using the FindCoordinator protocol, a few clients are taking as long as max.poll.interval.ms to do so. This delay in rediscovery is causing the group rebalance to be postponed, leading to a prolonged interruption in message consumption. Our Kafka server version is 2.3.1, but the clients are using version 1.1.1. We observed that after the client logs the message: ---- Group coordinator ... is unavailable or invalid, will attempt rediscovery --- it takes about 5 minutes before we see: --- Discovered group coordinator ... --- Unfortunately, due to the older client version (1.1.1), we lack more detailed logs for further insight. Has anyone experienced a similar delay in coordinator rediscovery...

  Hi   Could you please give some clues about the pitfalls of this solution below?   We have 3 ZooKeepers in 3 data centers. 6 Kafka nodes in 2 data centers replication factor=4 insync =3 unclean.leader.election.enable=false We use rack.awareness to distribute partitions across data centers.   We want to stay alive after failing 1 data center. When the data center goes down, we have at least 1 in-sync replica and 1 not insync   The cluster is available only for reading but not for writing.   To become available for writing, we temporarily make insync = 2 (using some automatic service), then after the data center becomes available, set it in sync = 3.   With best regards Evgeny

Hi Omer, We (NetApp Instaclustr) are pretty happy with Karapace (we contributed to it), and provide it as a managed service along with our Apache Kafka offering. I did a blog series on it a while back if you are interested in reading about my experiences with it, starting here with part 1 https://www.instaclustr.com/blog/exploring-karapace-part-1/ (all links at end) Regards, Paul Brebner From: karan alang < karan.alang@gmail.com > Date: Friday, 11 April 2025 at 9:09 am To: users@kafka.apache.org < users@kafka.apache.org > Subject: Re: Schema Registry options for Strimzi Kafka EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Thanks Omer .. I was checking the options and it seems this is one of the most popular Open Source options for Schema Registry. I'm planning on implementing the Apicurio Registry Operator with Strimzi Kafka. Do let me know if there are any specific pointers or issues you might have encountered OR I n...

Because it (usually) doesn't make sense to write to a "remote topic". Doing so would mean it no longer is a simple copy of the source topic. Only MM itself should be writing to a remote topic. That said, it may make sense to replicate write ACLs for the corresponding *source* topics on the remote cluster, however. That is something I've done with external tooling, but IMO it would be problematic for MM to do itself. MM doesn't modify source topics, and it might be surprising if it did. In particular, it would be difficult to determine which sources are "sources of truth" and which should be automatically updated. Ryanne On Fri, Apr 11, 2025, 11:35 AM Mehrtens, Mazrim <mmehrten@amazon.com.invalid> wrote: > Does anyone know why MirrorMaker2 doesn't replicate write ACLs? > > This is the logic MM2 uses for choosing ACLs to replicate, which excludes > GROUP resources and ALLOW WRITE permissions: > > >...

Does anyone know why MirrorMaker2 doesn't replicate write ACLs? This is the logic MM2 uses for choosing ACLs to replicate, which excludes GROUP resources and ALLOW WRITE permissions: https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java - L425-L433< https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L425-L433 > MirrorSourceConnector.java< https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java > ListAclBinding< https://app.slack.com/client/AclBinding > filteredBindings = rawBindings.get().stream() .filter(x -> x.pattern().resourceType() == ResourceType.TOPIC) .filter(x -> x.pattern().patternType() == PatternType.LITERAL) .filter(this::shouldReplicateAcl) .fil...

Hello Everyone , Recently we faced an issue where produce request to one topic was failing on one of the brokers. On further investigation, we found "java.lang.ClassCastException: org.apache.kafka.common.TopicPartition" in the stacktrace. The issue was mitigated by restart the problematic broker. We are using kafka version 2.8.2, and want to know if there is any existing reported bug with this version. Stacktrace: [2025-04-01T12:55:19,904Z] [INFO ] [kafka-scheduler-4] [kafka.log.LogSegment] Deleted time index /data/kafka/upsert-69/00000000003318491578.timeindex.deleted. java.lang.ClassCastException: org.apache.kafka.common.TopicPartition cannot be cast to java.lang.Long       at java.lang.Long.compareTo(Long.java:54)       at java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)       at java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:682)       at java.util.concurrent.ConcurrentSkipListMa...

Congrats TengYao! ------------------ Josep Prat Open Source Engineering Director, Aiven josep.prat@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B On Fri, Apr 11, 2025, 04:00 Luke Chen < showuon@gmail.com > wrote: > Congrats TengYao! Well deserved! > > Luke > > On Fri, Apr 11, 2025 at 1:51 AM Kuan Po Tseng < brandboat@apache.org > > wrote: > > > Congrats TengYao! > > > > Kuan-Po Tseng > > On 2025/04/10 15:54:58 "Matthias J. Sax" wrote: > > > The Project Management Committee (PMC) for Apache Kafka is pleased to > > > announce TengYao Chi as a new Kafka committer. > > > > > > TengYao has been active in the project since April of 2024 and has > > > contributed a great de...

Congrats TengYao! Well deserved! Luke On Fri, Apr 11, 2025 at 1:51 AM Kuan Po Tseng < brandboat@apache.org > wrote: > Congrats TengYao! > > Kuan-Po Tseng > On 2025/04/10 15:54:58 "Matthias J. Sax" wrote: > > The Project Management Committee (PMC) for Apache Kafka is pleased to > > announce TengYao Chi as a new Kafka committer. > > > > TengYao has been active in the project since April of 2024 and has > > contributed a great deal in that time. As a matter of fact, he has the > > most commits across all contributors in the last 12 month: > > https://github.com/apache/kafka/graphs/contributors?from=4%2F6%2F2024 > > > > Please join us in welcoming TengYao to their new role and responsibility > > in the Kafka community. > > > > > > Matthias J. Sax > > On behalf of the Apache Kafka PMC > > >

Thanks Omer .. I was checking the options and it seems this is one of the most popular Open Source options for Schema Registry. I'm planning on implementing the Apicurio Registry Operator with Strimzi Kafka. Do let me know if there are any specific pointers or issues you might have encountered OR I need to be aware of. Also, from what I understand - this has an Apache 2.0 license, so we use this without restrictions in commercial products as well. regds, Karan Alang On Fri, Mar 21, 2025 at 12:49 AM Ömer Şiar Baysal < osiarbaysal@gmail.com > wrote: > Hi, > > Like Strimzi, Red Hat folks has its schema registry offering called > Apicurio Registry which plays nicely with k8s through its operator. > > Currently Apicurio suit is getting regular updates. > > Good luck, > OSB > > > On Thu, Mar 20, 2025, 22:23 karan alang < karan.alang@gmail.com > wrote: > > > Hello - > > > > ...

Congrats TengYao! Kuan-Po Tseng On 2025/04/10 15:54:58 "Matthias J. Sax" wrote: > The Project Management Committee (PMC) for Apache Kafka is pleased to > announce TengYao Chi as a new Kafka committer. > > TengYao has been active in the project since April of 2024 and has > contributed a great deal in that time. As a matter of fact, he has the > most commits across all contributors in the last 12 month: > https://github.com/apache/kafka/graphs/contributors?from=4%2F6%2F2024 > > Please join us in welcoming TengYao to their new role and responsibility > in the Kafka community. > > > Matthias J. Sax > On behalf of the Apache Kafka PMC >

Kudos to TengYao! On Fri, Apr 11, 2025 at 12:15 AM Bruno Cadonna < cadonna@apache.org > wrote: > Well deserved! > > Congrats! > > Best, > Bruno > > On 10.04.25 18:01, Lucas Brutschy wrote: > > Congrats! > > > > On Thu, Apr 10, 2025 at 5:59 PM Chia-Ping Tsai < chia7712@gmail.com > > wrote: > >> > >> You are the best! TengYao > >> > >> Matthias J. Sax < mjsax@apache.org > 於 2025年4月10日 週四 下午11:55寫道： > >> > >>> The Project Management Committee (PMC) for Apache Kafka is pleased to > >>> announce TengYao Chi as a new Kafka committer. > >>> > >>> TengYao has been active in the project since April of 2024 and has > >>> contributed a great deal in that time. As a matter of fact, he has the > >>> most commits across all contributors in the last 12 month: > >>> https://github.com...

Well deserved! Congrats! Best, Bruno On 10.04.25 18:02, Sanskar Jhajharia wrote: > Congratulations Apoorv! > > > > On Thu, Apr 10, 2025 at 9:32 PM Lucas Brutschy > <lbrutschy@confluent.io.invalid> wrote: > >> Congrats! Well deserved, Apoorv! >> >> On Thu, Apr 10, 2025 at 5:59 PM Chia-Ping Tsai < chia7712@gmail.com > wrote: >>> >>> Welcome Apoorv as a committer! You deserve it! >>> >>> Matthias J. Sax < mjsax@apache.org > 於 2025年4月10日 週四 下午11:56寫道： >>> >>>> The Project Management Committee (PMC) for Apache Kafka is pleased to >>>> announce Apoorv Mittal as a new Kafka committer. >>>> >>>> Apoorv has been active in the project since October of 2023 and has >>>> contributed a great deal in that time. Most notably, he implemented >>>> KIP-714 (Client Metrics and Observability), and is...

Well deserved! Congrats! Best, Bruno On 10.04.25 18:01, Lucas Brutschy wrote: > Congrats! > > On Thu, Apr 10, 2025 at 5:59 PM Chia-Ping Tsai < chia7712@gmail.com > wrote: >> >> You are the best! TengYao >> >> Matthias J. Sax < mjsax@apache.org > 於 2025年4月10日 週四 下午11:55寫道： >> >>> The Project Management Committee (PMC) for Apache Kafka is pleased to >>> announce TengYao Chi as a new Kafka committer. >>> >>> TengYao has been active in the project since April of 2024 and has >>> contributed a great deal in that time. As a matter of fact, he has the >>> most commits across all contributors in the last 12 month: >>> https://github.com/apache/kafka/graphs/contributors?from=4%2F6%2F2024 >>> >>> Please join us in welcoming TengYao to their new role and responsibility >>> in the Kafka community. >>> >>> ...

Congratulations! Happy to contribute to Kafka with you. On 2025/04/10 15:54:58 "Matthias J. Sax" wrote: > The Project Management Committee (PMC) for Apache Kafka is pleased to > announce TengYao Chi as a new Kafka committer. > > TengYao has been active in the project since April of 2024 and has > contributed a great deal in that time. As a matter of fact, he has the > most commits across all contributors in the last 12 month: > https://github.com/apache/kafka/graphs/contributors?from=4%2F6%2F2024 > > Please join us in welcoming TengYao to their new role and responsibility > in the Kafka community. > > > Matthias J. Sax > On behalf of the Apache Kafka PMC >

Congratulations Apoorv! On Thu, Apr 10, 2025 at 9:32 PM Lucas Brutschy <lbrutschy@confluent.io.invalid> wrote: > Congrats! Well deserved, Apoorv! > > On Thu, Apr 10, 2025 at 5:59 PM Chia-Ping Tsai < chia7712@gmail.com > wrote: > > > > Welcome Apoorv as a committer! You deserve it! > > > > Matthias J. Sax < mjsax@apache.org > 於 2025年4月10日 週四 下午11:56寫道： > > > > > The Project Management Committee (PMC) for Apache Kafka is pleased to > > > announce Apoorv Mittal as a new Kafka committer. > > > > > > Apoorv has been active in the project since October of 2023 and has > > > contributed a great deal in that time. Most notably, he implemented > > > KIP-714 (Client Metrics and Observability), and is a major contributor > > > to KIP-932 (Queues for Kafka). > > > > > > Please join us in welcoming Apoorv to their new role and responsib...

Congrats TengYao! -Bill On Thu, Apr 10, 2025 at 11:59 AM Chia-Ping Tsai < chia7712@gmail.com > wrote: > You are the best! TengYao > > Matthias J. Sax < mjsax@apache.org > 於 2025年4月10日 週四 下午11:55寫道： > > > The Project Management Committee (PMC) for Apache Kafka is pleased to > > announce TengYao Chi as a new Kafka committer. > > > > TengYao has been active in the project since April of 2024 and has > > contributed a great deal in that time. As a matter of fact, he has the > > most commits across all contributors in the last 12 month: > > https://github.com/apache/kafka/graphs/contributors?from=4%2F6%2F2024 > > > > Please join us in welcoming TengYao to their new role and responsibility > > in the Kafka community. > > > > > > Matthias J. Sax > > On behalf of the Apache Kafka PMC > > >

Congrats! On Thu, Apr 10, 2025 at 5:59 PM Chia-Ping Tsai < chia7712@gmail.com > wrote: > > You are the best! TengYao > > Matthias J. Sax < mjsax@apache.org > 於 2025年4月10日 週四 下午11:55寫道： > > > The Project Management Committee (PMC) for Apache Kafka is pleased to > > announce TengYao Chi as a new Kafka committer. > > > > TengYao has been active in the project since April of 2024 and has > > contributed a great deal in that time. As a matter of fact, he has the > > most commits across all contributors in the last 12 month: > > https://github.com/apache/kafka/graphs/contributors?from=4%2F6%2F2024 > > > > Please join us in welcoming TengYao to their new role and responsibility > > in the Kafka community. > > > > > > Matthias J. Sax > > On behalf of the Apache Kafka PMC > >

Congrats! Well deserved, Apoorv! On Thu, Apr 10, 2025 at 5:59 PM Chia-Ping Tsai < chia7712@gmail.com > wrote: > > Welcome Apoorv as a committer! You deserve it! > > Matthias J. Sax < mjsax@apache.org > 於 2025年4月10日 週四 下午11:56寫道： > > > The Project Management Committee (PMC) for Apache Kafka is pleased to > > announce Apoorv Mittal as a new Kafka committer. > > > > Apoorv has been active in the project since October of 2023 and has > > contributed a great deal in that time. Most notably, he implemented > > KIP-714 (Client Metrics and Observability), and is a major contributor > > to KIP-932 (Queues for Kafka). > > > > Please join us in welcoming Apoorv to their new role and responsibility > > in the Kafka community. > > > > > > Matthias J. Sax > > On behalf of the Apache Kafka PMC > >

Congratulations Apoorv! Well deserved. -Bill On Thu, Apr 10, 2025 at 11:59 AM Matthias J. Sax < mjsax@apache.org > wrote: > The Project Management Committee (PMC) for Apache Kafka is pleased to > announce Apoorv Mittal as a new Kafka committer. > > Apoorv has been active in the project since October of 2023 and has > contributed a great deal in that time. Most notably, he implemented > KIP-714 (Client Metrics and Observability), and is a major contributor > to KIP-932 (Queues for Kafka). > > Please join us in welcoming Apoorv to their new role and responsibility > in the Kafka community. > > > Matthias J. Sax > On behalf of the Apache Kafka PMC >

Congratulation! Well-deserved! PoAn On 2025/04/10 15:54:58 "Matthias J. Sax" wrote: > The Project Management Committee (PMC) for Apache Kafka is pleased to > announce TengYao Chi as a new Kafka committer. > > TengYao has been active in the project since April of 2024 and has > contributed a great deal in that time. As a matter of fact, he has the > most commits across all contributors in the last 12 month: > https://github.com/apache/kafka/graphs/contributors?from=4%2F6%2F2024 > > Please join us in welcoming TengYao to their new role and responsibility > in the Kafka community. > > > Matthias J. Sax > On behalf of the Apache Kafka PMC >

Welcome Apoorv as a committer! You deserve it! Matthias J. Sax < mjsax@apache.org > 於 2025年4月10日 週四 下午11:56寫道： > The Project Management Committee (PMC) for Apache Kafka is pleased to > announce Apoorv Mittal as a new Kafka committer. > > Apoorv has been active in the project since October of 2023 and has > contributed a great deal in that time. Most notably, he implemented > KIP-714 (Client Metrics and Observability), and is a major contributor > to KIP-932 (Queues for Kafka). > > Please join us in welcoming Apoorv to their new role and responsibility > in the Kafka community. > > > Matthias J. Sax > On behalf of the Apache Kafka PMC >

You are the best! TengYao Matthias J. Sax < mjsax@apache.org > 於 2025年4月10日 週四 下午11:55寫道： > The Project Management Committee (PMC) for Apache Kafka is pleased to > announce TengYao Chi as a new Kafka committer. > > TengYao has been active in the project since April of 2024 and has > contributed a great deal in that time. As a matter of fact, he has the > most commits across all contributors in the last 12 month: > https://github.com/apache/kafka/graphs/contributors?from=4%2F6%2F2024 > > Please join us in welcoming TengYao to their new role and responsibility > in the Kafka community. > > > Matthias J. Sax > On behalf of the Apache Kafka PMC >

The Project Management Committee (PMC) for Apache Kafka is pleased to announce Apoorv Mittal as a new Kafka committer. Apoorv has been active in the project since October of 2023 and has contributed a great deal in that time. Most notably, he implemented KIP-714 (Client Metrics and Observability), and is a major contributor to KIP-932 (Queues for Kafka). Please join us in welcoming Apoorv to their new role and responsibility in the Kafka community. Matthias J. Sax On behalf of the Apache Kafka PMC

The Project Management Committee (PMC) for Apache Kafka is pleased to announce TengYao Chi as a new Kafka committer. TengYao has been active in the project since April of 2024 and has contributed a great deal in that time. As a matter of fact, he has the most commits across all contributors in the last 12 month: https://github.com/apache/kafka/graphs/contributors?from=4%2F6%2F2024 Please join us in welcoming TengYao to their new role and responsibility in the Kafka community. Matthias J. Sax On behalf of the Apache Kafka PMC

Could you share more info on this? Like how much memory is allocated and whats the usage? Topic config settings, is it configured for large messages, are you using compacted topics etc? On Tue, Apr 8, 2025 at 5:59 AM Manesh VM < manesh@saturnme.com > wrote: > My kafka app server is consuming high memory then we allocated could you > help us with the issue? > > -- > Regards, > Manesh V M > Phone : +971 0521366974 > Email : manesh@saturnme.com > Saturn ME | www.saturnme.com >

Hey! We recently worked on a massive migration project: migrating a petabyte-scale Hadoop cluster to Kubernetes, which is fully open-source and built for scale. This wasn't just a lift-and-shift. It involved running Spark, Trino, Airflow, JupyterHub, Superset, and HDFS (yes, still!) across a K8s-based platform with full automation via ArgoCD. It had to support multiple teams, secure access, and be easy to maintain. We did this for Play, Poland's largest telco, and shared the story at last year's Data & AI Warsaw Tech Summit. The full write-up is now live: 👉 https://getindata.com/blog/play-case-migrating-hadoop-cluster-kubernetes-open-source/ If you're exploring similar challenges or just curious about how we handled orchestration, resource management, or multi-tenant deployments, you'll probably find some takeaways. Also—this year's Data & AI Summit is around the corner. If you want to hear more like this (or chat with us), we...

My kafka app server is consuming high memory then we allocated could you help us with the issue? -- Regards, Manesh V M Phone : +971 0521366974 Email : manesh@saturnme.com Saturn ME | www.saturnme.com

Hi all, Community Over Code < https://communityovercode.org/ >, the ASF conference, will be held in Minneapolis, Minnesota, September 11-14, 2025. The call for presentations < https://communityovercode.org/call-for-presentations/ > is open now through April 21, 2025. I am one of the co-chairs for the stream processing track, and we would love to see you there and hope that you will consider submitting a talk. About the Streaming track: There are many top-level ASF projects which focus on and push the envelope for stream and event processing. ActiveMQ, Beam, Bookkeeper, Camel, Flink, Kafka, Pulsar, RocketMQ, and Spark are all house-hold names in the stream processing and analytics world at this point. These projects show that stream processing has unique characteristics requiring deep expertise. On the other hand, users need easy to apply solutions. The streaming track will host talks focused on the use cases and advances of these projects as we...

Hi Kafka team, Hope you're doing well. I'm investigating how to use the MirrorMaker 2 source connector to replicate data between different Kafka clusters. One scenario is that in the source Kafka cluster, we have multiple topics, then we expect all messages from these topics will be sent to one topic in the target Kafka. That is, all messages from multiple topics will be aggregated to one topic. We plan to use transformation to implement the feature. org.apache.kafka.connect.transforms.RegexRouter ( https://docs.confluent.io/kafka-connectors/transforms/current/regexrouter.html#description ) can update the record's topic using the configured regular expression and replacement string, which can be used to change the original topics to the same name. Here's the example: ``` transforms=RenameTopicSimple transforms.RenameTopicSimple.type=org.apache.kafka.connect.transforms.RegexRouter transforms.RenameTopicSimple.regex=.* transforms.RenameTopicSimple.r...

Hi Richard, Thanks for the pointer. Yes, I am aware of it, and I have altered the strings before sharing, so those are not actual credentials. Thanks. From: Richard Bosch < richard.bosch@axual.com > Date: Wednesday, 2 April 2025 at 8:57 pm To: users@kafka.apache.org < users@kafka.apache.org > Subject: Re: Kafka OAuth authentication using OKTA EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Hi Anup, I don't think I can help you with your problem, but it appears you've included some credentials in your message with the KAFKA_OPTS Please check that you haven't got an issue now with leaked credentials Kind regards, Richard Bosch Developer Advocate Axual BV https://urldefense.com/v3/__https://axual.com/__;!!Nhn8V6BzJA!RQeDvu-hdU1iOIzuvIsxylULaELy53HCaZJfeJcLN7B8lHOMb47C1xuILqQucvFp47gnbrO3J0JkdRxH6CPFWKq8XFMUaMo$ < https://urldefense.com/v3/__https:/axual.com/__;!!Nhn8V6BzJA!RQeDvu-hdU1iOIzuvIsxylULaELy...

Hi Anup, I don't think I can help you with your problem, but it appears you've included some credentials in your message with the KAFKA_OPTS Please check that you haven't got an issue now with leaked credentials Kind regards, Richard Bosch Developer Advocate Axual BV https://axual.com/ On Wed, Apr 2, 2025 at 4:58 PM Shirolkar, Anup <Anup.Shirolkar@netapp.com.invalid> wrote: > Hi Kirk, > > > > Sorry I didn't realise that my query was replied. > > I am following this blog > https://medium.com/@apoorv.adarsh/kafka-and-okta-a-delightful-duo-on-your-local-windows-environment-a5bab3456498 > > > > The change is I am performing the steps on Ubuntu. > > I have made necessary changes to adopt the windows suitable declarations > for Ubuntu. > > > > I am getting the error – page not found from okta. > > I am attaching the server.properties variant with the new d...

Hi Kirk,   Sorry I didn't realise that my query was replied. I am following this blog   https://medium.com/@apoorv.adarsh/kafka-and-okta-a-delightful-duo-on-your-local-windows-environment-a5bab3456498   The change is I am performing the steps on Ubuntu. I have made necessary changes to adopt the windows suitable declarations for Ubuntu.   I am getting the error – page not found from okta. I am attaching the server.properties variant with the new details I added in addition to the defaults also the jaas config. Here is the KAFKA_OPTS I exported –   export KAFKA_OPTS="-Djava.security.auth.login.config=/home/ubuntu/Downloads/kafka_2.12-3.7.2/config/kafka_sasl_oauth_jaas.conf -DclientId=0oanzubxpo3CA6xVV5d8 -DOAUTH_WITH_SSL=true -DOAUTH_LOGIN_SERVER=https://dev-83775998.okta.com/oauth2/default -DOAUTH_LOGIN_ENDPOINT=/oauth2/default/v1/token -DOAUTH_LOGIN_GRANT_TYPE=0oanzubxpo3CA6xVV5d8 -DOAUTH_LOGIN_SCOPE=kafka -DOAUTH_CHECK_AUD...

Consider the following Kafka controller setup. There are three controllers c1, c2 and c3, each on its own hardware. All controllers are voters and let's assume c1 is the leader. Assume new servers can be added as needed to replace broken one, but broken/lost servers cannot be brought back. If a new server is needed, it'd be c4 and then the one after that would be c5. Now, let's assume the controllers c2 and c3 are irreversibly lost for some reason, e.g. because of an unrecoverable hardware failure. In order to repair the controller cluster, i.e. to make it again have enough voters (at least two), we'd like to remove the voters c2 and c3 (make them observers) and add controllers c4 and c5 to the cluster as voters. In order to do this, we create those servers and they join the controller cluster as observers. However, adding c4 or c5 as a voter is currently not possible in this situation, nor is making c2 and c3 observers. The controller cluster is not able...

Hello Kafka users, We are really stuck in this issue where we cannot use the special character in the keystore (and truststore) password from the shell. We would we grateful for any possible solution. Regards, Deepak ________________________________ From: Deepak Jain < deepak.jain@cumulus-systems.com > Sent: Friday, March 28, 2025 10:18 AM To: users@kafka.apache.org < users@kafka.apache.org > Subject: Re: Kafka process fails to start when special character is present in Keystore password in SSL encryption and SASL authentication Caution: This email originated from outside of the organization. Please call and confirm with the sender before opening attachments or clicking links inside the email. Thanks Maciej for the prompt response. But we tried the Percent encoding and unicode encoding but it did not work either. Please note that we are able to pass the special character via server.properties but not with the zookeeper config in the shell script. ...

Thanks, Vruttant, for sharing.   it means we need to have a few MM2 deployments, onlybecause the first step is failing due to the batch limit. It would be more nice if MM2 will have some approach forbatch size on this phase     On Tuesday, April 1, 2025 at 01:13:57 PM GMT+3, Vruttant Mankad < vamankad@gmail.com > wrote: Hi, We faced a similar problem when mirroring huge amounts of topics and partitions . Kafka version 3.7.2 We saw an error of BoundedListTooLong < https://github.com/apache/kafka/blob/3.7.0/server-common/src/main/java/org/apache/kafka/server/mutable/BoundedListTooLongException.java > Excpetion in the kafka logs. According to kafka code < https://github.com/apache/kafka/blob/e3809682e2a0a312c6d5594fbc463b6cb0bb2273/metadata/src/main/java/org/apache/kafka/controller/QuorumController.java#L185C1-L186C1 >, kraft allows only 10k operations per user in a single transaction. Though ...