Kafka

Hello, My team is trying to setup multiple Kafka clusters on RHEL servers for our clients to use. I am wondering if there was any specific information that I could get to set the server.properties file to enable OAUTHBEARER as the sasl.mechanism. I would love to also receive a copy of the server.properties file that would have this enabled to easily compare our current implementation to the one provided. Thank you, Kerwn Shaw

Hi Team, Can you please add me to kafka group as i work on kafka in my company. Thanks, Prince

Hi Team, We are experiencing some odd behaviour of CooperativeStickyAssignor - during rebalancing of partitions a consumer receives new partition and keeps old ones. Still there is CommitFailedException received for one of the old partitions and the offsets are not retried, we receive directly the next batch and thus skipping some of the offsets.Partition in question is receivedPartition: [27] and as you can see offset 21551049 is received, then the exception and next received offset is  21551051. How can we handle this? Logs:  INFO;com.rewe.digital.mystique.kafka.consumer.listener.StringKafkaListener;Received kafka message for topic: [picking-job-v1], key: [d52e171e-7a53-4e8c-8f4a-220d8e493237], offset: [21551051],  receivedPartition: [27], time: 1725975726868, publishedOn: 1725975726848;INFO;org.springframework.kafka.listener.KafkaMessageListenerContainer;productrecall-pickingjob-message-consumer: partitions assigned: [picking-job-v1-20];INFO;...

Hello Kafka Team , I'm facing an issue with *Kafka* running in *KRaft mode* (without ZooKeeper) on a *Windows* environment, and I keep getting an AccessDeniedException in the logs when Kafka attempts to delete specific snapshot files. [2024-09-14 11:28:22,849] ERROR Error deleting snapshot files C:\tmp\kraft-combined-logs\__cluster_metadata-0\00000000000000943744-0000000001.checkpoint and C:\tmp\kraft-combined-logs\__cluster_metadata-0\00000000000000943744-0000000001.checkpoint.deleted (org.apache.kafka.snapshot.Snapshots) java.nio.file.AccessDeniedException: C:\tmp\kraft-combined-logs\__cluster_metadata-0\00000000000000943744-0000000001.checkpoint.deleted at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89) *Environment Details*: - *Kafka version*: Using KRaft mode (without ZooKeeper) - *OS*: Windows 10 - *Log retention settings*: - log.retention.bytes=1073741824 (1 GB) - log.segment.b...

Hi all, Thanks, everyone who took a look at RC0. Let's sink this RC since we discovered several problems: - the current error message for KAFKA-17543 is unclear (fixed by PR #17210) - docs/ops.html has not been updated for KIP-853 - some maven artifacts are missing - the documentation generation is slightly wonky I'll create a new RC in a day or two that will fix these issues. best, Colin On Wed, Sep 18, 2024, at 07:21, Jakub Scholz wrote: > One more thing I noticed - the documentation at > https://kafka.apache.org/39/documentation.html#brokerconfigs seems to be > missing all the configuration options for brokers, producers, consumers etc. > > Not sure if that is normal for RCs or if this is some bug in the docs > generator. > > Thanks & Regards > Jakub > > On Wed, Sep 18, 2024 at 5:07 AM Colin McCabe < cmccabe@apache.org > wrote: > >> Hello Kafka users, developers and client-developers, >...

One more thing I noticed - the documentation at https://kafka.apache.org/39/documentation.html#brokerconfigs seems to be missing all the configuration options for brokers, producers, consumers etc. Not sure if that is normal for RCs or if this is some bug in the docs generator. Thanks & Regards Jakub On Wed, Sep 18, 2024 at 5:07 AM Colin McCabe < cmccabe@apache.org > wrote: > Hello Kafka users, developers and client-developers, > > > > This is the first candidate for release of Apache Kafka 3.9.0. > > > - This is a major release, the final one in the 3.x line. (There may of > course be other minor releases in this line, such as 3.9.1.) > - Tiered storage will be considered production-ready in this release. > - This will be the final major release to feature the deprecated ZooKeeper > mode. > > This release includes the following KIPs: > - KIP-853: Support dynamically changing KRaft controller membersh...

Hi all, I think this is my error since I only "closed" one entry, and "dropped" the others. Sorry about that. Let me try the Maven upload task again when I'm in front of my laptop. best, Colin On Wed, Sep 18, 2024, at 04:37, 'Josep Prat' via kafka-clients wrote: > I can see the Stream Quickstart ones in the staging repo, but only these. > When I did the 3.8 release I realized that https://repository.apache.org/#stagingRepositories had 3 different entries (repositories) every time I tried to upload to maven central (1 was kind of faulty with only togdor jar file and no metadata, and the other 2 were for StreamQuickStart jars, and all the other jars). Taking a look there now, I see only the repository for the quickstart is present. Colin, did you happen to see the other repositories as well? > > Best, > > > On Wed, Sep 18, 2024 at 11:26 AM Luke Chen < showuon@gmail.com > wrote: >> Same here, no 3.9.0 a...

I can see the Stream Quickstart ones in the staging repo, but only these. When I did the 3.8 release I realized that https://repository.apache.org/#stagingRepositories had 3 different entries (repositories) every time I tried to upload to maven central (1 was kind of faulty with only togdor jar file and no metadata, and the other 2 were for StreamQuickStart jars, and all the other jars). Taking a look there now, I see only the repository for the quickstart is present. Colin, did you happen to see the other repositories as well? Best, On Wed, Sep 18, 2024 at 11:26 AM Luke Chen < showuon@gmail.com > wrote: > Same here, no 3.9.0 artifacts in the staging repo. > > Thanks. > Luke > > > On Wed, Sep 18, 2024 at 4:49 PM Jakub Scholz < jakub@scholz.cz > wrote: > > > Hi Colin, > > > > Thanks for the RC. > > > > It looks like the Maven artifacts are missing for 3.9.0 in the staging > > repo. Ar...

Same here, no 3.9.0 artifacts in the staging repo. Thanks. Luke On Wed, Sep 18, 2024 at 4:49 PM Jakub Scholz < jakub@scholz.cz > wrote: > Hi Colin, > > Thanks for the RC. > > It looks like the Maven artifacts are missing for 3.9.0 in the staging > repo. Are we still waiting for some staging job to complete? Or is that > some CDN issue and I'm the only one not seeing them? Or is that some > general problem? > > Thanks & Regards > Jakub > > On Wed, Sep 18, 2024 at 5:07 AM Colin McCabe < cmccabe@apache.org > wrote: > > > Hello Kafka users, developers and client-developers, > > > > > > > > This is the first candidate for release of Apache Kafka 3.9.0. > > > > > > - This is a major release, the final one in the 3.x line. (There may of > > course be other minor releases in this line, such as 3.9.1.) > > - Tiered storage will be considere...

Hi Colin, Thanks for the RC. It looks like the Maven artifacts are missing for 3.9.0 in the staging repo. Are we still waiting for some staging job to complete? Or is that some CDN issue and I'm the only one not seeing them? Or is that some general problem? Thanks & Regards Jakub On Wed, Sep 18, 2024 at 5:07 AM Colin McCabe < cmccabe@apache.org > wrote: > Hello Kafka users, developers and client-developers, > > > > This is the first candidate for release of Apache Kafka 3.9.0. > > > - This is a major release, the final one in the 3.x line. (There may of > course be other minor releases in this line, such as 3.9.1.) > - Tiered storage will be considered production-ready in this release. > - This will be the final major release to feature the deprecated ZooKeeper > mode. > > This release includes the following KIPs: > - KIP-853: Support dynamically changing KRaft controller membership > - KIP-...

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.9.0. - This is a major release, the final one in the 3.x line. (There may of course be other minor releases in this line, such as 3.9.1.) - Tiered storage will be considered production-ready in this release. - This will be the final major release to feature the deprecated ZooKeeper mode. This release includes the following KIPs: - KIP-853: Support dynamically changing KRaft controller membership - KIP-1057: Add remote log metadata flag to the dump log tool - KIP-1049: Add config log.summary.interval.ms to Kafka Streams - KIP-1040: Improve handling of null...

I'm wondering if I'm missing something about configuring SSL for Kraft. We currently have a Zookeeper cluster where all inter-broker & client communication uses SSL. We manage our own CA and generate all the necessary keystore/truststore jks files for our brokers and clients. We've had this in place for several years. I've recently been trying to set up a Kraft-based cluster. And since a Kraft controller is configured much the same as a Kafka broker is, I figured I would try to have the Kraft controllers communicate to each other over SSL, same as I have the brokers do. When I enable SSL for the Kraft controller I get this error on startup, "Invalid value javax.net.ssl.SSLHandshakeException: No available authentication scheme for configuration A client SSLEngine created with the provided settings can't connect to a server SSLEngine created with those settings." I am running everything in Docker containers, using Confluent's im...

Hi Team, How we can see old messages of kafka topics. can you please help with CLI command. -- Thanks & Regard Kiran Satpute (9921424521)

As pointed out by Josep Prat and James Olsen the analysis of the problem was incorrect. After sending my original post I realized that the addresses of the servers in the bootstrap.servers url were separated by a semicolon, instead of a comma. In the example error message, I manually typed the names of dummy servers, (correctly) using a comma to separate their values. I have tested v3.8.0 with a comma separated value and this works fine. My apologies for the inconvenience.

You should be all set. On 9/13/24 7:47 PM, 最红 wrote: > Hi, > > > > > I'd like to apply for a jira id to contribute to Kafka. > > > > > Jira ID: veryfirefly > > > > > Thank you

Hi, I'd like to apply for a jira id to contribute to Kafka. Jira ID: veryfirefly Thank you

Hi everyone, I am using Mirrormaker 2. From what I understand, mirrormaker2 does not store its offsets in the source cluster's __consumer_offsets topic. Rather, it stores its offsets in the destination cluster in the value defined by offset.storage.topic In the __consumer_offsets world, I could manually change offsets by doing this kafka-consumer-groups.sh --bootstrap-server kafka:9092 --group groupname --reset-offsets --to-latest--execute Is there any similar way to manipulate mirrormaker2, to tell it to start (re)consuming from a specific offset in a source topic? Thanks, -James

Hey everyone, Bumping this topic to see if anyone has experience migrating Streams jobs between clusters with Mirror Maker, and how you handled repartition topics. Thanks! Greg On Mon, Aug 12, 2024 at 8:55 AM Greg Harris < greg.harris@aiven.io > wrote: > Hi all, > > We were recently working to set up a MirrorMaker2 flow to migrate a > Streams job between an existing Kafka cluster and a new Kafka cluster. We > noticed the following behavior: > > * Streams sets infinite retention on the repartition topic, and uses the > AdminClient deleteRecords call to implement "active retention" by deleting > data that is no longer necessary. > * MirrorMaker2 mirrors the data to topic with infinite retention, but > _without_ the active AdminClient deleteRecords calls running, as it is not > known to be a repartition topic and the destination job is not yet active. > * The target topic grows without bound and exhausts disk...

Hello Kafka Team, i am using a three node kafka broker cluster but recently compression issues occur on brocker side. Please provide guillines. broker property: process.roles=broker,controller node.id =0 controller.quorum.voters=0@kafka-0-0.kafka.prod.svc.cluster.local :9093,1@kafka-1-0.kafka.prod.svc.cluster.local :9093,2@kafka-2-0.kafka.prod.svc.cluster.local:9093 listeners=PLAINTEXT://kafka-0-0.kafka.prod.svc.cluster.local:9092,CONTROLLER://kafka-0-0.kafka.prod.svc.cluster.local:9093 inter.broker.listener.name =PLAINTEXT advertised.listeners=PLAINTEXT://kafka-0-0.kafka.prod.svc.cluster.local:9092 controller.listener.names=CONTROLLER listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL log.dirs=/mnt/kafka/0 num.partitions=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 group.initia...

Hello Akash, I am getting below error logs sometimes when a kafka producer is going to produce some messages to the topic. *Logs:-* org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition `Error : org.apache.kafka.common.errors.OutOfOrderSequenceException: The broker received an out of order sequence number On Fri, Aug 30, 2024 at 9:04 AM Vikram Singh < vikram.singh@nciportal.com > wrote: > Hello Akash, > > I will try by changing compression type and test if the issue persists, > but for now the issue is resolved I guess. I just changed > replication_factor of the topic to 3. It was like a 3-node kafka cluster > and replication_factor was configured 2 for the topic. > > On Wed, Aug 28, 2024 at 2:55 PM A...

Looking at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(List<String>, ClientDnsLookup) shows that error message is referring to a singular url from the list. So the issue is earlier in the chain - the list is not being decoded from the CSV. > On 12 Sep 2024, at 03:03, Josep Prat <josep.prat@aiven.io.INVALID> wrote: > > Hi Bart, > > I'm not an expert in this area, but looking at the change [1] in question > and the usage of this regex, I'd say it was never the intention to accept > comma separated hosts with ports (and optional protocol). > The only 2 usages of HOST_PORT_PATTERN can be found in lines 538 [2] and > 548 [3] in the same Utils class. Also looking at the tests it seems the > expectation has always been to parse a single host with a port pattern > (with optional protocol). > > [1]: https://github.com/apache/kafka/pull/16048 > [2]: > https://github.com/apache/kafka/blob/3.8...

Hi Bart, I'm not an expert in this area, but looking at the change [1] in question and the usage of this regex, I'd say it was never the intention to accept comma separated hosts with ports (and optional protocol). The only 2 usages of HOST_PORT_PATTERN can be found in lines 538 [2] and 548 [3] in the same Utils class. Also looking at the tests it seems the expectation has always been to parse a single host with a port pattern (with optional protocol). [1]: https://github.com/apache/kafka/pull/16048 [2]: https://github.com/apache/kafka/blob/3.8/clients/src/main/java/org/apache/kafka/common/utils/Utils.java#L538 [3]: https://github.com/apache/kafka/blob/3.8/clients/src/main/java/org/apache/kafka/common/utils/Utils.java#L548 [4]: https://github.com/apache/kafka/blob/3.8/clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java#L104 Best, On Wed, Sep 11, 2024 at 4:23 PM Hamer, Bart < Bart.Hamer@tennet.eu > wrote: > Hi, > > O...

Hi, One of our Quarkus applications connects to Kafka topics using the kafka-clients  library. Quarkus provides support for Apache Kafka through the SmallRye Reactive Messaging  framework. According to the SmallRye documentation the bootstrap.servers  property in the application.properties file should contain a comma separated list (see the Configuration Reference  section on this page: https://smallrye.io/smallrye-reactive-messaging/smallrye-reactive-messaging/3.4/kafka/kafka.html ). With kafka-clients v3.7.1 this worked fine, but after upgrading to v3.8.0 starting the application fails with this exception: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer Caused by: org.apache.kafka.common.config.ConfigException: Invalid url in bootstrap.servers: kafka://test1:9094,kafka://test2:9094 This is caused by the changed regex pattern named HOST_PORT_PATTERN in the org.apache.kafka.common.util...

congrats! On Tue, Sep 10, 2024 at 5:02 PM Kirk True < kirk@kirktrue.pro > wrote: > Congratulations Jeff! > > > On Sep 9, 2024, at 6:25 PM, Matthias J. Sax < mjsax@apache.org > wrote: > > > > Congrats! > > > > On 9/9/24 12:34 PM, José Armando García Sancio wrote: > >> Congratulations Jeff! > >> On Mon, Sep 9, 2024 at 11:45 AM Justine Olshan > >> <jolshan@confluent.io.invalid> wrote: > >>> > >>> Congratulations Jeff! > >>> > >>> On Mon, Sep 9, 2024 at 8:33 AM Satish Duggana < > satish.duggana@gmail.com > > >>> wrote: > >>> > >>>> Congratulations Jeff! > >>>> > >>>> On Mon, 9 Sept 2024 at 18:37, Bruno Cadonna < cadonna@apache.org > > wrote: > >>>>> > >>>>> Congrats! Well deserved! > >>>>...

Congratulations Jeff! > On Sep 9, 2024, at 6:25 PM, Matthias J. Sax < mjsax@apache.org > wrote: > > Congrats! > > On 9/9/24 12:34 PM, José Armando García Sancio wrote: >> Congratulations Jeff! >> On Mon, Sep 9, 2024 at 11:45 AM Justine Olshan >> <jolshan@confluent.io.invalid> wrote: >>> >>> Congratulations Jeff! >>> >>> On Mon, Sep 9, 2024 at 8:33 AM Satish Duggana < satish.duggana@gmail.com > >>> wrote: >>> >>>> Congratulations Jeff! >>>> >>>> On Mon, 9 Sept 2024 at 18:37, Bruno Cadonna < cadonna@apache.org > wrote: >>>>> >>>>> Congrats! Well deserved! >>>>> >>>>> Best, >>>>> Bruno >>>>> >>>>> >>>>> >>>>> On 9/9/24 2:28 PM, Bill Bejeck wrote: >>>>...