Kafka

Hello, After upgrading from 2.7.2 to 3.5.1 some topics are missing a partition for one or two brokers. The kafka manager shows "Under replicated%" for the topic. Looking at the topic for some brokers (of 3) partitions are missing (in my case 1 partition). A rollback will restore the "Under replicated%" to 0 again (this is the wanted number). Is this a bug of kafka or the kafka manager? Best regards, Lud Antonie -- Met vriendelijke groet / Kind regards, *Lud Antonie* < https://www.coosto.com/ > Kennedyplein 101, 5611 ZS, Eindhoven +31(0)402492700 <0031402492700> www.coosto.com < https://twitter.com/CoostoNL > < https://www.facebook.com/coosto.solution > < https://www.instagram.com/coosto_com/ > < https://www.linkedin.com/company/coosto >

The doc states that fetch.max.bytes & max.partition.fetch.bytes are not absolute maximum. If the first record batch in the first non-empty > partition of the fetch is larger than this limit, the batch will still be > returned to ensure that the consumer can make progress. I am getting a bit confused. Let's say I have a configuration like below with sufficient messages in each partition - Partitions in a topic 20 - Single message size 2MB - Consumers 5 - max.poll.records 20 - fetch.max.bytes 50 MB - max.partition.fetch.bytes 1 MB. The broker config message.max.bytes and max.message.bytes is set to default 100MB If the consumer does a poll will it receive 20 records? If yes then there is no significance of fetch.max.bytes & max.partition.fetch.bytes with max.poll.records? - Java Kafka Client - 3.5.1 - Kafka Broker - 2.8.1

Hi All, We have enabled acl for 2 topics as per the user request. We have 100+ topics in the cluster.After that we are unable to view the LogSize and Lag in the cmak. Does acl cause this issue? Please advise. Environment details: Kafka : kafka_2.12-3.2.0 CMAK: 3.0.0.5 Number of Brokers: 2 No errors in the kafka logs. -- Thanks and Regards, Hari Mobile:9790756568

Hi - I have a few Kafka MirrorMaker deployments that I'm looking to migrate to using MirrorMaker2. KIP-382< https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 > indicates that MM2 will provide "drop-in replacement" for MirrorMaker. I'm unable to find any additional documentation regarding this sort of migration, however. Does any such documentation exist? If not, is the process of switching from MirrorMaker to Mirrormaker 2 as simple as stopping the original MirrorMaker deployment and then starting an equivalent MirrorMaker2 deployment? Thanks! Andrew Gehring

Hi Mickael, - Build from source (Java 17, Scala 2.13) - Run unit and integration tests - Run custom client apps using staging artifacts +1 (non binding) Thanks Fede On Sun, Nov 26, 2023 at 11:34 AM Jakub Scholz < jakub@scholz.cz > wrote: > > +1 non-binding. I used the staged Scala 2.13 artifacts and the staged Maven > repo for my tests. All seems to work fine. > > Thanks > Jakub > > On Fri, Nov 24, 2023 at 4:37 PM Mickael Maison < mimaison@apache.org > wrote: > > > Hello Kafka users, developers and client-developers, > > > > This is the first candidate for release of Apache Kafka 3.6.1. > > > > This is a bugfix release with several fixes, including dependency > > version bumps for CVEs. > > > > Release notes for the 3.6.1 release: > > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/RELEASE_NOTES.html > > > > *** Please download, test and vot...

Ah, yeah, IQ v2 was a pretty big feature so it hasn't yet been implemented across all parts of Kafka Streams. You'll notice that we're still actively putting out new KIPs trying to complete this feature. I don't think there's any particular reason that Global KTables can't be made to work with IQ v2, but no one's gotten around to doing it yet. I did a quick search and couldn't even find a JIRA ticket for this, so it doesn't seem to be on anyone's radar. If this is something you'd like to see implemented, go ahead and file a ticket for it on JIRA < https://issues.apache.org/jira/projects/KAFKA/issues/ >. Honestly it may have been overlooked completely. Sometimes the global table stuff can get overshadowed since it's relatively less common. So definitely go ahead and file a ticket to hopefully kick off the conversation. I'm not saying it'll get picked up right away, but if there's no ticket and no one ask...

+1 non-binding. I used the staged Scala 2.13 artifacts and the staged Maven repo for my tests. All seems to work fine. Thanks Jakub On Fri, Nov 24, 2023 at 4:37 PM Mickael Maison < mimaison@apache.org > wrote: > Hello Kafka users, developers and client-developers, > > This is the first candidate for release of Apache Kafka 3.6.1. > > This is a bugfix release with several fixes, including dependency > version bumps for CVEs. > > Release notes for the 3.6.1 release: > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/RELEASE_NOTES.html > > *** Please download, test and vote by Friday, December 1 > > Kafka's KEYS file containing PGP keys we use to sign the release: > https://kafka.apache.org/KEYS > > * Release artifacts to be voted upon (source and binary): > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/ > > * Maven artifacts to be voted upon: > https://repository.apache.org/content...

Hi. JoinGroup request is sent from the polling/user thread. In your example, the consumer instance will be removed from the group because it didn't join the group within the timeout. So the partition will be assigned to another consumer and be processed. 2023年11月26日(日) 18:09 Debraj Manna < subharaj.manna@gmail.com >: > Can someone let me know if the JoinRequest is sent by the consumer from the > polling/user thread or from the background heart-beat thread? > > If JoinRequest is being sent from the polling/user thread then in this case > if the poll user thread takes more than max.poll.interval.secs then the > consumer will remain disconnected from the broker for that long. For > example, if max.poll.interval.secs is 300 sec and if processing in the poll > thread takes 15 mins then for 15 mins the partition from which this > consumer was polling will remain idle and no message will be consumed from > that partition. Is my unde...

Can someone let me know if the JoinRequest is sent by the consumer from the polling/user thread or from the background heart-beat thread? If JoinRequest is being sent from the polling/user thread then in this case if the poll user thread takes more than max.poll.interval.secs then the consumer will remain disconnected from the broker for that long. For example, if max.poll.interval.secs is 300 sec and if processing in the poll thread takes 15 mins then for 15 mins the partition from which this consumer was polling will remain idle and no message will be consumed from that partition. Is my understanding correct? I am using Kafka client 3.5.1 with Apache Kafka broker 2.8.1 with all default settings on the consumer configs.

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.6.1. This is a bugfix release with several fixes, including dependency version bumps for CVEs. Release notes for the 3.6.1 release: https://home.apache.org/~mimaison/kafka-3.6.1-rc0/RELEASE_NOTES.html *** Please download, test and vote by Friday, December 1 Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://home.apache.org/~mimaison/kafka-3.6.1-rc0/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~mimaison/kafka-3.6.1-rc0/javadoc/ * Tag to be voted upon (off 3.6 branch) is the 3.6.1 tag: https://github.com/apache/kafka/releases/tag/3.6.1-rc0 PR for updating docs: https://github.com/apache/kafka-site/pull/568 * Documentation:...

+1 non-binding. I used the staged Scala 2.13 binaries and the staged Maven repo to run my tests and all seems to work fine. Thanks & Regards Jakub On Tue, Nov 21, 2023 at 11:09 AM Luke Chen < showuon@apache.org > wrote: > Hello Kafka users, developers and client-developers, > > This is the first candidate for release of Apache Kafka 3.5.2. > > This is a bugfix release with several fixes since the release of 3.5.1, > including dependency version bumps for CVEs. > > Release notes for the 3.5.2 release: > https://home.apache.org/~showuon/kafka-3.5.2-rc1/RELEASE_NOTES.html > > *** Please download, test and vote by Nov. 28. > > Kafka's KEYS file containing PGP keys we use to sign the release: > https://kafka.apache.org/KEYS > > * Release artifacts to be voted upon (source and binary): > https://home.apache.org/~showuon/kafka-3.5.2-rc1/ > > * Maven artifacts to be voted upon: > https://...

Hi, If you stop the service prior to stopping the server, it's considered as a graceful shutdown and leader migration happens in advance. This behaviour is controlled by controlled.shutdown.enable setting, which is true by default. You can read more about it here: https://kafka.apache.org/documentation/#basic_ops_restarting But when you just stop the server, without stopping the service first, leader failure should first be spotted (depending if you are using Zookeeper or KRaft this process is different) and election will be triggered afterwards, which generally takes more time comparing to a graceful shutdown. On Thu, Nov 23, 2023 at 12:40 PM Denis Santangelo < denis.santangelo@scorechain.com > wrote: > Hello Denis, > > I'm encountering a peculiar issue with my Kafka cluster. > > I've been running 8 brokers on version 3.4.0 for several months, and > everything seems to be functioning well. > > All my topics have at...

Hello Denis, I'm encountering a peculiar issue with my Kafka cluster. I've been running 8 brokers on version 3.4.0 for several months, and everything seems to be functioning well. All my topics have at least two replicas for each partition. However, I face a problem when I shut down a server running Kafka. In such cases, my producers are unable to use the cluster, receiving the error: "There is no leader for this topic-partition as we are in the middle of a leadership election." Interestingly, if I stop the Kafka service first and then shut down the server, the failover works seamlessly. Could anyone provide insights into what might be going wrong, or if this is an issue inherent to Kafka? Thank you, Denis Santangelo

Hi Alex Brekken, Sorry for the delayed response, I tried your fix, At first I got > Fatal error during KafkaServer startup. Prepare to shutdown" > "org.apache.kafka.common.KafkaException: Exception while determining if > ZooKeeper is secure > [java.security.auth.login.config=./../config/kafka_server_jaas.conf, > zookeeper.sasl.client=false, zookeeper.sasl.clientconfig=default:Client] > at > org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:75) > at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:441) > at kafka.server.KafkaServer.startup(KafkaServer.scala:191) > at kafka.Kafka$.main(Kafka.scala:109) > at kafka.Kafka.main(Kafka.scala) Then I set zookeeper.sasl.client=true > - 10.91.21.142 arjun-8481 - - - 23 > org.apache.zookeeper.client.ZooKeeperSaslClient respondToServer SEVERE > "23-11-2023 10:49:21:770" - "SASL authenti...

Hi Luke, - Compiled from source (Java 17 and Scala 2.13) - Ran unit and integration tests - Ran custom client apps using staging artifacts +1 (non binding) Thanks Fede On Wed, Nov 22, 2023 at 2:44 PM Josep Prat <josep.prat@aiven.io.invalid> wrote: > > Hi Luke, > > Thanks for running the release. > I did the following: > - Verified artifact's signatures and hashes > - Checked JavaDoc (with navigation to Oracle JavaDoc) > - Compiled source code > - Run unit tests and integration tests > - Run getting started with ZK and KRaft > > It gets a +1 from my side (non-binding) > > Best, > > On Tue, Nov 21, 2023 at 11:09 AM Luke Chen < showuon@apache.org > wrote: > > > Hello Kafka users, developers and client-developers, > > > > This is the first candidate for release of Apache Kafka 3.5.2. > > > > This is a bugfix release with several fixes since the release of 3....

Hi Luke, Thanks for running the release. I did the following: - Verified artifact's signatures and hashes - Checked JavaDoc (with navigation to Oracle JavaDoc) - Compiled source code - Run unit tests and integration tests - Run getting started with ZK and KRaft It gets a +1 from my side (non-binding) Best, On Tue, Nov 21, 2023 at 11:09 AM Luke Chen < showuon@apache.org > wrote: > Hello Kafka users, developers and client-developers, > > This is the first candidate for release of Apache Kafka 3.5.2. > > This is a bugfix release with several fixes since the release of 3.5.1, > including dependency version bumps for CVEs. > > Release notes for the 3.5.2 release: > https://home.apache.org/~showuon/kafka-3.5.2-rc1/RELEASE_NOTES.html > > *** Please download, test and vote by Nov. 28. > > Kafka's KEYS file containing PGP keys we use to sign the release: > https://kafka.apache.org/KEYS > > * Rele...

Hi Sophie, thanks a lot for you tip! I've implemented a StateListener - to block queries when the state does not equal RUNNING. This will work perfectly now for our use-case! In the meantime I noticed the InteractiveQuery API v2 and give it a try. Unfortunately it seems not to cope with GlobalKTable. When try to run this: return streams.query(StateQueryRequest.inStore(STORE_NAME).withQuery(KeyQuery.withKey(key))); I got: "Global stores do not yet support the KafkaStreams#query API. Use KafkaStreams#store instead." From my point of view it would be great if this will work and behave like with IN_MEMORY StoreType as it is straight forward to use. Do you see a chance to get InteractiveQueryV2 work with GlobalKTable? Kind regards, Christian -----Original Message----- From: Sophie Blee-Goldman < sophie@responsive.dev > Sent: Wednesday, November 22, 2023 1:51 AM To: Christian.Zuegner@ams-osram.com.invalid Cc: users@kafka.apache.org Subjec...

Just to make sure I understand the logs, you're saying the "new file processed" lines represent store queries, and presumably the com.osr.serKafkaStreamsService is your service that's issuing these queries? You need to wait for the app to finish restoring state before querying it. Based on this message -- "KafkaStreams has not been started, you can retry after calling start()" -- I assume you're kicking off the querying service right away and blocking queries until after KafkaStreams#start is called. But you need to wait for it to actually finish starting up, not just for start() to be called. The best way to do this is by setting a state listener via KafkaStreams#setStateListener, and then using this to listen in on the KafkaStreams.State and blocking the queries until the state has changed to RUNNING. In case you're curious about why this seems to work with in-memory stores but not with rocksdb, it seems like in the in-memory case,...

Hi, we have the following problem - a Kafka Topic ~20Megabytes is made available as GlobalKTable for queries. With using RocksDB the GKTable is ready for queries instantly even without having reading the data complete - all get() requests return null. After a few seconds the data is querieable correctly - but this is to late for our application. Once we switch to IN_MEMORY we get the expected behavior. The store is only ready after all data has been read from topic. How can we achieve the same behavior with the RocksDB setup? Snipet to build KafkaStreams Topology builder.globalTable( "topic-name", Consumed.with(Serdes.String(), Serdes.String()), Materialized.as(STORE_NAME).withStoreType(Materialized.StoreType.ROCKS_DB) ); Query the Table while (true) { try { return streams.store( StoreQueryParameters.fromNameAndType(FileCrawlerKafkaTopologyProducer.STORE_NAME, QueryableStoreTypes.keyValueStore()));...

Hi, I'm trying to change super.users config at runtime, without broker restart. The next command is used (I use the same approach for all brokers, but posting output only from one broker here not to overload): > /opt/confluent/bin/kafka-configs --bootstrap-server localhost:9093 > --command-config kafka.properties --entity-type brokers --entity-name 2 > --alter --add-config 'super.users=User:kafka;' I can see in the kafka.log it was applied successfully: > {"short_message":"Processing override for entityPath: brokers/2 with > config: HashMap(super.users -> User:kafka;)","full_message":"Processing > override for entityPath: brokers/2 with config: HashMap(super.users -> > User:kafka;)","timestamp":1.700571334002E9,"level":6,"facility":"logstash-gelf","LoggerName":"kafka.server.DynamicConfigManager","SourceSimpleClassName":...

As I know, MM2 cannot sync topics on same cluster as topics on source and target supposed to have same name or to come with prefix.sameName plz correct me, there is no topicA -> topicB replication -----Original Message----- From: Anders Engström < epirealer@gmail.com > Sent: Tuesday, November 21, 2023 12:58 To: users@kafka.apache.org Subject: Re: Messages streaming from topic to topic [You don't often get email from epirealer@gmail.com . Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] CAUTION: This email is from an external source. Please don't open any unknown links or attachments. Mirror Maker (2) might be a good solution.. It's "mostly" Kafka Connect, so it should be possible to do filtering etc. https://developers.redhat.com/articles/2023/11/13/demystifying-kafka-mirrormaker-2-use-cases-and-architecture#mirrormaker_2_internal_topics /Anders On Tue, Nov 21, 2023 at 11:55 AM Alexander Shapir...

Okay. For us both of them are working well for the use case mentioned by you. Thanks, Megh On Tue, Nov 21, 2023, 16:25 Alexander Shapiro (ashapiro) <Alexander.Shapiro@amdocs.com.invalid> wrote: > Thanks for instant reply > > Yes, I did > > I have checked till now the below: > Apache Storm > Apache Heron > Apache Samza > Apache Spark > Apache Flink > Apache NiFi > Confluent replicator > StreamSets Data collector > Envoy Filters > > > > > > -----Original Message----- > From: megh vidani < vidanimegh23@gmail.com > > Sent: Tuesday, November 21, 2023 12:46 > To: users@kafka.apache.org > Cc: dev < dev@kafka.apache.org >; kafka-clients < > kafka-clients@googlegroups.com > > Subject: Re: Messages streaming from topic to topic > > [You don't often get email from vidanimegh23@gmail.com . Learn why this is > important at https://aka.ms/LearnAb...

Mirror Maker (2) might be a good solution.. It's "mostly" Kafka Connect, so it should be possible to do filtering etc. https://developers.redhat.com/articles/2023/11/13/demystifying-kafka-mirrormaker-2-use-cases-and-architecture#mirrormaker_2_internal_topics /Anders On Tue, Nov 21, 2023 at 11:55 AM Alexander Shapiro (ashapiro) <Alexander.Shapiro@amdocs.com.invalid> wrote: > Thanks for instant reply > > Yes, I did > > I have checked till now the below: > Apache Storm > Apache Heron > Apache Samza > Apache Spark > Apache Flink > Apache NiFi > Confluent replicator > StreamSets Data collector > Envoy Filters > > > > > > -----Original Message----- > From: megh vidani < vidanimegh23@gmail.com > > Sent: Tuesday, November 21, 2023 12:46 > To: users@kafka.apache.org > Cc: dev < dev@kafka.apache.org >; kafka-clients < > kafka-clients@googlegroups.com ...