Kafka

The Apache Kafka community is pleased to announce the release for Apache Kafka 2.7.2 This is a bug fix release and it includes fixes and improvements from 26 JIRAs, including a fix for CVE-2021-38153. All of the changes in this release can be found in the release notes: https://downloads.apache.org/kafka/2.7.2/RELEASE_NOTES.html You can download the source and binary release (Scala 2.12 and 2.13) from: https://kafka.apache.org/downloads#2.7.2 --------------------------------------------------------------------------------------------------- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them. ** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more to...

The Apache Kafka community is pleased to announce the release for Apache Kafka 2.6.3 This is a bug fix release and it includes fixes and improvements from 11 JIRAs, including a fix for CVE-2021-38153. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/2.6.3/RELEASE_NOTES.html You can download the source and binary release (Scala 2.12 and 2.13) from: https://kafka.apache.org/downloads#2.6.3 --------------------------------------------------------------------------------------------------- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them. ** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more top...

Dave, i am not sure i get your point... it is not about lesser partitions, the issue is about the duplicate hash caused by default partitioner for 2 different string, which might be landing the 2 different keys into same partition On Sun, Nov 21, 2021 at 9:33 PM Dave Klein < daveklein@usa.net > wrote: > Another possibility, if you can pause processing, is to create a new topic > with the higher number of partitions, then consume from the beginning of > the old topic and produce to the new one. Then continue processing as > normal and all events will be in the correct partitions. > > Regards, > Dave > > > On Nov 21, 2021, at 7:38 AM, Pushkar Deole < pdeole2015@gmail.com > wrote: > > > > Thanks Luke, I am sure this problem would have been faced by many others > > before so would like to know if there are any existing custom algorithms > > that can be reused, > > > > Note that we also ...

Hi Rajat, > So to route old keys , I will have to route old keys first before I start accepting the new data. Right? The old keys will automatically be routed to different partitions based on the partitioner used (in partitioner.class config). > A separate partitioner code has to be executed post increase in partitions which will read all records in kafka topic and run partition algorithm and then push it to new partition number simply. , then we can start accepting new messages. Is my understanding correct? Basically that is correct, except that the partitioner won't read records in kafka topics, instead, it reads records in producer. The partitioner exists in producer. When sending records to brokers, the partitioner in producer will decide which topic partition to send to, and find the partition leader broker for that partition. So, take an example. Before increasing the partition count, keys are "key-0", key-1", ... "key-99" S...

Hi, When I use AdminClient to delete the topic, and immediately create the topic that was just deleted, it always indicates that the topic already exists, but I try to get all the existing topics, the topic does not exist. Following is the code I tested. package org.wzh.three2.kafka.producer; import org.apache.kafka.clients.admin.*; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.ProducerConfig; import org.apache.kafka.clients.producer.ProducerRecord; import org.apache.kafka.clients.producer.RecordMetadata; import org.apache.kafka.common.serialization.StringSerializer; import java.util.Collections; import java.util.Properties; import java.util.concurrent.ExecutionException; import java.util.concurrent.Future; import java.util.concurrent.TimeUnit; import java.util.concurrent.locks.LockSupport; public class ProducerClient { public static final String TOPIC = "topic-three2-kafka"; public stati...

Hi Mark, Kafka 3.1.0, which already reached feature freeze and will be released soon, is the version that introduces support for Java 17. I hope this helps. Best, ——— Josep Prat Aiven Deutschland GmbH Immanuelkirchstraße 26, 10405 Berlin Amtsgericht Charlottenburg, HRB 209739 B Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen m: +491715557497 w: aiven.io e: josep.prat@aiven.io On Sun, Nov 21, 2021, 16:32 Mark Anderson < manderson23@gmail.com > wrote: > What is the status of support for Java 17 in Kafka for both brokers and > clients? > > The docs for Kafka 3.0.0 state that Java 8 and Java 11 are supported. > > Thanks, > Mark >

Hi Luke Thanks for responding. So to route old keys , I will have to route old keys first before I start accepting the new data. Right? A separate partitioner code has to be executed post increase in partitions which will read all records in kafka topic and run partition algorithm and then push it to new partition number simply. , then we can start accepting new messages. Is my understanding correct? On Sun, Nov 21, 2021 at 6:19 PM Luke Chen < showuon@gmail.com > wrote: > Hello Rajat, > > I'm not sure what you mean to "reshuffle messages", because once the > messages are stored in brokers, they can't be modified anymore. > But if you want to make the previous added messages route to the same > partitions after partition increasing, you can write custom partitioner: > https://kafka.apache.org/documentation/#producerconfigs_partitioner.class > > So, for example, you added 10 partitions for some new keys (ex: key-10...