Kafka

Hey there folks! My team is working on migrating application code that handles wire formatting of messages: serialization, deserialization, encryption, and signature. We are moving to custom Ser/Des classes implementing this interface https://kafka.apache.org/24/javadoc/org/apache/kafka/common/serialization/Deserializer.html . These features used to live in a library and we figure that using the class implementations would give us more flexibility while keeping a canonical implementation of what SerDes looks like in our organization. Given that context, if there are any errors while processing the message on the consumer, how can I access the original bytes of the message? We need to access the original bytes because we would like to publish the original message that was encrypted and signed to a deadletter topic to be processed later on. Currently, the approach we considered viable, but an ugly workaround, is to set a header during the deserializing with the origina...

Hi, When using MirrorMaker for a forwarding flow like A->B, B->C what are the options for handling failure of B so that C could then mirror from A? From what I can see MirrorMaker has no concept of failover so C cannot failover to A. Can you please confirm this is correct? An option could be having multiple MirrorMaker instances - one to mirror from A and one to mirror from B - that could then be controlled by another process depending on the availability of A or B. Has anyone else had to handle these types of scenarios? Thanks, Mark

Thanks Stansilav for running the release. That must be exhausting but you executed it well. Regards, Apoorv Mittal +44 7721681581 On Wed, Feb 28, 2024 at 11:22 AM Divij Vaidya < divijvaidya13@gmail.com > wrote: > Thank you Stanislav for running the release, especially fixing the whole > mess with out of sync site docs in different branches. Really appreciate > your hard work on this one. > > Thank you all contributors! Your contributions is what makes Apache Kafka > community awesome <3 > > There are many impactful changes in this release but the one closest to my > heart is https://issues.apache.org/jira/browse/KAFKA-15046 . I am very glad > this is fixed. The P999 latency spikes were driving me crazy for a long > time now. > > -- > Divij Vaidya > > > > On Wed, Feb 28, 2024 at 10:06 AM Satish Duggana < satish.duggana@gmail.com > > wrote: > > > Thanks Stanislav for all yo...

Thank you Stanislav for running the release, especially fixing the whole mess with out of sync site docs in different branches. Really appreciate your hard work on this one. Thank you all contributors! Your contributions is what makes Apache Kafka community awesome <3 There are many impactful changes in this release but the one closest to my heart is https://issues.apache.org/jira/browse/KAFKA-15046 . I am very glad this is fixed. The P999 latency spikes were driving me crazy for a long time now. -- Divij Vaidya On Wed, Feb 28, 2024 at 10:06 AM Satish Duggana < satish.duggana@gmail.com > wrote: > Thanks Stanislav for all your hard work on running the release. Thanks > to all the contributors to this release. > > > On Wed, 28 Feb 2024 at 13:43, Bruno Cadonna < cadonna@apache.org > wrote: > > > > Thanks Stan and all contributors for the release! > > > > Best, > > Bruno > > > > On...

Thanks Stanislav for all your hard work on running the release. Thanks to all the contributors to this release. On Wed, 28 Feb 2024 at 13:43, Bruno Cadonna < cadonna@apache.org > wrote: > > Thanks Stan and all contributors for the release! > > Best, > Bruno > > On 2/27/24 7:01 PM, Stanislav Kozlovski wrote: > > The Apache Kafka community is pleased to announce the release of > > Apache Kafka 3.7.0 > > > > This is a minor release that includes new features, fixes, and > > improvements from 296 JIRAs > > > > An overview of the release and its notable changes can be found in the > > release blog post: > > https://kafka.apache.org/blog#apache_kafka_370_release_announcement > > > > All of the changes in this release can be found in the release notes: > > https://www.apache.org/dist/kafka/3.7.0/RELEASE_NOTES.html > > > > You can download the source and b...

Thanks Stan and all contributors for the release! Best, Bruno On 2/27/24 7:01 PM, Stanislav Kozlovski wrote: > The Apache Kafka community is pleased to announce the release of > Apache Kafka 3.7.0 > > This is a minor release that includes new features, fixes, and > improvements from 296 JIRAs > > An overview of the release and its notable changes can be found in the > release blog post: > https://kafka.apache.org/blog#apache_kafka_370_release_announcement > > All of the changes in this release can be found in the release notes: > https://www.apache.org/dist/kafka/3.7.0/RELEASE_NOTES.html > > You can download the source and binary release (Scala 2.12, 2.13) from: > https://kafka.apache.org/downloads#3.7.0 > > --------------------------------------------------------------------------------------------------- > > > Apache Kafka is a distributed streaming platform with four core APIs: > ...

Thanks for running this release, Stanislav! And thanks to all the contributors who helped implement all the bug fixes and new features we got to put out this time around. On Tue, Feb 27, 2024, 13:03 Stanislav Kozlovski < stanislavkozlovski@apache.org > wrote: > The Apache Kafka community is pleased to announce the release of > Apache Kafka 3.7.0 > > This is a minor release that includes new features, fixes, and > improvements from 296 JIRAs > > An overview of the release and its notable changes can be found in the > release blog post: > https://kafka.apache.org/blog#apache_kafka_370_release_announcement > > All of the changes in this release can be found in the release notes: > https://www.apache.org/dist/kafka/3.7.0/RELEASE_NOTES.html > > You can download the source and binary release (Scala 2.12, 2.13) from: > https://kafka.apache.org/downloads#3.7.0 > > > ----------------------------------------...

I want to migrate old Java code which uses org.apache.kafka:kafka-clients:1.0.1-mapr-1803 but I get missing Java methods: public Producer kafkaProducer() { final Properties producerproperties = new Properties(); producerproperties.put(ProducerConfig.STREAMS_BUFFER_TIME_CONFIG, kafkaProducerConfiguration.getBufferTimeInMilliSeconds()); producerproperties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaProducerConfiguration.getProducerBootstrapServers()); return new Producer<String, String>(producerproperties); } STREAMS_BUFFER_TIME_CONFIG (" streams.buffer.max.time.ms ") is missing. private void setHeaders(SimpleMapMessage message, ConsumerRecord<K, V> record) { message.setHeader("partition", String.valueOf(record.partition())); message.setHeader("producer", record.producer()); } producer() is missing. Do you know how I have to replace producer() into the latest kafka clien...

The Apache Kafka community is pleased to announce the release of Apache Kafka 3.7.0 This is a minor release that includes new features, fixes, and improvements from 296 JIRAs An overview of the release and its notable changes can be found in the release blog post: https://kafka.apache.org/blog#apache_kafka_370_release_announcement All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/3.7.0/RELEASE_NOTES.html You can download the source and binary release (Scala 2.12, 2.13) from: https://kafka.apache.org/downloads#3.7.0 --------------------------------------------------------------------------------------------------- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream of records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records pro...

Hi, ZK chroot allows the Zookeeper ensemble to be shared with other applications, including other Kafka clusters, without a conflict. Kafka brokers from different clusters can connect to a single ZK ensemble consists from 5 nodes (metadata), I will explain below why I have chosen 5 nodes. One can have many brokers which can be failed/replaced/removed/added at any time (data). These 5 nodes of ZK ensemble are distributed against multiple availability zones (3) makes metadata to be more highly available compared to having nodes in 2 AZs. AZ-A AZ-B AZ-C +-----------+ +------------+ +-----------+ |+---+ +---+| | +---+ | |+---+ +---+| ||ZK1| |ZK2|| | |ZK3| | ||ZK4| |ZK5|| |+---+ +---+| | +---+ | |+---+ +---+| +-----------+ +------------+ +-----------+ An enterprise may have two datacenter at their premises, for third availability zone one can have it as a ...

*(double-sending this to kafka-clients & users, as I forgot initially)* This vote passes with *10 +1 votes* (3 bindings) and no 0 or -1 votes. +1 votes PMC Members (binding): * Mickael Maison * Justine Olshan * Divij Vaidya Community (non-binding): * Proven Provenzano * Federico Valeri * Vedarth Sharma * Andrew Schofield * Paolo Patierno * Jakub Scholz * Josep Prat ---- 0 votes * No votes ---- -1 votes * No votes ---- Vote thread: https://lists.apache.org/thread/71djwz292y2lzgwzm7n6n8o7x56zbgh9 I'll continue with the release process and the release announcement will follow ASAP. On Mon, Feb 26, 2024 at 10:42 AM Stanislav Kozlovski < stanislav@confluent.io > wrote: > > This vote passes with *10 +1 votes* (3 bindings) and no 0 or -1 votes. > > +1 votes > > PMC Members (binding): > * Mickael Maison > * Justine Olshan > * Divij Vaidya > > Community (non-binding): > * Prov...