Kafka

Hi Kafka Team, I hope this email finds you well. I am reaching out to inquire about the functionalities and usage of the Apache Kafka product within our organization. As we explore opportunities to enhance our data processing and streaming capabilities, Apache Kafka has emerged as a promising solution. However, before fully integrating it into our systems, we seek comprehensive information regarding its features, implementation process, and potential benefits. Could you please provide detailed insights into the following aspects of Apache Kafka: 1. Product Overview: An overview of Apache Kafka, including its core functionalities, architecture, and how it facilitates data streaming and messaging. 2. Use Cases: Examples of real-world use cases where Apache Kafka has been successfully implemented, demonstrating its versatility and applicability across various industries and scenarios. 3. Implementation Guidance: Step-by-step guidance on how to set up and confi...

Hi Luke, Thanks for the answers. I understand what you are describing in terms of rationale for using just the availability of the controller port to determine the readiness of the controller, but that is not fully satisfying under an operational perspective, at least based on the lack of sufficient documentation on the matter. Based on my understanding of kraft, which I admit is not considerable, the controllers will host the cluster metadata partitions on disk and make them available for the brokers. So, presumably, one of the purposes of the controllers is to ensure that the metadata partitions are properly replicated. Hence, what happens even in a non k8s environment all controllers go down? What sort of outage does the wider cluster experience in that circumstance? A complete outage on the controllers is of course an extreme scenario, but a more likely one is that a disk of the controller goes offline and needs to be replaced. In this scenario, the controller will have to re...

Hello Frank, That's a good question. I think we all know there is no "correct" answer for this question. But I can share with you what our team did for it. Readiness: controller is listening on the controller.listener.names The rationale behind it is: 1. The last step for the controller node startup is to wait until all the SocketServer ports to be open, and the Acceptors to be started, and the controller port is one of them. 2. This controller listener is used to talk to other controllers (voters) to form the raft quorum, so if it is not open and listening, the controller is basically not working at all. 3. The controller listener is also used for brokers (observers) to get the updated raft quorum info and fetch metadata. Compared with Zookeeper cluster, which is the KRaft quorum is trying to replace with, the liveness/readiness probe that recommended in Kubernetes tutorial < https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/...

Hello, I have a question regarding the deployment of Kafka using Kraft controllers in a Kubernetes environment. Our current Kafka cluster is deployed on K8S clusters as statefulsets without operators and our brokers are configured to use a script which marks the containers as unready if under-replicated partitions exist. With this readiness check and a pod disruption budget of the minimum in sync replica - 1, we are able to perform rollout restarts of our brokers automatically without ever producing consumers and producers errors. We have started the processes of transitioning to Kraft and based on the recommended deployment strategy we are going to define dedicated nodes as controllers instead of using combined servers. However, defining nodes as controller does not seem to allow to use the same strategy for readiness check as the kafka-topics.sh does not appear to be executable on controller brokers. The question is: what is a reliable readiness check that can be used for Kr...

The main difference is the internal implementation. Semantically, both are equivalent. suppress() uses an in-memory buffer, while `emitStrategy()` does not, but modifies the upstream aggregation operator impl, and waits to send results downstream, and thus, it's RocksDB based. -Matthias On 4/12/24 10:37 AM, Ayoub wrote: > Hello, > > *[Not sure if my email went through as I was not subscribed to this mailing > list. Here is my original email]* > > I found that there are two ways to send only the final result of a windowed > groupBy, either using Suppressed > < https://kafka.apache.org/37/javadoc/org/apache/kafka/streams/kstream/KTable.html#suppress(org.apache.kafka.streams.kstream.Suppressed) >.untilWindowCloses > on the final KTable or EmitStrategy > < https://kafka.apache.org/37/javadoc/org/apache/kafka/streams/kstream/TimeWindowedKStream.html#emitStrategy(org.apache.kafka.streams.kstream.EmitStrategy) > ...

I don't think that there is any specific recommendation. However, there is an overall max-message-size config that you need to keep in mind. -Matthias On 4/16/24 9:42 AM, Gabriel Giussi wrote: > I have logic in my service to capture exceptions being thrown during > message processing and produce a new message to a different topic with > information about the error. The idea is to leave the message unmodified, > aka produce the exact same bytes to this new topic, therefore I'm planning > on adding the java exception as a header. > By looking at the documentation it is just an array of bytes and it doesn't > say anything about a max size but is there any recommendation about it? > https://kafka.apache.org/documentation/#recordheader >

Congrats Greg! On 4/15/24 10:44 AM, Hector Geraldino (BLOOMBERG/ 919 3RD A) wrote: > Congrats! Well deserved > > From: dev@kafka.apache.org At: 04/13/24 14:42:22 UTC-4:00To: dev@kafka.apache.org > Subject: [ANNOUNCE] New Kafka PMC Member: Greg Harris > > Hi all, > > Greg Harris has been a Kafka committer since July 2023. He has remained > very active and instructive in the community since becoming a committer. > It's my pleasure to announce that Greg is now a member of Kafka PMC. > > Congratulations, Greg! > > Chris, on behalf of the Apache Kafka PMC > >