Kafka

Hmmm... that's interesting... It seems that Kafka Streams "version probing" does not play well static group membership... Sounds like a "bug" to me -- well, more like a missing integration. Not sure right now, if/how we could fix it. Can you file a ticket? For now, I don't think you can do anything about it. Sorry. :( -Matthias On 2/27/23 6:50 AM, Vinoth Rengarajan wrote: > Hi Team, > > I am trying to upgrade my Kaka Streams application from 2.7.1 to 3.3.1. > Brokers are running on Kafka 2.7.1. The plan is to upgrade the clients > first and then then brokers > > I have already enabled the static membership in our application so that we > I am not expecting a rebalance. Below are the configs *(Stream Config & > Consumer Config)*. > > As mentioned earlier, the application is running on Kafka 2.7.1. I deployed > the latest version of the app with 3.3.1 streams libraries, and conf...

Hi guys, I am loadtesting a couple clusters one with local ssd disks and another one with ceph. Both clusters have the same amount of cpu/ram and they are configured the same way. im sending the same amount of messages and producing with linger.ms =0 and acks=all besides seeing higuer latencies on ceph for the most part, compared to local disk. There is something that I don't understand. On the local disk cluster. messages per second matches exactly the number of requests. but on the ceph cluster messages do not match total produce requests per second. and the only thing I can find is that the Producer purgatory in ceph kafka cluster has more request queued up than the local disk. Also RemoteTime-ms for producers is high, which could explain why there are more requests on the purgatory. To me , I think this means that the Producer is waiting to hear from all the acks. which are set to all. But I don't understand why the local disk Kafka cluster...

Paolo, Hi, that sounds like a perfect use case for Kafka, probably in conjunction with Kafka Connect and maybe Kafka streams to. I've built some demo Kafka IoT applications over the last 6 years, you may get some ideas from them. Summary of blogs here: https://www.linkedin.com/pulse/complete-guide-apache-kafka-developers-everything-i-know-paul-brebner/ There's also a Use Cases section (4) which would be useful. [ https://media.licdn.com/dms/image/C5612AQETlzxol4RL_Q/article-cover_image-shrink_720_1280/0/1649653491404?e=2147483647&v=beta&t=9dWpd4hIVIbO95efSwM2YnyFEbW-E0XiPhBxvmC0s-I ]< https://www.linkedin.com/pulse/complete-guide-apache-kafka-developers-everything-i-know-paul-brebner/ > A Complete Guide to Apache Kafka for Developers (or, everything I know about Kafka in one place)< https://www.linkedin.com/pulse/complete-guide-apache-kafka-developers-everything-i-know-paul-brebner/ > Paul Brebner, 11 April 2022 Last Updated 15 November 2022 ...

I would like to add additional information for manifests used to deploy strimzi. These should be deployed on SOURCE cluster. > https://github.com/yuwtennis/apache-kafka-apps/blob/master/strimzi/kafka/ephemeral-single-internal-listener-only.yaml > https://github.com/yuwtennis/apache-kafka-apps/blob/master/strimzi/kafka/ephemeral-single-with-external-listener.yaml This should be deployed on TARGET cluster. > https://github.com/yuwtennis/apache-kafka-apps/blob/master/strimzi/mm2/plain-source-tls-target.yaml Thanks, Yu On Sun, Feb 26, 2023 at 10:29 PM Yu Watanabe < yu.w.tennis@gmail.com > wrote: > > Hello. > > This looks like a strimzi configuration rather than kafka . But I > would like to help you anyway. > > > podIP keeps changing in GCP, hence when MirrorMaker tries to access the pod > > using the older IP, the pod is not found .. hence the DisconnectException > > I think the short answer would be don'...

Hello. This looks like a strimzi configuration rather than kafka . But I would like to help you anyway. > podIP keeps changing in GCP, hence when MirrorMaker tries to access the pod > using the older IP, the pod is not found .. hence the DisconnectException I think the short answer would be don't access pods directory access the "kafka-bootstrap" service resource for clients. https://strimzi.io/docs/operators/latest/configuring.html#ref-list-of-kafka-cluster-resources-str For the long answer, the answer will depend on your deployment model . For this I assumed that you are having a deployment model that is attached to this reply. I have also attached sample manifests to workout MM2. Sample terraform code is below. https://github.com/yuwtennis/iac-samples/tree/main/terraform/google_cloud/multi_region_private_container_clusters Sample manifests are below. https://github.com/yuwtennis/apache-kafka-apps/blob/master/strimzi/mm2/plain-sour...