Kafka

Which kafka version are you running? Which replication factor is being used? are producers using acks=all? On Wed, Aug 9, 2023 at 9:14 AM Tiansu Yu <tiansu.yu@klarna.com.invalid> wrote: > 1. We use cruise-control to actively balance the partitions across all > brokers. So point 1 could be ruled out. > 2. I am not sure how much this would impact the broker, as we do have some > exceptionally large partitions around. I have to check to know if they live > on the aforementioned broker. So far I don't see there is strong > correlation between total producer / consumer byte rates with CPU spikes on > this broker. > > Tiansu Yu > Engineer > Data Ingestion & Streaming > > Klarna Bank AB German Branch > Chausseestraße 117 > 10115 Berlin > Tel: +49 221 669 501 00 > klarna.de > > Klarna Bank AB, German Branch > Sitz: Berlin, Amtsgericht Charlottenburg HRB 217291 B > USt-Nr.: DE 815 867 3...

1. We use cruise-control to actively balance the partitions across all brokers. So point 1 could be ruled out.  2. I am not sure how much this would impact the broker, as we do have some exceptionally large partitions around. I have to check to know if they live on the aforementioned broker. So far I don't see there is strong correlation between total producer / consumer byte rates with CPU spikes on this broker.  Tiansu Yu Engineer Data Ingestion & Streaming Klarna Bank AB German Branch Chausseestraße 117 10115 Berlin Tel: +49 221 669 501 00 klarna.de Klarna Bank AB, German Branch Sitz: Berlin, Amtsgericht Charlottenburg HRB 217291 B USt-Nr.: DE 815 867 324   Zweigstelle der Klarna Bank AB (publ), AG schwedischen Rechts mit Hauptsitz in Stockholm, Schw. Gesellschaftsregister 556737-0431 Verwaltungsratsvorsitzender: Michael Moritz Geschäftsführender Direktor: Sebastian Siemiatkowski Leiter Zweigniederlassung: Yaron Shaer, Björn Petersen On 9. Aug 2023,...

Hi All, I have a streams application with 3 instances with application-id set to applicationV1. The application uses processor API with reading from source topics, processing the data and writing to destination topic. Currently it consumes from 6 source topics however we don't need to process data any more from 2 of those topics so we removed 2 topics from the source topics list. We have configured Datadog dashboard to report and alert on consumer lag so after removing the 2 source topics and deploying application, we started getting several alerts about consumer lag on applicationV1 consumer group which is underlying consumer group of the streams application. When we looked at the consumer group from kafka-cli, we could see that the consumer group is reporting lag against the topics removed from source topic list which is reflecting as increasing lag on Datadog monitoring. Can someone advise if this is expected behavior? In my opinion, this is not expected since...

Hi I can guess two problems here. 1. Either too many partition's concentrated on this broker compared to other broker 2. The partitions on this broker might have larger size as compared to the parition on other brokers please chech if all brokers are evenly balanced in terms of number of partitions and the total topic size on each broker. On Wed, 9 Aug 2023 at 1:29 PM, Tiansu Yu <tiansu.yu@klarna.com.invalid> wrote: > Hi Kafka community, > > We have an issue with our Kafka cluster from time to time, that a single > (one and only one) broker (leader) in the cluster reaches 100% CPU > utilisation. We could not see any apparent issue from the metrics. There is > no heap memory usage increase, no excessive connections made on the broker, > no misbehaving producers and consumers trying to dump or load excessively > during these periods. The only difference we could see is that thread usage > decreases during these period. Despite t...

Hi Kafka community,  We have an issue with our Kafka cluster from time to time, that a single (one and only one) broker (leader) in the cluster reaches 100% CPU utilisation. We could not see any apparent issue from the metrics. There is no heap memory usage increase, no excessive connections made on the broker, no misbehaving producers and consumers trying to dump or load excessively during these periods. The only difference we could see is that thread usage decreases during these period. Despite the problem, the service is still available (understandable from Kafka's perspective.)   We are trying to understand what else might be the cause of the issue and how we can mitigate them.  Tiansu Yu Engineer Data Ingestion & Streaming Klarna Bank AB German Branch Chausseestraße 117 10115 Berlin Tel: +49 221 669 501 00 klarna.de Klarna Bank AB, German Branch Sitz: Berlin, Amtsgericht Charlottenburg HRB 217291 B USt-Nr.: DE 815 867 324   Zweigstelle der Klar...

Hi Sahil, we value that you raise security issues. However, since this is a sensitive area there are guidelines to follow as François pointed out. Additionally to what François said, I would like to remind you again (as I did in https://kafka.apache.org/project-security ) to follow the Kafka-specific instructions published at https://kafka.apache.org/project-security if you find a security issue. Thanks for your interest in Apache Kafka. Best, Bruno On 8/7/23 10:04 AM, fpapon wrote: > Hi, > > Security threads should not be discussed on user/dev mailing list, > please use dedicated security mailing list: > > https://www.apache.org/security/ > > Regards, > > François > > On 07/08/2023 06:11, Sahil Sharma D wrote: >> Hi team, >> >> We have found below vulnerabilities in Kafka Version 3.3.1. >> >>    *   CVE-2023-34462(on 3PP Netty) ...

Hi, Security threads should not be discussed on user/dev mailing list, please use dedicated security mailing list: https://www.apache.org/security/ Regards, François On 07/08/2023 06:11, Sahil Sharma D wrote: > Hi team, > > We have found below vulnerabilities in Kafka Version 3.3.1. > > * CVE-2023-34462(on 3PP Netty) > * CVE-2023-35116: (on Jackson databind) > > > Can you please share the mitigation plan and impact of these CVEs. > > Regards, > Sahil > -- -- François