Kafka

Hi David, It did look like a bug. Could you file a bug in JIRA? And if you have time, welcome to investigate and submit PR for it. (My guess is there are some internal topics are included in `TotalFetchRequestsPerSec`, but not included in another one, but not sure) Thank you. Luke On Sat, Feb 25, 2023 at 2:31 AM David Ballano Fernandez < dfernandez@demonware.net > wrote: > Hi Pere, > > Thanks for replying. > > so what i can see is: > > *kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec* > and > *kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce}* > show the same/close number of producer requests and that makes sense. > > but that is not the same with fetch requests, I pull those metrics from > Kafka jmx "oneMinuterate" without any other calculations. and this is what > i see: > > kafka.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec ...

Hi Pere, Thanks for replying. so what i can see is: kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec and kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce} show the same/close number of producer requests  and that makes sense. but that is not the same with fetch requests, I pull those metrics from Kafka jmx "oneMinuterate" without any other calculations. and this is what i see: kafka.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec         3.1Million req/s kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Fetch}   97k req/s I always trusted " kafka.network:type=RequestMetrics,name=RequestsPerSec,request"  and I thought  "TotalFetchRequestsPerSec"  would be similar with the ability like you say to see per topic. but the difference is huge and I don't know what to trust. Thanks. On Mon, Feb 20, 2023 at 2:56 AM Pere Urbón Bayes < pere.urbon@gmail.com > wrote: Hi David...

I have a system which gathers logs from several clients. The client logs are produced with a key specifying which client it is. That way processing logs is guaranteed to be ordered by time per user. Getting metrics per the topic is good and all, but getting metrics per key allows you to analyze how many logs there are per client. Of course one can claim the entity responsible for processing the logs can expose those kinds of metrics. But this only gives you the metrics on the already processed logs, instead of merely reporting how much input per client got in. I can create a different consumer group only for that purpose, but ideally I would expect such information to be exposed by the broker itself. Given the broker already calculates the message rate, doing the same per key (while limiting the memory of course) wouldn't be unreasonable.

Hi Paolo, for use cases, high-level view you can read https://www.confluent.io/blog/iot-streaming-use-cases-with-kafka-mqtt-confluent-and-waterstream/ (disclaimer, this is marketing material from a vendor, but would be a good high level view of real cases). from your other question: > I'm wondering if Kafka could be the right choice to leave database pulling and which benefits it brings. Removing completely the database, not sure it makes sense, however you could build a pipeline like: device -> mqtt broker -> kafka -> [pre-process your data when need] -> database would that help? -- Pere On Thu, Feb 23, 2023 at 7:13 AM paolo francia < paolo.francia1988@gmail.com > wrote: > Hello, > I would ask if there are some cases/examples in which Kafka has been used > in the backend of an ingestion pipeline for IoT data, with the purpose to > make it scalable. > > My case is briefly doing this: > - a web api is ...

Hi Iirop, question: > If not, what is the best way to know how many messages with a specific key got inside a topic? why do require this information? is this a business metric? or are you looking at it from an operational point of view? I'm not aware of any metric that will give you the per-key/partition distribution, especially from the broker side. -- Pere On Fri, Feb 24, 2023 at 4:46 PM lirop kaykov < lironkaykov@gmail.com > wrote: > Hi, I have a question about monitoring kafka. > When using jmx exporter and kafka exporter, you get key metrics like topic > metrics (bytes in per second, messages in per second etc). > Is there a way to get bytes/messages per second per message key? > Meaning, getting the distribution of messages by key. > If not, what is the best way to know how many messages with a specific key > got inside a topic? > -- Pere Urbon-Bayes Software Architect https://twitter.com/purbon https...

Hi, I have a question about monitoring kafka. When using jmx exporter and kafka exporter, you get key metrics like topic metrics (bytes in per second, messages in per second etc). Is there a way to get bytes/messages per second per message key? Meaning, getting the distribution of messages by key. If not, what is the best way to know how many messages with a specific key got inside a topic?

Hi Tomasz, There are some configurations needed to be tuned in producer side. Try searching kafka producer tuning, you should find many good articles describing it. Good luck. Luke On Fri, Feb 24, 2023 at 6:16 AM Tomasz Sąciński < tomasz.sacinski@linuxpolska.pl > wrote: > Hello users. > > I test max throughput to a single kafka broker but results do not match > speed limits disk or network. > > /kafka-producer-perf-test.sh --topic perf-test --num-records 3000 > --record-size 1000000 --throughput -1 --producer-props > bootstrap.servers=localhost:9092 > > 3000 records sent, 98.872850 records/sec (94.29 MB/sec), 13.51 ms avg > latency, 528.00 ms max latency, 9 ms 50th, 15 ms 95th, 162 ms 99th, 350 ms > 99.9th. > > Smaller record size gets lower throughput. > > Is upper limit throughput for broker or something another limit throughput? > > What is bottleneck for this? > > There are detail...