Hi Tamar,
Interesting problem! Kafka Streams is actually the only Kafka API I haven’t done extensive performance testing/benchmarking on over the years.
A couple of thoughts to start with.
Is this behaviour only obvious at increasing load?
Do you have sufficient cluster resources, Kafka clients?
For Kafka Streams I recall that increasing the partitions (and therefore having sufficient clients and cluster resources) is the only real scalability knob?
And for Kafka Streams, windowing size and state store may impact scalability.
We tried Kafka Streams a few years internally for aggregating performance data and found resource usage too high and scalability poor, and resorted to custom Kafka clients. I’ve talked about this at C/C a few years ago, and here’s a bog from the internal team: https://www.instaclustr.com/blog/the-introduction-of-apache-kafka-infrastructure/
Others have used alternative streams processing tech including Flink, ClickHouse, RisingWave, etc.
...