Hi Tamar, Interesting problem! Kafka Streams is actually the only Kafka API I haven’t done extensive performance testing/benchmarking on over the years. A couple of thoughts to start with. Is this behaviour only obvious at increasing load? Do you have sufficient cluster resources, Kafka clients? For Kafka Streams I recall that increasing the partitions (and therefore having sufficient clients and cluster resources) is the only real scalability knob? And for Kafka Streams, windowing size and state store may impact scalability. We tried Kafka Streams a few years internally for aggregating performance data and found resource usage too high and scalability poor, and resorted to custom Kafka clients. I’ve talked about this at C/C a few years ago, and here’s a bog from the internal team: https://www.instaclustr.com/blog/the-introduction-of-apache-kafka-infrastructure/ Others have used alternative streams processing tech including Flink, ClickHouse, RisingWave, etc. I can ask around our wider team if that’s useful? Regards, Paul Brebner From: tami kikoz <tamikikoz@gmail.com> Date: Saturday, 8 August 2026 at 2:27 am To: users@kafka.apache.org <users@kafka.apache.org> Subject: Kafka Streams Abnormal Latency & Lag Spiking Under High Load [You don't often get email from tamikikoz@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Hi everyone, I am reaching out to the community for some guidance on a performance issue we are currently facing. To provide some context, our broader data processing pipeline relies heavily on Kafka to pass data sequentially between steps. Each processing step—many of which are Python services operating alongside our Kafka Streams application—processes an image and produces the result to the next topic in the chain. It is important to note that *I do not see any saturation or bottlenecks in these other pipeline steps*. The Python services are handling the throughput without issue. The performance degradation is entirely isolated to our Kafka Streams service, which handles real-time windowing and aggregations. We use branches, 3 minutes windows and calculate different aggregation fields. Messages are produced to repartition topics, changelog topics and main sink topic. We want to optimize the processing logic but from the metrics it is easily inferred that processing is not the bottleneck and does not spike under load. Due to the nature of the system, we have a strict requirement for a maximum of 2 seconds of end-to-end latency. *The Problem:* Under high load, our Kafka Streams service experiences massive lag spikes and abnormal latency. Messages are essentially "living" inside this specific service for up to 2.5 minutes before being processed, which completely breaches our 2-second SLA and degrades the real-time functionality of the system. To troubleshoot and stabilize the transaction commit rates and state stores, we have been testing a series of tuning adjustments in our prep environment. These changes have not yet been applied to production. *The Adjustments:* Our primary focus in the prep environment has been reducing overhead. The key shifts from our original baseline include: - *Commit Interval (commit.interval.ms <https://urldefense.com/v3/__http://commit.interval.ms__;!!Nhn8V6BzJA!SF04L67sl0Mh-Lxj24M3H1vXA7sbXkJLI3fqBtDt7ztXeh-lTpJJrWDataGsWZDvOP87ntOo2RJESaJ_I_vzXbI$ >)*: Increased from 100ms to 2000ms. - *Linger MS (linger.ms <https://urldefense.com/v3/__http://linger.ms__;!!Nhn8V6BzJA!SF04L67sl0Mh-Lxj24M3H1vXA7sbXkJLI3fqBtDt7ztXeh-lTpJJrWDataGsWZDvOP87ntOo2RJESaJ_XOBb6Ek$ >)*: Increased from 5ms to 10ms. - *Probing Rebalance Interval (probing.rebalance.interval.ms <https://urldefense.com/v3/__http://probing.rebalance.interval.ms__;!!Nhn8V6BzJA!SF04L67sl0Mh-Lxj24M3H1vXA7sbXkJLI3fqBtDt7ztXeh-lTpJJrWDataGsWZDvOP87ntOo2RJESaJ_h_p-gFc$ >)*: Extended from every 10 minutes to every 1 hour. - *Plus a few additional compression and batching changes outlined below.* Here is the full breakdown of the relevant configurations currently running across both environments for comparison: Production Environment (Current Baseline) - TOPOLOGY_OPTIMIZATION_CONFIG: all - COMMIT_INTERVAL_MS_CONFIG: 1000 *(Note: Previously tested at 100ms)* - CACHE_MAX_BYTES_BUFFERING_CONFIG: 0 - MAX_REQUEST_SIZE: 8388608 - AUTO_OFFSET_RESET_CONFIG: latest - COMPRESSION_TYPE_CONFIG: snappy - EVENT_WINDOW_SECONDS: 180 - SERIAL_WINDOW_SECONDS: 180 - REPARTITION_PURGE_INTERVAL_MS_CONFIG: 600000 - LINGER_MS_CONFIG: 5 Prep Environment (Currently Testing) - TOPOLOGY_OPTIMIZATION_CONFIG: all - COMMIT_INTERVAL_MS_CONFIG: 2000 - CACHE_MAX_BYTES_BUFFERING_CONFIG: 0 - MAX_REQUEST_SIZE: 8388608 - AUTO_OFFSET_RESET_CONFIG: latest - COMPRESSION_TYPE_CONFIG: lz4 - EVENT_WINDOW_SECONDS: 180 - SERIAL_WINDOW_SECONDS: 180 - REPARTITION_PURGE_INTERVAL_MS_CONFIG: 600000 - LINGER_MS_CONFIG: 10 - REBALANCE_INTERVAL_MS: 3600000 - STATE_STORE_KB: 32 - BATCH_SIZE_KB: 32 - MAX_POLL_RECORDS: 100 - FETCH_MAX_MB: 1 - MAX_PARTITION_FETCH_KB: 256 *Conclusion & Next Steps:* By widening the commit (from 100ms to 1000ms) and linger intervals, alongside extending the probing rebalance window (from 10 min to 60 min), our goal was to drastically reduce the I/O and CPU overhead tied to constant state commits and frequent partition reassignments. I have attached snapshots of our dashboard metrics (for prep and prod environments) to show the differences in behavior between the two environments. As you can see in the dashboards, the prep configurations have shown an improvement, but the performance is still lacking and falls well short of our required throughput. Because we are still struggling to handle the load efficiently within the Streams application, I am looking for general advice. Has anyone encountered a similar 2+ minute processing lag isolated to windowing aggregations under high load? I would greatly appreciate any insights, architectural suggestions, or feedback on what else we should be tuning to get our latency back down to the 2-second mark. We are using *Kafka Streams version 3.8.1*. I'm attaching a public drive folder with prep and prod metrics screenshots. I can't attach a direct Grafana dashboards links and the source code, so I know the context is a little narrowed. If you would like to hear more details about the use case, let me know. https://urldefense.com/v3/__https://drive.google.com/drive/folders/1bVuvUww1GHmq258yP_tOlSHcUGr1FbeI?usp=sharing__;!!Nhn8V6BzJA!SF04L67sl0Mh-Lxj24M3H1vXA7sbXkJLI3fqBtDt7ztXeh-lTpJJrWDataGsWZDvOP87ntOo2RJESaJ_ItrPOEw$ Thanks in advance, [Tamar Kikozashvili] [Data-Engineer, MOD] <https://urldefense.com/v3/__https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail__;!!Nhn8V6BzJA!SF04L67sl0Mh-Lxj24M3H1vXA7sbXkJLI3fqBtDt7ztXeh-lTpJJrWDataGsWZDvOP87ntOo2RJESaJ_BxFUbCY$ > ללא וירוסים.www.avast.com <https://urldefense.com/v3/__https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail__;!!Nhn8V6BzJA!SF04L67sl0Mh-Lxj24M3H1vXA7sbXkJLI3fqBtDt7ztXeh-lTpJJrWDataGsWZDvOP87ntOo2RJESaJ_BxFUbCY$ > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Comments
Post a Comment