Re: Kafka streams - message not materialized in intermediate topics

Hi Tomer,

From the description you've provided, it sounds to me like you have a
stateful operation.

The thing to keep in mind with stateful operations in Kafka Streams is that
every result is not written to the changelog and forwarded downstream.
Kafka Streams uses a cache for stateful operations and it's only on cache
flush either when it's full or when Kafka Streams commits (every 30 seconds
by default) that Kafka Streams writes the results of the stateful
operations to the changelog and forwards the records downstream to other
processors.

If you want every Kafka Streams to forward every record you'll need to set
the `StreamConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG` to 0.

If I haven't understood your experience accurately can you provide a few
more details?

Thanks,
Bill


On Wed, Oct 27, 2021 at 9:48 AM Tomer Cohen <ilan012@gmail.com> wrote:

> Hello Kafka team,
>
> I am seeing an odd behavior when using kafka streams.
>
> During periods of heavier volumes, there are messages coming in. However,
> they do look like they are processed by the adder/subtractor and are not
> materialized in the intermediate topics which causes them not to be
> outputted in the final topic.
>
> Is there any way to debug this or log out when a message is dropped in the
> stream and not processed for whatever reason?
>
> Thanks,
>
> Tomer
>