Hi team
So I ran into a complicated issue, something which I believe Kafka Streams
is not prepared for.
Basically my app is reading from two topics and joining them.
But when testing the in my staging environment I found, one topic moves
faster than the other one basically pushing stream time forward
Some partitions are even months apart. I found a question on SO detailing
something similar:
https://stackoverflow.com/questions/69126351/bulk-processing-data-through-a-join-with-kafka-streams-results-in-skipping-reco
The problem for me is that joins are no longer working. Setting a huge
grace period has somehow alleviated the problem for now but I don't that
that's the right approach and not all events join at the end anyway? Have
other users faced something similar, if so, how can it be resolved? Can we
somehow delay the processing to make them aligned some how
thanks
- Miguel
So I ran into a complicated issue, something which I believe Kafka Streams
is not prepared for.
Basically my app is reading from two topics and joining them.
But when testing the in my staging environment I found, one topic moves
faster than the other one basically pushing stream time forward
Some partitions are even months apart. I found a question on SO detailing
something similar:
https://stackoverflow.com/questions/69126351/bulk-processing-data-through-a-join-with-kafka-streams-results-in-skipping-reco
The problem for me is that joins are no longer working. Setting a huge
grace period has somehow alleviated the problem for now but I don't that
that's the right approach and not all events join at the end anyway? Have
other users faced something similar, if so, how can it be resolved? Can we
somehow delay the processing to make them aligned some how
thanks
- Miguel
Comments
Post a Comment