Thank you for the information. This is actually not my issue. In my
scenario the records were written to the KTable topic before the record was
written to the KStream topic. We only get a handful of these issues per
day, and they seem to happen when many transactions are being run through
the system. This stream application was previously running using a
GlobalKTable and we saw one incident in 30 days. Now with the KTable we see
them a few times a day. This is not happening during startup, these PODs
have been running for a few days. When we switched from GlobalKTable we
went from 3 running instances to 12 running instances of our application
(we have 12 partitions on each topic). We also moved from Kafka 2.8.0
client libraries to 3.0.0. When these happen we usually figure it out about
an hour after, and to fix the issue we just put the exact same message back
on the KStream topic and everything processes correctly.
Does GlobalKTable read records from the topic differently than KTable while
the stream application is running (after startup fully warmed up)?
On Sat, Oct 30, 2021 at 4:42 AM Gilles Philippart
<gilles.philippart@fundingcircle.com.invalid> wrote:
> Hi Chad, this talk around 24:00 clearly explains what you're seeing
> https://www.confluent.io/events/kafka-summit-europe-2021/failing-to-cross-the-streams-lessons-learned-the-hard-way/
> <
> https://www.confluent.io/events/kafka-summit-europe-2021/failing-to-cross-the-streams-lessons-learned-the-hard-way/
> >
>
> Gilles
>
> > On 30 Oct 2021, at 04:02, Chad Preisler <chad.preisler@gmail.com> wrote:
> >
> > Hello,
> >
> > I have a stream application that does a KStream to KTable left join. We
> > seem to be occasionally missing joins (KTable side is null).
> >
> > I'm wondering if someone can point me to the Kafka streams internal code
> > that reads records for the join? I've poked around the Kafka code base,
> but
> > there is a lot there. I imagine there is some consumer poll for each side
> > of the join, and possibly a background thread for reading the KTable
> topic.
> >
> > I figure there are several possible causes of this issue, and since
> nothing
> > is really jumping out in my code, I was going to start looking at the
> Kafka
> > code to see if there is something I can do to fix this.
> >
> > Thanks,
> > Chad
>
>
> --
>
>
>
>
> Funding Circle Limited is authorised and regulated by the Financial
> Conduct Authority under firm registration number 722513. Funding Circle is
> not covered by the Financial Services Compensation Scheme. Registered in
> England (Co. No. 06968588) with registered office at 71 Queen Victoria
> Street, London EC4V 4AY.
>
scenario the records were written to the KTable topic before the record was
written to the KStream topic. We only get a handful of these issues per
day, and they seem to happen when many transactions are being run through
the system. This stream application was previously running using a
GlobalKTable and we saw one incident in 30 days. Now with the KTable we see
them a few times a day. This is not happening during startup, these PODs
have been running for a few days. When we switched from GlobalKTable we
went from 3 running instances to 12 running instances of our application
(we have 12 partitions on each topic). We also moved from Kafka 2.8.0
client libraries to 3.0.0. When these happen we usually figure it out about
an hour after, and to fix the issue we just put the exact same message back
on the KStream topic and everything processes correctly.
Does GlobalKTable read records from the topic differently than KTable while
the stream application is running (after startup fully warmed up)?
On Sat, Oct 30, 2021 at 4:42 AM Gilles Philippart
<gilles.philippart@fundingcircle.com.invalid> wrote:
> Hi Chad, this talk around 24:00 clearly explains what you're seeing
> https://www.confluent.io/events/kafka-summit-europe-2021/failing-to-cross-the-streams-lessons-learned-the-hard-way/
> <
> https://www.confluent.io/events/kafka-summit-europe-2021/failing-to-cross-the-streams-lessons-learned-the-hard-way/
> >
>
> Gilles
>
> > On 30 Oct 2021, at 04:02, Chad Preisler <chad.preisler@gmail.com> wrote:
> >
> > Hello,
> >
> > I have a stream application that does a KStream to KTable left join. We
> > seem to be occasionally missing joins (KTable side is null).
> >
> > I'm wondering if someone can point me to the Kafka streams internal code
> > that reads records for the join? I've poked around the Kafka code base,
> but
> > there is a lot there. I imagine there is some consumer poll for each side
> > of the join, and possibly a background thread for reading the KTable
> topic.
> >
> > I figure there are several possible causes of this issue, and since
> nothing
> > is really jumping out in my code, I was going to start looking at the
> Kafka
> > code to see if there is something I can do to fix this.
> >
> > Thanks,
> > Chad
>
>
> --
>
>
>
>
> Funding Circle Limited is authorised and regulated by the Financial
> Conduct Authority under firm registration number 722513. Funding Circle is
> not covered by the Financial Services Compensation Scheme. Registered in
> England (Co. No. 06968588) with registered office at 71 Queen Victoria
> Street, London EC4V 4AY.
>
Comments
Post a Comment