Re: Stream to KTable internals

Hi Chad,
> I'm wondering if someone can point me to the Kafka streams internal code
that reads records for the join?
--> You can check StreamThread#pollPhase, where stream thread (main
consumer) periodically poll records. And then, it'll process each topology
node with these polled records in stream tasks (StreamTask#process).

Hope that helps.

Thanks.
Luke


On Sat, Oct 30, 2021 at 5:42 PM Gilles Philippart
<gilles.philippart@fundingcircle.com.invalid> wrote:

> Hi Chad, this talk around 24:00 clearly explains what you're seeing
> https://www.confluent.io/events/kafka-summit-europe-2021/failing-to-cross-the-streams-lessons-learned-the-hard-way/
> <
> https://www.confluent.io/events/kafka-summit-europe-2021/failing-to-cross-the-streams-lessons-learned-the-hard-way/
> >
>
> Gilles
>
> > On 30 Oct 2021, at 04:02, Chad Preisler <chad.preisler@gmail.com> wrote:
> >
> > Hello,
> >
> > I have a stream application that does a KStream to KTable left join. We
> > seem to be occasionally missing joins (KTable side is null).
> >
> > I'm wondering if someone can point me to the Kafka streams internal code
> > that reads records for the join? I've poked around the Kafka code base,
> but
> > there is a lot there. I imagine there is some consumer poll for each side
> > of the join, and possibly a background thread for reading the KTable
> topic.
> >
> > I figure there are several possible causes of this issue, and since
> nothing
> > is really jumping out in my code, I was going to start looking at the
> Kafka
> > code to see if there is something I can do to fix this.
> >
> > Thanks,
> > Chad
>
>
> --
>
>
>
>
> Funding Circle Limited is authorised and regulated by the Financial
> Conduct Authority under firm registration number 722513. Funding Circle is
> not covered by the Financial Services Compensation Scheme. Registered in
> England (Co. No. 06968588) with registered office at 71 Queen Victoria
> Street, London EC4V 4AY.
>