Kafka

>> Thanks for the information. I ran the code using Kafka locally. After >> submitting some records inside and outside of the time window and grace, >> the join performed as expected when running locally. That gives some hope :) > However, they never get into the join. How do you know this? Did you check the metric for dropper records? Maybe records are considers malformed and dropped? Are you using the same records in production and in your local test? >> Are there any settings for the stream client that would affect the join? Not that I can think of... There is one more internal config, but as long as data is flowing, it should not impact the result you see. >> Are there any settings on the broker side that would affect the join? No. The join is computed client side. Broker configs should not have any impact. > f I increase the log level for the streams API would that >> shed some light on what ...

Matthias, Thanks for the information. I ran the code using Kafka locally. After submitting some records inside and outside of the time window and grace, the join performed as expected when running locally. I'm not sure why the join is not working as expected when running against our actual brokers. We are peeking at the records for the streams and we are seeing the records get pulled. However, they never get into the join. It's been over 24 hours since the expected records were created, and there has been plenty of traffic to advance the stream time. Only records that have both a left and right side match are getting processed by the join. Are there any settings for the stream client that would affect the join? Are there any settings on the broker side that would affect the join? The outer join is just one part of the topology. Compared to running it locally there is a lot more data going through the app when running on our actual servers. If I increase ...

> I expect the join to >> execute after the 25 with one side of the join containing a record and the >> other being null Given that you also have a grace period of 5 minutes, the result will only be emitted after the grace-period passed and the window is closed (not when window end time is reached). > One has a >> naming convention of "KSTREAM_OUTERSHARED". I see a record there, but I'm >> not sure how to decode that message to see what is in it. What is the >> purpose of those messages? It's an internal store, that stores all records which are subject to be emitted as left/right join result, ie, if there is no inner join result. The format used is internal: https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/internals/LeftOrRightValueSerde.java Also note: time is based on event-time, ie, if the input stream stops to send new records, "stream-time" w...

Hello, I have a KStream to KStream outer join with a time difference of 25 minutes and 5 minutes of grace. When I get a record for one side of the join, but don't get a record on the other side of the join, I expect the join to execute after the 25 with one side of the join containing a record and the other being null. Is that correct? If it is correct, it's not working for me. I was poking around on the broker and saw some internal topics. I see the key I expected to execute the join on some of those topics. One has a naming convention of "KSTREAM_OUTERSHARED". I see a record there, but I'm not sure how to decode that message to see what is in it. What is the purpose of those messages? If I decode the message will it help me see when the join should have been executed? I also see the key on a topic with the naming convention "KSTREAM_OUTERTHIS". Are there any other topics that I should be looking at to troubleshoot this issue? ...

Congrats, Igor! On Thu, Apr 25, 2024 at 6:10 AM Matthias J. Sax < mjsax@apache.org > wrote: > Congrats! > > On 4/24/24 2:29 PM, Bill Bejeck wrote: > > Congrats Igor! > > > > -Bill > > > > On Wed, Apr 24, 2024 at 2:37 PM Tom Bentley < tbentley@redhat.com > wrote: > > > >> Congratulations Igor! > >> > >> On Thu, 25 Apr 2024 at 6:27 AM, Chia-Ping Tsai < chia7712@gmail.com > > wrote: > >> > >>> Congratulations, Igor! you are one of the best Kafka developers!!! > >>> > >>> Mickael Maison < mickael.maison@gmail.com > 於 2024年4月25日 週四 上午2:16寫道： > >>> > >>>> Congratulations Igor! > >>>> > >>>> On Wed, Apr 24, 2024 at 8:06 PM Colin McCabe < cmccabe@apache.org > > >> wrote: > >>>>> > >>>>> Hi all, > >>>>...

Congrats! On 4/24/24 2:29 PM, Bill Bejeck wrote: > Congrats Igor! > > -Bill > > On Wed, Apr 24, 2024 at 2:37 PM Tom Bentley < tbentley@redhat.com > wrote: > >> Congratulations Igor! >> >> On Thu, 25 Apr 2024 at 6:27 AM, Chia-Ping Tsai < chia7712@gmail.com > wrote: >> >>> Congratulations, Igor! you are one of the best Kafka developers!!! >>> >>> Mickael Maison < mickael.maison@gmail.com > 於 2024年4月25日 週四 上午2:16寫道： >>> >>>> Congratulations Igor! >>>> >>>> On Wed, Apr 24, 2024 at 8:06 PM Colin McCabe < cmccabe@apache.org > >> wrote: >>>>> >>>>> Hi all, >>>>> >>>>> The PMC of Apache Kafka is pleased to announce a new Kafka committer, >>>> Igor Soarez. >>>>> >>>>> Igor has been a Kafka contributor since 2019. In additi...

Dear Team Greetings for the day I have KAFKA running in a namespace on a k8s cluster exposed with NodePort. I took the backup of the whole namespace and restored it into a different namespace using velero. Now I have another kafka deployed on the 2nd namespace which is also exposed at some NodePorts. The issue which I am facing is that my KAFKA deployed on the first namespace is internally connected to the KAFKA deployed in the new namespace. It's using the service endpoints of the new KAFKA services. DEBUGGING: Listed the BROKERS and configuration of the kafka in the first namespace and not able to get any endpoints which are related to the KAFKA deployed in Second namespace. Please help me to resolve the issue. Thanks in advance for your help. *---------Thanks and Regards:* *Pulkit Saraf* Software Engineer (Blockchain) *Contact: *+919672432959 / +918056120595

I'll join Dima with the thanks, Luke. This seems to be indeed a good way of enforcing safe restarts. Thanks, Frank -- Francesco Burato | Software Development Engineer | Adobe | burato@adobe.com <mailto: burato@adobe.com > | c. +44 747 9029370<tel:+447479029370> From: Dima Brodsky < ddbrodsky@gmail.com > Date: Monday, 22 April 2024 at 05:16 To: users@kafka.apache.org < users@kafka.apache.org > Subject: Re: Kraft controller readiness checks EXTERNAL: Use caution when clicking on links or opening attachments. Thanks Luke, this helps for our use case. It does not cover the buildout of a new cluster where there are no brokers, but that should be remedied by kip 919 which looks to be resolved in 3.7.0. ttyl Dima On Sun, Apr 21, 2024 at 9:06 PM Luke Chen < showuon@gmail.com > wrote: > Hi Frank, > > About your question: > > Unless this is already available but not well publicised in the > documenta...

Thanks Luke, this helps for our use case. It does not cover the buildout of a new cluster where there are no brokers, but that should be remedied by kip 919 which looks to be resolved in 3.7.0. ttyl Dima On Sun, Apr 21, 2024 at 9:06 PM Luke Chen < showuon@gmail.com > wrote: > Hi Frank, > > About your question: > > Unless this is already available but not well publicised in the > documentation, ideally there should be protocol working on the controller > ports that answers to operational questions like "are metadata partitions > in sync?", "has the current controller converged with other members of the > quorum?". > > I'm sorry that KRaft controller is using raft protocol, so there is no such > "in-sync replica" definition like data replication protocol. What we did > for our check is described here > < > https://github.com/strimzi/proposals/blob/main/060-kafka-roller-kraf...

Hi Frank, About your question: > Unless this is already available but not well publicised in the documentation, ideally there should be protocol working on the controller ports that answers to operational questions like "are metadata partitions in sync?", "has the current controller converged with other members of the quorum?". I'm sorry that KRaft controller is using raft protocol, so there is no such "in-sync replica" definition like data replication protocol. What we did for our check is described here < https://github.com/strimzi/proposals/blob/main/060-kafka-roller-kraft.md#the-new-quorum-check >. In short, we use ` controller.quorum.fetch.timeout.ms ` and `replicaLastCaughtUpTimestamp` to determine if it's safe to roll this controller pod. Hope this helps. Thank you. Luke On Fri, Apr 19, 2024 at 5:06 PM Francesco Burato <burato@adobe.com.invalid> wrote: > Hi Luke, > > Thanks for t...

Not sure either, but it sounds like a bug to me. Can you reproduce this reliably? What version are you using? It would be best if you could file a Jira ticket and we can take it from there. -Matthias On 4/21/24 5:38 PM, Penumarthi Durga Prasad Chowdary wrote: > Hi , > I have an issue in kafka-streams while constructing kafka-streams state > store windows(TimeWindow and SessionWindow). While kafka-streams > processing data sometimes intermittent kafka-streams process throwing below > error > ThreadName: > kafka-streams-exec-0-test-store-6d676cf0-3910-4c25-bfad-ea2b98953db3-StreamThread-9 > TraceID: unknown CorelationID: eff36722-1430-4ffb-bf2e-c6e6cf6ae164 > Message: stream-client [ kafka-streams-exec-0-test-store > -6d676cf0-3910-4c25-bfad-ea2b98953db3] Replacing thread in the streams > uncaught exception handler > org.apache.kafka.streams.errors.StreamsException: failed to initialize > processor KSTREAM-AGGREGATE-0...

Hi , I have an issue in kafka-streams while constructing kafka-streams state store windows(TimeWindow and SessionWindow). While kafka-streams processing data sometimes intermittent kafka-streams process throwing below error ThreadName: kafka-streams-exec-0-test-store-6d676cf0-3910-4c25-bfad-ea2b98953db3-StreamThread-9 TraceID: unknown CorelationID: eff36722-1430-4ffb-bf2e-c6e6cf6ae164 Message: stream-client [ kafka-streams-exec-0-test-store -6d676cf0-3910-4c25-bfad-ea2b98953db3] Replacing thread in the streams uncaught exception handler org.apache.kafka.streams.errors.StreamsException: failed to initialize processor KSTREAM-AGGREGATE-0000000001 at org.apache.kafka.streams.processor.internals.ProcessorNode.init(ProcessorNode.java:115) at org.apache.kafka.streams.processor.internals.StreamTask.initializeTopology(StreamTask.java:986) at org.apache.kafka.streams.processor.internals.StreamTask.completeRestoration(StreamTask.java:271) at org.apache.kafka.stream...

Hi, I have *flink 1.19 *installation and start *sql-client locally*, like this: *./bin/sql-client.sh* then execute *SET 'table.sql-dialect' = 'hive'*; and have this exception: *[ERROR] Could not execute SQL statement. Reason:java.lang.ClassNotFoundException: org.antlr.runtime.tree.Tree* I'm pretty sure that* i do have antlr in classpath*, because i found it there using `ps` command (see text in *bold*): ps -Af | grep java | grep client | grep antlr 501 1131 993 0 11:30PM ttys000 0:03.27 java -XX:+IgnoreUnrecognizedVMOptions --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-...