Kafka

Hi Guozhang,   We can confirm the behavior with the 2.7.1 release. Appreciate all the help!   Cheers, Upesh   Upesh Desai ​  |  Senior Software Developer  |  udesai@itrsgroup.com www.itrsgroup.com From: Guozhang Wang <wangguoz@gmail.com> Date: Tuesday, March 30, 2021 at 2:10 PM To: Users <users@kafka.apache.org> Cc: Bart Lilje <blilje@itrsgroup.com> Subject: Re: Kafka Streams Processor API state stores not restored via changelog topics Great, I think https://issues.apache.org/jira/browse/KAFKA-12323 is indeed the root cause then. Note that this is only an issue with punctuation triggered events, where `context.timestamp()` would return 0 (and it is fixed in the yet-to-release 2.7.1/2.8.0). You can consider applying the patch if you could on top of 2.7.0, or wait for the new release; OR, if your production code does not actually use punctuation to write records to K...

Hello again, all, I just wanted to mention that I am aware of Justin's concerns in the 2.6.2 thread: https://lists.apache.org/thread.html/r2df54c11c10d3d38443054998bc7dd92d34362641733c2fb7c579b50%40%3Cdev.kafka.apache.org%3E I plan to make sure we address these concerns before the actual 2.8.0 release, but wanted to get RC0 out asap for testing. Thank you, John On Tue, 2021-03-30 at 16:37 -0500, John Roesler wrote: > Hello Kafka users, developers and client-developers, > > This is the first candidate for release of Apache Kafka > 2.8.0. This is a major release that includes many new > features, including: > > * Early-access release of replacing Zookeeper with a self- > managed quorum > * Add Describe Cluster API > * Support mutual TLS authentication on SASL_SSL listeners > * Ergonomic improvements to Streams TopologyTestDriver > * Logger API improvement to respect the hierarchy > * Request/response trace logs a...

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 2.8.0. This is a major release that includes many new features, including: * Early-access release of replacing Zookeeper with a self- managed quorum * Add Describe Cluster API * Support mutual TLS authentication on SASL_SSL listeners * Ergonomic improvements to Streams TopologyTestDriver * Logger API improvement to respect the hierarchy * Request/response trace logs are now JSON-formatted * New API to add and remove Streams threads while running * New REST API to expose Connect task configurations * Fixed the TimeWindowDeserializer to be able to deserialize keys outside of Streams (such as in the console consumer) * Streams resilient improvement: new uncaught exception handler * Streams resilience improvement: automatically recover from transient timeout exceptions Release notes for the 2.8.0 release: https://home.apache.org/~vvcephei/kafka-2.8.0-rc...

Hey Ning, I believe "if the CG offsets do not contain a pair of <topic, partition>, simply sync the offsets from source" could be the problematic behavior here? I'm not very familiar with Mirrormaker's internals, so can't say for sure. As I described previously, this "negative lag" problem happens when a target partition is empty (log end offset = 0) and the source CG offset is > 0. This scenario can be reached if a consumer processed some records which were then deleted (eg. by retention policy), or, like Alan encountered, when MM replication is set to start at "latest". When this is the case, Mirrormaker sets the target CG offset equal to the literal, not converted, source CG offset. The correct behavior here should be to set the target CG offset to 0 when the target partition is empty, not to the source CG offset. The negative lag can cause CGs to miss messages during a migration if new messages are sent betwee...

Great, I think https://issues.apache.org/jira/browse/KAFKA-12323 is indeed the root cause then. Note that this is only an issue with punctuation triggered events, where `context.timestamp()` would return 0 (and it is fixed in the yet-to-release 2.7.1/2.8.0). You can consider applying the patch if you could on top of 2.7.0, or wait for the new release; OR, if your production code does not actually use punctuation to write records to Kafka, then this issue would not actually impact you. Guozhang On Tue, Mar 30, 2021 at 11:56 AM Upesh Desai < udesai@itrsgroup.com > wrote: > Hi Guozhang, > > > > Great to hear we might have found the issue! > > > > To answer your question, the changelog record is generated by us calling > 'store.put(key,value)' from the punctuate callback, which makes sense then > because the timestamp would be 0 like you saw in your test as well. > > > > Best, > > Upesh ...

Hi Guozhang,   Great to hear we might have found the issue!   To answer your question, the changelog record is generated by us calling 'store.put(key,value)' from the punctuate callback, which makes sense then because the timestamp would be 0 like you saw in your test as well.   Best, Upesh   Upesh Desai ​  |  Senior Software Developer  |  udesai@itrsgroup.com www.itrsgroup.com From: Guozhang Wang <wangguoz@gmail.com> Date: Tuesday, March 30, 2021 at 1:00 PM To: Users <users@kafka.apache.org> Cc: Bart Lilje <blilje@itrsgroup.com> Subject: Re: Kafka Streams Processor API state stores not restored via changelog topics Hello Upesh, These are super helpful logs, and I think I'm very close to the root cause of it. You see, the written changelog record's timestamp is set to 0 (i.e. January 1st 1970 at midnight GMT), and hence given a reasonable Kafka server start time ...