Hi Yves, do you use a consumer interceptor or wrapper that constructs ConsumerRecord objects using a deprecated pre-4.0 constructor? We have seen these problems when people use these kinds of outdated overloads; see https://cwiki.apache.org/confluence/display/KAFKA/KIP-1094%3A+Add+a+new+constructor+method+with+nextOffsets+to+ConsumerRecords. If that is the case, the solution is to use the new overload. Cheers, Lucas On Fri, Apr 24, 2026 at 5:58 PM STROUCKEN Yves <yves.stroucken@soprasteria.com> wrote: > > Hi Bill, > > Thanks for the quick reply. > Here is some of the output I get with debug logging and kafka-consumer-groups.sh > > ./kafka-consumer-groups.sh \ > --bootstrap-server localhost:9092 \ > --describe \ > --group 'MyProcessingApplication' > > loging.level.org.apache.kafka.streams.processor.internals.StreamTask: DEBUG > > Starting with a clean topic with no events. > > output: > GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID > MyProcessingApplication myTopic.0 4 - 0 - MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer-6a48934d-d93f-41d5-a89b-4faecaaba720 / hostIp MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer > MyProcessingApplication myTopic.0 3 - 0 - MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer-6a48934d-d93f-41d5-a89b-4faecaaba720 / hostIp MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer > MyProcessingApplication myTopic.0 2 - 0 - MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer-6a48934d-d93f-41d5-a89b-4faecaaba720 / hostIp MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer > MyProcessingApplication myTopic.0 1 - 0 - MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer-6a48934d-d93f-41d5-a89b-4faecaaba720 / hostIp MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer > MyProcessingApplication myTopic.0 0 - 0 - MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer-6a48934d-d93f-41d5-a89b-4faecaaba720 / hostIp MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer > > after producing about 1356 events I get this: > > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_1] Prepared RUNNING task for committing > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_2] Prepared RUNNING task for committing > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_0] Prepared RUNNING task for committing > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_3] Prepared RUNNING task for committing > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_4] Prepared RUNNING task for committing > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_1] Checkpointable offsets {myTopic.0-1=118} > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_1] Finalized commit for RUNNING task with eos false enforce checkpoint false > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_2] Checkpointable offsets {myTopic.0-2=119} > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_2] Finalized commit for RUNNING task with eos false enforce checkpoint false > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_0] Checkpointable offsets {myTopic.0-0=113} > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_0] Finalized commit for RUNNING task with eos false enforce checkpoint false > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_3] Checkpointable offsets {myTopic.0-3=114} > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_3] Finalized commit for RUNNING task with eos false enforce checkpoint false > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_4] Checkpointable offsets {myTopic.0-4=126} > 2026-04-24 17:24:39.566 [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask - stream-thread [MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1] task [0_4] Finalized commit for RUNNING task with eos false enforce checkpoint false > > and the ' ./kafka-consumer-groups.sh' result again: > MyProcessingApplication myTopic.0 4 - 288 - MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer-6a48934d-d93f-41d5-a89b-4faecaaba720 /hostIp MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer > MyProcessingApplication myTopic.0 3 - 270 - MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer-6a48934d-d93f-41d5-a89b-4faecaaba720 /hostIp MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer > MyProcessingApplication myTopic.0 2 - 264 - MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer-6a48934d-d93f-41d5-a89b-4faecaaba720 /hostIp MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer > MyProcessingApplication myTopic.0 1 - 276 - MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer-6a48934d-d93f-41d5-a89b-4faecaaba720 /hostIp MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer > MyProcessingApplication myTopic.0 0 - 258 - MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer-6a48934d-d93f-41d5-a89b-4faecaaba720 /hostIp MyProcessingApplication-509fdb18-f2ec-43f9-a214-50c3d309fde5-StreamThread-1-consumer > > As you can see only 'LOG-END-OFFSET' is moving. Current-offset and LAG stay as '-'. > > On other environments the other numbers are filled in because they have run with older versions, and then the displayed lag number stays the same. > For example, below the consumer group shows large offset lag, while processing is in near realtime with only about 60 events a minute. > > MyProcessingApplication myTopic.0 5 2194852 2306908 112056 MyProcessingApplication -1fd70c5b-1f45-4a81-97b4-951c3b7ae861-StreamThread-1-consumer-a3176137-2d89-4278-98ae-c90b3e6fdeb7 /Host MyProcessingApplication -1fd70c5b-1f45-4a81-97b4-951c3b7ae861-StreamThread-1-consumer > MyProcessingApplication myTopic.0 0 2194725 2305917 111192 MyProcessingApplication -d44f97cf-62ed-499f-90aa-2c563301f2e6-StreamThread-1-consumer-6fbf8c49-00b1-407d-be73-a46524bf62ad /Host MyProcessingApplication -d44f97cf-62ed-499f-90aa-2c563301f2e6-StreamThread-1-consumer > MyProcessingApplication myTopic.0 1 2302267 2306596 4329 MyProcessingApplication -d6df029b-658e-4e1c-bc96-b1bd1bcc29be-StreamThread-1-consumer-255edc8a-92de-4c93-bf92-68915bd758e1 /Host MyProcessingApplication -d6df029b-658e-4e1c-bc96-b1bd1bcc29be-StreamThread-1-consumer > MyProcessingApplication myTopic.0 4 2303234 2306341 3107 MyProcessingApplication -0c0f2d0c-d1a7-415c-9b30-5e2f7dc80899-StreamThread-1-consumer-6ccfcdba-2182-4804-a0f4-ef4e361f2e26 /Host MyProcessingApplication -0c0f2d0c-d1a7-415c-9b30-5e2f7dc80899-StreamThread-1-consumer > MyProcessingApplication myTopic.0 3 2182863 2306026 123163 MyProcessingApplication -0a8893b3-3ca8-4e6b-9164-a271e13bd3b2-StreamThread-1-consumer-185b65b2-7ba8-40d3-8389-533d12b378bb /Host MyProcessingApplication -0a8893b3-3ca8-4e6b-9164-a271e13bd3b2-StreamThread-1-consumer > MyProcessingApplication myTopic.0 2 2301468 2306056 4588 MyProcessingApplication -dcb5ee39-0b95-42ce-a83a-44e1e0e0ad76-StreamThread-1-consumer-458d237a-28ec-47ec-9acb-786f5a2d5bc2 /Host MyProcessingApplication -dcb5ee39-0b95-42ce-a83a-44e1e0e0ad76-StreamThread-1-consumer > > There are no errors on the current log settings. > > I hope this is useful, please let me know if there is anything else I can do to help. > > Thanks, > Yves. > > > > On 24/04/2026, 16:29, "Bill Bejeck via users" <users@kafka.apache.org <mailto:users@kafka.apache.org>> wrote: > > > Sorry to hear about the issues. Can you provide log files to help diagnose > the problem? > > > Thanks, > Bill > > > On Fri, Apr 24, 2026 at 4:53 AM STROUCKEN Yves < > yves.stroucken@soprasteria.com <mailto:yves.stroucken@soprasteria.com>> wrote: > > > > Hello Kafka users, > > I am looking for help with a Kafka Streams offset commit issue that > > started after upgrading. > > We have a Spring Cloud Stream application using the Kafka Streams binder. > > Old working stack: > > > > * Spring Boot 3.5.9 > > * Spring Cloud 2025.0.0 > > * spring-cloud-stream-binder-kafka-streams 4.3.0 > > * kafka-streams 3.9.1 > > New stack with the issue: > > > > * Spring Boot 4.0.5 > > * Spring Cloud 2025.1.1 > > * spring-cloud-stream-binder-kafka-streams 5.0.1 > > * kafka-streams 4.1.2 > > Broker version: > > > > * Kafka brokers 3.9.0 > > Symptoms: > > > > * The Kafka Streams application processes new records normally. > > * However, committed offsets for the Streams application do not appear > > to advance. > > * In Kafka UI, consumer group lag stays high for all partitions, while > > application-level metrics show near real-time processing of new events. > > * On restart, the application starts reading from the beginning of the > > topic again. > > * When I set auto.offset.reset=none, startup fails with: > > org.apache.kafka.streams.errors.StreamsException: No valid committed > > offset found for input [myTopic] and no valid reset policy configured > > Important notes: > > > > * application.id is explicitly configured and static. > > * The same topology and configuration on the older stack works fine > > and retains offsets across restart. > > I have only a basic familiarity with the Kafka Streams internals, but > > while debugging StreamTask in 4.1.2, I see commit paths being reached even > > though no actual offsets seem to be committed. > > > > Has anyone seen this in 4.1.x, or does this match any known issue? > > Any pointers, similar bug reports, or things I should inspect next would > > be greatly appreciated. > > Thanks > > > > > > > > > > >
Comments
Post a Comment