Well, it's kinda expected behavior. It's a split brain problem.
In the end, you use the same `application.id / group.id` and thus the
committed offsets for the removed topics are still in
`__consumer_offsets` topics and associated with the consumer group.
If a tool inspects lags and compares the latest committed offsets to
end-offsets it looks for everything it finds in the `__consumer_offsets`
topics for the group in question -- the tool cannot know that you
changed the application and that is does not read from those topics any
longer (and thus does not commit any longer).
I am not sure from top of my head if you could do a manual cleanup for
the `application.id` and topics in question and delete the committed
offsets from the `__consumer_offsets` topic -- try to checkout `Admin`
client and/or the command line tools...
In know that it's possible to delete committed offsets for a consumer
group (if a group becomes inactive, the broker would also cleanup all
group metadata after a configurable timeout), but I am not sure if
that's for the entire consumer group (ie, all topic) or if you can do it
on a per-topic basis, too.
HTH,
-Matthias
On 8/16/23 2:11 AM, Pushkar Deole wrote:
> Hi streams Dev community @matthias, @bruno
>
> Any inputs on above issue? Is this a bug in the streams library wherein the
> input topic removed from streams processor topology, the underlying
> consumer group still reporting lag against those?
>
> On Wed, Aug 9, 2023 at 4:38 PM Pushkar Deole <pdeole2015@gmail.com> wrote:
>
>> Hi All,
>>
>> I have a streams application with 3 instances with application-id set to
>> applicationV1. The application uses processor API with reading from source
>> topics, processing the data and writing to destination topic.
>> Currently it consumes from 6 source topics however we don't need to
>> process data any more from 2 of those topics so we removed 2 topics from
>> the source topics list. We have configured Datadog dashboard to report and
>> alert on consumer lag so after removing the 2 source topics and deploying
>> application, we started getting several alerts about consumer lag on
>> applicationV1 consumer group which is underlying consumer group of the
>> streams application. When we looked at the consumer group from kafka-cli,
>> we could see that the consumer group is reporting lag against the topics
>> removed from source topic list which is reflecting as increasing lag on
>> Datadog monitoring.
>>
>> Can someone advise if this is expected behavior? In my opinion, this is
>> not expected since streams application no more has those topics as part of
>> source, it should not report lag on those.
>>
>
In the end, you use the same `application.id / group.id` and thus the
committed offsets for the removed topics are still in
`__consumer_offsets` topics and associated with the consumer group.
If a tool inspects lags and compares the latest committed offsets to
end-offsets it looks for everything it finds in the `__consumer_offsets`
topics for the group in question -- the tool cannot know that you
changed the application and that is does not read from those topics any
longer (and thus does not commit any longer).
I am not sure from top of my head if you could do a manual cleanup for
the `application.id` and topics in question and delete the committed
offsets from the `__consumer_offsets` topic -- try to checkout `Admin`
client and/or the command line tools...
In know that it's possible to delete committed offsets for a consumer
group (if a group becomes inactive, the broker would also cleanup all
group metadata after a configurable timeout), but I am not sure if
that's for the entire consumer group (ie, all topic) or if you can do it
on a per-topic basis, too.
HTH,
-Matthias
On 8/16/23 2:11 AM, Pushkar Deole wrote:
> Hi streams Dev community @matthias, @bruno
>
> Any inputs on above issue? Is this a bug in the streams library wherein the
> input topic removed from streams processor topology, the underlying
> consumer group still reporting lag against those?
>
> On Wed, Aug 9, 2023 at 4:38 PM Pushkar Deole <pdeole2015@gmail.com> wrote:
>
>> Hi All,
>>
>> I have a streams application with 3 instances with application-id set to
>> applicationV1. The application uses processor API with reading from source
>> topics, processing the data and writing to destination topic.
>> Currently it consumes from 6 source topics however we don't need to
>> process data any more from 2 of those topics so we removed 2 topics from
>> the source topics list. We have configured Datadog dashboard to report and
>> alert on consumer lag so after removing the 2 source topics and deploying
>> application, we started getting several alerts about consumer lag on
>> applicationV1 consumer group which is underlying consumer group of the
>> streams application. When we looked at the consumer group from kafka-cli,
>> we could see that the consumer group is reporting lag against the topics
>> removed from source topic list which is reflecting as increasing lag on
>> Datadog monitoring.
>>
>> Can someone advise if this is expected behavior? In my opinion, this is
>> not expected since streams application no more has those topics as part of
>> source, it should not report lag on those.
>>
>
Comments
Post a Comment