Hello Kafka community members,
I am reaching out to ask if there is a straightforward way to retrieve the
timestamp of when a consumer group commits an offset for a topic partition.
From what I understand, it is possible to obtain this information by
scanning or continuously reading from the `*__consumer_offsets*` internal
Kafka topic and storing the data externally. However, I am wondering if
there might be a better or more recommended approach that I may have
overlooked.
*Use Cases*
There are two main use cases I am considering:
1. This is for monitoring purposes. As a Kafka cluster maintainer, if a
developer notices that their consumers stopped consuming records from a
Kafka topic unintentionally, we can quickly use the last committed time to
roughly estimate when the issue may start occurring.
2. This is for finer control over consumer group offsets
retention. Currently, we manage consumer group offset retention using the `
*offsets.retention.minutes*` configuration. For some consumer groups, such
as those consuming from retry or dead-letter-queue topics, which often have
longer record retention and maybe infrequent new records, we would like to
either never delete their committed offsets or set a longer retention
period. We are considering implementing a service to handle this, but to do
so, we would need a way to determine the last time a consumer group
committed an offset.
*Exploration*
From my initial research, it appears that this information is already
managed by `*OffsetMetadataManager*` in `*OffsetAndMetadata*` (
https://github.com/apache/kafka/blob/4.0.0/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetAndMetadata.java#L56).
However, regretfully, it seems that this value is not exposed when using
the `*admin.listConsumerGroupOffsets*` method.
If anyone has insights or could point me to any previous discussions on
this topic, I would greatly appreciate it. I am also curious if there are
specific reasons why the committed timestamp is not exposed via the admin
client API.
Thank you very much for your time,
Pattarapon Kittisrisawai
I am reaching out to ask if there is a straightforward way to retrieve the
timestamp of when a consumer group commits an offset for a topic partition.
From what I understand, it is possible to obtain this information by
scanning or continuously reading from the `*__consumer_offsets*` internal
Kafka topic and storing the data externally. However, I am wondering if
there might be a better or more recommended approach that I may have
overlooked.
*Use Cases*
There are two main use cases I am considering:
1. This is for monitoring purposes. As a Kafka cluster maintainer, if a
developer notices that their consumers stopped consuming records from a
Kafka topic unintentionally, we can quickly use the last committed time to
roughly estimate when the issue may start occurring.
2. This is for finer control over consumer group offsets
retention. Currently, we manage consumer group offset retention using the `
*offsets.retention.minutes*` configuration. For some consumer groups, such
as those consuming from retry or dead-letter-queue topics, which often have
longer record retention and maybe infrequent new records, we would like to
either never delete their committed offsets or set a longer retention
period. We are considering implementing a service to handle this, but to do
so, we would need a way to determine the last time a consumer group
committed an offset.
*Exploration*
From my initial research, it appears that this information is already
managed by `*OffsetMetadataManager*` in `*OffsetAndMetadata*` (
https://github.com/apache/kafka/blob/4.0.0/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetAndMetadata.java#L56).
However, regretfully, it seems that this value is not exposed when using
the `*admin.listConsumerGroupOffsets*` method.
If anyone has insights or could point me to any previous discussions on
this topic, I would greatly appreciate it. I am also curious if there are
specific reasons why the committed timestamp is not exposed via the admin
client API.
Thank you very much for your time,
Pattarapon Kittisrisawai
Comments
Post a Comment