-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org
iQIzBAEBCgAdFiEE8osu2CcCCF5douGQu8PBaGu5w1EFAlzF16AACgkQu8PBaGu5
w1FRLhAApU42TawOMGsMmU2CexQ2UoXuI5Rbd5BTatWpUBfTgvK6UTGVesgcw5s/
5Zrv08tDnbSzRQRzn4AAnOcfZqFxtKtAEK7HIcDdyZqhP61rtNPh90Wz9xhEGUS1
+07OW2HiqgBKt46d+i866RL/54EIQmfQPpvBitI7NuAOKXYij3Q9t+QhQNgTHehj
3tBdGGcal3TT+xgTxd1vxKSQ78Kau9PRleEp6hA2v318LXPFJF5s7RmMcDjRvnEi
nJlwADcQx3Jt13ebFMcCDNOyTYqBvnVP765zpnH4bQw4XLwfRwR1Y1XIw3BcLYgD
68pLJrswnHSEKjI4bcaz+HYGBZ1GS22jx4z1b2ROu5cghqUZ5bdWTKf7DE+Qedpb
IsVoqrD7v6UC/W3gjgJSWTGM/BLzvgasvDDLy+OWu5VaGZekzO1RQeeHqxZGhnxI
ZE/Z6oYchHkNKlB1ZBS/KBnytlBERmVMxoS03weIlc2240u2jp4anyuHPqX8/8N2
GgXRo8cf35YlQ0efsJXyNkbEs+YtinwksRO8WhFoOK8idoIsPw7deRoH9gR9uprI
irfzzKZi5tLTEA6XJjpBf9gQ+WQBsw/W+9x5zCoCoeTvKNW9OKde+mbQWfw2m1sB
8DA+RAl+VuUzpun3RVJtq2xnRHoyFKPr2LOr7vJB0DYBTWOqo34=
=+7pL
-----END PGP SIGNATURE-----
This won't work if your topic is compacted though. Also, if you are
using transactions, it might not be accurate, depending on how many
transaction markers are in the topics.
-Matthias
On 4/28/19 2:59 PM, Peter Bukowinski wrote:
> You'll need to do this programmatically with some simple math. There's a binary included with kafka called kafka-run-class that you can use to expose earliest and latest offset information.
>
> This will return the earliest unexpired offsets for each partition in a topic:
>
> kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic TOPIC --time -2
>
> This will return the latest offset:
>
> kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic TOPIC --time -1
>
> With that info, subtract the latest from the earliest per partition, sum the results, and you'll have the number of messages available in your topic.
>
> -- Peter
>
>> On Apr 28, 2019, at 1:41 AM, jaaz jozz <jazzlofi2@gmail.com> wrote:
>>
>> Hello,
>> I want to count how many messages available in each topic in my kafka
>> cluster.
>> I understand that just looking at the latest offset available is not
>> correct, because older messages may have been already purged due to
>> retention policy.
>> So what is the correct way of counting that?
>>
>> Thanks,
>> Jazz.
Comment: GPGTools - https://gpgtools.org
iQIzBAEBCgAdFiEE8osu2CcCCF5douGQu8PBaGu5w1EFAlzF16AACgkQu8PBaGu5
w1FRLhAApU42TawOMGsMmU2CexQ2UoXuI5Rbd5BTatWpUBfTgvK6UTGVesgcw5s/
5Zrv08tDnbSzRQRzn4AAnOcfZqFxtKtAEK7HIcDdyZqhP61rtNPh90Wz9xhEGUS1
+07OW2HiqgBKt46d+i866RL/54EIQmfQPpvBitI7NuAOKXYij3Q9t+QhQNgTHehj
3tBdGGcal3TT+xgTxd1vxKSQ78Kau9PRleEp6hA2v318LXPFJF5s7RmMcDjRvnEi
nJlwADcQx3Jt13ebFMcCDNOyTYqBvnVP765zpnH4bQw4XLwfRwR1Y1XIw3BcLYgD
68pLJrswnHSEKjI4bcaz+HYGBZ1GS22jx4z1b2ROu5cghqUZ5bdWTKf7DE+Qedpb
IsVoqrD7v6UC/W3gjgJSWTGM/BLzvgasvDDLy+OWu5VaGZekzO1RQeeHqxZGhnxI
ZE/Z6oYchHkNKlB1ZBS/KBnytlBERmVMxoS03weIlc2240u2jp4anyuHPqX8/8N2
GgXRo8cf35YlQ0efsJXyNkbEs+YtinwksRO8WhFoOK8idoIsPw7deRoH9gR9uprI
irfzzKZi5tLTEA6XJjpBf9gQ+WQBsw/W+9x5zCoCoeTvKNW9OKde+mbQWfw2m1sB
8DA+RAl+VuUzpun3RVJtq2xnRHoyFKPr2LOr7vJB0DYBTWOqo34=
=+7pL
-----END PGP SIGNATURE-----
This won't work if your topic is compacted though. Also, if you are
using transactions, it might not be accurate, depending on how many
transaction markers are in the topics.
-Matthias
On 4/28/19 2:59 PM, Peter Bukowinski wrote:
> You'll need to do this programmatically with some simple math. There's a binary included with kafka called kafka-run-class that you can use to expose earliest and latest offset information.
>
> This will return the earliest unexpired offsets for each partition in a topic:
>
> kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic TOPIC --time -2
>
> This will return the latest offset:
>
> kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic TOPIC --time -1
>
> With that info, subtract the latest from the earliest per partition, sum the results, and you'll have the number of messages available in your topic.
>
> -- Peter
>
>> On Apr 28, 2019, at 1:41 AM, jaaz jozz <jazzlofi2@gmail.com> wrote:
>>
>> Hello,
>> I want to count how many messages available in each topic in my kafka
>> cluster.
>> I understand that just looking at the latest offset available is not
>> correct, because older messages may have been already purged due to
>> retention policy.
>> So what is the correct way of counting that?
>>
>> Thanks,
>> Jazz.
Comments
Post a Comment