Hello community,
We use Kafka extensively at our organisation.
Our SLAs are strict and require:-
— throughputs north of 1000TPS and
— 60ms latency per transaction
We run:-
— 6 brokers and 6 zookeepers
— brokers and zookeepers are hosted on ec2 instances with sufficient iops,
throughput and network bandwidth to meet our requirements
We were under the impression that using log levels (on brokers) such as
INFO or DEBUG would produce too many server logs and have an adverse impact
on performance.
Can someone with experience please confirm that log levels won't have an
adverse effect on latency and throughput of these clusters or vice-versa,
given that we come up with a strategy to clean out these logs on broker
servers after a defined unit of time ?
Does generation of system logs have potential to impact disk or compute
iops and slow us down ?
Thanks!
Posts
Hi Jesús,
I just fixed the links in the Downloads page. Thanks for reporting it.
Best,
------------------
Josep Prat
Open Source Engineering Director, Aiven
josep.prat@aiven.io | +491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B
On Wed, Oct 30, 2024, 18:21 Jesus Cea < jcea@jcea.es > wrote:
> On 30/10/24 18:18, Jesus Cea wrote:
> > On 30/10/24 18:15, Jesus Cea wrote:
> >> On 29/10/24 15:23, Josep Prat wrote:
> >>> The Apache Kafka community is pleased to announce the release for
> >>> Apache Kafka 3.8.1
> >>>
> >>> This is a bug fix release and it includes fixes and improvements.
> >>>
> >>> All of the changes in this release can be found in the release notes:
> >>> https://www.ap...
On 30/10/24 18:18, Jesus Cea wrote:
> On 30/10/24 18:15, Jesus Cea wrote:
>> On 29/10/24 15:23, Josep Prat wrote:
>>> The Apache Kafka community is pleased to announce the release for
>>> Apache Kafka 3.8.1
>>>
>>> This is a bug fix release and it includes fixes and improvements.
>>>
>>> All of the changes in this release can be found in the release notes:
>>> https://www.apache.org/dist/kafka/3.8.1/RELEASE_NOTES.html
>>
>> Webpage < https://kafka.apache.org/downloads > has some links to 3.8.0
>> instead of 3.8.1. Specifically, source code and digital signatures links.
>
> In fact, there are no digital signatures valid for 3.8.1.
Oh, digital signature available at
< https://downloads.apache.org/kafka/3.8.1/kafka_2.13-3.8.1.tgz.asc >.
--
Jesús Cea Avión _/_/ _/_/_/ _/_/_/
jcea@jcea.es - https://www.jcea.es/ ...
On 30/10/24 18:15, Jesus Cea wrote:
> On 29/10/24 15:23, Josep Prat wrote:
>> The Apache Kafka community is pleased to announce the release for
>> Apache Kafka 3.8.1
>>
>> This is a bug fix release and it includes fixes and improvements.
>>
>> All of the changes in this release can be found in the release notes:
>> https://www.apache.org/dist/kafka/3.8.1/RELEASE_NOTES.html
>
> Webpage < https://kafka.apache.org/downloads > has some links to 3.8.0
> instead of 3.8.1. Specifically, source code and digital signatures links.
In fact, there are no digital signatures valid for 3.8.1.
--
Jesús Cea Avión _/_/ _/_/_/ _/_/_/
jcea@jcea.es - https://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
Twitter: @jcea _/_/ _/_/ _/_/_/_/_/
jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy...
On 29/10/24 15:23, Josep Prat wrote:
> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 3.8.1
>
> This is a bug fix release and it includes fixes and improvements.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/3.8.1/RELEASE_NOTES.html
Webpage < https://kafka.apache.org/downloads > has some links to 3.8.0
instead of 3.8.1. Specifically, source code and digital signatures links.
--
Jesús Cea Avión _/_/ _/_/_/ _/_/_/
jcea@jcea.es - https://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
Twitter: @jcea _/_/ _/_/ _/_/_/_/_/
jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor e...
Hi Jianfeng,
The reason we don't support ZK mode is because the ZK is going to be
removed in v4.0.0, so we decided to only implement it in KRaft mode.
> I read the source code and seems kafka broker only updates the LogConfig
and then invokes maybeUpdateRemoteLogComponents to
cancel some tasks. So I want to know how if just update the log configs
without a v5 stopReplicas v5 rpc from controller to broker in zk mode?
The ZK mode is not completely implemented IIRC, and not being tested. So
I'd suggest you upgrade to KRaft for it.
Thank you.
Luke
On Mon, Oct 28, 2024 at 12:32 PM Jianfeng Lin
<jianfeng.lin@shopee.com.invalid> wrote:
> Hi kafka community, I read the KIP-950
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement#KIP950:TieredStorageDisablement-Disablement-ZookeeperBackedCluster
> >
> which
> describe the disablement of kafka tiered storage. It said that zk mode
...
The Apache Kafka community is pleased to announce the release for
Apache Kafka 3.8.1
This is a bug fix release and it includes fixes and improvements.
All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/3.8.1/RELEASE_NOTES.html
An overview of the release can be found in our announcement blog post:
https://kafka.apache.org/blog#apache_kafka_381_release_announcement
You can download the source and binary release (Scala <VERSIONS>) from:
https://kafka.apache.org/downloads#3.8.1
---------------------------------------------------------------------------------------------------
Apache Kafka is a distributed streaming platform with four core APIs:
** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.
** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.
** The Streams API ...
Hi all,
Thanks to all the reviewers!
This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes.
+1 votes
PMC Members:
* Mickael Maison
* Luke Chen
* Chia-Ping Tsai
Community:
* TengYao Chi
* Jiunn-Yang
* Federico Valeri
* Jakub Scholz
0 votes
* No votes
-1 votes
* No votes
I'll continue with the release process and the release announcement will
follow in the next few days.
Best,
------------------
Josep Prat
Open Source Engineering Director, Aiven
josep.prat@aiven.io | +491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B
On Tue, Oct 29, 2024, 01:12 Jakub Scholz < jakub@scholz.cz > wrote:
> +1 (non-binding) ... I used the staged Scala 2.13 binaries and Maven
> artifacts and run my tests. All seems to work fine. Thanks for the release.
>
> Jaku...
Yes
It seems that there is still no way to prevent partition assignment that may occur between broker shutdown and unregister (scale down). This topic seems to be under discussion in KIP-1066
On 2024/10/29 02:18:57 sunil chaudhari wrote:
> So if I understand this discussion correctly, there is no clear path to
> scale down cluster is it?
>
>
>
> On Tue, 29 Oct 2024 at 7:40 AM, Luke Chen < sh...@gmail.com > wrote:
>
> > Hi hayoung,
> >
> > This is the problem that KIP-1066
> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1066%3A+Mechanism+to+cordon+brokers+and+log+directories
> > >
> > is trying to address.
> > It is under discussion, and should be included in the upcoming releases.
> >
> > Thanks.
> > Luke
> >
> >
> >
> > On Tue, Oct 29, 2024 at 12:45 AM hayoung lee < l4...@gmail.com > wrote:
> >
> >...
So if I understand this discussion correctly, there is no clear path to
scale down cluster is it?
On Tue, 29 Oct 2024 at 7:40 AM, Luke Chen < showuon@gmail.com > wrote:
> Hi hayoung,
>
> This is the problem that KIP-1066
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1066%3A+Mechanism+to+cordon+brokers+and+log+directories
> >
> is trying to address.
> It is under discussion, and should be included in the upcoming releases.
>
> Thanks.
> Luke
>
>
>
> On Tue, Oct 29, 2024 at 12:45 AM hayoung lee < l4314538@gmail.com > wrote:
>
> > Hi Team,
> >
> > I am currently operating a Kafka cluster in KRaft mode and would like to
> > raise a few questions and suggestions regarding the cluster scale-down
> > process.
> >
> > Recently, I scaled down the cluster by removing one broker; however, I
> > encountered a situation where the removed ...
Hi hayoung,
This is the problem that KIP-1066
< https://cwiki.apache.org/confluence/display/KAFKA/KIP-1066%3A+Mechanism+to+cordon+brokers+and+log+directories >
is trying to address.
It is under discussion, and should be included in the upcoming releases.
Thanks.
Luke
On Tue, Oct 29, 2024 at 12:45 AM hayoung lee < l4314538@gmail.com > wrote:
> Hi Team,
>
> I am currently operating a Kafka cluster in KRaft mode and would like to
> raise a few questions and suggestions regarding the cluster scale-down
> process.
>
> Recently, I scaled down the cluster by removing one broker; however, I
> encountered a situation where the removed broker still appeared in the
> cluster, causing new partitions to be assigned to it. To address this, I
> was able to remove the broker from the controller metadata using the
> kafka-cluster.sh command. However, I could not find any documentation on
> this method, so I had to refer to t...
+1 (non-binding) ... I used the staged Scala 2.13 binaries and Maven
artifacts and run my tests. All seems to work fine. Thanks for the release.
Jakub
On Thu, Oct 17, 2024 at 10:27 PM Josep Prat <josep.prat@aiven.io.invalid>
wrote:
> Hello Kafka users, developers and client-developers,
>
> This is the second release candidate of Apache Kafka 3.8.1.
>
> This is a bugfix release with several fixes.
>
> Release notes for the 3.8.1 release:
> https://dist.apache.org/repos/dist/dev/kafka/3.8.1-rc1/RELEASE_NOTES.html
>
> **** Please download, test and vote by Tuesday, October 22, 9am ET*
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://dist.apache.org/repos/dist/dev/kafka/3.8.1-rc1/
>
> * Docker release artifacts to be voted upon:
> apache/kafka:3.8.1-rc1
> apache/kaf...
Hi,
Just wanting to check if something (that we could avoid if it's bad) is a performance concern in any way: Proliferation of record schemata. I believe Schemas are internally deduplicated as they are hashable, and are likely to be reused so the savings from deduplication is substantial. However, should we expect performance degradation (beyond having to read a bit more data to handle the additional Schema data) if we have, say, millions of unique record Schemata due to variations in Schema content?
Cheers
Paul
Hi Team,
I am currently operating a Kafka cluster in KRaft mode and would like to
raise a few questions and suggestions regarding the cluster scale-down
process.
Recently, I scaled down the cluster by removing one broker; however, I
encountered a situation where the removed broker still appeared in the
cluster, causing new partitions to be assigned to it. To address this, I
was able to remove the broker from the controller metadata using the
kafka-cluster.sh command. However, I could not find any documentation on
this method, so I had to refer to the Kafka source code to discover this
solution. The issue is now resolved, but I would like to ask if there is a
more streamlined approach for scaling down.
Additional Questions and Suggestions:
1.
*Automatic Cluster Exclusion*
I initially expected that a removed broker would be automatically
excluded from the cluster after a certain period. However, the broker
remained part of the cluster for quite s...
Hi Team,
I am currently operating a Kafka cluster in KRaft mode and would like to
raise a few questions and suggestions regarding the cluster scale-down
process.
Recently, I scaled down the cluster by removing one broker; however, I
encountered a situation where the removed broker still appeared in the
cluster, causing new partitions to be assigned to it. To address this, I
was able to remove the broker from the controller metadata using the
kafka-cluster.sh command. However, I could not find any documentation on
this method, so I had to refer to the Kafka source code to discover this
solution. The issue is now resolved, but I would like to ask if there is a
more streamlined approach for scaling down.
Additional Questions and Suggestions:
1.
*Automatic Cluster Exclusion*
I initially expected that a removed broker would be automatically
excluded from the cluster after a certain period. However, the broker
remained part of the cluster for quite s...
Hi kafka community, I read the KIP-950
< https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement#KIP950:TieredStorageDisablement-Disablement-ZookeeperBackedCluster >
which
describe the disablement of kafka tiered storage. It said that zk mode
won't be supported and I can't find out the reason, with confusion, I have
a question:
This KIP proposes an implement with stopReplicas v5 to let brokers know
which topic partitions should stop their remote storage functions when
"remote.log.copy.disable" is set to true, different from kraft mode, it
takes an extra rpc round. I read the source code and seems kafka broker
only updates the LogConfig and then invokes maybeUpdateRemoteLogComponents to
cancel some tasks. So I want to know how if just update the log configs
without a v5 stopReplicas v5 rpc from controller to broker in zk mode? And
why zk mode is not supported to disable?
Really appreciate if Luke Chen< showuon...
I read the code again. In `ApiVersionsResponseData._write()`, there is a
check of the version. If >=3, the API keys array is pre-prended by one byte
that contains the length of the following array. Else, it is an INT32 that
contains the length of the following array. So, the 0x3c in my case is the
count of APIs in the response i.e., 60. While the Wireshark GUI does show
the API Key and Version, it appears in square brackets. Therefore, I am
not sure, if Wireshark _always_ assumes that the response is version 3 or
higher.
On Sun, Oct 27, 2024 at 11:25 PM Steven Miller < steve@idrathernotsay.com >
wrote:
> There used to be a kafka dissector that I thought was shipped with
> Wireshark - does that decode this properly? I wasn't certain if you were
> trying to decode it yourself or if the kafka dissector wasn't working.
>
> I wrote something up here about using the dissector, but it's been almost
> a decade since then…
>
> ...
There used to be a kafka dissector that I thought was shipped with Wireshark - does that decode this properly? I wasn't certain if you were trying to decode it yourself or if the kafka dissector wasn't working.
I wrote something up here about using the dissector, but it's been almost a decade since then…
-Steve
> On Oct 27, 2024, at 12:42 PM, Chain Head < mrchainhead@gmail.com > wrote:
>
> (Apologies - I sent the same mail to the dev mailing list by mistake!)
>
> Hello,
> I set-up a simple Java client to publish a sample message to Kafka broker.
> I can see API Versions request v3 and response messages. I am able to map
> all the bytes as described here
> ( https://kafka.apache.org/protocol.html#The_Messages_ApiVersions ) for
> API
> Versions request and response except for 1 byte as shown below.
>
> - Length - INT32
> - Correlation ID - INT32
> - Error code - INT16
> - ? - 1 byte value ...
The correlation ID is in the response **header**.
On Sun, Oct 27, 2024 at 10:36 PM Ömer Şiar Baysal < osiarbaysal@gmail.com >
wrote:
> Hi,
>
> Could not able to find correlationID response for the Apiversions API, did
> you mean Produce API?
>
> Maybe it is throttleTimeSize ?
>
>
> https://github.com/a0x8o/kafka/blob/54eff6af115ee647f60129f2ce6a044cb17215d0/core/src/main/scala/kafka/api/ProducerResponse.scala#L68
>
> On Sun, Oct 27, 2024, 17:42 Chain Head < mrchainhead@gmail.com > wrote:
>
> > (Apologies - I sent the same mail to the dev mailing list by mistake!)
> >
> > Hello,
> > I set-up a simple Java client to publish a sample message to Kafka
> broker.
> > I can see API Versions request v3 and response messages. I am able to map
> > all the bytes as described here
> > ( https://kafka.apache.org/protocol.html#The_Messages_ApiVersions ) for
> > API
>...
Hi,
Could not able to find correlationID response for the Apiversions API, did
you mean Produce API?
Maybe it is throttleTimeSize ?
https://github.com/a0x8o/kafka/blob/54eff6af115ee647f60129f2ce6a044cb17215d0/core/src/main/scala/kafka/api/ProducerResponse.scala#L68
On Sun, Oct 27, 2024, 17:42 Chain Head < mrchainhead@gmail.com > wrote:
> (Apologies - I sent the same mail to the dev mailing list by mistake!)
>
> Hello,
> I set-up a simple Java client to publish a sample message to Kafka broker.
> I can see API Versions request v3 and response messages. I am able to map
> all the bytes as described here
> ( https://kafka.apache.org/protocol.html#The_Messages_ApiVersions ) for
> API
> Versions request and response except for 1 byte as shown below.
>
> - Length - INT32
> - Correlation ID - INT32
> - Error code - INT16
> - ? - 1 byte value 0x3c
> - Array of API keys, supported versions and tagged fields
> ...
(Apologies - I sent the same mail to the dev mailing list by mistake!)
Hello,
I set-up a simple Java client to publish a sample message to Kafka broker.
I can see API Versions request v3 and response messages. I am able to map
all the bytes as described here
( https://kafka.apache.org/protocol.html#The_Messages_ApiVersions ) for
API
Versions request and response except for 1 byte as shown below.
- Length - INT32
- Correlation ID - INT32
- Error code - INT16
- ? - 1 byte value 0x3c
- Array of API keys, supported versions and tagged fields
- Throttle - INT32
- Tagged fields.
Can you please help me understand?
Thanks.
Hi there,
I am currently working on a project for the logistic industry, and I think
Kafka will be a great fit for us.
I was wondering if you can recommend any software companies (maybe partner)
that can help us achieving this project.
Let me know if it's something we can help with.
Looking forward to hearing from you.
Alex
Hi Artur,
Thanks for the question!
In transactions there's no distinction between partitions within the same
topic and partitions in different topics. You can have transactions across
multiple topics, and multiple partitions in those topics.
Hope this is more clear,
Greg
On Fri, Oct 25, 2024 at 5:11 AM Artur Bondek < artur.bondek@zowie.ai > wrote:
> Hi,
>
> I would like to ask about the kafka producer transaction. Documentation
> states ( https://kafka.apache.org/documentation/#semantics ):
> Also beginning with 0.11.0.0, *the producer supports the ability to send
> messages to multiple topic partitions* using transaction-like semantics:
> i.e. either all messages are successfully written or none of them are.
>
> Does it mean that the producer supports transactions in multiple topics
> (and multiple partitions)? Or it's just for multiple partitions but within
> a single topic?
>
> Artur Bondek
>
Hi,
I would like to ask about the kafka producer transaction. Documentation
states ( https://kafka.apache.org/documentation/#semantics ):
Also beginning with 0.11.0.0, *the producer supports the ability to send
messages to multiple topic partitions* using transaction-like semantics:
i.e. either all messages are successfully written or none of them are.
Does it mean that the producer supports transactions in multiple topics
(and multiple partitions)? Or it's just for multiple partitions but within
a single topic?
Artur Bondek
Vinay, a "last access" timestamp would be much harder to implement, as it
would be a mutable property by nature. In contrast, the timestamp of a
topic creation is immutable and persisted (AFAIK).
I was really surprised to see that this information is not already
available through the Kafka Admin client. Any idea why? I would assume this
information to be valuable to a lot of things in the Kafka ecosystem.
I'm not too familiar with the process for suggesting enhancements to Kafka.
Would an addition like this require a KIP?
Regards,
Erik
On Mon, Oct 21, 2024 at 7:14 PM Vinay Bagare < vinay.bagare@gmail.com > wrote:
> I had been tracking this and my intention is to capture last access
> timestamp for identifying topic activity staleness,
> I feel we would probably benefit from having both Create/Last access
> Timestamp similar to what Erik is requesting (I hope I am understanding his
> requirement).
> For now, I am doing a hula...
Thanks Greg! It helps so much. This KIP seems familiar somehow, I should go through it again. Really appreciated! ---- Replied Message ---- From Greg Harris<greg.harris@aiven.io.INVALID> Date 10/25/2024 00:40 To dev@kafka.apache.org Cc users@kafka.apache.org Subject Re: doc clarification about meesage format Hey Xiang, Thanks for your questions! This is getting to the limit of my knowledge, but I'll answer as best I can. The partitionLeaderEpoch is only set once during the batch lifetime (during Produce), and is not mutated any other time. This includes when data is fetched by other replicas and by consumers, and when partition leadership changes. I believe this field is a record of which partitionLeaderEpoch was active at the time the batch was produced, and can be different for different batches within a partition as leadership changes. I wouldn't call this "outdated", as I think there is an intentional use for this historical leadership data in the log [1]....
Hi all!
Gentle ping on this thread.
Thanks!
------------------
Josep Prat
Open Source Engineering Director, Aiven
josep.prat@aiven.io | +491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B
On Thu, Oct 17, 2024, 22:27 Josep Prat < josep.prat@aiven.io > wrote:
> Hello Kafka users, developers and client-developers,
>
> This is the second release candidate of Apache Kafka 3.8.1.
>
> This is a bugfix release with several fixes.
>
> Release notes for the 3.8.1 release:
> https://dist.apache.org/repos/dist/dev/kafka/3.8.1-rc1/RELEASE_NOTES.html
>
> **** Please download, test and vote by Tuesday, October 22, 9am ET*
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to...
Hey Xiang,
Thanks for your questions! This is getting to the limit of my knowledge,
but I'll answer as best I can.
The partitionLeaderEpoch is only set once during the batch lifetime (during
Produce), and is not mutated any other time. This includes when data is
fetched by other replicas and by consumers, and when partition leadership
changes.
I believe this field is a record of which partitionLeaderEpoch was active
at the time the batch was produced, and can be different for different
batches within a partition as leadership changes. I wouldn't call this
"outdated", as I think there is an intentional use for this historical
leadership data in the log [1].
[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Epoch+rather+than+High+Watermark+for+Truncation
Thanks,
Greg
On Wed, Oct 23, 2024 at 8:07 PM Xiang Zhang < xiangzhang1128@gmail.com >
wrote:
> Thank you Greg for all the kn...
Hi community, I am trying to run/debug Kafka in IDEA, it works fine except the following SLF4J errors. I kind of know how to fix this, but it seems messy to me that a workaround is required when the code is just downloaded and built. I am just wondering whether this is due to some misconfigured setup or this can be fixed or avoided out of the box. I would love your thoughts on this. Thanks ! Xiang Zhang xiangzhang1128@gmail.com
Updating log.dirs using 'kafka-reassign-partitions' spamming 'ReplicaFetcher: Current offset 0 for partition 0 is out of range, ... Reset fetch offset to 0'
Hi Team,
I am currently struggling with below behavior. Does anybody notice something similar in the past or can help me with it?
In my scenario I am currently running a three Kafka brokers (1, 2, 3) cluster in Zookeeper mode.
I am using confluent community edition 7.7.1.
On broker 1 I have three log.dirs.
On broker 2 I have two log.dirs.
On broker 3 I have one log.dirs.
With kafka-reassign-partitions.sh I would like to move all data from dir2 to dir1 on broker 2.
My prepared topic-json file is prepared correctly, I am not doing a new replica distribution, just changing the log dirs.
When doing this either one or two brokers are spamming the log with:
"INFO [ReplicaFetcher replicaId=1, LeaderId=2, fetcherId=1] Reset fetch offset for partition test.partition-0 from 0 to the current local replica's end offset 0 (kafka.server.ReplicaFetcherThread)
INFO [ReplicaFetcher replicaId=1, LeaderId=2, fetcherId=1] current offset 0 for partition test.partition-0 is out o...
Thank you Greg for all the knowledge, some follow up questions.
Does partitionLeaderEpoch always reflect the latest leader election or an
old epoch can be allowed ? If it is the first case, then I agree
partitionLeaderEpoch should not be included in CRC computation. But it
raises some new questions for me, which is which roles will check the
checksum and under what circumstances? I am asking this because after the
producing process, any record in the broker log can have an outdated leader
epoch field once leader election happens, right ? Do they get updated ?
Sorry for all the questions, I have been using Kafka for several years and
want to dive deep into it a little bit. I have become more interested and
ready to find out on my own. But still look forward to your thoughts on
this if the questions above do make some sense.
Thanks,
XIang
Greg Harris <greg.harris@aiven.io.invalid> 于2024年10月24日周四 00:25写道：
> Hi Xiang,
>
> Thanks for your question...
I had been tracking this and my intention is to capture last access timestamp for identifying topic activity staleness,
I feel we would probably benefit from having both Create/Last access Timestamp similar to what Erik is requesting (I hope I am understanding his requirement).
For now, I am doing a hula dance (round about) by describing the topic and identifying its last activity (if anyone has better suggestions instead of this, please do let me know).
Best,
Vinay Bagare
From: Erik Godding Boye < egboye@gmail.com >
Date: Monday, October 21, 2024 at 10:17 AM
To: users@kafka.apache.org < users@kafka.apache.org >
Subject: Re: Get topic creationTimestamp through Admin Client (or Client)
Hi Ömer,
I don't disagree with you on the general principles for deletion, but let's
keep focus on the question I posted. :-) And yes, I am aware of the
LinkedIn state-machine for cleaning up metadata.
> What do you mean by "this must be done exter...
Hi Ömer,
I don't disagree with you on the general principles for deletion, but let's
keep focus on the question I posted. :-) And yes, I am aware of the
LinkedIn state-machine for cleaning up metadata.
> What do you mean by "this must be done external to " ?
I mean that I want to do this with brokers as my only interface (Client
and/or Admin Client). We are in the process of migrating from zookeeper to
KRaft mode, and I don't want the burden of having to coordinate this
migration into my toolbox. In addition, the clusters are insulated, so
zookeepers/controllers are unavailable to this app.
Wouldn't it make sense to include a creation timestamp in the topic
metadata returned to an Admin Client - if it is possible?
Regards,
Erik
On Mon, Oct 21, 2024 at 4:24 PM Ömer Şiar Baysal < osiarbaysal@gmail.com >
wrote:
> Hi Erik,
>
> What do you mean by "this must be done external to " ?
>
> If...
Hi Erik,
What do you mean by "this must be done external to " ?
If the cluster still uses ZK for metadata, you can construct a read-only
Zookeeper client to get zNode creation timestamp for the topic from ZK
node, which reflects the creation time.
Also determining unused topics can be tricky if consumers come and go by
irregular basis, for example some batch jobs for BI and/or monthly
generated reports, etc.
Check this blog post to see how LinkedIn does it.
https://engineering.linkedin.com/content/engineering/en-us/blog/2022/topicgc_how-linkedin-cleans-up-unused-metadata-for-its-kafka-clu
IMHO hard deletes should be avoided for any kind of data platform.
Kind regards,
OSB
On Sun, Oct 20, 2024, 14:24 Erik Godding Boye < egboye@gmail.com > wrote:
> Hi Kafka experts,
>
> I wrote a small utility application to terminate (delete) unused topics in
> our clusters. The application is hosted at
> https://github.com/statne...
Hi Kafka experts,
I wrote a small utility application to terminate (delete) unused topics in
our clusters. The application is hosted at
https://github.com/statnett/k3a-topic-terminator (but this is not an
attempt to promote the app; it's fresh ;-). We just got rid of 2816 topics
in one of our clusters, and now we want to find even more garbage.
So my question is: *Can the topic creation timestamp be obtained through a
Kafka Admin Client (or Client)?*
I think the answer is NO, but would it be possible to implement such a
feature? As a user, I would expect to find this information through the
TopicDescription
< https://kafka.apache.org/38/javadoc/org/apache/kafka/clients/admin/TopicDescription.html >
class.As far as I understand, this information is available in the cluster
- either within Zookeepers and/or on the __cluster_metadata topic. I have
tried to set up a consumer towards the metadata topic, but no luck so far.
I am very interested in potential wo...
Hello Kafka users, developers and client-developers,
This is the second release candidate of Apache Kafka 3.8.1.
This is a bugfix release with several fixes.
Release notes for the 3.8.1 release:
https://dist.apache.org/repos/dist/dev/kafka/3.8.1-rc1/RELEASE_NOTES.html
**** Please download, test and vote by Tuesday, October 22, 9am ET*
Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS
* Release artifacts to be voted upon (source and binary):
https://dist.apache.org/repos/dist/dev/kafka/3.8.1-rc1/
* Docker release artifacts to be voted upon:
apache/kafka:3.8.1-rc1
apache/kafka-native:3.8.1-rc1
* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/
* Javadoc:
https://dist.apache.org/repos/dist/dev/kafka/3.8.1-rc1/javadoc/
* Tag to be voted upon (off 3.8 branch) is the 3.8.1 tag:
https://github.com/apache/kafka/releases/tag/3.8.1-rc1
* Documen...