Kafka

Hi Tom, Thanks for running this release! I've done the following: - Checked signatures and checksums - Checked javadocs/maven artifacts - Built from source and run all tests with Java 11 - Ran quickstart on Scala 2.13 artifact with Java 11 It looks like the website has not been updated yet, I still only see 3.1.0. When you'll add 3.1.1, let's make sure we mention reload4j in the notable changes section. +1 (binding) Thanks, Mickael On Fri, Apr 29, 2022 at 11:12 AM Tom Bentley < tbentley@redhat.com > wrote: > > Hello Kafka users, developers and client-developers, > > This is the first candidate for release of Apache Kafka 3.1.1. > > Apache Kafka 3.1.1 is a bugfix release and 30 issues have been fixed > since 3.1.0. > > Release notes for the 3.1.1 release: > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/RELEASE_NOTES.html > > *** Please download, test and vote by 09:00 UTC, Friday 6th Ma...

Luke, I would argue that https://issues.apache.org/jira/browse/KAFKA-13636 is a critical defect as it can have a very serious impact. We run on AWS MSK which supports these versions: https://docs.aws.amazon.com/msk/latest/developerguide/supported-kafka-versions.html . We are currently on 2.7.2. I note that MSK does not support any 3.x (maybe they're not ready for the Zookeeper removal). So I suspect we're going to need a 2.x if MSK is going to adopt it any time soon. I'd be happier with a 2.7.3 incorporating KAFKA-13636 in order to minimise the risk of introducing other issues, or the 2.8.2 if that's not possible. What can we do to make this happen ASAP? Regards, James. On 29/04/2022, at 14:50, Luke Chen < showuon@gmail.com <mailto: showuon@gmail.com >> wrote: Hi James, So far, v2.8.2 is not planned, yet. And usually, the patch release only has one, that is, v2.8.0, and v2.8.1. But there are of course some exceptions that some re...

Luke, Do you know if 2.8.2 will be released anytime soon? It appears to be waiting on https://issues.apache.org/jira/browse/KAFKA-13805 for which fixes are available. Regards, James. On 11/04/2022, at 14:22, Luke Chen < showuon@gmail.com <mailto: showuon@gmail.com >> wrote: Hi James, This looks like this known issue KAFKA-13636 < https://issues.apache.org/jira/browse/KAFKA-13636 >, which should be fixed in the newer version. Thank you. Luke On Mon, Apr 11, 2022 at 9:18 AM James Olsen < james@inaseq.com <mailto: james@inaseq.com >> wrote: I recently observed the following series of events for a particular partition (MyTopic-6): 2022-03-18 03:18:28,562 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] 'executor-thread-2' [Consumer clientId=consumer-MyTopicService-group-3, groupId=MyTopicService-group] Setting offset for partition MyTopic-6 to the committed offset FetchPosition{offset=438, offs...

Liam, Thanks a lot for your help. I have not made metrics yet. I have just checked everything about file privileges again, all correct. Then I notice there's an error in state-change log as below, [2022-04-26 19:07:17,727] ERROR [Broker id=1] Topic ID in memory: 7LRTCKIRT5W5qZUrLQ_0wA does not match the topic ID for partition topic-test-0 received: o-iJg3N9Qk61rzKfdor5tA. (state.change.logger) I don't know whether the error will be the root cause. Regards, Peter -----Original Message----- From: Liam Clarke-Hutchinson < lclarkeh@redhat.com > Sent: 2022年4月28日 16:45 To: users@kafka.apache.org Subject: Re: How to deal with the Error: KAFKA_STORAGE_ERROR Hi Peter, Firstly, I'd check disk health, then I'd check owners and permissions on files in your log dir, eliminate those as issues. Secondly, are you tracking metrics on offline log dirs? Cheers, Liam On Thu, 28 Apr 2022 at 15:55, Dong, Peter A. (NSB - CN/Qingdao) <...

Hi Peter, Firstly, I'd check disk health, then I'd check owners and permissions on files in your log dir, eliminate those as issues. Secondly, are you tracking metrics on offline log dirs? Cheers, Liam On Thu, 28 Apr 2022 at 15:55, Dong, Peter A. (NSB - CN/Qingdao) < peter.a.dong@nokia-sbell.com > wrote: > > Greetings, Kafka specialists > > A strange issue in my Kafka instance that blocked my for a couple of days. > > 1. Cannot produce event to [topic-test] due to a KAFKA_STORAGE_ERROR > > 2. It seems to me the log segment file is not having any problem. > > I can dump log, index, timeindex with kafka-dump-log without seeing any > error. > > 3. No error when I tried to produce events to other topics. Kafka > log files are in a same disk partition. > > 4. Restart Kafka instance and zookeeper instance did not help. > > 5. I cannot find useful information ab...

Rebalancing happens mainly because of these reasons: You restart consumer Consumer host is not reachable You stop consumer All above situations are fine when you have sufficient number of consumers(threads) to read from the available partitions and all consumers are logically distributed across multiple consumer groups. In case you have grouped many consumers into same group reading from multiple topics then it may cause frequent rebalancing. Even if one consumer has availability issue whole group will go for rebalancing. Recently we gone thru same problem and what we did is, whole group is devided in many groups logically. Each group is associated with specific topic and consumers are distributed across those groups. Now in case of consumer fails, only that small group is rebalanced not others. I hope i have answered your question with my limited knowledge😊 Corrections are welcome! Regards, Sunil. On Thu, 28 Apr 2022 at 8:42 AM, Luke Chen ...

Greetings, Kafka specialists A strange issue in my Kafka instance that blocked my for a couple of days. 1. Cannot produce event to [topic-test] due to a KAFKA_STORAGE_ERROR 2. It seems to me the log segment file is not having any problem. I can dump log, index, timeindex with kafka-dump-log without seeing any error. 3. No error when I tried to produce events to other topics. Kafka log files are in a same disk partition. 4. Restart Kafka instance and zookeeper instance did not help. 5. I cannot find useful information about the error from server.log with TRACE level. Could you please let me know whether similar issue ever happened before? Where shall I go to dig further and continue my investigation? Thanks a lot! Peter The kafka client log kafka-console-producer --bootstrap-server 135.251.236.162:9092 --topic topic-test >[2022-04-28 11:12:10,925] WARN [Producer clientId=console-producer] Got error produce...

1. is it correctly the idea about B ? i think it maybe the major factor for rebalance in my case ,because consuming data is slowly. => Looks like so, but we cannot confirm that because we don't have other information. You should check the consumer log to see why the consumer leave group. 2. i have do an experiment for assert B. but not reproduce A：consumer fast or slow can't trigger it You should also adjust the heartbeat interval to allow the heartbeat detect the poll expiration. You can refer to this test: https://github.com/apache/kafka/blob/trunk/core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala#L167 3. Any tips on APIs or debug techniques to figure out rebalance causes On server side, you can check log like this: "Preparing to rebalance group xxx ... (reason: yyyyy) 4. How can I trigger it manually ? => same as question 2 5. Is it a bad idea to have the same Consumer Group (Same ID) consuming ...

Hi guys. I have a question about upgrades. I am currently on 2.7 or confluent opensource 6.1. since my last upgrade from confluent opensoure 5.x (kafka 2.0) my server.properties has this: inter.broker.protocol.version=2.7 log.message.format.version=2.0 following confluent notes for upgrades mentions that on my first restart those 2 configs should be set to: inter.broker.protocol.version=*2.7* log.message.format.version=*2.7* then before my second restart i should set: inter.broker.protocol.version=*2.8* log.message.format.version=2.7 and it seems i can leave message format to 2.7 Because the message format is the same in Confluent Platform 3.3.x (kafka 0.11.0.x) through 7.1.x (kafka 3.1.*), this step is optional if the upgrade is from Confluent Platform 3.3.x (kafka 0.11.0.*) or newer. Also I guess that is the reason log.message.format.version was set to 2.0 at the beginning. Am I understanding the upgrade process correctly? Thanks very muc...

HI： We are seeing unexpected rebalances in golang consumers, described below. 1. We have a topic with 36 partitions，and one consumer (lets name it consumer1) consuming it. 2. Run kafka in Docker and configuration：We use defaults 3. Consumer consuming data is slowly about cost 1s for one piece of data 4. All the consumers for topic A are in the same group 5. The rebalances are intermittent and hard to reproduce. We see no obvious errors in the logs. 6. No matter how to change the configuration that affects rebalance ，it always rebalance The configuration that affects rebalance below: max.poll.interval.ms max.poll.records request.timeout.ms session.timeout.ms As far as I am concerned, conditions of the rebalance contains： a consumer is considered DEAD by the group coordinator. A. when the consumer is busy, which means that no heartbea...

Hi Lavanya, Apache Kafka is a free open source software project and as such, doesn't provide paid support. However, numerous companies do offer a supported Kafka product, you will be able to find them via googling "kafka support" or similar. Cheers, Liam On Wed, 27 Apr 2022 at 02:20, Lavanya Voruganti < lavanyav@dhanyaayai.com > wrote: > Dear Kafka Team, > > *Greetings From Dhanyaayai..!!!* > > One of our client is having a requirement of Kafka Support with Quantity - > 04. > > Could you please help me with the reseller discounts for the same. > > Regards > Lavanya Reddy > +91 9701328158 >

Hi, Bruno. Thanks for the reply. Your understanding is correct. This is a regression introduced only in the 3.2 branch. Sorry for the late notice. Jun On Tue, Apr 26, 2022 at 10:04 AM Bruno Cadonna < cadonna@apache.org > wrote: > Hi Jun, > > Thank you for your message! > > Now I see how this issue was introduced in 3.2.0. The fix for the bug > described in KAFKA-12841 introduced it, right? I initially understood > that the PR you want to include is the fix for the bug described in > KAFKA-12841 which dates back to 2.6. > > I think that classifies as a regression. > > I will abort the voting and create a new release candidate. > > Best, > Bruno > > On 26.04.22 18:09, 'Jun Rao' via kafka-clients wrote: > > Hi, Bruno, > > > > Could we include https://github.com/apache/kafka/pull/12064 > > < https://github.com/apache/kafka/pull/12064 > in 3.2.0? This fixes an ...

Hi Jun, Thank you for your message! Now I see how this issue was introduced in 3.2.0. The fix for the bug described in KAFKA-12841 introduced it, right? I initially understood that the PR you want to include is the fix for the bug described in KAFKA-12841 which dates back to 2.6. I think that classifies as a regression. I will abort the voting and create a new release candidate. Best, Bruno On 26.04.22 18:09, 'Jun Rao' via kafka-clients wrote: > Hi, Bruno, > > Could we include https://github.com/apache/kafka/pull/12064 > < https://github.com/apache/kafka/pull/12064 > in 3.2.0? This fixes an > issue introduced in 3.2.0 where in some of the error cases, the producer > interceptor is called twice for the same record. > > Thanks, > > Jun > > On Tue, Apr 26, 2022 at 6:34 AM Bruno Cadonna < cadonna@apache.org > <mailto: cadonna@apache.org >> wrote: > > Hi all, > ...

Hi, Bruno, Could we include https://github.com/apache/kafka/pull/12064 in 3.2.0? This fixes an issue introduced in 3.2.0 where in some of the error cases, the producer interceptor is called twice for the same record. Thanks, Jun On Tue, Apr 26, 2022 at 6:34 AM Bruno Cadonna < cadonna@apache.org > wrote: > Hi all, > > This is a gently reminder to vote for the first candidate for release of > Apache Kafka 3.2.0. > > I added the 3.2 documentation to the kafka site. That means > https://kafka.apache.org/32/documentation.html works now. > > A successful system tests run can be found here: > https://jenkins.confluent.io/job/system-test-kafka/job/3.2/24/ > > Thank you to Michal for voting on the release candidate. > > Best, > Bruno > > On 15.04.22 21:05, Bruno Cadonna wrote: > > Hello Kafka users, developers and client-developers, > > > > This is the first candidate for release of...

Hi all, This is a gently reminder to vote for the first candidate for release of Apache Kafka 3.2.0. I added the 3.2 documentation to the kafka site. That means https://kafka.apache.org/32/documentation.html works now. A successful system tests run can be found here: https://jenkins.confluent.io/job/system-test-kafka/job/3.2/24/ Thank you to Michal for voting on the release candidate. Best, Bruno On 15.04.22 21:05, Bruno Cadonna wrote: > Hello Kafka users, developers and client-developers, > > This is the first candidate for release of Apache Kafka 3.2.0. > > * log4j 1.x is replaced with reload4j (KAFKA-9366) > * StandardAuthorizer for KRaft (KIP-801) > * Send a hint to the partition leader to recover the partition (KIP-704) > * Top-level error code field in DescribeLogDirsResponse (KIP-784) > * kafka-console-producer writes headers and null values (KIP-798 and > KIP-810) > * JoinGroupRequest and LeaveGroupRequest...

Dear Kafka Team, *Greetings From Dhanyaayai..!!!* One of our client is having a requirement of Kafka Support with Quantity - 04. Could you please help me with the reseller discounts for the same. Regards Lavanya Reddy +91 9701328158

Hi Amanda, Apache Kafka is an open source project. You can download, install, and use the software for free. Some companies offer Apache Kafka as a cloud service or as an on-prem software. You can find those companies by asking google. Best, Bruno On 25.04.22 17:11, Amanda Jampaulo wrote: > Hi Team, good day! > > We are SantoDigital, a Brazilian company and Google's premier partner. > > We would like to know more about the Apache Kafka solution. We have a > customer who is interested in your solution. > > We want to know how we can make the contract for the solution, what the > price of the list and what the price is for resale and how is the billing? > > If this is not the correct channel for this type of question, please, we > kindly ask you to direct our email to the responsible team. > > We are looking forward to hearing from you. > > Best regards, > > Amanda Jampaulo > ...

Hi Team, good day! We are SantoDigital, a Brazilian company and Google's premier partner. We would like to know more about the Apache Kafka solution. We have a customer who is interested in your solution. We want to know how we can make the contract for the solution, what the price of the list and what the price is for resale and how is the billing? If this is not the correct channel for this type of question, please, we kindly ask you to direct our email to the responsible team. We are looking forward to hearing from you. Best regards, Amanda Jampaulo Sales Operations Manager +55 11 98444-4509 santodigital.com.br < https://www.santodigital.com.br > [image: LinkedIn] < https://www.linkedin.com/company/santodigital/ > [image: Instagram] < https://www.instagram.com/santodigital/ > [image: YouTube] < https://www.youtube.com/channel/UCw4UVLYRFUyYUDdLwmmlfcA > [image: Facebook] < https://www.facebook.com/santodigital > [i...

Hi All, My setup is shown in the attached JPEG file. In my setup, I have a stretch cluster spread across 2 data-centres (Geographically distant). The network latency as measured by ping round-trip-time is about 50ms. There are 4 brokers in each DC. In each of these data-centres, I have a Kafka Producer application and a KStream application. My test involves the following : Make producer in DC1 produce records at the rate of say about 3000 messages per second. Shutdown all brokers in DC2, so as to simulate a site-outage, for about 30 minutes. When I re-start the brokers in DC2, I encounter the following errors in the logs of my KStream application. (Note : The Kafka producer application does not seem to suffer from any such errors). Obviously, I can see that the replica lag in the brokers of DC2 has increased, but is gradually reducing due to the brokers in DC2 now trying to fetch records from DC1 brokers. However, the KStream application shuts down and cannot be started up ...

Hi Luke, After moving all data from one disk to another on every Kafka broker, I went through some weird behaviour I wanted to highlight here. Basically the disk storage kept increasing even though there is no change on `bytes in` metric per broker. After investigation, I've seen that all segment log files in the new log.dir had a modification time set to the moment when the copy had been done. So I guess the process applying the retention policy (log cleaner?) uses that timestamp to check whether the segment file should be deleted or not. So I ended with a lot more data than we supposed to store, since we are basically doubling the retention time of all the freshly moved data. It seems a bit off that the Kafka-reassign-partition command doesn't handle that somehow when moving the data. Do you please think of a way to avoid the issue? Do you confirm that the log cleaner uses the modification date of the segment log files in the FS to check whether ...

Hello all, I have executed some produce/consume system tests which all passed. Also everything passed from https://github.com/tombentley/kafka-verify-rc - checking signatures, checksums, with gradle unit & integration tests, etc. Good from me (non-binding). Thank you pi 15. 4. 2022 o 21:05 Bruno Cadonna < cadonna@apache.org > napísal(a): > Hello Kafka users, developers and client-developers, > > This is the first candidate for release of Apache Kafka 3.2.0. > > * log4j 1.x is replaced with reload4j (KAFKA-9366) > * StandardAuthorizer for KRaft (KIP-801) > * Send a hint to the partition leader to recover the partition (KIP-704) > * Top-level error code field in DescribeLogDirsResponse (KIP-784) > * kafka-console-producer writes headers and null values (KIP-798 and > KIP-810) > * JoinGroupRequest and LeaveGroupRequest have a reason attached (KIP-800) > * Static membership protocol lets the leader skip assig...

Hi Matt, Apache Kafka is a free open source software project, not too much swag on hand ;) Cheers, Liam On Tue, 19 Apr 2022 at 02:39, Matt Gilbert < matthewgabrielgilbert@gmail.com > wrote: > I just wanted to reach out and thank you for all you do. I am a Senior > Test Engineer and wanted to reach out to see if you had any swag or gear, I > would love to represent your company! >

I just wanted to reach out and thank you for all you do. I am a Senior Test Engineer and wanted to reach out to see if you had any swag or gear, I would love to represent your company!

Hi Lorcan, Thanks for your interest! The instructions for subscribing are available here: https://kafka.apache.org/contact Thanks, John On Fri, Apr 15, 2022, at 11:28, Lorcan Cooke wrote: > Hi, > > > I would like to subscribe to users@kafka.apache.org please. > > > Regards, > > — > Lorcan

Alex, MM2 is very conservative wrt replicating ACLs. Usually only MM2 is supposed to write to remote topics, so it usually doesn't make sense to replicate WRITE for other identities. Currently we only sync READ to remote topics, and we don't touch non-remote topics. If your use-case requires replicating WRITE permission, you'll need to do it manually. Ryanne On Fri, Apr 15, 2022, 2:40 PM Alex Zuroff < azuroff@gmail.com > wrote: > Hi, > > We are using MirrorMaker 2 (version 2.6.2 - to match our cluster version) > to migrate applications from one cluster to another, and as such, need the > topic ACLs to be the same on both clusters. The ACLs are being replicated, > but the operation is being set to "READ", even if the operation in the > source cluster was "ALL" or "WRITE". > > Here's an example ACL: > > Old cluster - > Current ACLs for resource `ResourcePattern(resourceTy...

Hi, We are using MirrorMaker 2 (version 2.6.2 - to match our cluster version) to migrate applications from one cluster to another, and as such, need the topic ACLs to be the same on both clusters. The ACLs are being replicated, but the operation is being set to "READ", even if the operation in the source cluster was "ALL" or "WRITE". Here's an example ACL: Old cluster - Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=gms-price-logic-detail, patternType=LITERAL)`: (principal=User:CN=MirrorMaker_DEV, host=*, operation=ALL, permissionType=ALLOW) (principal=User:CN=WGS_DEV, host=*, operation=ALL, permissionType=ALLOW) New cluster - Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=gms-price-logic-detail, patternType=LITERAL)`: (principal=User:CN=WGS_DEV, host=*, operation=READ, permissionType=ALLOW) (principal=User:CN=MirrorMaker_DEV, host=*, operation=READ, permissionType=ALLOW) If I manually ...