Kafka

Hi Team, We have a scenario where our application is processing messages at a slower rate. We want the consumer to stop fetching messages from the broker and re-fetch when an application is ready to process again. We have fetch.max.bytes but that doesn't manage the buffer memory. If my understanding is correct then there can be multiple fetch requests and all the messages fetched must be kept in buffer memory and poll() in consumer fetches from this buffer memory. Can we have a way to control this buffer memory? For example, If we keep this memory at 10MB then it can fetch messages and keep them in the buffer but once the buffer is full then it should not fetch more and keep more in memory. Once the poll happens and messages are decreased in the buffer then it can buffer more by fetching again. Thanks, Divya

Hi! I'm considering to use the following configuration for my producers and consumers: client.dns.lookup="use_all_dns_ips" So that I only have a single DNS entry to manage the list of brokers. I also have another cluster connected via MirrorMaker 2, which serves as a failover. My question is, if I was to change the DNS entry to point to the 2nd cluster, is there a delay before clients pick up the new A Records due to caching? If yes, can caching be disabled? Thank you.

Hi Luke, Many thanks for your continued work on this release! To verify, I: - Built from source using Java 11 with both: - - the 3.4.1-rc3 tag on GitHub - - the kafka-3.4.1-src.tgz artifact from https://home.apache.org/~showuon/kafka-3.4.1-rc3/ - Checked signatures and checksums - Ran the quickstart using the kafka_2.13-3.4.1.tgz artifact from https://home.apache.org/~showuon/kafka-3.4.1-rc3/ with Java 11 and Scala 13 in KRaft mode - Ran all unit tests - Ran all integration tests for Connect and MM2 +1 (binding) Cheers, Chris On Tue, May 30, 2023 at 11:16 AM Mickael Maison < mickael.maison@gmail.com > wrote: > Hi Luke, > > I built from source with Java 11 and Scala 2.13 and ran the unit and > integration tests. It took a few retries to get some of them to pass. > I verified signatures and hashes and also ran the zookeeper quickstart. > > +1 (binding) > > Thanks, > Mickael > > On Sat, May 27, 2023 at ...

Hello ! I am working on an exactly once stream processors in Python, using aiokafka client library. My program stores a state in memory, that is recovered from a changelog topic, like in kafka streams. On each processing loop, I am consuming messages, producing messages to an output topics and to my changelog topic, within a transaction. When I need to restart a runner, to restore the state in memory, I have a routine consuming the changelog topic from the beginning to the "end" with a read_commited isolation level. Here I am struggling to define when to stop my recovery : * my current (maybe) working solution is to loop over "poll" until poll is not returning any messages anymore * I tried to do more something based on the end offests, the checking the consumer position, but with control messages at the end of the partition, I am running into an issue where position is one below end offsets, and doesn't go further I had a quick look to ...

Hi Luke, I built from source with Java 11 and Scala 2.13 and ran the unit and integration tests. It took a few retries to get some of them to pass. I verified signatures and hashes and also ran the zookeeper quickstart. +1 (binding) Thanks, Mickael On Sat, May 27, 2023 at 12:58 PM Jakub Scholz < jakub@scholz.cz > wrote: > > +1 (non-binding) ... I used the staged binaries and Maven artifacts to run > my tests and all seems to work fine. > > Thanks for running the release. > > Jakub > > On Fri, May 26, 2023 at 9:34 AM Luke Chen < showuon@gmail.com > wrote: > > > Hello Kafka users, developers and client-developers, > > > > This is the 4th candidate for release of Apache Kafka 3.4.1. > > > > This is a bugfix release with several fixes since the release of 3.4.0. A > > few of the major issues include: > > - core > > KAFKA-14644 < https://issues.apache.org/jira/browse...

Hi Edward, if you use partition number and/or partition offset you are potentially building up a technical debt and tightly coupling the pipeline. What if you want to change the numbers of partitions in the future? Also there is what you mentioned about moving to another Kafka cluster where you have no control over setting the initial offset number. If you need to sink data from Kafka to a DB, the most robust way is via a sink connector (if you care about EOS). There are several available: PostgreSQL, Cassandra, Mongodb, etc. As for the "queriability" on the DB side, I'd recommend designing your system to use the data in the payload/event themselves. Or to make it simple have it indexed by the event key and timestamp. I know timestamp can have different meaning in Kafka, but as long as the same key is coming from the same producer it should be fine to keep the order of the events. In case not, I'd see if there is way to have the timestamp as part of the ...

HI We have a Kafka requirement as per the below scope: *Scope of work for Kafka Cluster and Zookeeper Support:* We would require complete Kafka and zookeeper support to ensure that 100% uptime and no data loss is maintained. Ensuring high performance is assured. On few instances there might be cases where data consumption is not happening both the teams (customer application team and Kafka support team) need to work together in getting it resolved. Kafka Upgrades, scale and optimize. Review of production upgrade plans and provide guidance to maintain the SLAs. Audit & Preventative Maintenance. Assessment of reliability, scalability, latency, throughput, monitoring, geo-replication, log management, and hardware. Security, Meet all of the security goals, including authorization, authentication, and encryption. All stages of the data pipeline including app/producer generation of data, transformation, and loading. Resolve upgrade issues if any Support Zookeeper ...

Oh, the Kafka parallel consumer may help potentially? https://www.instaclustr.com/blog/improving-apache-kafka-performance-and-scalability-with-the-parallel-consumer-part-2/ Paul From: Edvard Fagerholm < edvard.fagerholm@gmail.com > Date: Tuesday, 30 May 2023 at 6:55 am To: users@kafka.apache.org < users@kafka.apache.org > Subject: Patterns for generating ordered streams NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi there, Kafka is new to us, so we don't have operational experience from it. I'm building a system that would be broadcasting events to groups of clients. Kafka consumers handle the broadcasting. In other words, kafka topic --- consumers --- clients We need to maintain the order of the events as they come through Kafka and clients need to be able to retrieve from a DB any events posted while they were offline. This m...

Hi Edvard, interesting problem – I've had similar problems with high fan out use cases, but only for demo applications where I'm more interested in scale than order – e.g. have a look at this list of blogs, examples include Anomalia Machina for Kafka+Cassandra, and Kongo, Kafka+IoT. https://www.linkedin.com/pulse/complete-guide-apache-kafka-developers-everything-i-know-paul-brebner/ Regards, Paul From: Edvard Fagerholm < edvard.fagerholm@gmail.com > Date: Tuesday, 30 May 2023 at 6:55 am To: users@kafka.apache.org < users@kafka.apache.org > Subject: Patterns for generating ordered streams NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi there, Kafka is new to us, so we don't have operational experience from it. I'm building a system that would be broadcasting events to groups of clients. Kafka consumers handle the broadcasting. ...

Hi there, Kafka is new to us, so we don't have operational experience from it. I'm building a system that would be broadcasting events to groups of clients. Kafka consumers handle the broadcasting. In other words, kafka topic --- consumers --- clients We need to maintain the order of the events as they come through Kafka and clients need to be able to retrieve from a DB any events posted while they were offline. This means that the consumer also writes each event to a DB. The number of clients is in the hundreds of millions. What I'm trying to understand here is what I can use to sort the events in the DB. A natural candidate would be to include the Kafka partition offset as a range key in the DB and when doing queries use it for sorting as it would guarantee the same order. Timestamps from the consumer I would not use, since they aren't idempotent and a consumer could fail before acking its last batch. A problem with the partition offset that I...

When I set fetch.max.wait.ms to greater than one minute, I don't pull any data after more than one minute that's my config

+1 (non-binding) ... I used the staged binaries and Maven artifacts to run my tests and all seems to work fine. Thanks for running the release. Jakub On Fri, May 26, 2023 at 9:34 AM Luke Chen < showuon@gmail.com > wrote: > Hello Kafka users, developers and client-developers, > > This is the 4th candidate for release of Apache Kafka 3.4.1. > > This is a bugfix release with several fixes since the release of 3.4.0. A > few of the major issues include: > - core > KAFKA-14644 < https://issues.apache.org/jira/browse/KAFKA-14644 > Process > should stop after failure in raft IO thread > KAFKA-14946 < https://issues.apache.org/jira/browse/KAFKA-14946 > KRaft > controller node shutting down while renouncing leadership > KAFKA-14887 < https://issues.apache.org/jira/browse/KAFKA-14887 > ZK session > timeout can cause broker to shutdown > - client > KAFKA-14639 < https://issues.apache.org/jira/browse...

Hi Luke, - Checked signature, checksum, libs, and license file - Build from source with Java 17 and Scala 2.13 - Ran full unit and integration test suites - Ran Java app with staging Maven artifacts against ZK and KRaft multi-node clusters +1 (non binding) Thanks Fede On Fri, May 26, 2023 at 1:42 PM Josep Prat <josep.prat@aiven.io.invalid> wrote: > > Hi Luke, > > Thanks again for running this! > > I ran the following validation steps: > - Built from source with Java 11 and Scala 2.13 > - Signatures and hashes of the artifacts generated > - Navigated through Javadoc including links to JDK classes > - Run the unit tests > - Run integration tests > - Run the quickstart in KRaft and Zookeeper mode > -- For KRaft, I looked at the process running Kafka and confirmed that the > spamming log message is not present anymore ("Generated a metadata delta > between...") > - Checked that the diff in LI...

Hi Luke, Thanks again for running this! I ran the following validation steps: - Built from source with Java 11 and Scala 2.13 - Signatures and hashes of the artifacts generated - Navigated through Javadoc including links to JDK classes - Run the unit tests - Run integration tests - Run the quickstart in KRaft and Zookeeper mode -- For KRaft, I looked at the process running Kafka and confirmed that the spamming log message is not present anymore ("Generated a metadata delta between...") - Checked that the diff in LICENSE-binary is identical than previous RC -- For each difference of the previous one I checked the binary on the scala 2.13 package had the version specified in the file This gets a +1 from me. Best On Fri, May 26, 2023 at 9:33 AM Luke Chen < showuon@gmail.com > wrote: > Hello Kafka users, developers and client-developers, > > This is the 4th candidate for release of Apache Kafka 3.4.1. > > This is a bugfix rel...

Hello! We have a setup where clients are not connecting to Kafka directly, but rather using a middle layer API for both producing and consuming messages. This layer is centrally managed by the same tech team that is operating Kafka, so we don't see a use for the Kafka client quotas mechanism. At the same time, we see that on each of our Kafka brokers a lot of G1GC Old Generation heap space is used by the Sensor[1] objects with inactiveSensorExpirationTimeMs=3_600_000 (=1 hour), which are apparently used for throughput calculation for enforcing quotas. In a heap dump we find millions of such objects with the total retained heap size in the order of magnitude of more than a GB. We know that our API code is responsible for creating consumers more often than a typical client would do, and we've taken some measures to improve it (namely, using a pool of consumers where they are needed for a short time — to be reused later). That change did improve the heap occup...

Hello Kafka users, developers and client-developers, This is the 4th candidate for release of Apache Kafka 3.4.1. This is a bugfix release with several fixes since the release of 3.4.0. A few of the major issues include: - core KAFKA-14644 < https://issues.apache.org/jira/browse/KAFKA-14644 > Process should stop after failure in raft IO thread KAFKA-14946 < https://issues.apache.org/jira/browse/KAFKA-14946 > KRaft controller node shutting down while renouncing leadership KAFKA-14887 < https://issues.apache.org/jira/browse/KAFKA-14887 > ZK session timeout can cause broker to shutdown - client KAFKA-14639 < https://issues.apache.org/jira/browse/KAFKA-14639 > Kafka CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle - connect KAFKA-12558 < https://issues.apache.org/jira/browse/KAFKA-12558 > MM2 may not sync partition offsets correctly KAFKA-14666 < https://issues.apache.org/jira/browse/KAFKA-14666 > MM2 should...

Hello Kafka users, developers and client-developers, This is the 3rd candidate for release of Apache Kafka 3.4.1. This is a bugfix release with several fixes since the release of 3.4.0. A few of the major issues include: - core KAFKA-14644 < https://issues.apache.org/jira/browse/KAFKA-14644 > Process should stop after failure in raft IO thread KAFKA-14946 < https://issues.apache.org/jira/browse/KAFKA-14946 > KRaft controller node shutting down while renouncing leadership KAFKA-14887 < https://issues.apache.org/jira/browse/KAFKA-14887 > ZK session timeout can cause broker to shutdown - client KAFKA-14639 < https://issues.apache.org/jira/browse/KAFKA-14639 > Kafka CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle - connect KAFKA-12558 < https://issues.apache.org/jira/browse/KAFKA-12558 > MM2 may not sync partition offsets correctly KAFKA-14666 < https://issues.apache.org/jira/browse/KAFKA-14666 > MM2 should...

*In what version(s) of Spring for Apache Kafka are you seeing this issue?* <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> <version>2.2.4.RELEASE</version> *Describe the bug* I have enabled spring-kafka in lower version of springboot (2.1.3), below is my configuration spring: kafka: ###########【Kafka集群】########### bootstrap-servers: jaas: enabled: true loginModule: org.apache.kafka.common.security.plain.PlainLoginModule options: username: password: properties: security.protocol: SASL_PLAINTEXT sasl.mechanism: PLAIN producer: retries: 3 # 重试次数 acks: all # 应答级别:多少个分区副本备份完成时向生产者发送ack确认(可选0、1、all/-1) batch-size: 16384 # 批量大小 buffer-memory: 33554432 # 生产端缓冲区大小 # Kafka提供的序列化和反序列化类 key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.a...

Hi Mickael, I just wanted to point out that I think the documentation you recently merged on Kafka site regarding the 3.5.0 version has a problem when it states the version number and the sub-menu that links to previous versions. Left a comment here: https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939 Best, On Tue, May 23, 2023 at 9:29 AM Josep Prat < josep.prat@aiven.io > wrote: > Hi Mickael, > > I can +1 this candidate. I verified the following: > - Built from source with Java 17 and Scala 2.13 > - Signatures and hashes of the artifacts generated > - Navigated through Javadoc including links to JDK classes > - Run the unit tests > - Run integration tests > - Run the quickstart in KRaft and Zookeeper mode > > Best, > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison < mimaison@apache.org > > wrote: > >> Hello Kafka users, developers and client-developers, >> >...

Hello, I'm working with a global employer that is looking to hire someone to take lead on the modernisation of their on-premises environment. This person will need to be based in Singapore or London and must know Python. Consequently, I had hoped that some members of this group may like to discuss I can be reached using "JamesBTobin (at) Gmail (dot) Com". Kind regards, James

Hi Mickael, I can +1 this candidate. I verified the following: - Built from source with Java 17 and Scala 2.13 - Signatures and hashes of the artifacts generated - Navigated through Javadoc including links to JDK classes - Run the unit tests - Run integration tests - Run the quickstart in KRaft and Zookeeper mode Best, On Mon, May 22, 2023 at 5:30 PM Mickael Maison < mimaison@apache.org > wrote: > Hello Kafka users, developers and client-developers, > > This is the first candidate for release of Apache Kafka 3.5.0. Some of the > major features include: > - KIP-710: Full support for distributed mode in dedicated MirrorMaker > 2.0 clusters > - KIP-881: Rack-aware Partition Assignment for Kafka Consumers > - KIP-887: Add ConfigProvider to make use of environment variables > - KIP-889: Versioned State Stores > - KIP-894: Use incrementalAlterConfig for syncing topic configurations > - KIP-900: KRaft kafka-storage.sh API addi...

Hi Vic, Yes, JDK 8 is still supported in kafka 3.x.x. It'll become unsupported after kafka 4.0.0. Thanks. Luke On Tue, May 23, 2023 at 9:56 AM Vic Xu < vic.xu.development@gmail.com > wrote: > Hi Greg > > I know JDK 8 will solve the problem certainly, but I wondering if JDK8 has > been deprecated after Kafka 3? Can I use JDK 8 with Kafka 3.2.1? Thank you. > > > On 2023/05/22 20:55:01 Greg Harris wrote: > > Vic, > > > > While you can certainly try that, I don't know if that will solve the > problem. > > The reason why JDK11 appears relevant in this context is that a class > > was removed between JDK8 and JDK11. I don't know if a replacement > > stack inspector with better performance was added to JDK17 and used > > within log4j2. > > If you were to try to solve this with a JDK version change, a > > downgrade to 8 may solve the problem, since the log4j library woul...

Hi Greg I know JDK 8 will solve the problem certainly, but I wondering if JDK8 has been deprecated after Kafka 3? Can I use JDK 8 with Kafka 3.2.1? Thank you. On 2023/05/22 20:55:01 Greg Harris wrote: > Vic, > > While you can certainly try that, I don't know if that will solve the problem. > The reason why JDK11 appears relevant in this context is that a class > was removed between JDK8 and JDK11. I don't know if a replacement > stack inspector with better performance was added to JDK17 and used > within log4j2. > If you were to try to solve this with a JDK version change, a > downgrade to 8 may solve the problem, since the log4j library would > use a different stack inspector. > > Greg > > On Sun, May 21, 2023 at 11:30 PM Vic Xu < vi...@gmail.com > wrote: > > > > Hi Greg, > > > > I found another possible solution that is upgrade JDK from 11 to 17. Do you recommend this solution...

Hi Alexandre, looks interesting. Would you consider submitting something (related to performance) to the Community over Code Performance Engineering track please? https://www.linkedin.com/pulse/call-papers-2nd-performance-engineering-track-over-code-brebner/ Thanks, Paul From: Alexandre Strapacao Guedes Vianna < asgv@cin.ufpe.br > Date: Tuesday, 23 May 2023 at 9:04 am To: users@kafka.apache.org < users@kafka.apache.org > Subject: Data Stream Processing applications testing NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hey everyone, I wanted to share my latest paper, "A Grey Literature Review on Data Stream Processing Applications Testing," in the Journal of Systems and Software (JSS), Elsevier. This paper provides unique industry insights, addresses the challenges faced in Data Stream Processing (DSP) application testing, and exp...

Hey everyone, I wanted to share my latest paper, "A Grey Literature Review on Data Stream Processing Applications Testing," in the Journal of Systems and Software (JSS), Elsevier. This paper provides unique industry insights, addresses the challenges faced in Data Stream Processing (DSP) application testing, and explores many test approaches and tools. It bridges the gap between academia and industry. For a limited time, you can access the paper for free via the following link: https://authors.elsevier.com/a/1h7LmbKHpCdKp I believe this paper could provide valuable knowledge and support for our community, and I would be delighted to hear any feedback or thoughts you may have after reading it. Happy reading! Best Regards, Alexandre Vianna

Also I'll point it out here so that everyone is aware, but I don't think it necessarily warrants a new RC on it's own: This patch: https://github.com/apache/kafka/pull/13005 is present on trunk/3.5.0, but is not present in 3.4.0/3.4.1-RC1. It is not a regression strictly, since it was already a problem in 3.4.0. Impact: when any MM2 connector starts, it attempts to create a topic. if that topic exists (as it will on every connector restart), it prints an ERROR log with a TopicExistsException. This is annoying and causes ERROR log noise, but does not otherwise affect the operation or correctness of the connectors. If something else forces a new RC, perhaps consider rolling this into the next RC. Thanks, Greg On Mon, May 22, 2023 at 1:47 PM Greg Harris < greg.harris@aiven.io > wrote: > > Hi Luke, > > I performed a test upgrade of MM2 from 3.4.0 to 3.4.1-RC1, and > verified that the new offset translation logic worked as inten...