Kafka

Given that duplicate tombstones are not a correctness issue, the bug is not considered critical. Of course, we want to merge the fix, but there was no capacity so far to review the PR. But given your inquire about it, its priority was just bumped up :) Sorry of the inconvenience, but Kafka is a very busy project and sometimes it takes much longer to get fixes done due to limited capacity on the committer side to review and merge PR :( -Matthias On 8/30/24 1:36 AM, Vogel, Kevin, DP EPS, BN, extern, external wrote: > Yes, that's it. Thank you. Unfortunately, the issue is not fixed yet. There is a PR from March '24. This bug seems urgent to me. Are there any plans about merging the PR? > > -Kevin > > -----Ursprüngliche Nachricht----- > Von: Matthias J. Sax < mjsax@apache.org > > Gesendet: Freitag, 30. August 2024 04:19 > An: users@kafka.apache.org > Betreff: Re: Unexpected Tombstone records after kafka stream...

yes. thank you. this answers my question. From: Josep Prat < josep.prat@aiven.io > Date: Friday, August 30, 2024 at 8:38 AM To: users@kafka.apache.org < users@kafka.apache.org > Cc: Tim Carroll < TCarroll@perforce.com > Subject: Re: community supported releases You don't often get email from josep.prat@aiven.io . Learn why this is important< https://aka.ms/LearnAboutSenderIdentification > Hi Tim, I replied to your question on the previous email you sent. Didn't you receive it? Hi Tim, Thanks for your interest in Apache Kafka. The page you mention is still contains valid information regarding the End of Life cycle: Apache Kafka maintains the latest 3 versions. Currently, this means 3.6.2, 3.7.1 and 3.8.0. In a few weeks we hope to have released 3.9.0, which would then imply that 3.6.2 has reached EOL. Apache Kafka doesn't use GitHub for releases, but rather the ASF mechanisms, under kafka.apache.org/downloads < http://k...

Hi Tim, I replied to your question on the previous email you sent. Didn't you receive it? *Hi Tim,* *Thanks for your interest in Apache Kafka.* *The page you mention is still contains valid information regarding the End of Life cycle: Apache Kafka maintains the latest 3 versions. Currently, this means 3.6.2, 3.7.1 and 3.8.0.* *In a few weeks we hope to have released 3.9.0, which would then imply that 3.6.2 has reached EOL.* *Apache Kafka doesn't use GitHub for releases, but rather the ASF mechanisms, under kafka.apache.org/downloads < http://kafka.apache.org/downloads > you can find all different releases. In there you can find the 3 actively maintained releases and older ones (EOL) under the archive section.* *Hopefully this helps. Let me know if you have further questions.* Best, On Fri, Aug 30, 2024 at 2:17 PM Tim Carroll <TCarroll@perforce.com.invalid> wrote: > hi all. > > can someone point me to a reference tha...

Dear Team, Subject: Assistance Request: Connecting Oracle Database with Apache Kafka for Bidirectional Replication with PostgreSQL Dear [Recipient's Name], I hope this email finds you well. I am reaching out to request your assistance with setting up a connection between an Oracle database and Apache Kafka using Debezium CDC (Change Data Capture) for real-time data streaming. My goal is to establish bidirectional replication between the Oracle database and a PostgreSQL database on a Windows environment. The replication process involves using Kafka Connect with JDBC connectors to facilitate seamless data flow between the two databases. Specifically, I need guidance on configuring and deploying the necessary components, including the Debezium Oracle connector for capturing changes from the Oracle database, the Kafka Connect framework for data streaming, and the JDBC connector to replicate data to the PostgreSQL database. Additionally, I am interested in underst...

Yes, that's it. Thank you. Unfortunately, the issue is not fixed yet. There is a PR from March '24. This bug seems urgent to me. Are there any plans about merging the PR? -Kevin -----Ursprüngliche Nachricht----- Von: Matthias J. Sax < mjsax@apache.org > Gesendet: Freitag, 30. August 2024 04:19 An: users@kafka.apache.org Betreff: Re: Unexpected Tombstone records after kafka streams update 3.7.1 Sounds like https://issues.apache.org/jira/browse/KAFKA-16394 -Matthias On 8/29/24 02:35, Vogel, Kevin, DP EPS, BN, extern, external wrote: > Hello there, > > I searched the Apache Jira for a bug report on this topic but couldn't find one. Maybe anyone else has noticed something similar or knows more about this. > > After we updated our Spring Boot Kafka Streams application kafka-streams dependency from 3.6.2 to 3.7.1, we noticed some failing tests. The expected behavior of the streams-processor is, to join two input topics togethe...

Hello Akash, I will try by changing compression type and test if the issue persists, but for now the issue is resolved I guess. I just changed replication_factor of the topic to 3. It was like a 3-node kafka cluster and replication_factor was configured 2 for the topic. On Wed, Aug 28, 2024 at 2:55 PM Akash Jain < akashjain0802@gmail.com > wrote: > Also what is the version of Java you are running? > > On Wed, Aug 28, 2024 at 11:44 AM Akash Jain < akashjain0802@gmail.com > > wrote: > > > Hi Vikram, please share some more details: > > > > 1. producer version > > 2. broker version > > 3. broker side config you mentioned is "snappy", can you check what is > > it on topic level as well > > 4. do you face this issue when you use any other algorithm instead of > > snappy, say lz4? Try with topic compression.type set to producer and > > producer compression.t...

Sounds like https://issues.apache.org/jira/browse/KAFKA-16394 -Matthias On 8/29/24 02:35, Vogel, Kevin, DP EPS, BN, extern, external wrote: > Hello there, > > I searched the Apache Jira for a bug report on this topic but couldn't find one. Maybe anyone else has noticed something similar or knows more about this. > > After we updated our Spring Boot Kafka Streams application kafka-streams dependency from 3.6.2 to 3.7.1, we noticed some failing tests. The expected behavior of the streams-processor is, to join two input topics together and produce a single output record. Then the test injects a third record with a null value and expects another output record with a null value. But since the update we are getting two records with null value instead of one. > > I stripped down the processor to the absolute minimum, so you have some example code. Don't worry about the strange names. We are working with a legacy system and are unfortunately bound ...

Hi Tim, Thanks for your interest in Apache Kafka. The page you mention is still contains valid information regarding the End of Life cycle: Apache Kafka maintains the latest 3 versions. Currently, this means 3.6.2, 3.7.1 and 3.8.0. In a few weeks we hope to have released 3.9.0, which would then imply that 3.6.2 has reached EOL. Apache Kafka doesn't use GitHub for releases, but rather the ASF mechanisms, under kafka.apache.org/downloads you can find all different releases. In there you can find the 3 actively maintained releases and older ones (EOL) under the archive section. Hopefully this helps. Let me know if you have further questions. Best, ------------------ Josep Prat Open Source Engineering Director, Aiven josep.prat@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B ...

hi all. can someone point me to a reference that describes the apache kafka release lifecycle and/or a page that has EoL dates for various versions? i'm looking for information on what versions of the product are currently supported by the community, and dates/information that outlines when that support will expire. * it looks like this page is dated - https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan#TimeBasedReleasePlan-WhatIsOurEOLPolicy * releases are not tracked using github devices - https://github.com/apache/kafka/releases * there are release tags - https://github.com/apache/kafka/tags , but there isn't any associated lifecycle attribution. any insights much appreciated. thanks, tim This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.

hi all. can someone point me to a reference that describes the apache kafka release lifecycle and/or a page that has EoL dates for various versions? i'm looking for information on what versions of the product are currently supported by the community, and dates/information that outlines when that support will expire. * it looks like this page is dated - https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan#TimeBasedReleasePlan-WhatIsOurEOLPolicy * releases are not tracked using github devices - https://github.com/apache/kafka/releases * there are release tags - https://github.com/apache/kafka/tags , but there isn't any associated lifecycle attribution. any insights much appreciated. thanks, tim This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.

Hello Kafka brokers are designed to be dumb i.e. they simply act as a pipe to transmit messages from producer to consumer. They don't have capabilities to selectively filter or perform any analytics/transformation of the messages in flights. In fact, a brokers doesn't ever read the key/value pair in a message (unless compaction is desired in which case it reads the keys). It's a long winded way to say, no, you cannot push filtering to the broker/server. You would either need to do it client-side manually or use some stream processing to consume "raw" messages, perform the transformation/filtering/enrichment and produce them back again into an "enriched" topic. Your consumes can then consume from the "enriched" topic. I am not super familiar but I believe one of the choices of such a stream processing library could be Kafka streams which is provided with Apache Kafka. Note that it would still perform filtering at the clien...

Hi Kafka users, I have a producer consumer setup where I want the consumer to get only a subset of messages based on some conditions. Trying to figure out whether there are any configuration on the Kafka side that we can do to achieve this . Most of the solutions I see is to filter them at the consumer end but I would like to push that up to the Kafka configuration level How do I achieve this ? Sent from my iPhone

Thanks, Luke. Both the static and dynamic options for configuring unclean leader election now work in 3.9, thanks to your help. Dima, if you have only a transient need to do an unclean leader election, using the kafka-leader-election.sh script is still the best way. It avoids the risk for forgetting to turn off unclean leader election afterwards, and it's faster. You should only set the configurations for unclean leader election if you want to leave it on forever. (i.e. you genuinely don't care about losing data, and just always want a leader.) best, Colin On Wed, Aug 28, 2024, at 23:11, Dima Brodsky wrote: > Thank you! This helps alot > > Sent from Mobile > > > On Wed, Aug 28, 2024 at 19:48 Luke Chen < showuon@gmail.com > wrote: > >> Hi Dima, >> >> Before v3.9.0, brokers in KRaft mode didn't honor >> `unclean.leader.election.enable` config. >> So, the static config and dynamic config didn...

Hello there, I searched the Apache Jira for a bug report on this topic but couldn't find one. Maybe anyone else has noticed something similar or knows more about this. After we updated our Spring Boot Kafka Streams application kafka-streams dependency from 3.6.2 to 3.7.1, we noticed some failing tests. The expected behavior of the streams-processor is, to join two input topics together and produce a single output record. Then the test injects a third record with a null value and expects another output record with a null value. But since the update we are getting two records with null value instead of one. I stripped down the processor to the absolute minimum, so you have some example code. Don't worry about the strange names. We are working with a legacy system and are unfortunately bound to them: @Bean public BiFunction< KStream<Gidpf01iKey, Gidpf01iValue>, KTable<String, Gidpf01gAggregate>, KStream<UuidString, Teil>> proces...

[re-send] Hi All,  Trying to avoid the rebalance operation on the kafka connect client (2 node/distributed mode) with Kafka 2.5.1 The connect client side properties were updated from default to higher value.session.timeout.ms =300000 heartbeat.interval.ms =100000 While this helps in avoid rebalances, a node restart (accidental failures) was resulting in continuous rebalancing of tasks whenever a heartbeat is sent from both nodes causing the creation of new generation. (infinite-rebalancing-loop) To mitigate it, tried to set group.initial.rebalance.delay.ms =100000 (on broker side), same as heartbeat . while this helped in preventing the formation of new generation during the start up time (initial group formation)., the chance of  node failures after the initial group formation and leading to the " (infinite-rebalancing-loop)". Could not find the property on the broker side to delay the rebalance for all scenarios and not only during the initial stages. Plea...

Hi All,  Trying to avoid the rebalance operation on the kafka connect client (2 node/distributed mode) with Kafka 2.5.1 The connect client side properties were updated from default to higher value.session.timeout.ms =300000 heartbeat.interval.ms =100000 While this helps in avoid rebalances, a node restart (accidental failures) was resulting in continuous rebalancing of tasks whenever a heartbeat is sent from both nodes causing the creation of new generation. (infinite-rebalancing-loop) To mitigate it, tried to set group.initial.rebalance.delay.ms =100000 (on broker side), same as heartbeat . while this helped in preventing the formation of new generation during the start up time (initial group formation)., the chance of  node failures after the initial group formation and leading to the " (infinite-rebalancing-loop)". Could not find the property on the broker side to delay the rebalance for all scenarios and not only during the initial stages. Please suggest/ ...

Thank you! This helps alot Sent from Mobile On Wed, Aug 28, 2024 at 19:48 Luke Chen < showuon@gmail.com > wrote: > Hi Dima, > > Before v3.9.0, brokers in KRaft mode didn't honor > `unclean.leader.election.enable` config. > So, the static config and dynamic config didn't work. > The only thing works before v3.9.0 is using he admin API or cli to run > `kafka-leader-election.sh` with `unclean` option. > > Thanks. > Luke > > On Thu, Aug 29, 2024 at 10:39 AM Dima Brodsky < ddbrodsky@gmail.com > wrote: > > > Thanks Luke, > > > > I meant in pre 3.9.x, like 3.6.x or 3.7.x branch for example. > > > > ttyl > > Dima > > > > > > On Wed, Aug 28, 2024 at 7:36 PM Luke Chen < showuon@gmail.com > wrote: > > > > > Hi Dima, > > > > > > > I assume that unclean leader election exists as a static config in > > k...

Hi Dima, Before v3.9.0, brokers in KRaft mode didn't honor `unclean.leader.election.enable` config. So, the static config and dynamic config didn't work. The only thing works before v3.9.0 is using he admin API or cli to run `kafka-leader-election.sh` with `unclean` option. Thanks. Luke On Thu, Aug 29, 2024 at 10:39 AM Dima Brodsky < ddbrodsky@gmail.com > wrote: > Thanks Luke, > > I meant in pre 3.9.x, like 3.6.x or 3.7.x branch for example. > > ttyl > Dima > > > On Wed, Aug 28, 2024 at 7:36 PM Luke Chen < showuon@gmail.com > wrote: > > > Hi Dima, > > > > > I assume that unclean leader election exists as a static config in > kafka > > kraft, but is still unavailable dynamically? Or is it unavailable in > > general? > > > > Let me make it clear. In v3.9.0, the `unclean.leader.election.enable` in > > KRaft will work, either static config or dynamic c...

Thanks Luke, I meant in pre 3.9.x, like 3.6.x or 3.7.x branch for example. ttyl Dima On Wed, Aug 28, 2024 at 7:36 PM Luke Chen < showuon@gmail.com > wrote: > Hi Dima, > > > I assume that unclean leader election exists as a static config in kafka > kraft, but is still unavailable dynamically? Or is it unavailable in > general? > > Let me make it clear. In v3.9.0, the `unclean.leader.election.enable` in > KRaft will work, either static config or dynamic config. > It's just the behavior changes when dynamically enabling the > `unclean.leader.election.enable`, which will not trigger unclean leader > election immediately after config change. The detailed behavior change is > as described above. > > Let me know if it's still not clear. > > Thanks. > Luke > > > On Thu, Aug 29, 2024 at 1:58 AM Dima Brodsky < ddbrodsky@gmail.com > wrote: > > > Hi Luke, > > ...

Hi Dima, > I assume that unclean leader election exists as a static config in kafka kraft, but is still unavailable dynamically? Or is it unavailable in general? Let me make it clear. In v3.9.0, the `unclean.leader.election.enable` in KRaft will work, either static config or dynamic config. It's just the behavior changes when dynamically enabling the `unclean.leader.election.enable`, which will not trigger unclean leader election immediately after config change. The detailed behavior change is as described above. Let me know if it's still not clear. Thanks. Luke On Thu, Aug 29, 2024 at 1:58 AM Dima Brodsky < ddbrodsky@gmail.com > wrote: > Hi Luke, > > I assume that unclean leader election exists as a static config in kafka > kraft, but is still unavailable dynamically? Or is it unavailable in > general? > > Thanks! > ttyl > Dima > > > On Wed, Aug 28, 2024 at 3:45 AM Luke Chen < showuon@gmai...

Hi Luke, I assume that unclean leader election exists as a static config in kafka kraft, but is still unavailable dynamically? Or is it unavailable in general? Thanks! ttyl Dima On Wed, Aug 28, 2024 at 3:45 AM Luke Chen < showuon@gmail.com > wrote: > Hi all, > > In v3.9.0, we're planning to support the `unclean.leader.election.enable` > config in KRaft. Currently, the implementation PR > < https://github.com/apache/kafka/pull/16866 > is under review now. But I'd > like to raise the discussion to let you know there will be one behavior > change when enabling `unclean.leader.election.enable` config dynamically in > KRaft. > > scenario: > 1. broker [0, 1] config are all set `unclean.leader.election.enable=false`. > 2. topic A is created with 1 partition, 2 replication factors and > `unclean.leader.election.enable=false`. Leader of the partition 0 is 0. ISR > is [0]. > 3. broker 0 is down, whi...

Well deserved! Best, Bruno On 8/28/24 6:37 PM, Bill Bejeck wrote: > Congrats Lianet! > > On Wed, Aug 28, 2024 at 12:32 PM Matthias J. Sax < mjsax@apache.org > wrote: > >> Congrats! Very well deserved! >> >> On 8/28/24 9:18 AM, Justine Olshan wrote: >>> Congratulations Lianet!! >>> Justine >>> >>> On Wed, Aug 28, 2024 at 8:58 AM David Arthur < mumrah@gmail.com > wrote: >>> >>>> Congrats, Lianet! >>>> >>>> On Wed, Aug 28, 2024 at 11:48 AM Mickael Maison < >> mickael.maison@gmail.com > >>>> wrote: >>>> >>>>> Congratulations Lianet! >>>>> >>>>> On Wed, Aug 28, 2024 at 5:40 PM Josep Prat <josep.prat@aiven.io.invalid >>> >>>>> wrote: >>>>>> >>>>>> Congrats Lianet! >>>>>> >...

Congrats Lianet! On Wed, Aug 28, 2024 at 12:32 PM Matthias J. Sax < mjsax@apache.org > wrote: > Congrats! Very well deserved! > > On 8/28/24 9:18 AM, Justine Olshan wrote: > > Congratulations Lianet!! > > Justine > > > > On Wed, Aug 28, 2024 at 8:58 AM David Arthur < mumrah@gmail.com > wrote: > > > >> Congrats, Lianet! > >> > >> On Wed, Aug 28, 2024 at 11:48 AM Mickael Maison < > mickael.maison@gmail.com > > >> wrote: > >> > >>> Congratulations Lianet! > >>> > >>> On Wed, Aug 28, 2024 at 5:40 PM Josep Prat <josep.prat@aiven.io.invalid > > > >>> wrote: > >>>> > >>>> Congrats Lianet! > >>>> > >>>> On Wed, Aug 28, 2024 at 5:38 PM Chia-Ping Tsai < chia7712@apache.org > > >>> wrote: > >>>> > >>>...

Congrats! Very well deserved! On 8/28/24 9:18 AM, Justine Olshan wrote: > Congratulations Lianet!! > Justine > > On Wed, Aug 28, 2024 at 8:58 AM David Arthur < mumrah@gmail.com > wrote: > >> Congrats, Lianet! >> >> On Wed, Aug 28, 2024 at 11:48 AM Mickael Maison < mickael.maison@gmail.com > >> wrote: >> >>> Congratulations Lianet! >>> >>> On Wed, Aug 28, 2024 at 5:40 PM Josep Prat <josep.prat@aiven.io.invalid> >>> wrote: >>>> >>>> Congrats Lianet! >>>> >>>> On Wed, Aug 28, 2024 at 5:38 PM Chia-Ping Tsai < chia7712@apache.org > >>> wrote: >>>> >>>>> Congratulations, Lianet!!! >>>>> >>>>> On 2024/08/28 15:35:23 David Jacot wrote: >>>>>> Hi all, >>>>>> >>>>>> The PMC of Apache Kafka is plea...

Hi all, In v3.9.0, we're planning to support the `unclean.leader.election.enable` config in KRaft. Currently, the implementation PR < https://github.com/apache/kafka/pull/16866 > is under review now. But I'd like to raise the discussion to let you know there will be one behavior change when enabling `unclean.leader.election.enable` config dynamically in KRaft. scenario: 1. broker [0, 1] config are all set `unclean.leader.election.enable=false`. 2. topic A is created with 1 partition, 2 replication factors and `unclean.leader.election.enable=false`. Leader of the partition 0 is 0. ISR is [0]. 3. broker 0 is down, which leaves ISR to empty [ ], and leader none. 4. dynamically alter config using admin API to set `unclean.leader.election.enable=true`, no matter it changes to the topic A, or to broker 1, or default broker. In ZK mode, with this scenario, it'll trigger unclean leader election immediately, and broker 1 will be elected as the leader, wh...

Also what is the version of Java you are running? On Wed, Aug 28, 2024 at 11:44 AM Akash Jain < akashjain0802@gmail.com > wrote: > Hi Vikram, please share some more details: > > 1. producer version > 2. broker version > 3. broker side config you mentioned is "snappy", can you check what is > it on topic level as well > 4. do you face this issue when you use any other algorithm instead of > snappy, say lz4? Try with topic compression.type set to producer and > producer compression.type to anything other than snappy > 5. Do you face this issue when you disable compression? > 6. You also mention that you are facing this issue randomly, can you > elaborate more? Like you have different producer versions - and a > specific producer shows this problem? Or does a specific broker show this > problem? Have you observed any pattern at all? > > Do you see these exceptions in br...

Hi Vikram, please share some more details: 1. producer version 2. broker version 3. broker side config you mentioned is "snappy", can you check what is it on topic level as well 4. do you face this issue when you use any other algorithm instead of snappy, say lz4? Try with topic compression.type set to producer and producer compression.type to anything other than snappy 5. Do you face this issue when you disable compression? 6. You also mention that you are facing this issue randomly, can you elaborate more? Like you have different producer versions - and a specific producer shows this problem? Or does a specific broker show this problem? Have you observed any pattern at all? Do you see these exceptions in broker? I guess yes. Broker will decompress the messages. It is likely because the version of snappy that is used in producer is 'far away' from the version of snappy in broker. On Tue, Aug 27, 2024 at 8:00 A...