Kafka

Hi Alexandre, thanks for your reply. I updated my example to produce a message to the compacted topic ("table") at a lower timestamp, but I'm still not getting the expected result unfortunately. The scenario is now: 1. Message published to A ("input") with timestamp t_0 2. Three seconds of wall clock time elapses 3. Message published to B ("table") with timestamp (t_0 - 1) On Tue, Aug 30, 2022 at 12:15 AM Alexandre Brasil < alexandre.brasil@gmail.com > wrote: > Hi Derek, > > What max.task.idle.ms does is set a wait time for the stream application > to > wait for new messages when one or more > input topics have no messages after a poll. In your case, the application > polls for the first time and finds a message on > topic A ("input") and no messages on topic B ("table"). Since you have > max.task.idle.ms set to 5000, it waits up to five > seconds for messages to arriv...

Hi, I have one topic with 300 partitions. I have 4 KSQL instances with 8 threads each on 8 core machine. Topic has lag of around million records. Can I increase number of threads equal to number of partitions so hat lag will be reduced? Or I have to reduce partitions to match total number of threads? please advise how do I reduce lag on source topic. Thanks

Hi Derek, What max.task.idle.ms does is set a wait time for the stream application to wait for new messages when one or more input topics have no messages after a poll. In your case, the application polls for the first time and finds a message on topic A ("input") and no messages on topic B ("table"). Since you have max.task.idle.ms set to 5000, it waits up to five seconds for messages to arrive on topic B. When you produce your second message three seconds later, the app will process the messages. Kafka Streams will process the messages from both topics in timestamp order, but since both of your messages have the same timestamp, my guess is that it's processing the message from the stream ("input") first and it finds nothing to join to on "table". My guess is that if you tweak the second message timestamp to be lower than the timestamp of the second message you'll get the result you want. Regards, Alexandre O...

unsubscribe On Mon, Aug 29, 2022 at 8:56 AM Derek Mok < derek.mok9201@gmail.com > wrote: > Hi, I'd like some help with understanding how exactly max.task.idle.ms > works. I have a topology that consumes from an input topic A, and a join > operator that enriches the topic A messages with a KTable from a compacted > topic B. The enriched messages are output to topic C. > > If I set max.task.idle.ms to 5000, what should be the expected behaviour > in > the following scenario (assuming all messages have the same key and topics > have same partition count): > > 1. Message published to A with timestamp t_0 > 2. Three seconds of wall clock time elapses > 3. Message published to B with timestamp t_0 > > My understanding is that a message should be output to topic C containing > the enriched result after max.task.idle.ms elapses since it should account > for the late producer of topic B. The join oper...

Hi, I'd like some help with understanding how exactly max.task.idle.ms works. I have a topology that consumes from an input topic A, and a join operator that enriches the topic A messages with a KTable from a compacted topic B. The enriched messages are output to topic C. If I set max.task.idle.ms to 5000, what should be the expected behaviour in the following scenario (assuming all messages have the same key and topics have same partition count): 1. Message published to A with timestamp t_0 2. Three seconds of wall clock time elapses 3. Message published to B with timestamp t_0 My understanding is that a message should be output to topic C containing the enriched result after max.task.idle.ms elapses since it should account for the late producer of topic B. The join operator should only be invoked after max.task.idle.ms has elapsed. However, what actually happens is that nothing is published in topic C. I suspect my understanding of max.task.idle....

Hi Chris, Great to hear you found a way to work around it now. Could you share your solution here in case other people got stuck on the same issue? Thank you Luke On Fri, Aug 26, 2022 at 11:57 PM Chris Peart < chris@peart.me.uk > wrote: > > Thanks Luke, > The problem I was having was that the consumer group was not inactive even > with all the consumers stopped. > Managed to work around this now. > > Many Thanks > Chris > > > On 25 Aug 2022, at 10:12 am, Luke Chen < showuon@gmail.com > wrote: > > Hi Chris, > > > >> is there a way to force the offset forward by an increment of 1 just for > > this topic? > > kafka-consumer-groups.sh script has an option `--shift-by` to shift > current > > offset by 'n'. > > I think this is what you are looking for. > > > > Thank you > > Luke > > > > On Thu, Aug 25, 2022 at 4:25 PM Chris P...

 Thanks Luke, The problem I was having was that the consumer group was not inactive even with all the consumers stopped. Managed to work around this now. Many Thanks Chris > On 25 Aug 2022, at 10:12 am, Luke Chen < showuon@gmail.com > wrote: > Hi Chris, > >> is there a way to force the offset forward by an increment of 1 just for > this topic? > kafka-consumer-groups.sh script has an option `--shift-by` to shift current > offset by 'n'. > I think this is what you are looking for. > > Thank you > Luke > > On Thu, Aug 25, 2022 at 4:25 PM Chris Peart < chris@peart.me.uk > wrote: > >> >> >> Hi All, >> >> I'm trying to reset a kafka offset for a topic in a consumer group, i >> have stopped all the consumers using the consumer group but i'm still >> receiving the message that the current state is stable. >> >> Is there a w...

Are you upgrading your Kafka servers or your applications' client libraries? We upgraded our client libraries and then ran into an incompatibility when the upgraded client ran with an older broker cluster. But the issue was fixed by simply adding an ACL for IdempotentWrite on the brokers. This page was a very valuable explanation of this issue: https://kpow.io/articles/kafka-producer-breaking-change/ -- Patrick Taylor Sr. Principal Software Engineer ptaylor@snaplogic.com Mobile: 303-594-8979 Boulder, Colorado (US Mountain Time) < http://www.snaplogic.com/about-us/jobs > *SnapLogic Inc | 1825 South Grant Street | San Mateo CA | USA * This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system; you may not copy this message or disclose its contents to anyone. The integ...

Hi Chris, > is there a way to force the offset forward by an increment of 1 just for this topic? kafka-consumer-groups.sh script has an option `--shift-by` to shift current offset by 'n'. I think this is what you are looking for. Thank you Luke On Thu, Aug 25, 2022 at 4:25 PM Chris Peart < chris@peart.me.uk > wrote: > > > Hi All, > > I'm trying to reset a kafka offset for a topic in a consumer group, i > have stopped all the consumers using the consumer group but i'm still > receiving the message that the current state is stable. > > Is there a way to put the consumer group to an inactive state after > stopping all the consumers or is there a way to force the offset forward > by an increment of 1 just for this topic? > > Many Thanks, > > Chris

1. Is kafka 2.3.0 going end of life ,If yes then what is the expected date? -> Kafka supports last 3 releases. REF: https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan#TimeBasedReleasePlan-WhatIsOurEOLPolicy ? 2. Is kafka 3.1.0 backward compatible to 2.3.0? -> Since 2.3 to 3.1 has one major release (through 3.0), some deprecated features are removed. You can refer to this doc for upgrade guide: https://kafka.apache.org/documentation/#upgrade_3_1_0 , and check for release note for each release. Thanks Luke On Thu, Aug 25, 2022 at 3:42 PM Fred Bai < piaobomengxiang@gmail.com > wrote: > +1 > Me too, We consider upgrading Kafka to 3.X from Kafka 2.X, but don't know > the compatibility. > > thx > > Ankit Saran < saranankit1511@gmail.com > 于2022年8月23日周二 22:21写道： > > > Hi Team, > > We are planning to upgrade kafka version from 2.3.0 to 3.1.0 , We have > > below queries rega...

Hi All, I'm trying to reset a kafka offset for a topic in a consumer group, i have stopped all the consumers using the consumer group but i'm still receiving the message that the current state is stable. Is there a way to put the consumer group to an inactive state after stopping all the consumers or is there a way to force the offset forward by an increment of 1 just for this topic? Many Thanks, Chris

Hi Sunil Chaudhari : Excuse me, is Kafka 3.1.0 backward compatible to 2.3.0? We also need to upgrade to Kafka3.x from 2.x. thx. sunil chaudhari < sunilmchaudhari05@gmail.com > 于2022年8月16日周二 23:41写道： > Hi Sowjanya, > I am not technical support from Kafka, but I can help you in this. > Recently I upgraded one of the confluent version, so I will try to help > you. > Please let me know what exactly you need. > > On Tue, 16 Aug 2022 at 7:43 PM, sowjanya reddy < > sowjanyabairapureddy7@gmail.com > wrote: > > > I Team, > > > > I need contact details of technical support on kafka upgrade. > > > > > > > > Thanks and regards, > > Sowjanya. > > >

+1 Me too, We consider upgrading Kafka to 3.X from Kafka 2.X, but don't know the compatibility. thx Ankit Saran < saranankit1511@gmail.com > 于2022年8月23日周二 22:21写道： > Hi Team, > We are planning to upgrade kafka version from 2.3.0 to 3.1.0 , We have > below queries regarding the same > > 1. Is kafka 2.3.0 going end of life ,If yes then what is the expected date? > 2. Is kafka 3.1.0 backward compatible to 2.3.0? > > Please help us with the above queries, Thanks in advance. > > Regards, > Ankit Saran >

Hi John, Please find the full stack trace with 3 frames from the app removed that are process/state-store related functions. {"log":"2022-08-17 15:21:32,890 [ERROR] Util: Failed saving key=cx_dev_sys:c3d376b6e62f11ecbf8256eb17370719:SG18KW70HN to stateStore with Exception={}\n\njava.lang.UnsupportedOperationException: this should not happen: timestamp() is not supported in standby tasks.\n\n\tat org.apache.kafka.streams.processor.internals.ProcessorContextImpl.throwUnsupportedOperationExceptionIfStandby(ProcessorContextImpl.java:352)\n\n\tat org.apache.kafka.streams.processor.internals.ProcessorContextImpl.timestamp(ProcessorContextImpl.java:328)\n\n\tat org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.log(ChangeLoggingKeyValueBytesStore.java:136)\n\n\tat org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:78)\n\n\tat org.apache.kafka.streams.state.internals.ChangeLogging...

Hi Suresh, Sorry for the trouble! Are you able to provide the rest of the stack trace? It shouldn't be possible to call put() on a store in a standby task, so we need to see the stack frames that show what is calling it. Thanks, John On Tue, Aug 23, 2022, at 05:08, Suresh Rukmangathan wrote: > Hi, > > We have a Kafka streams application with "num.standby.replicas" = 1 and > with replication-factor of 2 for that topic. The application is crashing > with the following stack trace. > > java.lang.UnsupportedOperationException: this should not happen: > timestamp() is not supported in standby tasks.\n\n\tat > > org.apache.kafka.streams.processor.internals.ProcessorContextImpl.throwUnsupportedOperationExceptionIfStandby(ProcessorContextImpl.java:352)\n\n\tat > > org.apache.kafka.streams.processor.internals.ProcessorContextImpl.timestamp(ProcessorContextImpl.java:328)\n\n\tat > > org.apache.kafka.str...

Hi, We have a Kafka streams application with "num.standby.replicas" = 1 and with replication-factor of 2 for that topic. The application is crashing with the following stack trace. java.lang.UnsupportedOperationException: this should not happen: timestamp() is not supported in standby tasks.\n\n\tat org.apache.kafka.streams.processor.internals.ProcessorContextImpl.throwUnsupportedOperationExceptionIfStandby(ProcessorContextImpl.java:352)\n\n\tat org.apache.kafka.streams.processor.internals.ProcessorContextImpl.timestamp(ProcessorContextImpl.java:328)\n\n\tat org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.log(ChangeLoggingKeyValueBytesStore.java:136)\n\n\tat org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:78)\n\n\tat org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:32)\n\n\tat org.apache.kafka.streams...

Hi Team, We are planning to upgrade kafka version from 2.3.0 to 3.1.0 , We have below queries regarding the same 1. Is kafka 2.3.0 going end of life ,If yes then what is the expected date? 2. Is kafka 3.1.0 backward compatible to 2.3.0? Please help us with the above queries, Thanks in advance. Regards, Ankit Saran

Hi Raj, I think for your proble Rabbitmq RPC usecase is more suitable. By the way its not recommended to make such tight couple between 2 services if it's possible. Regards On Fri, Jul 29, 2022, 12:06 PM Richard Bosch < richard.bosch@axual.com > wrote: > Hi Raj, > > The Kafka design is based on fully decoupled producers and consumers, where > the responsibility of the producer ends after a record has been > successfully produced. > Since a producer is not aware of other applications that might be consuming > the record it can continue with its own flow. > The same is true for a consumer, they are usually not aware which > application is producing a record. only where to read a record and how to > deserialize and interpret the record. > This allows for new applications to be added as producers or consumers of > the data. > > This is one of the reasons why Kafka does not contain a builtin feature to > send an...

Hi, Are you using multiple MM2 instances? If yes, there is a known issue around that, the Connect REST API is missing from MM2 instances, causing clusters to fail. There is a dormant KIP trying to fix it: https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters Daniel 2022. 08. 20. 0:21 keltezéssel, An, Hongguo (CORP) írta: > Hi: > My MM2 got this error after it starts for a while and then the logs are flooded with this error, what does it mean and how should I fix it? > > > Failed to reconfigure connector's tasks (MirrorCheckpointConnector), retrying after backoff > > Thanks > Andrew > > > This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended ...

Hi: My MM2 got this error after it starts for a while and then the logs are flooded with this error, what does it mean and how should I fix it? Failed to reconfigure connector's tasks (MirrorCheckpointConnector), retrying after backoff Thanks Andrew This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

Hi Sophie, thanks for your reply, im a big fan of your videos. About the issue the logger comes from > org.apache.kafka.clients.consumer.internals.AbstractCoordinator After the message I don't see any other log or exception, just the stream stop consuming messages. As you mention yea looks like the heartbeat thread is stuck. I already debug my topology and I was not able to reproduce the issue on my local, but this is happening so frequently on production. Our application have multiple streams running in the same app with different > StreamsConfig.APPLICATION_ID_CONFIG Like myapp-stream1 myapp-stream2 Myapp-stream3 The issue happens so randomly, some times to one stream other times to another. But for example when it happen to myapp-stream2, other streams continue working good, in some cases happen to only one, other times to two or more streams. Our topology contains a multiple branches where depending on some conditions we delegate to differen...

Well it sounds like your app is getting stuck somewhere in the poll loop so it's unable to call poll again within the session timeout, as the error message indicates -- it's a bit misleading as it says "Sending LeaveGroup request to coordinator" which implies it's *currently* sending the LeaveGroup, but IIRC this error actually comes from the heartbeat thread -- just a long way of clarifying that the reason you don't see the state go into REBALANCING is because the StreamThread is stuck and can't rejoin the group by calling #poll So...what now? I know your question was how to detect this, but I would recommend first trying to take a look into your application topology to figure out where, and *why*, it's getting stuck (sorry for the "q: how do I do X? answ. don't do X, do Y" StackOverflow-type response -- happy to help with that if we really can't resolve the underlying issue, I'll give it some thought since I...

Hey Peter Try clearing the local state -- if you have stateful tasks then by default Streams will use rocksdb to store records locally in directories specific to/named after that task. This is presumably why you're seeing errors related to "the task for peek node missing in old nodes" You can delete the local state manually, which is generally recommended, or by running `KafkaStreams#cleanup` -- you just want to make sure to remove the #cleanup before you resume running normally, in case of a restart for whatever reason (since wiping out the local state can take a while to restore from the changelogs) Of course, just wiping out the local state alone will get you past any problems related to that, but you might have to fully reset the application (or might just have lost your current data/state) if eg the changelog or repartition topic names changed. In general upgrading topologies is hard to do in a compatible way, but there are a few tricks/feat...

Hello, I am looking for help regarding topologies and tasks in KS. Mostly where this information is stored outside of KS app. My case is upgrading the KS topology from v1 to v2, the topology is different (f.e adding peek node in v2). I made a change in code, prepared a jar and deployed. The issue that I observed was when old and new versions of app are up and running (by mistake). The task for peek node is missing in old nodes hence deployment crashes. My next steps were shutting down all nodes and deployment without old nodes. But this did not help. New deployment crashed too and I was not able to figure out why. The topology just stopped ingestion from the source topic. No error in logs. My hypothesis is that the former deployment of (KS topology v1 and v2 at once) leaves some sort of ephemeral state in the kafka cluster itself. And here is my question. what kind of state except messages in repartition and changelog topics, consumer offsets are stored by...

Thank you Sunil ,Peter Raph and Richard for your kind inputs.Much appreciated. On Wed, Aug 17, 2022 at 6:46 AM sunil chaudhari < sunilmchaudhari05@gmail.com > wrote: > You can try this, if you know what prometheus and how its installed > configured. > > > https://www.confluent.io/blog/monitor-kafka-clusters-with-prometheus-grafana-and-confluent/ > > > On Wed, 17 Aug 2022 at 2:25 AM, Peter Bukowinski < pmbuko@gmail.com > wrote: > > > Richard recently answered your query. A kafka cluster does not keep track > > of lag on behalf of external consumers and it therefore is not available > in > > JMX. This is why tools like Burrow were written. The java kafka consumer > > published consumer lag metrics, and perhaps some other third-party > clients > > do, as well. > > > > > On Aug 16, 2022, at 12:05 PM, Kafka Life < lifekafka999@gmail.com > > wrote: > > > ...

You can try this, if you know what prometheus and how its installed configured. https://www.confluent.io/blog/monitor-kafka-clusters-with-prometheus-grafana-and-confluent/ On Wed, 17 Aug 2022 at 2:25 AM, Peter Bukowinski < pmbuko@gmail.com > wrote: > Richard recently answered your query. A kafka cluster does not keep track > of lag on behalf of external consumers and it therefore is not available in > JMX. This is why tools like Burrow were written. The java kafka consumer > published consumer lag metrics, and perhaps some other third-party clients > do, as well. > > > On Aug 16, 2022, at 12:05 PM, Kafka Life < lifekafka999@gmail.com > wrote: > > > > Hello Experts, Any info or pointers on my query please. > > > > > > > > On Mon, Aug 15, 2022 at 11:36 PM Kafka Life < lifekafka999@gmail.com > > wrote: > > > >> Dear Kafka Experts > >> we need to monitor the...

Richard recently answered your query. A kafka cluster does not keep track of lag on behalf of external consumers and it therefore is not available in JMX. This is why tools like Burrow were written. The java kafka consumer published consumer lag metrics, and perhaps some other third-party clients do, as well. > On Aug 16, 2022, at 12:05 PM, Kafka Life < lifekafka999@gmail.com > wrote: > > Hello Experts, Any info or pointers on my query please. > > > > On Mon, Aug 15, 2022 at 11:36 PM Kafka Life < lifekafka999@gmail.com > wrote: > >> Dear Kafka Experts >> we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0 >> versions of kafka in Grafana. >> >> 1/ What is the correct path for JMX metrics to evaluate Consumer Lag in >> kafka cluster. >> >> 2/ I had thought it is FetcherLag but it looks like it is not as per the >> link below. >> >> https://...

Hi Guys, I'm having an issue with a kafka stream app, at some point I get a consumer leave group message. Exactly same issue described to another person here: https://stackoverflow.com/questions/61245480/how-to-detect-a-kafka-streams-app-in-zombie-state But the issue is that stream state is continuing reporting that the stream is running, but it's not consuming anything, but the stream never rejoin the consumer group, so my application with only one replica stop consuming. I have a health check on Kubernetes where I expose the stream state to then restart the pod. But as the kafka stream state it's always running when the consumer leaves the group, the app is still healthy in zombie state, so I need to manually go and restart the pod. Is this a bug? Or is there a way to check what is the stream consumer state to then expose as healt check for my application? This issue really happen randomly, usually all the Mondays. I'm using Kafka 2.8.1 and m...

Hello Experts, Any info or pointers on my query please. On Mon, Aug 15, 2022 at 11:36 PM Kafka Life < lifekafka999@gmail.com > wrote: > Dear Kafka Experts > we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0 > versions of kafka in Grafana. > > 1/ What is the correct path for JMX metrics to evaluate Consumer Lag in > kafka cluster. > > 2/ I had thought it is FetcherLag but it looks like it is not as per the > link below. > > https://www.instaclustr.com/support/documentation/kafka/monitoring-information/fetcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metric%20aggregates%20lag,in%20sync%20with%20partitions%20that%20it%20is%20replicating > . > > Could one of you experts please guide on which JMX i should use for > consumer lag apart from kafka burrow or such intermediate tools > > Thanking you in advance > >

Hi Sowjanya, I am not technical support from Kafka, but I can help you in this. Recently I upgraded one of the confluent version, so I will try to help you. Please let me know what exactly you need. On Tue, 16 Aug 2022 at 7:43 PM, sowjanya reddy < sowjanyabairapureddy7@gmail.com > wrote: > I Team, > > I need contact details of technical support on kafka upgrade. > > > > Thanks and regards, > Sowjanya. >

Hi, I don't think that the Kafka Broker exposes the current lag as a metric. It's the reason most of those intermediate tools, like Kafka Exporter, exist. The lag information is available to the client itself, but that is based on the received topic metadata and can be out of date. Kind regards, Richard Bosch Developer Advocate Axual BV https://axual.com/ On Mon, Aug 15, 2022 at 8:07 PM Kafka Life < lifekafka999@gmail.com > wrote: > Dear Kafka Experts > we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0 > versions of kafka in Grafana. > > 1/ What is the correct path for JMX metrics to evaluate Consumer Lag in > kafka cluster. > > 2/ I had thought it is FetcherLag but it looks like it is not as per the > link below. > > https://www.instaclustr.com/support/documentation/kafka/monitoring-information/fetcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metr...

Replying to myself I ended by making a small python utility that do the work. On 12/08/2022 12:50, Raphael Mazelier wrote: > Hi kafka community, > > I've been an happy user of kafka since years. Today I faced a problem > that begins to make mad... > > So we have the project to migrate from and old (and unmonitored, > etc..) kafka cluster to multiple new fresh clusters. Great. > > So the plan was clear, identify source topics and source groups, use > mirromarker2 with whitelist configuration and migrate consumer by > consumer on the fresh cluster, then producer. > > For that we need that consumer  group should be synced at some point > (if not every consumer will restart from its own strategy). > > That were the problem begins. Mirrormaker2 seems to works well > replicating topics, but on consumer it's very erratic to say the > least. My temporary conclusion is that mm2 only "replicate...

I Team, I need contact details of technical support on kafka upgrade. Thanks and regards, Sowjanya.

Dear Kafka Experts we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0 versions of kafka in Grafana. 1/ What is the correct path for JMX metrics to evaluate Consumer Lag in kafka cluster. 2/ I had thought it is FetcherLag but it looks like it is not as per the link below. https://www.instaclustr.com/support/documentation/kafka/monitoring-information/fetcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metric%20aggregates%20lag,in%20sync%20with%20partitions%20that%20it%20is%20replicating . Could one of you experts please guide on which JMX i should use for consumer lag apart from kafka burrow or such intermediate tools Thanking you in advance

Hi kafka community, I've been an happy user of kafka since years. Today I faced a problem that begins to make mad... So we have the project to migrate from and old (and unmonitored, etc..) kafka cluster to multiple new fresh clusters. Great. So the plan was clear, identify source topics and source groups, use mirromarker2 with whitelist configuration and migrate consumer by consumer on the fresh cluster, then producer. For that we need that consumer  group should be synced at some point (if not every consumer will restart from its own strategy). That were the problem begins. Mirrormaker2 seems to works well replicating topics, but on consumer it's very erratic to say the least. My temporary conclusion is that mm2 only "replicate" empty consumer group (which can make sense?), but inspecting the offset show me that they are not sync. I tried to dig into config/log and I never find something that works reliably. (btw the document...

Hi, I made a question on StackOverflow about this subject but did not receive any advice. Could you look at it? https://stackoverflow.com/questions/73309760/consumer-not-polling-any-messages-with-random-group-id

thank you. i will try fluent-bit for now. Raphael Mazelier wrote: > maybe you can you try another log shipper to validate this assumption; > there are plenty and fluentd is already an old (and buggy and slow one); > you can try fluent-bit or vector maybe. >

Hi, this seems a fluentd issue not a kafka one. maybe you can you try another log shipper to validate this assumption; there are plenty and fluentd is already an old (and buggy and slow one); you can try fluent-bit or vector maybe. -- Raphael Mazelier On 12/08/2022 04:58, pengyh wrote: > do you konw if the current fluentd supports kafka 3 (my version is > 3.2.1)? > > fluentd dry-run outputs: > > 2022-08-12 10:47:24 +0800 [info]: gem > 'fluent-plugin-calyptia-monitoring' version '0.1.3' > 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-elasticsearch' > version '5.2.3' > 2022-08-12 10:47:24 +0800 [info]: gem > 'fluent-plugin-flowcounter-simple' version '0.1.0' > 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-kafka' version > '0.18.0' > 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-metrics-cmetrics' > version '0.1....

do you konw if the current fluentd supports kafka 3 (my version is 3.2.1)? fluentd dry-run outputs: 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-calyptia-monitoring' version '0.1.3' 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.3' 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0' 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-kafka' version '0.18.0' 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-metrics-cmetrics' version '0.1.2' 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-opensearch' version '1.0.7' 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-prometheus' version '2.0.3' 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.1.0' 2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-recor...

Hi Guozhang, Thank you for your response. When the records do come through, the right side is null. I can provide you the full source code if you would like to take a look at it. This is just an example, so there is very little code. It is in a private gitlab repository right now. If you want access, feel free to email directly and I can provide you the link and access. Thanks, Chad On Tue, Aug 9, 2022 at 6:11 PM Guozhang Wang < wangguoz@gmail.com > wrote: > Hello Chad, > > Here are a few thoughts on top of my head: for left joins, we would keep > those received records from the left side that have NOT found a match on > the right side in a separate temporary store (this is only recently > improved, but since you're already on 3.2.1 it's the case indeed). When > later e.g. a right hand side record arrives and found a match on the > temporary left hand side "no-matching-yet" store, we would delete from that > st...

Hello Chad, Here are a few thoughts on top of my head: for left joins, we would keep those received records from the left side that have NOT found a match on the right side in a separate temporary store (this is only recently improved, but since you're already on 3.2.1 it's the case indeed). When later e.g. a right hand side record arrives and found a match on the temporary left hand side "no-matching-yet" store, we would delete from that store and emit the join result. But if no matches found as the join window elapsed, we would still emit those records from the "no-matching-yet" store and emit the join result as (left, null). In your case, I think the arrival of the second record advances the inferred stream time, and hence after that time advanced the first record, originally in the "no-matching-yet" store, are doomed to not found a match as join window already expires, so we would emit that record, but as I said, when that hap...

Hi All, I am trying to set up encryption on a self-managed Kafka cluster running Kafka 3.0, jdk-11, using certificates signed with an internal root CA in pem format. I am running into the following error when I try to restart Kafka. *org.apache.kafka.common.errors.InvalidConfigurationException: Invalid PEM keystore configsCaused by: java.io.IOException: overrun, bytes = 1196at javax.crypto.EncryptedPrivateKeyInfo.<init>(EncryptedPrivateKeyInfo.java:98) ~[?:?]at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.privateKey(DefaultSslEngineFactory.java:512) ~[kafka-clients-3.0.0.jar:?]* The kafka config file looks as follows. Any pointers on how to resolve the error would be greatly appreciated. # TODO: update protocol after first full-deployment of a new Kafka version inter.broker.protocol.version=2.8 listener.security.protocol.map =PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL listeners=PLAINTEXT:...

Hi All, I am trying to set up encryption on a self-managed Kafka cluster running Kafka 3.0, jdk-11, using certificates signed with an internal root CA in pem format. I am running into the following error when I try to restart Kafka. *org.apache.kafka.common.errors.InvalidConfigurationException: Invalid PEM keystore configsCaused by: java.io.IOException: overrun, bytes = 1196 at javax.crypto.EncryptedPrivateKeyInfo.<init>(EncryptedPrivateKeyInfo.java:98) ~[?:?] at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.privateKey(DefaultSslEngineFactory.java:512) ~[kafka-clients-3.0.0.jar:?]* The kafka config file looks as follows. Any pointers on how to resolve the error would be greatly appreciated. # TODO: update protocol after first full-deployment of a new Kafka version inter.broker.protocol.version=2.8 listener.security.protocol.map =PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL listeners=PLAINTEX...

Thanks for driving the release, David! Best, Bruno On 03.08.22 17:46, Kirk True wrote: > Thanks for driving this, David! > > On Mon, Aug 1, 2022, at 4:45 PM, David Arthur wrote: >> The Apache Kafka community is pleased to announce the release for >> Apache Kafka 3.2.1 >> >> This is a bugfix release with several fixes since the release of >> 3.2.0. A few of the major issues include: >> >> * KAFKA-14062 OAuth client token refresh fails with SASL extensions >> * KAFKA-14079 Memory leak in connectors using errors.tolerance=all >> * KAFKA-14024 Cooperative rebalance regression causing clients to get stuck >> >> >> All of the changes in this release can be found in the release notes: >> >> https://www.apache.org/dist/kafka/3.2.1/RELEASE_NOTES.html >> >> >> You can download the source and binary release (Scala 2.12 and 2.13) from: >> >> http...

Hello, I'm doing a stream to stream leftjoin. Here is what I am seeing when I test the code. - I write a record to the left side topic. The stream app reads the message and the deserializer gets triggered. However, the join is not triggered at this time. - I write another record to the left side topic (different key) and I see the deserializer get called for the topic. I see the deserializer gets called a second time for a store-changelog topic and it deserializes the first record. The leftjoin code is executed for the first record submitted. This behavior isn't even consistent. Some records on the left never get processed. Why are all the records not processed right away or at all? My join window is just 500ms. I'm using the Kafka 3.2.1 client. Here is a code snippet of the leftjoin. KStream<String, Party> partyStream = streamsBuilder.stream(PARTY_TOPIC, Consumed.with(Serdes.String(), partySerde)); K...

Hello: I have a client considering moving to KRaft. The page here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-833%3A+Mark+KRaft+as+Production+Ready indicates that 3.3 would be the production-ready version of KRaft and available this month. Is this month still a valid target? Thanks, Greg

Hi Podunk, If you're saying the error: ERROR Exiting JVM with code 0 (org.apache.zookeeper.util.ServiceUtils), I think this is a bug in Kafka. I guess it's because we didn't close ZooKeeperAdmin before exit. Please open a JIRA ticket for it, and welcome to file a PR for it! But for the ZK shell, it works well. It returned [0] to you since you only have one broker in the cluster with id 0. For me, I used to enter the ZK shell, and type command, like this: > bin/zookeeper-shell.sh localhost:2181 Connecting to localhost:2181 Welcome to ZooKeeper! .... # then, type commands like in normal shell ls /brokers [ids, seqid, topics] ls /brokers/ids [0] ... Hope that helps. Luke On Wed, Aug 3, 2022 at 7:32 PM < podunk@gmx.com > wrote: > Anyone? > > > > > Sent: Saturday, July 30, 2022 at 9:14 PM > From: podunk@gmx.com > To: users@kafka.apache.org > Subject: An existing connection was forcibly closed by ...