Kafka

Congratulations Sato :-) Regards, Dhiraj On Thu, Dec 29, 2022 at 10:17 AM Bill Bejeck < bbejeck@gmail.com > wrote: > Congratulations Satish! > > -Bill > > On Thu, Dec 29, 2022 at 11:05 AM Satish Duggana < satish.duggana@gmail.com > > wrote: > > > Thank you, everyone! > > > > On Wed, 28 Dec 2022 at 00:02, Matthias J. Sax < mjsax@apache.org > wrote: > > > > > > Congrats! > > > > > > On 12/27/22 10:20 AM, Kirk True wrote: > > > > Congrats, Satish! > > > > > > > > On Fri, Dec 23, 2022, at 10:07 AM, Jun Rao wrote: > > > >> Hi, Everyone, > > > >> > > > >> The PMC of Apache Kafka is pleased to announce a new Kafka committer > > Satish > > > >> Duggana. > > > >> > > > >> Satish has been a long time Kafka contributor since 2017. He is the ...

hi, I guess this would mean a deserialisation error, meaning he data you're trying to read, is not as expected by the Confluent Avro deserializer. https://www.youtube.com/watch?v=GWBk9LIzy-A < https://www.youtube.com/watch?v=GWBk9LIzy-A > > On 29 Dec 2022, at 16:47, Дмитрий Лушников < lushnikov.dmitry@gmail.com > wrote: > > Anders, maybe you can help me? > MirrorMaker 2 stops transfer messages after setting up Schema Registry > Transfer SMT. > > The logs show these errors: > > Caused by: org.apache.kafka.common.errors.SerializationException: > Unknown magic byte! > > Caused by: cricket.jmoore.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: > Schema 1 not found; error code: 40403 > > I can't figure out what to do with the *magic byte* and who needed *Schema > 1*? > > https://stackoverflow.com/questions/74952744/mirrormaker-2-stops-transfer-messages-a...

Sounds like a SpringBoot issue rather than a KS issues. -Matthias On 12/29/22 2:45 AM, Nawal Sah wrote: > Hi, > > My SpringBoot stream application works fine in a fresh start of the > clustered environment. > But when I restart one of the pods out of two pods, I start getting the > below exception from "KafkaStreams.store". I debugged the state but found > it was RUNNING. > > *org.apache.kafka.streams.errors.InvalidStateStoreException: The state > store, wkstore, may have migrated to another instance. * > *at > org.apache.kafka.streams.state.internals.WrappingStoreProvider.stores(WrappingStoreProvider.java:67) > * > *at > org.apache.kafka.streams.state.internals.CompositeReadOnlyWindowStore.fetchAll(CompositeReadOnlyWindowStore.java:175)* > > To fix this, I must restart all the pods/instances or the cluster. > > *Steps to reproduce* > > 1. Kafka Stream application environment ...

Congratulations Satish! -Bill On Thu, Dec 29, 2022 at 11:05 AM Satish Duggana < satish.duggana@gmail.com > wrote: > Thank you, everyone! > > On Wed, 28 Dec 2022 at 00:02, Matthias J. Sax < mjsax@apache.org > wrote: > > > > Congrats! > > > > On 12/27/22 10:20 AM, Kirk True wrote: > > > Congrats, Satish! > > > > > > On Fri, Dec 23, 2022, at 10:07 AM, Jun Rao wrote: > > >> Hi, Everyone, > > >> > > >> The PMC of Apache Kafka is pleased to announce a new Kafka committer > Satish > > >> Duggana. > > >> > > >> Satish has been a long time Kafka contributor since 2017. He is the > main > > >> driver behind KIP-405 that integrates Kafka with remote storage, a > > >> significant and much anticipated feature in Kafka. > > >> > > >> Congratulations, Satish! > > >...

Thank you, everyone! On Wed, 28 Dec 2022 at 00:02, Matthias J. Sax < mjsax@apache.org > wrote: > > Congrats! > > On 12/27/22 10:20 AM, Kirk True wrote: > > Congrats, Satish! > > > > On Fri, Dec 23, 2022, at 10:07 AM, Jun Rao wrote: > >> Hi, Everyone, > >> > >> The PMC of Apache Kafka is pleased to announce a new Kafka committer Satish > >> Duggana. > >> > >> Satish has been a long time Kafka contributor since 2017. He is the main > >> driver behind KIP-405 that integrates Kafka with remote storage, a > >> significant and much anticipated feature in Kafka. > >> > >> Congratulations, Satish! > >> > >> Thanks, > >> > >> Jun (on behalf of the Apache Kafka PMC) > >> > >

Anders, maybe you can help me? MirrorMaker 2 stops transfer messages after setting up Schema Registry Transfer SMT. The logs show these errors: Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte! Caused by: cricket.jmoore.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema 1 not found; error code: 40403 I can't figure out what to do with the *magic byte* and who needed *Schema 1*? https://stackoverflow.com/questions/74952744/mirrormaker-2-stops-transfer-messages-after-setting-up-schema-registry-transfer

Hi, My SpringBoot stream application works fine in a fresh start of the clustered environment. But when I restart one of the pods out of two pods, I start getting the below exception from "KafkaStreams.store". I debugged the state but found it was RUNNING. *org.apache.kafka.streams.errors.InvalidStateStoreException: The state store, wkstore, may have migrated to another instance. * *at org.apache.kafka.streams.state.internals.WrappingStoreProvider.stores(WrappingStoreProvider.java:67) * *at org.apache.kafka.streams.state.internals.CompositeReadOnlyWindowStore.fetchAll(CompositeReadOnlyWindowStore.java:175)* To fix this, I must restart all the pods/instances or the cluster. *Steps to reproduce* 1. Kafka Stream application environment 'kafka-streams', version: '3.0.1' 'kafka-clients', version: '3.0.1' 2. Create a cluster environment with at least two replication factors of the Kafka Stream application. 3. Restar...

Hi Nicolas, > Any Idea some may have to limit log compaction disk usage ? That's a pretty big issue we're having, compaction of a few topics is using far more IO than everything else on the broker... I don't know any other better solution except make the `log.cleaner.io.max.bytes.per.second` lower than you expected, maybe 512? As I explained earlier, based on how throttler works currently, there will be a 300ms time slot of full speed read/write. And it will make the final IO speed higher than you expected. However, Apache Kafka is an open source project, so, welcome to make a contribution to make it better! Let me know if you're interested. Thank you. Luke On Wed, Dec 28, 2022 at 6:25 PM Nicolas Carlot <nicolas.carlot@chronopost.fr.invalid> wrote: > Okay, thx for the explaination Luc. > When looking for logs on a bigger topic, I can see the IO limitation kinda > works. > I have like15MB/s when having configured >...

Okay, thx for the explaination Luc. When looking for logs on a bigger topic, I can see the IO limitation kinda works. I have like15MB/s when having configured log.cleaner.io.max.bytes.per.second=1024.0 I'm not very comfortable with this. Setting the limit rate to 1Kb/s, ending with 15Mb/s, seems a bit akward. Any Idea some may have to limit log compaction disk usage ? That's a pretty big issue we're having, compaction of a few topics is using far more IO than everything else on the broker... Log cleaner thread 0 cleaned log kafka-stream-parcel360.lt_event360.multiprofiles-lteventmulti2-KSTREAM.RETAINER.kafka-stream-parcel360.lt_event360.multiprofiles-lteventmulti2-changelog-8 (dirty section = [34819497, 44148502]) 1 372,1 MB of log processed in 182,2 seconds (7,5 MB/sec). Indexed 1 143,2 MB in 84,9 seconds (13,5 Mb/sec, 46,6% of total time) Buffer utilization: 9,2% Cleaned 1 372,1 MB in 97,3 seconds (14,1 Mb/sec, 53,4% of t...

Hi Nicolas, The throttler in log cleaner is just a simple throttler to control the IO speed by periodically (every 300ms) check the IO rate so far, and sleep some time to slow it down. That is, in your case, it could be: log cleaner runs for 300ms with full IO speed -> check -> throttle (sleep to reach desired rate) -> run 300ms with full speed -> done So, when printing the log, you will see the rate is higher than you expected. At least I ran in kafka v3.3.1 and it works well. Thank you. Luke On Thu, Dec 22, 2022 at 11:31 PM Nicolas Carlot <nicolas.carlot@chronopost.fr.invalid> wrote: > Hello users, > > I'm trying to use the log.cleaner.io.max.bytes.per.second configuration to > limit disk usage when the log cleaner runs, but it doesn't seem to work as > expected. > When looking at the logs, I can see it still uses far more than configured. > > This is the output with log.cleaner.io.max.bytes.per.second=1...

Congrats! On 12/27/22 10:20 AM, Kirk True wrote: > Congrats, Satish! > > On Fri, Dec 23, 2022, at 10:07 AM, Jun Rao wrote: >> Hi, Everyone, >> >> The PMC of Apache Kafka is pleased to announce a new Kafka committer Satish >> Duggana. >> >> Satish has been a long time Kafka contributor since 2017. He is the main >> driver behind KIP-405 that integrates Kafka with remote storage, a >> significant and much anticipated feature in Kafka. >> >> Congratulations, Satish! >> >> Thanks, >> >> Jun (on behalf of the Apache Kafka PMC) >> >

+1 (non-binding). I used the staged Scala 2.13 binaries and Maven repo to run my tests. All seems to work fine. Thanks for running the release. Jakub On Wed, Dec 21, 2022 at 11:22 PM Chris Egerton <chrise@aiven.io.invalid> wrote: > Hello Kafka users, developers and client-developers, > > This is the second candidate for release of Apache Kafka 3.3.2. > > This is a bugfix release with several fixes since the release of 3.3.1. A > few of the major issues include: > > * KAFKA-14358 Users should not be able to create a regular topic name > __cluster_metadata > KAFKA-14379 Consumer should refresh preferred read replica on update > metadata > * KAFKA-13586 Prevent exception thrown during connector update from > crashing distributed herder > > > Release notes for the 3.3.2 release: > https://home.apache.org/~cegerton/kafka-3.3.2-rc1/RELEASE_NOTES.html > > *** Please download, test and vote by Friday, ...

Hello users, I'm trying to use the log.cleaner.io.max.bytes.per.second configuration to limit disk usage when the log cleaner runs, but it doesn't seem to work as expected. When looking at the logs, I can see it still uses far more than configured. This is the output with log.cleaner.io.max.bytes.per.second=1024 [2022-12-22 16:14:49,462] INFO [kafka-log-cleaner-thread-0]: Log cleaner thread 0 cleaned log __consumer_offsets-9 (dirty section = [13509895330, 13510879425]) 100,2 MB of log processed in 1,7 seconds (60,1 MB/sec). Indexed 99,7 MB in 0,8 seconds (117,5 Mb/sec, 50,8% of total time) Buffer utilization: 0,1% Cleaned 100,2 MB in 0,8 seconds (122,2 Mb/sec, 49,2% of total time) Start size: 100,2 MB (988642 messages) End size: 0,5 MB (4547 messages) 99,5% size reduction (99,5% fewer messages) I did update the value trough kafak-config this way: kafka-configs.sh --bootstrap-server localh...

Hello Kafka users, developers and client-developers, This is the second candidate for release of Apache Kafka 3.3.2. This is a bugfix release with several fixes since the release of 3.3.1. A few of the major issues include: * KAFKA-14358 Users should not be able to create a regular topic name __cluster_metadata KAFKA-14379 Consumer should refresh preferred read replica on update metadata * KAFKA-13586 Prevent exception thrown during connector update from crashing distributed herder Release notes for the 3.3.2 release: https://home.apache.org/~cegerton/kafka-3.3.2-rc1/RELEASE_NOTES.html *** Please download, test and vote by Friday, January 6, 2023, 10pm UTC (this date is chosen to accommodate the various upcoming holidays that members of the community will be taking and give everyone enough time to test out the release candidate, without unduly delaying the release) Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache...

Hello! My name is Kasia and I would like to share with you our tutorials. After reading, you can also visit our GitHub, to see examples on how to work with the data online generator. Read about solution that support use-cases such us: 🔹Analysis of user web sessions for web analytics. 🔹Funnel analysis, cohort analysis and many more, to understand how product users interact with the product, behave, convert or drop out. 🔹Fraud/anomaly or threat detection to prevent them quickly. 🔹Product recommendations, suggestions and many more… *LINK < https://getindata.com/blog/data-online-generation-for-event-stream-processing/?utm_source=mailing&utm_medium=kafka2&utm_campaign=grupy&utm_id=getindata >* In this one, we would like to prove that Iceberg can be used not only in #batchprocessing < https://www.linkedin.com/feed/hashtag/?keywords=batchprocessing&highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6934831698933125120 > but also can be...

Congratulations, Luke! -John On Mon, Dec 19, 2022, at 22:08, Luke Chen wrote: > Thank you all! :) > > Luke > > On Tue, Dec 20, 2022 at 3:02 AM Matthias J. Sax < mjsax@apache.org > wrote: > >> Congrats! Well deserved! >> >> On 12/19/22 9:37 AM, Divij Vaidya wrote: >> > Congratulations Luke! >> > -- >> > Divij Vaidya >> > >> > >> > >> > On Mon, Dec 19, 2022 at 10:09 AM Bruno Cadonna < cadonna@apache.org > >> wrote: >> > >> >> Congrats, Luke! >> >> >> >> Best, >> >> Bruno >> >> >> >> On 17.12.22 03:31, ziming deng wrote: >> >>> Congratulations, Luke! >> >>> Well deserved. >> >>> >> >>> -- >> >>> Ziming >> >>> >> >>>> On Dec 17, 2022, at 08:08, Sophie Ble...

There are also some errors like: After being scheduled for shutdown, the orphan task threw an uncaught exception. A newer instance of this task might be already running Any help? Thanks Hongguo An On 12/20/22, 9:26 AM, "An, Hongguo (CORP)" <Hongguo.An@ADP.com.INVA <mailto: Hongguo.An@ADP.com.INVA >LID> wrote: WARNING: Do not click links or open attachments unless you recognize the source of the email and know the contents are safe. ********************************************************************** Hi: I am running MM2, when it starts, it copies data from source to destination, but after a while, it stopped data sync. If I restart it, it will copy some data again, then stopped data sync. The process is still running but no data is synched. With some errors like: Graceful stop of task MirrorSourceConnector-0 failed. Appreciate any help! Thanks Hongguo An This message and any attachments are intended only for th...

Congrats Josep! On Tue, Dec 20, 2022, 21:56 Matthias J. Sax < mjsax@apache.org > wrote: > Congrats! > > On 12/20/22 12:01 PM, Josep Prat wrote: > > Thank you all! > > > > ——— > > Josep Prat > > > > Aiven Deutschland GmbH > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > m: +491715557497 > > > > w: aiven.io > > > > e: josep.prat@aiven.io > > > > On Tue, Dec 20, 2022, 20:42 Bill Bejeck < bbejeck@gmail.com > wrote: > > > >> Congratulations Josep! > >> > >> -Bill > >> > >> On Tue, Dec 20, 2022 at 1:11 PM Mickael Maison < > mickael.maison@gmail.com > > >> wrote: > >> > >>> Congratulations Josep! ...

Congrats! On 12/20/22 12:01 PM, Josep Prat wrote: > Thank you all! > > ——— > Josep Prat > > Aiven Deutschland GmbH > > Immanuelkirchstraße 26, 10405 Berlin > > Amtsgericht Charlottenburg, HRB 209739 B > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > m: +491715557497 > > w: aiven.io > > e: josep.prat@aiven.io > > On Tue, Dec 20, 2022, 20:42 Bill Bejeck < bbejeck@gmail.com > wrote: > >> Congratulations Josep! >> >> -Bill >> >> On Tue, Dec 20, 2022 at 1:11 PM Mickael Maison < mickael.maison@gmail.com > >> wrote: >> >>> Congratulations Josep! >>> >>> On Tue, Dec 20, 2022 at 6:55 PM Bruno Cadonna < cadonna@apache.org > >> wrote: >>>> >>>> Congrats, Josep! >>>> >>>> Well deserved! >>>> >>>...

The warning should have come first though I'm sorry it's already disseminated - you can remove me from your addresses from now on if you like but I enjoy reading them and learning a bit and I don't know how to code etc- I was in customer service and I'm a fan of the software but I'm not intelligent enough to be a coder myself but I may have work eventually that overlaps with this knowledge so just check your mailing lists for any information you wouldn't wish to share with someone that may be in the field eventually. Thanks On Tue, Dec 20, 2022 at 12:27 PM Schehrzade < schehrzade.syed.1@gmail.com > wrote: > I have received this in error! > > On Tue, Dec 20, 2022 at 12:27 PM An, Hongguo (CORP) > <Hongguo.An@adp.com.invalid> wrote: > >> Hi: >> I am running MM2, when it starts, it copies data from source to >> destination, but after a while, it stopped data sync. If I restart it, it >> wi...

I have received this in error! On Tue, Dec 20, 2022 at 12:27 PM An, Hongguo (CORP) <Hongguo.An@adp.com.invalid> wrote: > Hi: > I am running MM2, when it starts, it copies data from source to > destination, but after a while, it stopped data sync. If I restart it, it > will copy some data again, then stopped data sync. > The process is still running but no data is synched. > With some errors like: > Graceful stop of task MirrorSourceConnector-0 failed. > > Appreciate any help! > > Thanks > Hongguo An > > > This message and any attachments are intended only for the use of the > addressee and may contain information that is privileged and confidential. > If the reader of the message is not the intended recipient or an authorized > representative of the intended recipient, you are hereby notified that any > dissemination of this communication is strictly prohibited. If you have > received this commu...

Hi: I am running MM2, when it starts, it copies data from source to destination, but after a while, it stopped data sync. If I restart it, it will copy some data again, then stopped data sync. The process is still running but no data is synched. With some errors like: Graceful stop of task MirrorSourceConnector-0 failed. Appreciate any help! Thanks Hongguo An This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

Thanks, then I know. On Tue, Dec 20, 2022 at 2:14 PM Luke Chen < showuon@gmail.com > wrote: > Hi Simon, > > This is a known issue in kafka v3.3.1, and will be fixed in kafka v3.3.2 > and v3.4.0. > Here's the JIRA issue: https://issues.apache.org/jira/browse/KAFKA-14337 > > Thank you > Luke > > On Tue, Dec 20, 2022 at 7:13 PM Simon Dahlbacka < simon.dahlbacka@gmail.com > > > wrote: > > > I am running a single node cluster in KRaft mode on > > confluentinc/cp-kafka:7.3.0-1-ubi8 > > aka kafka 3.3.x > > > > I cannot recreate a topic 'base.m3.ocusma' (but I can > create,delete,create > > other topics just fine. > > > > the topic is deleted: > > > > ls /opt/kafka/data-0/logs/base.m3.ocusma > > > > ls: cannot access '/opt/kafka/data-0/logs/base.m3.ocusma': No such file > or > > directory > > > ...

Hi Simon, This is a known issue in kafka v3.3.1, and will be fixed in kafka v3.3.2 and v3.4.0. Here's the JIRA issue: https://issues.apache.org/jira/browse/KAFKA-14337 Thank you Luke On Tue, Dec 20, 2022 at 7:13 PM Simon Dahlbacka < simon.dahlbacka@gmail.com > wrote: > I am running a single node cluster in KRaft mode on > confluentinc/cp-kafka:7.3.0-1-ubi8 > aka kafka 3.3.x > > I cannot recreate a topic 'base.m3.ocusma' (but I can create,delete,create > other topics just fine. > > the topic is deleted: > > ls /opt/kafka/data-0/logs/base.m3.ocusma > > ls: cannot access '/opt/kafka/data-0/logs/base.m3.ocusma': No such file or > directory > > > When I try to create the topic I see the following in the kafka log: > > [2022-12-20 09:38:15,882] WARN [Controller 0] createTopics: failed with > unknown server exception NoSuchElementException at epoch 93 in 1311 us. > Renounc...

I am running a single node cluster in KRaft mode on confluentinc/cp-kafka:7.3.0-1-ubi8 aka kafka 3.3.x I cannot recreate a topic 'base.m3.ocusma' (but I can create,delete,create other topics just fine. the topic is deleted: ls /opt/kafka/data-0/logs/base.m3.ocusma ls: cannot access '/opt/kafka/data-0/logs/base.m3.ocusma': No such file or directory When I try to create the topic I see the following in the kafka log: [2022-12-20 09:38:15,882] WARN [Controller 0] createTopics: failed with unknown server exception NoSuchElementException at epoch 93 in 1311 us. Renouncing leadership and reverting to the last committed offset 2037058. (org.apache.kafka.controller.QuorumController) Tue, Dec 20 2022 11:38:15 am java.util.NoSuchElementException Tue, Dec 20 2022 11:38:15 am at org.apache.kafka.timeline.SnapshottableHashTable$CurrentIterator.next(SnapshottableHashTable.java:167) Tue, Dec 20 2022 11:38:15 am at org.apache.kafka.timeline.Snapshottable...