Kafka

Kafka 中文文档 - ApacheCN Q: <img ... src="/10/images/...png"&gt;; src should be&nbsp;&nbsp; <img ... src="/images/...png"&gt;;

Hi José, I verified the signatures and ran all the unit tests, as well as the Streams integration tests with: > ./gradlew -version > > ------------------------------------------------------------ > Gradle 7.4.2 > ------------------------------------------------------------ > > Build time: 2022-03-31 15:25:29 UTC > Revision: 540473b8118064efcc264694cbcaa4b677f61041 > > Kotlin: 1.5.31 > Groovy: 3.0.9 > Ant: Apache Ant(TM) version 1.10.11 compiled on July 10 2021 > JVM: 1.8.0_342 (Private Build 25.342-b07) > OS: Linux 5.15.0-48-generic amd64 I'm +1 (binding), pending system test results. Thanks, -John On Fri, Sep 30, 2022, at 11:46, Bill Bejeck wrote: > Hi, > > I did the following to validate the release: > > 1. Validated all checksums, signatures > 2. Built from source and ran all the unit tests > 3. Ran the ZK and KRaft quic...

Hi All, I have a scenario where I want to send data from elasticsearch to Mongodb through kafka and while researching I came across Kafka connect. Through Kafka connect is it possible to have the elasticsearch as a source connector that will send data/messages from elasticsearch to kafka? I came across Kafka sink connector which can receive messages/Data from the Kafka server through topics. In case of Mongodb sink connector, what would be the behaviour if I have larger data that is greater than the maximum size of document in mongodb which is 16MB. Is there a way to handle this? Also, Is there a prerequisite to have docker setup before installing the connectors Thanks & Regards, Namita

Hi, I did the following to validate the release: 1. Validated all checksums, signatures 2. Built from source and ran all the unit tests 3. Ran the ZK and KRaft quickstart 4. Ran the Raft single quorum test 5. Ran the Kafka Streams quick start +1(binding) pending successful system test run Thanks, Bill On Fri, Sep 30, 2022 at 5:30 AM David Jacot <djacot@confluent.io.invalid> wrote: > Hey, > > I performed the following validations: > * Verified all checksums and signatures. > * Built from source and ran unit tests. > * Ran the first quickstart steps for both ZK and KRaft. > * Spotchecked the Javadocs. > > I am +1 (binding), assuming that the system tests look good. > > Thanks for running the release. > > Best, > David > > On Fri, Sep 30, 2022 at 2:23 AM José Armando García Sancio > <jsancio@confluent.io.invalid> wrote: > > > > On Thu, Sep 29, 2022 a...

Hi again, Tomasz, Your issue is really bugging me, since I'm pretty sure it shouldn't be happening. I went ahead and added an integration test with your exact scenario, as I understand it: https://github.com/apache/kafka/pull/12706 The test passes for me. Do you think you can check it out and try adjusting the test setup until you're able to reproduce the behavior you're seeing? If you can do that, I think we will get to the bottom of it. Thanks, -John On Fri, Sep 30, 2022, at 09:51, John Roesler wrote: > Hi Tomasz, > > Thanks for trying that out. It's not the way I'd expect it to work. I > don't remember if there were any follow-up bugs that have been solved > in subsequent releases. Just as a long shot, perhaps you can try it on > the latest release (3.3.0)? > > Otherwise, I think the best path forward would be to file a bug report > on the Apache Kafka Jira with enough information to reproduce th...

Hi Tomasz, Thanks for trying that out. It's not the way I'd expect it to work. I don't remember if there were any follow-up bugs that have been solved in subsequent releases. Just as a long shot, perhaps you can try it on the latest release (3.3.0)? Otherwise, I think the best path forward would be to file a bug report on the Apache Kafka Jira with enough information to reproduce the issue (or if you're able to provide a repro, that would be awesome). Thanks, and sorry for the trouble. -John On Tue, Sep 27, 2022, at 03:15, Tomasz Gac wrote: > I upgraded to kafka streams 3.0.0 with positive task.max.idle.ms and it did > not help. > When lag is large, the application still consumes data batches without > interleaving. > > > > wt., 27 wrz 2022 o 05:51 John Roesler < vvcephei@apache.org > napisał(a): > >> Hi Tomasz, >> >> Thanks for asking. This sounds like the situation that we fixed in Apache ...

Happy to help, Namita. Regards, Jim Ziesig On Fri, Sep 30, 2022 at 7:51 AM Namita Jaokar < jaokarnamita9@gmail.com > wrote: Hi James, Thanks alot James for the solution provided. The below configurations did work when producer and consumer applications were checked with the scenarios mentioned. Thanks & Regards, Namita Jaokar On Tue, Sep 27, 2022 at 7:35 PM James Ziesig < james.ziesig@broadcom.com .invalid> wrote: > Hi Namita, > > This is likely a listener configuration issue.  The bootstrap > server setting only gets you so far, your client must be able to contact > the server via a listener address.  I have seen this occur on a few systems > where a hostname could not be identified by Kafka, so the server advertises > a listener on only localhost:9092.  Since your producer is not on the same > system, it will not connect. > > You can try updating the listeners and advertised.listeners properties...

Hi James, Thanks alot James for the solution provided. The below configurations did work when producer and consumer applications were checked with the scenarios mentioned. Thanks & Regards, Namita Jaokar On Tue, Sep 27, 2022 at 7:35 PM James Ziesig <james.ziesig@broadcom.com.invalid> wrote: > Hi Namita, > > This is likely a listener configuration issue. The bootstrap > server setting only gets you so far, your client must be able to contact > the server via a listener address. I have seen this occur on a few systems > where a hostname could not be identified by Kafka, so the server advertises > a listener on only localhost:9092. Since your producer is not on the same > system, it will not connect. > > You can try updating the listeners and advertised.listeners properties in > your server configuration file to use the actual hostname (assuming the > producer can reach the server via the hostname). > > ...

Hey, I performed the following validations: * Verified all checksums and signatures. * Built from source and ran unit tests. * Ran the first quickstart steps for both ZK and KRaft. * Spotchecked the Javadocs. I am +1 (binding), assuming that the system tests look good. Thanks for running the release. Best, David On Fri, Sep 30, 2022 at 2:23 AM José Armando García Sancio <jsancio@confluent.io.invalid> wrote: > > On Thu, Sep 29, 2022 at 2:39 PM José Armando García Sancio > < jsancio@confluent.io > wrote: > > Please download, test and vote by Tuesday, October 4, 9am PT. > > The vote will be open for 72 hours. Please vote by Sunday, October 2nd, 3 PM PT. > > Thanks! > -- > -José

On Thu, Sep 29, 2022 at 2:39 PM José Armando García Sancio < jsancio@confluent.io > wrote: > Please download, test and vote by Tuesday, October 4, 9am PT. The vote will be open for 72 hours. Please vote by Sunday, October 2nd, 3 PM PT. Thanks! -- -José

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.3.1. This release fixes two issues with the 3.3.0 release. * [KAFKA-14259] - BrokerRegistration#toString throws an exception, terminating metadata replay * [KAFKA-14265] - Prefix ACLs may shadow other prefix ACLs Release notes for the 3.3.1 release: https://home.apache.org/~jsancio/kafka-3.3.1-rc0/RELEASE_NOTES.html Please download, test and vote by Tuesday, October 4, 9am PT. Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://home.apache.org/~jsancio/kafka-3.3.1-rc0/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~jsancio/kafka-3.3.1-rc0/javadoc/ * Tag to be voted upon (off 3.3 branch) is the 3.3.1 tag: https://github.com/apache/...

For those interested, here's a PR to fix the HTML rendering issue: https://github.com/apache/kafka-site/pull/446 -David On Wed, Sep 28, 2022 at 9:45 AM David Arthur < davidarthur@apache.org > wrote: > > Thanks Divij, we made a bunch of documentation changes at the last minute for this release, so we're trying to figure out when the issue was introduced. I'm hoping we can fix it before we send out the announcement. > > Thanks for calling it out :) > > Best, > David > > On Wed, Sep 28, 2022 at 09:11 Divij Vaidya < divijvaidya13@gmail.com > wrote: >> >> Please ignore my previous email. Seems like that is a known issue and we a >> plan to fix it after the release. >> >> Divij Vaidya >> >> >> >> On Wed, Sep 28, 2022 at 3:08 PM Divij Vaidya < divijvaidya13@gmail.com > >> wrote: >> >> > Hey folks >> > >> > I notice...

Thanks Divij, we made a bunch of documentation changes at the last minute for this release, so we're trying to figure out when the issue was introduced. I'm hoping we can fix it before we send out the announcement. Thanks for calling it out :) Best, David On Wed, Sep 28, 2022 at 09:11 Divij Vaidya < divijvaidya13@gmail.com > wrote: > Please ignore my previous email. Seems like that is a known issue and we a > plan to fix it after the release. > > Divij Vaidya > > > > On Wed, Sep 28, 2022 at 3:08 PM Divij Vaidya < divijvaidya13@gmail.com > > wrote: > > > Hey folks > > > > I noticed a non-blocking bug with the documentation page where the arrow > > to left nav overlaps with the text and a blue color vertical bar appears > at > > the right side. Please see the highlighted elements in the attached > image. > > In contrast, the current documentation page does not have ...

Please ignore my previous email. Seems like that is a known issue and we plan to fix it after the release. Divij Vaidya On Wed, Sep 28, 2022 at 3:08 PM Divij Vaidya < divijvaidya13@gmail.com > wrote: > Hey folks > > I noticed a non-blocking bug with the documentation page where the arrow > to left nav overlaps with the text and a blue color vertical bar appears at > the right side. Please see the highlighted elements in the attached image. > In contrast, the current documentation page does not have this bug. > > Reproducer: > 1. Visit https://kafka.apache.org/33/documentation.html on a > chrome browser. > 2. Observe that arrow to expand left nav overlaps with text. > 3. Remove "33/" from the url to observe the current documentation. > 4. Observe that the current website does not have this bug. > > > Divij Vaidya > > > > On Tue, Sep 27, 2022 at 8:35 PM David Arthur < davidar...

Hey folks I noticed a non-blocking bug with the documentation page where the arrow to left nav overlaps with the text and a blue color vertical bar appears at the right side. Please see the highlighted elements in the attached image. In contrast, the current documentation page does not have this bug. Reproducer: 1. Visit  https://kafka.apache.org/33/documentation.html on a chrome browser. 2. Observe that arrow to expand left nav overlaps with text. 3. Remove "33/" from the url to observe the current documentation.  4. Observe that the current website does not have this bug. Divij Vaidya On Tue, Sep 27, 2022 at 8:35 PM David Arthur < davidarthur@apache.org > wrote: I re-ran the failing system tests last night and got passing builds for each. There is still some flakiness it seems. Round trip test: http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1664223686--apache--3.3--9b8a48ca2a/2022-09-26--...

Hi Namita, You need to use an address that the producer system can reach on the server system as the listener address.  If you are using an IP address in the bootstrap server value on the producer, you likely need to use the IP address in the listener as well.  Either that or add the IP address and hostname to /etc/hosts.  I believe your producer is on Windows so it might be best to stick with the IP address. Good luck, Jim Ziesig On Wed, Sep 28, 2022 at 8:21 AM Namita Jaokar < jaokarnamita9@gmail.com > wrote: Hi James, As suggested, I did try below steps but still my producer is unable to connect with the consumer. Below are the changes checked upon: *Producer changes* In My java producer configuration class which is at client network and different system, Changed the configuration of bootstrap server from localhost to IP address of the system wherein my consumer and kafka server is present  configs.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, " 000.00...

Hi James, As suggested, I did try below steps but still my producer is unable to connect with the consumer. Below are the changes checked upon: *Producer changes* In My java producer configuration class which is at client network and different system, Changed the configuration of bootstrap server from localhost to IP address of the system wherein my consumer and kafka server is present configs.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, " 000.000.0.000:9092 "); - IP Address of SERVER_HOSTNAME.com: *Server Changes:* The Server changes which is on a different network and system.Made changes to the advertised.listeners=PLAINTEXT://SERVER_HOSTNAME.com:9092 SERVER_HOSTNAME is the hostname of the system where the kafka server and consumer both are Present. After making these changes, restarted zookeeper and kafka server. After making the above changes, When the producer application in java is started , the producer throws the below exception When ...

Yes, exactly. I can prepare a PR to add this page. Regards, Francois On 28/09/2022 12:02, Bruno Cadonna wrote: > Hi, > > Ah, I see you were not looking for actual commercial support but > rather for the page itself. > > Best, > Bruno > > On 28.09.22 11:31, Jean-Baptiste Onofré wrote: >> Hi, >> >> +1, yes it makes sense to me. >> >> Regards >> JB >> >> On Wed, Sep 28, 2022 at 11:26 AM fpapon < fpapon@apache.org > wrote: >>> >>> Hi Bruno, >>> >>> Thanks for your reply, I'm looking for a commercial support about >>> services consulting like we can have in others Apache project like: >>> >>> https://camel.apache.org/manual/commercial-camel-offerings.html >>> >>> https://activemq.apache.org/support >>> >>> I think it could be nice to add this on the Kafka website....

Hi, Ah, I see you were not looking for actual commercial support but rather for the page itself. Best, Bruno On 28.09.22 11:31, Jean-Baptiste Onofré wrote: > Hi, > > +1, yes it makes sense to me. > > Regards > JB > > On Wed, Sep 28, 2022 at 11:26 AM fpapon < fpapon@apache.org > wrote: >> >> Hi Bruno, >> >> Thanks for your reply, I'm looking for a commercial support about >> services consulting like we can have in others Apache project like: >> >> https://camel.apache.org/manual/commercial-camel-offerings.html >> >> https://activemq.apache.org/support >> >> I think it could be nice to add this on the Kafka website. >> >> Regards, >> >> Francois >> >> On 28/09/2022 11:05, Bruno Cadonna wrote: >>> Hi Francois, >>> >>> I am not aware of such a page on the Apache Kafka website. ...

Hi, +1, yes it makes sense to me. Regards JB On Wed, Sep 28, 2022 at 11:26 AM fpapon < fpapon@apache.org > wrote: > > Hi Bruno, > > Thanks for your reply, I'm looking for a commercial support about > services consulting like we can have in others Apache project like: > > https://camel.apache.org/manual/commercial-camel-offerings.html > > https://activemq.apache.org/support > > I think it could be nice to add this on the Kafka website. > > Regards, > > Francois > > On 28/09/2022 11:05, Bruno Cadonna wrote: > > Hi Francois, > > > > I am not aware of such a page on the Apache Kafka website. > > > > There are a variety of companies that sell Kafka as a self-hosted > > platform or as a Cloud-hosted service. > > > > Those companies include Confluent (disclaimer: I work for them), Red > > Hat, AWS, Aiven, Instaclustr, Cloudera, and more. ...