Kafka

This worked for me, thank you for the help! Andrew Greer -----Original Message----- From: Daniel Hinojosa < dhinojosa@evolutionnext.com > Sent: Tuesday, August 10, 2021 2:33 PM To: users@kafka.apache.org Subject: [EXTERNAL]Re: KSQLdb Stream, Getting Topic Key CAUTION:This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. I had to verify myself, I wrote an example that uses a key that is not in the payload and I did where I specify the KEY and now it is a field. Notice the state down below CREATE STREAM my_avro_orders (total BIGINT, shipping VARCHAR, state VARCHAR KEY, discount DOUBLE, gender VARCHAR) WITH (kafka_topic='my_avro_orders', value_format='AVRO'); This came out as the following with a describe: ksql> describe my_avro_orders; Name : MY_AVRO_ORDERS Field | Type ----------------------------------- ...

Hello all, I have a question about the Group Membership lifecycle of Kafka Streams, or more specific about when Kafka Streams does leave the consumer group (in case of dynamic membership). My expectation was, that a call to the method KafkaStreams.close() also sends a LeaveGroup request to the coordination (if dynamic membership is used). However, its seems that this is not the case (at least in my case the request was not send). Only if I explicitly call KafkaStreams.removeStreamThread() a LeaveGroup request is sent to the coordinator. I used the WordCount example located in https://github.com/confluentinc/kafka-streams-examples to evaluate this. Is this how Kafka Streams is intended to work and if yes, what do you recommend to achieve that Kafka Streams leaves the group when shutting down the application? For example, one situation where I don't want to wait for the session timeout is when downscaling an application. Thanks. Best Regards, Uwe

Hello Mike, this is a bug which I also have already noticed. This has already been fixed ( https://github.com/apache/kafka/pull/10935 ) and it will be released with Kafka 3.0.0 ( https://issues.apache.org/jira/browse/KAFKA-13003 ). [ https://opengraph.githubassets.com/0ca691b1fbe7df32144cb42a8cca70a806b01d3faa4d8a533771acd38bba100f/apache/kafka/pull/10935 ]< https://github.com/apache/kafka/pull/10935 > KAFKA-13003: In kraft mode also advertise configured advertised port instead of socket port by ueisele · Pull Request #10935 · apache/kafka< https://github.com/apache/kafka/pull/10935 > In Kraft mode Apache Kafka 2.8.0 does advertise the socket port instead of the configured advertised port. A broker given with the following configuration listeners=PUBLIC:// 0.0.0.0:19092 ,REPLICATI... github.com I created a Docker Image of Kafka 3.0.0 SNAPSHOT with the fix included ( https://hub.docker.com/layers/154845427/ueisele/apache-kafka-server/3.0.0-SNAPSHOT/images/s...

Hello, We are using Kafka for data uploading via SSL. While doing the SSL certificate expiry test, we found that Kafka checks the expiry of keystore and does not start when the current date exceed the validity end date of keystore and dump the following exception in server.log ------------------------------------------------------------------------------------START-OF-EXCEPTION--------------------------------------------------------------------------------------------------------------------------------------------------------- [2021-10-08 20:01:39,731] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) org.apache.kafka.common.KafkaException: org.apache.kafka.common.config.ConfigException: Invalid value javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed for configuration A client SSLEngine created with the provided settings can't c...

Greetings, I'm trying to create a container (suitable for testing purposes) that runs a single-instance (and "zookeeper free") Kafka instance using the "KRaft" functionality in 2.8.0. In the configuration file, I have configured the listeners as follows: listeners=CONTROLLER:// 127.0.0.1:9093 ,INTERNAL:// 127.0.0.1:9094 ,EXTERNAL:// 0.0.0.0:9092 inter.broker.listener.name =INTERNAL advertised.listeners=INTERNAL:// 127.0.0.1:9094 ,EXTERNAL:// 127.0.0.1:55032 controller.listener.names=CONTROLLER listener.security.protocol.map=CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT Then I forward a random port to port 9092 inside the container. However, when I connect to the container from the outside, the client sees an advertised port of 9092, which is only configured in the "listeners" key: $ kafkacat -L -b 127.0.0.1:55033 Metadata for all topics (from broker -1: 127.0.0.1:55033/bootstrap ): 1 brokers: broke...

I have 2 Kafka setup, A and B. A is a cluster with 3 instances running on the same machine, while B is a stand alone in another machine. I tried to use Mirror Maker to replicate from A to B. The config file is as follows: ----- clusters = A, BA.bootstrap.servers = host1:9091, host1:9092, host1:9093B.bootstrap.servers = host2:9092 A->B.enabled = trueB->A.enabled = false replication.factor = 1 checkpoints.topic.replication.factor=1heartbeats.topic replication.factor=1offset-syncs.topic.replication.factor=1 offset.storage.replication.factor=1status.storage.replication.factor=1config.storage.replication.factor=1--- I execute mirror maker as follows: bin/connect-mirror-maker.sh config/mirror.maker.properties But when I run the mirror maker on the 'B' side I can see that it keeps creating 'heartbeats' topic. So at some point I will end up having so many 'A.heartbeats' topics like below. And it will just keep growing until I kill the mir...

Glad you figured it out. If you think the docs should be improved, we take PRs :) -- In the end, it's a broader community effort to have good docs... -Matthias On 6/11/21 12:10 PM, Richard Rossel wrote: > Thanks Matthias for your help, > I was able to find the issue, it was actually my Serdes definition. It > is very hard to find those > type problems, the execution trace log are not very helpful, and > documentation regarding custom > serdes are not that great. > > Thanks for the tip regarding the need of materializing tables, I had > no idea about the "push down" process. > Do you know where I can find some documentation regarding that process? > > > On Tue, Jun 8, 2021 at 5:54 PM Matthias J. Sax < mjsax@apache.org > wrote: >> >> Can you create a ticket? There is at least one bug I found looking into >> the code. (At least in `trunk` -- what version are you using? -- we >...