Kafka

Hi Florin, A colleague of mine wrote this piece[1] inspired by your question. Let me know if this answers your question. [1]: https://ftisiot.net/posts/kafka-consume-http/ Best, On Wed, Aug 30, 2023 at 9:00 AM Spico Florin < spicoflorin@gmail.com > wrote: > Hello! > > I'm looking for opensource solutions for consuming Kafka messages in > browser. > I found to interesting articles > 1. > > https://www.confluent.io/blog/consuming-messages-out-of-apache-kafka-in-a-browser/ > 2. https://ably.com/topic/websockets-kafka > > Unfortunately: > - the first is from 2019 and does not provide any reference to solution and > a github repository. > - the second one is a proprietary solution of Ably. > > Can you please advice what open source solutions are using for consuming > Kafka messages in browser? > > Thank you, > Florin > -- [image: Aiven] < https://www.aiven.io > ...

Ok I get it: SASL must be enabled on controllers listener too. Not only the brokers. ACL works perfectly now. Seems to be a lack of documentation (?) Le mer. 30 août 2023 à 11:10, charlie vuillemez < charlie.vuillemez@gmail.com > a écrit : > Hi all, > > Maybe a bug... but I'm facing an issue when trying to add a Kafka ACL > (Kafka is confluent version 7.5.0-ccs). > > First, I can list ACLs (so that's why I think my client and broker > configuration is ok). > Result is empty, cause I have no ACLs: > > $ kafka-acls --bootstrap-server broker01:9093 --command-config > /etc/kafka/client-ssl.properties --list > > Here is the /etc/kafka/client-ssl.properties: > > sasl.mechanism=PLAIN > security.protocol=SASL_SSL > ssl.truststore.type=PEM > ssl.truststore.location=/etc/kafka/ssl/truststore.pem > sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule > requir...

Hi all, Maybe a bug... but I'm facing an issue when trying to add a Kafka ACL (Kafka is confluent version 7.5.0-ccs). First, I can list ACLs (so that's why I think my client and broker configuration is ok). Result is empty, cause I have no ACLs: $ kafka-acls --bootstrap-server broker01:9093 --command-config /etc/kafka/client-ssl.properties --list Here is the /etc/kafka/client-ssl.properties: sasl.mechanism=PLAIN security.protocol=SASL_SSL ssl.truststore.type=PEM ssl.truststore.location=/etc/kafka/ssl/truststore.pem sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="admin" \ password="xxxxxxxx"; When adding the first ACL, I get the following exception: kafka-acls --bootstrap-server broker01:9093 --command-config /etc/kafka/client-ssl.properties --add --allow-principal "User:test" --topic "test" --operation Read Adding ACLs for resource `ResourcePattern(res...

Hello! I'm looking for opensource solutions for consuming Kafka messages in browser. I found to interesting articles 1. https://www.confluent.io/blog/consuming-messages-out-of-apache-kafka-in-a-browser/ 2. https://ably.com/topic/websockets-kafka Unfortunately: - the first is from 2019 and does not provide any reference to solution and a github repository. - the second one is a proprietary solution of Ably. Can you please advice what open source solutions are using for consuming Kafka messages in browser? Thank you, Florin

Hey Hemanth! Thank you for asking about Mirror Maker 2! Offset translation is not so simple, so I'll summarize the main functionality and leave some pointers into the code for you to examine yourself. 1. After MirrorSourceTask writes a record, it receives a commitRecord callback [1] with information about what offset the record has in the destination. 2. The MirrorSourceTask sends this information (an OffsetSync) [2] to a persistent topic (the Offset Syncs topic.) 3. The MirrorCheckpointTask uses an OffsetSyncStore to read the Offset Syncs from the Offset Syncs topic [3] and store them in memory. 4. The OffsetSyncStore provides a translateDownstream method that reads from memory and translates the offset [4]. 5. The translation picks an offset sync before the offset being translated, to obtain a downstream offset which must be earlier than where the requested offset could be written. This changed recently in KAFKA-12468. 6. The MirrorCheckpointTask uses the tra...

Hi, We're using Mirror Maker 2 for replicating messages from our primary to secondary cluster, it's one way replication. We're also replicating by the consumer group offsets by adding the below properties sync.group.offsets.enabled=true sync.group.offsets.interval.seconds=5 emit.checkpoints.interval.seconds=5 Wanted to know how the translation of consumer group happens from source to destination cluster, especially need to know "how current offset and log end offset translation happens from source to destination kafka cluster" Thanks in advance. -- Regards, Hemanth Savasere

Hello! Registration is still open for the upcoming Community Over Code NA event in Halifax, NS! We invite you to register for the event https://communityovercode.org/registration/ Apache Committers, note that you have a special discounted rate for the conference at US$250. To take advantage of this rate, use the special code sent to the committers@ list by Brian Proffitt earlier this month. If you are in need of an invitation letter, please consult the information at https://communityovercode.org/visa-letter/ Please see https://communityovercode.org/ for more information about the event, including how to make reservations for discounted hotel rooms in Halifax. Discounted rates will only be available until Sept. 5, so reserve soon! --Rich, for the event planning team

Hello Greg Thanks a *lot* for your help on this. Indeed the empty poll is not the issue for us. As mentioned, our setup is a poll every 24 hours. So the `stop()` being stuck due to the `poll()` is hitting us hard. I did a trace today on my dev environment, I can indeed see this waiting log entry every 100 ms < https://github.com/confluentinc/kafka-connect-jdbc/blob/master/src/main/java/io/confluent/connect/jdbc/source/JdbcSourceTask.java#L427 >. Then I call a DEL on this connector, and the stop is not processed until the next loop in the `poll()`. Your initial diagnosis is 100% correct. You mentioned other connectors already changed to support a stop signal from the same thread. Would u have any concrete connector impl to point me? Kind regards Robson On Mon, 21 Aug 2023 at 19:23, Greg Harris <greg.harris@aiven.io.invalid> wrote: > Hey Robson, > > Thanks for opening an issue on the JDBC repo, I think this is > certainly relevant f...

Hey Robson, Thanks for opening an issue on the JDBC repo, I think this is certainly relevant feedback for the connector developers. I commented on the issue with a potential regression that I saw, you can try downgrading your connector to see if the behavior improves. I also know that kafka-connect-jdbc received a patch to improve this behavior when no data is being emitted: https://github.com/confluentinc/kafka-connect-jdbc/pull/947 but I'm not sure if that is relevant to your situation. Thanks! Greg On Mon, Aug 21, 2023 at 6:53 AM Robson Hermes < robsonhermes@gmail.com > wrote: > > No, it stops them also. > The problem is precisely what Greg described, now the stop signal comes > from the same thread. So any source task which is running in a blocking way > will not process the stop signal until the current poll finishes. > So would need to patch source jdbc connector. > > On Mon, 21 Aug 2023 at 15:48, sunil chaudhari < su...

How can I stop getting these updates ? On Mon, Aug 21, 2023 at 9:01 AM Robson Hermes < robsonhermes@gmail.com > wrote: > This email was sent from an external source so please treat with caution. > > No, it stops them also. > The problem is precisely what Greg described, now the stop signal comes > from the same thread. So any source task which is running in a blocking way > will not process the stop signal until the current poll finishes. > So would need to patch source jdbc connector. > > On Mon, 21 Aug 2023 at 15:48, sunil chaudhari < sunilmchaudhari05@gmail.com > > > wrote: > > > I think when you delete connector it removes the task and workers > continues > > to run. > > When you stop it actually stops the worker. > > Both different things. > > Point to be noted is Worker has connector. > > So connector should be removed before stopping the worker. > > > > ...

No, it stops them also. The problem is precisely what Greg described, now the stop signal comes from the same thread. So any source task which is running in a blocking way will not process the stop signal until the current poll finishes. So would need to patch source jdbc connector. On Mon, 21 Aug 2023 at 15:48, sunil chaudhari < sunilmchaudhari05@gmail.com > wrote: > I think when you delete connector it removes the task and workers continues > to run. > When you stop it actually stops the worker. > Both different things. > Point to be noted is Worker has connector. > So connector should be removed before stopping the worker. > > Though I am not expert in this. > > On Mon, 21 Aug 2023 at 7:10 PM, Robson Hermes < robsonhermes@gmail.com > > wrote: > > > Hello Sunil > > > > I'm not calling a stop, I'm straight deleting the connectors with the > > DELETE. Stopping the connector is done...

I think when you delete connector it removes the task and workers continues to run. When you stop it actually stops the worker. Both different things. Point to be noted is Worker has connector. So connector should be removed before stopping the worker. Though I am not expert in this. On Mon, 21 Aug 2023 at 7:10 PM, Robson Hermes < robsonhermes@gmail.com > wrote: > Hello Sunil > > I'm not calling a stop, I'm straight deleting the connectors with the > DELETE. Stopping the connector is done internally during deletion. > > Regards > Robson > > On Mon, 21 Aug 2023 at 15:36, sunil chaudhari < sunilmchaudhari05@gmail.com > > > wrote: > > > You have to remove connectors first using delete api > > and then stop the connector > > > > On Thu, 17 Aug 2023 at 2:51 AM, Robson Hermes < robsonhermes@gmail.com > > > wrote: > > > > > Hello > > > > ...

Hello Sunil I'm not calling a stop, I'm straight deleting the connectors with the DELETE. Stopping the connector is done internally during deletion. Regards Robson On Mon, 21 Aug 2023 at 15:36, sunil chaudhari < sunilmchaudhari05@gmail.com > wrote: > You have to remove connectors first using delete api > and then stop the connector > > On Thu, 17 Aug 2023 at 2:51 AM, Robson Hermes < robsonhermes@gmail.com > > wrote: > > > Hello > > > > I'm using kafka connect 7.4.0 to read data from Postgres views and write > to > > another Postgres tables. So using JDBC source and sink connectors. > > All works good, but whenever I stop the source connectors via the rest > api: > > > > DEL http://kafka-connect:8083/connectors/connector_name_here > > > > The connector stops fine, but not the task: > > > > > > Graceful stop of connector (connector-nam...

You have to remove connectors first using delete api and then stop the connector On Thu, 17 Aug 2023 at 2:51 AM, Robson Hermes < robsonhermes@gmail.com > wrote: > Hello > > I'm using kafka connect 7.4.0 to read data from Postgres views and write to > another Postgres tables. So using JDBC source and sink connectors. > All works good, but whenever I stop the source connectors via the rest api: > > DEL http://kafka-connect:8083/connectors/connector_name_here > > The connector stops fine, but not the task: > > > Graceful stop of connector (connector-name-here) succeeded. > > Graceful stop of task (task-name-here) failed. > > > It only happens with the *source* connector tasks. The sink connector > and tasks shutdown gracefully and fine. > > The timeout for task shutdown has been increased, but didn't help: > > task.shutdown.graceful.timeout.ms =60000 > > > > The co...

Hello Greg (sorry about the duplicate e-mail, forgot to cc users mailing list) Thanks a lot for your detailed reply. I'm using JDBC Source connectors from kafka-connect-jdbc < https://github.com/confluentinc/kafka-connect-jdbc >. Indeed the `poll()` implementation is blocked, so it only processes a `stop()` when it returns from the current `poll()`execution. There was a change in the past to fix a similar problem < https://github.com/confluentinc/kafka-connect-jdbc/pull/677 >, but not involving `stop()` from the same thread. I've just raised one < https://github.com/confluentinc/kafka-connect-jdbc/issues/1360 >. Unfortunately setting a lower poll interval is not an option for me, as this is being used in a heavy data load operation, being executed only once per day. Will see if I'm able to come up with a change, although not sure yet. First time using kafka, kafka connect and kafka connect jdbc =D Kind regards Robson On Thu, 17 A...

@matthias what are the alternatives to get rid of this issue? When the lag starts increasing, we have alerts configured on our monitoring system in Datadog which starts sending alerts and alarms to reliability teams. I know in kafka the inactive consumer group is cleared up after 7 days however not sure if that is the case with topics that were consumed previously and not consumed now. Does creation of new consumer group (setting a different application.id ) on streams application an option here? On Thu, Aug 17, 2023 at 7:03 AM Matthias J. Sax < mjsax@apache.org > wrote: > Well, it's kinda expected behavior. It's a split brain problem. > > In the end, you use the same ` application.id / group.id ` and thus the > committed offsets for the removed topics are still in > `__consumer_offsets` topics and associated with the consumer group. > > If a tool inspects lags and compares the latest committed offsets to > end-offsets i...

Well, it's kinda expected behavior. It's a split brain problem. In the end, you use the same ` application.id / group.id ` and thus the committed offsets for the removed topics are still in `__consumer_offsets` topics and associated with the consumer group. If a tool inspects lags and compares the latest committed offsets to end-offsets it looks for everything it finds in the `__consumer_offsets` topics for the group in question -- the tool cannot know that you changed the application and that is does not read from those topics any longer (and thus does not commit any longer). I am not sure from top of my head if you could do a manual cleanup for the ` application.id ` and topics in question and delete the committed offsets from the `__consumer_offsets` topic -- try to checkout `Admin` client and/or the command line tools... In know that it's possible to delete committed offsets for a consumer group (if a group becomes inactive, the broker w...

Hi Robson, Thank you for the detailed bug report. I believe the behavior that you're describing is caused by this flaw: https://issues.apache.org/jira/browse/KAFKA-15090 which is still under discussion. Since the above flaw was introduced in 3.0, source connectors need to return from poll() before the graceful shutdown timeout to avoid the error. You may be able to mitigate the error if the connector allows you to reduce its poll timeout/interval to something less than the graceful timeout, but that will depend on the specific connector implementation, so check the documentation for your connector. I know some implementations have received patches to compensate for this behavior in the framework, so also consider upgrading or checking release notes for your connectors. As for the effects of this error: whenever a non-graceful stop occurs, the runtime will immediately close the producer so that the task will not be able to write any further records. However, i...

Hello I'm using kafka connect 7.4.0 to read data from Postgres views and write to another Postgres tables. So using JDBC source and sink connectors. All works good, but whenever I stop the source connectors via the rest api: DEL http://kafka-connect:8083/connectors/connector_name_here The connector stops fine, but not the task: Graceful stop of connector (connector-name-here) succeeded. Graceful stop of task (task-name-here) failed. It only happens with the *source* connector tasks. The sink connector and tasks shutdown gracefully and fine. The timeout for task shutdown has been increased, but didn't help: task.shutdown.graceful.timeout.ms =60000 The connectors are running once per day (during the night) to load a lot of data, and the error happens when I try to delete the connectors in the middle of the day. That is, they are not actually executing/loading any data, it has finished already. offset.flush.interval.ms =10000 in develop...

Hey Folks, I have a process which aims to measure time lag (in Scala). When the process bootstraps it looks at the history of offsets and collect the offset that existed for different timestamps (7 days ago, 6 days ago... etc in more frequency as it gets closer to *now*). In order to do that it uses the "consumer.offsetsForTimes" method and keeps that information in memory. I observed that in some cases, mainly in *some* compacted topics partitions (observed on a few partitions from __consumer_offsets and __transaction_state) that the resulted offsets arent monotonically increasing as times are getting closer to now. In these specific cases (again, not all partitions) I see for example (a few data points from many): time (in seconds)=1691512025 offset=16101908 1691550724/15121538 1691645078/15473125 1691789229/15473125 1692104539/16078952 1692116770/16101908 1692116809/16101908 1692116833/16101908 .. .. Code looks like: BootstrapTimes.reve...

Hi streams Dev community @matthias, @bruno Any inputs on above issue? Is this a bug in the streams library wherein the input topic removed from streams processor topology, the underlying consumer group still reporting lag against those? On Wed, Aug 9, 2023 at 4:38 PM Pushkar Deole < pdeole2015@gmail.com > wrote: > Hi All, > > I have a streams application with 3 instances with application-id set to > applicationV1. The application uses processor API with reading from source > topics, processing the data and writing to destination topic. > Currently it consumes from 6 source topics however we don't need to > process data any more from 2 of those topics so we removed 2 topics from > the source topics list. We have configured Datadog dashboard to report and > alert on consumer lag so after removing the 2 source topics and deploying > application, we started getting several alerts about consumer lag on > applicationV1 consumer...

Hi Neil, You should be good to go now. Thanks for your interest in contributing to Apache Kafka! Cheers, Chris On Tue, Aug 15, 2023 at 12:28 PM Neil Buesing < buesing.dev@gmail.com > wrote: > Looking to make the minor fix to the documentation for a bug I reported , > KAFKA-13945, so I need to get my JIRA ID added. > > username: nbuesing > > Thanks, > Neil > > >if you have not been added to the list, send an email to the users@kafka > mailing list to request for it). >

Looking to make the minor fix to the documentation for a bug I reported , KAFKA-13945, so I need to get my JIRA ID added. username: nbuesing Thanks, Neil >if you have not been added to the list, send an email to the users@kafka mailing list to request for it).

Hey, I want to set up Kafka with SASL_SSL in a docker enviroment kafka should be albe to recives message encrypted over the puplic internet in addition, telegraf grafana and more are used in the backend everything runs flawlessly only the zookeeper and kafka should be allowed to communicate on the internet kafka should be encrypted with SASL_SSL SCRAM-SHA-512 the connection between kafka and zookeeper should run via DIGEST-MD5 but i can't find a solution because i always get the following errors ERROR SASL authentication failed using login context 'Client'. (org.apache.zookeeper.client.ZooKeeperSaslClient ERROR Error occurred while connecting to Zookeeper server[zookeeper:2181,zookeeper:2182]. Authentication failed. javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null. i also add the full log ===> Configuring ... Running in Zookeeper mode... SSL is enabled. SAS...

Hello Team, I am trying to sync 2 kafka clusters using mirrormaker 2. But only replicate latest data coming to source topics not sure how to make it replicate from the starting of the topic partitions instead of latest offsets.? in mirrormaker 1 we can do that with `auto.offset.reset` consumer setting however not sure how to achieve same with mirrormaker 2 . Could you please guide me on that. Thank you, Uday

When you said "The only difference we could see is that thread usage decreases during these period", did you mean thread usage increases? You can monitor the usage of two different thread pools, network thread pool and requestHandler thread pool. If none of them are high and yet, you have a large CPU spike, then it is probably due to background threads responsible for cleaning/compaction or some other JVM process that runs on your broker. If network pool is high, then it's probably due to large number of requests. If requestHandler is high then a small number of requests are causing large CPU spikes while processing. Some reasons that can happen is if you send data in format V1 but server expects in V2 and hence, CPU will be spent in converting from V1 to V2 format on the server. This becomes worse when data is compressed because the server has to do decompression-compression-decompression-compression before actually writing to the disk. Another reason could...

Hi Romulo, You should be able to assign tasks to yourselves. Thanks. Luke On Sat, Aug 5, 2023 at 9:10 PM Romulo Oliveira < romulosnoliveira@gmail.com > wrote: > I added a new Jira task in order to contribute to a new feature for Kafka > Connectors TimestampConverter transformer: > https://issues.apache.org/jira/browse/KAFKA-15310 . > I intend to work on it, but I do not have permission to assign this task to > my user. Can someone add the permission or at least the assign the task to > me? > > Romulo Souza >

Point 2 may impact if the size of partitions is too big. too many log segments will cause those many iops I am not expert though On Wed, 9 Aug 2023 at 6:43 PM, Tiansu Yu <tiansu.yu@klarna.com.invalid> wrote: > 1. We use cruise-control to actively balance the partitions across all > brokers. So point 1 could be ruled out. > 2. I am not sure how much this would impact the broker, as we do have some > exceptionally large partitions around. I have to check to know if they live > on the aforementioned broker. So far I don't see there is strong > correlation between total producer / consumer byte rates with CPU spikes on > this broker. > > Tiansu Yu > Engineer > Data Ingestion & Streaming > > Klarna Bank AB German Branch > Chausseestraße 117 > < https://www.google.com/maps/search/Chausseestra%C3%9Fe+117+10115+Berlin?entry=gmail&source=g > > 10115 Berlin > < https://www.google.com/maps/s...

The Kafka version is 3.2. Replication factor is defined per topic, but minimum 3. We have no control over every producer, but we do recommend our users to set acks as all.  Another angle we are considering is that there might be some misbehaving clients using Kafka low level APIs that might not show up on the metrics, e.g. Spark Streaming or Flink.  Tiansu Yu Engineer Data Ingestion & Streaming Klarna Bank AB German Branch Chausseestraße 117 10115 Berlin Tel: +49 221 669 501 00 klarna.de Klarna Bank AB, German Branch Sitz: Berlin, Amtsgericht Charlottenburg HRB 217291 B USt-Nr.: DE 815 867 324   Zweigstelle der Klarna Bank AB (publ), AG schwedischen Rechts mit Hauptsitz in Stockholm, Schw. Gesellschaftsregister 556737-0431 Verwaltungsratsvorsitzender: Michael Moritz Geschäftsführender Direktor: Sebastian Siemiatkowski Leiter Zweigniederlassung: Yaron Shaer, Björn Petersen On 9. Aug 2023, at 15:15, Samuel Cantero <scanterog@gmail.com> wrote: Which...