Hi,
Our application uses kafka streams that reads from a source topic, does
processing on records and produces processed record on destination topic
through the use of external producer i.e. the producer created via kafka
producer API.
Does this model still guarantee exactly once semantic or it won't?
Currently we are using at_least_once, however the question is how streams
handles offset commits here?
Though the event is produced using synchronous API, could there be
possibility of event loss in case streams commit offset before external
producer hasn't produced event on destination topic yet?
Posts
On Tue, Jun 29, 2021 at 5:45 PM Péter Sinóros-Szabó
<peter.sinoros-szabo@wise.com.invalid> wrote:
> Hey,
>
> we had the same issue as you.
>
> I checked the code and it chooses the first live replica from the
> assignment list. So if you describe a topic with kafka-topics, you will see
> the brokers list that has the replica of each partition. For example:
> [1001, 1002, 1003]. If that is the list, Kafka will choose the first
> replica that is available (is online) in that list.
>
That was our understanding of the relevant code as well (unless the
assignment sequence is ordered in a way that most-in-sync replica goes
first, which is doubtful):
def offlinePartitionLeaderElection(assignment: Seq[Int], isr: Seq[Int],
liveReplicas: Set[Int], uncleanLeaderElectionEnabled: Boolean,
controllerContext: ControllerContext): Option[Int] = {
assignment.find(id => liveReplicas.contains(id) &&
isr.contains(id...
Hey,
we had the same issue as you.
I checked the code and it chooses the first live replica from the
assignment list. So if you describe a topic with kafka-topics, you will see
the brokers list that has the replica of each partition. For example:
[1001, 1002, 1003]. If that is the list, Kafka will choose the first
replica that is available (is online) in that list.
We use "acks=all" and "min.insync.replicas=2", so that should mean that
even if the leader is down and the rest of the replicas fall out of the
ISR, one of the follower replicas should have up to date data. You can
compare the two follower replicas with kafka-dump-tool to see which are
more up-to-date. If you run a partition reassignment, you can change the
order of the followers in the assignment list and then trigger an unclean
leader election for the reassigned partitions. So it seems that this way,
assuming the use of "acks=all" and "min.insync.replicas=2", we c...
Hi All,
I am trying to replicate my kafka cluster to another one using mirror maker
2. This is uni-directional replication. Both these clusters are at
different locations.
This replication happens fine when the load is low. However, when the load
increases I usually get timeout error in my logs and it never comes out of
this error.
To fix this problem, I tried reduce the value of buffer.memory in the mm
config, and even though it reads the property from the config and initially
prints the updated value in the log in the producer config section, the
default value of 32 MB is being used.
Can you please suggest on how to get this working?
Thanks a lot for your help.
In the docs, section "5.4 Log", there is a diagram with the title "Kafka
Log Implementation". The right column of the diagram depicts "Segment
Files": "topic/34477849968.kafka" at the top, "topic/82796232652.kafka"
at the bottom. Inside the file boxes, the individual messages are shown:
"Message 34477849968", ...
Surprisingly, the box at the top shows the same messages as the box at
the bottom.
Is this just a copy-past-error? And the first message inside the box at
the bottom should be "Message 82796232652"?
This looks like a. list of trusted root certs to me. Did you configure the
ssl keystore, trust store correctly? did you use truststore as key store?
for example?
[image: Confluent] < https://www.confluent.io >
Wu Shilin
Solution Architect
+6581007012
Follow us: [image: Blog]
< https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog >[image:
Twitter] < https://twitter.com/ConfluentInc >[image: LinkedIn]
< https://www.linkedin.com/company/confluent/ >[image: Slack]
< https://slackpass.io/confluentcommunity >[image: YouTube]
< https://youtube.com/confluent >
[image: Kafka Summit] < https://www.kafka-summit.org/ >
On Mon, Jun 28, 2021 at 6:30 PM meghna murthy < meghna.agni@gmail.com > wrote:
> Hi Team ,
>
> Please help in resolving the below issue why Kafka is not sending the
> certificate which is present in it's trustedstor...
Hi Team , Please help in resolving the below issue why Kafka is not sending the certificate which is present in it's trustedstore during Certificate Request to client in the Distinguished names. Is kafka will send it default certificate entries if yes is there any way to override in ssl properties ?? Regards, Meghana On Fri, 25 Jun 2021 at 12:18, meghna murthy < meghna.agni@gmail.com > wrote: Hi Team , When ssl.client.auth=required is set , Srver is sending Certificate request with DN with junk certificates to client . Server has to send what certificates we have deployed since its sending junk certificate handshake is failing. Attaching the screenshot of junk certificates sent in Certificate request. Are these default certificate sent by kafka ?? I have attached pcap also for the same Please help in resolving issue. Regards, Meghna
On Mon, Jun 21, 2021 at 12:33 PM Oleksandr Shulgin <
oleksandr.shulgin@zalando.de > wrote:
>
> In summary: is there a risk of data loss in such a scenario? Is this
risk avoidable and if so, what are
> the prerequisites?
Apologies if I messed up line breaks and that made reading harder. O:-)
The question boils down to: is replica selection completely random in case
of unclean leader election or not?
Regards,
--
Alex
Hello Nitay,
I have not heard someone else reporting similar things that may point to a
bug still.. Maybe you could try to reproduce the issue by first starting a
brand new app in 2.5, and then follow the upgrade path (with config
overrides) to 2.6 and see if it is easily reproducible, and if yes create a
JIRA ticket?
Guozhang
On Wed, Jun 23, 2021 at 3:50 PM Nitay Kufert <nitay.k@is.com.invalid> wrote:
> Bumping for the off chance that during this time some sort of a bug was
> reported that might explain this behaviour.. i will feel more comfortable
> bumping our kafka versions this way :)
>
> On Wed, Feb 24, 2021 at 12:48 PM Nitay Kufert < nitay.k@ironsrc.com > wrote:
>
> > I guess it's possible but very unlikely because it works perfectly with
> > all the previous versions and the current one? (2.5.1)
> > Why did a change in the version introduce NULLS there?
> >
> > On Tue, Feb 23, 2021 at 9:1...
Hello Alan,
I've added you to the contributors list and also assigned the ticket to you.
Cheers,
Guozhang
On Fri, Jun 25, 2021 at 5:19 PM Alan Artigao < alanartigao@gmail.com > wrote:
> Hi there!
>
> I'm working on KAFKA-12995
> < https://issues.apache.org/jira/browse/KAFKA-12995# > and I need to be in
> the contributors list of Apache Kafka in order to assign it to me.
>
> Username in JIRA is: *aartigao*
>
> Cheers!
>
--
-- Guozhang
Hello Miguel and Samson,
Just to add to what Dave just stated and to summarize the use cases a bit:
The Kafka documentation has a good summary here with the APIs that are
provided within the project
https://kafka.apache.org/documentation/#api
To add to what's in the documentation, we have a couple of scenarios for
you to consider:
(a) Just fetching events from a kafka topic and processing it using your
own mechanisms and logic. This is when you use the Consumer API
https://kafka.apache.org/documentation/#consumerapi
With this API, you provide the logic to filter, process or merge events
across multiple topics if you need to
You can also use the Confluent Parallel Consumer for this, especially when
you don't want to limit your scaling capabilities to the number of
partitions in the topics you are consuming from.
https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/
(b) Fetching data from one or more Kafka topics, j...
Yes, Kafka Consumer and Kafka Streams are just libraries. My point with that, is that it's not difficult to switch from one to the other as your needs evolve.
There are several ways that Kafka Streams aids in processing. It provides a rich set of functions for transforming, filtering, branching, etc. Also it manages state for any stateful processing, like aggregations, joins, etc. If you don't need any of these and are just consuming events and writing them to a database, Kafka Consumer will work fine. But if your needs change, you can switch to Kafka Streams later.
Also, if you really are just consuming to write to a DB, you may want to consider Kafka Connect.
Let me know if this is unclear.
Thanks,
Dave
> On Jun 26, 2021, at 7:05 AM, SuarezMiguelC <SuarezMiguelC@protonmail.com.invalid> wrote:
>
> DaveKlein, in the reply email of "Kafka Streams" on the question to use Kafka Streams or just a consumer you specified:
>
...
DaveKlein, in the reply email of "Kafka Streams" on the question to use Kafka Streams or just a consumer you specified:
> But Streams and Consumer are just libraries, so start with Consumer and if you find yourself doing more processing, consider moving to Kafka Streams.
I though Kafka Consumer was also just a library, and didn't knew Streams helped with processing, can you elaborate on this?
Thanks for sharing your knowledge!
Miguel Suárez
If you are not doing any transformation or filtering of the data before writing it to the db, then you're probably better off with a consumer or Kafka Connect.
Kafka Streams shines when you have processing to do on the data as it is consumed. Especially if you do any stateful transformations, since Streams will safely manage that state for you.
But Streams and Consumer are just libraries, so start with Consumer and if you find yourself doing more processing, consider moving to Kafka Streams.
Either way, it's a lot of fun!
Dave
> On Jun 25, 2021, at 5:09 PM, Samson Adeyemi < adesamson77@gmail.com > wrote:
>
> Hi,
>
> I'm looking over your documentation at docs.confluent.io on Kafka Streams,
> but I couldn't find the information I'm looking for. What I need to know
> are:
>
> 1. When should one use Kafka Stream instead of regular Kafka Consumer?
> 2. If I'm not performing data transformation, but ...
Hi,
Done. Thanks for your interest in Apache Kafka!
-Bill
On Wed, Jun 23, 2021 at 6:51 PM code@uweeisele.eu < code@uweeisele.eu > wrote:
> Hello together,
>
> could you please add user ueisele to the list of contributors in the
> Apache Kafka Jira?
>
> Thanks
>
Hi,
I'm looking over your documentation at docs.confluent.io on Kafka Streams,
but I couldn't find the information I'm looking for. What I need to know
are:
1. When should one use Kafka Stream instead of regular Kafka Consumer?
2. If I'm not performing data transformation, but simply reading from Kafka
and saving the data to a DB, should I use Kafka Stream? What advantages do
I have over Kafka Consumer?
3. How does Kafka Stream provide fault tolerance over regular Kafka
Consumer?
Please advise. Thanks.
Samson
I think similar issue is being discussed in other email thread. On Fri, 25 Jun 2021 at 6:09 PM, meghna murthy < meghna.agni@gmail.com > wrote: Hi Team , When ssl.client.auth=required is set , Srver is sending Certificate request with DN with junk certificates to client . Server has to send what certificates we have deployed since its sending junk certificate handshake is failing. Attaching the screenshot of junk certificates sent in Certificate request. Are these default certificate sent by kafka ?? I have attached pcap also for the same Please help in resolving issue. Regards, Meghna
Hi there!
I'm working on KAFKA-12995
< https://issues.apache.org/jira/browse/KAFKA-12995# > and I need to be in
the contributors list of Apache Kafka in order to assign it to me.
Username in JIRA is: *aartigao*
Cheers!
Thank you Luke, it makes sense.
I have made the update on my application. Thanks all for your feedback!
On 2021/06/24 02:26:49, Luke Chen < showuon@gmail.com > wrote:
> Hi Tao,
> The Round-Robin assignor is OK, for sure.
> But since the *StickyAssignor* doesn't get affected by this bug, I'd
> suggest you use it. After all, the StickyAssignor will have better
> performance because it preserves the existing assignments as much as
> possible to reduce the overheads to re-assign the topic partitions/tasks.
>
> Thank you.
> Luke
>
> On Thu, Jun 24, 2021 at 10:13 AM Tao Huang < sandy.huangtao@gmail.com > wrote:
>
> > Thank you Sophie and Luke for the confirmation.
> >
> > @Luke, the reason I think the assignor strategy may not play an important
> > role in my application is that, my application workflow does not rely on
> > partition assigned, what it does is just to poll the eve...
Hi,
We have a 13 node Kafka cluster and each broker has multiple disks and all
topics have replication factor 3.
Broker 6 had a hardware issue and required a complete OS reload (Linux) and
2 disk replacements. Now I installed Kafka again on this node with the same
broker id 6 but started to get an exception from all producers -
*[Error 6] NotLeaderForPartitionError:
ProduceResponsePayload(topic=u'amyTopic', partition=7, error=6, offset=-1)*
I am assuming that since I am using the same broker ID, it (zookeeper?
or controller broker?) is expecting data in the disk which got replaced or
some other meta info that might get wiped out during OS reload. What are
the options I have to *add this node back to the cluster* without much
disturbance to the cluster and without data loss? Should I use a new broker
ID for this node and then repartition the data of every topic as we do
after adding a new node? We have a lot of data (a few hundred TB) in the
cluster and I ...
Yes , already had SSL configuration in server.properties
On Fri, Jun 25, 2021, 14:03 Ran Lupovich < ranlupovich@gmail.com > wrote:
> You open seperate topics with no reason... I feel you are spamming the
> mailing list .... your need to share more information to be able to
> investigate and help you... is your server.properties defines the SSL
> enabled in your port you are trying to connect? You need to setup step by
> step according to manauls and it work for you... it is not just client
> change that need to be done , it needs server change to enable the ssl on
> that port.
>
> בתאריך יום ו׳, 25 ביוני 2021, 11:29, מאת Anjali Sharma <
> sharma.anjali.2699@gmail.com >:
>
> > 1.Was trying for mtls by setting SSL.client.auth=required
> > 2. Had imported the trustore, keystore and everything on client side
> > 3. Need to consume messages on client which we are not able to see
> >
> > Can you...
You open seperate topics with no reason... I feel you are spamming the
mailing list .... your need to share more information to be able to
investigate and help you... is your server.properties defines the SSL
enabled in your port you are trying to connect? You need to setup step by
step according to manauls and it work for you... it is not just client
change that need to be done , it needs server change to enable the ssl on
that port.
בתאריך יום ו׳, 25 ביוני 2021, 11:29, מאת Anjali Sharma <
sharma.anjali.2699@gmail.com >:
> 1.Was trying for mtls by setting SSL.client.auth=required
> 2. Had imported the trustore, keystore and everything on client side
> 3. Need to consume messages on client which we are not able to see
>
> Can you help with this?
>
>
> On Fri, Jun 25, 2021, 13:54 M. Manna < manmedia@gmail.com > wrote:
>
> > 1. What is it that you've tried ?
> > 2. What config changes have you made?
> ...