Hello All,
Any insights on this issue?
Thanks,
Kiran
On Sun, 23 Apr 2023 at 1:58 PM, kiran kumar < kiran.cse507@gmail.com > wrote:
> Hello Kafka Folks,
>
> I have encountered an issue with Kafka 3.2 version.
> Environment:
> OS: RHEL 7.9
> Java: 1.8
> Kafka: 3.2
>
> There seems to one of the replica is stuck in __consumer_offsets topic.
>
> I have the replication factor set as 3 and its 5 node kafka cluster. I
> could see that out of 2 replicas one replica is not getting in sync with
> rest of the two replicas. I have trued to delete the partition folder and
> performed the rolling restart of Kafka. I have also tried to reassign the
> partition to another bode but that node as well in same state. I have tried
> to expand the replicas to 5 partition reassignment tool but all the new
> replicas are at the same state.
> Even though, there two nodes which are good, the third node is unable to
> co...
Posts
Hi:
When I am running MM2 (3.4.0), any new message/topic got synced immediately.
However, the group offset takes long time to sync, (around 10m), is there any setting I am missing?
I have following setting for the sync:
enabled: true
topics: NASHCM.KAFKA.MIGRATION.OFFSET.TEST
sync.group.offsets: true
Global setings:
sync.topic.acls.enabled: false
tasks.max: 10
emit.checkpoints.enabled: true
sync.group.offsets.enabled: true
replication.policy.separator:
Thanks
Andrew
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachmen...
Hello Kafka users,
I'm facing an issue with a Kafka cluster, specifically with the
__consumer_offsets topic.
There seems to be an imbalance in the number of commit messages across its
partitions. Most of the commit messages are concentrated in a single
partition, which is causing high CPU usage on the broker handling that
partition.
I have already verified that the topic partitions' leaders are
well-balanced across the six brokers.
However, a specific consumer group (The largest one, with many members
consuming from multiple topics, based on Spring Kafka) generates a large
number of commit messages, and they all end up in the same partition #37.
My understanding is that, by default, all commit messages sent by a
particular consumer group for a specific topic partition are directed to a
single partition of the __consumer_offsets topic, determined by hashing the
consumer group id and the topic partition. In our case, this default
partitioning strategy seems to b...
Dear Kafka Experts
How can we check for a particular offset number in Apache kafka 3.2.3
version.Could you please share some light.
The kafka_console_consumer tool is throwing class not found error.
./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
--topic your-topic
--group your-consumer-group
--zookeeper localhost:2181
Many thanks Samuel. Will go thru this.
On Tue, Apr 25, 2023 at 9:03 PM Samuel Delepiere <
samuel.delepiere@celer-tech.com > wrote:
> Hi,
>
> I use a combination of the Prometheus JMX exporter (
> https://github.com/prometheus/jmx_exporter ) and the Prometheus Kafka
> exporter ( https://github.com/danielqsj/kafka_exporter ).
> The consumer lag metrics come from the latter.
>
> I can then output the data in Grafana
>
>
> Regards,
>
> Sam.
>
>
>
> On 25 Apr 2023, at 16:26, Kafka Life < lifekafka999@gmail.com > wrote:
>
> Dear Kafka Experts
>
> Could you please suggest good metrics exporter for consumer lag and topic
> level metrics apart from Linkedin kafka burrow for the kafka broker
> cluster.
>
>
>
> *This message, including any attachments, may include private, privileged
> and confidential information and is intended only for the personal and
> confide...
Adding the user-mailing list. Seems relevant to everybody.
On 4/20/23 2:45 AM, Divij Vaidya wrote:
> Thank you Matthias for your comments.
>
> I agree with you that the decision should be driven based on strong
> community ask as it introduces a significant overhead on the maintainers. I
> was hoping that more folks (users of Kafka) would contribute to this thread
> with their opinion but perhaps, I need to find alternative ways to get data
> about Kafka version usage in the wild. Given the effort of migrating major
> versions (2.x to 3.x), I am actually surprised that we don't hear more
> often from the users about the community's 12 month EOL policy.
>
> I will get back on this thread once I have more data to support the
> proposal.
>
> --
> Divij Vaidya
>
>
>
> On Thu, Apr 20, 2023 at 3:52 AM Matthias J. Sax < mjsax@apache.org > wrote:
>
>> While I understand the desire, I ...
Hi Santosh
You can use ASF's self service portal to request an account. It would be
approved by a committer and they will provide you with the necessary
permissions. You can also find this documented at
https://kafka.apache.org/contributing
--
Divij Vaidya
On Tue, Apr 25, 2023 at 7:28 PM Santhosh Kumar < greetsanthosh@gmail.com >
wrote:
> Hi Team,
>
> I would also like to contribute to open source Kafka, can you please add me
> to
> Kafka contributor group.Please find my details below:
>
> Username: Santhosh_Gopal
> Email: greetsanthosh@gmail.com
>
> Thanks
> Santhosh Gopal
>
> On Tue, Apr 25, 2023 at 1:19 PM <info@greenbox-environmental.com.invalid>
> wrote:
>
> > On 2023-04-22 02:58, Luke Chen wrote:
> > > Hi Atul,
> > >
> > > I've added you to the contribution group in JIRA.
> > > I didn't find your account in wiki system.
> ...
Yes, it is possible to have a separate Kraft controller from the brokers in
Apache Kafka. The Kraft controller is responsible for managing the overall
state of the Kafka cluster, such as managing topics, partitions, and
replica assignments.
To set up a separate Kraft controller, you will need to configure the Kafka
brokers to point to the new controller using the controller.quorum.voters
property in the broker configuration file. This property should be set to a
comma-separated list of hostname:port pairs that correspond to the Kraft
controller instances.
Additionally, you will need to configure the Kraft controller instances to
communicate with each other using the controller.quorum.type property in
the controller configuration file. This property should be set to static if
you are using a fixed set of controllers, or zookeeper if you are using
ZooKeeper for controller discovery.
There is documentation available on setting up a separate Kraft controller
in the Apa...
Hi Team,
I would also like to contribute to open source Kafka, can you please add me
to
Kafka contributor group.Please find my details below:
Username: Santhosh_Gopal
Email: greetsanthosh@gmail.com
Thanks
Santhosh Gopal
On Tue, Apr 25, 2023 at 1:19 PM <info@greenbox-environmental.com.invalid>
wrote:
> On 2023-04-22 02:58, Luke Chen wrote:
> > Hi Atul,
> >
> > I've added you to the contribution group in JIRA.
> > I didn't find your account in wiki system.
> > Let me know if you need wiki permission to create a KIP.
> >
> > Thank you.
> > Luke
> >
> > On Fri, Apr 21, 2023 at 10:15 PM Atul Sharma < atulsharmajwr@gmail.com >
> > wrote:
> >
> >> Hi Team,
> >>
> >> I would like to contribute to open source Kafka, can you please add me
> >> to
> >> Kafka contributor group.
> >>
> >> Username: atush...
On 2023-04-19 03:53, Fabio Pardi wrote:
>> Is it possible to have a separate Kraft controller from the brokers?
>> Is there any documentation on setting this up?
>>
>>
>
> Hi,
>
> you should be able to achieve such setup running kafka in 'controller'
> mode.
>
> https://kafka.apache.org/documentation/#brokerconfigs_process.roles
>
>
> regards,
>
> fabio pardi
> Agileos Consulting
Hi,
New information, setup running kafka in 'controller' mode.
Thanks,
On 2023-04-21 15:51, Matthias J. Sax wrote:
> Congrats Mickael!
>
> And thanks a lot for taking on this additional task! Glad to have you!
>
>
> -Matthias
>
> On 4/21/23 9:40 AM, Viktor Somogyi-Vass wrote:
>> Jun, thank you for all your hard work! Also, congrats Mickael, it is
>> very
>> well deserved :)
>>
>> Best,
>> Viktor
>>
>> On Fri, Apr 21, 2023, 18:15 Adam Bellemare < adam.bellemare@gmail.com >
>> wrote:
>>
>>> Thank you for all your hard work Jun - that's a decade-long legacy!
>>> And congratulations to you Mickael!
>>>
>>> On Fri, Apr 21, 2023 at 11:20 AM Josep Prat
>>> <josep.prat@aiven.io.invalid>
>>> wrote:
>>>
>>>> Thanks Jun for your work as Chair all these years!
>>>> Congratulations Mickael!
>>>>
>>>> Best,
>>>...
On 2023-04-22 02:53, Luke Chen wrote:
> Hi Akshay,
>
> Thanks for reporting the issue.
> It looks like a bug.
> Could you open a JIRA < https://issues.apache.org/jira/browse/KAFKA >
> ticket
> to track it?
>
> Thank you.
> Luke
>
>
> On Fri, Apr 21, 2023 at 10:16 PM Akshay Kumar
> <akshaykumar@ameyo.com.invalid>
> wrote:
>
>> Hello team,
>>
>> - We are using the zookeeper less Kafka (kafka Kraft).
>> - The cluster is having 3 nodes.
>> - One of the nodes gets automatically shut down randomly.
>> - Checked the logs but didn't get the exact reason.
>> - Sharing the logs below. Kafka version - 3.3.1
>>
>> *Logs - *
>>
>> [2023-04-13 01:49:17,411] WARN [Controller 1] Renouncing the
>> leadership
>> due to a metadata log event. We were the leader at epoch 37110, but in
>> the
>> new...
On 2023-04-22 02:58, Luke Chen wrote:
> Hi Atul,
>
> I've added you to the contribution group in JIRA.
> I didn't find your account in wiki system.
> Let me know if you need wiki permission to create a KIP.
>
> Thank you.
> Luke
>
> On Fri, Apr 21, 2023 at 10:15 PM Atul Sharma < atulsharmajwr@gmail.com >
> wrote:
>
>> Hi Team,
>>
>> I would like to contribute to open source Kafka, can you please add me
>> to
>> Kafka contributor group.
>>
>> Username: atusharm
>> Email: atulsharmajwr@gmail.com
>>
>> Thanks
>>
Hi Team,
New information, kafka.
Thank
On 2023-04-22 11:23, David Arthur wrote:
> Akshay, this looks a lot like
> https://issues.apache.org/jira/browse/KAFKA-14035 which was fixed for
> 3.3.0. Can you upload complete controller logs to that JIRA (or a new
> one
> if you prefer)?
>
> Thanks!
> David
>
> On Sat, Apr 22, 2023 at 2:54 AM Luke Chen < showuon@gmail.com > wrote:
>
>> Hi Akshay,
>>
>> Thanks for reporting the issue.
>> It looks like a bug.
>> Could you open a JIRA < https://issues.apache.org/jira/browse/KAFKA >
>> ticket
>> to track it?
>>
>> Thank you.
>> Luke
>>
>>
>> On Fri, Apr 21, 2023 at 10:16 PM Akshay Kumar
>> <akshaykumar@ameyo.com.invalid>
>> wrote:
>>
>> > Hello team,
>> >
>> > - We are using the zookeeper less Kafka (kafka Kraft).
>> > - The cluster is having 3 nodes.
>...
On 2023-04-24 11:19, Margaret Figura wrote:
> Hi Arvid,
>
> Just as another user of Kafka, I suspect the claim has to do with the
> producer's ability to perform batching and compression across produced
> data, so it should be more efficient in some cases, especially with
> smaller volumes of data and larger linger.ms settings.
>
> However, like you, I've found that a single KafkaProducer instance can
> sometimes become a bottleneck. For instance, threads can be blocked in
> certain cases when they are producing to the same partition. In our
> application, we found a producer per thread wasn't necessary, but
> adding a small pool of producers (e.g. 4 producers shared by 32
> threads) helped to improve performance. It probably varies quite a bit
> on your use-case and configuration.
>
> I do wonder if the Kafka team would consider it a bug in the cases
> where a single producer isn't faster, or if th...
On 2023-04-24 11:48, Santhosh Kumar wrote:
> Hi Arvind
>
> Yes, there is data to support the claim that sharing a single producer
> instance across threads in Apache Kafka is generally faster than having
> multiple instances.
>
> The reason for this is that a single producer instance can take
> advantage
> of batch processing, which allows it to send multiple messages to Kafka
> in
> a single network request. This reduces the network overhead and can
> result
> in higher throughput.
>
> On the other hand, if multiple producer instances are used, each
> instance
> will have its own network connection to Kafka, resulting in higher
> network
> overhead. Additionally, each instance will have to manage its own queue
> of
> messages to send, which can increase the overall processing overhead.
>
> Several benchmark tests have been conducted to compare the performance
> of
> single ve...
On 2023-04-25 11:32, Samuel Delepiere wrote:
> Hi,
>
> I use a combination of the Prometheus JMX exporter
> ( https://github.com/prometheus/jmx_exporter ) and the Prometheus Kafka
> exporter ( https://github.com/danielqsj/kafka_exporter ).
> The consumer lag metrics come from the latter.
>
> I can then output the data in Grafana
>
> Regards,
>
> Sam.
>
>> On 25 Apr 2023, at 16:26, Kafka Life < lifekafka999@gmail.com > wrote:
>>
>> Dear Kafka Experts
>>
>> Could you please suggest good metrics exporter for consumer lag and
>> topic
>> level metrics apart from Linkedin kafka burrow for the kafka broker
>> cluster.
>
> _This message, including any attachments, may include private,
> privileged and confidential information and is intended only for the
> personal and confidential use of the intended recipient(s). If the
> reader of this message is not...
Hi, I use a combination of the Prometheus JMX exporter ( https://github.com/prometheus/jmx_exporter ) and the Prometheus Kafka exporter ( https://github.com/danielqsj/kafka_exporter ). The consumer lag metrics come from the latter. I can then output the data in Grafana Regards, Sam. On 25 Apr 2023, at 16:26, Kafka Life < lifekafka999@gmail.com > wrote: Dear Kafka Experts Could you please suggest good metrics exporter for consumer lag and topic level metrics apart from Linkedin kafka burrow for the kafka broker cluster.
Dear Kafka Experts
Could you please suggest good metrics exporter for consumer lag and topic
level metrics apart from Linkedin kafka burrow for the kafka broker cluster.
Hi Arvind
Yes, there is data to support the claim that sharing a single producer
instance across threads in Apache Kafka is generally faster than having
multiple instances.
The reason for this is that a single producer instance can take advantage
of batch processing, which allows it to send multiple messages to Kafka in
a single network request. This reduces the network overhead and can result
in higher throughput.
On the other hand, if multiple producer instances are used, each instance
will have its own network connection to Kafka, resulting in higher network
overhead. Additionally, each instance will have to manage its own queue of
messages to send, which can increase the overall processing overhead.
Several benchmark tests have been conducted to compare the performance of
single versus multiple producer instances. For example, one test conducted
by Confluent found that using a single producer instance was up to 50%
faster than using multiple instances.
The...
Hi Arvid,
Just as another user of Kafka, I suspect the claim has to do with the producer's ability to perform batching and compression across produced data, so it should be more efficient in some cases, especially with smaller volumes of data and larger linger.ms settings.
However, like you, I've found that a single KafkaProducer instance can sometimes become a bottleneck. For instance, threads can be blocked in certain cases when they are producing to the same partition. In our application, we found a producer per thread wasn't necessary, but adding a small pool of producers (e.g. 4 producers shared by 32 threads) helped to improve performance. It probably varies quite a bit on your use-case and configuration.
I do wonder if the Kafka team would consider it a bug in the cases where a single producer isn't faster, or if they'd consider it a doc bug where the recommendation should be a bit more nuanced.
Hope that helps,
Meg
-----Original Message-----...
Hi!
I have a question about executing a Kafka producer application, utilizing
multiple threads.
In the documentation for Kafka producers (
https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html )
it says that "...sharing a single producer instance across threads will
generally be faster than having multiple instances.".
I am mainly wondering if there is any data to verify this claim?
The reason for this is that I have carried out rather extensive performance
tests, with varying levels of computational load per message produced,
using between 1 and 24 threads, and it seems that there is really no
situation in which a single, shared producer achieves a higher performance
than
if each executing thread is assigned its own producer instance.
Kind regards,
Arvid
Hello team, We are using the zookeeper less Kafka ( kafka Kraft). The cluster is having 3 nodes. One of the nodes gets automatically shut down randomly. Checked the logs but didn't get the exact reason. Kafka version - 3.3.1 Attaching the log files. Time - 2023-04-21 16:28:23 state-change.log - https://drive.google.com/file/d/1eS-ShKlhGPsIJoybHndlhahJnucU8RWA/view?usp=share_link server.log - https://drive.google.com/file/d/1Ov5wrQIqx2AS4J7ppFeHJaDySsfsK588/view?usp=share_link Regards, Akshay Kumar Senior Software Engineer ll | AMEYO +91-8556063696 Latest from the Ameyo Blog Disclaimer: The information in this communication is confidential and may be legally privileged. It is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking action in reliance of the contents of this information ...
Hello Kafka Folks,
I have encountered an issue with Kafka 3.2 version.
Environment:
OS: RHEL 7.9
Java: 1.8
Kafka: 3.2
There seems to one of the replica is stuck in __consumer_offsets topic.
I have the replication factor set as 3 and its 5 node kafka cluster. I
could see that out of 2 replicas one replica is not getting in sync with
rest of the two replicas. I have trued to delete the partition folder and
performed the rolling restart of Kafka. I have also tried to reassign the
partition to another bode but that node as well in same state. I have tried
to expand the replicas to 5 partition reassignment tool but all the new
replicas are at the same state.
Even though, there two nodes which are good, the third node is unable to
completed sync and partitions to 5 nodes are also stuck at the same state
as the other faulty node.
Is there any way to recover the Kafka partition replica and get it sync
with other two of the good nodes ?
Thanks and regards
Kiran
...
Akshay, this looks a lot like
https://issues.apache.org/jira/browse/KAFKA-14035 which was fixed for
3.3.0. Can you upload complete controller logs to that JIRA (or a new one
if you prefer)?
Thanks!
David
On Sat, Apr 22, 2023 at 2:54 AM Luke Chen < showuon@gmail.com > wrote:
> Hi Akshay,
>
> Thanks for reporting the issue.
> It looks like a bug.
> Could you open a JIRA < https://issues.apache.org/jira/browse/KAFKA > ticket
> to track it?
>
> Thank you.
> Luke
>
>
> On Fri, Apr 21, 2023 at 10:16 PM Akshay Kumar
> <akshaykumar@ameyo.com.invalid>
> wrote:
>
> > Hello team,
> >
> > - We are using the zookeeper less Kafka (kafka Kraft).
> > - The cluster is having 3 nodes.
> > - One of the nodes gets automatically shut down randomly.
> > - Checked the logs but didn't get the exact reason.
> > - Sharing the logs below. Kafka version - 3...
Hi Atul,
I've added you to the contribution group in JIRA.
I didn't find your account in wiki system.
Let me know if you need wiki permission to create a KIP.
Thank you.
Luke
On Fri, Apr 21, 2023 at 10:15 PM Atul Sharma < atulsharmajwr@gmail.com >
wrote:
> Hi Team,
>
> I would like to contribute to open source Kafka, can you please add me to
> Kafka contributor group.
>
> Username: atusharm
> Email: atulsharmajwr@gmail.com
>
> Thanks
>
Hi Akshay,
Thanks for reporting the issue.
It looks like a bug.
Could you open a JIRA < https://issues.apache.org/jira/browse/KAFKA > ticket
to track it?
Thank you.
Luke
On Fri, Apr 21, 2023 at 10:16 PM Akshay Kumar <akshaykumar@ameyo.com.invalid>
wrote:
> Hello team,
>
> - We are using the zookeeper less Kafka (kafka Kraft).
> - The cluster is having 3 nodes.
> - One of the nodes gets automatically shut down randomly.
> - Checked the logs but didn't get the exact reason.
> - Sharing the logs below. Kafka version - 3.3.1
>
> *Logs - *
>
> [2023-04-13 01:49:17,411] WARN [Controller 1] Renouncing the leadership
> due to a metadata log event. We were the leader at epoch 37110, but in the
> new epoch 37111, the leader is (none). Reverting to last committed offset
> 28291464. (org.apache.kafka.controller.QuorumController)
> [2023-04-13 01:49:17,531] INFO [RaftManager nodeId=1] Comple...
Congrats Mickael!
And thanks a lot for taking on this additional task! Glad to have you!
-Matthias
On 4/21/23 9:40 AM, Viktor Somogyi-Vass wrote:
> Jun, thank you for all your hard work! Also, congrats Mickael, it is very
> well deserved :)
>
> Best,
> Viktor
>
> On Fri, Apr 21, 2023, 18:15 Adam Bellemare < adam.bellemare@gmail.com > wrote:
>
>> Thank you for all your hard work Jun - that's a decade-long legacy!
>> And congratulations to you Mickael!
>>
>> On Fri, Apr 21, 2023 at 11:20 AM Josep Prat <josep.prat@aiven.io.invalid>
>> wrote:
>>
>>> Thanks Jun for your work as Chair all these years!
>>> Congratulations Mickael!
>>>
>>> Best,
>>>
>>> ———
>>> Josep Prat
>>>
>>> Aiven Deutschland GmbH
>>>
>>> Alexanderufer 3-7, 10117 Berlin
>>>
>>> Amtsgericht Ch...
Hello team, We are using the zookeeper less Kafka (kafka Kraft). The cluster is having 3 nodes. One of the nodes gets automatically shut down randomly. Checked the logs but didn't get the exact reason. Sharing the logs below. Kafka version - 3.3.1 Logs - [2023-04-13 01:49:17,411] WARN [Controller 1] Renouncing the leadership due to a metadata log event. We were the leader at epoch 37110, but in the new epoch 37111, the leader is (none). Reverting to last committed offset 28291464. (org.apache.kafka.controller.QuorumController) [2023-04-13 01:49:17,531] INFO [RaftManager nodeId=1] Completed transition to Unattached(epoch=37112, voters=[1, 2, 3], electionTimeoutMs=982) (org.apache.kafka.raft.QuorumState) [2023-04-13 02:00:33,902] WARN [Controller 1] Renouncing the leadership due to a metadata log event. We were the leader at epoch 37116, but in the new epoch 37117, the leader is (none). Reverting to last committed offset 28292807. (org.apache.kafka.controller.QuorumController) [2023...
Hi Team,
I would like to contribute to open source Kafka, can you please add me to
Kafka contributor group.
Username: atusharm
Email: atulsharmajwr@gmail.com
Thanks
We have a broker that is part of a 20 node kafka cluster that is dead.
We wanted to rebalance the broken node out of the cluster by using the
kafka-reassign-partitions.sh
We get the following error when we try to reassign partitions:
Error: org.apache.kafka.common.errors.TimeoutException: Timed out waiting
for a node assignment. Call: incrementalAlterConfigs
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a
node assignment. Call: incrementalAlterConfigs
How can we perform a partition reassignment if a node is down/dead?
The Kafka Version we run is: 3.2.0
Thanks
Baargav
Hey Doug
Yes. You can choose to have nodes which act as "controllers-only" by
setting "process.roles=controller". You can find this information at
https://kafka.apache.org/documentation.html#kraft_role
--
Divij Vaidya
On Tue, Apr 18, 2023 at 5:28 PM Doug Whitfield
<dwhitfield@perforce.com.invalid> wrote:
> Hi folks,
>
> Is it possible to have a separate Kraft controller from the brokers? Is
> there any documentation on setting this up?
>
> Thanks,
> Doug Whitfield
>
>
> This e-mail may contain information that is privileged or confidential. If
> you are not the intended recipient, please delete the e-mail and any
> attachments and notify us immediately.
>
>
> Is it possible to have a separate Kraft controller from the brokers? Is there any documentation on setting this up?
>
>
Hi,
you should be able to achieve such setup running kafka in 'controller' mode.
https://kafka.apache.org/documentation/#brokerconfigs_process.roles
regards,
fabio pardi
Agileos Consulting
Hi folks,
Is it possible to have a separate Kraft controller from the brokers? Is there any documentation on setting this up?
Thanks,
Doug Whitfield
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Hi, Lee,
Would you be able to see which Kafka API is generating the traffic?
This is provided by the MBean
kafka.network:type=RequestMetrics,name=RequestsPerSec,request=*,version=([0-9]+)
[1].
Thanks,
Alexandre
[1] https://kafka.apache.org/documentation/#monitoring
Le dim. 16 avr. 2023 à 18:22, Lee Carroll
<lee.a.carroll@googlemail.com.invalid> a écrit :
>
> Hi Kirk,
>
> Points 1 and 2: yes we have both logging and metrics. I'll collate
> Point 3 concurrency refers to the spring boot kafka concurrent container. 2
> threads will be made available to the consumer group to consume 30
> partitions making up a single topic.
> point 4: consumer- Kafka Consumer Configurations for Confluent Platform |
> Confluent Documentation
> < https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#fetch-max-wait-ms >
>
> On Thu, 13 Apr 2023 at 16:03, Kirk True < kirk@kirktrue.p...
Hi Kirk,
Points 1 and 2: yes we have both logging and metrics. I'll collate
Point 3 concurrency refers to the spring boot kafka concurrent container. 2
threads will be made available to the consumer group to consume 30
partitions making up a single topic.
point 4: consumer- Kafka Consumer Configurations for Confluent Platform |
Confluent Documentation
< https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#fetch-max-wait-ms >
On Thu, 13 Apr 2023 at 16:03, Kirk True < kirk@kirktrue.pro > wrote:
> Hi Lee,
>
> Some questions:
>
> 1. Can you enable metrics for your consumer(s)? See [1] and [2] for links
> on consumer-level and fetch-level metrics, respectively.
> 2. Can you enable more detailed logging?
> 3. What do you mean specifically by "concurrency of 2?"
> 4. Can you verify the name of the fetch.wait.max.ms configuration? I
> don't see that configuration option. ...