Re: Consumer has high network bandwidth when consuming no messages

Hi, Lee,

Would you be able to see which Kafka API is generating the traffic?
This is provided by the MBean
kafka.network:type=RequestMetrics,name=RequestsPerSec,request=*,version=([0-9]+)
[1].

Thanks,
Alexandre

[1] https://kafka.apache.org/documentation/#monitoring

Le dim. 16 avr. 2023 à 18:22, Lee Carroll
<lee.a.carroll@googlemail.com.invalid> a écrit :
>
> Hi Kirk,
>
> Points 1 and 2: yes we have both logging and metrics. I'll collate
> Point 3 concurrency refers to the spring boot kafka concurrent container. 2
> threads will be made available to the consumer group to consume 30
> partitions making up a single topic.
> point 4: consumer- Kafka Consumer Configurations for Confluent Platform |
> Confluent Documentation
> <https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#fetch-max-wait-ms>
>
> On Thu, 13 Apr 2023 at 16:03, Kirk True <kirk@kirktrue.pro> wrote:
>
> > Hi Lee,
> >
> > Some questions:
> >
> > 1. Can you enable metrics for your consumer(s)? See [1] and [2] for links
> > on consumer-level and fetch-level metrics, respectively.
> > 2. Can you enable more detailed logging?
> > 3. What do you mean specifically by "concurrency of 2?"
> > 4. Can you verify the name of the fetch.wait.max.ms configuration? I
> > don't see that configuration option. Is that a broker configuration or a
> > consumer configuration?
> >
> > Thanks,
> > Kirk
> >
> > [1] https://kafka.apache.org/documentation/#consumer_group_monitoring
> > [2] https://kafka.apache.org/documentation/#consumer_fetch_monitoring
> >
> > > On Apr 13, 2023, at 4:52 AM, Lee Carroll <lee.a.carroll@googlemail.com.INVALID>
> > wrote:
> > >
> > > Hi All,
> > > I've a spring boot kafka consumer group, consuming a topic with 30
> > > partitions with a concurrency of 2 (staging set up).
> > >
> > > The network traffic in and out was very high >200k per second even when
> > > consuming no messages.
> > >
> > > I've adjusted fetch.wait.max.ms to be significantly higher (15,000) than
> > > the default which has had a significant impact but the network is still
> > > seeing ~30k a second.
> > >
> > > Is this to be expected?
> > >
> > > Cheers Lee C
> >
> >