Hi,
I am using kafka-connect-file-pulse connector and scanning around 20K
files. After the scan step, the whole connect cluster is becoming
unresponsive. I can not even access localhost:8083/connectors/ URL. It is
giving request timeout.
I have observed the below errors from the connect logs. Did anyone face
this issue?
Please advise if I am doing something wrong.
[2021-03-31 16:21:58,920] INFO Scanning local file system directory
'/apps/datafiles_1/cm_dir/QA1/'
(io.streamthoughts.kafka.connect.filepulse.scanner.LocalFileSystemScanner:241)
[2021-03-31 16:22:57,586] WARN [Worker clientId=connect-1,
groupId=connect-cluster] This member will leave the group because consumer
poll timeout has expired. This means the time between subsequent calls to
poll() was longer than the configured max.poll.interval.ms , which typically
implies that the poll loop is spending too much time processing messages.
You can address this either by increasing max.poll.interval.m...
Posts
Hello. I was wondering if someone can help answer my question. I'm trying
to run MirrorMaker 2 in distributed mode using SSL. I have the distributor
running in SSL but when I can't get the curl REST api to do so. I saw that
kif-208 fixed this but I can't seem to implement it.
in my mm2-dist.prop file I have set:
////
listeners= https://localhost:8443
security.protocol=SSL
ssl.truststore.location=/home/ec2-user/kafka_2.13-2.7.0/cert/kafka.client.truststore.jks
////
my connector.json file look like this:
////
{
"name": "mm2-connect-cluster",
"config":{
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"connector.client.config.override.policy": "All",
"name": "mm2-connect-cluster",
"topics": "test.*",
"tasks.max": "1",
"source.cluster.alias...
Hey Georg,
I'm currently working out the issues with the licensing that Justin brought
up earlier along with the
other release manager(s). I expect to be back on track with a new RC under
vote by the beginning
of next week.
Here's the ticket for anyone interested in following along:
https://issues.apache.org/jira/browse/KAFKA-12593
Sorry for the delay. I appreciate everyone's patience
Sophie
On Wed, Mar 31, 2021 at 11:19 AM Georg Friedrich <
georg.friedrich@webfleet.com > wrote:
> Hi Sophie,
>
> Is there a new RC yet?
> Thanks for driving the release.
>
> Kind regards
> Georg
>
> -----Original Message-----
> From: Sophie Blee-Goldman <sophie@confluent.io.INVALID>
> Sent: Friday, March 19, 2021 11:44 PM
> To: Bruno Cadonna < bruno@confluent.io >
> Cc: dev < dev@kafka.apache.org >; users@kafka.apache.org ;
> kafka-clients@googlegroups.com
> Subject: Re: [kafka-clients] [...
Hi Sophie,
Is there a new RC yet?
Thanks for driving the release.
Kind regards
Georg
-----Original Message-----
From: Sophie Blee-Goldman <sophie@confluent.io.INVALID>
Sent: Friday, March 19, 2021 11:44 PM
To: Bruno Cadonna < bruno@confluent.io >
Cc: dev < dev@kafka.apache.org >; users@kafka.apache.org ; kafka-clients@googlegroups.com
Subject: Re: [kafka-clients] [VOTE] 2.6.2 RC0
Thanks Bruno. I agree this qualifies as a blocker since it was a regression in 2.6 and may result in data loss. I'll roll a new RC with the fix
On Fri, Mar 19, 2021 at 7:03 AM 'Bruno Cadonna' via kafka-clients < kafka-clients@googlegroups.com > wrote:
> Hi Sophie,
>
> Correction to my last e-mail: The bug does not break eos, but it
> breaks at-least-once.
>
> Bruno
>
> On 19.03.21 14:54, Bruno Cadonna wrote:
> > Hi Sophie,
> >
> > Please have a look at the following bug report:
> >
> ...
Hello, I'm working on creating an application that leverages Kafka and Kafka Streams. I have some issues with application startup that I've been unable to solve myself even with the help of my team mates, so I'm writing here in the hopes that someone could offer help. I'd be very grateful. Application description: The application is running in AWS and uses the AWS MSK service for the operation of the brokers. The application is running in parallel on multiple nodes, typically we have 3 but it's meant to scale to tens if needed. The number of brokers is also currently 3. Kafka version is 2.6.0, both in MSK and in the Kafka libraries included with the application. The application is running in US west coast, while the Kafka brokers are in Europe, so there is some network lag between. (There's another group of 3 servers running also in Europe, with a different application id configured, so the servers in a given geographic location...
Great to hear! Always a pleasure.
Guozhang
On Tue, Mar 30, 2021 at 8:04 PM Upesh Desai < udesai@itrsgroup.com > wrote:
> Hi Guozhang,
>
>
>
> We can confirm the behavior with the 2.7.1 release. Appreciate all the
> help!
>
>
>
> Cheers,
>
> Upesh
>
>
> Upesh Desai | Senior Software Developer | * udesai@itrsgroup.com *
> < udesai@itrsgroup.com >
> * www.itrsgroup.com * < https://www.itrsgroup.com/ >
> < https://www.itrsgroup.com/ >
>
> *From: *Guozhang Wang < wangguoz@gmail.com >
> *Date: *Tuesday, March 30, 2021 at 2:10 PM
> *To: *Users < users@kafka.apache.org >
> *Cc: *Bart Lilje < blilje@itrsgroup.com >
> *Subject: *Re: Kafka Streams Processor API state stores not restored via
> changelog topics
>
> Great, I think https://issues.apache.org/jira/browse/KAFKA-12323 is indeed
> the root cause then. Note that this is only...