Kafka Connect Distributed Mode Issues

Hi,
I am using kafka-connect-file-pulse connector and scanning around 20K
files. After the scan step, the whole connect cluster is becoming
unresponsive. I can not even access localhost:8083/connectors/ URL. It is
giving request timeout.


I have observed the below errors from the connect logs. Did anyone face
this issue?

Please advise if I am doing something wrong.


[2021-03-31 16:21:58,920] INFO Scanning local file system directory
'/apps/datafiles_1/cm_dir/QA1/'
(io.streamthoughts.kafka.connect.filepulse.scanner.LocalFileSystemScanner:241)
[2021-03-31 16:22:57,586] WARN [Worker clientId=connect-1,
groupId=connect-cluster] This member will leave the group because consumer
poll timeout has expired. This means the time between subsequent calls to
poll() was longer than the configured max.poll.interval.ms, which typically
implies that the poll loop is spending too much time processing messages.
You can address this either by increasing max.poll.interval.ms or by
reducing the maximum size of batches returned in poll() with
max.poll.records.
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1051)
[2021-03-31 16:22:57,586] INFO [Worker clientId=connect-1,
groupId=connect-cluster] *Member
connect-1-064cf0bf-b834-40d2-9e72-e61b229157c4 sending LeaveGroup request
to coordinator URL:9092* (id: 2147483646 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:822)
[2021-03-31 16:23:24,562] ERROR Request to leader to reconfigure connector
tasks failed
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1037)


*org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Request
timed out*
at
org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:97)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1034)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2021-03-31 16:23:24,562] ERROR* Failed to reconfigure connector's tasks,
retrying after backoff:
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:958)*
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Request
timed out
at
org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:97)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1034)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:51

--
Regards,
Himanshu Shukla