Re: Connect task configuration limits?

Hi Ryan,

There are some limits, as the configs are stored inside an internal
topic of Connect. So the usual message size and producer request size
limitations apply.
You can reconfigure the internal topic to allow larger messages than the
default (I think it's 1MB), and the producer max request size for the
Connect worker.

Besides that, there is one extra bottleneck - Connect followers might
forward the task configs to the leader through the REST API. The default
request size limit of the REST client can also cause an issue, not sure
if there is a way to reconfigure that.

I think you should be able to find some kind of a related error message,
probably in the leader of the cluster, or the worker which was hosting
the connector itself.

Daniel

2022. 05. 19. 17:01 keltezéssel, Ryan Slominski írta:
> Hi,
> Are there limits to the size of configuration data passed via the
> taskConfigs method of the Connector class? I'm observing a situation
> where if I use a large configuration no tasks are created, and no log
> messages appear in the connect log file. Using a smaller configuration
> works. If there are limits, can I increase them? Also, it's probably a
> good idea if Kafka were to log a warning message of some kind in this
> scenario instead of silently failing. I'm using a custom Source Connector
> and I have documented steps to reproduce the issue using Docker compose
> here:
>
> https://github.com/JeffersonLab/epics2kafka/issues/11
>
> Thanks for any insights!
>
> Ryan
>

--
Ezt az e-mailt az Avast víruskereső szoftver átvizsgálta.
https://www.avast.com/antivirus