Hi James,
A JIRA would be helpful. It looks like something we should fix.
Ismael
On Mon, Mar 30, 2020 at 8:17 PM James Olsen <james@inaseq.com> wrote:
> Christophe,
>
> See "Problems when Consuming from multiple Partitions" in the list
> archive. I'll forward you the full conversation privately. It includes
> debug logs that demonstrate fetches being discarded and refetched, but only
> after one or more full expirations of the fetch.max.wait.ms even though
> messages are available. It appears to be due to the re-fetch only querying
> one of the Partitions allocated to the Consumer, so if that Partition is
> empty you get the full delay even though messages are available (and in
> fact already fetched) from another Partition.
>
> The exact behaviour differs depending on the server and client versions.
> I tried 2.2.1 and 2.3.1 servers with a 2.4.0 client. Both introduced
> delays and/or some partitions not being processed at all.
>
> The problems were only observed where a Consumer subscribed to multiple
> Partitions of the same Topic.
>
> I haven't raised an issue for it as I would have no expectation of it
> being fixed. I just made my client and server versions match and
> implemented a workaround for the issue I wanted the 2.4.0 client to resolve.
>
> Regards, James.
>
> On 31/03/2020, at 12:16, JALADY Christophe <christophe.jalady@laposte.fr
> <mailto:christophe.jalady@laposte.fr>> wrote:
>
> There are serious latency issues when mixing different client and server
> version
>
> Could you be more specific ? Link to any issue ?
>
> Thanks by advance !
>
> Christophe
> ________________________________
> De : James Olsen <james@inaseq.com<mailto:james@inaseq.com>>
> Envoyé : vendredi 27 mars 2020 01:48
> À : users@kafka.apache.org<mailto:users@kafka.apache.org> <
> users@kafka.apache.org<mailto:users@kafka.apache.org>>
> Objet : Re: Reg : Slowness in Kafka
>
> Also check your Kafka Client and Server versions. There are serious
> latency issues when mixing different client and server versions IF your
> consumers handle multiple partitions.
>
> On 27/03/2020, at 12:59, Chris Larsen <clarsen@confluent.io<mailto:
> clarsen@confluent.io>> wrote:
>
> Hi Vidhya,
>
> How many tasks are you running against the topic? How many partitions are
> on the topic? Can you post the connector config anonymized?
>
> Best,
> Chris
>
> On Thu, Mar 26, 2020 at 17:58 Vidhya Sakar <sakar.black@gmail.com<mailto:
> sakar.black@gmail.com>> wrote:
>
> Hi Team,
>
> The Kafka consumer is reading only 8 records per second.We have implemented
> apache Kafka and confluent connect S3. The confluent connect S3 collects
> the records and pushes it to S3 bucket.
> In this process, we are seeing some slowness like on an average only 8
> records is being processed for a second. am planning to have a higher
> throughout results, so that where there is a higher data load, it should
> process more number of records.
>
> Thanks in advance.
>
> --
>
> *Chris Larsen*
>
> Systems Engineer | Confluent
>
> 847.274.3735
>
> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> <https://www.confluent.io/blog/>
>
>
> Post-scriptum La Poste
>
> Ce message est confidentiel. Sous reserve de tout accord conclu par
> ecrit entre vous et La Poste, son contenu ne represente en aucun cas un
> engagement de la part de La Poste. Toute publication, utilisation ou
> diffusion, meme partielle, doit etre autorisee prealablement. Si vous
> n'etes pas destinataire de ce message, merci d'en avertir immediatement
> l'expediteur.
>
>
A JIRA would be helpful. It looks like something we should fix.
Ismael
On Mon, Mar 30, 2020 at 8:17 PM James Olsen <james@inaseq.com> wrote:
> Christophe,
>
> See "Problems when Consuming from multiple Partitions" in the list
> archive. I'll forward you the full conversation privately. It includes
> debug logs that demonstrate fetches being discarded and refetched, but only
> after one or more full expirations of the fetch.max.wait.ms even though
> messages are available. It appears to be due to the re-fetch only querying
> one of the Partitions allocated to the Consumer, so if that Partition is
> empty you get the full delay even though messages are available (and in
> fact already fetched) from another Partition.
>
> The exact behaviour differs depending on the server and client versions.
> I tried 2.2.1 and 2.3.1 servers with a 2.4.0 client. Both introduced
> delays and/or some partitions not being processed at all.
>
> The problems were only observed where a Consumer subscribed to multiple
> Partitions of the same Topic.
>
> I haven't raised an issue for it as I would have no expectation of it
> being fixed. I just made my client and server versions match and
> implemented a workaround for the issue I wanted the 2.4.0 client to resolve.
>
> Regards, James.
>
> On 31/03/2020, at 12:16, JALADY Christophe <christophe.jalady@laposte.fr
> <mailto:christophe.jalady@laposte.fr>> wrote:
>
> There are serious latency issues when mixing different client and server
> version
>
> Could you be more specific ? Link to any issue ?
>
> Thanks by advance !
>
> Christophe
> ________________________________
> De : James Olsen <james@inaseq.com<mailto:james@inaseq.com>>
> Envoyé : vendredi 27 mars 2020 01:48
> À : users@kafka.apache.org<mailto:users@kafka.apache.org> <
> users@kafka.apache.org<mailto:users@kafka.apache.org>>
> Objet : Re: Reg : Slowness in Kafka
>
> Also check your Kafka Client and Server versions. There are serious
> latency issues when mixing different client and server versions IF your
> consumers handle multiple partitions.
>
> On 27/03/2020, at 12:59, Chris Larsen <clarsen@confluent.io<mailto:
> clarsen@confluent.io>> wrote:
>
> Hi Vidhya,
>
> How many tasks are you running against the topic? How many partitions are
> on the topic? Can you post the connector config anonymized?
>
> Best,
> Chris
>
> On Thu, Mar 26, 2020 at 17:58 Vidhya Sakar <sakar.black@gmail.com<mailto:
> sakar.black@gmail.com>> wrote:
>
> Hi Team,
>
> The Kafka consumer is reading only 8 records per second.We have implemented
> apache Kafka and confluent connect S3. The confluent connect S3 collects
> the records and pushes it to S3 bucket.
> In this process, we are seeing some slowness like on an average only 8
> records is being processed for a second. am planning to have a higher
> throughout results, so that where there is a higher data load, it should
> process more number of records.
>
> Thanks in advance.
>
> --
>
> *Chris Larsen*
>
> Systems Engineer | Confluent
>
> 847.274.3735
>
> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> <https://www.confluent.io/blog/>
>
>
> Post-scriptum La Poste
>
> Ce message est confidentiel. Sous reserve de tout accord conclu par
> ecrit entre vous et La Poste, son contenu ne represente en aucun cas un
> engagement de la part de La Poste. Toute publication, utilisation ou
> diffusion, meme partielle, doit etre autorisee prealablement. Si vous
> n'etes pas destinataire de ce message, merci d'en avertir immediatement
> l'expediteur.
>
>
Comments
Post a Comment