Kafka

Very good description with pictures in the book Kafka: The Definitive Guide https://www.oreilly.com/library/view/kafka-the-definitive/9781491936153/ch04.html -hans > On Mar 26, 2020, at 12:00 PM, sunil chaudhari < sunilmchaudhari05@gmail.com > wrote: > > Again.... > A consumer can have one or more consumer thread. > The analogy of 12 partitions and 4 consumer is true when each consumer has > 3 consumer threads. > Please don't skip the important factor "consumer thread" in this matter. > > If you run each consumer with threads then you may need max 3 consumers for > that consumer group. > > If you have 12 partitions and you run 4 consumers with 4 consumer threads > then 4 threads will be idle at any time T1. > > I hope this is clear. > > Thanks, > Sunil. > > On Thu, 26 Mar 2020 at 7:52 PM, Hans Jespersen < hans@confluent.io > wrote: > >>> As per my und...

Also check your Kafka Client and Server versions. There are serious latency issues when mixing different client and server versions IF your consumers handle multiple partitions. > On 27/03/2020, at 12:59, Chris Larsen < clarsen@confluent.io > wrote: > > Hi Vidhya, > > How many tasks are you running against the topic? How many partitions are > on the topic? Can you post the connector config anonymized? > > Best, > Chris > > On Thu, Mar 26, 2020 at 17:58 Vidhya Sakar < sakar.black@gmail.com > wrote: > >> Hi Team, >> >> The Kafka consumer is reading only 8 records per second.We have implemented >> apache Kafka and confluent connect S3. The confluent connect S3 collects >> the records and pushes it to S3 bucket. >> In this process, we are seeing some slowness like on an average only 8 >> records is being processed for a second. am planning to have a higher >> throug...

Hi Vidhya, How many tasks are you running against the topic? How many partitions are on the topic? Can you post the connector config anonymized? Best, Chris On Thu, Mar 26, 2020 at 17:58 Vidhya Sakar < sakar.black@gmail.com > wrote: > Hi Team, > > The Kafka consumer is reading only 8 records per second.We have implemented > apache Kafka and confluent connect S3. The confluent connect S3 collects > the records and pushes it to S3 bucket. > In this process, we are seeing some slowness like on an average only 8 > records is being processed for a second. am planning to have a higher > throughout results, so that where there is a higher data load, it should > process more number of records. > > Thanks in advance. > -- *Chris Larsen* Systems Engineer | Confluent 847.274.3735 Follow us: Twitter < https://twitter.com/ConfluentInc > | blog < https://www.confluent.io/blog/ >

-----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl59QeYACgkQO4miYXKq /OjLFg//S7oP0N/ajojOed4hgATZzAHwpRJ+/FtOEjaTXk3VBByberO3e6eJr+ff QGz4HhdaeqLu3xAUfn2/lfVnL0qf4AhSh8RhjFSOJrC7iKZ+3lvzuJ0JMTipYZ4Y q0TTHiDLsZGAYjC9CyFmKKjy5UBWiuywZuVp3ei8bJwWi9RVC1r8AGmDinSMnEGS Ql4xlTvloXXl1qjuKpQdRMR1UkvSx/Wy77J/hXAwt+bKHsEIE29XX5/z8uFPPej3 sdzYvzmBcdC/H93nI8c5cQ5fLjrRF+TPuNmSSoMDkydc9hLUCZrk+8p4awpv+w1Q DNEF0ZAC6iGZDp05UtdeJChcVIbCea2kJh6iPO36oL8A5xt04ncSQ0j9e2cWUu+T 5KVdOwo3fKkifcGAzPT04Jc7razy7ITQ+BpcstRar9l5/tG/p+IwJ6yoft1MKFus AVNFSbLQTxcFCXnyU3HKeqlKl2UfIzj/PcqAcBoR2JmByXw9keS/ypgZMimf7QJN ekBPDg88B7srXzhSbmq/ZILCp8Jg4xH3wJW6UGLhvwx7cfI4GF8+5sxwL4i5f+7u JWOSeJRU68ptTniBTRT5Wui66NJx6DvsLXhvfR/HrgTqmC8mu4VMfKQTYMQE+cKZ 5vcerAFZD1ZAdHp3hUjT85j3fw84NylAgmA8+Uok0ue33ODXYak= =vRt2 -----END PGP SIGNATURE----- Your code looks correct to me. If you write into the store, you should also be able to read it back from the store. Can you reproduce the issue usi...

-----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl59P6sACgkQO4miYXKq /Og/7xAApj65zJIFGJOleVN8kEasLuOljodtZoY9i1DkfkqALIjjT5pZqtGVMyL1 7aZLCQG9wtTevVy/W/hojNmRwMqHnxtP7fDjmulmy1cEUtPEKJ71uFz6F4oHkJi3 RhaMzFBuvayGQs3cwLCSgNLl/Igo5nUCorNeUulrzL3/osjAJc2rAzQX6uN/mJNG fnmRXwiK/6V3l23m4tKU+OWFu7gHwX9yX01EZQ9WsJZ2w5uq20VPB/PddZVa7j6+ NpJcF/IgGLHQi4ZUZs5icoNqWr8r0Ym8Y5E1CazdRNv/uTGAVwnNyX8QiX5E1y/n 4pz0n+lubO1uUzmovwfFczw6EpqPVfVB+6Qq7fL+O+LHvC1jrQthlFoM54yA51mw /GZGlGFrj+oQUcY3FNfWJcC+TiChTljUQcO0YFU5AabxsBiZdTp39F5MOQF+J+Hg nS8gCFdtpoBw8dVfL5CWzQjSOO2cfvXaSvTyIfxR63M5U5fgRHJzH1JCZa4dDrDJ 4N4kqa+Wo3RbVYzJ82NIz9sU83MbwU8huP/p1PEJfl3YBzq2ZVBWtqXHJGUmAy5s kXkdb9NDeAlgxT866ujZKGQ09b9dYo4bhXTJdry9BSqIg1+6y0ZOAQ0/onYrFQNB Wh1ysmGBabPKRh8jNeOnjNW4sX+SujV6NhvGAGRJ9uuGcvxqbMQ= =k3Zk -----END PGP SIGNATURE----- ` max.poll.intervall.ms ` is the maximum allowed time between two calls to `poll()`. Hence, this config seems to be unrelated. For the background h...

Yes it should be going much faster than that. Something is wrong in your setup. -hans > On Mar 26, 2020, at 5:58 PM, Vidhya Sakar < sakar.black@gmail.com > wrote: > > Hi Team, > > The Kafka consumer is reading only 8 records per second.We have implemented > apache Kafka and confluent connect S3. The confluent connect S3 collects > the records and pushes it to S3 bucket. > In this process, we are seeing some slowness like on an average only 8 > records is being processed for a second. am planning to have a higher > throughout results, so that where there is a higher data load, it should > process more number of records. > > Thanks in advance.

Hi Team, The Kafka consumer is reading only 8 records per second.We have implemented apache Kafka and confluent connect S3. The confluent connect S3 collects the records and pushes it to S3 bucket. In this process, we are seeing some slowness like on an average only 8 records is being processed for a second. am planning to have a higher throughout results, so that where there is a higher data load, it should process more number of records. Thanks in advance.