Kafka

-----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIzBAEBCgAdFiEE8osu2CcCCF5douGQu8PBaGu5w1EFAl03TmYACgkQu8PBaGu5 w1GA1w//QvXvATiHWJ5TEGO8xAexDCnhrVPbQYogPwV0Z+VDWweUgMJQVj+1Zf/c lpGyqIuAo4vS6aTqNfF+JfPCKL/hM3AwcDxNofCDDJ02PklMpuymLLk44XyfkE8B j/XTPRl0n31LeSQ4tj/edyO3Ejq8jNkP/feGIe5Gf8cTbMsvFvTQqrnIGJGKuvbw puChv6Ua9vO1Ja4h1uujzBQLbMnL8SXcsTV3juQQd8A5tEW+L6ZnddhfZ38yhIvF 7KS9KKdt9RymYuswXRg6tOBEw2QLs6XX6pAPVnvrU0Q5LT2ufi62n5MRXMYyqjjz canceGaw+DLuubiMfFMC9kmZb/bKJh9MqzCiA3QVvSBeSFgCca1G+cffDPrnFbLw I/Y9zdkDbhjtaQ1DVcoSr9JFeYXgIg3KQFCM/UVAZ0ATQWvUC+ehgTpncvI4MZIh o03hj9F2wqF2t/LTVRAMW/54tUkDjJQF9yDPZh/aVFGQof2zMO5hAbYLk1MRFbbx uxtwMuOIQSKk5Ehbq4hO+QMyYXoQrw8oabAsBP4lyJZP69arBWR4xDgYLECh2EJn r/79Wr04JcdTx2F6ACoCC5m4xwkEJ9Yx3XAYATtLM8/3SlaTI2L3SAaHCZssEWeu 6k/EhoGKYFNHNTiwnRQpthSOZ1Gd0RJC3zrq83MODkGKtuagpd8= =DI8i -----END PGP SIGNATURE----- If you need to group-by an attribute/field in the value, you will need to extract the grouping attribute/fi...

Hi Harper, For the producers, I am using 1.1.0. As for the brokers, I do not have idea as we are using Microsoft Event Hubs as our broker. Here's the error logs that I have when I run my producer and consumers: 19/07/12 15:22:20.354 ERROR reactor.kafka.receiver.internals.DefaultKafkaReceiver: Unexpected exception org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topics': Error reading array of size 825844834, only 33 bytes available at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:76) at org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeSubscription(ConsumerProtocol.java:96) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:340) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:520) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1100(AbstractCoordina...

Hi Kyla, What message format version are your producers and brokers using? That exception looks very similar to: https://stackoverflow.com/questions/39227213/why-kafka-0-10-console-producer-cannot-send-messages-to-kafka-0-9 On Mon, Jul 22, 2019 at 6:10 AM Kyla Palos < kyla.palos@gmail.com > wrote: > I have a producer that would send messages to a topic with 4 partitions. > Then, 4 consumers would consume from that topic. Most of the time, it works > properly. But then, I would sometimes experience this error: > org.apache.kafka.common.protocol.types.SchemaException: Error reading field > 'topics': Error reading array of size 825844834, only 33 bytes available > > Can anyone give me a background on why this is happening and how to fix > this? > > Thanks! >

Hello.   I was using Kafka  2.1.1 and  facing a problem where our consumers sometimes intermittently stop consuming from one or two of the partitions.  My config 

Hi Guozhang,  Thanks very much, this was really useful. Jamie -----Original Message----- From: Guozhang Wang < wangguoz@gmail.com > To: users < users@kafka.apache.org > Sent: Tue, 23 Jul 2019 2:40 Subject: Re: Truncation Hi Jamie, The most relevant materials I can think of would be in KIP-101: https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Epoch+rather+than+High+Watermark+for+Truncation Although it is a bit out-dated it still contains most significant design key ideas here. Guozhang On Thu, Jul 18, 2019 at 11:47 PM Jamie <jamiedd13@aol.co.uk.invalid> wrote: > Hi All, > Could someone confirm what truncation happens when a partition changes > from a follower to a leader and why? > Any help would be greatly appreciated. > Many Thanks, > Jamie -- -- Guozhang

Hi Jamie, The most relevant materials I can think of would be in KIP-101: https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Epoch+rather+than+High+Watermark+for+Truncation Although it is a bit out-dated it still contains most significant design key ideas here. Guozhang On Thu, Jul 18, 2019 at 11:47 PM Jamie <jamiedd13@aol.co.uk.invalid> wrote: > Hi All, > Could someone confirm what truncation happens when a partition changes > from a follower to a leader and why? > Any help would be greatly appreciated. > Many Thanks, > Jamie -- -- Guozhang

Hello Raman, since you are using Consumer and you are concerning about the member-failure triggered rebalance, I think KIP-429 is most relevant to your scenario. As Matthias mentioned we are working on getting it in to the next release 2.4. Guozhang On Sat, Jul 20, 2019 at 6:36 PM Matthias J. Sax < matthias@confluent.io > wrote: > Static-Group membership ships with AK 2.3 (the open tickets of the KIP > are minor): > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances > > There is also KIP-415 for Kafka Connect in AK 2.3: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect > > > > Currently WIP is KIP-429 and KIP-441: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol > > > https...