Re: Producer slowness and Exception on broker restart - kafka.common.KafkaException: Unknown group metadata version 1

Hi,

We also did version upgrade and then degraded to old due to some issue.
But that we done ~1 year before.
Do I need to follow this to solve this issue?

http://mail-archives.apache.org/mod_mbox/kafka-dev/201907.mbox/%3CJIRA.13246826.1563975667000.40318.1563975720170@Atlassian.JIRA%3E

In this he follow below steps to evicted the broken records with
temporarily setting the segment time to a very low value and deactivation
of compaction.

/opt/kafka/bin/kafka-topics.sh --alter --config segment.ms=900000
--topic __consumer_offsets
--zookeeper localhost:2181
/opt/kafka/bin/kafka-topics.sh --alter --config cleanup.policy=delete
--topic __consumer_offsets
--zookeeper localhost:2181
< wait for the cleaner to clean up >
/opt/kafka/bin/kafka-topics.sh --alter --config segment.ms=604800000
--topic __consumer_offsets
--zookeeper localhost:2181
/opt/kafka/bin/kafka-topics.sh --alter --config cleanup.policy=compact
--topic __consumer_offsets
--zookeeper localhost:2181{code}




On Wed, Nov 27, 2019 at 10:33 PM Upendra Yadav <upendra1024@gmail.com>
wrote:

> Hi Team,
>
> In my production setup, I'm getting below exception(for 4-5
> __consumer_offsets partitions) while restarting kafka broker(s).
> Before restart, my producers(sync) are very slow. for some messages its
> taking 5 seconds to acknowledge.
> But after restart, all messages wishing 2 milliseconds.
> After sometime once again same producer slowness.
> Before restart was able to produce 7k messages, but after restart able to
> push 1lakh messages in 10 minutes.
>
> Kafka broker and client version- 10.0.0.1
> Using latest producer java client for this version.
> But consumers are old scala api.
>
> Exception:
>
> [2019-11-27 05:54:43,724] INFO [Group Metadata Manager on Broker 3]:
> Loading offsets and group metadata from [__consumer_offsets,34]
> (kafka.coordinator.GroupMetadataManager)
> [2019-11-27 05:54:43,735] ERROR [Group Metadata Manager on Broker 3]:
> Error in loading offsets from [__consumer_offsets,34]
> (kafka.coordinator.GroupMetadataManager)
> kafka.common.KafkaException: Unknown group metadata version 1
> at
> kafka.coordinator.GroupMetadataManager$.schemaForGroup(GroupMetadataManager.scala:764)
> at
> kafka.coordinator.GroupMetadataManager$.readGroupMessageValue(GroupMetadataManager.scala:934)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1$$anonfun$apply$mcV$sp$1.apply(GroupMetadataManager.scala:412)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1$$anonfun$apply$mcV$sp$1.apply(GroupMetadataManager.scala:383)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
> at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at kafka.message.MessageSet.foreach(MessageSet.scala:71)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply$mcV$sp(GroupMetadataManager.scala:383)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply(GroupMetadataManager.scala:374)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply(GroupMetadataManager.scala:374)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:231)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:239)
> at
> kafka.coordinator.GroupMetadataManager.kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1(GroupMetadataManager.scala:374)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$loadGroupsForPartition$1.apply$mcV$sp(GroupMetadataManager.scala:353)
> at
> kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
> at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:56)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>