Hi Koushik
Seems there is something lead to didn't replicated in time, so the follower
was kicked out from ISR.
May i know if you have any chance can lead to that issue, e.g. the
throughput is too high to can not complete replication in time or there is
record can not replicated to follower as some configs fo max size not
proper?
Best,
Lisheng
Koushik Chitta <kchitta@microsoft.com.invalid> 于2019年8月29日周四 上午2:03写道：
> Hi All,
>
> We had a topic partition(with 5 replication) going offline when leader of
> the partition was down. Below is some analysis
>
> Kafka server - 1.1 , relevant config (replica.fetch.wait.max.ms = 500,
> replica.fetch.min.bytes = 50000, replica.lag.time.max.ms=10000)
> Topic partition (Test.Request-3) - replication 5 , Replica List [17,
> 425222741, 425222681, 423809494,425222740] , unclean leader election =
> false
>
> Sequence of events.
>
>
> 1. Leader(425222740) of the partition is down.
>
>
>
> 1. Controller detects the offline broker.
>
> [2019-08-26 13:00:22,037] INFO [Controller id=423809469] Newly added
> brokers: , deleted brokers: 425222740, all live brokers:
> 15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,423809443,423809444,423809450,423809458,423809463,423809464,423809469,423809474,423809494,425218574,425222675,425222681,425222741,425222745
> (kafka.controller.KafkaController)
>
>
>
> 1. Controller sends update metadata request , but observes only leader
> in the isr. Please note that none of the isr metrics "underminisr",
> "isrshirnk" are captured before or after the partition going offline.
> [2019-08-26 13:00:05,333] TRACE [Controller id=423809469 epoch=206]
> Sending UpdateMetadata request PartitionState(controllerEpoch=204,
> leader=425222740, leaderEpoch=435, isr=[425222740], zkVersion=804,
> replicas=[425222740, 423809494, 17, 425222741, 425222681],
> offlineReplicas=[]) to brokers Set(425222740, 423809458, 425222741, 24,
> 425218574, 25, 423809474, 26, 27, 19, 20, 21, 22, 425222681, 23, 423809450,
> 15, 16, 423809443, 423809494, 425222675, 17, 423809444, 18, 423809469,
> 425222745, 423809463, 28, 423809464, 29) for Test.Request-3
> (state.change.logger)
>
>
> 1. New leader election fails under strategy
> OfflinePartitionLeaderElectionStrategy since no replicas are in ISR list.
>
>
>
> 1. All replicas see replica fetch request errors as they cannot connect
> to leader.
>
> Any pointers on why the ISR list was shrunk just before the leader went
> down forcing the partition to go offline.
>
> Thanks,
> Koushik
>
Seems there is something lead to didn't replicated in time, so the follower
was kicked out from ISR.
May i know if you have any chance can lead to that issue, e.g. the
throughput is too high to can not complete replication in time or there is
record can not replicated to follower as some configs fo max size not
proper?
Best,
Lisheng
Koushik Chitta <kchitta@microsoft.com.invalid> 于2019年8月29日周四 上午2:03写道：
> Hi All,
>
> We had a topic partition(with 5 replication) going offline when leader of
> the partition was down. Below is some analysis
>
> Kafka server - 1.1 , relevant config (replica.fetch.wait.max.ms = 500,
> replica.fetch.min.bytes = 50000, replica.lag.time.max.ms=10000)
> Topic partition (Test.Request-3) - replication 5 , Replica List [17,
> 425222741, 425222681, 423809494,425222740] , unclean leader election =
> false
>
> Sequence of events.
>
>
> 1. Leader(425222740) of the partition is down.
>
>
>
> 1. Controller detects the offline broker.
>
> [2019-08-26 13:00:22,037] INFO [Controller id=423809469] Newly added
> brokers: , deleted brokers: 425222740, all live brokers:
> 15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,423809443,423809444,423809450,423809458,423809463,423809464,423809469,423809474,423809494,425218574,425222675,425222681,425222741,425222745
> (kafka.controller.KafkaController)
>
>
>
> 1. Controller sends update metadata request , but observes only leader
> in the isr. Please note that none of the isr metrics "underminisr",
> "isrshirnk" are captured before or after the partition going offline.
> [2019-08-26 13:00:05,333] TRACE [Controller id=423809469 epoch=206]
> Sending UpdateMetadata request PartitionState(controllerEpoch=204,
> leader=425222740, leaderEpoch=435, isr=[425222740], zkVersion=804,
> replicas=[425222740, 423809494, 17, 425222741, 425222681],
> offlineReplicas=[]) to brokers Set(425222740, 423809458, 425222741, 24,
> 425218574, 25, 423809474, 26, 27, 19, 20, 21, 22, 425222681, 23, 423809450,
> 15, 16, 423809443, 423809494, 425222675, 17, 423809444, 18, 423809469,
> 425222745, 423809463, 28, 423809464, 29) for Test.Request-3
> (state.change.logger)
>
>
> 1. New leader election fails under strategy
> OfflinePartitionLeaderElectionStrategy since no replicas are in ISR list.
>
>
>
> 1. All replicas see replica fetch request errors as they cannot connect
> to leader.
>
> Any pointers on why the ISR list was shrunk just before the leader went
> down forcing the partition to go offline.
>
> Thanks,
> Koushik
>
Comments
Post a Comment