Hi Fabio Pardi,
Thanks for your prompt response.
Split brain was our suspicion and we are investigating other possibilities.
Perhaps our understanding of the problem might be incorrect at the moment.
The issue started when one of the broker instances went down abruptly (3
brokers, 3 zookeepers) and the cluster was unstable.
Later, we were able to restart the affected broker instance followed by
rolling restart of other 2 brokers. The cluster was stabilized at this
point.
However, we noticed un-repl partitions and Preferred Replica imbalance
irregularities.
[xxx(user):/xxx/install/1.0.0/bin] ./kafka-topics.sh --describe --zookeeper
zookeeper1:2181 --under-replicated-partitions
Topic: ABC Partition: 3 Leader: 31 Replicas: 31,21,11
Isr: 31,11
Topic: __consumer_offsets Partition: 1 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 3 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: __consumer_offsets Partition: 7 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 9 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: __consumer_offsets Partition: 13 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 15 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: __consumer_offsets Partition: 19 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 21 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: __consumer_offsets Partition: 25 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 27 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: __consumer_offsets Partition: 31 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 33 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: __consumer_offsets Partition: 37 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 43 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 45 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: __consumer_offsets Partition: 49 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: _kafka_lenses_alerts Partition: 0 Leader: 31
Replicas: 31,21,11 Isr: 31,11
Topic: _kafka_lenses_alerts_settings Partition: 0 Leader: 31
Replicas: 31,21,11 Isr: 31,11
Topic: _kafka_lenses_processors Partition: 0 Leader: 31
Replicas: 31,21,11 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 0 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 4 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 6 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 10 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 12 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 16 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 18 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 22 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 24 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: connect-kfkxxxprd-status Partition: 3 Leader: 31
Replicas: 21,31,11 Isr: 31,11
On Tue, Jul 27, 2021 at 9:46 AM Fabio Pardi <f.pardi@portavita.eu> wrote:
>
>
> On 27/07/2021 09:19, Sridhar Rao wrote:
> > Hi Everyone,
> >
> > Recently we noticed a high number of under-replicated-partitions after
> > zookeeper split brain issue.
> > We tried fixing the issue by executing ./kafka-reassign-partitions.sh
> > procedure. However Kafka refuses to re-assign the partitions in ISR and
> > un-repl partitions remain the same.
> >
> > Kafka version: Apache Kafka 1.0.0
> >
> > Any advice in this regard would be highly appreciated. Please feel free
> to
> > request any missing information. Thank you!
> >
>
> Zookeeper needs a quorum to elect a leader therefore a split brain is
> unlikely. Can you describe how split brain occurred? (number of instances,
> what happened and how it was resolved)
>
> Given that, if your topic/partitions topology did not change, (that I
> know) even a split brain should not affect your data, therefore your ISR.
>
> Running the below command should tell you which partitions are under
> replicated
>
> kafka-topics.sh --describe
>
> I think it might be a good starting point to understand what is going on.
>
>
> To blindly reassign partitions is in my experience not an ideal solution,
> because you will have data shuffling around unnecessarily.
>
>
> regards,
>
> fabio pardi
>
>
Thanks for your prompt response.
Split brain was our suspicion and we are investigating other possibilities.
Perhaps our understanding of the problem might be incorrect at the moment.
The issue started when one of the broker instances went down abruptly (3
brokers, 3 zookeepers) and the cluster was unstable.
Later, we were able to restart the affected broker instance followed by
rolling restart of other 2 brokers. The cluster was stabilized at this
point.
However, we noticed un-repl partitions and Preferred Replica imbalance
irregularities.
[xxx(user):/xxx/install/1.0.0/bin] ./kafka-topics.sh --describe --zookeeper
zookeeper1:2181 --under-replicated-partitions
Topic: ABC Partition: 3 Leader: 31 Replicas: 31,21,11
Isr: 31,11
Topic: __consumer_offsets Partition: 1 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 3 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: __consumer_offsets Partition: 7 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 9 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: __consumer_offsets Partition: 13 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 15 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: __consumer_offsets Partition: 19 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 21 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: __consumer_offsets Partition: 25 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 27 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: __consumer_offsets Partition: 31 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 33 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: __consumer_offsets Partition: 37 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 43 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: __consumer_offsets Partition: 45 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: __consumer_offsets Partition: 49 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: _kafka_lenses_alerts Partition: 0 Leader: 31
Replicas: 31,21,11 Isr: 31,11
Topic: _kafka_lenses_alerts_settings Partition: 0 Leader: 31
Replicas: 31,21,11 Isr: 31,11
Topic: _kafka_lenses_processors Partition: 0 Leader: 31
Replicas: 31,21,11 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 0 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 4 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 6 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 10 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 12 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 16 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 18 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 22 Leader: 31
Replicas: 21,11,31 Isr: 31,11
Topic: connect-kfkxxxprd-offset Partition: 24 Leader: 31
Replicas: 31,11,21 Isr: 31,11
Topic: connect-kfkxxxprd-status Partition: 3 Leader: 31
Replicas: 21,31,11 Isr: 31,11
On Tue, Jul 27, 2021 at 9:46 AM Fabio Pardi <f.pardi@portavita.eu> wrote:
>
>
> On 27/07/2021 09:19, Sridhar Rao wrote:
> > Hi Everyone,
> >
> > Recently we noticed a high number of under-replicated-partitions after
> > zookeeper split brain issue.
> > We tried fixing the issue by executing ./kafka-reassign-partitions.sh
> > procedure. However Kafka refuses to re-assign the partitions in ISR and
> > un-repl partitions remain the same.
> >
> > Kafka version: Apache Kafka 1.0.0
> >
> > Any advice in this regard would be highly appreciated. Please feel free
> to
> > request any missing information. Thank you!
> >
>
> Zookeeper needs a quorum to elect a leader therefore a split brain is
> unlikely. Can you describe how split brain occurred? (number of instances,
> what happened and how it was resolved)
>
> Given that, if your topic/partitions topology did not change, (that I
> know) even a split brain should not affect your data, therefore your ISR.
>
> Running the below command should tell you which partitions are under
> replicated
>
> kafka-topics.sh --describe
>
> I think it might be a good starting point to understand what is going on.
>
>
> To blindly reassign partitions is in my experience not an ideal solution,
> because you will have data shuffling around unnecessarily.
>
>
> regards,
>
> fabio pardi
>
>
Comments
Post a Comment