Kafka

Hey folks, I'm quite confused here. I have a random partition that has 4 replications instead of 3. I don't have a clue how to fix it or what the ramifications of this are either. There are no repartitioning happening or reassignments or anything at all. This looks like the metadata is just wrong. That node 38 id on that partition shouldn't have existed at all. Can somebody tell me what's going on here? Do I have to worry? How do I fix it? What do I need to worry about with the inconsistency here? How will stores be affected? Etc? This is on a kafka 3.6.2 cluster using kraft servers. bash-4.2# KAFKA_HEAP_OPTS="-Xms64m -Xmx200m" /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker16:9092 --describe --topic iris.insight Topic: iris.insight TopicId: b-QgeEnGRmKc1Dfutxq4XQ PartitionCount: 15 ReplicationFactor: 3 Configs: segment.bytes=1073741824, retention.ms =604800000 Topic: iris.insight Partition: 0 Leader: 28 Replicas: 28,29,30 Isr: 28,30,29 Topic: iri...

I would suggest going from 2.6.2 to 3.6.2 and then stopping and waiting until 3.7.1 comes out. There is a pretty serious flaw in 3.7.0 that hit us and we ended up having to downgrade to 3.6.2 to recover. The bug is fixed but not going to be released until 3.7.1 comes out. The bug prevented metadata from being readable and thus anytime a node restarted (including the kraft server itself), it could never come back online or be unregistered from the cluster either. Sejal [Patel] | CTO sejal@playerzero.ai <mailto: sejal@playerzero.ai > | 470.440.1255 On Thu Jun 13, 2024, 03:26 PM GMT, Jose Manuel Monroy <mailto: jose.monroy@888williamhill.com > wrote: > Hi there, > We have few clusters in AWS world and being forced to bump cluster version. > So, we are considering options to be in safer way possible, mitigating risks, and without outage as possible (our business is 24/7). > Upgrade would be from 2.2.1 / 2.6.2 to 3.7.0 (MSK recommended version). > T...

Hi there,   We have few clusters in AWS world and being forced to bump cluster version.   So, we are considering options to be in safer way possible, mitigating risks, and without outage as possible (our business is 24/7).   Upgrade would be from 2.2.1 / 2.6.2 to 3.7.0 (MSK recommended version).   Things to consider:   Recommended configurations in place so rolling restart is not impacting producers/consumers Clients from different technologies (Java official client/Java SpringBoot Spring Cloud Stream/Alpakka Scala/NodeJS) around those versions 2.2.1 cluster version -> TLS SSL SASL/SCRAM authenticated 2.2.6 cluster version -> unathenticated We don't use Kafka streams   Could you please which would be your recommendation in terms of upgrade brokers? Can be done directly or in several steps? We should upgrade clients before than brokers? If so, in which way?   Client side, any recommendation of clients? (exampl...

Hello, we have a requirement to make kafka connector more fault tolerant for our use, where in we don't want them to fail for some kinds of errors, i.e error where bootstrap broker are missing or if we don't have sufficient permission to read data from topic (we are reading from aws msk). we tried using basic error handing for such exception where in, in the poll method we tried to swallow SaslException and config exceptions where bootstrap brokers are missing, and retry after t amount of time but this seems to make the connector make lots of unassigned task corresponding to even a connector even with max.tasks set to 1. 1. unclear as to why that happens 2. looking for guidance on more standardised way to ensure resilient connectors that don't transition to fail state on some errors which we expect can happen

Hi Sebastian, Thanks for your question! A consumer joining a group should always notify the group coordinator and the leader consumer, and the partition.assignment.strategy [1] will determine the rebalance protocol, and which partitions are affected by the rebalance. For example, I expect that the Range, RoundRobin, and Sticky assignors would be quite disruptive, with additional consumers triggering a full revocation of all partitions. The CooperativeStickyAssignor should perform a much less invasive cooperative rebalance that avoids revoking unnecessary partitions [2]. It also appears that consumers undergoing a cooperative rebalance are permitted to continue processing data [3]. So under good conditions with modern versions and configurations, extra consumers joining the group should trigger a rebalance, but the rebalance should have minimal impact on processing. Thanks, Greg [1] https://kafka.apache.org/documentation.html#consumerconfigs_partition.assignm...

While I haven't reached that point because with repartitioning and aggregation and various other things happening with in the stream having additional topics a simple 5 input topics with 5 partitions can end up with hundreds of total topic partitions overall. But I'm 99% confident that it will rebalance because part of that is understanding who is available even, who has which data, latency of work being done etc. While something might be perfectly balanced 1 second, it might not be the next in theory. But if it makes you feel better, it should only rebalance for a split second if it issue seeing a reason to shift work around. But I'm more confident that you'll not be spinning up more consumers than you have total partitions to start with cause there are a lot of partitions happening with the Kafka stores each having some as well. Sejal [Patel] | CTO sejal@playerzero.ai <mailto: sejal@playerzero.ai > | 470.440.1255 On Wed Jun 12, 2024, 10:25 AM GMT...

Hello, If I have a consumer group with more members than the number of partition of a topic, adding a consumer to the group will still trigger a rebalancing of partitions to the group? Imagine the partitions are already perfectly balanced, ie each consumer has 1 partition. Then reblancing won't be of any use in theory. So does Kafka still triggers a rebalancing? Thanks , Sébastien