Hi Marcus,
1. If you change REPLICATION_FACTOR_CONFIG without resetting the
application (or deleting the changelog and repartition topics) and
redeploy the Streams application, the replication factor of the internal
topics will not change. The replication factor will only change for new
deployments, i.e., Streams applications with a new application ID or
Streams applications that where reset. In both cases the internal topics
will be newly created.
2. Changing the replication factor of a topic directly on the brokers
should be fine. Kafka Streams should not re-create the internal topics
and not throw any exceptions.
3. Unfortunately, I do not know the answer to this question. Hopefully
somebody else can answer it.
I answered your questions to the best of my knowledge. The only way to
confirm my answers is to test (preferably in a test environment).
Best,
Bruno
On 28.04.21 17:34, Marcus Horsley-Rai wrote:
> Hi All,
>
> I'm in a sub-optimal situation whereby I have some Kafka Streams apps
> deployed to production, but the default replication factor set on the
> brokers was 1 when they were first deployed.
> As such, any state store changelog topics, and re-partition topics
> therefore have RF 1 also.
>
> I'm familiar with the bin/kafka-reassign-partitions.sh tool and how to use
> that.
> I've since also found the streams replication.factor
> (StreamsConfig.REPLICATION_FACTOR_CONFIG) setting that can be supplied in
> the streams apps config.
>
> My questions are:
>
> 1. Will simply changing the value of REPLICATION_FACTOR_CONFIG and
> re-deploying have any effect on already-created internal topics?
> 2. Conversely, should I just change the RF of the internal topics using
> the re-assign-partitions tool? Is that safe to do whilst the apps are still
> running?
> 3. (not Streams-specific) If a broker that was the leader of
> partition(s) for a topic with RF 1 died (i.e. no replicas), and was
> non-recoverable - would the only way to restore service to those partitions
> be to delete the topic and re-create? (with data loss, understandably). I
> couldn't seem to achieve this using preferred/unclean leader election, nor
> using the re-assign-partitions tool, since that requires all brokers to be
> healthy.
>
> Many thanks in advance for any answers,
>
> Marcus
>
1. If you change REPLICATION_FACTOR_CONFIG without resetting the
application (or deleting the changelog and repartition topics) and
redeploy the Streams application, the replication factor of the internal
topics will not change. The replication factor will only change for new
deployments, i.e., Streams applications with a new application ID or
Streams applications that where reset. In both cases the internal topics
will be newly created.
2. Changing the replication factor of a topic directly on the brokers
should be fine. Kafka Streams should not re-create the internal topics
and not throw any exceptions.
3. Unfortunately, I do not know the answer to this question. Hopefully
somebody else can answer it.
I answered your questions to the best of my knowledge. The only way to
confirm my answers is to test (preferably in a test environment).
Best,
Bruno
On 28.04.21 17:34, Marcus Horsley-Rai wrote:
> Hi All,
>
> I'm in a sub-optimal situation whereby I have some Kafka Streams apps
> deployed to production, but the default replication factor set on the
> brokers was 1 when they were first deployed.
> As such, any state store changelog topics, and re-partition topics
> therefore have RF 1 also.
>
> I'm familiar with the bin/kafka-reassign-partitions.sh tool and how to use
> that.
> I've since also found the streams replication.factor
> (StreamsConfig.REPLICATION_FACTOR_CONFIG) setting that can be supplied in
> the streams apps config.
>
> My questions are:
>
> 1. Will simply changing the value of REPLICATION_FACTOR_CONFIG and
> re-deploying have any effect on already-created internal topics?
> 2. Conversely, should I just change the RF of the internal topics using
> the re-assign-partitions tool? Is that safe to do whilst the apps are still
> running?
> 3. (not Streams-specific) If a broker that was the leader of
> partition(s) for a topic with RF 1 died (i.e. no replicas), and was
> non-recoverable - would the only way to restore service to those partitions
> be to delete the topic and re-create? (with data loss, understandably). I
> couldn't seem to achieve this using preferred/unclean leader election, nor
> using the re-assign-partitions tool, since that requires all brokers to be
> healthy.
>
> Many thanks in advance for any answers,
>
> Marcus
>
Comments
Post a Comment