Hi Kota,
Thanks for your questions!
> - Are there any circumstances where the target (MSK) offsets could be
> overwritten by smaller source offsets while consumers are already running
> on MSK?
> - I assume that this kind of behavior is not intended by design.
No, as soon as you start the first consumer on the target cluster, MM2
offset syncing will stop for that group and offsets can only be changed by
the target consumers. This is a property enforced by the consumer group
coordinator, not a MM2 feature specifically.
However, it is extremely likely that the target consumer will get some
messages that already had offsets committed by the source consumers, even
if you stop the source consumers first. Your consumer application should be
able to tolerate this re-delivery if you want to do a live/staged migration
(consumers before producers) like you described.
> - Will new messages written to onpre-kafka continue to replicate to MSK
> via MM2, and remain readable by consumers from MSK?
Yes, the MirrorSourceConnector will continue mirroring the topic. Consumers
on the target cluster will be able to read the mirrored messages and commit
offsets to the target cluster.
Hope this helps,
Greg
On Thu, Sep 11, 2025 at 12:50 PM Kota Yagi <mdryzk64smsh@gmail.com> wrote:
> Summary
> I'm migrating from an on-prem Apache Kafka cluster ("onpre-kafka") to an
> AWS MSK cluster using MirrorMaker 2 (MM2).
> After enabling one-way replication (onpre-kafka → MSK), I plan to cut
> consumers over to MSK while producers still write to onpre-kafka.
> I'd like to confirm the exact offset semantics.
>
> Topology
>
> - Source: onpre-kafka
> - Target: MSK
> - Replication: MM2 (MirrorSourceConnector + MirrorHeartbeatConnector +
> MirrorCheckpointConnector)
> - Direction: source → target (one-way)
> - Producers: continue writing to onpre-kafka
> - Consumers: switch from onpre-kafka to MSK after offset translation
>
> Scenario
>
> 1. MM2 replicates topics from onpre-kafka to MSK.
> 2. I start consuming from MSK (consumers move), while producers still
> write to onpre-kafka.
> 3. MSK consumer offsets advance (e.g., 100 → 102). onpre-kafka consumer
> group offsets remain at 100.
>
>
> Questions
>
> - Are there any circumstances where the target (MSK) offsets could be
> overwritten by smaller source offsets while consumers are already
> running
> on MSK?
> - I assume that this kind of behavior is not intended by design.
> - Will new messages written to onpre-kafka continue to replicate to MSK
> via MM2, and remain readable by consumers from MSK?
> - I assume that consumers can keep consuming new messages from MSK
> that are replicated via MM2.
>
>
> Thanks in advance!
>
Thanks for your questions!
> - Are there any circumstances where the target (MSK) offsets could be
> overwritten by smaller source offsets while consumers are already running
> on MSK?
> - I assume that this kind of behavior is not intended by design.
No, as soon as you start the first consumer on the target cluster, MM2
offset syncing will stop for that group and offsets can only be changed by
the target consumers. This is a property enforced by the consumer group
coordinator, not a MM2 feature specifically.
However, it is extremely likely that the target consumer will get some
messages that already had offsets committed by the source consumers, even
if you stop the source consumers first. Your consumer application should be
able to tolerate this re-delivery if you want to do a live/staged migration
(consumers before producers) like you described.
> - Will new messages written to onpre-kafka continue to replicate to MSK
> via MM2, and remain readable by consumers from MSK?
Yes, the MirrorSourceConnector will continue mirroring the topic. Consumers
on the target cluster will be able to read the mirrored messages and commit
offsets to the target cluster.
Hope this helps,
Greg
On Thu, Sep 11, 2025 at 12:50 PM Kota Yagi <mdryzk64smsh@gmail.com> wrote:
> Summary
> I'm migrating from an on-prem Apache Kafka cluster ("onpre-kafka") to an
> AWS MSK cluster using MirrorMaker 2 (MM2).
> After enabling one-way replication (onpre-kafka → MSK), I plan to cut
> consumers over to MSK while producers still write to onpre-kafka.
> I'd like to confirm the exact offset semantics.
>
> Topology
>
> - Source: onpre-kafka
> - Target: MSK
> - Replication: MM2 (MirrorSourceConnector + MirrorHeartbeatConnector +
> MirrorCheckpointConnector)
> - Direction: source → target (one-way)
> - Producers: continue writing to onpre-kafka
> - Consumers: switch from onpre-kafka to MSK after offset translation
>
> Scenario
>
> 1. MM2 replicates topics from onpre-kafka to MSK.
> 2. I start consuming from MSK (consumers move), while producers still
> write to onpre-kafka.
> 3. MSK consumer offsets advance (e.g., 100 → 102). onpre-kafka consumer
> group offsets remain at 100.
>
>
> Questions
>
> - Are there any circumstances where the target (MSK) offsets could be
> overwritten by smaller source offsets while consumers are already
> running
> on MSK?
> - I assume that this kind of behavior is not intended by design.
> - Will new messages written to onpre-kafka continue to replicate to MSK
> via MM2, and remain readable by consumers from MSK?
> - I assume that consumers can keep consuming new messages from MSK
> that are replicated via MM2.
>
>
> Thanks in advance!
>
Comments
Post a Comment