Replying to myself I ended by making a small python utility that do the
work.
On 12/08/2022 12:50, Raphael Mazelier wrote:
> Hi kafka community,
>
> I've been an happy user of kafka since years. Today I faced a problem
> that begins to make mad...
>
> So we have the project to migrate from and old (and unmonitored,
> etc..) kafka cluster to multiple new fresh clusters. Great.
>
> So the plan was clear, identify source topics and source groups, use
> mirromarker2 with whitelist configuration and migrate consumer by
> consumer on the fresh cluster, then producer.
>
> For that we need that consumer group should be synced at some point
> (if not every consumer will restart from its own strategy).
>
> That were the problem begins. Mirrormaker2 seems to works well
> replicating topics, but on consumer it's very erratic to say the
> least. My temporary conclusion is that mm2 only "replicate" empty
> consumer group (which can make sense?), but inspecting the offset show
> me that they are not sync.
>
> I tried to dig into config/log and I never find something that works
> reliably. (btw the documentation of mirrormaker2 is just horrible, we
> don't what config stanza is supposed to be applied, and reading the
> log is .. hmm ... ).
>
> I also try confluent replicator; which is also replicating well
> topics, but not consumer group.
>
> So first question:
> - is someone have already succeed making work mm2 with consumer group
> translation/replication?
>
> B Plan:
> Assuming I can shut my consumers for some time, assuming also I have
> the exact same topics/part topology so I don't translation.
>
>
> I have in the idea to find a way to take manually a snapshot of each
> consumer group involved, and recreate them on the destination cluster.
>
> It should be possible (after all mm2 is supposed to do it); and after
> all it's just a matter of manually committing offset by partition.
> kafka-topics propose an option to reset offset to earliest or a time
> (maybe its sufficient we will see) ; but I would have prefer a tool to
> create the exact same consumer group in the dest cluster.
>
> Is someone have already done that? and know an tool to do it?
>
> Any help appreciated :)
>
> --
> Raphael Mazelier
>
>
work.
On 12/08/2022 12:50, Raphael Mazelier wrote:
> Hi kafka community,
>
> I've been an happy user of kafka since years. Today I faced a problem
> that begins to make mad...
>
> So we have the project to migrate from and old (and unmonitored,
> etc..) kafka cluster to multiple new fresh clusters. Great.
>
> So the plan was clear, identify source topics and source groups, use
> mirromarker2 with whitelist configuration and migrate consumer by
> consumer on the fresh cluster, then producer.
>
> For that we need that consumer group should be synced at some point
> (if not every consumer will restart from its own strategy).
>
> That were the problem begins. Mirrormaker2 seems to works well
> replicating topics, but on consumer it's very erratic to say the
> least. My temporary conclusion is that mm2 only "replicate" empty
> consumer group (which can make sense?), but inspecting the offset show
> me that they are not sync.
>
> I tried to dig into config/log and I never find something that works
> reliably. (btw the documentation of mirrormaker2 is just horrible, we
> don't what config stanza is supposed to be applied, and reading the
> log is .. hmm ... ).
>
> I also try confluent replicator; which is also replicating well
> topics, but not consumer group.
>
> So first question:
> - is someone have already succeed making work mm2 with consumer group
> translation/replication?
>
> B Plan:
> Assuming I can shut my consumers for some time, assuming also I have
> the exact same topics/part topology so I don't translation.
>
>
> I have in the idea to find a way to take manually a snapshot of each
> consumer group involved, and recreate them on the destination cluster.
>
> It should be possible (after all mm2 is supposed to do it); and after
> all it's just a matter of manually committing offset by partition.
> kafka-topics propose an option to reset offset to earliest or a time
> (maybe its sufficient we will see) ; but I would have prefer a tool to
> create the exact same consumer group in the dest cluster.
>
> Is someone have already done that? and know an tool to do it?
>
> Any help appreciated :)
>
> --
> Raphael Mazelier
>
>
Comments
Post a Comment