Re: Help on MM2 -- only copy new data

Thanks, Jamie:
I tried, but still not working☹, here is the updated properties file:
clusters=CDL,PC
CDL.bootstrap.servers=xxx

PC.bootstrap.servers=xx
PC.security.protocol=SSL

consumer.auto.offset.reset=earliest
group.id=test-mm2-2

CDL->PC.enabled=true
topics= .*
groups= .*
topic.blacklist= .*[\~\.]internal, .*\.replica, __.*, heartbeats,nas\.vantage.*,INTEGRATION.*, ADPRM\..*, ADP_LIFION_INTEGRATION\..*, AES\..*, BENEFITS\..*, DMX\..*, NASBILLING\..*, TAAS\..*
groups.blacklist= console-consumer-.*, connect-.*, __.*

tasks.max= 10
#emit.checkpoints.enabled= true
#sync.group.offsets.enabled= true
#offset.lag.max= 9223372036854775807

replication.factor= 3
checkpoints.topic.replication.factor= 3
heartbeats.topic.replication.factor= 3
offset-syncs.topic.replication.factor= 3
offset.storage.replication.factor= 3
status.storage.replication.factor= 3
config.storage.replication.factor= 3
transaction.state.log.replication.factor= 3

On 6/13/22, 4:25 PM, "Jamie" <jamiedd13@aol.co.uk.INVALID> wrote:


WARNING: Do not click links or open attachments unless you recognize the source of the email and know the contents are safe.

**********************************************************************
Hi An,
Try setting consumer.auto.offset.reset=earliest
This will mean if the consumers in the MM2 tasks don't have an offset for a topic partition they will start at the beginning.
If you've already enabled the below you might have to use a new group.id so that the consumers no longer have an offset that is valid. Since when you stated MM2 with below config it will now have an offset stored in the offset storage topic for each topic partition being consumed and therefore the auto.offset.reset config won't be applied.
Thanks,
Jamie


Sent from the all-new AOL app for iOS


On Tuesday, June 14, 2022, 12:19 am, An, Hongguo (CORP) <Hongguo.An@ADP.com.INVALID> wrote:

Hi:
I am trying to use MM2 ( 3.3.0) to migration Kafka to a new server, but looks like it only copys new data, any existing data is not copied.

Here is the connection properties:


clusters=CDL,PC

CDL.bootstrap.servers=xxx



PC.bootstrap.servers=xxx

PC.security.protocol=SSL



CDL->PC.enabled=true

topics= .*

groups= .*

topic.blacklist= .*[\~\.]internal, .*\.replica, __.*, heartbeats,nas\.vantage.*,INTEGRATION.*, ADPRM\..*, ADP_LIFION_INTEGRATION\..*, AES\..*, BENEFITS\..*, DMX\..*, NASBILLING\..*, TAAS\..*

groups.blacklist= console-consumer-.*, connect-.*, __.*



tasks.max= 10

#emit.checkpoints.enabled= true

#sync.group.offsets.enabled= true

#offset.lag.max= 9223372036854775807



replication.factor= 3

checkpoints.topic.replication.factor= 3

heartbeats.topic.replication.factor= 3

offset-syncs.topic.replication.factor= 3

offset.storage.replication.factor= 3

status.storage.replication.factor= 3

config.storage.replication.factor= 3

transaction.state.log.replication.factor= 3



Thanks in advance for any help


This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.