Kafka

+1. Colin On Fri, Dec 27, 2019 at 12:29 PM Guozhang Wang < guozhang@confluent.io > wrote: > Hmm, I think that's pushed by someone by mistake, we can delete it. > > > Guozhang > > > On Thu, Dec 26, 2019 at 12:18 PM Matthias J. Sax < matthias@confluent.io > > wrote: > >> Should we delete `master` ? (Was not even aware it exists...) >> >> >> -Matthias >> >> On 12/25/19 9:50 AM, M. Manna wrote: >> > +1 with what John mentioned. >> > >> > Master is more like a template that gets created for new repo. It's not >> in >> > use for any Kafka activities (not that we know of). >> > >> > Regards, >> > >> > >> > On Wed, 25 Dec 2019 at 17:04, John Roesler < vvcephei@apache.org > wrote: >> > >> >> Hi Sachin, >> >> >> >> Trunk is the basis for de...

Hello Ryanne, thank you, that helps to get a better understanding. We'll just wait until something better is available and until then use the legacy-mode of MM2... Best regards Sebastian On 30-Dec-19 7:04 PM, Ryanne Dolan wrote: >> Is there a way to prevent that from happening? > Unfortunately there is no tooling (yet?) to manipulate Connect's offsets, > so it's difficult to force MM2 to skip ahead, reset, etc. > > One approach is to use Connect's Simple Message Transform feature. This > enables you to filter the messages being replicated, e.g. based on > timestamps, s.t. only recent messages are ever replicated. It's possible to > configure MM2 to use an existing SMT or you can write your own as a plug-in. > > Ryanne > > On Thu, Dec 26, 2019, 12:25 PM Sebastian Schmitz < > sebastian.schmitz@propellerhead.co.nz > wrote: > >> Hello Ryanne, >> >> Is there a way to ...

Hi Ryanne, > > Failed to fetch offsets > I have not encountered this. Is it possible one of the clusters is/was unreachable? Are any of the clusters using Kerberos or SSL that may be misconfigured? Clusters were reachable and working fine according to our monitoring. It might be some network issue between AWS regions, but it is not likely. I will keep monitoring this issue and let you know if it happens again. No Kerberos or SSL is used. Just plain text, with almost default configuration on both clusters. > > Plugin class loader... > I'll look into this. We use an old JVM on this: openjdk version "1.8.0_222", just guessing it may be the reason. Peter On Fri, 27 Dec 2019 at 19:11, Ryanne Dolan < ryannedolan@gmail.com > wrote: > Hey Peter. > > > No Authorizer is configured on the broker > > You'll need to disable ACL sync to avoid this error. It's harmless tho. > > > Failed to f...

> Is there a way to prevent that from happening? Unfortunately there is no tooling (yet?) to manipulate Connect's offsets, so it's difficult to force MM2 to skip ahead, reset, etc. One approach is to use Connect's Simple Message Transform feature. This enables you to filter the messages being replicated, e.g. based on timestamps, s.t. only recent messages are ever replicated. It's possible to configure MM2 to use an existing SMT or you can write your own as a plug-in. Ryanne On Thu, Dec 26, 2019, 12:25 PM Sebastian Schmitz < sebastian.schmitz@propellerhead.co.nz > wrote: > Hello Ryanne, > > Is there a way to prevent that from happening? We have two separate > clusters with some topics being replicated to the second one for > reporting. If we replicate everything again that reporting would > probably have some problems. > > Yes, I wondered when the Networking-guys would come and complain about > me using too...

Sebastian, you can drop in a custom jar in the "Connect plug-in path" and MM2 will be able to load it. That enables you to implement your own ReplicationPolicy (and other pluggable interfaces) without compiling everything. In an upcoming release we'll have a "LegacyReplicationPolicy" that does not rename topics. It's possible "SimpleReplicationPolicy" is a better name. Be advised that some features depend on correct ReplicationPolicy semantics, which LegacyReplicationPolicy will explicitly break. For example, MM2 cannot prevent cycles if topics are not renamed (or some other similar mechanism is used). Ryanne On Sun, Dec 29, 2019, 7:41 PM Sebastian Schmitz < sebastian.schmitz@propellerhead.co.nz > wrote: > Hello, > > I found that it's using the DefaultReplicationPolicy that always returns > "sourceClusterAlias + separator + topic" with only the separator being > configurable in the confi...

Hi, I was able to resolve this issue by changing the name of server to docker container name. Thanks, Asmath On Wed, Dec 25, 2019 at 8:30 PM KhajaAsmath Mohammed < mdkhajaasmath@gmail.com > wrote: Hi, I am trying to do POC for kafka CDC with database and ingest it into kafka. I have been trying to make it work from past 5 days and had no luck. can someone help me what is wrong in my approach. Error Received: {"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nUnable to connect: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.\nYou can also find the above list of errors at the endpoint `/{connectorType}/config/validate`" I am able to connect mysql in docker container and also from my laptop using sqlconnector with host as localhost and port 3306,I even tried with IP address of database and n...

Hello, I found that it's using the DefaultReplicationPolicy that always returns "sourceClusterAlias + separator + topic" with only the separator being configurable in the configuration-file with REPLICATION_POLICY_SEPARATOR. It seems like I need a different ReplicationPolicy, like a SimpleReplicationPolicy which always returns "topic" for the formatRemoteTopic, then. But that would mean that I can't download the Binaries and have to build the whole thing myself after adding the new Policy-file!? Or I could create a PR for a SimpleReplicationPolicy to be in some future build... Any suggestions for this? Thanks Sebastian On 30-Dec-19 1:39 PM, Sebastian Schmitz wrote: > Hello, > > another thing I found and didn't find any configuration in the KIP yet > was that if I have two clusters (source and target) and a topic > "replicateme" on the source-cluster it will get replicated to the > ta...

Hello, another thing I found and didn't find any configuration in the KIP yet was that if I have two clusters (source and target) and a topic "replicateme" on the source-cluster it will get replicated to the target-cluster as "source.replicateme". How can I stop it from adding the cluster-name in front of the topic-name on target-cluster? Thanks Sebastian On 27-Dec-19 7:24 AM, Sebastian Schmitz wrote: > Hello Ryanne, > > Is there a way to prevent that from happening? We have two separate > clusters with some topics being replicated to the second one for > reporting. If we replicate everything again that reporting would > probably have some problems. > > Yes, I wondered when the Networking-guys would come and complain about > me using too much bandwidth on the VPN-Link ;) > > Thanks > > Sebastian > > On 24-Dec-19 1:11 PM, Ryanne Dolan wrote: >> Glad to hear you are repli...