Re: Kafka connect issue with mysql - Debezium CDC

Hi,

I was able to resolve this issue by changing the name of server to docker container name.

Thanks,

Asmath

On Wed, Dec 25, 2019 at 8:30 PM KhajaAsmath Mohammed <mdkhajaasmath@gmail.com> wrote:

Hi,

I am trying to do POC for kafka CDC with database and ingest it into kafka. I have been trying to make it work from past 5 days and had no luck. can someone help me what is wrong in my approach.

Error Received:

{"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nUnable to connect: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.\nYou can also find the above list of errors at the endpoint `/{connectorType}/config/validate`"

I am able to connect mysql in docker container and also from my laptop using sqlconnector with host as localhost and port 3306,I even tried with IP address of database and no luck.

ALso tried various options of changing binding address for mysql and my assumption is that connection is being made but somehow it is not processing.

I can also see the request is sent to kafka connector where I can see the same error in distributed.log

Here are my code snippets.

Jar used is debezium-connector-mysql-1.0.0.Final

Script file to call the rest api request.

#!/bin/bash
curl -X POST http://localhost:8083/connectors  -H 'Content-Type: application/json' -H 'Accept: application/json'  -d @./lms_dev_source_jdbc_config.json

JSON file to send request to connector.

{
  "name": "mysql-connector-flattened",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "database.hostname":"localhost",
    "database.port": "3306",
    "database.user": "root",
    "database.password": "Bismillah1!",
    "database.server.id": "4",
    "database.server.name": "recipes_database",
    "database.history.kafka.bootstrap.servers": "localhost:9092",
    "database.history.kafka.topic": "dbhistory.fullfillment" ,  
    "include.schema.changes": "true" ,
    "transforms": "unwrap,changetopic",
    "transforms.unwrap.type": "io.debezium.transforms.UnwrapFromEnvelope",
    "transforms.changetopic.type":"org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.changetopic.regex":"(.*)",
    "transforms.changetopic.replacement":"$1-smt"
  }
}

DockerCompse File:

version: '2'

services:
  db:
    image: mysql:8.0
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    environment:
      MYSQL_DATABASE: db
      MYSQL_USER: debezium
      MYSQL_PASSWORD: Bismillah1!
      MYSQL_ROOT_PASSWORD: Bismillah1!
    ports:
      - 3306:3306
    expose:
      - 3306

  kafka-cluster:

    image: landoop/fast-data-dev:cp3.3.0
    restart: always
    environment:
      ADV_HOST: 127.0.0.1         # Change to 192.168.99.100 if using Docker Toolbox
      RUNTESTS: 0                 # Disable Running tests so the cluster starts faster
    ports:
      - 2181:2181                 # Zookeeper
      - 3030:3030                 # Landoop UI
      - 8081-8083:8081-8083       # REST Proxy, Schema Registry, Kafka Connect ports
      - 9581-9585:9581-9585       # JMX Ports
      - 9092:9092                 # Kafka Broker