My IDE which is sitting outside of the kube cluster creates a producer that attempts to connect to kafka using the cluster dns name of the headless service

Hi!

The Kafka chart being used is here https://github.com/bitnami/charts/tree/master/bitnami/kafka

| chart | application |
| kafka-1.10.1 | 2.2.0 |


$ kubectl get svc -n kafka

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kafka-janitha ClusterIP 10.103.152.8 <none> 9092/TCP 5d19h
kafka-janitha-headless ClusterIP None <none> 9092/TCP 5d19h
kafka-janitha-zookeeper ClusterIP 10.108.191.161 <none> 2181/TCP,2888/TCP,3888/TCP 5d19h
kafka-janitha-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 5d19h

I am then port-forwarding like so

kubectl port-forward --namespace postgresql svc/postgres-janitha-postgresql 5432:5432 --address 0.0.0.0 &
kubectl port-forward --namespace kafka svc/kafka-janitha 9092:9092 --address 0.0.0.0 &
kubectl port-forward --namespace activemq svc/activemq-tideworks-activemq 61616:61616 --address 0.0.0.0 &

I have my application properties set like so where the ip is the private ip of the node (master). My IDE is running outside the cluster.

tc.db.jdbc.url=jdbc:postgresql://172.31.31.192:5432/tcb_configuration
#
activemq.base.url=tcp://172.31.31.192:61616?wireFormat.maxInactivityDuration=-1
#
kafka.bootstrap.servers=172.31.31.192:9092

The application log states the following

17:36:45.617 [kafka-producer-network-thread | producer-3] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Give up sending metadata request since no node is available
17:36:45.618 [kafka-producer-network-thread | producer-5] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-5] Give up sending metadata request since no node is available
17:36:45.667 [kafka-producer-network-thread | producer-3] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Give up sending metadata request since no node is available
17:36:45.668 [kafka-producer-network-thread | producer-5] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-5] Give up sending metadata request since no node is available
17:36:45.717 [kafka-producer-network-thread | producer-3] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Give up sending metadata request since no node is available
17:36:45.718 [kafka-producer-network-thread | producer-5] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-5] Give up sending metadata request since no node is available
17:36:45.768 [kafka-producer-network-thread | producer-3] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Give up sending metadata request since no node is available
17:36:45.768 [kafka-producer-network-thread | producer-5] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-5] Give up sending metadata request since no node is available
17:36:45.818 [kafka-producer-network-thread | producer-3] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Give up sending metadata request since no node is available
17:36:45.818 [kafka-producer-network-thread | producer-5] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-5] Give up sending metadata request since no node is available
17:36:45.868 [kafka-producer-network-thread | producer-3] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Give up sending metadata request since no node is available
17:36:45.869 [kafka-producer-network-thread | producer-5] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-5] Give up sending metadata request since no node is available
17:36:45.918 [kafka-producer-network-thread | producer-3] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Give up sending metadata request since no node is available
17:36:45.919 [kafka-producer-network-thread | producer-5] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-5] Give up sending metadata request since no node is available
17:36:45.969 [kafka-producer-network-thread | producer-3] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Initialize connection to node kafka-janitha-0.kafka-janitha-headless.kafka.svc.cluster.local:9092 (id: 1001 rack: null) for sending metadata request
17:36:45.969 [kafka-producer-network-thread | producer-3] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Initiating connection to node kafka-janitha-0.kafka-janitha-headless.kafka.svc.cluster.local:9092 (id: 1001 rack: null)
17:36:45.969 [kafka-producer-network-thread | producer-5] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-5] Give up sending metadata request since no node is available
17:36:45.969 [kafka-producer-network-thread | producer-3] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Error connecting to node kafka-janitha-0.kafka-janitha-headless.kafka.svc.cluster.local:9092 (id: 1001 rack: null)
java.io.IOException: Can't resolve address: kafka-janitha-0.kafka-janitha-headless.kafka.svc.cluster.local:9092
at org.apache.kafka.common.network.Selector.doConnect(Selector.java:223) ~[kafka-clients-1.0.2.jar:?]
at org.apache.kafka.common.network.Selector.connect(Selector.java:202) ~[kafka-clients-1.0.2.jar:?]
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:793) [kafka-clients-1.0.2.jar:?]
at org.apache.kafka.clients.NetworkClient.access$700(NetworkClient.java:62) [kafka-clients-1.0.2.jar:?]
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:944) [kafka-clients-1.0.2.jar:?]
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:848) [kafka-clients-1.0.2.jar:?]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:458) [kafka-clients-1.0.2.jar:?]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239) [kafka-clients-1.0.2.jar:?]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163) [kafka-clients-1.0.2.jar:?]
at java.lang.Thread.run(Thread.java:844) [?:?]
Caused by: java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:112) ~[?:?]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[?:?]
at org.apache.kafka.common.network.Selector.doConnect(Selector.java:221) ~[kafka-clients-1.0.2.jar:?]
... 9 more

I think this is because my IDE which is sitting outside of the kube cluster is creates a producer that attempts to connect to kafka using the cluster dns name of the headless service.

I haven't finished checking whether the application will run inside the kube cluster, but likely yes given that the following works

$ kubectl create -f https://k8s.io/examples/admin/dns/busybox.yaml

$ kubectl -n default exec -ti busybox -- nslookup kafka-janitha-0.kafka-janitha-headless.kafka.svc.cluster.local
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name: kafka-janitha-0.kafka-janitha-headless.kafka.svc.cluster.local
Address 1: 192.168.150.22 192-168-150-22.kafka-janitha.kafka.svc.cluster.local

Is sort of setup possible? What am I missing / what do I not know?

Thanks

Janitha