How to communicate two docker containers from two different hosts

Docker for AWS
1

Expected behavior

Actual behavior

Additional Information

Steps to reproduce the behavior

  1. …
  2. …
2 
$ docker network create -d overlay mynet
62b36zpsbweq6w2nss7rce12y

$ docker run -d --name spam --net mynet alpine nc -l -p 8080
722aec95b2ff377566ee09cd782d1158d69af1a3276b042476592b7013613760

$ docker run -d --net mynet alpine sh -c 'echo hello spam | nc spam 8080'
15bd93b13405464ea99d5b86f26dbc578d30b4879df4ce7f51f3d9bfc0c82a2e

$ docker logs spam
hello spam

You can do similar effect with docker service networking.

3

Hi,

I have a problem related to overlay networks. Here is an example for docker service that works:

~ $ docker network create -d overlay test
~ $ docker service create --name test-sink --network test alpine sh -c 'while true; do nc -l -p 8080; done'
~ $ docker service create --name test-spam --network test alpine sh -c 'echo hello spam | nc test-sink 8080'

Below is another example where listening netcat is bound to containers interface and it does not work (hello messages wont get through). Is this supposed to work like this or is this a bug? I’m asking since I have problem starting cluster with official Cassandra containers where Cassandra refuses to bind to 0.0.0.0 (I have to specify some address for Cassandra).

~ $ docker network create -d overlay test
0d8fe93f139e707ba5dabb225b6737307f4348fa1b529bd9780a25d4cc930648
~ $ docker service create --name test-sink --network test alpine sh -c 'while true; do nc -l -s test-sink -p 8080; done'
~ $ docker service create --name test-spam --network test alpine sh -c 'echo hello spam | nc test-sink 8080'
4

Hi

I’m new to docker. As of my understand in your problem need to expose Port ips while running containers. did you check with this. i think it will work but not sure.

5

In your second example why specify -s test-sink? test-sink is not a local address or even the hostname. it’s simply the case that dns server in docker returns the container ip for that dns entry in containers on that network

        -s ADDR Local address

Why doesn’t binding to 0.0.0.0 work? What’s the error?

6

org.apache.cassandra.exceptions.ConfigurationException: listen_address cannot be a wildcard address (0.0.0.0)

From cassandra.yaml:

You must change this if you want multiple nodes to be able to communicate!

Set listen_address OR listen_interface, not both. Interfaces must correspond
to a single address, IP aliasing is not supported.
Leaving it blank leaves it up to InetAddress.getLocalHost(). This
will always do the Right Thing if the node is properly configured
(hostname, name resolution, etc), and the Right Thing is to use the
address associated with the hostname (it might not be).

Setting listen_address to 0.0.0.0 is always wrong.

7

Weird that they won’t let you do this, but I think you may want to set listen_interface instead. If you set that to eth0 I think that may resolve to the correct IP address. This seems to be the correct interface for resolving on the overlay but I’m not sure how consistent eth0 / eth1 corresponding to a given network is.

$ docker exec -ti aa36 ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:0A:00:04:03
          inet addr:10.0.4.3  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::42:aff:fe00:403%32722/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1206 (1.1 KiB)  TX bytes:648 (648.0 B)

eth1      Link encap:Ethernet  HWaddr 02:42:AC:12:00:03
          inet addr:172.18.0.3  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe12:3%32722/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1148 errors:0 dropped:0 overruns:0 frame:0
          TX packets:468 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1605587 (1.5 MiB)  TX bytes:26253 (25.6 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1%32722/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:2075 (2.0 KiB)  TX bytes:2075 (2.0 KiB)

$ docker exec -ti aa36 curl 10.0.4.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
8

Update: @mrjana suggested to me that the interfaces might change, so the above approach is not good. Try setting the listen_address to service_name.network (e.g., if the service is called cassandra and network is called mynet, set it to cassandra.mynet).

9

Thanks Nathanleclaire,

I’m able to communivate two containers from different hosts using overlay global network.

thanks.

2 Likes
11

Thanks, got it working now. setting cassandra_listen_on_broadcast_address and listen_address correctly helped.

2 Likes
12

I am having the same issue:
Running first instance of cassandra in node “vm1”
Output of netstat -ntl inside container

tcp 0 0 127.0.0.1:42735 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.11:36303 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:9042 0.0.0.0:* LISTEN
tcp 0 0 10.0.0.7:7000 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:7199 0.0.0.0:* LISTEN

cannot telnet from other container in other node in the same overlay network to 10.0.0.7 7000

I’ve done what @nathanleclaire suggested

Try setting the listen_address to service_name.network

My env variables for Cassandra:

CASSANDRA_LISTEN_ADDRESS: “test_cassandra-1.test_sandd-net”
CASSANDRA_BROADCAST_ADDRESS: “cassandra-1”

cassandra-1 resolves to 10.0.0.7 but I get connection refused on this port
I had to prefix service name and network name with “test” because I ran a stack in a swarm.

PS Found the issue I had to attach port also to 10.0.0.6 (hostname --ip-address)