How to configure Swarm Service Discovery (1.12)

I’m running Docker for Windows 1.12.0-rc2 :heart_eyes: and trying to configure the new Swarm service discovery following this latest blog post. I think I’m missing a step in configuring the networking. I expect to be able to create a Docker service named ‘pending’ on port 3000 and navigate to ‘http://pending:3000’, but I am not able to resolve the service name. What am I missing here?

Steps to reproduce the behavior

1.) $ git clone https://github.com/AndrewBell/wet-tech
2.) $ docker compose build
3.) $ docker swarm init
4.) $ docker network create -d overlay mynet
5.) $ docker service create --replicas 1 --name pending --publish 3000:3000/tcp --network mynet recursivechaos/pending-service
6.) $ curl pending:3000

Playing around with this some more, in my Docker for Windows settings, I enabled “Expose container ports on localhost” under the Network tab. I can now ‘curl localhost:3000’, but when I scale via ‘docker service scale pending=3’, I am always routed to a single container.

Information

$docker info

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 14
Server Version: 1.12.0-rc2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 22
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host bridge overlay null
Swarm: active
 NodeID: 0yklbyqd7c5abuumxhok7xriu
 IsManager: Yes
 Managers: 1
 Nodes: 1
 CACertHash: sha256:b1f2eea025dd15b755f82e33e91014cdd1e640744532b7d722444f8e09d907ce
Runtimes: default
Default Runtime: default
Security Options: seccomp
Kernel Version: 4.4.13-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.951 GiB
Name: moby
ID: W5LB:TEKC:ACBQ:B2KF:HWF3:2IE4:ZGL7:I2LJ:UIHU:RATN:3B2N:6SQJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8

$ docker service inspect pending -p

ID:             cth0pdwlfyyoshcxt7ht1qwlf
Name:           pending
Mode:           Replicated
 Replicas:      1
Placement:
 Strategy:      Spread
UpdateConfig:
 Parallelism:   0
ContainerSpec:
 Image:         recursivechaos/pending-service
Resources:
Reservations:
Limits:
Networks: enndjmsxqdr9sg909zpttgfgmPorts:
 Name =
 Protocol = tcp
 TargetPort = 3000
 PublishedPort = 3000

$ docker network ls

NETWORK ID          NAME                DRIVER              SCOPE
6fccb3626b08        bridge              bridge              local
cb39a667eb69        docker_gwbridge     bridge              local
6e5b882d316a        host                host                local
1njmqwgiij8i        ingress             overlay             swarm
enndjmsxqdr9        mynet               overlay             swarm
6ffdc31c69e8        none                null                local
5b7bd030cff2        wettech_default     bridge              local

Judging by your inspect and info commands, you do have a single service task running.

Try running

docker service ls

to see what services and how many tasks currently run.

@umkus

Here’s what I see

$ docker service ls

ID            NAME     REPLICAS  IMAGE                           COMMAND
1y3i82dpeaa8  pending  1/1       recursivechaos/pending-service 

$ docker ps

CONTAINER ID        IMAGE                                   COMMAND             CREATED             STATUS              PORTS               NAMES
4ce90a15e2bd        recursivechaos/pending-service:latest   "npm start"         6 hours ago         Up 4 seconds        3000/tcp            pending.1.73weeotu20c4pbpvowbddbusb

Right, a single replica for that service.
So the scale command didn’t work or wasn’t run.
docker service scale pending=3 should fire up 2 extra containers. And the REPLICAS value should immediately change from 1/1 to something else, when your run docker service ls again.

Scaling works fine, I am able to use the scale command, and swarm will add additional containers. The problem I’m having is the service discover aspect of this. I should be able to curl pending:3000 and get a response for one of the containers, but I am not able to resolve the address.

Could you have forgotten to update /etc/hosts?

Refer

Hi, what do you mean by that? Don’t the hosts files get updated by docker themselves?
I am stuck at the same position. I want to have containers access services provided by other containers. From reading the docs I would expect the containers to be able to do “ping pending” and get the IP of the pending container. Is this correct? Does docker swarm start a DNS server somewhere in the background that should provide these mappings?

I’m on Windows, but I judging by the blogs, tutorials, and demos I was made to believe I didn’t need to muck around with any hosts files.

The blog post I followed goes through the same steps as the tutorial. Is there something I’m missing? Also, the tutorial doesn’t cover service discovery.

As of 1.12, what you described should be the desired behavior. Docker Swarm runs it’s own internal DNS service to route services by name.

I recreated your example, and yes, I needed to add:

127.0.0.1 pending

to /etc/hosts

Otherwise browser doesn’t know how to open the requested address.

My /etc/hosts on Mac host wasn’t updated.

Folks, I don’t think Docker for Mac/Windows is intended to publish services by hostname / DNS name lookup on the host computer at all (perhaps a maintainer of these projects can step in and correct me if I’m wrong). They expose the published ports from those services for access on localhost (or docker in Windows’s case).

In the above example, pending:3000 isn’t intended to be available from these tools. The ports get forwarded to localhost on OSX and a docker hostname on Windows. So it’d be:

localhost:3000 on OSX
docker:3000 on Windows

e.g. on Mac:

$ docker service create --publish 80:80 nginx
2oh65essgvf2xd2rg7r85c18x

$ # wait a second if nginx image not available locally..

$ curl localhost
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

p.s. – Editing /etc/hosts is not neccessary.

After reading your answer I think I was misunderstanding how the service routing works. I’d gone to David Gageot’s talk at DockerCon, and he was routing service to service via service names, which you can see embedded in his code, but I think what you said clicked. Externally these services won’t be available by name. I assume this functionality should be present internally though. Looks like everything is working as expected then.

@recursivechaos Oh yes, inside of containers on the same docker network you should be able to see a DNS entry for another container (or service) which resolves properly to that container’s virtual IP.

How can we use those DNS names outside?, Should we still use a KV for these?

You cannot today, however it should not be too much effort to rig up a service using $PROXY of your choice to resolve requests to the relevant downstream containers. e.g., you could expose a HAProxy service on port 80/443 which routes requests for foo.mycompany.com to service foo. (This is all just my own personal ideas, no de jure or even de facto solutions have emerged clearly yet).

Thanks, that’s how we solve these problems today, but I ask if there is something new in 1.12 related to these points.

I haven’t found anything about --endpoint-mode Endpoint mode (vip or dnsrr)??

Many Thanks

Well, the DNS entry inside service will automatically load balance to all instances of the service. So configuring your proxy is easier.

What is that flag for?