Docker Community Forums

Share and learn in the Docker community.

Service in container not reachable after a while when deployed using docker swarm with overlay network


(Tabka) #1

Issue type:

After deploying my stack in the swarm using a compose file everything work perfectly in the beginning but after a period of time (some times few hours , half day or even a day ) some container are not responding an we can access the service exposed on a specific port.
When I try to debug the issue with the command docker stack services test :
x43gw7k8sf57 test_webapi replicated 1/1 cm.registry.own:5000/cm/webapi:latest *:8888->8888/tcp

As you can see for the main container I can see that the post is published even if it’s not responding (I even checked inside the container and all the services are running perfectly and there is no issue with the api)

But when checking with docker ps

CONTAINER ID        IMAGE                                                 COMMAND                  CREATED              STATUS                  PORTS                                NAMES
7b823a6b5448        cm.registry.own:5000/cm/watcher:latest                "/bin/sh -c 'pytho..."   About a minute ago   Up About a minute                                            test_watcher.1.oelh0mwquvk2se41mknxjqzz3
1750165170ab        cm.registry.own:5000/cm/monitoring.collector:latest   "/bin/sh -c 'pytho..."   26 minutes ago       Up 26 minutes                                                test_monitoring-collector.1.aq7anztwh9lqgyoep8dp7zpk0
3466a8cce086        cm.registry.own:5000/cm/webapi:latest                 "/bin/bash /opt/cl..."   41 minutes ago       Up 41 minutes                                                test_webapi.1.t8v18681iiyu1jfy7xp3ccw2x
ed581690dca9        cm.registry.own:5000/cm/monit:latest                  "/bin/sh -c 'monit..."   42 minutes ago       Up 41 minutes                                                test_monit.1.wxut53t5gtv3to8g6dr6pp3m7
5ba1f81c5d5f        cm.registry.own:5000/cm/websocket:latest              "/bin/sh -c 'pytho..."   42 minutes ago       Up 41 minutes                                                test_ui-websocket.1.tk8qi3v14kdinv1tjd27ughid
815d9d06fb53        cm.registry.own:5000/cm/updater:latest                "python3 server.py"      42 minutes ago       

Up 41 minutes 8765/tcp test_updater.1.igfisma4hed5zhdbhro3homnw

As you can see here the port is not exposed like the other container even though in the compose file the port is published ( ports: - “8888:8888”) and even when I inspect the container :

docker inspect --format='{{.NetworkSettings.Ports}}' 3466a8cce086
map[]

As you can see no port is mapped and for other containers I find all the ports published.

I made the test using swarm with 2 nodes and even in a single node (deploying only on the manager) , the webapi container stop responding every time

OS Version/build:

Ubuntu 16.04.3 LTS (installed on a baremetall server not a VM)
Linux localhost 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

App version:

docker version
Client:
Version: 17.09.1-ce
API version: 1.32
Go version: go1.8.3
Git commit: 19e2cf6
Built: Thu Dec 7 22:24:23 2017
OS/Arch: linux/amd64

Server:
Version: 17.09.1-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: 19e2cf6
Built: Thu Dec 7 22:23:00 2017
OS/Arch: linux/amd64
Experimental: false