Hello all,
I would need help figuring out, whether it is networking problem at host or infrastructure level, or in Docker itself.
Long story short: We are hosting Docker services on Linux servers using simple Docker setup (no Swarm). And after a while we spotted, that with a time running, after making multiple deployments, ports get being not accessible from outside the host. So I would like to get an advice, whether it is a Docker issue, or something with network setup.
Whole story: There are multiple services running, and due to picked application design we are allocating strict ports for each service (9009, 8555, …).
sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
37f6d899a2c5 some.registry.com:5000/someapp.host:1.0.0 "dotnet SomeApp" 5 weeks ago Up 5 weeks 0.0.0.0:9009->9009/tcp projectname_someapp_1
For deploying the services we are using docker-compose files:
version: '2'
services:
someapp:
image: some.registry.com:5000/someapp.host:1.0.0
restart: always
ports:
- "9009:9009"
environment:
- LOGS_PATH=/data/logs/
- STORAGE_PATH=/data/storage/
- CONFIG_PATH=/data/config/
volumes:
- /storage/someapp/logs:/data/logs
- /storage/someapp/config:/data/config
- /storage/someapp/storage:/data/storage
networks:
- back_net
networks:
back_net:
driver: bridge
driver_opts:
com.docker.network.enable_ipv6: "false"
com.docker.network.bridge.name: "back"
Everything was going smoothly. However, after we reduced number of deploymets/daemon restarts, ports started being inaccessible. So making a healthcheck:
curl -X GET http://127.0.0.1:9009/healthcheck
curl -X GET http://localhost:9009/healthcheck
curl -X GET http://10.10.10.10:9009/healthcheck
curl -X GET http://someserver:9009/healthcheck
Running the same commands from inside the Docker container return healthcheck response, however there is different situation, when running outside the container.
Running commands with localhost returns the correct data, but when we use host IP address or hostname, we have the following errors:
ERROR: curl: (7) Failed connect to 10.10.10.10:9009; Connection refused
ERROR: curl: (7) Failed connect to someserver:9009; Connection refused
The situation appeared for 40 more servers and 80 more services (random ports).
Anyone has faced the same problem? Are we doing something wrong here? Is it network problem or Docker one?
Additional information: As for deployment we are always doing:
sudo docker-compose -p projectname -f docker-compose.yml down
#some additional steps on host
sudo docker-compose -p projectname -f docker-compose.yml up -d
There is information about out Docker daemon:
sudo docker info
Containers: 9
Running: 9
Paused: 0
Stopped: 0
Images: 349
Server Version: 1.12.1
Storage Driver: devicemapper
Pool Name: vg_root-docker--latest--pool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 26.99 GB
Data Space Total: 31.26 GB
Data Space Available: 4.264 GB
Metadata Space Used: 8.52 MB
Metadata Space Total: 54.53 MB
Metadata Space Available: 46.01 MB
Thin Pool Minimum Free Space: 3.125 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.107-RHEL7 (2015-10-14)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host null overlay
Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 24
Total Memory: 62.63 GiB
Name: someserver
ID: 3AXL:PMF7:T4P3:7ALA:ZNYE:LFRB:FTTV:XE2Z:2TZG:MASP:HKRD:LNCV
Docker Root Dir: /var/lib/docker-latest
Debug Mode (client): false
Debug Mode (server): false
Registry: https://registry.access.redhat.com/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
some.registry.com:5000
127.0.0.0/8
Registries: registry.access.redhat.com (secure), docker.io (secure)