I would need help figuring out, whether it is networking problem at host or infrastructure level, or in Docker itself.
Long story short: We are hosting Docker services on Linux servers using simple Docker setup (no Swarm). And after a while we spotted, that with a time running, after making multiple deployments, ports get being not accessible from outside the host. So I would like to get an advice, whether it is a Docker issue, or something with network setup.
Whole story: There are multiple services running, and due to picked application design we are allocating strict ports for each service (9009, 8555, …).
sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 37f6d899a2c5 some.registry.com:5000/someapp.host:1.0.0 "dotnet SomeApp" 5 weeks ago Up 5 weeks 0.0.0.0:9009->9009/tcp projectname_someapp_1
For deploying the services we are using docker-compose files:
version: '2' services: someapp: image: some.registry.com:5000/someapp.host:1.0.0 restart: always ports: - "9009:9009" environment: - LOGS_PATH=/data/logs/ - STORAGE_PATH=/data/storage/ - CONFIG_PATH=/data/config/ volumes: - /storage/someapp/logs:/data/logs - /storage/someapp/config:/data/config - /storage/someapp/storage:/data/storage networks: - back_net networks: back_net: driver: bridge driver_opts: com.docker.network.enable_ipv6: "false" com.docker.network.bridge.name: "back"
Everything was going smoothly. However, after we reduced number of deploymets/daemon restarts, ports started being inaccessible. So making a healthcheck:
curl -X GET http://127.0.0.1:9009/healthcheck curl -X GET http://localhost:9009/healthcheck curl -X GET http://10.10.10.10:9009/healthcheck curl -X GET http://someserver:9009/healthcheck
Running the same commands from inside the Docker container return healthcheck response, however there is different situation, when running outside the container.
Running commands with localhost returns the correct data, but when we use host IP address or hostname, we have the following errors:
ERROR: curl: (7) Failed connect to 10.10.10.10:9009; Connection refused ERROR: curl: (7) Failed connect to someserver:9009; Connection refused
The situation appeared for 40 more servers and 80 more services (random ports).
Anyone has faced the same problem? Are we doing something wrong here? Is it network problem or Docker one?
Additional information: As for deployment we are always doing:
sudo docker-compose -p projectname -f docker-compose.yml down #some additional steps on host sudo docker-compose -p projectname -f docker-compose.yml up -d
There is information about out Docker daemon:
sudo docker info
Containers: 9 Running: 9 Paused: 0 Stopped: 0 Images: 349 Server Version: 1.12.1 Storage Driver: devicemapper Pool Name: vg_root-docker--latest--pool Pool Blocksize: 524.3 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: Metadata file: Data Space Used: 26.99 GB Data Space Total: 31.26 GB Data Space Available: 4.264 GB Metadata Space Used: 8.52 MB Metadata Space Total: 54.53 MB Metadata Space Available: 46.01 MB Thin Pool Minimum Free Space: 3.125 GB Udev Sync Supported: true Deferred Removal Enabled: true Deferred Deletion Enabled: true Deferred Deleted Device Count: 0 Library Version: 1.02.107-RHEL7 (2015-10-14) Logging Driver: journald Cgroup Driver: systemd Plugins: Volume: local Network: bridge host null overlay Authorization: rhel-push-plugin Swarm: inactive Runtimes: docker-runc runc Default Runtime: docker-runc Security Options: seccomp Kernel Version: 3.10.0-327.el7.x86_64 Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo) OSType: linux Architecture: x86_64 Number of Docker Hooks: 2 CPUs: 24 Total Memory: 62.63 GiB Name: someserver ID: 3AXL:PMF7:T4P3:7ALA:ZNYE:LFRB:FTTV:XE2Z:2TZG:MASP:HKRD:LNCV Docker Root Dir: /var/lib/docker-latest Debug Mode (client): false Debug Mode (server): false Registry: https://registry.access.redhat.com/v1/ WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled Insecure Registries: some.registry.com:5000 127.0.0.0/8 Registries: registry.access.redhat.com (secure), docker.io (secure)