Docker Community Forums

Share and learn in the Docker community.

Connection refused when accessing using host machine IP address/hostname

docker

(Tomas Maconko) #1

Hello all,

I would need help figuring out, whether it is networking problem at host or infrastructure level, or in Docker itself.

Long story short: We are hosting Docker services on Linux servers using simple Docker setup (no Swarm). And after a while we spotted, that with a time running, after making multiple deployments, ports get being not accessible from outside the host. So I would like to get an advice, whether it is a Docker issue, or something with network setup.

Whole story: There are multiple services running, and due to picked application design we are allocating strict ports for each service (9009, 8555, …).

sudo docker ps -a
CONTAINER ID        IMAGE                                                         COMMAND                  CREATED             STATUS              PORTS                                                                                        NAMES
37f6d899a2c5        some.registry.com:5000/someapp.host:1.0.0     "dotnet SomeApp"   5 weeks ago         Up 5 weeks          0.0.0.0:9009->9009/tcp                                                                       projectname_someapp_1

For deploying the services we are using docker-compose files:

version: '2'
services:     
     
  someapp:
    image: some.registry.com:5000/someapp.host:1.0.0
    restart: always
    ports:
      - "9009:9009"
    environment:
      - LOGS_PATH=/data/logs/
      - STORAGE_PATH=/data/storage/
      - CONFIG_PATH=/data/config/
    volumes:
      - /storage/someapp/logs:/data/logs
      - /storage/someapp/config:/data/config
      - /storage/someapp/storage:/data/storage
    networks:
      - back_net
            
networks:
  back_net:
    driver: bridge
    driver_opts:
      com.docker.network.enable_ipv6: "false"
      com.docker.network.bridge.name: "back"

Everything was going smoothly. However, after we reduced number of deploymets/daemon restarts, ports started being inaccessible. So making a healthcheck:

curl -X GET http://127.0.0.1:9009/healthcheck
curl -X GET http://localhost:9009/healthcheck
curl -X GET http://10.10.10.10:9009/healthcheck
curl -X GET http://someserver:9009/healthcheck

Running the same commands from inside the Docker container return healthcheck response, however there is different situation, when running outside the container.

Running commands with localhost returns the correct data, but when we use host IP address or hostname, we have the following errors:

ERROR: curl: (7) Failed connect to 10.10.10.10:9009; Connection refused
ERROR: curl: (7) Failed connect to someserver:9009; Connection refused

The situation appeared for 40 more servers and 80 more services (random ports).

Anyone has faced the same problem? Are we doing something wrong here? Is it network problem or Docker one?

Additional information: As for deployment we are always doing:

sudo docker-compose -p projectname -f docker-compose.yml down

#some additional steps on host

sudo docker-compose -p projectname -f docker-compose.yml up -d

There is information about out Docker daemon:

sudo docker info
Containers: 9
 Running: 9
 Paused: 0
 Stopped: 0
Images: 349
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: vg_root-docker--latest--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 26.99 GB
 Data Space Total: 31.26 GB
 Data Space Available: 4.264 GB
 Metadata Space Used: 8.52 MB
 Metadata Space Total: 54.53 MB
 Metadata Space Available: 46.01 MB
 Thin Pool Minimum Free Space: 3.125 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2015-10-14)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host null overlay
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 24
Total Memory: 62.63 GiB
Name: someserver
ID: 3AXL:PMF7:T4P3:7ALA:ZNYE:LFRB:FTTV:XE2Z:2TZG:MASP:HKRD:LNCV
Docker Root Dir: /var/lib/docker-latest
Debug Mode (client): false
Debug Mode (server): false
Registry: https://registry.access.redhat.com/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 some.registry.com:5000
 127.0.0.0/8
Registries: registry.access.redhat.com (secure), docker.io (secure)