Connection Refused when connecting two containers in docker-compose

General
1

Hi all!

I’m new to (the AWESOME) docker and I’m facing a problem when trying to connect to a container from within another. The wierd stuff is that some containers can indeed be contacted, and others, configured in the same way, not. I tried a zillion fixes found while crawling google, nothing works. I guess this is a simple noob mistake though.

Here’s my docker-compose file:

version: '3.4'

services:

  mssql.db.posm:
    image: "microsoft/mssql-server-linux"
    environment:
        SA_PASSWORD: "mypassword"
        ACCEPT_EULA: "Y"
        MSSQL_PID: "Express"
    volumes:
      - mssql-data:/var/opt/mssql
    ports:
      - "1433:1433"
    networks:
      posm:

  api.posm:
    image: ${DOCKER_REGISTRY}posm.api
    build:
      context: .
      dockerfile: Posm.Api/Dockerfile
    expose:
      - "6869"
    ports:
      - "6869:80"
    networks:
      posm:

  cloud.subscription:
    image: ${DOCKER_REGISTRY}cloud.subscription
    build:
      context: ./Services
      dockerfile: Cloud.Subscription/Dockerfile
    ports:
      - "80"
    networks:
      posm:

  catalogmanager.services.posm:
    image: ${DOCKER_REGISTRY}posm.services.catalogmanager
    build:
      context: ./Services
      dockerfile: CatalogManager/Dockerfile
    ports:
      - "80"
    networks:
      posm:
        aliases:
          - services.posm
          - catalogmanager

  productmanager.services.posm:
    image: ${DOCKER_REGISTRY}posm.services.productmanager
    build:
      context: ./Services
      dockerfile: ProductManager/Dockerfile
    ports:
      - "80"
    networks:
      posm:
        aliases:
          - services.posm
          - productmanager

  localizer.services.posm:
    image: ${DOCKER_REGISTRY}posm.services.localizer
    build:
      context: ./Services
      dockerfile: Localizer/Dockerfile
    ports:
      - "80"
    networks:
      posm:
        aliases:
          - services.posm
          - localizer

networks:

  posm:

volumes:

  mssql-data:

So, when I connect to the api.posm, the latter is able to successfully connect to mssql.db.posm and to cloud.subscription. It however fails, with a connection refused error, to connect to ‘catalogmanager’ and ‘productmanager’, even though I can resolve their hostname from within api.posm… What is happening?

Thanks in advance for your precious help!

PS: I created a copy of that post on stackoverflow to maximize my chances to get an answer (asp.net - Connection Refused when connecting two containers in docker-compose - Stack Overflow)

2

Is api.posm service attempting to connect to host port 80?
From the compose-file web page, describing ‘ports:’

" Either specify both ports ( HOST:CONTAINER ), or just the container port (an ephemeral host port is chosen)."

So, catalogmanager.services.posm is on some ephemeral host port, not 80.

If api.posm is attempting to connect to the host interface port 80, then the above matters.

Secondly, I’m confused by the dots in the service name. I’ve never used them. I’ve also not attempted to use fqdn type names as service names. No need. I wonder if the dns client inside docker is confused too. I don’t know if this is root cause. Are you opposed to using simple service names (without the dots)? I don’t think it will resolve the connection problem. I do not know if its a problem.

3

Hello jjfraney!

First of all, thanks for you answer.

Is api.posm service attempting to connect to host port 80?

Well, from my understanding, the HOST port is necessary if I want to contact the container from outside of the vm, isn’t it? api.posm connects to the catalogmanager.services.posm on port 80 indeed, but from within the vm, as it does for the cloud.subscription service, which is working on port 80.

So, catalogmanager.services.posm is on some ephemeral host port, not 80.

So then why can cloud.subscription be contacted on 80? Is it because it is the first being given the port, and the other services cannot get it? I though that many services could have the same port, but maybe I might have misunderstood something, as I said I’m a noob regarding docker.

Secondly, I’m confused by the dots in the service name. I’ve never used them (…)

Well, I always liked FQDN, what can I say :stuck_out_tongue: ? It seems to be working fine though(except for those two damned services of course), even if I must say I used the network aliases anyways for interservice connection. I’ll check if that’s the cause.

4

the HOST port is necessary if I want to contact the container from outside of the vm, isn’t it?

Yes. You didn’t specify which ip address api.posm was using. There is no need to use ‘ports’ if no client connects to the service through the host interface, but you have it there for catalogmanager.services.posm. If api.posm was attempting host:80, then it could get connection refused.

One reason for connection refused could be that catalogmanager.service.posm is not actually healthy. It could look like its running, but maybe in a restart cycle, or it hasn’t opened 80 for some reason.

I notice you have alias ‘services.pom’ for both catalogmanager and productmanager. I don’t think that can work: two nodes with the same hostname? No. Maybe this prevents one from joining the network.

You can run a connectivity test, if api.posm has curl or ping. exec into api.posm and run curl to the catalogmanager.services.posm endpoint, or ping it.

You can run ‘docker ps’ to see what uptime of catalogmanager. If its short, then you have an unhealthy service.

Maybe you can verify health by looking at catalogmanager logs?

5

One reason for connection refused could be that catalogmanager.service.posm is not actually healthy

I checked. It has been up for as long of other services, and logs are empty. So I guess the service is healthy.

I notice you have alias ‘services.pom’ for both catalogmanager and productmanager. I don’t think that can work: two nodes with the same hostname? No. Maybe this prevents one from joining the network.

I thought I read in some articles you could ‘tag’ multiple services with the same alias, for discovery purposes. I might have gotten this wrong. I changed the docker-compose file so that no service reproduce the alias of any other. It changes nothing.

You can run a connectivity test, if api.posm has curl or ping. exec into api.posm and run curl to the catalogmanager.services.posm endpoint, or ping it.

I did it. Dns resolves as expected with service name or aliases. It just refuses the connection on port 80 and I tried a port scan on it, without any success (I must say I did not run through the whole of it, though, but I made sure all relevant ports have been tested, without success)

One piece of information I had forgotten to provide is the result from ‘docker ps’ command, maybe it might help you help me :wink:

CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                            NAMES
ecd04e4139c8        posm.services.catalogmanager:dev   "tail -f /dev/null"      11 minutes ago      Up 11 minutes       0.0.0.0:32794->80/tcp            dockercompose15927292303634697816_catalogmanager.services.posm_1
86af927db007        posm.services.localizer:dev        "tail -f /dev/null"      11 minutes ago      Up 11 minutes       0.0.0.0:32793->80/tcp            dockercompose15927292303634697816_localizer.services.posm_1
85d4b3c28335        cloud.subscription:dev             "tail -f /dev/null"      11 minutes ago      Up 11 minutes       0.0.0.0:32792->80/tcp            dockercompose15927292303634697816_cloud.subscription_1
eb535a710290        posm.api:dev                       "tail -f /dev/null"      11 minutes ago      Up 11 minutes       6869/tcp, 0.0.0.0:6869->80/tcp   dockercompose15927292303634697816_api.posm_1
d3c5c31526d8        posm.services.productmanager:dev   "tail -f /dev/null"      11 minutes ago      Up 11 minutes       0.0.0.0:32791->80/tcp            dockercompose15927292303634697816_productmanager.services.posm_1
3b63e4251f12        microsoft/mssql-server-linux       "/opt/mssql/bin/sqls…"   11 minutes ago      Up 11 minutes       0.0.0.0:1433->1433/tcp           dockercompose15927292303634697816_mssql.db.posm_1
6

I’m flat out of suggestions. Im forced to doubt that the application actualy opens port 80 for listening. That seems far more likely than the alternate: a bug in docker. Proving the port is open could be done with something like netstat. Is that available in the catalogmanager container? Netstat -l i think.

7

Hmmm, Im feeling a bit stupid here: I tried to run netsat into my catalogmanager container, only to realize there was an exception being thrown by my service, which was, as you suggested, not starting up. Wierd thing is that my debbuger should have informed me somehow…

Sorry for that, and thanks again for your time!