Container can't connect to another container on same host using external DNS or host IP

I have the following setup…

  • Public DNS entry to point to a digital ocean droplet, lets call it foo.com

  • On the DO droplet, docker is running with a docker compose file with multiple services

  • The docker compose file is using the default bridge network

  • One service, lets call it service A, in the compose file handles http/https traffic (ports 80, 443)

  • Service B needs to do http(s) requests to service A and uses the public DNS foo.com as the host when performing these requests because I don’t want B to depend on A running on the same docker host and network. B and A will most likely end up on separate hosts/networks down the road.

From container B, when I do curl http://foo.com the request just times out.

Doing ping foo.com from inside B correctly resolves to the docker host IP address (the DO droplet) and the pings return.

My expectation for the curl request is that DNS would resolve to the docker host IP address, send the request there, which should then be be handled by the docker client and handed off to service A because ports 80, 443 are exposed.

Performing curl http://foo.com from the docker host or any other computer connected to the internet works correctly and requests are routed to service A. Why will this simple request not work from within the containers?

is this from inside the container b? not clear from the text.

i thing timed out means the host did not respond, not that the host was not found…
you could test by changing the foo.com to foo1.com (which should be a non existent host name),
and should return host not found

when I do that I get

curl fribble.sam.com
curl: (6) Could not resolve host: fribble.sam.com

Thanks for the reply. I clarified the ping test, yes it is from inside B.

Doing curl for an invalid host does result in a Could not resolve host response. So DNS resolution does seem to be working.

I used tcpdump to look at the traffic and could see that DNS did seem to be resolving. Here is a capture of that. I’ve replaced the IP of foo.com with foo.com so the actual IP is kept private.
172.28.0.3 is the IP address of container B
172.28.0.6 is the IP address of container A
172.28.0.1 is the docker gateway

I’m not experienced in tcpdump to make too many conclusions from this logging, but maybe it is helpful for someone else.

01:44:24.677632 IP 172.28.0.3.51806 > 67.207.67.2.53: 25680+ AAAA? staging.mochilafulfillment.com. (48)
01:44:24.677733 IP 67.207.67.2.53 > 172.28.0.3.46475: 56602 1/0/0 A 159.89.41.110 (64)
01:44:24.709364 IP 67.207.67.2.53 > 172.28.0.3.51806: 25680 0/1/0 (107)
01:44:24.737688 IP 172.28.0.3.45636 > foo.com.443: Flags [S], seq 687390928, win 29200, options [mss 1460,sackOK,TS val 130634531 ecr 0,nop,wscale 7], length 0
01:44:25.735689 IP 172.28.0.3.45636 > foo.com.443: Flags [S], seq 687390928, win 29200, options [mss 1460,sackOK,TS val 130634781 ecr 0,nop,wscale 7], length 0
01:44:27.739669 IP 172.28.0.3.45636 > foo.com.443: Flags [S], seq 687390928, win 29200, options [mss 1460,sackOK,TS val 130635282 ecr 0,nop,wscale 7], length 0
01:44:29.683621 ARP, Request who-has 172.28.0.3 tell 172.28.0.1, length 28
01:44:29.683657 ARP, Request who-has 172.28.0.1 tell 172.28.0.3, length 28
01:44:29.683669 ARP, Reply 172.28.0.1 is-at 02:42:72:a6:6d:cf, length 28
01:44:29.683680 ARP, Reply 172.28.0.3 is-at 02:42:ac:1c:00:03, length 28

I’d like to understand this issue and resolve it, but in the mean time I did come up with a workaround by specifying foo.com as an alias for service A in the docker compose file. Seems to work so far, I’m not sure if there are any potential issues in doing this down the road. Its definitely a band-aid I’d like to not keep.

Did this ever get resolved? I seem to have hit an identical issue: I’ve tried moving a docker based server (happens to be sonarqube) onto a machine I used for building. The build containers can’t seem to reach sonarqube. The sonarqube service is actually proxied behind an nginx container: they are setup using docker-compose and using a network declared as

networks:
sonarnet:
driver: bridge

between then. The other containers are all “standalone” - I start and stop them manually as required. [They are Jenkins build slaves].

OK if I use --net=host it fixes my specific issue. Have to say I think the default is possibly wrong, but there we are. Not yet tried a scenario where I have a mini-swarm (say two or three containers run via docker-compose.yml) which use a service which happen to be implemented by an independent container on the same VM. I would like to know how to solve that…

I’m with the same issue here. Can’t reach thorough internal DNS the container in the same machine on a specific port

I know this is an old thread, but it also seems to be the only place on the Internet I found that discusses the same problem I had recently. Above, @johnfo mentions --net=host as a solution, and while it was not clear to me how to apply that in my case (a docker newbie here), their message helped me to figure it out. Posting here my case and my solution, hoping it helps someone coming from Google with the same or similar issue.

TL;DR:

Using network_mode: host in compose.yaml fixed the issue.

The details:

I have 3 docker containers on a single host machine. One container is a Caddy server that works as a reverse proxy for two websites (alpha.com and beta.com). The other two containers serve those websites on ports 3001 and 3002. I have a compose file to manage all three containers.

In this common setup, I have one special requirement: Alpha needs to connect to Beta using its external public URL https://beta.com/*. Since that piece of code inside Alpha works both on the server-side (generating HTML pages), as well as on the client side (web app working in the browser), I can’t just replace the address with the internal docker hostname http://beta:3002/. I need one “universal” URL that works both on the server and in the visitor’s browser.

# compose.yaml

services:
  caddy:
    container_name: caddy
    image: caddy:latest
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config

  alpha:
    container_name: alpha
    build:
      context: ./alpha/
    environment:
      PORT: 3001
    restart: unless-stopped

  beta:
    container_name: beta
    build:
      context: ./beta/
    environment:
      PORT: 3002
    restart: unless-stopped

volumes:
  caddy_data:
  caddy_config:

# Caddyfile

alpha.com {
    reverse_proxy alpha:3001
}
beta.com {
    reverse_proxy beta:3002
}

With the config above, Alpha couldn’t connect the https://beta.com/*. What helped me to fix it is to move docker network to the host level, using network_mode: host in each of compose’s services. With this, I also removed no longer required setting ports in the caddy service, and replaced hostnames in Caddyfile with localhost. So now my configs look like this:

# compose.yaml

services:
  caddy:
    container_name: caddy
    image: caddy:latest
    restart: unless-stopped
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config
    network_mode: host

  alpha:
    container_name: alpha
    build:
      context: ./alpha/
    environment:
      PORT: 3001
    restart: unless-stopped
    network_mode: host

  beta:
    container_name: beta
    build:
      context: ./beta/
    environment:
      PORT: 3002
    restart: unless-stopped
    network_mode: host

volumes:
  caddy_data:
  caddy_config:

# Caddyfile

alpha.com {
    reverse_proxy localhost:3001
}
beta.com {
    reverse_proxy localhost:3002
}

And now Alpha can successfully connect to https://beta.com/* from inside its container.

Hoping this helps!

It seems to be very similar to another topic we are currently discussing

Edited:

On the other hand your situation could have been solved by network aliases. Example:

services:
  test:
    depends_on:
      - httpd
    image: nicolaka/netshoot
    command:
      - nslookup
      - yourdomain.com

  httpd:
    image: httpd:2.4
    networks:
      default:
        aliases:
          - yourdomain.com

Run the test:

docker compose up

And the domain will be resolved to the ip address of the container.

No, unfortunately, it couldn’t (I actually had tried aliases, without success).

The thing is, DNS resolution was not an issue. In my old setup, I could successfully run ping beta.com from inside Alpha container (which naturally resolves the hostname to the external IP address), and when I did curl https://beta.com/ it showed me that it connected to the right IP address (meaning DNS resolution part worked fine), but then it was never getting a reply and the connection just hanged.

Also, even if DNS resolution was a problem, the alias would only solve one part of the issue: hostname. To make the URL working, we also need to match port (443 vs 3002) and TLS termination (URL starts with “https://”). The port part is probably easy to fix, but directly connecting to Beta container with HTTPS is not really possible (the whole point of Caddy in my setup is to provide TLS termination for Alpha and Beta).

I didn’t say DNS resolution was the problem, but instead of connecting to the host (and back to another container) you could connect to that other container directly. That “other container” could be caddy as well if Caddy and the client container have a common network and on that common network Caddy has the alias.

I actually understood that DNS resolution worked. That’s why I linked the other topic as related, because the issue is using a host port from a container while that host port redirects back to another container. It works on my machine, but firewalls can change it. I’m not sure if I remember correctly, but I think I had this problem years ago when I used CentOS and firewalld was configured to block these requests.

Aliases may not be a solution for you, but if you share how you tried, it’s possible we recognize what was wrong if there was anything wrong.

Running services on the host network is not the best idea (I’m sure you know that), network isolation is used for a reason :slight_smile:

I’m glad you were able to solve your problem by using host network and thank you for sharing your solution, but as a next step, I recommend starting to find a better way.

@rimelek, thanks for your reply.

I don’t have that config anymore, but it was pretty much the same as in your snippet above (the only difference is that I named the network “my_net” instead of “default”, and set all the containers (i.e. services in compose) to that network.

Hmm… I don’t quite understand how to do that (as I said, I have very little experience with Docker). Does it mean that caddy container would have the “beta.com” alias, instead of beta container? For the simple case of three containers (caddy + two app servers), could you please provide an example of compose.yaml that implements this magic?

I get that network isolation makes sense in general, but I don’t see any disadvantage for “host-level” network in my case, where I only use that host for those three docker containers (there’s just nothing else on that machine, but OS, docker engine, and firewall). I’d appreciate if you could point out possible issues with my setup. What should I take into account?

P.S. Regarding the linked topic. I read it but it doesn’t really look relevant for the issue I had (perhaps due to my limited knowledge on the topic). AFAICS, main topic there is DNS, which worked fine for me, and the other issue is Internet access, which I also didn’t have problems with: all the URLs worked fine from inside the container, the only URL that didn’t work was Beta, run by the neighbor container.

Aren’t all traffic goes through the proxy server anyway? The point is that you add the alias to the container which has to have that hostname. If that is a proxy, that you have to add it to that container. If the traffic doesn’t go through the proxy, then the alias has to be addedd to the container that handles the request.

It’s okay in your case probably, but if anyone can hack hack the proxy or any container that is listening on a public port and also on the host network will have access to your local network. Even to your localhost on which you could have only locally listening services like a local admin interface or something you didn’t even know about. For example I usually don’t let my databases listen on public ports, only on localhost and I can access it from my machine remotely using SSH tunnel securely. And sometimes containers listen on multiple ports internally and you don’t forward all ports from the host, only what you need. Only one container (actually one process) can listen on one port, so you couldn’t run multiple instance of one container on host network.

The DNS is just the service that had to be accessible. The problem was the network the same way as in your case. I reproduce the issue with a webserver since I don’t want to run DNS server on my machine.

Don’t let yourself to be confused by the topic title :slight_smile: (and some of the comments) The “internet access” issue was just the result of the DNS issue caused by the locally running DNS server.

@rimelek, thank you for the detailed answer.

I guess next time I can play with the server, I’m going to test two ideas: the alias on the proxy container as described above, and then checking if the firewall somehow involved. If I learn anything useful there, I’ll make sure to share here.