Network - Could not resolve host

General
1

Using this docker-compose file:

version: '3'

services:
  hello:
    image: nginxdemos/hello
    ports:
      - 7080:80
  tool:
    image: wbitt/network-multitool
    tty: true

networks:
  default:
    name: test-network

If I curl from the host, it works.

❯ curl -s -o /dev/null -v http://192.168.1.102:7080
* Expire in 0 ms for 6 (transfer 0x8088b0)
*   Trying 192.168.1.102...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x8088b0)
* Connected to 192.168.1.102 (192.168.1.102) port 7080 (#0)
> GET / HTTP/1.1
> Host: 192.168.1.102:7080
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.23.1
< Date: Sun, 10 May 2071 00:06:00 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
< Expires: Sun, 10 May 2071 00:05:59 GMT
< Cache-Control: no-cache
<
{ [6 bytes data]
* Connection #0 to host 192.168.1.102 left intact

If I try to contact another container from within the network, it fails.

❯ docker exec -it $(gdid tool) curl -s -o /dev/null -v http://hello
* Could not resolve host: hello
* Closing connection 0

Is this intended behaviour? I thought networks within the same network (and using docker-compose) are meant to be able to talk by their service name?

2

Assumed gdid tool returns the id of the tool container from this compose project, that container should be able to resolve the hello container by its service name and access the application that runs inside it without any issues. Thus said, no it is not intended behavior → something is wrong.

3

It does. It’s a simple bash function to get the docker id with grep/awk.

function gdid() {
        docker ps | grep $1 | awk '{print $1}'
}

Where do you suggest I start with debugging this issue?
The IP address of the network seems to resolve.

❯ docker inspect $(gdid hello) | jq '.[].NetworkSettings.Networks."test-network".IPAddress'
"172.22.0.3"

Checking the docker daemon debug log:

DEBU[2022-09-11T19:39:49.213098919+01:00] Name To resolve: hello.
DEBU[2022-09-11T19:39:49.213352418+01:00] [resolver] lookup name hello. present without IPv6 address
DEBU[2022-09-11T19:39:49.213176604+01:00] Name To resolve: hello.
DEBU[2022-09-11T19:39:49.214040451+01:00] [resolver] lookup for hello.: IP [172.22.0.3]
4

You can achieve this with a single command:docker ps --filter name="${1}" -q

I doubt that the dns-based service discover for user defined networks does not work. After all the default network (even though you gave it a different external name) is a user defined network.

@rimelek: do you have an idea how to debug this issue? I don’t realy have one…

5

Yeah I’m kind of stumped. I checked the resolv.conf and it has the following, which I don’t think is an issue.

nameserver 127.0.0.11
options ndots:0
6

Yep, that’s the resolver for the built-in dns server for service discovery.

7

Only thing I can think of, but I am unsure as I have disabled it and tried, is that I do run a PiHole DNS server.

8

I would be surprised. The built-in dns server will use whatever is declared in the hosts /etc/resolv.conf as upstream resolver. The request should have been handled by the built-in dns itself without even hitting any of the upstream resolvers.

Can you try the command curl -s -o /dev/null -v http://hello.test-network in your tool container?

9

I’ve been playing around and when I try to ping, I receive “Try again” several times, and eventually, it’ll go through.

[...]
bash-5.1# ping hello
ping: hello: Try again
bash-5.1# ping hello
PING hello (172.22.0.3) 56(84) bytes of data.
64 bytes from network-test-hello-1.test-network (172.22.0.3): icmp_seq=1 ttl=64 time=664810 ms
6

This is what I get in the debug log for “Try again”:

DEBU[2022-09-11T20:22:23.192330322+01:00] Name To resolve: hello.
DEBU[2022-09-11T20:22:23.193173883+01:00] [resolver] lookup name hello. present without IPv6 address
DEBU[2022-09-11T20:22:23.193962296+01:00] [resolver] lookup for hello.: IP [172.22.0.3]

This is what I get in the log when it is successful:

DEBU[2022-09-11T20:23:13.355696829+01:00] IP To resolve 3.0.22.172
DEBU[2022-09-11T20:23:13.355785828+01:00] [resolver] lookup for IP 3.0.22.172: name network-test-hello-1.test-network

What is that IP?

bash-5.1# curl -s -o /dev/null -v http://hello.test-network
* Could not resolve host: hello.test-network
* Closing connection 0
10

I would try it with an unmodified default network and I would check the LAN network. Colliding IP addresses are always jolly joker for me, if I don’t have any other idea :slight_smile:

On the other hand

If it works randomly, then the other Jolly Joker is the MTU, but none of these jolly jokers should matter. in this case I think. Still worth running ip addr to see the IP addresses and MTUs.

@nathun How did you create the “test-network” and where do you run Docker? Physical Linux machine, virtual machine, WSL2 on Windows? Also how did you install Docker?

I tried the compose file and the name resolution worked perfectly with a simple test network (docker network create test-network) in an Ubuntu virtual machine created by multipass.

Update:

I missed that post. So does it mean you run Docker on Rasberry py?

11

Yeah, I do. Is that’s what maybe the issue?

12

That should have worked as well. A compose service is registered by its {service name} and {service name}.{network name}. Are you sure the hello container is running stable and does not restart all the time? This would at least explain why it works occasionally. Apart from that, @rimelek brought up some very good points regarding a possible ip collision. Since you are running it on a RPi, your os, installation method and docker version indeed are valuable. You only get “the real deal” if you use a debian/ubuntu or centos OS and install Docker from Docker’s repositories - everything else may or may not differ in aspects of the functionality.

The fqdn network-test-hello-1.test-network looks strange. It should be hello.test-network
By any chance is this a swarm stack deployment using an overlay network?

13

I am going to try and install Ubuntu on the Raspberry Pi and I will get back to you.

Docker version is Docker version 20.10.12, build e91ed57

❯ cat /etc/*release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
14

Before doing that, I would suggest removing the docker installation you have and install it according Install Docker Engine on Debian | Docker Documentation (see update below). Based on the content of the os-release file, you already use a Debian based system.

Update: the repositories do not support Raspbian. You have to install docker using the convenience script

15

Exactly the same issue. I’ll give Ubuntu a shot and see what happens.

16

My bad. what we see is {container name}.{network name}. {container name} consists of {project name}-{service name}-{replica number}. So forget my question regarding it being a swarm stack deployment.

Btw. you compose file works like a charm on Ubuntu 20.04 x86_64:

me@myhost:~$ compose=$(cat <<EOF
version: '3'

services:
  hello:
    image: nginxdemos/hello
    ports:
      - 7080:80
  tool:
    image: wbitt/network-multitool
    tty: true

networks:
  default:
    name: test-network

EOF
)

me@myhost:~$ echo "${compose}" | docker compose --file - --project-name test up -d
[+] Running 2/0
 ⠿ Container test-tool-1   Running                                                                                                         0.0s
 ⠿ Container test-hello-1  Running                                                                                                            0.0s

me@myhost:~$ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED         STATUS                 PORTS                                              NAMES
02a789f52619   nginxdemos/hello               "/docker-entrypoint.…"   4 minutes ago   Up 4 minutes           0.0.0.0:7080->80/tcp                               test-hello-1
3fefa4b3e67a   wbitt/network-multitool        "/bin/sh /docker/ent…"   4 minutes ago   Up 4 minutes           80/tcp, 443/tcp, 1180/tcp, 11443/tcp               test-tool-1

me@myhost:~$ docker exec -ti $(docker ps --filter name=tool-1 -q) bash

bash-5.1# ping -c1 hello
PING hello (172.28.0.3) 56(84) bytes of data.
64 bytes from test-hello-1.test-network (172.28.0.3): icmp_seq=1 ttl=64 time=0.042 ms

--- hello ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.042/0.042/0.042/0.000 ms

bash-5.1# ping -c1 hello.test-network
PING hello.test-network (172.28.0.3) 56(84) bytes of data.
64 bytes from test-hello-1.test-network (172.28.0.3): icmp_seq=1 ttl=64 time=0.026 ms

--- hello.test-network ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.026/0.026/0.026/0.000 ms
bash-5.1#
exit

me@myhost:~$ echo "${compose}" | docker compose --file - --project-name test down
+] Running 3/3
 ⠿ Container test-hello-1  Removed                                                                                                            0.3s
 ⠿ Container test-tool-1   Removed                                                                                                            0.3s
 ⠿ Network test-network    Removed
17

I couldn’t find my SD card reader so had to buy a new one. If I have any further issues I will let you know, though I don’t think it will be as complex and as weird as this on Ubuntu, haha.

Thank you for your help, have a great day :slight_smile: