Temporary failure resolving deb.debian.org inside all docker containers

[yfranasiuk]

Hello, I have a problem with no internet access in Docker containers on Ubuntu Server 22.04.2 (freshly installed).

For example, when I create a container with “docker run -it debain:latest /bin/bash” and run the “apt update” command, I get the error “Temporary failure resolving ‘deb.debian.org’”.

However, when I run the container with the “–network common” flag after creating a network with “docker network create --driver bridge common”, everything works and there is internet access.

What could be the problem when the host has a working internet connection?

• OS Version/build: Ubuntu Server 22.04.2
• App version:

Client: Docker Engine - Community
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        a5ee5b1
 Built:             Thu Feb  9 19:47:01 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       bc3805a
  Built:            Thu Feb  9 19:47:01 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.18
  GitCommit:        2456e983eb9e37e47538f59ea18f2043c9a73640
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[rimelek (Ákos Takács)]

It looks like your default Docker bridge is broken or your LAN network has an IP address from the same IP range. It could also happen when you are running Docker in a WSL 2 distribution on Windows which uses a random, but similar ip range. The default Docker bridge has the following subnet: 172.17.0.0/16

I moved the topic to DockerEngine, as the shared version indicates that you are not using Docker Desktop.
Also, please, use code blocks when sharing code, terminal outputs, logs, everything with special characters thst otherwise would be interpreted by the forums markdown filter or as a BB code. I edited your post to make it more readable.

How to format your forum posts

The Docker forum is using Discourse, which supports several ways to format your posts: Markdown; see commonmark.org/help for an interactive tutorial BBCode A small selection of HTML Any combination of the above Blank lines for formatting and readability Quite often you’ll need an empty line above specific formatting to make it work. You also need to type blank lines to create proper paragraphs; just pressing Return once and then typing a new block of text right below a previous block only…

[yfranasiuk]

I have encountered an issue where my containers are not working with the standard bridge in Docker.

When I run the command:
docker run --network host -it debian /bin/bash
everything works fine, but when I try to run the container using the bridge network:
docker run --network bridge -it debian /bin/bash
there is no internet connection inside the container.

Here are the settings in iptables:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

Here are the settings in ip route:

default via 10.100.66.254 dev ens18 proto static 
10.100.66.0/24 dev ens18 proto kernel scope link src 10.100.66.100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown

Do you know how to fix this?

[rimelek (Ákos Takács)]

There are no allow rules in your iptables.

Can you check the logs?

sudo journalctl -e -u docker

For some people removing the the following file and restarting Docker helped:

/var/lib/docker/network/files/local-kv.db

This is where Docker stores network related information.

We still can’t rule out that there is another intersecting subnet in your local network although it is not your LAN, but it could be connected to through the gateway/router.

Please check the following commands to see if you can access a webserver running in the container connecting to the default bridge:

docker run -d --name net-test nginx
curl -L $(docker container inspect net-test --format '{{ .NetworkSettings.IPAddress }}')

or if you have wget and not curl

docker run -d --name net-test nginx
wget -O-  $(docker container inspect net-test --format '{{ .NetworkSettings.IPAddress }}')

[yfranasiuk]

I deleted /var/lib/docker/network/files/local-kv.db, but it had no effect.

The problem cannot be in the intersection of subnets, because I changed the ip address of the docker bridge network to another and the result is the same.

I have determined that there is internet inside the containers by pinging 8.8.8.8, however when I ping google.com I get an error, most likely a problem getting the domains ip addresses. This problem is only when I use the standard docker bridge network, when I use my own docker network, the dns in the container works correctly and google.com is pinged.

On the host machine I use 8.8.8.8 and 8.8.4.4 dns.

log.txt (339.0 KB)

[rimelek (Ákos Takács)]

I can’t check the whole log now, but you could check the resolv.conf in the containers (working and non-working)

cat /etc/resolv.conf

My new tutorial about the name resolution issue could also be relevant: https://learn-docker.it-sziget.hu/en/latest/pages/advanced/kernel-namespaces-network.html#working-with-docker-s-network-namespaces

[yfranasiuk]

on my host machine:

nameserver 127.0.0.53
options edns0 trust-ad
search inteldim

inside docker containers:

nameserver 8.8.8.8
nameserver 8.8.4.4
search inteldim

[rimelek (Ákos Takács)]

Is there any difference between the working and non-working containers? Is this resolv.conf the same in each container?

[yfranasiuk]

Yes it’s config in all containers.
In working (custom docker network) and non-working (bridge) containers, the dns in the configs are the same

nameserver 8.8.8.8
nameserver 8.8.4.4
search inteldim

[rimelek (Ákos Takács)]

Please try to following commands:

docker run --rm nicolaka/netshoot:v0.9 nslookup google.com
docker run --rm nicolaka/netshoot:v0.9 nslookup google.com 8.8.8.8
docker run --rm nicolaka/netshoot:v0.9 nslookup google.com 8.8.4.4

[aalmazanarbs]

I have the same problem, same Ubuntu version and Docker fresh install. I have the problem with default and custom bridge networks. Also the problem occurs with custom DNS server.

Output for your commands:

$ docker run --rm nicolaka/netshoot:v0.9 nslookup google.com
;; communications error to 192.168.168.3#53: host unreachable
;; communications error to 192.168.168.3#53: host unreachable
;; communications error to 192.168.168.3#53: host unreachable
;; no servers could be reached

$ docker run --rm nicolaka/netshoot:v0.9 nslookup google.com 8.8.8.8
;; communications error to 8.8.8.8#53: host unreachable
;; communications error to 8.8.8.8#53: host unreachable
;; communications error to 8.8.8.8#53: host unreachable
;; no servers could be reached

[rimelek (Ákos Takács)]

Since you showed that the containers contain the same resolv.conf and the nameservers are the nameservers of Google, the first result shouldn’t have happened.Do you recognize the IP address in the first output?

Update:

It is also strange that you mentioned you could ping 8.8.8.8, but when you are using nslookup, the host is unreachable. As if the result depended on the protocol. so ping can send icmp packages but communicating with the DNS server on port 53/UDP doesn’t work. If it is the case, there must be a firewall software on the host which is configured only for the default bridge.

[rimelek (Ákos Takács)]

@rimelek

Since you showed that the containers contain the same resolv.conf and the nameservers are the nameservers of Google, the first result shouldn’t have happened.Do you recognize the IP address in the first output?

@yfranasiuk I am sorry I thought you posted the nslookup commands I asked for, but I can see now it was @aalmazanarbs. Can you share the results too?

@aalmazanarbs

I have the same problem, same Ubuntu version and Docker fresh install. I have the problem with default and custom bridge networks. Also the problem occurs with custom DNS server.

Output for your commands:

Since you have unreachable host error, it doesn’t look like the same error. Since we now the original issue in this topic was about name resolution only, I am going to change the topic name accordingly. Can you create a new topic so we can continue the discusssion about “host unreachable” there?

[aalmazanarbs]

Hi @rimelek, it is the same error. As you have said, there is something in the host blocking all request (unreachable host was my custom DNS server).

I tried to check iptables and firewalld configuration but since it is hell I deleted all the iptables rules, the docker zone and reinstalled docker (previously purged) and got it working.

[rimelek (Ákos Takács)]

@aalmazanarbs

it is the same error

Same as what?

@yfranasiuk

I get the error “Temporary failure resolving deb.debian.org”.

@aalmazanarbs

;; no servers could be reached

@aalmazanarbs

(unreachable host was my custom DNS server).

Which one do you consider to be custom?

@aalmazanarbs

;; communications error to 8.8.8.8#53: host unreachable

@aalmazanarbs

;; communications error to 192.168.168.3#53: host unreachable

Even if you get Temporary name resolution failure that must be the consequence of the unreachable host error while @yfranasiuk found out that the network worked, but the name resolution didn’t. I talked about a similar issue i the tutorial I shared before in this topic. This is a direct link to the name resolution issue caused by network issue: LINK

and a link to the part of the video where I mention this issue: “Everything you must know about Docker networks and the network namespace” at 20:03

PS.: I forgot to rename the topic when I wrote I would, but I did it now.

[yfranasiuk]

$ docker run --rm nicolaka/netshoot:v0.9 nslookup google.com
;; communications error to 8.8.8.8#53: timed out
;; communications error to 8.8.8.8#53: timed out
;; communications error to 8.8.8.8#53: timed out
;; communications error to 8.8.8.4#53: timed out
;; no servers could be reached

$ docker run --rm nicolaka/netshoot:v0.9 nslookup google.com 8.8.8.8
;; communications error to 8.8.8.8#53: timed out
;; communications error to 8.8.8.8#53: timed out
;; communications error to 8.8.8.8#53: timed out
;; no servers could be reached

$ docker run --rm nicolaka/netshoot:v0.9 nslookup google.com 8.8.4.4
;; communications error to 8.8.4.4#53: timed out
;; communications error to 8.8.4.4#53: timed out
;; communications error to 8.8.4.4#53: timed out
;; no servers could be reached

[yfranasiuk]

You can rename the topic as this dns problem is not only in Ubuntu Server, I changed the guest OS to Fedora Server 37 and I get the same problem, most likely it’s not an OS problem.

[rimelek (Ákos Takács)]

@yfranasiuk

You can rename the topic … most likely it’s not an OS problem.

Okay, I renamed it again, but as the creator of this topic, you can rename it too.

Since you get the same result in on different Linux distributions, it still looks like it is either a bug in Docker 23.0.1 or there is something in your environment outside the Docker host. The problem is that I don’t know how something outside the Docker host could affect only the default bridge and only nslookup, but I don’t experience this issue in my test virtual machine so I don’t think it is a bug in Docker. Or if it is, it must depend on your environment.

You can try to change the ip address of the default bridge and see if it fixes anything

https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file.

/etc/docker/daemon.json

{
    "bip": "10.88.0.1/24"
}

systemctl restart docker

@aalmazanarbs I have noticed that indeed you both had no servers could be reached error message at the end despite the different error message before that. I still haven’t checked if you created a new topic, but did you also test pinging the ip addresses?

Since I have no other idea, you could also change the default ip addresses, not just the ip of the default bridge but the address pool as well. Since existing networks would not change, you will need to create a new network to have the new ip address.

{
    "bip": "10.88.0.1/16",
    "default-address-pools": [
      {
        "base": "10.89.0.1/16",
        "size": 24
      }
    ]
}

Of course, make sure the IP addresses can be used in your local network.

Unfortunately I have no other idea

[narayananiyer86]

I’m facing similar issues. I initially thought it could be issues on Ubuntu end. Here is my detailed question on stack to which I’m not finding a solution yet. Long story short. My docker container fails to fetch the repos.

Unable to build docker file for Jitsi web

[rimelek (Ákos Takács)]

Have you tried my suggestions in my previous comments?
In addition to those, please, try the build without Docker Compose only with the “docker build” command and with buildkit turned off

DOCKER_BUILDKIT=0 docker build . -t IMAGENAME

Note that this is just an example command. The important part that you should try is starting the build command with DOCKER_BUILDKIT=0. Depending on your docker version Docker will warn you that legacy build is deprecated, but I would like to test a theory.

Can you also share the output of the following commands?

docker info
docker version

Update

You have a COPY rootfs/ / line in your Dockerfile. What is in the rootfs folder? Do you overwrite something that is required for name resolution?

Note: It is usually recommended to share dockerfiles and compose files here too instead of referring to another forum. We don’t controll those forums and sites, topics can disappear and not everyone likes to follow external links.

Update 2

I found your new topc. Let’s continue it there then…

Service Webserver failed to build : Temporary failure/Failed to Fetch

I’m getting the stated error when I attempt to build my container. I cant seem to understand what’s wrong. I’m on Ubuntu 22.04 and using docker-compose to build my image. Here is the detailed trace → Building myapp_webserver Step 1/15 : ARG JITSI_REPO=jitsi Step 2/15 : ARG BASE_TAG=stable Step 3/15 : FROM ${JITSI_REPO}/base:${BASE_TAG} ---> 319b2e366502 Step 4/15 : LABEL org.opencontainers.image.title="Jitsi Meet" ---> Using cache ---> c5cc34716be6 Step 5/15 : LABEL org.opencontainers.image…