Docker Community Forums

Share and learn in the Docker community.

Can't pull any image (Client.Timeout exceeded while awaiting headers)

I’ve installed Docker on my ODROID N2 with Ubuntu 18.04 installed. It’s an arm64 device. I’m running something between 12 to 16 container with docker-compose on my system. My intention was to install another image and then it ended up with the error
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
and it seems to be a known issue regarding my Google researches. I could fix it for one day with the solution of @efranelas in the thread Docker pull results in "Request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)".

But after one day the pull was so slow that 8kb/minute were downloaded and often ended up in an unexspected EOF. I’ve used one random proxy of the list https://free-proxy-list.net/. I thought I could choose another one and try it again to increase the download speed. And since this point nothing works. No other proxy and also not the old one. Depending on which proxy IP I’ve chose the error is changing. For example forbidden or proxyconnect tcp: dial tcp: connect: connection refused or moved permanently. Normally I should not need any proxy because I’m in my home network directly connected to my router.

At the moment I have no docker proxy active while creating the following logs.

My resolv.conf:

nameserver 8.8.4.4
nameserver 8.8.8.8
nameserver 127.0.0.53
search fritz.box

The google nameserver I’ve added manually because that was also an solution attempt.

systemd-resolve --status output:

Global
         DNS Servers: 8.8.4.4
                      8.8.8.8
          DNS Domain: fritz.box
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 4 (docker0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 3 (br-5e7b27d1cc10)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no


Link 2 (eth0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.178.1
                      fd00::3681:c4ff:fe3d:7e33
          DNS Domain: ~.
                      fritz.box

There are many more links but I think you don’t need them.

iconfig docker0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:5c:eb:2f:e3  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

sudo service docker status

● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─http-proxy.conf
   Active: active (running) since Tue 2020-03-24 00:16:33 UTC; 39min ago
     Docs: https://docs.docker.com
 Main PID: 2680 (dockerd)
    Tasks: 175
   CGroup: /system.slice/docker.service
           ├─2680 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
           ├─3629 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9100 -container-ip 172.21.0.2 -container-port 9100
           ├─3657 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9090 -container-ip 172.21.0.3 -container-port 9090
           ├─3675 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8181 -container-ip 172.26.0.2 -container-port 8181
           ├─3730 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9001 -container-ip 172.18.0.3 -container-port 9000
           ├─3763 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 3000 -container-ip 172.21.0.4 -container-port 3000
           ├─3863 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -container-ip 172.21.0.5 -container-port 8080
           ├─3950 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9093 -container-ip 172.21.0.6 -container-port 9093
           ├─4162 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 30033 -container-ip 172.25.0.2 -container-port 30033
           ├─4248 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 10011 -container-ip 172.25.0.2 -container-port 10011
           ├─4358 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8000 -container-ip 172.18.0.3 -container-port 8000
           ├─4407 /usr/bin/docker-proxy -proto udp -host-ip 0.0.0.0 -host-port 9987 -container-ip 172.25.0.2 -container-port 9987
           ├─4826 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 3129 -container-ip 172.22.0.2 -container-port 3129
           ├─5071 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8081 -container-ip 172.18.0.5 -container-port 80
           ├─5103 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 4443 -container-ip 172.18.0.6 -container-port 4443
           ├─5163 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.18.0.6 -container-port 443
           └─5208 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.18.0.6 -container-port 80

Mar 24 00:21:06 odroid dockerd[2680]: time="2020-03-24T00:21:06.695573161Z" level=error msg="Handler for POST /v1.35/images/create returned error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Time
Mar 24 00:21:21 odroid dockerd[2680]: time="2020-03-24T00:21:21.709825278Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiti
Mar 24 00:21:21 odroid dockerd[2680]: time="2020-03-24T00:21:21.709929155Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exce
Mar 24 00:21:21 odroid dockerd[2680]: time="2020-03-24T00:21:21.710020073Z" level=error msg="Handler for POST /v1.35/images/create returned error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Time
Mar 24 00:26:21 odroid dockerd[2680]: time="2020-03-24T00:26:21.103187320Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Mar 24 00:26:21 odroid dockerd[2680]: time="2020-03-24T00:26:21.103341530Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Mar 24 00:26:21 odroid dockerd[2680]: time="2020-03-24T00:26:21.103447865Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Mar 24 00:33:32 odroid dockerd[2680]: time="2020-03-24T00:33:32.418212085Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Mar 24 00:33:32 odroid dockerd[2680]: time="2020-03-24T00:33:32.418402587Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Mar 24 00:33:32 odroid dockerd[2680]: time="2020-03-24T00:33:32.418530880Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.out exceeded while awaiting headers)"

docker info:

Client:
 Debug Mode: false

Server:
 Containers: 16
  Running: 16
  Paused: 0
  Stopped: 0
 Images: 22
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.9.216-69
 Operating System: Ubuntu 18.04.4 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 6
 Total Memory: 3.625GiB
 Name: odroid
 ID: SI6X:XFPI:53TH:EXNC:7Z5H:YA24:E5PS:FRDL:NX3M:W4DL:4JJU:SDKS
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

cat /etc/*release:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Thank you for any help!