Cannot wget/curl/git/apt-get to internet from within docker container

Reproduction Steps:

  1. Execute “sudo apt-get update” within container
user@11611da3d377:/projects$ sudo apt-get update
0% [Waiting for headers] [Waiting for headers]

Expected behavior:
Able to do sudo apt-get update and install software.

Observed behavior:
I can observe that the apt-get update stops at [waiting for headers].

I troubleshoot the issue with the following steps. Confirm that able to ping but when doing wget/curl it seems to be able to connect but not getting response.

ubuntu@dev14-04:~$ docker run -itd --name=container1 busybox
545bf50a46d798602e21d04b177ab9500c48af5c8b37b2773204067631507211
ubuntu@dev14-04:~$ docker attach container1
/ # ping -w3 www.google.com
PING www.google.com (202.75.147.25): 56 data bytes
64 bytes from 202.75.147.25: seq=0 ttl=53 time=24.949 ms
64 bytes from 202.75.147.25: seq=1 ttl=53 time=22.862 ms
64 bytes from 202.75.147.25: seq=2 ttl=53 time=22.751 ms

--- www.google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 22.751/23.520/24.949 ms
/ # wget http://www.eclipse.org/che/
Connecting to www.eclipse.org (198.41.30.198:80)

In another container with curl

user@11611da3d377:/projects$ curl -O https://github.com/lhwong/datasciencecoursera/blob/master/HelloWorld.md
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0

I don’t have the same issue when running on Windows.

OS and version: Ubuntu 16.04
Docker version: 1.12.0

Additional information:

  • Problem started happening recently, didn’t happen in an older version of Che: No
  • Problem can be reliably reproduced, doesn’t happen randomly: Yes

The problem persists when firewall is disabled.

In container:

/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:03
          inet addr:172.17.0.3  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe11:3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:63 errors:0 dropped:0 overruns:0 frame:0
          TX packets:67 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:8423 (8.2 KiB)  TX bytes:5336 (5.2 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

The network bridge seems OK:

ubuntu@dev14-04:~$ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "6e86e16d0e75de89442fc4dc672928c35cdd1e66d886f8eacb61beb0361f6006"                                                                                                             ,
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "11611da3d3772e15c528d58170bc515f605b33393e4e506f304841fdc1893d08":                                                                                                              {
                "Name": "ubuntu",
                "EndpointID": "7cf5e799e3a2bade5c8b6632cf30f3c7c499d5a60abf6c946                                                                                                             dd2601bdeec3f7b",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            },
            "129d1d5af39a1f5d0a823f91d89ae41b48a2ca240b2190616d36dd75c217cd27":                                                                                                              {
                "Name": "container1",
                "EndpointID": "560a488340a965f158809feccbd7343ebe3a80ddcd2e06419                                                                                                             23171322407b1c4",
                "MacAddress": "02:42:ac:11:00:03",
                "IPv4Address": "172.17.0.3/16",
                "IPv6Address": ""
            },
            "545bf50a46d798602e21d04b177ab9500c48af5c8b37b2773204067631507211":                                                                                                              {
                "Name": "container2",
                "EndpointID": "146d32a022c46daebc5cbf286ae29dac47ca615fcc4a11a2b                                                                                                             b405a0b9317ad70",
                "MacAddress": "02:42:ac:11:00:04",
                "IPv4Address": "172.17.0.4/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

It seems to be issue of the version of the ubuntu container?

ubuntu@ws:~$ docker run -ti ubuntu /bin/bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
43db9dbdcb30: Already exists
2dc64e8f8d4f: Already exists
670a583e1b50: Already exists
183b0bfcd10e: Already exists
Digest: sha256:c6674c44c6439673bf56536c1a15916639c47ea04c3d6296c5df938add67b54b
Status: Downloaded newer image for ubuntu:latest
root@9f681f26f5a9:/# apt-get update
0% [Waiting for headers]

Nothing happens after this

ubuntu@ws:~$ docker run -ti ubuntu:14.04 /bin/bash
root@d9dd5b1c102b:/# sudo apt-get update
Ign http://archive.ubuntu.com trusty InRelease
12% [Waiting for headers]

Nothing happens after that.

ubuntu@ws:~$ docker run -ti 32bit/ubuntu:14.04 /bin/bash
...
root@35a6986323dd:/# sudo apt-get update
Ign http://extras.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty InRelease
Get:1 http://extras.ubuntu.com trusty Release.gpg [72 B]
Hit http://extras.ubuntu.com trusty Release
Ign http://extras.ubuntu.com trusty Release
Ign http://extras.ubuntu.com trusty/main i386 Packages/DiffIndex
Hit http://extras.ubuntu.com trusty/main i386 Packages
Ign http://extras.ubuntu.com trusty/main Translation-en
Ign http://security.ubuntu.com trusty-security InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
Get:2 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Ign http://archive.ubuntu.com trusty-backports InRelease
Ign http://security.ubuntu.com trusty-security Release
Get:3 http://archive.ubuntu.com trusty Release.gpg [933 B]
Ign http://security.ubuntu.com trusty-security/main i386 Packages/DiffIndex
Get:4 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]
Ign http://security.ubuntu.com trusty-security/restricted i386 Packages/DiffIndex
Get:5 http://archive.ubuntu.com trusty-backports Release.gpg [933 B]
Ign http://security.ubuntu.com trusty-security/universe i386 Packages/DiffIndex
Hit http://archive.ubuntu.com trusty Release
Ign http://security.ubuntu.com trusty-security/multiverse i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-updates Release
Ign http://archive.ubuntu.com trusty-backports Release
Hit http://archive.ubuntu.com trusty/main i386 Packages
Hit http://archive.ubuntu.com trusty/restricted i386 Packages
Hit http://archive.ubuntu.com trusty/universe i386 Packages
Hit http://archive.ubuntu.com trusty/multiverse i386 Packages
Hit http://archive.ubuntu.com trusty/main Translation-en
Hit http://archive.ubuntu.com trusty/multiverse Translation-en
Hit http://archive.ubuntu.com trusty/restricted Translation-en
Hit http://archive.ubuntu.com trusty/universe Translation-en
Ign http://archive.ubuntu.com trusty-updates/main i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-updates/restricted i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-updates/universe i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-updates/multiverse i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-backports/main i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-backports/restricted i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-backports/universe i386 Packages/DiffIndex
Ign http://archive.ubuntu.com trusty-backports/multiverse i386 Packages/DiffIndex
Get:6 http://archive.ubuntu.com trusty-backports/restricted Translation-en [28 B]
Get:7 http://archive.ubuntu.com trusty-backports/restricted i386 Packages [28 B]
Ign http://security.ubuntu.com trusty-security/main Translation-en
Ign http://security.ubuntu.com trusty-security/multiverse Translation-en
Ign http://security.ubuntu.com trusty-security/restricted Translation-en
Ign http://security.ubuntu.com trusty-security/universe Translation-en
Err http://security.ubuntu.com trusty-security/main i386 Packages
  404  Not Found [IP: 91.189.91.26 80]
Err http://security.ubuntu.com trusty-security/restricted i386 Packages
  404  Not Found [IP: 91.189.91.26 80]
Err http://security.ubuntu.com trusty-security/universe i386 Packages
  404  Not Found [IP: 91.189.91.26 80]
Err http://security.ubuntu.com trusty-security/multiverse i386 Packages
  404  Not Found [IP: 91.189.91.26 80]
Ign http://archive.ubuntu.com trusty-updates/main Translation-en
Ign http://archive.ubuntu.com trusty-updates/multiverse Translation-en
Ign http://archive.ubuntu.com trusty-updates/restricted Translation-en
Ign http://archive.ubuntu.com trusty-updates/universe Translation-en
100% [Waiting for headers]

Takes long time to update. More than 2 hours still has not complete on a fast line.

I have the exact same problem…
and I didn’t find a solution yet…

The problem is in the bridge MTU…
The only solution I managed to find is this:
sudo iptables -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

setting the --mtu in the daemon options does not work for me on Docker 1.12, ubuntu 16.04.

We encountered the same problem across all our hosts.
The problem can to be reproducible constantly using any images.

Basically curl/wget requests does not complete
curl www. google. com -vvvv

  • About to connect() to www. google. com port 80 (#0)
  • Trying 216.58.219.196…
  • Connected to www.google.com (216.58.219.196) port 80 (#0)

GET / HTTP/1.1
User-Agent: curl/7.29.0
Host: www.google.com
Accept: /

We found that mtu on the host and containers are same

ifconfig | grep mtu
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
vetha3d31c4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

We also took a tracedump.
it actually returns a valid http response back
but for whatever reason it can’t finish the connection.

It still happens sometimes to me. Restarting the Docker service is enough to get it work for me:

sudo systemctl restart docker

Problem

I ran into similar problem couple of days ago when i started to deploy my containers into new VM environment. I have docker swarm running, user defined docker overlay network and containers attached to that network. And by some strange reason containers failed to communicate with SOME of the servers in outside world. Most of the curl/wget requests worked fine, but with some specific servers they failed (https://atlassian.com for example). They did reach connected status and after that just hung there. Curl on host machine worked just fine. Same containers and swarm setup on my local machine and on AWS worked just fine. But in this new VM it did not.

wget on host machine

$ wget https://www.atlassian.com/software
> --2019-10-25 10:38:18--  https://www.atlassian.com/software
> Resolving www.atlassian.com (www.atlassian.com)... 18.184.99.151, 18.184.99.149, 18.184.99.150
> Connecting to www.atlassian.com (www.atlassian.com)|18.184.99.151|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Syntax error in Set-Cookie: wac_user_detected=0;Domain=www.atlassian.com;Path=/;Max-Age=-1 at position 62.
> Length: 96011 (94K) [text/html]
> Saving to: ‘software’
> 2019-10-25 10:38:18 (2.02 MB/s) - ‘software’ saved [96011/96011]

wget on docker container in swarm

$ docker exec 66ad061529ac wget https://www.atlassian.com/software
> --2019-10-25 10:39:13--  https://www.atlassian.com/software
> Resolving www.atlassian.com (www.atlassian.com)... 18.184.99.150, 18.184.99.149, 18.184.99.151
> Connecting to www.atlassian.com (www.atlassian.com)|18.184.99.150|:443... connected.

Solution

It took me three days of trials to finally get it to work and it all came down to MTU. On my new VM it was 1450, while anywhere else it has always been 1500, and so is docker default. So i needed to tell docker to use the same value: 1450. Here are the steps i had to go through.

First check if your mtu even needs changing

# check VM-s mtu
$ ifconfig ens3 | grep mtu
> ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450

# check docker_gwbridge mtu
$ ifconfig docker_gwbridge | grep mtu
> docker_gwbridge: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

Second, change docker default mtu

$ sudo nano /etc/docker/daemon.json
# add definition "mtu": 1450

Third, as by some reason the docker_gwbridge didnt care about mtu change in daemon.json, i needed to recreate docker_gwbridge manually (inspired by https://docs.docker.com/network/overlay/)

$ docker swarm leave -f
$ sudo service docker stop
$ sudo ip link set docker_gwbridge down
$ sudo ip link del dev docker_gwbridge
$ sudo service docker start
$ docker network rm docker_gwbridge
$ docker network create \
  --opt com.docker.network.bridge.name=docker_gwbridge \
  --opt com.docker.network.bridge.enable_icc=false \
  --opt com.docker.network.bridge.enable_ip_masquerade=true \
  --opt com.docker.network.driver.mtu=1450 \
  docker_gwbridge

Fourth: recreate my own overlay network

$ docker network rm my-swarm-network
$ docker network create \
  --driver overlay \
  --attachable \
  --opt com.docker.network.driver.mtu=1450 \
  my-swarm-network

Now when i re-deploy my services, all curl/wget requests work fine. I hope it helps.

Still, i’m wondering

…why curl fails only on some endpoints. One, that i actually needed to communicate to, uses Apache and Atlassian server is globaledge-envoy, while others that i tested against and where curl succeeded, are Nginx servers…
And one other note i made, is that Apache server responded with Transfer-Encdong: chunked and Content-length unspecified. While Atlassian response had Syntax error in it “Syntax error in Set-Cookie: wac_user_detected=0;Domain=…” is it possible, that unspecified content length, or possibly invalid content length broken by syntax error and with mtu conflict there was no way to recover packages…?
Any ideas…?

1 Like