Docker Community Forums

Share and learn in the Docker community.

Windows 1709 containers on overlay network can't access internet, but can access each other

swarm

(Gh2k) #1

I’m using a mixed-platform (linux and Windows) swarm mode swarm. I have a VM for Windows Server 1709 and another for Linux.

I’ve created a docker stack with a few Linux and Windows containers defined, all attached to an overlay network.

Ingress routing works - I can publish ports on containers and access them on the external IP of either VM.

Inter-container networking works. I can access containers (and services running on them) from other containers by their internal names, across hosts. (e.g. access Linux containers from Windows and vice versa.)

DNS works. I can ping www.google.com and get an IP address back.

Internet access from Linux containers works. I can ping www.google.com from Linux containers and get ping responses.

Internet access from Windows containers does not work. When I ping www.google.com from a Windows container I get ‘request timed out’, even though I can look up the IP.

Internet access does work from standalone Windows containers.

Expected behavior

Able to access the internet from Windows 1709 containers running in a swarm and attached to an overlay network.

Actual behavior

No internet access from Windows 1709 containers running in swarm mode and attached to an overlay network.

Information

Docker version:

-> docker version
Client:
 Version:	18.02.0-ce
 API version:	1.34 (downgraded from 1.36)
 Go version:	go1.9.2
 Git commit:	fc4de44
 Built:	Mon Feb 12 08:56:54 2018
 OS/Arch:	linux/amd64
 Experimental:	false
 Orchestrator:	swarm

Server:
 Engine:
  Version:	17.10.0-ee-preview-3
  API version:	1.34 (minimum version 1.24)
  Go version:	go1.8.4
  Git commit:	b8571fd
  Built:	Fri Oct  6 18:01:48 2017
  OS/Arch:	windows/amd64
  Experimental:	true
  • the output of:
    • DockerDebugInfo.ps1 using Powershell on Windows
      debuginfo.txt (189.2 KB)

docker network ls on the Windows host:

-> docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
jt0sood49kbh        ingress             overlay             swarm
e70385f46f64        nat                 nat                 local
2ca023c080a7        none                null                local
dji43hv3bxib        test_overlay        overlay             swarm

Inspect the test_overlay network:

-> docker network inspect test_overlay
[
    {
        "Name": "test_overlay",
        "Id": "dji43hv3bxib92ldqjxrqiklc",
        "Created": "2018-02-19T07:06:43.0971711-08:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "86bd96a670c8e4ca471e8d858a8f6a0f0eacccc1a06ffa40c7be170f40091a65": {
                "Name": "windowsping.1.tszyfp0wk6w452erf2e445i5y",
                "EndpointID": "a3bffa491338b805ec756c7d20015ef403a8b0da6198975298a8bf7ed5735ea4",
                "MacAddress": "00:15:5d:e5:7c:de",
                "IPv4Address": "10.0.1.12/24",
                "IPv6Address": ""
            },
            "test_overlay-sbox": {
                "Name": "test_overlay-endpoint",
                "EndpointID": "c282173916e6bc6060a66f9b10e590e7ad2155d8f52a70c213ed626cd778db54",
                "MacAddress": "00:15:5d:e5:77:25",
                "IPv4Address": "10.0.1.3/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4098",
            "com.docker.network.windowsshim.hnsid": "5543adfa-7091-434a-b3a7-3bacaa9ce81f"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "41bd519bb433",
                "IP": "192.168.84.130"
            },
            {
                "Name": "vagrant-1709-73b05f3333e5",
                "IP": "192.168.84.128"
            }
        ]
    }
]

Inspect the ‘nat’ network:

-> docker network inspect nat
[
    {
        "Name": "nat",
        "Id": "e70385f46f6416ee615bdc366735ea99be54d977704172719f0abf9268e2f141",
        "Created": "2018-02-19T06:22:43.3933081-08:00",
        "Scope": "local",
        "Driver": "nat",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "windows",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.25.128.0/20",
                    "Gateway": "172.25.128.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "ed30df8ec3171ac954a3e67545c129fe678fd7d829ce621bcf64a4ae5cfa6229": {
                "Name": "windowsping_nonet.1.yo86r9c6vk5yh0iueqyckvl1j",
                "EndpointID": "6e422453c4ee008208064571381232daa16023c7d83815632a364049af126397",
                "MacAddress": "00:15:5d:22:97:36",
                "IPv4Address": "172.25.140.73/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.windowsshim.hnsid": "a2cae4d3-0c70-4946-be12-f388cac4e922",
            "com.docker.network.windowsshim.networkname": "nat"
        },
        "Labels": {}
    }
]

Steps to reproduce the behavior

  1. On a linux box, run docker swarm init
  2. On a Windows Server 1709 box run docker swarm join
  3. Create an overlay network: docker network create -d overlay test_overlay
  4. Create a Linux service that runs ping (note: this works)
-> docker service create --network test_overlay --constraint "node.platform.os == linux" --name linuxping debian ping www.google.com

-> docker service logs linuxping
linuxping.1.lehid1veibtj@gentoo-minimal    | 64 bytes from 216.58.204.36: icmp_seq=1805 ttl=127 time=7.934 ms
linuxping.1.lehid1veibtj@gentoo-minimal    | 64 bytes from 216.58.204.36: icmp_seq=1806 ttl=127 time=7.922 ms
linuxping.1.lehid1veibtj@gentoo-minimal    | 64 bytes from 216.58.204.36: icmp_seq=1807 ttl=127 time=8.146 ms
linuxping.1.lehid1veibtj@gentoo-minimal    | 64 bytes from 216.58.204.36: icmp_seq=1808 ttl=127 time=8.085 ms
  1. Create a Windows service that runs ping on the overlay network (note: this does not work)
-> docker service create --network test_overlay --constraint "node.platform.os == windows" --name windowsping microsoft/windowsservercore:1709 ping -t www.google.com

-> docker service logs windowsping
windowsping.1.tszyfp0wk6w4@vagrant-1709    | Pinging www.google.com [216.58.211.100] with 32 bytes of data:
windowsping.1.tszyfp0wk6w4@vagrant-1709    | Request timed out.
windowsping.1.tszyfp0wk6w4@vagrant-1709    | Request timed out.
windowsping.1.tszyfp0wk6w4@vagrant-1709    | Request timed out.
windowsping.1.tszyfp0wk6w4@vagrant-1709    | Request timed out.
windowsping.1.tszyfp0wk6w4@vagrant-1709    | Request timed out.
windowsping.1.tszyfp0wk6w4@vagrant-1709    | Request timed out.
windowsping.1.tszyfp0wk6w4@vagrant-1709    | Request timed out.
  1. Create a Windows service that runs ping on the default network (note: this works)
-> docker service create --constraint "node.platform.os == windows" --name windowsping_nonet microsoft/windowsservercore:1709 ping -t www.google.com

-> docker service logs windowsping_nonet
windowsping_nonet.1.yo86r9c6vk5y@vagrant-1709    | Reply from 216.58.211.100: bytes=32 time=15ms TTL=127
windowsping_nonet.1.yo86r9c6vk5y@vagrant-1709    | Reply from 216.58.211.100: bytes=32 time=14ms TTL=127
windowsping_nonet.1.yo86r9c6vk5y@vagrant-1709    | Reply from 216.58.211.100: bytes=32 time=15ms TTL=127
windowsping_nonet.1.yo86r9c6vk5y@vagrant-1709    | Reply from 216.58.211.100: bytes=32 time=15ms TTL=127
windowsping_nonet.1.yo86r9c6vk5y@vagrant-1709    | Reply from 216.58.211.100: bytes=32 time=15ms TTL=127
windowsping_nonet.1.yo86r9c6vk5y@vagrant-1709    | Reply from 216.58.211.100: bytes=32 time=15ms TTL=127

Note: For the sake of brevity I haven’t shown inter-container communication here, but I can confirm that Windows and Linux containers can talk to each other over the overlay network and can expose services on the ingress network without issue.

In conclusion

When connected to the overlay network, Windows containers can’t access the internet. When connected to the NAT network, they can.

I think it’s possible that this is due to the fact that Linux uses the docker_gwbridge network for this purpose. Windows doesn’t seem to want to connect containers on an overlay network to the nat network as well, so I’m not sure what to do here.

Help appreciated!


(Shaun777) #2

Same problem with overlay network on a single windows host in swarm mode. Simply innit swarm mode and your container can no longer connect to the internet. Inter container communication is perfect. Will resolve dns for google but ping request will timeout. I will further note that the host machine is also not reachable.


(Gh2k) #3

Good to know I’m not the only one!

Yeah, anything beyond the overlay network’s gateway is unreachable. It might help to ‘me-too’ on https://github.com/docker/for-win/issues/1726 - it hasn’t had any attention as yet.


(Bilalhaider) #4

I am on windows server 2016 and exploring Windows 1709. Needed to have services that can talk on a hybrid swarm.

Internet access from Windows containers would be needed so this would be a blocker for me. Nonetheless, I am also interested in Inter-host (mixed) container communication. Do you people get this working by virtue of win 1709? I only know that ingress routing support starts from win 1709. Can anybody confirm that the Inter-host communication also comes along.

The github issue is closed now.


(Artisticcheese) #5

I don’t believe routing mesh is GA or available yet, it requires EE-preview build


(Bilalhaider) #6

Yeah, as per this announcement from microsoft, routing mesh is available to Win 1709 + Docker EE preview build.

What I would like to know about is the inter-host (mixed) containers communication on the overlay network, which I can’t get to work on Win 2016 with EE preview builds, as the windows services cannot resolve names of the services running on linux. Is win 1709 the way to go for this too?


(Gh2k) #7

It does work but I couldn’t get internet access from the Windows containers at the same time. One thing I found is that if you have a second NIC in your Windows hosts (which is quite common with Vagrant for example) then it’s quite possible for your overlay network stuff to be bound to the wrong adapter.

There’s a comment on my issue second-from-bottom which has a potential resolution if this is your issue: https://github.com/docker/for-win/issues/1366

I’d worked around my Windows containers needing to access the internet anyway, so haven’t tried it. Instead I reconfigured my environment so that my Windows VMs only have 1 network adapter.


(Bilalhaider) #8

Thanks for pointing out, I will try that and will post here my findings for reference