Docker Community Forums

Share and learn in the Docker community.

No external connection using secondary network

docker

(Steve) #1

Hello.

I can’t get an external connection using the secondary interface inside a container.

Steps to reproduce the behavior

  1. Create bridge
  2. Connect to container as secondary network
  3. Request google.de 80 using nc with and without secondary network.
  4. (cleanup)
docker network create --driver=bridge --subnet=172.42.0.0/24 test_network \
&& docker run --rm -dit --name networktest --net=default alpine:3.8 sh \
&& docker network connect --ip=172.42.0.2 test_network networktest \
&& docker exec networktest sh -c 'printf "GET / HTTP/1.0\r\n\r\n" | nc -w 1 google.de 80 >/dev/null && echo "eth0: got response" || echo "eth0: no response"' \
&& docker exec networktest sh -c 'printf "GET / HTTP/1.0\r\n\r\n" | nc -w 1 -s 172.42.0.2 google.de 80 >/dev/null && echo "eth1: got response" || echo "eth1: no response"' \
&& docker rm -f networktest >/dev/null \
&& docker network rm test_network >/dev/null

Connects on book2docker - Tiny Core Linux - Docker 17.12.0-ce :slight_smile:

eth0: got response
eth1: got response

Doesn’t connect using Ubuntu 18.04 or CentOs 7 as Docker-Host :frowning:

eth0: got response
eth1: no response

As far as I can tell, all seems “identical”.

docker network inspect test_network
route -n
ifconfig

except iptables -S

# Tiny Core Linux
...
-A DOCKER-ISOLATION -i docker0 -o br-5856fcd16cd7 -j DROP
-A DOCKER-ISOLATION -i br-5856fcd16cd7 -o docker0 -j DROP

# Ubuntu
...
-A DOCKER-ISOLATION-STAGE-1 -i br-9771e9429386 ! -o br-9771e9429386 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-2 -o br-9771e9429386 -j DROP

What am I missing?
Please ask, If you need any more informations.

Cheers,
Steve

Output of docker version:

Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:48:22 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:19:08 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 5
Server Version: 18.09.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-39-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.947GiB
Name: networktest
ID: UOFN:XNTE:REDJ:PW3I:T3PN:NK4Y:CVPU:6VDQ:5BSL:SXZM:AFIJ:XGWO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 docker.mailingwork.local
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

Output of docker network inspect test_network:

[
    {
        "Name": "test_network",
        "Id": "9771e9429386c40c68606a4d5bc11fb8b31a98b0082a20b49b92f0e25d20c166",
        "Created": "2018-11-16T14:24:20.413739169Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.42.0.0/24"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "7f0efbeed0e9200f089aface518c04bd35c9e2a3005bb6e50155143ff4fa74e0": {
                "Name": "networktest",
                "EndpointID": "5acfa6a40ac8321db4ee5251c1fdb86a3ce39a098fdb260c0ae4b8e3f727535b",
                "MacAddress": "02:42:ac:2a:00:02",
                "IPv4Address": "172.42.0.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Output of route -n:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG    100    0        0 enp0s3
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 enp0s3
10.0.2.2        0.0.0.0         255.255.255.255 UH    100    0        0 enp0s3
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.42.0.0      0.0.0.0         255.255.255.0   U     0      0        0 br-9771e9429386
192.168.56.0    0.0.0.0         255.255.255.0   U     0      0        0 enp0s8

Output of ifconfig:

br-9771e9429386: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.42.0.1  netmask 255.255.255.0  broadcast 172.42.0.255
        inet6 fe80::42:99ff:fe64:26c3  prefixlen 64  scopeid 0x20<link>
        ether 02:42:99:64:26:c3  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12  bytes 936 (936.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:84ff:fe51:d3e0  prefixlen 64  scopeid 0x20<link>
        ether 02:42:84:51:d3:e0  txqueuelen 0  (Ethernet)
        RX packets 222453  bytes 9048846 (9.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 234090  bytes 466451886 (466.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
        inet6 fe80::a00:27ff:fe56:743d  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:56:74:3d  txqueuelen 1000  (Ethernet)
        RX packets 833216  bytes 777066076 (777.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 323488  bytes 19821538 (19.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.56.42  netmask 255.255.255.0  broadcast 192.168.56.255
        inet6 fe80::a00:27ff:fe86:4985  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:86:49:85  txqueuelen 1000  (Ethernet)
        RX packets 12647  bytes 1377055 (1.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 11141  bytes 1938398 (1.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 370  bytes 32794 (32.7 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 370  bytes 32794 (32.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth547f586: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::60c0:3ff:fe2c:565e  prefixlen 64  scopeid 0x20<link>
        ether 62:c0:03:2c:56:5e  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 24  bytes 1872 (1.8 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethf9133e3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::a406:3cff:fefa:b5f7  prefixlen 64  scopeid 0x20<link>
        ether a6:06:3c:fa:b5:f7  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14  bytes 1116 (1.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Output of iptables -S:

-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-9771e9429386 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-9771e9429386 -j DOCKER
-A FORWARD -i br-9771e9429386 ! -o br-9771e9429386 -j ACCEPT
-A FORWARD -i br-9771e9429386 -o br-9771e9429386 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-9771e9429386 ! -o br-9771e9429386 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-9771e9429386 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN

Additional environment details (AWS, VirtualBox, physical, etc.)

Tested on TCL, CentOS and Ubuntu on several VMs (VirtualBox, VMware ESXi, OnApp).


(Steve) #2

I noticed one more difference.
But the change to -P FORWARD ACCEPT did not change the behavior in my case.


(Steve) #3

I also tested with a container which does only use test_network as eth0.

That works as expected.
iptables -S were identical though.

2nd example

docker network create --driver=bridge --subnet=172.42.0.0/24 test_network \
&& docker run --rm -dit --name networktest --net=test_network --ip=172.42.0.2 alpine:3.8 sh \
&& docker exec networktest sh -c 'printf "GET / HTTP/1.0\r\n\r\n" | nc -w 1 google.de 80 >/dev/null && echo "eth0: got response" || echo "eth0: no response"' \
&& docker rm -f networktest >/dev/null \
&& docker network rm test_network >/dev/null

Is there a mechanism in docker, which allows eth1 to connect to the internet on one docker-host and does not on another docker-host with another os?