No external connection using secondary network

sseypt · November 16, 2018, 2:47pm

Hello.

I can’t get an external connection using the secondary interface inside a container.

Steps to reproduce the behavior

Create bridge
Connect to container as secondary network
Request google.de 80 using nc with and without secondary network.
(cleanup)

docker network create --driver=bridge --subnet=172.42.0.0/24 test_network \
&& docker run --rm -dit --name networktest --net=default alpine:3.8 sh \
&& docker network connect --ip=172.42.0.2 test_network networktest \
&& docker exec networktest sh -c 'printf "GET / HTTP/1.0\r\n\r\n" | nc -w 1 google.de 80 >/dev/null && echo "eth0: got response" || echo "eth0: no response"' \
&& docker exec networktest sh -c 'printf "GET / HTTP/1.0\r\n\r\n" | nc -w 1 -s 172.42.0.2 google.de 80 >/dev/null && echo "eth1: got response" || echo "eth1: no response"' \
&& docker rm -f networktest >/dev/null \
&& docker network rm test_network >/dev/null

Connects on book2docker - Tiny Core Linux - Docker 17.12.0-ce

eth0: got response
eth1: got response

Doesn’t connect using Ubuntu 18.04 or CentOs 7 as Docker-Host

eth0: got response
eth1: no response

As far as I can tell, all seems “identical”.

docker network inspect test_network
route -n
ifconfig

except iptables -S

# Tiny Core Linux
...
-A DOCKER-ISOLATION -i docker0 -o br-5856fcd16cd7 -j DROP
-A DOCKER-ISOLATION -i br-5856fcd16cd7 -o docker0 -j DROP

# Ubuntu
...
-A DOCKER-ISOLATION-STAGE-1 -i br-9771e9429386 ! -o br-9771e9429386 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-2 -o br-9771e9429386 -j DROP

What am I missing?
Please ask, If you need any more informations.

Cheers,
Steve

Output of docker version:

Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:48:22 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:19:08 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 5
Server Version: 18.09.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-39-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.947GiB
Name: networktest
ID: UOFN:XNTE:REDJ:PW3I:T3PN:NK4Y:CVPU:6VDQ:5BSL:SXZM:AFIJ:XGWO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 docker.mailingwork.local
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

Output of docker network inspect test_network:

[
    {
        "Name": "test_network",
        "Id": "9771e9429386c40c68606a4d5bc11fb8b31a98b0082a20b49b92f0e25d20c166",
        "Created": "2018-11-16T14:24:20.413739169Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.42.0.0/24"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "7f0efbeed0e9200f089aface518c04bd35c9e2a3005bb6e50155143ff4fa74e0": {
                "Name": "networktest",
                "EndpointID": "5acfa6a40ac8321db4ee5251c1fdb86a3ce39a098fdb260c0ae4b8e3f727535b",
                "MacAddress": "02:42:ac:2a:00:02",
                "IPv4Address": "172.42.0.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Output of route -n:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG    100    0        0 enp0s3
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 enp0s3
10.0.2.2        0.0.0.0         255.255.255.255 UH    100    0        0 enp0s3
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.42.0.0      0.0.0.0         255.255.255.0   U     0      0        0 br-9771e9429386
192.168.56.0    0.0.0.0         255.255.255.0   U     0      0        0 enp0s8

Output of ifconfig:

br-9771e9429386: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.42.0.1  netmask 255.255.255.0  broadcast 172.42.0.255
        inet6 fe80::42:99ff:fe64:26c3  prefixlen 64  scopeid 0x20<link>
        ether 02:42:99:64:26:c3  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12  bytes 936 (936.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:84ff:fe51:d3e0  prefixlen 64  scopeid 0x20<link>
        ether 02:42:84:51:d3:e0  txqueuelen 0  (Ethernet)
        RX packets 222453  bytes 9048846 (9.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 234090  bytes 466451886 (466.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
        inet6 fe80::a00:27ff:fe56:743d  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:56:74:3d  txqueuelen 1000  (Ethernet)
        RX packets 833216  bytes 777066076 (777.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 323488  bytes 19821538 (19.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.56.42  netmask 255.255.255.0  broadcast 192.168.56.255
        inet6 fe80::a00:27ff:fe86:4985  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:86:49:85  txqueuelen 1000  (Ethernet)
        RX packets 12647  bytes 1377055 (1.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 11141  bytes 1938398 (1.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 370  bytes 32794 (32.7 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 370  bytes 32794 (32.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth547f586: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::60c0:3ff:fe2c:565e  prefixlen 64  scopeid 0x20<link>
        ether 62:c0:03:2c:56:5e  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 24  bytes 1872 (1.8 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethf9133e3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::a406:3cff:fefa:b5f7  prefixlen 64  scopeid 0x20<link>
        ether a6:06:3c:fa:b5:f7  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14  bytes 1116 (1.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Output of iptables -S:

-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-9771e9429386 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-9771e9429386 -j DOCKER
-A FORWARD -i br-9771e9429386 ! -o br-9771e9429386 -j ACCEPT
-A FORWARD -i br-9771e9429386 -o br-9771e9429386 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-9771e9429386 ! -o br-9771e9429386 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-9771e9429386 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN

Additional environment details (AWS, VirtualBox, physical, etc.)

Tested on TCL, CentOS and Ubuntu on several VMs (VirtualBox, VMware ESXi, OnApp).

sseypt · November 19, 2018, 10:35am

I noticed one more difference.
But the change to -P FORWARD ACCEPT did not change the behavior in my case.

sseypt · November 19, 2018, 10:44am

I also tested with a container which does only use test_network as eth0.

That works as expected.
iptables -S were identical though.

2nd example

docker network create --driver=bridge --subnet=172.42.0.0/24 test_network \
&& docker run --rm -dit --name networktest --net=test_network --ip=172.42.0.2 alpine:3.8 sh \
&& docker exec networktest sh -c 'printf "GET / HTTP/1.0\r\n\r\n" | nc -w 1 google.de 80 >/dev/null && echo "eth0: got response" || echo "eth0: no response"' \
&& docker rm -f networktest >/dev/null \
&& docker network rm test_network >/dev/null

Is there a mechanism in docker, which allows eth1 to connect to the internet on one docker-host and does not on another docker-host with another os?

Topic		Replies	Views
New bridge network is not working well Docker Hub	0	299	January 27, 2021
Network issue with custom created bridge General docker	1	1194	June 23, 2016
Bridge connection not working General	3	4393	November 17, 2020
Docker Networking assistance please General	1	2270	October 12, 2018
Internal network between containers without external network access General docker	9	87686	December 14, 2020

No external connection using secondary network

Steps to reproduce the behavior

2nd example

Related topics