Docker Community Forums

Share and learn in the Docker community.

How can diagnose issue with Docker not routing my packets?

docker

(U35tpus) #1

The issue is I can’t access LAN hosts from my Docker container.

I have Docker 17.09.1-ce-mac42 (21090) installed on Mac OS (10.11.3)

MacBook-Pro-mac:hms mac$ docker info
Containers: 52
 Running: 2
 Paused: 0
 Stopped: 50
Images: 401
Server Version: 17.09.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.49-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952GiB
Name: moby
ID: PYN3:SSYQ:U5PZ:TRA7:HXIV:OXVV:GCYO:BRN3:YZJQ:WT4R:ZSFM:BICJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 43
 Goroutines: 64
 System Time: 2017-12-28T11:00:23.80904173Z
 EventsListeners: 2
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

My host ifconfig:

en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ether e0:f8:47:2c:52:c6 
        inet 192.168.1.194 netmask 0xffffff00 broadcast 192.168.1.255
        media: autoselect
        status: active
    en2: flags=963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX> mtu 1500
        options=60<TSO4,TSO6>
        ether d2:00:14:6f:82:c0 
        media: autoselect <full-duplex>
        status: inactive
    fw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 4078
        lladdr 00:3e:e1:ff:fe:46:f8:2c 
        media: autoselect <full-duplex>
        status: inactive
    p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304
        ether 02:f8:47:2c:52:c6 
        media: autoselect
        status: inactive
    awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1484
        ether ba:8d:d9:4c:bf:47 
        inet6 fe80::b88d:d9ff:fe4c:bf47%awdl0 prefixlen 64 scopeid 0x9 
        nd6 options=1<PERFORMNUD>
        media: autoselect
        status: active
    bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=63<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 42:6c:8f:25:9c:00 
        Configuration:
            id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
            maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
            root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
            ipfilter disabled flags 0x2
        member: en2 flags=3<LEARNING,DISCOVER>
                ifmaxaddr 0 port 6 priority 0 path cost 0
        media: <unknown type>
        status: inactive
    utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1400
        inet 10.66.20.26 --> 10.66.20.26 netmask 0xffffffff 

Host routing table:

mac$ netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
default            192.168.1.254      UGSc          419       35     en1
10                 10.66.20.26        UGSc            3        0   utun0
10.66.20.26        10.66.20.26        UH             30        0   utun0
10.201.0.2         10.66.20.26        UGHS           13     1172   utun0
10.201.0.3         10.66.20.26        UGHS            1        8   utun0
38.102.149/24      10.66.20.26        UGSc            1        0   utun0
38.102.149.254     192.168.1.254      UGHS            3    87350     en1
65.98.94.64/26     10.66.20.26        UGSc            1        0   utun0
70.42.139/24       10.66.20.26        UGSc            1        0   utun0
108.166.96.52      10.66.20.26        UGHS            1        0   utun0
115.113.154.120    10.66.20.26        UGHS            1        0   utun0
127                127.0.0.1          UCS             1        0     lo0
127.0.0.1          127.0.0.1          UH             14   498710     lo0
140.239.3/24       10.66.20.26        UGSc            1        0   utun0
169.254            link#5             UCS             1        0     en1
172.16             10.66.20.26        UGSc            1        0   utun0
172.18             10.66.20.26        UGSc            1        0   utun0
172.19             10.66.20.26        UGSc            6        0   utun0
172.22             10.66.20.26        UGSc            1        0   utun0
172.23             10.66.20.26        UGSc            1        0   utun0
172.31.0.48/28     10.66.20.26        UGSc            1        0   utun0
172.31.0.240/28    10.66.20.26        UGSc            1        0   utun0
172.200.1/24       10.66.20.26        UGSc            1        0   utun0
192.168.1          link#5             UCS             6        0     en1
192.168.1.20       link#5             UHLWIi          1        0     en1
192.168.1.26       cc:2f:71:ff:31:c9  UHLWIi          1        4     en1    756
192.168.1.77       link#5             UHLWIi          1        0     en1
192.168.1.156      link#5             UHLWIi          1        0     en1
192.168.1.194/32   link#5             UCS             1        0     en1
192.168.1.254      link#5             UHCS            1        0     en1
192.168.1.254/32   link#5             UCS             2        0     en1
192.168.1.254      60:45:cb:18:7f:58  UHLWIir       421       79     en1   1193
192.168.1.255      link#5             UHLWbI          1       55     en1
203.153.13.136     10.66.20.26        UGHS            1        0   utun0
208.83.244         10.66.20.26        UGSc            1        0   utun0
208.83.245         10.66.20.26        UGSc            1        0   utun0
208.83.246         10.66.20.26        UGSc            1        0   utun0
208.116.21.240/28  10.66.20.26        UGSc            1        0   utun0
255.255.255.255/32 link#5             UCS             1        0     en1

Docker compose file:

version: '3.3'
services:
  web:
    build:
      context: .
      dockerfile: Dockerfile
    image: "bf:latest"
    container_name: "bf"
    ports:
     - "8080:8080"
    networks:
      - testnet
  redis:
    image: "redis:alpine"
    container_name: "bfr"
    ports:
     - "6379:6379"
    networks:
      - testnet
networks:
  testnet:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.28.0.0/16

Neither of LAN hosts is not available from inside docker containers:

mac$ docker exec -it bfr /bin/sh
/data # traceroute 172.19.13.63
traceroute to 172.19.13.63, 30 hops max, 46 byte packets
 1  172.16.4.254 (172.16.4.254)  0.006 ms  0.007 ms  0.005 ms
 2  *  *  *
 3  *  *  *
 4  *  *  *
 5  *  *  *
 6  *  *  *
 7  *  *  *

Same host 172.19.13.63 (routed via utun0 as per host routing table printed above) is tracerouted outside of container on the host itself:

 mac$ traceroute -n 172.19.13.63
traceroute to 172.19.13.63 (172.19.13.63), 64 hops max, 52 byte packets
 1  10.200.200.200  380.897 ms  206.423 ms  206.332 ms
 2  10.66.8.1  208.638 ms  205.951 ms  205.722 ms
 3  10.66.250.1  205.885 ms  206.526 ms  205.710 ms
 4  172.19.13.63  205.980 ms  295.470 ms  307.070 ms

I also found the following in the logs (syslog -k Sender Docker):

Dec 28 13:34:03 MacBook-Pro-mac Docker[13675] : DNS lookup syslog A: syslog.corp.ooma.com <IN|54001> [A (10.66.12.76)]
Dec 28 13:39:07 MacBook-Pro-mac Docker[13675] : DNS lookup 254.4.16.172.in-addr.arpa PTR: NoSuchRecord

Apparently docker tried to route packets to 172.19.13.63 via 172.19.13.63 (as opposed to 10.200.200.200 as mac os host does) and failed.

How can I investigate further and fix this issue?