Docker Container not reachable from the outside (only after ping from the inside) - MACVLAN

Hi all,

I have a strange issue with my docker environment since last Friday, before Friday it was running for half a year without any issues.

Since Friday I only can reach my containers from the outside, if I ping the user machine who wants to reach the container beforehand from the container itself. The issue is with all containers not only the ones who run a http server (below it is just an test container)-

Any idea?

Environment:

  1. Docker is running on a raspberry pi model 4

  2. I use macvlan with define/ unique IPs and MACs as I want to access the docker containers directly by IP

  3. Network is created with the following command: docker network create -d macvlan --subnet=192.168.178.0/24 --gateway=192.168.178.1 -o parent=eth0 pub_net

  4. Container start command (all my containers are started the same way, even this is just a test container): docker run –d --name testNetwork --restart unless-stopped --net=pub_net --ip=192.168.178.220 --mac-address 00:00:00:20:20:20 --hostname testNetwork --dns=192.168.178.21 nodered/node-red

Host information:

ip route

default via 192.168.178.1 dev eth0 proto dhcp src 192.168.178.20 metric 202 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.178.0/24 dev eth0 proto dhcp scope link src 192.168.178.20 metric 202

ip link

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 
    link/ether dc:a6:32:22:9b:7e brd ff:ff:ff:ff:ff:ff 
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:75:c9:25:19 brd ff:ff:ff:ff:ff:ff

docker network ls

NETWORK ID          NAME                DRIVER              SCOPE 
b858750d92dc        bridge              bridge              local 
637ad6cfdbd0        host                host                local 
e873e63ce306        none                null                local 
c2c032586768        pub_net             macvlan             local 

Container:

ip address

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 
    inet 127.0.0.1/8 scope host lo 
       valid_lft forever preferred_lft forever 
46: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 00:00:00:20:20:20 brd ff:ff:ff:ff:ff:ff 
    inet 192.168.178.220/24 brd 192.168.178.255 scope global eth0 
       valid_lft forever preferred_lft forever

ip route

default via 192.168.178.1 dev eth0 
192.168.178.0/24 dev eth0 scope link  src 192.168.178.220

Thanks, and Best
Cox42

Hello, I have the exact same issue.
I am trying to set up pihole on an raspberry pi 2 on a container via macvlan interface. The container is not responding to ARP replies before I ping the outside host from which I want to access the container, from within the container. After I ping a host from the container, that host will populate it’s arp table with the container’s mac address.
Below, please find the details of my setup:

Docker info:

Client:
 Debug Mode: false

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 19.03.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.97-v7+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 926.1MiB
 Name: raspberrypi
 ID: SWGQ:PDUG:JZEZ:N4QH:NG7H:HUE6:ZSW5:NXBE:GZZT:NUXZ:2HLH:53XK
 Docker Root Dir: /srv/dev-disk-by-label-docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

Container compose file:

version: '2'

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    mac_address: d0:ca:ab:cd:ef:01
    cap_add:
      - NET_ADMIN
    networks:
      pihole_network:
        ipv4_address: 192.168.100.199
    dns:
      - 127.0.0.1
      - 8.8.8.8
    ports:
      - "443:443/tcp"
      - "53:53/tcp"
      - "53:53/udp"
      - "67:67/udp"
      - "80:80/tcp"
    environment:
      TZ: 'Europe/Bucharest'
      VIRTUAL_HOST: pihole.home.geo
      WEBPASSWORD: "XXXXXX"
      DNS1: 1.1.1.1
      DNS2: 1.0.0.1
      DNSMASQ_LISTENING: all
      ServerIP: 192.168.100.199
    restart: unless-stopped
    volumes:
      - /etc/pihole
      - /etc/dnsmasq.d

networks:
  pihole_network:
    driver: macvlan
    driver_opts:
      parent: eth0
    ipam:
      config:
        - subnet: 192.168.100.0/24
          gateway: 192.168.100.1
          ip_range: 192.168.100.192/28
          aux_address: 192.168.100.100

Docker Networks:

NETWORK ID          NAME                DRIVER              SCOPE
a49c40eed7cb        bridge              bridge              local
a66e512eb428        host                host                local
7560b65028f8        none                null                local
151b447a370b        pi_pihole_network   macvlan             local```

Docker macvlan network:
```$ docker network inspect pi_pihole_network
[
    {
        "Name": "pi_pihole_network",
        "Id": "151b447a370bd8806f358644f77761f1dd817fd14d632ea428a2cb1233e99cad",
        "Created": "2020-03-04T18:26:01.78449681Z",
        "Scope": "local",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.100.0/24",
                    "IPRange": "192.168.100.192/28",
                    "Gateway": "192.168.100.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "b257249c4dd2e3fad43184d18c75ae96f67fb64a11d55ae168dab3891f623aea": {
                "Name": "pihole",
                "EndpointID": "51e1bfd3c51a9f221400e9a08270dee055a532524b9a361005fb7a674497ac67",
                "MacAddress": "d0:ca:ab:cd:ef:01",
                "IPv4Address": "192.168.100.199/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "parent": "eth0"
        },
        "Labels": {}
    }
]

Docker Container:

$ docker network inspect pi_pihole_network
[
    {
        "Name": "pi_pihole_network",
        "Id": "151b447a370bd8806f358644f77761f1dd817fd14d632ea428a2cb1233e99cad",
        "Created": "2020-03-04T18:26:01.78449681Z",
        "Scope": "local",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.100.0/24",
                    "IPRange": "192.168.100.192/28",
                    "Gateway": "192.168.100.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "b257249c4dd2e3fad43184d18c75ae96f67fb64a11d55ae168dab3891f623aea": {
                "Name": "pihole",
                "EndpointID": "51e1bfd3c51a9f221400e9a08270dee055a532524b9a361005fb7a674497ac67",
                "MacAddress": "d0:ca:ab:cd:ef:01",
                "IPv4Address": "192.168.100.199/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "parent": "eth0"
        },
        "Labels": {}
    }
]

In order to permit network communication between the container and the host network I also created another macvlan interface with eth0 as parent interface:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:85:cd:5d brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.30/24 brd 192.168.100.255 scope global dynamic eth0
       valid_lft 5959328sec preferred_lft 5959328sec
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:1a:7c:ac:14 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
5: veth40e3e9d@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether 3a:77:72:1b:d0:41 brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: macvlan0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 86:97:a1:79:ee:dd brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.195/28 scope global macvlan0
       valid_lft forever preferred_lft forever
pi@raspberrypi:/etc/network $ sudo ip link  show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether b8:27:eb:85:cd:5d brd ff:ff:ff:ff:ff:ff
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 02:42:1a:7c:ac:14 brd ff:ff:ff:ff:ff:ff
5: veth40e3e9d@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default
    link/ether 3a:77:72:1b:d0:41 brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: macvlan0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 86:97:a1:79:ee:dd brd ff:ff:ff:ff:ff:ff

IP route of raspberry:

$ ip route
default via 192.168.100.1 dev eth0
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.100.0/24 dev eth0 proto kernel scope link src 192.168.100.30
192.168.100.192/28 dev macvlan0 proto kernel scope link src 192.168.100.195

IP route of pihole container:

default via 192.168.100.1 dev eth0
192.168.100.0/24 dev eth0 proto kernel scope link src 192.168.100.199

To reproduce the issue I will try to reach the container from my PC, which is connected to the network via an eth switch. I will flush the arp cache form both the pi and my PC, restart the pihole container and then try to connect to it from my PC. Ping will fail until I ping the PC from the container because the container is not responding to ARP requests, therefore, my doesn’t have the MAC address of the pihole container.

PC IP addr: 192.168.100.4
PI IP addr: 192.168.100.30
Container IP addr: 192.168.100.199

Below, is an arp capture after flushing the arp cache on the pc and pi, and restarting the container:

pi@raspberrypi:~ $ cat tcpdump.arp
23:32:16.981417 ARP, Request who-has _gateway tell 192.168.100.199, length 28
23:32:16.982087 ARP, Reply _gateway is-at 20:f1:7c:2d:de:4e (oui Unknown), length 46
23:32:22.424704 ARP, Request who-has _gateway tell raspberrypi, length 28
23:32:22.425232 ARP, Reply _gateway is-at 20:f1:7c:2d:de:4e (oui Unknown), length 46
23:33:05.144724 ARP, Request who-has _gateway tell 192.168.100.199, length 28
23:33:05.145358 ARP, Reply _gateway is-at 20:f1:7c:2d:de:4e (oui Unknown), length 46

Below is a tcpdump and ping output while trying to ping the container (.199) from my pc (.4):

ping 192.168.100.199

Pinging 192.168.100.199 with 32 bytes of data:
Reply from 192.168.100.4: Destination host unreachable.
Reply from 192.168.100.4: Destination host unreachable.
Reply from 192.168.100.4: Destination host unreachable.
Reply from 192.168.100.4: Destination host unreachable.

23:35:35.489507 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:35:36.089711 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:35:37.089188 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:35:38.097340 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:35:39.089430 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:35:40.090133 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:35:41.064716 ARP, Request who-has _gateway tell raspberrypi, length 28
23:35:41.065268 ARP, Reply _gateway is-at 20:f1:7c:2d:de:4e (oui Unknown), length 46
23:35:41.094970 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:35:42.090042 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:35:43.089270 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:35:44.096895 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:35:45.089774 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:35:46.090225 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:36:59.304728 ARP, Request who-has _gateway tell raspberrypi, length 28
23:36:59.305463 ARP, Reply _gateway is-at 20:f1:7c:2d:de:4e (oui Unknown), length 46
23:37:07.590683 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:37:08.602561 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:37:09.590424 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:37:10.589920 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:37:11.595971 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:37:12.590187 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:37:13.590145 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:37:14.596994 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:37:15.590554 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46
23:37:16.590022 ARP, Request who-has 192.168.100.199 tell 192.168.100.4, length 46

I am running on docker on:

$ uname -a
Linux raspberrypi 4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l GNU/Linux
$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

Is there something wrong with this config? I’ve been debugging this for 2 days and running out of options. I will try next to get a capture on the docker macvlan interface from within a container and see what I get.

Any help or explanation on why this is happening would be appreciated.

Thank you,
George

Hi all,

I solved the issue thanks with an update of the kernel - looks like an ARP issue.
Link to the Solution

Best
Cox42

this worked! thank you for the info!

Regards,
George

Hello,
I have the same problem with a new container of ioBroker for a newly installed Raspberry Pi 4. I updated the kernel from 4.19.97-v7l+ to 4.19.114-v7l+ using sudo rpi-update, but I still have the same behaviour after rebooting. Any other ideas for a Linux newbie?
Best regards, Steffen

Hello,

I have also the same issue on Raspian Buster after update on my rpi4.
So no further update helps. So my “complete” infrastructure is “down”.

Also a clean install of the os does not work. Need help.

Thank you Regards
Thorsten

My problem was giving the docker container the IP 250. It seems that the FritzBox has a problem to handle this specific IP. I switched the configuration to IP 249 and it worked.

Hello,

I also tried to go to raspian lite “bullseye” (Testing), but no change.
Only going back to an raspian lite of 26. Sept 2019 helps.

But this could not be the solution.

Regards
Thorsten