Docker macvlan network on OpenWRT x86

General
1

Hi, I’m trying to use OpenWRT x86/64 as a Docker host to run pihole attached to the LAN network of OpenWRT. I have done this before with an Ubuntu host where pihole is a docker container, running on the Ubuntu host connected to bridged Ethernet network. I had to do the macvlan workaround so that the Ubuntu Host could ping the pihole LAN IP.

With the new release of OpenWRT 21.02, I thought of moving the pihole container into OpenWRT as it supports Docker. However when I’m trying the basic macvlan setup like below, I’m getting an error when starting the container.

docker-compose.yml
...
networks:
  internal:
  lan:
    name: lan
    driver: macvlan
    driver_opts:
      parent: eth2
    ipam:
      config:
        - subnet: 192.168.1.0/24
          gateway: 192.168.1.1
error output:
root@OpenWrt:~/docker-data# docker-compose -f docker-compose.yml up pihole
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Starting pihole ... error

ERROR: for pihole  Cannot start service pihole: failed to create the macvlan port: operation not supported

ERROR: for pihole  Cannot start service pihole: failed to create the macvlan port: operation not supported
ERROR: Encountered errors while bringing up the project.

So it begs to ask, are macvlan networks supported by docker on OpenWRT?

2

I am not sure if there is actualy an official docker package for OpenWRT from Docker itself… You might want to ask the question to whoever maintains the docker package for OpenWRT.

3

Was able to figure it out, needed to install the kmod-macvlan to fix it.

4

Congratz! You still might to want check if your kernel supports/enables required modules to prevent further surprises:

wget https://github.com/moby/moby/raw/master/contrib/check-config.sh
chmod +x check-config.sh
./check-config.sh
5

Thanks for the info, just learned that there’s such a great script.

Output 
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: nonexistent??
    (see https://github.com/tianon/cgroupfs-mount)
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: missing
- CONFIG_CGROUP_FREEZER: missing
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: missing
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_CGROUP_BPF: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: missing
    (cgroup swap accounting is currently not enabled, you can enable it by setting boot option "swapaccount=1")
- CONFIG_LEGACY_VSYSCALL_NONE: enabled
    (containers using eglibc <= 2.13 will not work. Switch to
     "CONFIG_VSYSCALL_[NATIVE|EMULATE]" or use "vsyscall=[native|emulate]"
     on kernel command line. Note that this will disable ASLR for the,
     VDSO which may assist in exploiting security vulnerabilities.)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: missing
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: missing
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_SECURITY_SELINUX: missing
- CONFIG_SECURITY_APPARMOR: missing
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: missing
- CONFIG_EXT4_FS_SECURITY: missing
    enable these ext4 configs if you are using ext3 or ext4 as backing filesystem
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled (as module)
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: missing
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

I did face some issues later on and I’m wondering. I have in my docker-compose.yaml the following:

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    hostname: pihole.lan
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "67:67/udp"
      - "80:80/tcp"
...
    networks:
      lan:
        ipv4_address: 192.168.1.11
...
networks:
  lan:
    name: lan
    driver: macvlan
    driver_opts:
      parent: eth2.20
    ipam:
      config:
        - subnet: 192.168.1.0/24
          gateway: 192.168.1.1

When I try starting the container, the following messages appear in the syslog:

 daemon.err dockerd[2734]: time="2021-09-21T23:52:31.428747251Z" level=warning msg="macvlan driver does not support port mappings"
 daemon.err dockerd[2734]: time="2021-09-21T23:52:31.428983631Z" level=warning msg="macvlan driver does not support port exposures"

Getting into the container to try pinging anything also fails:

root@OpenWrt:~# docker exec -it pihole bash
root@pihole:/# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 23ms

root@pihole:/# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
--- 192.168.1.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

root@pihole:/# exit
root@pihole:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
15: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:c0:a8:01:0e brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.1.11/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
16: eth1@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1
       valid_lft forever preferred_lft forever

Appreciate to hear your thoughts on this, I think the warnings are saying that the macvlan setup didn’t work right?

6

The output of check-config.sh is kind of worying. In the “Generally Necessary” section some kernel modules are listed as missing and the cgroup hierarchy is listed as "nonexistent?? ". You realy should share this output with the maintainer of the docker packages.

I am surprised that this works with ip_range, which defines the cidr inside the subnet used for macvlan child interfaces. This ip range must be outside your dhcp range and should have no other existing devicess in it. On the cli there is a syntax to exclude single ips from the range using aux-address, but I have no idea how it translates to a compose file configuration.
I am sure you already know this, but it’s worth knowing that a macvlan client interface is not allowed to communicate with it’s parent interface - this is restriction implemented in the kernel and not a docker “feature”

Published ports only work for bridge networks (or if swarm mode is enabled and used in overlay networks). Neither host, nor macvlan networks allow or required publised ports. Just remove them.