Failing at deploying containers with macvlan - help?

General
1

I have a Raspberry Pi 4 with Docker running 5 containers on 10.0.20.x / macvlan20, 1 on 10.0.40.x / macvlan40, and a few others like Portainer & Watchtower that don’t need to be on those. It works great. I’d like to rebuild the entire thing in a VM on a different device.

I’ve got a 2013 Mac Pro, the trashcan version. Not great, but they’re cheap on eBay and reasonably upgradable - up to 12 core/24 thread, 128GB RAM, internal NVMe storage, about $300. Fresh install of macOS 12, fresh install of VirtualBox 6.1. I’ve tried using Ubuntu Server 22.04.1, Photon OS 4 Rev2, and Debian 11.5 as the guest OS, and I can get Docker running just fine, I can get a container running just fine, but as soon as I try and put it on a macvlan network, it doesn’t work. I can see the container in my UniFi network controller - it sees the hostname, the MAC address, and the IP address I’ve set, but I can’t connect via the web portal for containers that have web portals (PiHole, Home Assistant, Homebridge), and if I ping them from my laptop I get no response on any container attempted (Unbound, NTP). (I’ve been using the laptop to ping them because I’ve read that you can’t ping the containers from the host, and the host is downstairs and headless anyway.)

I’ve tried using different network adapter types in VirtualBox (Intel PRO/10000 MT Desktop, Intel PRO/1000 MT Server, PCnet-FAST III), I’ve made sure Promiscuous Mode is on in the VirtualBox settings (tried Allow VMs and Allow All), I’ve set up the macvlan networks through command line and Portainer, and the outcome doesn’t change.

The Mac Pro is wired over ethernet to UniFi switch, and the port profile is set to “All”. I’ve also tried IPvlan for kicks, no difference. I’ve tried deploying containers manually through Portainer, using compose files in Portianer’s Stacks, and straight through command line - when they’re not on macvlan they work great, when they’re on macvlan they don’t work. And none of the containers throw up any error messages.

The command I’ve been using to make the macvlan network is

sudo docker network create -d macvlan --subnet=10.0.20.0/24 --gateway=10.0.20.1 -o parent=eth0.20 macvlan20

(Depending on the VM, sometimes it’s eth0, sometimes it’s enp0s3, I always check every time I try a new VM.)

(The original goal was to run ESXi on the Mac Pro 2013, since it’s supposed to be fully supported - 7 U3 or something like that. But ESXi broke itself twice, and too many features were locked away in vSphere that I decided to try VirtualBox instead. I had the exact same experience with containers working fine no matter the guest OS, and containers + macvlan failing.)

2

Back in the days, when I was still using ESXi, macvlan worked like a charm. I have no idea whether macvlan an VBox actualy works. Did you try to configure the vnic to use bridge mode?

I am running containers without any problems in KVN vm’s running on nodes of my Proxmox cluster.

3

VM > Network > Adapter 1 > Attached To > Bridged Adapter

As I was doing some searching, I came across this NetworkChuck video, and his demo was using docker containers inside a VirtualBox guest OS, so I figured I’d try VirtualBox as a second option.

4

If you already use the adapter in bridged mode and enabled promiscuous mode, we have seen in the video that it should work. Though, he did set the interface in promiscuous mode using ip link set ${nic} promisc on from the command line within the vm. I don’t recall that I would ever have required it at all for macvlan to work.

Anyway, usually if things are not explainable, it boils down to the used installation respo itself. Did you install the docker-ce package from docker repositories, the docker package from OS repositories or the snap package?

5

I’m able to run that and put the VM into promiscuous mode, but that doesn’t change the behavior. (I’m also not able to get it to stick after a reboot, and I haven’t come across an article that tells me how to do it in 2022 - the articles from 2014 I find when I search reference a config file that doesn’t seem to exist.)

Screen Shot 2022-09-12 at 6.22.21 PM
Screen Shot 2022-09-12 at 6.22.21 PM2204×1890 994 KB

I install the Docker repositories myself, using the instructions on Docker’s website. I screen share from my laptop to the Mac Pro, set up the VM, run the first update/upgrade, and then switch to SSH-ing to the machine from my laptop. Open the install page on monitor one, Terminal on monitor two, copy and paste. (Laptop is wired, 10G fiber to Thunderbolt 3 to SFP+ adapter.)

6

I mentioned it just to highlight that it’s not required, even though it was the first thing networkchuck did.
So you did install docker-ce from the docker repositories.

Your configuration checks all the required boxes to macvlan work. I have no idea why it isn’t working.
You do try to access the service inside the macvlan container using the container ports, correct?

7

You do try to access the service inside the macvlan container using the container ports, correct?

I think so? Like Homebridge should be 10.0.20.6:8581, Home Assistant should be 10.0.20.7:8123? Unbound & NTP don’t have any web access like that.

(Some of these probably don’t need to have individual IPs, I don’t think Homebridge and Home Assistant have any conflicting ports. It’s mainly PiHole - I want a normal instance of PiHole running on my main VLAN, using Unbound to resolve. Then I want a second instance of PiHole on another VLAN, that uses a different DNS resolving for other things on the network.)

I’m also doing all my testing with the working Raspberry Pi turned off, so there won’t be any port conflicts with the setup that worked, and I give pretty much everything a static IP so there wont be port conflicts there.

8

As a follow up, I’ve found that it works fine if I set up a regular macvlan network. It’s only when I try and tag it that I have an issue.

If I deploy a container without altering the network, it works. I can access the container through a browser (when appropriate), I can ping it, it works.
If I create a macvlan network on enp0s3, it works. I can access the container through a browser (when appropriate), I can ping it, it works.
As soon as I try and tag the macvlan network - like enp0s3.20 or enp0s.40, the container deploys, thinks nothing is wrong, my router recognizes a device, but it cannot be accessed and pings go unanswered.

9

You might want to try if it works with ipvlan.

While macvlan provides a mac per macvlan child interface, ipvlan uses the same mac for all ipvlan child interfaces. ipvlan can act on layer 2 (~=as bridge to the host interface, where the child interfaces are attached) or layer 3 (~=router between host network and the ipvlan network).

But according Docker Networking: macvlans with VLANs – HiCube, it should also work with macvlan.