Accessing udp for check_dhcp from a container

[gregbryant]

i’m new to docker, so the following is pretty easy for anyone to verify.

i want to run nagios from inside a container. this can be done with:

docker run -d -p 25:25 -p 80:80 quantumobject/docker-nagios:latest

almost everything works. the exception is this (from inside the container):

/usr/local/nagios/libexec/check_dhcp

which will fail:

CRITICAL: No DHCPOFFERs were received.

if i run check_dhcp outside the container, on the host, it works. i get:

OK: Received 1 DHCPOFFER(s), max lease time = 86400 sec.

i suppose i need to make udp ports 67 and 68 available to the container, so, running:

docker run -d --expose 67 --expose 68 -p 67:67/udp -p 68:68/udp -p 25:25/tcp -p 80:80/tcp quantumobject/docker-nagios:latest

gives me this error:

Error starting userland proxy: listen udp 0.0.0.0:68: bind: address already in use

caused by dhclient on the host. but, many processes can listen to the same ports, so I do this:

docker run -d --expose 67 --expose 68 -p 6700:67/udp -p 6800:68/udp -p 25:25/tcp -p 80:80/tcp quantumobject/docker-nagios:latest

which runs. and then i add these iptables rules … admittedly just a guess:

iptables -A FORWARD -i eth0 -o docker0 -p udp -m udp --sport 68 --dport 6800 -j ACCEPT
iptables -A FORWARD -i docker0 -o eth0 -p udp -m udp --sport 6800 --dport 68 -j ACCEPT
iptables -A FORWARD -i eth0 -o docker0 -p udp -m udp --sport 67 --dport 6700 -j ACCEPT
iptables -A FORWARD -i docker0 -o eth0 -p udp -m udp --sport 6700 --dport 67 -j ACCEPT

the behavior hasn’t changed. check_dhcp works on the host, but fails inside the container.

please help a newbie. what’s the correct approach here?
[running ubuntu15x10 4.2.0-16-generic, and Docker version 1.6.2, build 7c8fca2]

[programmerq]

I did a bit of googling, and found this: https://exchange.nagios.org/components/com_mtree/attachment.php?link_id=1817&cf_id=24

Is that the check_dhcp script you are trying to run?

I would expect that to fail in a container since docker containers don’t run dhcp. the ip address management is handled by the daemon so you don’t need to worry about things like networking inside the container.

[gregbryant]

Hi Jeff,

Thank you for the quick response.

The nagios check_dhcp is in c, built from nagios-plugins-2.1.1/plugins-root/check_dhcp.c … you can see the code in their repository here: https://github.com/nagios-plugins/nagios-plugins/blob/master/plugins-root/check_dhcp.c

You wrote that “docker containers don’t run dhcp”, so I’m guessing we’re not on the same page yet, because you also wrote “ip address management is handled by the daemon”, and that’s not what check_dhcp is for.

check_dhcp is a linux application that looks for dhcp servers, either on its subnet generally, or at a specific server IP. It broadcasts udp on port 67, and listens for a response on its own port 68. A web server does something similar, albeit with a different protocol, on port 80. So, in that sense (admittedly not the first thing that comes to mind when people think “dhcp”) docker probably does do dhcp. I should also be able to put a dhcp server in a linux container.

But, my question is, why are these udp packets not getting through to the container’s eth0 interface? I’m certain some networking architect at docker already has an approach in mind, because otherwise there wouldn’t be a udp directive in docker run -p … but, surprisingly, I can find nothing on this topic except for cryptic and indirect references to the userland proxy, which fails when you try to bind a container to port 68 directly (because it already has a listener).

If you could get someone’s attention on this, I’d really appreciate it. It isn’t possible to use any network monitoring software on docker without a resolution.

Thank you!

[programmerq]

Ah, understood. I thought that check_dhcp might be looking at the local server’s dhcp lease to make sure there was a valid lease. I should never make assumptions.

In the default docker network strategy is to create a veth interface for the conatiner, and that is NATed.

The docker NAT interface does not include a dhcp request forward component.

It might be appropriate to run your check_dhcp check in a container that uses --net=host. Basically that will cause the container’s networking setup to skip the docker network isolation bits, and it’ll run directly on the host’s interfaces.

[gregbryant]

Thanks again Jeff.

Your –net=host suggestion certainly solved this problem …

docker run -d --net=host -quantumobject/docker-nagios

… lets check_dhcp work properly!

I wasn’t able to get your other suggestion to work. Unfortunately, that’s kind of limiting. Looking towards the future, –net=host kind of encourages one container per machine.

Maybe I’m doing it badly? I substituted the generated vethxxxxxxx interface for docker0, in the iptables rules at the top of this thread.

You said “The docker NAT interface does not include a dhcp request forward component”. I assume you’re talking about the veth interface now. But I’m not sure how dhcp is involved. My iptables rules just get the kernel to forward packets received on one interface’s port 68 to the docker-daemon-generated veth interface. Of course, the veth interface driver doesn’t need to accept udp, but, again, I’d be surprised by that, since docker run -p has a udp directive. So, could you find the veth driver-writer and ask if my iptables rules could work, with a bit of tweaking?

Thank you!

[programmerq]

So, it’s been some time since I’ve dealt with dhcp at this level, so apologies for any gaps in my explanation. Let me brush up on my rfc2131. there we go! (https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol#Operation gave me a good reminder about how all this works)

So, when your DHCP client wants to get a lease, it sends out a DHCPDISCOVER udp broadcast. This broadcast packet only goes out on the network the client is connected to.

Since in your case, the dhcp client is on the docker0 network, and not on the subnet that your host is on, the DHCPDISCOVER packet never makes it to your dhcp server.

Since your dhcp server doesn’t get that packet, it never sends out a DHCPOFFER packet in response.

Essentially the docker0 network is analagous to your home router. Your container is analagous to your laptop connected to your LAN port, and the dhcp server is analagous to your ISP’s dhcp server. I wouldn’t be able to just set up a port forward for udp 68 in to my laptop and start being able to get leases from the DHCP server.

I’m not sure what it would take to make your DHCP request/response work through a NAT. Using --net=host gets rid of the NAT stuff, but it isn’t the only possible approach.

You could potentially do a docker network create and give it an address range that is routable from your network, and set up a bridge to a physical interface.

[gregbryant]

My theory was that the original docker run, at the top of this thread, sends our container’s udp packets to the real subnet (not just to the docker-internal subnet, as you suggested) – but somehow the response isn’t getting back to the docker container. A bit of …

tcpdump udp

… listening on the docker host, and on another machine on the real subnet, demonstrates this: the dhcp message is getting out to the real network. This is what you would expect: the ports are exposed, the docker daemon is mapping 67, 68 to 6700, 6800, and the iptables rules forward these to 67, 68 on the real network interface. No problem there. The dhcp server even responds based on the IP address of the immediate sender, and so its reply gets back to our host machine!

So why does it still not work?

The dhcp offer response to the discover request contains the docker container’s internal IP address. Our host perhaps doesn’t know where this machine is. The tcpdump is:

IP [dhcp server] > [docker container internal ip address]: 3049 NXDomain* 0/1/0 (108)

This gave me new search keywords, and I found somewhat related docker material on a thread – a link to which I cannot provide, because this forum prevents more than two links per discussion. To quote:

Disabling the docker-proxy by adding --userland-proxy=false to the ExecStart line in /lib/systemd/system/docker.service, and then bouncing docker with

systemctl daemon-reload 
systemctl restart docker
systemctl enable docker

I did this, and check_dhcp still failed, but the NXDomain was replaced, with something equally odd:

IP [dhcp server] > [docker container internal ip address]: 58824 2/0/0 CNAME vs1.nagios.org., A 96.126.126.159 (66)

Now, that looks like a confused dhcp packet to me. It’s probable that someone outside the container is inspecting the packet, noting that the real destination IP and the internal origin IP are different, and so it’s never forwarded to the container’s internal interface.

That’s a wild guess. Again, the authors of the docker daemon would be able to make sense of this behavior.

Next, I will try your suggestion for an appropriately configured docker network.

Thank you!

[programmerq]

So when you do -p 6700:67/udp -p 6800:68/udp, that is for incoming connections only. Those shouldn’t have any bearing on the broadcast packets trying to get from the container to the dhcp server.

It is interesting that the dhcp server is able to see the container’s IP address, and include it in its DHCPOFFER. As for what looks like a DNS response, I have no idea what might be happening.