I have a docker container that hosts a Debian installation. This container is run by a docker engine installed on a Linux VM which is also a Debian. The VM is running under virtual box on an Intel Mac with OSX Monterey.
I have a python script that is run in the container that launches periodically a ping command by executing the ping utility on the OS with a hostname fetched from a database running in another container.
Problem is that for an IP that corresponds to a host that I know is down, the ping command is able to ping it with the correct IP. It sometimes pings and doesn’t ping even in the same command execution.
I tried logging into the command line of the container and do the same ping command and the behaviour is the same.
I have set the container to have --network=host and put my LAN DNS server in resolv.conf, so the names are correctly resolved.
The VM under which the docker engine runs is configured with NAT and I use port forwarding to expose my application that is running in other containers that form part of the group.
I don’t understand why with the correct ip for a given hostname given by my DNS server, the ping command is able to ping a host that is powered off and does so with sequences in which a reply is received and after no reply is received.
One thing that I recognise it may not help is the fact of running docker engine under a VM in the bare metal, instead of running it directly on the Mac, but I am still new to this and it was more straight forward to use a linux vm.
The Mac where the VM is located is on the 192.168.2.x subnet. The machine I am pinging is on this same subnet. My DNS server is 192.168.2.93. I have it configured in /etc/resolv.conf file of the Debian installation of the container.
I do ‘ping hostname.mydomain.net’ and I sporadically get replies on a correctly resolved IP address for that host. If I do that same ping outside of the container in the VM, I see the correct behaviour which is the pings consistently get no reply.
It doesn’t seem likely, but Docker networks can also get IPs from the same IP range. Docker would recognize already used IP ranges on the same machine, but if your virtual machine has a completely different network using NAT to access the internet, Docker would not recognize that so ping could sometimes try to rach the actual VM and sometimes container. With my limited networking skills, I would assume always trying to reach the closer container IP, but it is one guess.
Or I could imagine this behavior when there is another VM or anything in the network with the same IP. If two machines have the same IP, that could cause a similar behavior.
Two unlikely schenarios, I know. You could try tracepath, or traceroute instead of ping. If you are lucky, that could reveal the difference in routing.