[networking] How does container reach host?

loreno · August 24, 2019, 12:54pm

I’m trying to understand some networking concepts. Here’s my setup, regarding network interfaces and their IPs.

macbook has:

en0 - 192.168.1.105/24

I’m running 2 containers:
ubuntu with bridge network: (docker run -it ubuntu)

eth0 - 172.17.0.2

ubuntu with host network (docker run -it --network host ubuntu)

eth0 - 192.168.65.3
docker0 - 172.17.0.1

I also looked into the Linux VM that docker is using internally (by invoking screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty):

eth0 - 192.168.65.3
docker0 - 172.17.0.1

My observations:

I can’t ping neither Linux VM nor the containers from my macbook
I can ping macbook’s en0 from the containers and from the Linux VM - how is that possible? I tried doing trace route, for example here’s its output from ubuntu with bridge network container:

root@35a432368f5d:/# traceroute 192.168.1.105
traceroute to 192.168.1.105 (192.168.1.105), 30 hops max, 60 byte packets
 1  172.17.0.1 (172.17.0.1)  1.261 ms  1.147 ms  1.116 ms
 2  192.168.1.105 (192.168.1.105)  46.154 ms  46.089 ms  46.056 ms

What is 172.17.0.1? Is it some virtual router that Linux VM is connected to? That would explain how the “signal” is able to reach my MacBook. I guess both the Linux VM and my macbok are connected to this router. The only thing that breaks this theory is th fact that ifconfig on my MacBook does not show any connection starting with 172.17…

I’d be really grateful if someone could explain that to me. I tried drawing some schematics to have a better look at it, but I cannot figure this out.

meyay · August 24, 2019, 1:29pm

On mac and Windows, Docker is run in a vm. By default the vm is nat’ed, additionaly the bridged network itself actly like a nat .

Usualy home lan setups have some sort of internet router that encapsulates the local lan with a specific ip range and translating outgoing calls with a NAT and mapping their responses automaticly to the request origin in your local lan. While requests orginated from the internet require portforwarding to forward incomming traffic to a specific ip in the local lan. Lets asume your mac is in the local lan. your mac can access everything in the internt, but at the same time, no one from the internat can use your local ip to connecto your mac. Correct so far?

Well, the docker world behaves exactly like this. You can not ping an ip behind a NAT, unless there is routing. But here is the thing: it will be either routing or NAT.

loreno · August 24, 2019, 2:12pm

@meyay Thanks for nice analogy. However, where is this docker NAT? I understand that docker uses some hidden Linux VM to operate. However, I don’t see how my host (Mac) communicates with this VM.
When I setup VM in VMWare, for example, I can see that my Mac has a virtual network adapter (vmnet8), which is in the same network as my VMWare VM. Regarding docker’s Linux VM, I don’t see any network interface on my host, that would be in the same network as the VM.
Here’re the interfaces of docker VM:

docker-desktop:~# ifconfig
docker0   Link encap:Ethernet  HWaddr 02:42:A1:2A:15:B4
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          inet6 addr: fe80::42:a1ff:fe2a:15b4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8789 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18511 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:362344 (353.8 KiB)  TX bytes:20028932 (19.1 MiB)

eth0      Link encap:Ethernet  HWaddr 02:50:00:00:00:01
          inet addr:192.168.65.3  Bcast:192.168.65.255  Mask:255.255.255.0
          inet6 addr: fe80::50:ff:fe00:1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:37889 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19733 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:39391525 (37.5 MiB)  TX bytes:1125481 (1.0 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:140 (140.0 B)  TX bytes:140 (140.0 B)

veth98f2cf3 Link encap:Ethernet  HWaddr 32:6F:C1:9B:9B:B6
          inet6 addr: fe80::306f:c1ff:fe9b:9bb6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8789 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18529 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:485390 (474.0 KiB)  TX bytes:20030280 (19.1 MiB)

Here are the interfaces on my host Mac:

marcin@MacBook-Pro-Marcin ~ $ ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
	options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
	inet 127.0.0.1 netmask 0xff000000
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	nd6 options=201<PERFORMNUD,DAD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
XHC20: flags=0<> mtu 0
XHC0: flags=0<> mtu 0
XHC1: flags=0<> mtu 0
VHC128: flags=0<> mtu 0
en5: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether ac:de:48:00:11:22
	inet6 fe80::aede:48ff:fe00:1122%en5 prefixlen 64 scopeid 0x8
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect (100baseTX <full-duplex>)
	status: active
ap1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
	ether f2:18:98:9e:20:2d
	media: autoselect
	status: inactive
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether f0:18:98:9e:20:2d
	inet6 fe80::c2f:6a57:fb71:117b%en0 prefixlen 64 secured scopeid 0xa
	inet 192.168.1.105 netmask 0xffffff00 broadcast 192.168.1.255
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active
p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304
	ether 02:18:98:9e:20:2d
	media: autoselect
	status: inactive
awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1484
	ether 22:d8:08:79:f6:40
	inet6 fe80::20d8:8ff:fe79:f640%awdl0 prefixlen 64 scopeid 0xc
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: active
en1: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=60<TSO4,TSO6>
	ether 4e:00:80:e0:c7:01
	media: autoselect <full-duplex>
	status: inactive
en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=60<TSO4,TSO6>
	ether 4e:00:80:e0:c7:00
	media: autoselect <full-duplex>
	status: inactive
en3: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=60<TSO4,TSO6>
	ether 4e:00:80:e0:c7:05
	media: autoselect <full-duplex>
	status: inactive
en4: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=60<TSO4,TSO6>
	ether 4e:00:80:e0:c7:04
	media: autoselect <full-duplex>
	status: inactive
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=63<RXCSUM,TXCSUM,TSO4,TSO6>
	ether 4e:00:80:e0:c7:01
	Configuration:
		id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
		maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
		root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
		ipfilter disabled flags 0x2
	member: en1 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 13 priority 0 path cost 0
	member: en2 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 14 priority 0 path cost 0
	member: en3 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 15 priority 0 path cost 0
	member: en4 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 16 priority 0 path cost 0
	nd6 options=201<PERFORMNUD,DAD>
	media: <unknown type>
	status: inactive
utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000
	inet6 fe80::61a0:7a4d:b8e:ebca%utun0 prefixlen 64 scopeid 0x12
	nd6 options=201<PERFORMNUD,DAD>
utun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
	inet6 fe80::8255:eaf8:6eda:b752%utun1 prefixlen 64 scopeid 0x13
	nd6 options=201<PERFORMNUD,DAD>
vmnet1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether 00:50:56:c0:00:01
	inet 192.168.198.1 netmask 0xffffff00 broadcast 192.168.198.255
vmnet8: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether 00:50:56:c0:00:08
	inet 172.16.66.1 netmask 0xffffff00 broadcast 172.16.66.255

How is the traffic routed between host and docker’s Linux VM? This is the part that I’m missing

meyay · August 24, 2019, 3:30pm

You might want to read the docs about networking:

loreno · August 24, 2019, 3:36pm

@meyay I’ve gone through this article already while searching for an answer. It doesn’t really answer my question. Is there some “magic” going on between Docker’s Linux VM and host OS? How is the traffic routed between them? They are not in any common network.

meyay · August 24, 2019, 3:44pm

The magic is pretty much a vm running in a NATed enviornment, and portforwading for incomming traffic, controlled by docker. Neither for incomming, nor for outoing traffic routing is involved.

If you publish a port, it is publised in the vm and portfowarding is created from your hosts port to your vms port. There is still no routing involved.

Try with vmware: create a vm with a nat network interface and install the docker engine there. Run your container and publish a port. Will you be able to reach the private ip of the vm from your mac? No you won’t. Will you be able to access the publised port of the container? No you won’t. Once you create port forwarding rules to your vm, you will be able to map specific host ports to specific vm ports…

the docker cli does the same with a xhyve vm on mac, but hiddes all the nast stuff from you.

loreno · August 24, 2019, 4:47pm

@meyay I didn’t fully understand your example. I followed what you said:

I have a VM in VMWare with following network setup I guess, this is the NAT setting.
I installed docker in the VM and run container: docker run -p 8000:80 -d nginx
I check the IP of my VM, which is: 172.16.66.132
I accessed the address 172.16.66.132:8000 from my Mac - it works.

I guess, I misunderstood something? In the setup above, I can access the container, because my VM, although behind NAT, is accessible from my host. My host is connected to the private network of the VM, because VMWare installed the proper network adapter, which is connected to this private network.

meyay · August 24, 2019, 5:00pm

uhm, this is not suppposed to worked with a single vNIC in NAT mode.

I am not sure if I understood you correct: did you add two vNics to your VM? One connected via NAT and one connected to a Host only network? The example only works, if you have a single vNic that works as NAT. At leats on the Windows version of vmware this would have proofen my point.

loreno · August 24, 2019, 5:03pm

@meyay - No, I added just one. Here are the interfaces that I have there:

marcin@edge:~$ ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:82ff:fe2d:df7c  prefixlen 64  scopeid 0x20<link>
        ether 02:42:82:2d:df:7c  txqueuelen 0  (Ethernet)
        RX packets 24  bytes 4018 (4.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 46  bytes 3950 (3.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.66.132  netmask 255.255.255.0  broadcast 172.16.66.255
        inet6 fe80::20c:29ff:fe68:cce4  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:68:cc:e4  txqueuelen 1000  (Ethernet)
        RX packets 186131  bytes 245284819 (245.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 91278  bytes 6311871 (6.3 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 2525  bytes 213119 (213.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2525  bytes 213119 (213.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vetha662cf9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::f41c:32ff:fe55:ab7a  prefixlen 64  scopeid 0x20<link>
        ether f6:1c:32:55:ab:7a  txqueuelen 0  (Ethernet)
        RX packets 24  bytes 4354 (4.3 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 60  bytes 5026 (5.0 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

My host computer has many many interfaces, one of them is this one:

vmnet8: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether 00:50:56:c0:00:08
	inet 172.16.66.1 netmask 0xffffff00 broadcast 172.16.66.255

That’s why connection works - host is connected to the NATed private network on the VM.

meyay · August 24, 2019, 5:11pm

Maybe, I was to quick to suggest vmware to do the test. I am surprised that you host has an ip of the nated interface. This ruins the point I wanted to proof

loreno · August 24, 2019, 5:14pm

@meyay That’s how NATting works - something needs to translate the addresses and know both of the “sides”. My host takes this role by being connected to both the “sides”.
No problem, it was still a nice exercise for me

I still wonder where is this port-forwarder (or rather packer forwarder?) between host and docker VM…

meyay · August 24, 2019, 5:25pm

You are right - this is howing NATting work

Though, I would have expected it to work like it happens on windows (or how it happens with virtualbox) - isolated from the host.

meyay · August 24, 2019, 7:22pm

Now I did my testing with vmware and realized my network configuration there was wildly messed up. You are right vmnet8 is the network supposed to be used for nat. And seems I never paid attention that It actualy makes the host act as the nat gateway. Well, I learned something new today: where vmware desktop place their NAT gateway.

Well, anyway VirtualBox works the way I was describing initialy. It does the nating underneath the hood, which makes your host ip treated same way like a random internet ip is treated by your local network’s nat gateway. In this situation there is no routing and you need to have portforwarding.

Topic		Replies	Views
Macvlan network and host to container connectity General	3	16587	May 21, 2020
Can reach container on MACVLAN from host, but not from rest of network General	1	3522	April 8, 2021
Connection between host and container with own ip General docker	3	1342	January 12, 2020
Ip address is accessible from docker host, but not from docker container running on the host Docker Desktop docker , macos	0	1087	September 23, 2021
Accessing containers inside VM from other hosts on network (macvlan driver) General	1	2917	July 31, 2018

[networking] How does container reach host?

Related topics