@strayerror
Good timing, I just implemented the above the other day and got a nice externalised helper setup for this. I plan to turn it into a module that can be installed and used when needed. My implementation is a bit different to the above, but it behaves exactly the same way as Docker for Windows so i opted to head in that direction.
To summarise the issue, currently Docker for Mac uses hyperkit under the hood to provide a Host Virtual Machine on which the containers a created. However the configuration for this virtual machine is hard coded into the Docker for Mac executable and is not configurable in any way. Why does this matter? Well one part of this configuration is the network settings. How the Host Virtual Machine talks to your physical machine. Currently this has been hardcoded to “virtio-vpnkit”. What driver does is it creates a vpn of sorts (using a unix socket on your physical machine) and bind specific port to your physical machine (like a nat, or exactly a nat?). However this means you cannot directly access the containers via their internal IP as there is no network interface which bridges between the physical machine and the Host virtual machine, thus you cannot create a network route to get to them.
What the above, and my implementation (hack!) do is inject an additional driver argument into the binary which adds a tap interface to the Host Virtual Machine. This then creates a eth1 interface on the Host VM which binds to the /dev/tap1 block device. Whenever the Host VM starts and opens this file descriptor the tap system will create a network interface called tap1. This is the bridge between the Host VM and the physical machine. From here you can assign an IP to the interface on your physical machine and an IP address to the interface inside the Host VM and create a route to you desired container network segment. Thus giving you access to the container via their internal IP address.
Why is this needed? Why not just map to the physical machines ports? Well in some instances ports are not configurable (SDKs have hardcoded ports etc) and if you need to run up a cluster with many of these containers you will run into port conflicts. To avoid port conflicts the solution is to have multiple IPs you can connect to which all listen on the same port. But to do this you need a virtual network where this IP segment will live. Thus the need for the above hacks.
Ideally what Docker for Mac (not hyperkit, as it really is not a hyperkit issue, they provide the arguments to do it) should be doing is allowing the user to configure the drivers attached to the Host Virtual Machine. What shape or form this takes is entirely up to the Docker for Mac team, but looking above and inside all the related issues, something is required by the community.
Now as for your other question “where does the 192.168.65.1 address come from?” I’m assuming you’re talking about the ip address of the eth0 interface inside the Host Virtual Machine right? If so, try and ping that ip from you physical machine (not using docker commands), you will not be able to reach this IP address. The reason for this is because this IP only exists within the Host VM and has not been bridged to your Physical Machine. Because you can’t route to “192.168.65.1” you also can’t route over “192.168.65.1” to get to your container IPs.
In theory you could implement a bridge on your physical machine which connects the unix socket a virtual network interface, but currently I am unaware of any solution which can do this (other than tap which is above and required a different driver to work). Also it is much more complicated than the above and honestly if we have the above this would give power to the user to decide how they want that to work, if for some reason they need to change it.
If something above doesn’t make sense, or you still need further details please let me know.
P.S. This problem doesn’t exist on any other platform, it is purely a Docker for Mac issue. On windows there is a hvint0 interface which is a bridge to the physical machine. And on Linux there is no VM being run in the first place, so you can just create the routes without issues.