Beta 9 and localhost connection via net=host

kolis · April 29, 2016, 7:18am

Expected behavior

In native linux docker, and in docker toolbox (virtualbox), and with earlier beta 8 it was possible to connect to other containers using localhost, from a container that has mounted the /var/run/docker.sock socket.

Specifically I am using the consul registrator image GitHub - gliderlabs/registrator: Service registry bridge for Docker with pluggable adapters
and the example command line works in all other docker environments except in beta 9

$ docker run -d \
    --name=registrator \
    --net=host \
    --volume=/var/run/docker.sock:/tmp/docker.sock \
    gliderlabs/registrator:latest \
      consul://localhost:8500

Update:
I now realise that the relevant setting is probably net=host, not the docker.sock mounting. But still, the problem is the same. I am also attaching a reproducable docker-compose.yml

Actual behavior

Can not connect to consul container using localhost adress

Information

the output of:
- pinata diagnose -u on OSX

pinata diagnose -u
OS X: version 10.10.5 (build: 14F27)
Docker.app: version v1.11.0-beta9
Running diagnostic tests:
[OK] docker-cli
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x
Docker logs are being collected into /tmp/20160429-091710.tar.gz
Most specific failure is: No error was detected
Your unique id is: CAC737C7-7929-4413-BE2D-A2E944F6C282
Please quote this in all correspondence.

-  `DockerDebugInfo.ps1` using Powershell on Windows

a reproducible case if this is a bug, Dockerfiles FTW

registrator:
  image: gliderlabs/registrator
  command: -retry-attempts -1 -retry-interval 500 consul://localhost:8500
  net: host
  volumes:
    - "/var/run/docker.sock:/tmp/docker.sock"

consul:
  image: gliderlabs/consul-server
  entrypoint: /bin/consul agent -bootstrap -server -data-dir /var/consul -client 0.0.0.0
  ports:
   - "8400:8400"
   - "8500:8500"
   - "8600:53/udp"

page URL if this is a docs issue or the name of a man page
host distribution and version ( OSX 10.10.x, OSX 10.11.x, Windows, etc )

Steps to reproduce the behavior

Run the embedded docker-compose.yml

the expected output is something like

registrator_1  | 2016/04/29 07:37:45 Starting registrator v7 ...
registrator_1  | 2016/04/29 07:37:45 Using consul adapter: consul://localhost:8500
registrator_1  | 2016/04/29 07:37:45 Connecting to backend (0/-1)
registrator_1  | 2016/04/29 07:37:46 consul: current leader  
registrator_1  | 2016/04/29 07:37:46 Listening for Docker events ...
registrator_1  | 2016/04/29 07:37:46 Syncing services on 2 containers
registrator_1  | 2016/04/29 07:37:46 added: 2c22a7941225 docker:case_consul_1:8500
registrator_1  | 2016/04/29 07:37:46 ignored: 2c22a7941225 port 8302 not published on host
registrator_1  | 2016/04/29 07:37:46 ignored: 2c22a7941225 port 8600 not published on host
registrator_1  | 2016/04/29 07:37:46 ignored: 2c22a7941225 port 8300 not published on host
registrator_1  | 2016/04/29 07:37:46 added: 2c22a7941225 docker:case_consul_1:53:udp
registrator_1  | 2016/04/29 07:37:46 added: 2c22a7941225 docker:case_consul_1:8400
registrator_1  | 2016/04/29 07:37:46 ignored: 2c22a7941225 port 8600 not published on host
registrator_1  | 2016/04/29 07:37:46 ignored: 2c22a7941225 port 8301 not published on host
registrator_1  | 2016/04/29 07:37:46 ignored: 2c22a7941225 port 8302 not published on host
registrator_1  | 2016/04/29 07:37:46 ignored: 2c22a7941225 port 8301 not published on host
registrator_1  | 2016/04/29 07:37:46 ignored: ed047dc191ab no published ports

but when it is not working you get

registrator_1  | 2016/04/29 07:33:53 Connecting to backend (1/-1)
registrator_1  | 2016/04/29 07:33:54 Connecting to backend (2/-1)
registrator_1  | 2016/04/29 07:33:54 Connecting to backend (3/-1)
registrator_1  | 2016/04/29 07:33:55 Connecting to backend (4/-1)
registrator_1  | 2016/04/29 07:33:56 Connecting to backend (5/-1)
registrator_1  | 2016/04/29 07:33:56 Connecting to backend (6/-1)
...

bsander · May 11, 2016, 7:37am

This is still a problem on beta 11.

jstarker · May 17, 2016, 4:28pm

I would just like to also state that we are also experiencing the problem. Especially when running and experimenting with kubernetes locally. Just wanted to see if this is at least being looked at or if there is a way to enable ssh forwarding or connecting to the local VM as a workaround until a better solution is found.

Otherwise, this seems to force developers to use docker-machine again. Great product so far!

crowsnestjosh · May 19, 2016, 8:43pm

Seems to still exist on beta 12.

mcgeeco · May 20, 2016, 2:57pm

Having this as known issue or something would have saved me a lot of time…

therealplato · May 20, 2016, 5:28pm

this is hurting me too, hoping for a fix
ref kubernetes/issues/25520

rpierce99 · May 21, 2016, 6:21pm

This is also a blocker for my use case, subscribing!

frenchben · May 26, 2016, 6:56pm

@kolis @bsander @crowsnestjosh @mcgeeco @rpierce99 think the real blocker here, is the fact that some of the networking magic is not explained/missed.

Looking at the https://github.com/gliderlabs/registrator getting started guide, it becomes relevant that relying on the host to do anything instead of using a first-class network is simply WRONG (IMHO)

Doing this properly looks something like this.

Create a docker network for all your glider stuff

docker network create gliderlabs

Create consul and attach it to that network

docker run -d --name=consul -p 8500:8500 --net=gliderlabs gliderlabs/consul-server -bootstrap

Port mapping here -p 8500:8500 is only to allow the curl ‘test’ to work - it’s not needed for registrator to work, since they’ll talk over the docker network. Anyhow…

$ curl localhost:8500/v1/catalog/services
{"consul":[]}

Once that’s confirmed to be up and running get your registrator up with the proper network settings (not the host network but our gliderlabs network)

docker run -d \
    --name=registrator \
    --net=gliderlabs \
    --volume=/var/run/docker.sock:/tmp/docker.sock \
    gliderlabs/registrator:latest \
      consul://consul:8500

The magic here is that using the docker network and attaching the above containers to the same network, allows containers to talk to each other by name.

Remember how we used --name=consul when we created the consul container?
That’s why we’re using consul:8500 when we create the registrator container.

Finally checking it works:

$ docker logs registrator
2016/05/26 18:47:46 Starting registrator v7 ...
2016/05/26 18:47:46 Using consul adapter: consul://consul:8500
2016/05/26 18:47:46 Connecting to backend (0/0)
2016/05/26 18:47:46 consul: current leader  172.19.0.2:8300
2016/05/26 18:47:46 Listening for Docker events ...
2016/05/26 18:47:46 Syncing services on 2 containers

rpierce99 · May 26, 2016, 11:17pm

Using a network to connect multiple docker containers, and actually being able to talk to the host, are two different things. I am trying to do the latter. In my case we have migrated a web app to docker, but postgres still runs on the host machine, so docker needs to be able to connect to postgres on localhost:5432 to make it work the way it works on linux.

frenchben · May 27, 2016, 2:25am

How did you do this before?

You mentioned having a postgres db - You can export your data, push it to a volume and start a Postgres container with that volume as the source for the data.

rpierce99 · May 27, 2016, 2:59am

I’m confused by your question, the whole topic of this post is that --net=“host” on mac doesn’t work like it does on linux. On linux, when you specify that you want host networking, localhost is the host, or the container, whatever happens to be listening on the specified port. This has never worked for docker mac with kitematic because of the VM, --net=“host” in that situation was the virtual machine itself, not the host. I expected a native docker for mac would make it work like linux does.

dipthegeezer · May 31, 2016, 9:22am

Hi

I’m getting the exact same issue running a single server using the official consul image. I’m using beta13 docker.

docker run --net=host -e ‘CONSUL_LOCAL_CONFIG={“skip_leave_on_interrupt”: true}’ consul agent -server -bind=127.0.0.1 -bootstrap-expect=1 -ui

I can’t access the services on my host via the described ports. For example

curl 127.0.0.1:8500/ui/

I have a linux box and this all works fine on there.

My mac environment

OS X: version 10.11.5 (build: 15F34) Docker.app: version v1.11.1-beta13.1 Running diagnostic tests: [OK] Moby booted [OK] driver.amd64-linux [OK] vmnetd [OK] osxfs [OK] db [OK] slirp [OK] menubar [OK] environment [OK] Docker [OK] VT-x

jlamillan · June 7, 2016, 9:03pm

+1 This post nails the heart of the issue IMO.

Subscribed to this thread since this is a blocker for me.

ricklusive · June 9, 2016, 5:10pm

+1 Huge blocker for many of my devs. Just need to connect from container to host. (eg container connects to postgres running on the host by simply connecting to “localhost:5432”)

snowgirl · June 10, 2016, 12:48am

“Faster and more reliable: no more VirtualBox! The Docker engine is running in an Alpine Linux distribution on top of an xhyve Virtual Machine on Mac OS X or on a Hyper-V VM on Windows, and that VM is managed by the Docker application. You don’t need docker-machine to run Docker for Mac and Windows.”

I’m having difficulty finding any documentation from Docker that states that Docker for Mac is anything other than docker with a different VM…

This is not something that can just be “fixed”. Because Docker is still running on a VM. It is not running “natively”.

In order to make this work as some are naïvely expecting it to work, would require so much instrumentation and hooks that you may as well just start using the -p flag and the NAT setup by Docker on the bridged interface.

Docker was not designed to run with --net=host, it was designed to use Compose’s EXPOSE directive and the -P flag, or the -p flag.

frenchben · June 15, 2016, 5:18am

@ricklusive @jlamillan

Please see my answer: