Networking issues in beta8: errno 526

I’m experiencing issues reliably starting containers with network = hostnet and native/port-forwarding = true. This is manifesting with an error when starting the container:

ERROR: for boots driver failed programming external connectivity on endpoint boots (f836b9399866a2fbe6b5dc5827f7a428acd69362da4b6a66af05e155dfd9ea0f): Error starting userland proxy: write /port/0.0.0.0:4400/ctl: errno 526

I’m using docker-compose. The docker-compose.yml file has 8 containers with about 20 ports between them, all specified as 5432:5432. This error does not occur when starting containers for the first time after a reboot, but happens reliably after stopping and restarting containers a few times.

It may be worth noting that I cannot resolve docker.local.

I’ve tried using network = nat but it seems to keep reverting to hostnet when I enable port forwarding.

Expected behavior

Containers start as expected.

Actual behavior

Some containers start successfully, others (the same ones, repeatedly) fail to start with an error, ex.:

ERROR: for boots driver failed programming external connectivity on endpoint boots (f836b9399866a2fbe6b5dc5827f7a428acd69362da4b6a66af05e155dfd9ea0f): Error starting userland proxy: write /port/0.0.0.0:4400/ctl: errno 526

Steps to reproduce the behavior

  1. Enable native/port-forwarding
  2. Start the containers with docker-compose create && docker-compse start
  3. Stop the containers with docker-compose down
  4. Repeat steps 3 and 4 until step 3 has one or more containers fail with errno 526.

Information

I’ve tried reinstalling Beta 8, and am using the default beta 8 configuration with the only change being that I’ve turned on native/port-forwarding.

Rebooting fixes the issue, allowing me to start the containers for the first time. Subsequent restarts of the containers fail. pinata restart sometimes fixes the issue.

> pinata list
...
network = hostnet (docker-ipv4=192.168.65.2, host-ipv4=192.168.65.1)
...
native/port-forwarding = true 

Pinata diagnose looks OK:

> pinata diagnose -u
OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.0-beta8.2
Running diagnostic tests:
[OK]      docker-cli
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x
Docker logs are being collected into /tmp/20160421-152415.tar.gz
Most specific failure is: No error was detected
Your unique id is: 04F763D7-6CE6-47A4-A93B-1CF232968064
Please quote this in all correspondence.

docker-compose.yml:

boots:
  image: CENSORED
  ports:
    - "4400:4400"
    - "4700:4700"
    - "4800:4800"
    - "5100:5100"
    - "5200:5200"
    - "5400:5400"
    - "5500:5500"
    - "5600:5600"
    - "5700:5700"
    - "5900:5900"
    - "6100:6100"
    - "6300:6300"
    - "6400:6400"

consul:
  image: CENSORED
  ports:
    - "8500:8500"

elasticsearch:
  image: CENSORED
  ports:
    - "9200:9200"
  volumes:
    - "${DOCKER_VOLUME_ROOT}/elasticsearch:/data"

memcached:
  image: CENSORED
  command: -m 32
  ports:
    - "11211:11211"

postgres:
  image: CENSORED
  ports:
    - "5432:5432"
  volumes:
    - "${DOCKER_VOLUME_ROOT}/postgres:/var/lib/postgresql/data"

redis:
  image: CENSORED
  ports:
    - "6379:6379"
  volumes:
    - "${DOCKER_VOLUME_ROOT}/redis:/data"

riak:
  image: CENSORED
  ports:
    - "8087:8087"
    - "8098:8098"
  volumes:
    - "${DOCKER_VOLUME_ROOT}/riak:/var/lib/riak"

nsqd:
  image: CENSORED
  command: nsqd -broadcast-address=127.0.0.1 -lookupd-tcp-address=127.0.0.1:4160 -max-body-size=163962880 -max-msg-size=40990720 -mem-queue-size=0 -data-path=/var/lib/nsqd
  ports:
    - "4150:4150"
    - "4151:4151"
  volumes:
    - "${DOCKER_VOLUME_ROOT}/nsqd:/var/lib/nsqd"

nsqlookupd:
  image: CENSORED
  command: nsqlookupd -broadcast-address=127.0.0.1
  ports:
    - "4160:4160"
    - "4161:4161"

nsqadmin:
  image: CENSORED
  command: nsqadmin -lookupd-http-address=127.0.0.1:4161
  ports:
    - "4171:4171"

I’m running OSX 10.11.4.

1 Like

Facing the exact same issue, here’s the output from pinata diagnose -u:

OS X: version 10.11.3 (build: 15D21)
Docker.app: version v1.11.0-beta8.2
Running diagnostic tests:
[OK] docker-cli
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x
Docker logs are being collected into /tmp/20160422-102752.tar.gz
Most specific failure is: No error was detected
Your unique id is: A03B1D41-FB4D-45BA-9E0F-D304695BA4EC
Please quote this in all correspondence.

+1 Seeing this as well.

I had one of the ports previously bound on the host OS when I tried to bing up the containers initially but even after shutting off the service that was using the port and destroying the docker network and containers and recreating I get the same kind of errors.

I got the same errorno, when I run mysql with docker-compose
ERROR: for db driver failed programming external connectivity on endpoint dockersummerweb_db_1 (cc3f318c22f1f50063f063997a4fc7fc57499b2a9100fab19e8a66fdc854c715): Error starting userland proxy: write /port/0.0.0.0:3306/ctl: errno 526

I notice the same for a Rails app container. Seems to resolve after an unspecified amount of time. I believe it may be related to native/port-forwarding = true setting.

Same here: I am on beta8 with native/port-forwarding enabled. When I docker run -p port:port some/image stop it with CTL-C and re-run the docker-run command, I get this Error starting userland proxy .. error 526.

I had this issue, too. Turns out, that the mysql port (3306) was already in use (in my case by MAMP).
Stopping the process using this port helped!