Fatal iptables/host port errors on 'docker run' after updating to beta18

Expected behavior

Running my containers works as it did in the previous version.

Actual behavior

After update to beta18, I re-launched my containers from a script and got this error:

+ docker run -d --restart=always --net=wnet -v /Users/kdh/walri.com/pvs/certs:/certs --name proxy-mail -h proxy-mail -p 9930-9999:9930-9999 -p 465:465 -p 587:587 walr.io/mail-proxy
b99fbdc78cfa5104b59ea8472f95a57f4077f3deda157106a9b9ed3f7f7adb4d
docker: Error response from daemon: driver failed programming external connectivity on endpoint proxy-mail (f0830724d5513700da999f88b9b6552cf72b6099c8c302585f938d0af24be6f8): iptables failed: iptables --wait -t nat -A POSTROUTING -p tcp -s 172.18.0.24 -d 172.18.0.24 --dport 9943 -j MASQUERADE: Fatal error: exception Unix.Unix_error(Unix.ENOTCONN, "open", "/var/log/service-port-opener.log")
 (exit status 2).

I stopped/removed all containers and reran my startup script and the above container worked but the next container started in the script failed to launch:

+ docker run -d --restart=always --net=wnet -v /Users/kdh/walri.com/pvs/certs:/certs --name proxy-wmail -h proxy-wmail -p 2587:587 walr.io/wmail-proxy
cf097a89a90be634622d7cbce2b59f220f0fd2be95cde9367a059e5f8f9b240c
docker: Error response from daemon: driver failed programming external connectivity on endpoint proxy-wmail (a03ff1d4df8be9a681dcb19553bd513f9c6f1075760df42b07359f8ffd8368ce): Bind for 0.0.0.0:2587 failed: port is already allocated.

I then restarted Docker and reran my script but failed on the same proxy-wmail container run with the same error. So, I ‘reset’ Docker (through the Preferences pane) which apparently re-installs Docker on my Mac, rebuilt my images and executed my startup script. Now, the script runs to completion and all containers are running.

Information

OSX 10.11.5
Version 1.12.0-rc3-beta18 (build: 9969)

Steps to reproduce the behavior

Not really reproducible as the first iptables error did not re-occur on the subsequent rerun. The second error about the host port already being allocated was reproducible (on 2 runs), but went away with the full reset of docker. When I did the initial upgrade to beta18, all containers were running before upgrading, but after the beta upgrade, all containers were stopped so I removed them and ran my startup script (which resulted in the initial iptables error above).

Full Reset resulted in success.

Also had this error on Docker for Mac, a few versions after, as above, this issue is intermittent, and restarting docker and resetting the containers seems to fix it. It has happened a few times now though.

I also have a start up script that has around 10 docker containers running on the same network


Version information:
OSX 10.11.4
Version 1.12.0-rc4-beta20 (build: 10404)
32ce3bdb164320603f1c2d1fb2ba32a43477ef3d


Output from Terminal:
docker: Error response from daemon: driver failed programming external connectivity on endpoint DockerContainer01 (afebd431818162f45d8f3702c58c115189440c4cc499e4ed5d32c3b5ea266a04): iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 8083 -j DNAT --to-destination 172.25.10.11:4567 ! -i br-84444c6851c8: Fatal error: exception Unix.Unix_error(Unix.ENOTCONN, "open", "/var/log/service-port-opener.log") (exit status 2). 5eb625f483f5fa234f4178f9abed4d3bd451f82e19b11b5e3723ae882ed41139 docker: Error response from daemon: Mounts denied: closed. ac07f1d1c236373d3f4381e3ce5fde2a3e8a99213f1d5203a5ac05ac2feb040a docker: Error response from daemon: driver failed programming external connectivity on endpoint DockerContainer2 (94a8f76248be68416166d9d24a34867358c7b333b31e311df63c35ae915eb45b): iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 9990 -j DNAT --to-destination 172.25.10.11:9990 ! -i br-84444c6851c8: Fatal error: exception Unix.Unix_error(Unix.ENOTCONN, "open", "/var/log/service-port-opener.log") (exit status 2).


Contents of /var/log/service-port-opener.log in the xhyve VM.

port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; POSTROUTING; -d; 127.0.0.11; -j; DOCKER_POSTROUTING] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; DOCKER_OUTPUT; -d; 127.0.0.11; -p; udp; --dport; 53; -j; DNAT; --to-destination; 127.0.0.11:40305] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; DOCKER_POSTROUTING; -s; 127.0.0.11; -p; udp; --sport; 40305; -j; SNAT; --to-source; :53] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; DOCKER_OUTPUT; -d; 127.0.0.11; -p; tcp; --dport; 53; -j; DNAT; --to-destination; 127.0.0.11:38507] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; DOCKER_POSTROUTING; -s; 127.0.0.11; -p; tcp; --sport; 38507; -j; SNAT; --to-source; :53] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -L; -n] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; --version] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -C; OUTPUT; -d; 127.0.0.11; -j; DOCKER_OUTPUT] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -N; DOCKER_OUTPUT] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; OUTPUT; -d; 127.0.0.11; -j; DOCKER_OUTPUT] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -C; POSTROUTING; -d; 127.0.0.11; -j; DOCKER_POSTROUTING] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -N; DOCKER_POSTROUTING] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; POSTROUTING; -d; 127.0.0.11; -j; DOCKER_POSTROUTING] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; DOCKER_OUTPUT; -d; 127.0.0.11; -p; udp; --dport; 53; -j; DNAT; --to-destination; 127.0.0.11:36364] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; DOCKER_POSTROUTING; -s; 127.0.0.11; -p; udp; --sport; 36364; -j; SNAT; --to-source; :53] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; DOCKER_OUTPUT; -d; 127.0.0.11; -p; tcp; --dport; 53; -j; DNAT; --to-destination; 127.0.0.11:42002] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; DOCKER_POSTROUTING; -s; 127.0.0.11; -p; tcp; --sport; 42002; -j; SNAT; --to-source; :53] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -L; -n] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; --version] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -C; OUTPUT; -d; 127.0.0.11; -j; DOCKER_OUTPUT] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -N; DOCKER_OUTPUT] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; OUTPUT; -d; 127.0.0.11; -j; DOCKER_OUTPUT] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -C; POSTROUTING; -d; 127.0.0.11; -j; DOCKER_POSTROUTING] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -N; DOCKER_POSTROUTING] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; POSTROUTING; -d; 127.0.0.11; -j; DOCKER_POSTROUTING] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; DOCKER_OUTPUT; -d; 127.0.0.11; -p; udp; --dport; 53; -j; DNAT; --to-destination; 127.0.0.11:38029] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; DOCKER_POSTROUTING; -s; 127.0.0.11; -p; udp; --sport; 38029; -j; SNAT; --to-source; :53] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; DOCKER_OUTPUT; -d; 127.0.0.11; -p; tcp; --dport; 53; -j; DNAT; --to-destination; 127.0.0.11:42884] port_forwarding=true intercepted arguments [/usr/local/sbin/iptables; --wait; -t; nat; -I; DOCKER_POSTROUTING; -s; 127.0.0.11; -p; tcp; --sport; 42884; -j; SNAT; --to-source; :53]

`