Upgrade to beta19 failed to restart container w/ a published port

I just upgraded to RC4 and one of my running containers did not restart. I am running a couple dozen containers, all run with restart=always and around half of them publishing a port to my mac. Only one container failed to restart. So, I tried to start it manually with:

$ docker start proxy-wmail
Error response from daemon: driver failed programming external connectivity on endpoint proxy-wmail (36e49462c4d86f3ce79c5e3514f0df033ae68c37172165a4310aa504a6fcf624): Bind for 0.0.0.0:2587 failed: port is already allocated
Error: failed to start containers: proxy-wmail

I then removed this proxy-wmail container and tried to run it again and got the same error:

$ docker run -d --restart=always --net=wnet -v /Users/kdh/walri.com/pvs/certs:/certs --name proxy-wmail -h proxy-wmail -p 2587:587 walr.io/wmail-proxy
555e88351bbecb72151fd0c03a2d9253565895769966ea50312391c248a19e6d
docker: Error response from daemon: driver failed programming external connectivity on endpoint proxy-wmail (1a706d10f0903a0823962f5a4c0281c5f4b70f0346ddb78ff6c456cff483e9eb): Bind for 0.0.0.0:2587 failed: port is already allocated.

This resulted in a container stuck in the Created state.

So, I removed this failed container and entered the Docker VM to look at the /port directory:

$ ls /port
README                                  tcp:0.0.0.0:9938:tcp:172.18.0.10:9938   tcp:0.0.0.0:9969:tcp:172.18.0.10:9969
tcp:0.0.0.0:1961:tcp:172.18.0.5:1961    tcp:0.0.0.0:9939:tcp:172.18.0.10:9939   tcp:0.0.0.0:9970:tcp:172.18.0.10:9970
tcp:0.0.0.0:2587:tcp:172.18.0.25:587    tcp:0.0.0.0:9940:tcp:172.18.0.10:9940   tcp:0.0.0.0:9971:tcp:172.18.0.10:9971
tcp:0.0.0.0:32768:tcp:172.18.0.4:1961   tcp:0.0.0.0:9941:tcp:172.18.0.10:9941   tcp:0.0.0.0:9972:tcp:172.18.0.10:9972
tcp:0.0.0.0:32769:tcp:172.18.0.6:1961   tcp:0.0.0.0:9942:tcp:172.18.0.10:9942   tcp:0.0.0.0:9973:tcp:172.18.0.10:9973
tcp:0.0.0.0:32770:tcp:172.18.0.2:1961   tcp:0.0.0.0:9943:tcp:172.18.0.10:9943   tcp:0.0.0.0:9974:tcp:172.18.0.10:9974
tcp:0.0.0.0:32771:tcp:172.18.0.12:1961  tcp:0.0.0.0:9944:tcp:172.18.0.10:9944   tcp:0.0.0.0:9975:tcp:172.18.0.10:9975
tcp:0.0.0.0:32772:tcp:172.18.0.22:1961  tcp:0.0.0.0:9945:tcp:172.18.0.10:9945   tcp:0.0.0.0:9976:tcp:172.18.0.10:9976
tcp:0.0.0.0:32773:tcp:172.18.0.15:1961  tcp:0.0.0.0:9946:tcp:172.18.0.10:9946   tcp:0.0.0.0:9977:tcp:172.18.0.10:9977
tcp:0.0.0.0:32774:tcp:172.18.0.25:1961  tcp:0.0.0.0:9947:tcp:172.18.0.10:9947   tcp:0.0.0.0:9978:tcp:172.18.0.10:9978
tcp:0.0.0.0:32775:tcp:172.18.0.20:1961  tcp:0.0.0.0:9948:tcp:172.18.0.10:9948   tcp:0.0.0.0:9979:tcp:172.18.0.10:9979
tcp:0.0.0.0:32776:tcp:172.18.0.17:1961  tcp:0.0.0.0:9949:tcp:172.18.0.10:9949   tcp:0.0.0.0:9980:tcp:172.18.0.10:9980
tcp:0.0.0.0:32777:tcp:172.18.0.23:1961  tcp:0.0.0.0:9950:tcp:172.18.0.10:9950   tcp:0.0.0.0:9981:tcp:172.18.0.10:9981
tcp:0.0.0.0:32778:tcp:172.18.0.24:1961  tcp:0.0.0.0:9951:tcp:172.18.0.10:9951   tcp:0.0.0.0:9982:tcp:172.18.0.10:9982
tcp:0.0.0.0:32779:tcp:172.18.0.11:1961  tcp:0.0.0.0:9952:tcp:172.18.0.10:9952   tcp:0.0.0.0:9983:tcp:172.18.0.10:9983
tcp:0.0.0.0:32784:tcp:172.18.0.6:1961   tcp:0.0.0.0:9953:tcp:172.18.0.10:9953   tcp:0.0.0.0:9984:tcp:172.18.0.10:9984
tcp:0.0.0.0:32790:tcp:172.18.0.12:1961  tcp:0.0.0.0:9954:tcp:172.18.0.10:9954   tcp:0.0.0.0:9985:tcp:172.18.0.10:9985
tcp:0.0.0.0:32792:tcp:172.18.0.14:1961  tcp:0.0.0.0:9955:tcp:172.18.0.10:9955   tcp:0.0.0.0:9986:tcp:172.18.0.10:9986
tcp:0.0.0.0:3587:tcp:172.18.0.7:587     tcp:0.0.0.0:9956:tcp:172.18.0.10:9956   tcp:0.0.0.0:9987:tcp:172.18.0.10:9987
tcp:0.0.0.0:443:tcp:172.18.0.5:443      tcp:0.0.0.0:9957:tcp:172.18.0.10:9957   tcp:0.0.0.0:9988:tcp:172.18.0.10:9988
tcp:0.0.0.0:465:tcp:172.18.0.10:465     tcp:0.0.0.0:9958:tcp:172.18.0.10:9958   tcp:0.0.0.0:9989:tcp:172.18.0.10:9989
tcp:0.0.0.0:587:tcp:172.18.0.10:587     tcp:0.0.0.0:9959:tcp:172.18.0.10:9959   tcp:0.0.0.0:9990:tcp:172.18.0.10:9990
tcp:0.0.0.0:80:tcp:172.18.0.5:80        tcp:0.0.0.0:9960:tcp:172.18.0.10:9960   tcp:0.0.0.0:9991:tcp:172.18.0.10:9991
tcp:0.0.0.0:9930:tcp:172.18.0.10:9930   tcp:0.0.0.0:9961:tcp:172.18.0.10:9961   tcp:0.0.0.0:9992:tcp:172.18.0.10:9992
tcp:0.0.0.0:9931:tcp:172.18.0.10:9931   tcp:0.0.0.0:9962:tcp:172.18.0.10:9962   tcp:0.0.0.0:9993:tcp:172.18.0.10:9993
tcp:0.0.0.0:9932:tcp:172.18.0.10:9932   tcp:0.0.0.0:9963:tcp:172.18.0.10:9963   tcp:0.0.0.0:9994:tcp:172.18.0.10:9994
tcp:0.0.0.0:9933:tcp:172.18.0.10:9933   tcp:0.0.0.0:9964:tcp:172.18.0.10:9964   tcp:0.0.0.0:9995:tcp:172.18.0.10:9995
tcp:0.0.0.0:9934:tcp:172.18.0.10:9934   tcp:0.0.0.0:9965:tcp:172.18.0.10:9965   tcp:0.0.0.0:9996:tcp:172.18.0.10:9996
tcp:0.0.0.0:9935:tcp:172.18.0.10:9935   tcp:0.0.0.0:9966:tcp:172.18.0.10:9966   tcp:0.0.0.0:9997:tcp:172.18.0.10:9997
tcp:0.0.0.0:9936:tcp:172.18.0.10:9936   tcp:0.0.0.0:9967:tcp:172.18.0.10:9967   tcp:0.0.0.0:9998:tcp:172.18.0.10:9998
tcp:0.0.0.0:9937:tcp:172.18.0.10:9937   tcp:0.0.0.0:9968:tcp:172.18.0.10:9968   tcp:0.0.0.0:9999:tcp:172.18.0.10:9999

As you can see there are a lot of ports allocated including the port for tcp:0.0.0.0:2587:tcp:172.18.0.25:587 which is a port that wasn’t cleaned up from before the update to RC4.

I then removed all containers:

$ docker stop $(docker ps -a -q)
$ docker rm -v $(docker ps -a -q)

And checked the /port directory again:

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
$ docker run -it --rm --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh
/ # ls port
README                                  tcp:0.0.0.0:32784:tcp:172.18.0.6:1961   tcp:0.0.0.0:32792:tcp:172.18.0.14:1961
tcp:0.0.0.0:2587:tcp:172.18.0.25:587    tcp:0.0.0.0:32790:tcp:172.18.0.12:1961

Note that some other ports were not removed for other containers that were removed, but many were removed.

I think there is a bug in cleaning up/removing containers that can sometimes result in the port mapping still remaining after the container is finished.

I “reset to factory defaults” and rebuilt my images and all started fine to work-around this bug.