I just upgraded to RC4 and one of my running containers did not restart. I am running a couple dozen containers, all run with restart=always and around half of them publishing a port to my mac. Only one container failed to restart. So, I tried to start it manually with:
$ docker start proxy-wmail
Error response from daemon: driver failed programming external connectivity on endpoint proxy-wmail (36e49462c4d86f3ce79c5e3514f0df033ae68c37172165a4310aa504a6fcf624): Bind for 0.0.0.0:2587 failed: port is already allocated
Error: failed to start containers: proxy-wmail
I then removed this proxy-wmail container and tried to run it again and got the same error:
$ docker run -d --restart=always --net=wnet -v /Users/kdh/walri.com/pvs/certs:/certs --name proxy-wmail -h proxy-wmail -p 2587:587 walr.io/wmail-proxy
555e88351bbecb72151fd0c03a2d9253565895769966ea50312391c248a19e6d
docker: Error response from daemon: driver failed programming external connectivity on endpoint proxy-wmail (1a706d10f0903a0823962f5a4c0281c5f4b70f0346ddb78ff6c456cff483e9eb): Bind for 0.0.0.0:2587 failed: port is already allocated.
This resulted in a container stuck in the Created state.
So, I removed this failed container and entered the Docker VM to look at the /port directory:
$ ls /port
README tcp:0.0.0.0:9938:tcp:172.18.0.10:9938 tcp:0.0.0.0:9969:tcp:172.18.0.10:9969
tcp:0.0.0.0:1961:tcp:172.18.0.5:1961 tcp:0.0.0.0:9939:tcp:172.18.0.10:9939 tcp:0.0.0.0:9970:tcp:172.18.0.10:9970
tcp:0.0.0.0:2587:tcp:172.18.0.25:587 tcp:0.0.0.0:9940:tcp:172.18.0.10:9940 tcp:0.0.0.0:9971:tcp:172.18.0.10:9971
tcp:0.0.0.0:32768:tcp:172.18.0.4:1961 tcp:0.0.0.0:9941:tcp:172.18.0.10:9941 tcp:0.0.0.0:9972:tcp:172.18.0.10:9972
tcp:0.0.0.0:32769:tcp:172.18.0.6:1961 tcp:0.0.0.0:9942:tcp:172.18.0.10:9942 tcp:0.0.0.0:9973:tcp:172.18.0.10:9973
tcp:0.0.0.0:32770:tcp:172.18.0.2:1961 tcp:0.0.0.0:9943:tcp:172.18.0.10:9943 tcp:0.0.0.0:9974:tcp:172.18.0.10:9974
tcp:0.0.0.0:32771:tcp:172.18.0.12:1961 tcp:0.0.0.0:9944:tcp:172.18.0.10:9944 tcp:0.0.0.0:9975:tcp:172.18.0.10:9975
tcp:0.0.0.0:32772:tcp:172.18.0.22:1961 tcp:0.0.0.0:9945:tcp:172.18.0.10:9945 tcp:0.0.0.0:9976:tcp:172.18.0.10:9976
tcp:0.0.0.0:32773:tcp:172.18.0.15:1961 tcp:0.0.0.0:9946:tcp:172.18.0.10:9946 tcp:0.0.0.0:9977:tcp:172.18.0.10:9977
tcp:0.0.0.0:32774:tcp:172.18.0.25:1961 tcp:0.0.0.0:9947:tcp:172.18.0.10:9947 tcp:0.0.0.0:9978:tcp:172.18.0.10:9978
tcp:0.0.0.0:32775:tcp:172.18.0.20:1961 tcp:0.0.0.0:9948:tcp:172.18.0.10:9948 tcp:0.0.0.0:9979:tcp:172.18.0.10:9979
tcp:0.0.0.0:32776:tcp:172.18.0.17:1961 tcp:0.0.0.0:9949:tcp:172.18.0.10:9949 tcp:0.0.0.0:9980:tcp:172.18.0.10:9980
tcp:0.0.0.0:32777:tcp:172.18.0.23:1961 tcp:0.0.0.0:9950:tcp:172.18.0.10:9950 tcp:0.0.0.0:9981:tcp:172.18.0.10:9981
tcp:0.0.0.0:32778:tcp:172.18.0.24:1961 tcp:0.0.0.0:9951:tcp:172.18.0.10:9951 tcp:0.0.0.0:9982:tcp:172.18.0.10:9982
tcp:0.0.0.0:32779:tcp:172.18.0.11:1961 tcp:0.0.0.0:9952:tcp:172.18.0.10:9952 tcp:0.0.0.0:9983:tcp:172.18.0.10:9983
tcp:0.0.0.0:32784:tcp:172.18.0.6:1961 tcp:0.0.0.0:9953:tcp:172.18.0.10:9953 tcp:0.0.0.0:9984:tcp:172.18.0.10:9984
tcp:0.0.0.0:32790:tcp:172.18.0.12:1961 tcp:0.0.0.0:9954:tcp:172.18.0.10:9954 tcp:0.0.0.0:9985:tcp:172.18.0.10:9985
tcp:0.0.0.0:32792:tcp:172.18.0.14:1961 tcp:0.0.0.0:9955:tcp:172.18.0.10:9955 tcp:0.0.0.0:9986:tcp:172.18.0.10:9986
tcp:0.0.0.0:3587:tcp:172.18.0.7:587 tcp:0.0.0.0:9956:tcp:172.18.0.10:9956 tcp:0.0.0.0:9987:tcp:172.18.0.10:9987
tcp:0.0.0.0:443:tcp:172.18.0.5:443 tcp:0.0.0.0:9957:tcp:172.18.0.10:9957 tcp:0.0.0.0:9988:tcp:172.18.0.10:9988
tcp:0.0.0.0:465:tcp:172.18.0.10:465 tcp:0.0.0.0:9958:tcp:172.18.0.10:9958 tcp:0.0.0.0:9989:tcp:172.18.0.10:9989
tcp:0.0.0.0:587:tcp:172.18.0.10:587 tcp:0.0.0.0:9959:tcp:172.18.0.10:9959 tcp:0.0.0.0:9990:tcp:172.18.0.10:9990
tcp:0.0.0.0:80:tcp:172.18.0.5:80 tcp:0.0.0.0:9960:tcp:172.18.0.10:9960 tcp:0.0.0.0:9991:tcp:172.18.0.10:9991
tcp:0.0.0.0:9930:tcp:172.18.0.10:9930 tcp:0.0.0.0:9961:tcp:172.18.0.10:9961 tcp:0.0.0.0:9992:tcp:172.18.0.10:9992
tcp:0.0.0.0:9931:tcp:172.18.0.10:9931 tcp:0.0.0.0:9962:tcp:172.18.0.10:9962 tcp:0.0.0.0:9993:tcp:172.18.0.10:9993
tcp:0.0.0.0:9932:tcp:172.18.0.10:9932 tcp:0.0.0.0:9963:tcp:172.18.0.10:9963 tcp:0.0.0.0:9994:tcp:172.18.0.10:9994
tcp:0.0.0.0:9933:tcp:172.18.0.10:9933 tcp:0.0.0.0:9964:tcp:172.18.0.10:9964 tcp:0.0.0.0:9995:tcp:172.18.0.10:9995
tcp:0.0.0.0:9934:tcp:172.18.0.10:9934 tcp:0.0.0.0:9965:tcp:172.18.0.10:9965 tcp:0.0.0.0:9996:tcp:172.18.0.10:9996
tcp:0.0.0.0:9935:tcp:172.18.0.10:9935 tcp:0.0.0.0:9966:tcp:172.18.0.10:9966 tcp:0.0.0.0:9997:tcp:172.18.0.10:9997
tcp:0.0.0.0:9936:tcp:172.18.0.10:9936 tcp:0.0.0.0:9967:tcp:172.18.0.10:9967 tcp:0.0.0.0:9998:tcp:172.18.0.10:9998
tcp:0.0.0.0:9937:tcp:172.18.0.10:9937 tcp:0.0.0.0:9968:tcp:172.18.0.10:9968 tcp:0.0.0.0:9999:tcp:172.18.0.10:9999
As you can see there are a lot of ports allocated including the port for tcp:0.0.0.0:2587:tcp:172.18.0.25:587 which is a port that wasn’t cleaned up from before the update to RC4.
I then removed all containers:
$ docker stop $(docker ps -a -q)
$ docker rm -v $(docker ps -a -q)
And checked the /port directory again:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ docker run -it --rm --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh
/ # ls port
README tcp:0.0.0.0:32784:tcp:172.18.0.6:1961 tcp:0.0.0.0:32792:tcp:172.18.0.14:1961
tcp:0.0.0.0:2587:tcp:172.18.0.25:587 tcp:0.0.0.0:32790:tcp:172.18.0.12:1961
Note that some other ports were not removed for other containers that were removed, but many were removed.
I think there is a bug in cleaning up/removing containers that can sometimes result in the port mapping still remaining after the container is finished.
I “reset to factory defaults” and rebuilt my images and all started fine to work-around this bug.