We recently did the update to 1.11.1-cs1 on our staging environment, up from the old version 1.9.1-cs2 and now our HAproxy is showing the servers going down when they get redeployed but whenthey come back online HAProxy keeps showing them as offline, and I have to redeploy the HAproxy. This is only happening in our staging environment, and only since we did the update. Any suggestions?
Sounds very similar to the long standing issue with redeploys using this image: https://github.com/docker/dockercloud-haproxy/issues/4
That problem looks like the issue I am having, thank you. Hopefully HAProxy will release a fix.
HAProxy doesn’t appear to be the issue, rather the image that the DockerCloud team have built around HAProxy seems to have trouble with redeploys.
The thinking was that the problem was due to HAProxy detecting that the network wasn’t up (since before the update Weave sometimes wasn’t online by the time the instance declared itself ‘online’ to the DockerCloud API), but we’ve now got that fix (https://github.com/weaveworks/weave/releases/tag/v1.5.2) and the issue is still around.
I have no idea where the issue could be since after a redeploy I’ve verified that web access (via CURL) is possible between the HAProxy service and the linked web services so it must be something to do with HAProxy marking the backend as bad.
Hopefully they’ll track down the problem soon.