Yeah I’ve found that either redeploying the linked container/s or redeploying the HAProxy container fixes the issue.
I thought it might be something to do with auto redeploy but I can replicate the problem with manual redeploy’s of services too.
My HAProxy log doesn’t show any errors, I get the following:
Old instance being terminated
[haproxy-1]2016-05-20T10:31:26.665761971Z INFO:haproxy:Docker Cloud Event: container d630621f-ba5e-4a27-848e-f79c563b7d6f is terminated
Notice the below lines, even though the instance has been terminated the new configuration still contains the backend server:
[haproxy-1]2016-05-20T10:31:28.366169048Z backend SERVICE_ZIONTECH
[haproxy-1]2016-05-20T10:31:28.366172422Z server ZIONTECH_1 10.7.0.2:80 check inter 1500 rise 1 fall 3
New instance coming online
503 Error when navigating to hostname
[haproxy-1]2016-05-20T10:31:28.374777792Z INFO:haproxy:Docker Cloud Event: container 09a3a974-b588-4151-ad28-efb95f9cea55 is running
[haproxy-1]2016-05-20T10:31:29.629915095Z INFO:haproxy:HAProxy configuration remains unchanged
So at this point you can’t navigate to the application, you just get a 503 error, so I manually redeployed the HAProxy service, interestingly the generated configuration is identical:
After Manual Redeploy:
2016-05-20T10:34:08.015827697Z backend SERVICE_ZIONTECH
2016-05-20T10:34:08.015830897Z server ZIONTECH_1 10.7.0.2:80 check inter 1500 rise 1 fall 3
I think the issue might actually be with HAProxy thinking the backend is unreachable. Notice that when the old instance is terminated the generated configuration still actually contains the terminated service, then when the new instance comes online the configuration hasn’t changed since the backend is identical and so HAProxy doesn’t get reloaded. Perhaps reloading HAProxy even when the configuration remains unchanged will be a shortcut around this problem since it will detect that the backend is now reachable again?