If we redeploy a service thats attached to our load balancer it will (nearly every time) result in a 503 error when attempting to navigate to the VIRTUAL_HOST until we restart the HAProxy service itself. Sometimes a service will redeploy with no problem, its very random.
I realise the HAProxy image is more of an added benefit than a core part of the DockerCloud service but it would be great if we could get some more activity surrounding this issue. Perhaps its a configuration problem at our end but it seems strange that it ‘sometimes’ works fine.
I’m having the same issue, and as you said, it’s random.
I have to redeploy the linked containers some times and then everything is ok.
Take a look to the console. It’s like if some times don’t load properly the virtual hosts, but I don’t know the reason.
Yeah I’ve found that either redeploying the linked container/s or redeploying the HAProxy container fixes the issue.
I thought it might be something to do with auto redeploy but I can replicate the problem with manual redeploy’s of services too.
My HAProxy log doesn’t show any errors, I get the following:
========================
Auto Redeploy:
Old instance being terminated
[haproxy-1]2016-05-20T10:31:26.665761971Z INFO:haproxy:Docker Cloud Event: container d630621f-ba5e-4a27-848e-f79c563b7d6f is terminated
Notice the below lines, even though the instance has been terminated the new configuration still contains the backend server:
[haproxy-1]2016-05-20T10:31:28.366169048Z backend SERVICE_ZIONTECH
[haproxy-1]2016-05-20T10:31:28.366172422Z server ZIONTECH_1 10.7.0.2:80 check inter 1500 rise 1 fall 3
==========================
Auto Redeploy:
New instance coming online
503 Error when navigating to hostname
So at this point you can’t navigate to the application, you just get a 503 error, so I manually redeployed the HAProxy service, interestingly the generated configuration is identical:
========================
After Manual Redeploy:
2016-05-20T10:34:08.015827697Z backend SERVICE_ZIONTECH
2016-05-20T10:34:08.015830897Z server ZIONTECH_1 10.7.0.2:80 check inter 1500 rise 1 fall 3
I think the issue might actually be with HAProxy thinking the backend is unreachable. Notice that when the old instance is terminated the generated configuration still actually contains the terminated service, then when the new instance comes online the configuration hasn’t changed since the backend is identical and so HAProxy doesn’t get reloaded. Perhaps reloading HAProxy even when the configuration remains unchanged will be a shortcut around this problem since it will detect that the backend is now reachable again?
From GitHub:
Looks like it has something to do with the overlay network. When you redeploy a service, it takes some time for our network daemon to attach the container to the network, which will cause 503 error for a short period of time.
Currently, we are implementing and testing to use docker plugin for the setup of the overlay network, which means the container will be attached to the overlay network before it gets started. This will help mitigate the 503 issue.