Dockercloud-haproxy - Linked service redeploy = 503 Error

If we redeploy a service thats attached to our load balancer it will (nearly every time) result in a 503 error when attempting to navigate to the VIRTUAL_HOST until we restart the HAProxy service itself. Sometimes a service will redeploy with no problem, its very random.

An open issue exists on GitHub and has been open since February: Service redeploy results in 503 error · Issue #4 · docker-archive/dockercloud-haproxy · GitHub

I realise the HAProxy image is more of an added benefit than a core part of the DockerCloud service but it would be great if we could get some more activity surrounding this issue. Perhaps its a configuration problem at our end but it seems strange that it ‘sometimes’ works fine.

1 Like

I’m having the same issue, and as you said, it’s random.
I have to redeploy the linked containers some times and then everything is ok.
Take a look to the console. It’s like if some times don’t load properly the virtual hosts, but I don’t know the reason.

Yeah I’ve found that either redeploying the linked container/s or redeploying the HAProxy container fixes the issue.

I thought it might be something to do with auto redeploy but I can replicate the problem with manual redeploy’s of services too.

My HAProxy log doesn’t show any errors, I get the following:

========================
Auto Redeploy:
Old instance being terminated

[haproxy-1]2016-05-20T10:31:26.665761971Z INFO:haproxy:Docker Cloud Event: container d630621f-ba5e-4a27-848e-f79c563b7d6f is terminated

Notice the below lines, even though the instance has been terminated the new configuration still contains the backend server:

[haproxy-1]2016-05-20T10:31:28.366169048Z backend SERVICE_ZIONTECH
[haproxy-1]2016-05-20T10:31:28.366172422Z server ZIONTECH_1 10.7.0.2:80 check inter 1500 rise 1 fall 3

==========================
Auto Redeploy:
New instance coming online
503 Error when navigating to hostname

[haproxy-1]2016-05-20T10:31:28.374777792Z INFO:haproxy:Docker Cloud Event: container 09a3a974-b588-4151-ad28-efb95f9cea55 is running
[haproxy-1]2016-05-20T10:31:29.629915095Z INFO:haproxy:HAProxy configuration remains unchanged

So at this point you can’t navigate to the application, you just get a 503 error, so I manually redeployed the HAProxy service, interestingly the generated configuration is identical:

========================
After Manual Redeploy:

2016-05-20T10:34:08.015827697Z backend SERVICE_ZIONTECH
2016-05-20T10:34:08.015830897Z server ZIONTECH_1 10.7.0.2:80 check inter 1500 rise 1 fall 3

I think the issue might actually be with HAProxy thinking the backend is unreachable. Notice that when the old instance is terminated the generated configuration still actually contains the terminated service, then when the new instance comes online the configuration hasn’t changed since the backend is identical and so HAProxy doesn’t get reloaded. Perhaps reloading HAProxy even when the configuration remains unchanged will be a shortcut around this problem since it will detect that the backend is now reachable again?

From GitHub:
Looks like it has something to do with the overlay network. When you redeploy a service, it takes some time for our network daemon to attach the container to the network, which will cause 503 error for a short period of time.

Currently, we are implementing and testing to use docker plugin for the setup of the overlay network, which means the container will be attached to the overlay network before it gets started. This will help mitigate the 503 issue.

1 Like

Ah, great, nice to see its being looked into.

This is supposedly fixed, though it does require a node upgrade/restart:
https://forums.docker.com/t/docker-cloud-release-notes-09-27-2016/22655