Scaling a service down causes ports to 'hang'

General
, ,
1

Expected behavior

Scaling a service down reduces the number of available docker instances (replicas) that can reply on a port request as accessible through the listener on the ELB

Actual behavior

Scaling a service down make requests to ‘hang’ on the ELB. Scaling the service up again resolves the ‘hanging’ behavior.

Additional Information

Using aws-v1.12.1-beta5 stack.
Using default setup (3 managers, 5 workers, all t2.micro)
Uploaded a custom image with a stand alone “http ping service” on ECR.

Steps to reproduce the behavior

  1. Successfully start service with command:
    docker service create \ --name [PING_SERVICE_NAME] \ --with-registry-auth \ --publish 8080:8080 \ --replicas 3 \ --constraint 'node.role == worker' \ [ECR_IMAGE_LOCATION]

  2. From a local host, recurrently execute curl in a script on URL: http://[DefaultDNSTarget]:8080, Successful: 3 different hosts, output continuously, e.g.:
    …
    16:34:34: *** Host: f076c4a02222 - Time: 14:34:34.973 - LocalIP4: 172.18.0.3 , 10.255.0.11 , 10.255.0.13 , 127.0.0.1 - ExternalIP: 52.18.155.169 ***
    16:34:35: *** Host: f076c4a02222 - Time: 14:34:35.034 - LocalIP4: 172.18.0.3 , 10.255.0.11 , 10.255.0.13 , 127.0.0.1 - ExternalIP: 52.18.155.169 ***
    16:34:35: *** Host: a5e3b7615bee - Time: 14:34:35.095 - LocalIP4: 172.18.0.3 , 10.255.0.11 , 10.255.0.12 , 127.0.0.1 - ExternalIP: 52.210.36.43 ***
    16:34:35: *** Host: a5e3b7615bee - Time: 14:34:35.156 - LocalIP4: 172.18.0.3 , 10.255.0.11 , 10.255.0.12 , 127.0.0.1 - ExternalIP: 52.210.36.43 ***
    16:34:35: *** Host: a5e3b7615bee - Time: 14:34:35.229 - LocalIP4: 172.18.0.3 , 10.255.0.11 , 10.255.0.12 , 127.0.0.1 - ExternalIP: 52.210.36.43 ***
    16:34:35: *** Host: 67160944c84b - Time: 14:34:35.291 - LocalIP4: 172.18.0.3 , 10.255.0.11 , 10.255.0.14 , 127.0.0.1 - ExternalIP: 52.50.217.199 ***

  3. Upscaling to 4 replicas, using command:
    docker service scale [PING_SERVICE_NAME]=4
    Successful: 4 hosts: output similar continuously

  4. Downscaling to 2 replicas, using command:
    docker service scale [PING_SERVICE_NAME]=2
    Expected: 2 hosts: output continuously, Actual: using curl timeouts shows 2 hosts are responding, but also 2 requests end in ‘hanging’ curl execution.

  5. Removing the service (command: docker service rm [PING_SERVICE_NAME]) and starting over results in more hanging requests and even scaling up doesn’t work anymore.

2

It looks like the issue is similar as reported here: https://github.com/docker/docker/issues/25698
Especially the following comment marks the behavior after scaling up and down: https://github.com/docker/docker/issues/25698#issuecomment-243245745

UPDATE: 28/9
[…]
There would be an update to Docker for AWS with a 1.12.2-rc1 engine version shortly.
See: https://github.com/docker/docker/issues/25698#issuecomment-250074968
[…]