Expected behavior
Scaling a service down reduces the number of available docker instances (replicas) that can reply on a port request as accessible through the listener on the ELB
Actual behavior
Scaling a service down make requests to ‘hang’ on the ELB. Scaling the service up again resolves the ‘hanging’ behavior.
Additional Information
Using aws-v1.12.1-beta5 stack.
Using default setup (3 managers, 5 workers, all t2.micro)
Uploaded a custom image with a stand alone “http ping service” on ECR.
Steps to reproduce the behavior
-
Successfully start service with command:
docker service create \ --name [PING_SERVICE_NAME] \ --with-registry-auth \ --publish 8080:8080 \ --replicas 3 \ --constraint 'node.role == worker' \ [ECR_IMAGE_LOCATION] -
From a local host, recurrently execute curl in a script on URL: http://[DefaultDNSTarget]:8080, Successful: 3 different hosts, output continuously, e.g.:
…
16:34:34: *** Host: f076c4a02222 - Time: 14:34:34.973 - LocalIP4: 172.18.0.3 , 10.255.0.11 , 10.255.0.13 , 127.0.0.1 - ExternalIP: 52.18.155.169 ***
16:34:35: *** Host: f076c4a02222 - Time: 14:34:35.034 - LocalIP4: 172.18.0.3 , 10.255.0.11 , 10.255.0.13 , 127.0.0.1 - ExternalIP: 52.18.155.169 ***
16:34:35: *** Host: a5e3b7615bee - Time: 14:34:35.095 - LocalIP4: 172.18.0.3 , 10.255.0.11 , 10.255.0.12 , 127.0.0.1 - ExternalIP: 52.210.36.43 ***
16:34:35: *** Host: a5e3b7615bee - Time: 14:34:35.156 - LocalIP4: 172.18.0.3 , 10.255.0.11 , 10.255.0.12 , 127.0.0.1 - ExternalIP: 52.210.36.43 ***
16:34:35: *** Host: a5e3b7615bee - Time: 14:34:35.229 - LocalIP4: 172.18.0.3 , 10.255.0.11 , 10.255.0.12 , 127.0.0.1 - ExternalIP: 52.210.36.43 ***
16:34:35: *** Host: 67160944c84b - Time: 14:34:35.291 - LocalIP4: 172.18.0.3 , 10.255.0.11 , 10.255.0.14 , 127.0.0.1 - ExternalIP: 52.50.217.199 *** -
Upscaling to 4 replicas, using command:
docker service scale [PING_SERVICE_NAME]=4
Successful: 4 hosts: output similar continuously -
Downscaling to 2 replicas, using command:
docker service scale [PING_SERVICE_NAME]=2
Expected: 2 hosts: output continuously, Actual: using curl timeouts shows 2 hosts are responding, but also 2 requests end in ‘hanging’ curl execution. -
Removing the service (command:
docker service rm [PING_SERVICE_NAME]) and starting over results in more hanging requests and even scaling up doesn’t work anymore.