Load Balancing Failures during replica increase

Docker Engine Swarm
1

Hi to all,

I have create a swarm which has 7 nodes (where one act as leader).

node-00:~ $ docker node ls
ID                           HOSTNAME      STATUS  AVAILABILITY  MANAGER STATUS
2ppyw801xfembiixctuqk86vf    node-05  Ready   Active
5egoipc7j9327ssi19l0uivpm *  node-00  Ready   Active        Leader
5p4s2animcyqhblgb3wu9v5z3    node-01  Ready   Active
6300xgqdrzx1g8dfagc54xi8d    node-03  Ready   Active
87z1wkceg2np04uejr48q8zrj    node-02  Ready   Active
b2uj8vz8q9k7b289ln3ztoznw    node-04  Ready   Active
em2kh1md0hqq4y0ibl2gxz63n    node-06  Ready   Active

Furthermore I have created an overlay network via:

docker network create --driver overlay ovr-net

Now I have created a service via:

docker service create --mode=replicated --network ovr-net --replicas=1 -p 8080:8080 -p 8081:8081 --name xyz abc/name:1.0.0

The service container (xyz) has been created with a Dockerfile which looks like this:

EXPOSE 8080 8081
CMD ["java", "-Xmx32m", "-jar", "/usr/local/service/xyz.jar", "server"]
# The healthcheck
HEALTHCHECK --interval=8s --timeout=3s \
  CMD curl -f http://localhost:8081/healthcheck || exit 1

So now I try to access the service via curl -k -s http://192.168.0.150:8080/uuid where I get a UUID so far so good. (The IP is of the swarm node which acts as leader. I know it shouldn’t matter).

Now I try to increase the number of replicas via:

docker service update --replicas 2 xyz

If I try to access the application via curl (using a script):

#!/bin/bash
nodes=$(seq -f "%05g" 1 100000)
for i in $nodes; do
  echo -n "$i "
  curl -k http://192.168.0.150:8080/uuid
  echo ""
done;

I will get error messages about timeouts which looks like the load balancer will switch to an instance which not yet has been ready for running (health state not yet reached).

curl: (7) Failed to connect to 192.168.0.150 port 8080: Connection refused

I had expected that I will never see those connection refused etc. cause the swarm will transparently handle the increase of replicas for me…

May be I miss a point here? Does someone has an idea what I’m doing wrong?

The whole docker images etc. are running on the following version of Docker:

node$ docker version
Client:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 03:23:36 2016
 OS/Arch:      linux/arm

Server:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 03:23:36 2016
 OS/Arch:      linux/arm

Kind regards
Karl Heinz Marbaise