Expected behavior
DNS resolution within a container should resolve to the same addresses / address set regardless of where you are in the swarm.
Actual behavior
DNS results seem to be split. On different containers / nodes, we end up with a different set of addresses within the swarm. Usually this only shows the addresses of the containers that are contained on the node, instead of the entire swarm.
The Leader’s containers mainly tend to show the entire result set. However, in this particular example, it did not.
This behavior seems to sometimes, not occur at all. However, scaling services up and down dynamically seems to exacerbate the issue.
Additional Information
In this particular case, we’re setup with a swarm that has 3 managers and 3 workers. However, I have experimented locally, and I also get this issue any time another node joins the swarm in general. So as little as 1 manager, and 1 worker has caused me to see this issue.
Here’s the current setup of our AWS swarm
docker node ls
Output:
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
21mxfguzwtolu2j1d2aj8yy09 ip-192-168-34-102.us-west-2.compute.internal Ready Active
2ouwmt4yusc3i3bm7664y4g67 ip-192-168-34-101.us-west-2.compute.internal Ready Active
7skp2j2exwxlj2evgyic76l17 ip-192-168-33-16.us-west-2.compute.internal Ready Active
a9ca38s8jjqapsyqq7hq8xs2d ip-192-168-34-250.us-west-2.compute.internal Ready Active Leader
cqe8ebej4dagx15gj9i32ebrv ip-192-168-33-158.us-west-2.compute.internal Ready Active Reachable
e5qcyskt9um0untan9hoyg69l * ip-192-168-34-251.us-west-2.compute.internal Ready Active Reachable
In this case, to simplify matters, we have two services we’ve created, and that are being run. I’ve configured the desire service to have a mode of DNSRR, in order to avoid VIP issues. This is somewhat necessary, because we do have situations requiring callbacks to specific ip addresses.
docker service inspect desire
Output:
[
{
"ID": "brpeop27lyeuxgmbkpjbxlaxi",
"Version": {
"Index": 547
},
"CreatedAt": "2016-08-31T19:23:55.258811173Z",
"UpdatedAt": "2016-08-31T19:23:55.258811173Z",
"Spec": {
"Name": "desire",
"TaskTemplate": {
"ContainerSpec": {
"Image": "<custom ecr repository>/desire:1.0.0"
},
"Resources": {
"Limits": {},
"Reservations": {}
},
"RestartPolicy": {
"Condition": "any",
"MaxAttempts": 0
},
"Placement": {}
},
"Mode": {
"Replicated": {
"Replicas": 12
}
},
"UpdateConfig": {
"Parallelism": 1,
"FailureAction": "pause"
},
"Networks": [
{
"Target": "ecldlbxrrepztpk8lhct6nszr"
}
],
"EndpointSpec": {
"Mode": "dnsrr"
}
},
"Endpoint": {
"Spec": {}
},
"UpdateStatus": {
"StartedAt": "0001-01-01T00:00:00Z",
"CompletedAt": "0001-01-01T00:00:00Z"
}
}
]
However, I’ve scaled one of the services to 12, in order to highlight the issue, and ensure an even spread across the nodes.
docker service ls
Output:
ID NAME REPLICAS IMAGE COMMAND
bn4w5rfpe2mt awareness 1/1 <custom ecr repository>/awareness:1.0.0
brpeop27lyeu desire 12/12 <custom ecr repository>/desire:1.0.0
In this case, I’m going to be picking a service that exists on the current node, however, to demonstrate the spread, I will show the following results first.
docker service ps desire
Output:
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
6q06y8l2e6gyhd1b8f7gz0ae5 desire.1 <custom ecr repository>/desire:1.0.0 ip-192-168-33-158.us-west-2.compute.internal Running Running 5 minutes ago
b8jbog3047os34l2x4q52it0t desire.2 <custom ecr repository>/desire:1.0.0 ip-192-168-34-251.us-west-2.compute.internal Running Running 5 minutes ago
541u33w2hj5el6tunmwhvny1k desire.3 <custom ecr repository>/desire:1.0.0 ip-192-168-34-102.us-west-2.compute.internal Running Running 5 minutes ago
7vs122a1578vtje9mwlkbtlaz desire.4 <custom ecr repository>/desire:1.0.0 ip-192-168-33-158.us-west-2.compute.internal Running Running 5 minutes ago
7xu05ai16ik1c007wzks8m4qq desire.5 <custom ecr repository>/desire:1.0.0 ip-192-168-34-251.us-west-2.compute.internal Running Running 5 minutes ago
dhhubwb4kxnbxovcubt9v2786 desire.6 <custom ecr repository>/desire:1.0.0 ip-192-168-34-102.us-west-2.compute.internal Running Running 5 minutes ago
0sqdqyqvtjwfl0np5o91nnwqz desire.7 <custom ecr repository>/desire:1.0.0 ip-192-168-34-250.us-west-2.compute.internal Running Running 5 minutes ago
21brjwzuk3tcq0io0z99b7cds desire.8 <custom ecr repository>/desire:1.0.0 ip-192-168-34-101.us-west-2.compute.internal Running Running 5 minutes ago
6cbl1cfj7wntleg4kf0nuatm9 desire.9 <custom ecr repository>/desire:1.0.0 ip-192-168-34-101.us-west-2.compute.internal Running Running 5 minutes ago
db0h0xljv05mh8jw0w1v42qnu desire.10 <custom ecr repository>/desire:1.0.0 ip-192-168-33-16.us-west-2.compute.internal Running Running 5 minutes ago
borwe5p4jceyngr9d07k0fxcx desire.11 <custom ecr repository>/desire:1.0.0 ip-192-168-34-250.us-west-2.compute.internal Running Running 5 minutes ago
0wj1z9dqwo61079k9zn3pr9gp desire.12 <custom ecr repository>/desire:1.0.0 ip-192-168-33-16.us-west-2.compute.internal Running Running 5 minutes ago
Now I will pick a single container from the node I am currently on, in order to demonstrate the results.
docker ps
Output:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d2f70f3ffbd0 <custom ecr repository>/desire:1.0.0 "/usr/sbin/desire" 8 minutes ago Up 8 minutes 80/tcp desire.5.7xu05ai16ik1c007wzks8m4qq
3d3fd83909a1 <custom ecr repository>/desire:1.0.0 "/usr/sbin/desire" 8 minutes ago Up 8 minutes 80/tcp desire.2.b8jbog3047os34l2x4q52it0t
612d960d97e4 docker4x/controller:aws-v1.12.1-beta5 "loadbalancer run --l" 19 hours ago Up 19 hours editions_controller
9afba9b67a42 docker4x/shell-aws:aws-v1.12.1-beta5 "/entry.sh /usr/sbin/" 19 hours ago Up 19 hours 0.0.0.0:22->22/tcp modest_thompson
636401cb0a50 docker4x/guide-aws:aws-v1.12.1-beta5 "/entry.sh" 19 hours ago Up 19 hours prickly_almeida
In this case, d2f70f3ffbd0 looks good enough for my tastes. So let’s exec into it with shell.
Command:
docker exec -it d2f70f3ffbd0 sh
/ #
now, a simple nslookup to query desire. I am expecting all 12 addresses.
nslookup desire
Output:
nslookup: can't resolve '(null)': Name does not resolve
Name: desire
Address 1: 10.255.0.13 d2f70f3ffbd0
Address 2: 10.255.0.10 desire.2.4muoyz3tyimf1vm8t1cx3fwza.ingress
Address 3: 10.255.0.11 desire.1.6q06y8l2e6gyhd1b8f7gz0ae5.ingress
Address 4: 10.255.0.18 residual.1.7x9yalx33utte9zl3jsryl7bl.ingress # this is interesting, since no other services are up, and it doesn't even have the same name.
Address 5: 10.255.0.19 desire.4.7vs122a1578vtje9mwlkbtlaz.ingress
Compared to results from a container on the leader. First need to ssh there, so I have at this point.
Welcome to Docker!
~ $
Just to double check.
docker node ls
Output:
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
21mxfguzwtolu2j1d2aj8yy09 ip-192-168-34-102.us-west-2.compute.internal Ready Active
2ouwmt4yusc3i3bm7664y4g67 ip-192-168-34-101.us-west-2.compute.internal Ready Active
7skp2j2exwxlj2evgyic76l17 ip-192-168-33-16.us-west-2.compute.internal Ready Active
a9ca38s8jjqapsyqq7hq8xs2d * ip-192-168-34-250.us-west-2.compute.internal Ready Active Leader
cqe8ebej4dagx15gj9i32ebrv ip-192-168-33-158.us-west-2.compute.internal Ready Active Reachable
e5qcyskt9um0untan9hoyg69l ip-192-168-34-251.us-west-2.compute.internal Ready Active Reachable
Time to pick a container again.
docker ps
Output:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8323da3da8b8 <custom ecr repository>/desire:1.0.0 "/usr/sbin/desire" 28 minutes ago Up 28 minutes 80/tcp desire.7.0sqdqyqvtjwfl0np5o91nnwqz
bde4723b51cd <custom ecr repository>/desire:1.0.0 "/usr/sbin/desire" 28 minutes ago Up 28 minutes 80/tcp desire.11.borwe5p4jceyngr9d07k0fxcx
918d8cb49af1 docker4x/controller:aws-v1.12.1-beta5 "loadbalancer run --l" 19 hours ago Up 19 hours editions_controller
b00a0e9c452b docker4x/shell-aws:aws-v1.12.1-beta5 "/entry.sh /usr/sbin/" 19 hours ago Up 19 hours 0.0.0.0:22->22/tcp desperate_stonebraker
f7e0d5ccfd3e docker4x/guide-aws:aws-v1.12.1-beta5 "/entry.sh" 19 hours ago Up 19 hours small_archimedes
8323da3da8b8 looks like a good one.
docker exec -it 8323da3da8b8 sh
/ #
And then, another nslookup attempt
nslookup desire
Output:
nslookup: can't resolve '(null)': Name does not resolve
Name: desire
Address 1: 10.255.0.9 8323da3da8b8
Address 2: 10.255.0.10 desire.2.4muoyz3tyimf1vm8t1cx3fwza.ingress
Address 3: 10.255.0.17 desire.11.borwe5p4jceyngr9d07k0fxcx.ingress
Both of these nodes are getting two different sets of a results. Scaling to 12 in this case may be a little extreme, but this issue is apparent with 1 service with 3 replicas, on 2 different nodes.
Steps to reproduce the behavior
Since the behavior exists in the overlay network in the swarm in general, the following steps work in my local environment to reproduce.
- Initialize a Swarm
- Add another node to the Swarm
- Create a persistent service, and scale it to 3
- Attach to a container in each node, and run
nslookupfor the service name.