Description
Swarm mode routing mesh not working, instead, it is working like using host mode by default.
We were deploying a swarm of 3 masters nodes and 8 worker nodes, each of them in a different instance of a cloud service OpenStack using Terraform and Ansible. The swarm and routing mesh was working perfectly since it stopped working and started working like in a host mode. We didn’t change anything nor done any update or deploy new services. We tried to restart the swarm and re-deploy the swarm and all services, but nothing worked, we couldn’t make it work in routing mesh mode again. So, we decided to destroy all instances and start from scratch (the issue reported below). We did a clean installation of Ubuntu 18.04 LTS and docker as we did before. Then we set 1 master node and 2 workers (this time manually) and deploy one service, but the swarm is still working like in host mode.
The only way to access the services is by the IP address of the node where it is running, otherwise, there is no answer (time out). We tried to access using the IP of the manager or the other worker instances, but it is not possible to access to the service. That is why we supposed that the swarm is using host mode by default instead of the ingress network and routing mesh.
We also tried with different services like Mongo or Cassandra but the behaviour is the same, the swarm looks like working using host mode. You can only access the service by using the instance IP address where the service is running.
Any ideas to how to bypass the host most and go back to the routing mesh?
We want to access to any service only by using the IP address of the manager nodes which are supossed to be in Drain mode.
Steps to reproduce the issue:
- [manager]
sudo docker swarm init --advertise-addr 158.39.201.14 - [worker-0]
sudo docker swarm join --token SWMTKN-1-3np0cy0msnfurecckl4863hkftykuqkgeq998s1hix6jsoiarq-758o52hyma iyzv74w3u1yzltt 158.39.201.14:2377 - [worker-1]
sudo docker swarm join --token SWMTKN-1-3np0cy0msnfurecckl4863hkftykuqkgeq998s1hix6jsoiarq-758o52hyma iyzv74w3u1yzltt 158.39.201.14:2377 - [manger]
sudo docker stack deploy -c docker-compose.yml nh
Describe the results you received:
curl http://[worker-0-ip]:8089/bigdata 200 OK
curl http://[worker-1-ip]:8089/bigdata FAIL TIME OUT
Describe the results you expected:
curl http://[worker-0-ip]:8089/bigdata 200 OK
curl http://[worker-1-ip]:8089/bigdata 200 OK
Additional information you deem important (e.g. issue happens only occasionally):
This issue was not happening 2 days ago and suddently it started happening. We didn’t made any modification nor touch the servers.
docker-compose.yml
version: '3.7'
networks:
news-hunter:
name: &network news-hunter
x-network: &network-base
networks:
- *network
services:
blazegraph:
<<: *network-base
image: lyrasis/blazegraph:2.1.5
ports:
- published: 8089
target: 8080
deploy:
placement:
constraints:
- node.role == worker
IP tables of manager, worker-1 and worker-2 (all are the same): sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-INGRESS all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
DROP all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (2 references)
target prot opt source destination
Chain DOCKER-INGRESS (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp dpt:8089
ACCEPT tcp -- anywhere anywhere state RELATED,ESTABLISHED tcp spt:8089
RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target prot opt source destination
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Manager ports: sudo netstat -tuplen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 101 46731 14980/systemd-resol
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 17752 865/sshd
tcp6 0 0 :::22 :::* LISTEN 0 17757 865/sshd
tcp6 0 0 :::8089 :::* LISTEN 0 306971 24992/dockerd
tcp6 0 0 :::2377 :::* LISTEN 0 301970 24992/dockerd
tcp6 0 0 :::7946 :::* LISTEN 0 301986 24992/dockerd
udp 0 0 127.0.0.53:53 0.0.0.0:* 101 46730 14980/systemd-resol
udp 0 0 158.39.201.14:68 0.0.0.0:* 100 46591 14964/systemd-netwo
udp 0 0 0.0.0.0:4789 0.0.0.0:* 0 302125 -
udp6 0 0 fe80::f816:3eff:fef:546 :::* 100 46586 14964/systemd-netwo
udp6 0 0 :::7946 :::* 0 301987 24992/dockerd
Worker ports: sudo netstat -tuplen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 101 44998 15283/systemd-resol
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 15724 1010/sshd
tcp6 0 0 :::22 :::* LISTEN 0 15726 1010/sshd
tcp6 0 0 :::8089 :::* LISTEN 0 300227 25355/dockerd
tcp6 0 0 :::7946 :::* LISTEN 0 283636 25355/dockerd
udp 0 0 0.0.0.0:4789 0.0.0.0:* 0 285465 -
udp 0 0 127.0.0.53:53 0.0.0.0:* 101 44997 15283/systemd-resol
udp 0 0 158.39.201.15:68 0.0.0.0:* 100 233705 15247/systemd-netwo
udp6 0 0 :::7946 :::* 0 283637 25355/dockerd
udp6 0 0 fe80::f816:3eff:fee:546 :::* 100 48229 15247/systemd-netwo
Services running: sudo docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
m7eha88ff4wm nh_blazegraph replicated 1/1 lyrasis/blazegraph:2.1.5 *:8089->8080/tcp
Stack: sudo docker stack ps nh
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
tqkd9t4i03yt nh_blazegraph.1 lyrasis/blazegraph:2.1.5 nh-worker-0 Running Running 3 hours ago
Output of docker version:
Client: Docker Engine - Community
Version: 19.03.6
API version: 1.40
Go version: go1.12.16
Git commit: 369ce74a3c
Built: Thu Feb 13 01:27:49 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.6
API version: 1.40 (minimum version 1.12)
Go version: go1.12.16
Git commit: 369ce74a3c
Built: Thu Feb 13 01:26:21 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
Output of docker info:
Client:
Debug Mode: false
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: 19.03.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: hpcm67vxrmkm1wvlhfqbjevox
Is Manager: true
ClusterID: gnl96swlf7o3a976oarvjrazt
Managers: 1
Nodes: 3
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 158.39.201.14
Manager Addresses:
158.39.201.14:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-74-generic
Operating System: Ubuntu 18.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.852GiB
Name: nh-manager-0
ID: PHBO:E6UZ:RNJL:5LVU:OZXW:FM5M:GQVW:SCAQ:EEQW:7IIW:GARL:AUHI
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
We are working on a OpenStack IaaS cloud provider.
Out workload can expect more than 1000 http request per minute from external sources and more than 5000 requests between nodes.
Crossposted: