All "docker swarm or node" commands fail with "Error response from daemon: 404 page not found" (Cluster on Microsoft Azure)

Hello,

I have deployed a cluster on azure with azure docker swarm template from portal.azure.com.

I have one master VM on azure and 5 nodes (2 started and 3 stopped) on 5 VMSS on azure.

I’m running 6 instances of nginx server (only for spreading test on two nodes).

swarm-agent-D3C8D65D000000 : node #1
swarm-agent-D3C8D65D000001 : node #2

xxx@swarm-master-D3C8D65D-0:~$ docker -H tcp://localhost:2375 ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
45c8ee5774ca nginx “nginx -g 'daemon off” About an hour ago Up About an hour 10.0.0.4:32787->80/tcp, 10.0.0.4:32786->443/tcp swarm-agent-D3C8D65D000000/distracted_keller
ee7a065ae45e nginx “nginx -g 'daemon off” About an hour ago Up About an hour 10.0.0.5:32797->80/tcp, 10.0.0.5:32796->443/tcp swarm-agent-D3C8D65D000001/awesome_lamarr
97fecbbae3ce nginx “nginx -g 'daemon off” 2 days ago Up 2 days 10.0.0.4:32785->80/tcp, 10.0.0.4:32784->443/tcp swarm-agent-D3C8D65D000000/prickly_wright
1b3cb5730e6e nginx “nginx -g 'daemon off” 2 days ago Up 2 days 10.0.0.5:32795->80/tcp, 10.0.0.5:32794->443/tcp swarm-agent-D3C8D65D000001/jovial_galileo
2fec5638a34e nginx “nginx -g 'daemon off” 2 days ago Up 2 days 10.0.0.4:32783->80/tcp, 10.0.0.4:32782->443/tcp swarm-agent-D3C8D65D000000/ecstatic_heisenberg
8277d44b81c2 nginx “nginx -g 'daemon off” 2 days ago Up 2 days 10.0.0.5:32793->80/tcp, 10.0.0.5:32792->443/tcp swarm-agent-D3C8D65D000001/goofy_fermi

Swarm agent is started on master (version 1.1.0) :

xxx@swarm-master-D3C8D65D-0:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0b357adcd51f swarm:1.1.0 “/swarm manage --repl” 5 days ago Up About an hour 0.0.0.0:2375->2375/tcp containers_swarm_1
a56b803be2be progrium/consul “/bin/start -server -” 5 days ago Up About an hour 0.0.0.0:8300-8302->8300-8302/tcp, 0.0.0.0:8400->8400/tcp, 53/tcp, 53/udp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:8301-8302->8301-8302/udp containers_consul_1

Now, I want to stop node #2. To do this I would like to drain node #2.

xxx@swarm-master-D3C8D65D-0:~$ docker -H tcp://localhost:2375 node ls # To list nodes
Error response from daemon: 404 page not found

Or

xxx@swarm-master-D3C8D65D-0:~$ docker -H tcp://localhost:2375 swarm join-token worker
Error response from daemon: 404 page not found

Etc …

All commands “docker swarm or node” fail with : " Error response from daemon: 404 page not found".

Correctly, it’s impossible to drain and remove the swarm agent #2 to stop the VMSS #2.

Do you have any idea to resolve this problem?

Thank you in advance to any one who may be able to give me some ideas and excuse me for my poor English.

Arnaud


System and docker swarm information :

$ docker -H tcp://localhost:2375 info
Containers: 6
Running: 6
Paused: 0
Stopped: 0
Images: 13
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 2
swarm-agent-D3C8D65D000000: 10.0.0.4:2375
└ Status: Healthy
└ Containers: 3
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 3.528 GiB
└ Labels: executiondriver=, kernelversion=3.19.0-65-generic, operatingsystem=Ubuntu 14.04.4 LTS, storagedriver=aufs
└ Error: (none)
└ UpdatedAt: 2016-10-10T07:33:41Z
swarm-agent-D3C8D65D000001: 10.0.0.5:2375
└ Status: Healthy
└ Containers: 3
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 3.528 GiB
└ Labels: executiondriver=, kernelversion=3.19.0-65-generic, operatingsystem=Ubuntu 14.04.4 LTS, storagedriver=aufs
└ Error: (none)
└ UpdatedAt: 2016-10-10T07:34:16Z
Plugins:
Volume:
Network:
Swarm:
NodeID:
Is Manager: false
Node Address:
Security Options:
Kernel Version: 3.19.0-65-generic
Operating System: linux
Architecture: amd64
CPUs: 2
Total Memory: 7.056 GiB
Name: 0b357adcd51f
Docker Root Dir:
Debug Mode (client): false
Debug Mode (server): false
WARNING: No kernel memory limit support

$ docker -H tcp://localhost:2375 version
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:22:43 2016
OS/Arch: linux/amd64

Server:
Version: swarm/1.1.0
API version: 1.22
Go version: go1.5.3
Git commit: a0fd82b
Built: Thu Feb 4 08:55:18 UTC 2016
OS/Arch: linux/amd64

$ uname -a

Linux swarm-master-D3C8D65D-0 3.19.0-65-generic #73~14.04.1-Ubuntu SMP Wed Jun 29 21:05:22 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I got exactly the same issue did you find a way to solve it?