I have a strange issue where containers on one node of my swarm cannot attach to the overlay network. Containers on all the other nodes can attach without problems. The Swarm consists of 5 nodes, 4 running Windows 11 and one running Oracle Linux 8. The problem only occurs on one of the Windows 11 machines. All nodes joined the swarm as managers.
The network was created on the machine that initialized the swarm using the following command
docker network create --driver overlay --attachable YADAMU-NET
I have no idea why this is happening, from what I can tell the configuration of the machine where containers can attached successfully, and the one where they cannot are identical.
On one of the Windows Machines that works (Surface Book)
C:\Development\YADAMU>book
C:\Development\YADAMU>set DOCKER_HOST=ssh://yadamu@surface-book
C:\Development\YADAMU>set COMPOSE_PARAMIKO_SSH=1
C:\Development\YADAMU>set DOCKER_ENGINE=windows
C:\Development\YADAMU>docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Build with BuildKit (Docker Inc., v0.6.3)
compose: Docker Compose (Docker Inc., v2.1.1)
scan: Docker Scan (Docker Inc., 0.9.0)
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 102
Server Version: 20.10.10
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics internal l2bridge l2tunnel nat null overlay private transparent
Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
Swarm: active
NodeID: y321c2y4gpmgzqoqtungznort
Is Manager: true
ClusterID: lz46jnnxhytbqlmpe588l91m0
Managers: 5
Nodes: 5
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.1.191
Manager Addresses:
192.168.1.102:2377
192.168.1.109:2377
192.168.1.191:2377
192.168.1.234:2377
192.168.1.236:2377
Default Isolation: hyperv
Kernel Version: 10.0 22000 (22000.1.amd64fre.co_release.210604-1628)
Operating System: Windows 10 Pro Version 2009 (OS Build 22000.318)
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 15.92GiB
Name: Surface-Book
ID: DL62:IY4O:HBP7:SVMM:GPM2:3TR7:AX62:FZDE:KANM:4FLP:CNFX:4VPN
Docker Root Dir: C:\ProgramData\Docker
Debug Mode: false
Registry: [ **https://index.docker.io/v1/** ](https://index.docker.io/v1/)
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
C:\Development\YADAMU>docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d0912b0ce31f yadamu/regression:latest “powershell c:\YADAM…” 19 hours ago Up 19 hours YADAMU-01
C:\Development\YADAMU>docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
y321c2y4gpmgzqoqtungznort * Surface-Book Ready Active Reachable 20.10.10
p5tfas4520ooxuadvf7ffseip YADAMU-DB5 Ready Active Reachable 20.10.10
blymzs14eit8tg1dtoec66f9f YADAMU-LAP Ready Active Reachable 20.10.10
u0wdlrxitieaxhf5916rg7eep yadamu-db3 Ready Active Leader 20.10.10
kn5eb83hguu5tj5fbfsf1a4s5 yadamu-db4 Ready Active Reachable 20.10.10
C:\Development\YADAMU>docker network ls
NETWORK ID NAME DRIVER SCOPE
wauow2q7bhqe YADAMU-NET overlay swarm
ioolqilt0hg9 ingress overlay swarm
0aa305232ea1 nat nat local
9d5115f91080 none null local
C:\Development\YADAMU>docker run --name CMDLINE -d --entrypoint powershell --network YADAMU-NET yadamu/commandline:latest -command wait-event
f9c9a4607a5cde461f93002bdc94a0162954237f6350c7b8d1841cbcacc5be9a
C:\Development\YADAMU>
As you can see the container was able to start and connect to the nework. However on a second machine (YADAMU-NET) when I execute the docker run, it fails with a Timeout.
C:\Development\YADAMU>lap
C:\Development\YADAMU>set DOCKER_HOST=ssh://yadamu@yadamu-lap
C:\Development\YADAMU>set COMPOSE_PARAMIKO_SSH=1
C:\Development\YADAMU>set DOCKER_ENGINE=windows
C:\Development\YADAMU>docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Build with BuildKit (Docker Inc., v0.6.3)
compose: Docker Compose (Docker Inc., v2.1.1)
scan: Docker Scan (Docker Inc., 0.9.0)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 73
Server Version: 20.10.10
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics internal l2bridge l2tunnel nat null overlay private transparent
Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
Swarm: active
NodeID: blymzs14eit8tg1dtoec66f9f
Is Manager: true
ClusterID: lz46jnnxhytbqlmpe588l91m0
Managers: 5
Nodes: 5
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.1.109
Manager Addresses:
192.168.1.102:2377
192.168.1.109:2377
192.168.1.191:2377
192.168.1.234:2377
192.168.1.236:2377
Default Isolation: hyperv
Kernel Version: 10.0 22000 (22000.1.amd64fre.co_release.210604-1628)
Operating System: Windows 10 Pro Version 2009 (OS Build 22000.318)
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 7.898GiB
Name: YADAMU-LAP
ID: KNYA:JWHF:IME2:MMMK:COP3:YESE:6J6V:R2R3:P6CS:TNHS:AS6K:CV47
Docker Root Dir: C:\ProgramData\Docker
Debug Mode: false
Registry: [ **https://index.docker.io/v1/** ](https://index.docker.io/v1/)
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
C:\Development\YADAMU>docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
C:\Development\YADAMU>docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
y321c2y4gpmgzqoqtungznort Surface-Book Ready Active Reachable 20.10.10
p5tfas4520ooxuadvf7ffseip YADAMU-DB5 Ready Active Reachable 20.10.10
blymzs14eit8tg1dtoec66f9f * YADAMU-LAP Ready Active Reachable 20.10.10
u0wdlrxitieaxhf5916rg7eep yadamu-db3 Ready Active Leader 20.10.10
kn5eb83hguu5tj5fbfsf1a4s5 yadamu-db4 Ready Active Reachable 20.10.10
C:\Development\YADAMU>docker network ls
NETWORK ID NAME DRIVER SCOPE
wauow2q7bhqe YADAMU-NET overlay swarm
ioolqilt0hg9 ingress overlay swarm
faf009c20b2b nat nat local
5fa8dce99454 none null local
C:\Development\YADAMU>docker run --name CMDLINE -d --entrypoint powershell --network YADAMU-NET yadamu/commandline:latest -command wait-event
bdcccf35a353bd891310fa1da4eb3210b7da33aab21e3bb19bed6fa48728b6fc
docker: Error response from daemon: attaching to network failed, make sure your network options are correct and check manager logs: context deadline exceeded.
C:\Development\YADAMU>
Any ideas why this might be. All the machine are connected via Physical Ethernet cables to the same switch. The hosts can all ping each other…
Surface-Book:
Pinging yadamu-db3 [192.168.1.234] with 32 bytes of data:
Reply from 192.168.1.234: bytes=32 time=1ms TTL=128
Reply from 192.168.1.234: bytes=32 time=1ms TTL=128
Reply from 192.168.1.234: bytes=32 time=1ms TTL=128
Reply from 192.168.1.234: bytes=32 time=1ms TTL=128
Ping statistics for 192.168.1.234:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 1ms, Maximum = 1ms, Average = 1ms
C:\Users\yadamu>
Yadamu-Lap:
C:\Users\Mark D Drake>ping -4 yadamu-db3
Pinging yadamu-db3 [192.168.1.234] with 32 bytes of data:
Reply from 192.168.1.234: bytes=32 time<1ms TTL=128
Reply from 192.168.1.234: bytes=32 time<1ms TTL=128
Reply from 192.168.1.234: bytes=32 time=1ms TTL=128
Reply from 192.168.1.234: bytes=32 time=1ms TTL=128
Ping statistics for 192.168.1.234:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 1ms, Average = 0ms
C:\Users\Mark D Drake>
Any pointers much appreciated