Running a docker stack deploy
on a Windows Server 2016 VM running on a VMware host running ESXi is very slow and fails to start the containers in a timely manner.
The same setup works correctly on a windows host running Hyper-V
Expected Behaviour
Docker containers start within 30 seconds and do not fail during startup
Observed Behaviour
The docker containers fail multiple times on startup. Dump from the windows event log follows:
Further Information
Message
failed to deactivate service binding for container suredrop_ngix-reverse-proxy.1.3w8hx311d73qrpj871zqfjsvz [node.id=y1at653lonky66q3bkv83nuxp module=node/agent error=No such container: suredrop_ngix-reverse-proxy.1.3w8hx311d73qrpj871zqfjsvz]
fatal task error [module=node/agent/taskmanager node.id=y1at653lonky66q3bkv83nuxp task.id=r3vdvlgyxgkjspijn4ru7ath0 service.id=qisky6g8ctz8uuo9obd5qmev7 error=starting container failed: container dd131874b8be53f66ae7e5f465f934efbded7d54ec6fcb220e3007d3cf7626fb encountered an error during Start: failure in a Windows system call: This operation returned because the timeout period expired. (0x5b4)]
dd131874b8be53f66ae7e5f465f934efbded7d54ec6fcb220e3007d3cf7626fb cleanup: failed to delete container from containerd: no such container
rmServiceBinding deleteServiceInfoFromCluster suredrop_ngix-reverse-proxy 2e0a7d0dfc78a4c005c1bc53a1d0eb0912a1eff1fc9fc80787fad5b5c2972ee6 aborted c.serviceBindings[skey] !ok
deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 2e0a7d0dfc78a4c005c1bc53a1d0eb0912a1eff1fc9fc80787fad5b5c2972ee6 oxombl6m65pnmrwy0yyi4gfax err:cannot delete entry endpoint_table with network id oxombl6m65pnmrwy0yyi4gfax and key 2e0a7d0dfc78a4c005c1bc53a1d0eb0912a1eff1fc9fc80787fad5b5c2972ee6 does not exist or is already being deleted
failed to start container [module=libcontainerd namespace=moby error=container dd131874b8be53f66ae7e5f465f934efbded7d54ec6fcb220e3007d3cf7626fb encountered an error during Start: failure in a Windows system call: This operation returned because the timeout period expired. (0x5b4)]
StartComputeSystem dd131874b8be53f66ae7e5f465f934efbded7d54ec6fcb220e3007d3cf7626fb:: Did not complete within 4m0s. This may indicate a platform issue. If it appears to be making no forward progress, obtain the stacks and see is there is a syscall stuck in the platform API for a significant length of time.
fatal task error [module=node/agent/taskmanager error=starting container failed: container 2aca15e54d8ee1ea748a9eb75d6a0ec10da0c0ec99cf2b77cce490c8c4059032 encountered an error during CreateProcess: failure in a Windows system call: The RPC server is unavailable. (0x6ba) extra info: {"CommandLine":"C:\\ServiceMonitor.exe w3svc","WorkingDirectory":"C:\\","Environment":{"COMPLUS_NGenProtectedProcess_FeatureEnabled":"0","ConnectionString":"Data Source=database-core;Initial Catalog=Core;User Id=sa;Password=gemalto.1234567","CoreEndpoint":"http://core","ROSLYN_COMPILER_LOCATION":"c:\\\\RoslynCompilers\\\\tools","ResourcesURL":"http://resources"},"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]} task.id=908s47o0c12pbxyen6a66gdqg service.id=iqvypbo7amtus1l8ekcvj6zf1 node.id=y1at653lonky66q3bkv83nuxp]
2aca15e54d8ee1ea748a9eb75d6a0ec10da0c0ec99cf2b77cce490c8c4059032 cleanup: failed to delete container from containerd: no such container
rmServiceBinding deleteServiceInfoFromCluster suredrop_azure-ad-sync d73d605e5c85cb8d215ecce0d23d2e7ed5bac5a346a688ed8d35ba2b34ded4dc aborted c.serviceBindings[skey] !ok
deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for d73d605e5c85cb8d215ecce0d23d2e7ed5bac5a346a688ed8d35ba2b34ded4dc oxombl6m65pnmrwy0yyi4gfax err:cannot delete entry endpoint_table with network id oxombl6m65pnmrwy0yyi4gfax and key d73d605e5c85cb8d215ecce0d23d2e7ed5bac5a346a688ed8d35ba2b34ded4dc does not exist or is already being deleted
CreateProcess() failed [module=libcontainerd namespace=moby container=2aca15e54d8ee1ea748a9eb75d6a0ec10da0c0ec99cf2b77cce490c8c4059032 error=container 2aca15e54d8ee1ea748a9eb75d6a0ec10da0c0ec99cf2b77cce490c8c4059032 encountered an error during CreateProcess: failure in a Windows system call: The RPC server is unavailable. (0x6ba) extra info: {"CommandLine":"C:\\ServiceMonitor.exe w3svc","WorkingDirectory":"C:\\","Environment":{"COMPLUS_NGenProtectedProcess_FeatureEnabled":"0","ConnectionString":"Data Source=database-core;Initial Catalog=Core;User Id=sa;Password=gemalto.1234567","CoreEndpoint":"http://core","ROSLYN_COMPILER_LOCATION":"c:\\\\RoslynCompilers\\\\tools","ResourcesURL":"http://resources"},"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}]
fatal task error [module=node/agent/taskmanager task.id=vju9jdm5666orifvgipbq7cpn service.id=oslrym5c3uy4609w3r4i24gad node.id=y1at653lonky66q3bkv83nuxp error=task: non-zero exit (2147500037)]
fatal task error [module=node/agent/taskmanager task.id=qhp5bak54uq4yuizb1pdzu2cx service.id=a8hl4bx4sdut6mcqqy362kyex error=task: non-zero exit (2147500037) node.id=y1at653lonky66q3bkv83nuxp]
dd131874b8be53f66ae7e5f465f934efbded7d54ec6fcb220e3007d3cf7626fb cleanup: failed to delete container from containerd: no such container
2aca15e54d8ee1ea748a9eb75d6a0ec10da0c0ec99cf2b77cce490c8c4059032 cleanup: failed to delete container from containerd: no such container
7c71de51943e30b1b4d1e3082867fb939bbca9f3d78fb68e4f54e2fdc420bf60 cleanup: failed to delete container from containerd: no such container
0c295177e54d8da1fdb185cb3788632d54dff60df2c269627d124612c07c1156 cleanup: failed to delete container from containerd: no such container
dd998b8093e3e2a127af2158c525742c78e1df0bff2999ee6dc12a12cad686d6 cleanup: failed to delete container from containerd: no such container
3f5291d19654402e55c236bd82515e1df10c3c7b15ae5479dfeef33e3bc3afc5 cleanup: failed to delete container from containerd: no such container
0ab7b6b454d092371f8c79a60d2a77c2d508858f78f7cc48e34256c212c50369 cleanup: failed to delete container from containerd: no such container
4338a81e1607761de46a167f6bf38d10a8911e4bcc158bb8be6b256bf9bff44e cleanup: failed to delete container from containerd: no such container
remove task failed [error=error while removing network: unknown network suredrop_webnet id oxombl6m65pnmrwy0yyi4gfax task.id=15uw3rqf9yrtrbm67if37b710 node.id=y1at653lonky66q3bkv83nuxp module=node/agent]
network suredrop_webnet remove failed: error while removing network: unknown network suredrop_webnet id oxombl6m65pnmrwy0yyi4gfax [node.id=y1at653lonky66q3bkv83nuxp module=node/agent]
remove task failed [node.id=y1at653lonky66q3bkv83nuxp module=node/agent error=error while removing network: unknown network suredrop_webnet id oxombl6m65pnmrwy0yyi4gfax task.id=zz47fk2ygzxipmt5y32dcgse1]
network suredrop_webnet remove failed: error while removing network: unknown network suredrop_webnet id oxombl6m65pnmrwy0yyi4gfax [node.id=y1at653lonky66q3bkv83nuxp module=node/agent]
remove task failed [task.id=pzei14ewj0waxffvd4r9y4iz9 node.id=y1at653lonky66q3bkv83nuxp module=node/agent error=error while removing network: unknown network suredrop_webnet id oxombl6m65pnmrwy0yyi4gfax]
network suredrop_webnet remove failed: error while removing network: unknown network suredrop_webnet id oxombl6m65pnmrwy0yyi4gfax [node.id=y1at653lonky66q3bkv83nuxp module=node/agent]
remove task failed [error=error while removing network: unknown network suredrop_webnet id oxombl6m65pnmrwy0yyi4gfax node.id=y1at653lonky66q3bkv83nuxp module=node/agent task.id=bidmqeokksaxmefedg23l0ggp]
network suredrop_webnet remove failed: error while removing network: unknown network suredrop_webnet id oxombl6m65pnmrwy0yyi4gfax [node.id=y1at653lonky66q3bkv83nuxp module=node/agent]
Error (Unable to complete atomic operation, key modified) deleting object [endpoint oxombl6m65pnmrwy0yyi4gfax aebd42ab5f9ba54a903c5deb8963a5b3649fc262d806e7d64cca064973a25cb2], retrying....
failed to deactivate service binding for container suredrop_storage-server-backup.1.pzei14ewj0waxffvd4r9y4iz9 [node.id=y1at653lonky66q3bkv83nuxp module=node/agent error=No such container: suredrop_storage-server-backup.1.pzei14ewj0waxffvd4r9y4iz9]
failed to deactivate service binding for container suredrop_azure-ad-sync.1.bidmqeokksaxmefedg23l0ggp [node.id=y1at653lonky66q3bkv83nuxp module=node/agent error=No such container: suredrop_azure-ad-sync.1.bidmqeokksaxmefedg23l0ggp]
Failed to set datapath keys in driver: not implemented
Failed the node discovery in driver: not implemented
Failed to set datapath keys in driver overlay: not implemented
fatal task error [service.id=iqvypbo7amtus1l8ekcvj6zf1 node.id=y1at653lonky66q3bkv83nuxp module=node/agent/taskmanager task.id=b8n7n0uouzn7a2dvacxwq8mcs error=task: non-zero exit (1073807364)]
fatal task error [node.id=y1at653lonky66q3bkv83nuxp module=node/agent/taskmanager task.id=k7c6lavdj37b4ziijlcvyec83 service.id=j325exuizl9gsk0fhiy9i82ab error=task: non-zero exit (1073807364)]
fatal task error [service.id=p4hu9y5jvt3d46w1cypketkyf node.id=y1at653lonky66q3bkv83nuxp module=node/agent/taskmanager task.id=q4jr2y3cc2isqashs33dg91tr error=task: non-zero exit (1073807364)]
fatal task error [task.id=27c7pjf6cneop2k34s8n849g8 error=task: non-zero exit (1073807364) service.id=qisky6g8ctz8uuo9obd5qmev7 node.id=y1at653lonky66q3bkv83nuxp module=node/agent/taskmanager]
fatal task error [node.id=y1at653lonky66q3bkv83nuxp module=node/agent/taskmanager task.id=2xpsulm41zyslb4kzb5ds0c32 service.id=a8hl4bx4sdut6mcqqy362kyex error=task: non-zero exit (1073807364)]
fatal task error [task.id=qbbfc7sh1sc9wsoy3v49huyho service.id=m1klczgr2h1ipfejm585szcx5 node.id=y1at653lonky66q3bkv83nuxp error=task: non-zero exit (1073807364) module=node/agent/taskmanager]
fatal task error [error=task: non-zero exit (4294967295) service.id=xctoohxlejzgoxto6mueauulp node.id=y1at653lonky66q3bkv83nuxp module=node/agent/taskmanager task.id=culy063zv7nqfol8elloov1dk]
fatal task error [module=node/agent/taskmanager task.id=a440xz7r8ygchd8n2exgwsgy6 service.id=5gy27rv8i3b23xjfd87kfoxnk node.id=y1at653lonky66q3bkv83nuxp error=task: non-zero exit (1073807364)]
fatal task error [service.id=oslrym5c3uy4609w3r4i24gad node.id=y1at653lonky66q3bkv83nuxp module=node/agent/taskmanager task.id=7e4xvbr1ve6xmp524gesl8uks error=task: non-zero exit (1073807364)]
error creating cluster object [module=node node.id=y1at653lonky66q3bkv83nuxp error=name conflicts with an existing object]
Error occurred when creating network insufficient vnis(0) passed to overlay. Windows driver requires VNIs to be prepopulated
Error occurred when creating network insufficient vnis(0) passed to overlay. Windows driver requires VNIs to be prepopulated
driver error deleting endpoint gateway_e7988b2c2106 : HNS failed with error : Element not found.
e7988b2c2106728246c3f148bad4ea86a7fdfa34c4a39bb3f5e56c6762821e5f cleanup: failed to delete container from containerd: no such container
Failed to update datapath keys in driver overlay: not implemented
Failed to update datapath keys in driver overlay: not implemented
Further Information
HOST
CPU Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
1 x Socket
4 x Cores
8 x Logical Processors
16GB RAM
VM
2 x CPUs
8GB RAM
100GB Disk, 57GB Free
COMPOSE FILE
version: "3.3"
services:
# ----------------------------------------------------------------------------------------------------
azure-ad-sync:
# -----------------------------------------
# change these to suite your environment
# -----------------------------------------
environment:
- CoreEndpoint=http://core
- ResourcesURL=http://resources
- ConnectionString=${ConnectionString}
# -----------------------------------------
image: suredrop/azure-ad-sync:${tag}
volumes:
- type: volume
source: azure-ad-sync-shared
target: c:\shared
volume:
nocopy: true
deploy:
replicas: 1
endpoint_mode: dnsrr
restart_policy:
condition: on-failure
networks:
webnet:
# ----------------------------------------------------------------------------------------------------
rest-api:
# -----------------------------------------
# change these to suite your environment
# -----------------------------------------
environment:
- PingValue=pong
- Tag=${tag}
- ConnectionString=${ConnectionString}
- CoreEndpoint=http://core
- ResourcesURL=http://resources
- syslog_enabled=${syslog_enabled}
- syslog_destination=${syslog_destination}
- syslog_port=${syslog_port}
- syslog_app=${syslog_app}
- splunk_enabled=${splunk_enabled}
- splunk_destination=${splunk_destination}
- splunk_token=${splunk_token}
# -----------------------------------------
image: suredrop/rest-api:${tag}
volumes:
- type: volume
source: restapi-shared
target: c:\shared
volume:
nocopy: true
deploy:
replicas: 1
endpoint_mode: dnsrr
restart_policy:
condition: on-failure
networks:
webnet:
# ----------------------------------------------------------------------------------------------------
core:
# -----------------------------------------
# change these to suite your environment
# -----------------------------------------
environment:
- PingValue=pong
- smtpServer=${smtpServer}
- smtpPort=${smtpPort}
- smtpSSLEnabled=${smtpSSLEnabled}
- smtpUsername=${smtpUsername}
- smtpPassword=${smtpPassword}
- ConnectionString=${ConnectionString}
- AccessKeyId=
- SecretAccessKey=
- emailFrom=${emailFrom}
# -----------------------------------------
image: suredrop/core:${tag}
volumes:
- type: volume
source: core-shared
target: c:\shared
volume:
nocopy: true
deploy:
replicas: 1
endpoint_mode: dnsrr
restart_policy:
condition: on-failure
networks:
webnet:
# ----------------------------------------------------------------------------------------------------
resources:
# -----------------------------------------
# change these to suite your environment
# -----------------------------------------
environment:
- PingValue=pong
- ResourcePath=C:\inetpub\wwwroot\Resources
# -----------------------------------------
image: suredrop/resources:${tag}
volumes:
- type: volume
source: resources-shared
target: c:\shared
volume:
nocopy: true
deploy:
replicas: 1
endpoint_mode: dnsrr
restart_policy:
condition: on-failure
networks:
webnet:
# ----------------------------------------------------------------------------------------------------
storage-server:
# -----------------------------------------
# change these to suite your environment
# -----------------------------------------
environment:
- PingValue=pong
- StorageBase=${storage_primary_base}
- StorageUsername=${storage_primary_username}
- StoragePassword=${storage_primary_password}
- S3Endpoint=${storage_primary_S3Endpoint}
- S3Bucket=${storage_primary_S3Bucket}
- AWSKey=${storage_primary_AWSKey}
- AWSSecret=${storage_primary_AWSSecret}
# -----------------------------------------
image: suredrop/${storage_primary_type}:${tag}
volumes:
- type: volume
source: storage-primary-shared
target: c:\shared
volume:
nocopy: true
deploy:
replicas: 1
endpoint_mode: dnsrr
restart_policy:
condition: on-failure
networks:
webnet:
# ----------------------------------------------------------------------------------------------------
storage-server-backup:
# -----------------------------------------
# change these to suite your environment
# -----------------------------------------
environment:
- PingValue=pong
- StorageBase=${storage_backup_base}
- StorageUsername=${storage_backup_username}
- StoragePassword=${storage_backup_password}
- S3Endpoint=${storage_backup_S3Endpoint}
- S3Bucket=${storage_backup_S3Bucket}
- AWSKey=${storage_backup_AWSKey}
- AWSSecret=${storage_backup_AWSSecret}
# -----------------------------------------
image: suredrop/${storage_backup_type}:${tag}
volumes:
- type: volume
source: storage-backup-shared
target: c:\shared
volume:
nocopy: true
deploy:
replicas: 1
endpoint_mode: dnsrr
restart_policy:
condition: on-failure
networks:
webnet:
# ----------------------------------------------------------------------------------------------------
ngix-reverse-proxy:
# -----------------------------------------
# change these to suite your environment
# -----------------------------------------
# -----------------------------------------
image: suredrop/ngix-reverse-proxy:${tag}
volumes:
- type: volume
source: ngix-reverse-proxy-shared
target: c:\shared
volume:
nocopy: true
deploy:
replicas: 1
endpoint_mode: dnsrr
restart_policy:
condition: on-failure
ports:
- target: 443
published: 443
protocol: tcp
mode: host
networks:
webnet:
# ----------------------------------------------------------------------------------------------------
networks:
webnet:
volumes:
restapi-shared:
core-shared:
resources-shared:
storage-primary-shared:
storage-backup-shared:
azure-ad-sync-shared:
ngix-reverse-proxy-shared:
Windows Version
Host Name: RPN-VPPPQ01
OS Name: Microsoft Windows Server 2016 Standard
OS Version: 10.0.14393 N/A Build 14393
OS Manufacturer: Microsoft Corporation
OS Configuration: Member Server
OS Build Type: Multiprocessor Free
Registered Owner: Windows User
Registered Organization:
Product ID: 00377-60000-00000-AA934
Original Install Date: 29-4-2019, 14:32:48
System Boot Time: 1-5-2019, 09:12:36
System Manufacturer: VMware, Inc.
System Model: VMware7,1
System Type: x64-based PC
Processor(s): 4 Processor(s) Installed.
[01]: Intel64 Family 6 Model 60 Stepping 3 GenuineIntel ~3392 Mhz
[02]: Intel64 Family 6 Model 60 Stepping 3 GenuineIntel ~3392 Mhz
[03]: Intel64 Family 6 Model 60 Stepping 3 GenuineIntel ~3392 Mhz
[04]: Intel64 Family 6 Model 60 Stepping 3 GenuineIntel ~3392 Mhz
BIOS Version: VMware, Inc. VMW71.00V.9318676.B64.1807270745, 27-7-2018