Windows server 2016 (version 1607), DOCKER EE -> docker run jams/freezes

Docker for windows EE (running in windows server 2016).

What could be the problem when my docker run command jams/freezes when i try to run a image ?
I just cant figure out what the problem is.
Also if you know how to clean up the docker installation so that i can start from scratch -> it could be nice :slight_smile:
I have tried to install docker in many different ways when it always behaved in this way that run command jams/hangs/freezes

So installation is done into a windows server 2016 VM (versio 1607) and in that VM i try to run the commands. Am i doing this wrong or should i always run commands from a client computer, eg. windows 10 ?
So basically what i want to know is dockerd.exe the docker host engine and is docker.exe the client and if so, should i be able to start containers from withing the windows server 2016 VM ?

Expected behavior

Should probably start some kind of dotnet example website or a powershell cmd if that kind of application is tried to be runned.

Actual behavior

Container does not start or get running !?

PS C:\docker\DockerEE> Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
PS C:\docker\DockerEE> Install-Package -Name docker -ProviderName DockerMsftProvider

Restart remote pc (windows server 2016 VM)

PS C:\docker\DockerEE> docker run microsoft/dotnet-samples
Unable to find image ‘microsoft/dotnet-samples:latest’ locally
latest: Pulling from microsoft/dotnet-samples
bce2fbc256ea: Already exists
cb1aafb71473: Pull complete
9a610418c539: Pull complete
02f6d73c6635: Pull complete
d45a0c8fbf08: Pull complete
bfbd165ab478: Pull complete
9c391b35df9d: Pull complete
9cc393d354f1: Pull complete
a7ccb79ff00a: Pull complete
095a6fbea33b: Pull complete
dd4d8b87b1ed: Pull complete
Digest: sha256:abad29ef2d1d9f2531c590175b07205eee08c08899260020818ef5aede3baede
Status: Downloaded newer image for microsoft/dotnet-samples:latest
--------------------------------------------
Freezes / jams here “forever”
--------------------------------------------

DOCKERD --DEBUG command shows:
… (plenty of data before this)
time=“2018-03-01T14:48:38.118426000+02:00” level=debug msg="Registering POST, /plugins/pull"
time=“2018-03-01T14:48:38.121427300+02:00” level=debug msg="Registering POST, /plugins/{name:.}/push"
time=“2018-03-01T14:48:38.123427300+02:00” level=debug msg="Registering POST, /plugins/{name:.
}/upgrade"
time=“2018-03-01T14:48:38.124427600+02:00” level=debug msg="Registering POST, /plugins/{name:.}/set"
time=“2018-03-01T14:48:38.154427700+02:00” level=debug msg="Registering POST, /plugins/create"
time=“2018-03-01T14:48:38.167429400+02:00” level=debug msg="Registering GET, /distribution/{name:.
}/json"
time=“2018-03-01T14:48:38.169429900+02:00” level=debug msg="Registering GET, /networks"
time=“2018-03-01T14:48:38.170429700+02:00” level=debug msg="Registering GET, /networks/"
time=“2018-03-01T14:48:38.171429500+02:00” level=debug msg="Registering GET, /networks/{id:.+}"
time=“2018-03-01T14:48:38.172429800+02:00” level=debug msg="Registering POST, /networks/create"
time=“2018-03-01T14:48:38.174429800+02:00” level=debug msg="Registering POST, /networks/{id:.}/connect"
time=“2018-03-01T14:48:38.177430200+02:00” level=debug msg="Registering POST, /networks/{id:.
}/disconnect"
time=“2018-03-01T14:48:38.178430000+02:00” level=debug msg="Registering POST, /networks/prune"
time=“2018-03-01T14:48:38.179431700+02:00” level=debug msg="Registering DELETE, /networks/{id:.*}"
time=“2018-03-01T14:48:38.181431300+02:00” level=info msg="API listen on //./pipe/docker_engine"
time=“2018-03-01T14:48:38.182429200+02:00” level=debug msg="Config reload - waiting signal at Global\docker-daemon-config-15748"
time=“2018-03-01T14:48:47.788045900+02:00” level=info msg="Processing signal ‘interrupt’"
time=“2018-03-01T14:48:47.789037800+02:00” level=debug msg="start clean shutdown of all containers with a 15 seconds timeout…"
time=“2018-03-01T14:48:47.793039800+02:00” level=debug msg=“Clean shutdown succeeded”

C:\Windows\system32>

dockerd jams at the bolded line listening for API calls or what ever.

Information

Same issue with each thing i try to run. eg.
docker run microsoft/nanoserver

PS C:\docker\DockerEE> docker system info
Containers: 25
Running: 0
Paused: 0
Stopped: 25
Images: 4
Server Version: 17.06.2-ee-6
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: l2bridge l2tunnel nat null overlay transparent
Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.2007.amd64fre.rs1_release.171231-1800)
Operating System: Windows Server 2016 Standard
OSType: windows
Architecture: x86_64
CPUs: 2
Total Memory: 16GiB
Name: ***
ID: ***
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Username: trimblesystestdocker
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

PS C:\docker\DockerEE> docker version
Client:
Version: 17.06.2-ee-6
API version: 1.30
Go version: go1.8.3
Git commit: e75fdb8
Built: Mon Nov 27 22:46:09 2017
OS/Arch: windows/amd64

Server:
Version: 17.06.2-ee-6
API version: 1.30 (minimum version 1.24)
Go version: go1.8.3
Git commit: e75fdb8
Built: Mon Nov 27 22:55:16 2017
OS/Arch: windows/amd64
Experimental: false

Steps to reproduce the behavior

Reproduced like shown above -> just install docker ee for windows server, restart and type docker run…

Anyone ?

Tips needed :slight_smile:

Additional notes_

If i run dockerd.exe from cmd as admin with --debug flag, i can se docker jamming at command:

DEBU[2018-03-05T22:25:49.508700700+02:00] HCSShim::Container::Start id=***********************************************************

I finally got a Windows 10 machine and i got the containers up and running using docker for windows CE…
BUT, it would be nice if the server EE version would work…

Am facing similar issue… any solution around this issue ?

This is environmental issue. It works fine on EE for me.

PS C:\Windows\System32\WindowsPowerShell\v1.0> docker info | select-string  Server

Server Version: 17.06.2-ee-6
Operating System: Windows Server 2016 Standard
Debug Mode (server): false

My version is 17.06.2-ee-7… will this make issue… ?
PS Z:> docker info | select-string Server

Server Version: 17.06.2-ee-7
Operating System: Windows Server 2016 Standard
Debug Mode (server): false

Works for me on ee-7 too

i just did all debug…Even the Debug-ContainerHost debug says all are fine… Still it is not working for me… any help will be appreciated,

docker just gets hung after docker run command…

PS Z:> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest 7500fb3fd7cf 13 days ago 1.11GB
hello-world nanoserver 7500fb3fd7cf 13 days ago 1.11GB
microsoft/iis latest 6d91070fab75 2 weeks ago 10.8GB
microsoft/windowsservercore latest 8d57c0984795 2 weeks ago 10.5GB
microsoft/nanoserver latest 2bee0eeea2d3 2 weeks ago 1.11GB
microsoft/dotnet-samples dotnetapp-nanoserver e9ae30d43834 4 months ago 1.17GB

PS Z:> docker run -it -d microsoft/dotnet-samples:dotnetapp-nanoserver
c0bbece387279b5c068f5a9fe63fd177b81bdfb3c6755359093712d0b49cd752

No output after this… it gets hung

PS Z:> Invoke-WebRequest https://aka.ms/Debug-ContainerHost.ps1 -UseBasicParsing | Invoke-Expression
Checking for common problems
Container Host OS Product Name: Windows Server 2016 Standard
Container Host OS Build Label: 14393.2125.amd64fre.rs1_release.180301-2139
Describing Windows Version and Prerequisites
[+] Is Windows 10 Anniversary Update or Windows Server 2016 71ms
[+] Has KB3192366, KB3194496, or later installed if running Windows build 14393 30ms
[+] Is not a build with blocking issues 23ms
[+] Has ‘Containers’ feature installed 7.58s
Describing Docker is installed
[+] A Docker service is installed - ‘Docker’ or ‘com.Docker.Service’ 43ms
[+] Service is running 23ms
[+] Docker.exe is in path 2.1s
[+] Docker is registered in the EventLog service 15ms
Describing User has permissions to use Docker daemon
[+] docker.exe should not return access denied 33ms
Describing Windows container settings are correct
[+] Do not have DisableVSmbOplock set to 1 107ms
[+] Do not have zz values set 24ms
[+] Do not have FDVDenyWriteAccess set to 1 27ms
Describing The right container base images are installed
[+] At least one of ‘microsoft/windowsservercore’ or ‘microsoft/nanoserver’ should be installed 113ms
Describing Container network is created
[+] At least one local container network is available 1.76s
[+] At least one NAT, Transparent, or L2Bridge Network exists 104ms
[+] NAT Network’s vSwitch is internal 13ms
[+] A Windows NAT is configured if a Docker NAT network exists 48ms
[+] Specified Network Gateway IP for NAT network is assigned to Host vNIC 49ms
[+] NAT Network’s internal prefix does not overlap with external IP’ 24ms
Showing output from: docker info
Containers: 45
Running: 0
Paused: 0
Stopped: 45
Images: 5
Server Version: 17.06.2-ee-7
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: l2bridge l2tunnel nat null overlay transparent
Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.2125.amd64fre.rs1_release.180301-2139)
Operating System: Windows Server 2016 Standard
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 7.999GiB
Name: xxx
ID: EOZC:TF65:JD7P:S6SN:YPKL:BF4S:ECQE:RKU3:3HC3:5G56:U6WG:R6AV
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: xxxxxxxxxx
Https Proxy: xxxxx
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

Showing output from: docker version
Client:
Version: 17.06.2-ee-7
API version: 1.30
Go version: go1.8.7
Git commit: 925df35
Built: Fri Mar 16 22:29:37 2018
OS/Arch: windows/amd64

Server:
Engine:
Version: 17.06.2-ee-7
API version: 1.30 (minimum version 1.24)
Go version: go1.8.7
Git commit: 925df35
Built: Fri Mar 16 22:39:05 2018
OS/Arch: windows/amd64
Experimental: false

Showing output from: docker network ls
NETWORK ID NAME DRIVER SCOPE
36b0268a67f5 nat nat local
e4a18641f054 none null local

What are you trying to do? You are asking to run container with interactive output and in detached mode at the same time docker run -it -d microsoft/dotnet-samples:dotnetapp-nanoserver

even i tried without any switch same error happens

PS Z:> restart-service docker
WARNING: Waiting for service ‘Docker Engine (docker)’ to stop…
WARNING: Waiting for service ‘Docker Engine (docker)’ to stop…
WARNING: Waiting for service ‘Docker Engine (docker)’ to stop…
WARNING: Waiting for service ‘Docker Engine (docker)’ to stop…
WARNING: Waiting for service ‘Docker Engine (docker)’ to stop…
WARNING: Waiting for service ‘Docker Engine (docker)’ to stop…
WARNING: Waiting for service ‘Docker Engine (docker)’ to stop…
PS Z:> docker container run microsoft/dotnet-samples:dotnetapp-nanoserver

no output after this…

Did you see my previous comment? What are you trying to do asking to run container both in detached mode and interactively at the same time?

I just want to see any one container running even in detached mode or in iterative mode… Am not able to run any container as of now.

Apps seems to be webservice apps and hence you will not see any output when you run it. See description for this repository which tells how to test it

Above container microsoft/dotnet-samples:dotnetapp-nanoserver should start and prints the hello world message, and then exits.

Tried below one which launch a webpage under http://localhost:8000. This also not working.

PS Z:> docker run --rm -p 8000:80 microsoft/dotnet-samples:aspnetapp
Unable to find image ‘microsoft/dotnet-samples:aspnetapp’ locally
aspnetapp: Pulling from microsoft/dotnet-samples
bce2fbc256ea: Already exists
5b557cd80a98: Already exists
274172f3d5fc: Pull complete
1e0ae2b4436b: Pull complete
dce3e0a93430: Pull complete
7c608a861666: Pull complete
79058720e771: Pull complete
1c26954a3446: Pull complete
7ebf72e10ef6: Pull complete
0f3c665ddb2f: Pull complete
b63746d92d06: Pull complete
71b01428a44f: Pull complete
Digest: sha256:fe6ca703cb4f1d7d494e663100048044c76f5be100008c25e810f97f49480629
Status: Downloaded newer image for microsoft/dotnet-samples:aspnetapp

gets hung after this…

The container goes to create status and it gets hung same with all container,

PS Z:> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
17d3654a64d2 microsoft/dotnet-samples:aspnetapp “dotnet aspnetapp.dll” 7 minutes ago Dead gallant_minsky
e25ea877760c microsoft/dotnet-samples:dotnetapp-nanoserver “dotnet dotnetapp.dll” 13 minutes ago Created clever_wilson
11834f764c11 microsoft/dotnet-samples:dotnetapp-nanoserver “dotnet dotnetapp.dll” 18 minutes ago Created nifty_poitras
b500a6d7df63 microsoft/dotnet-samples:dotnetapp-nanoserver “dotnet dotnetapp.dll” 20 minutes ago Created tender_bohr
5234dc9df974 microsoft/dotnet-samples:dotnetapp-nanoserver “dotnet dotnetapp.dll” 20 minutes ago Created zealous_ardinghelli
4f2a1db190ae microsoft/dotnet-samples:dotnetapp-nanoserver “dotnet dotnetapp.dll” 21 minutes ago Created infallible_elion
4e65a74f0a21 microsoft/dotnet-samples:dotnetapp-nanoserver “dotnet dotnetapp.dll” 4 hours ago Created competent_heyrovsky
fe6e4a4a50d5 microsoft/dotnet-samples:dotnetapp-nanoserver “dotnet dotnetapp.dll” 4 hours ago Created infallible_lumiere

once hung then without docker service restart non of the docker commands working…

it’s not hung, it just does not write anything to STDOUT. You shall access URL on port 8000 on your localhost

Yes am doing that… I just issues cmd

docker run --rm -p 8000:80 microsoft/dotnet-samples:aspnetapp

above command gets hung and its not returning back the cursor.

Then opened the browser to check the url http://localhost:8000/. It is down

I opened one more powershell to check the status of the docker since already opened PS got hung…
Issued the command docker ps -a.
No output for this command also. After docker service restart then the docker ps -a command start working… I check the docker daemon logs… it just stop here "HCSShim::Container::Start " when eve i run a container

Logs:
time=“2018-03-27T16:05:06.289816100-05:00” level=debug msg=“HCSShim::CreateContainer id=c0bbece387279b5c068f5a9fe63fd177b81bdfb3c6755359093712d0b49cd752 config={“SystemType”:“Container”,“Name”:“c0bbece387279b5c068f5a9fe63fd177b81bdfb3c6755359093712d0b49cd752”,“Owner”:“docker”,“IsDummy”:false,“VolumePath”:”\\?\Volume{5eab574a-31f2-11e8-9130-0050569d7abe}",“IgnoreFlushesDuringBoot”:true,“LayerFolderPath”:“C:\ProgramData\docker\windowsfilter\c0bbece387279b5c068f5a9fe63fd177b81bdfb3c6755359093712d0b49cd752”,“Layers”:[{“ID”:“01af109d-f13c-5315-9fe6-8ba5d5b42219”,“Path”:“C:\ProgramData\docker\windowsfilter\23adb9e6e8915263ba375b929846b697284d0d62a0f86abb4d9f5cd87d996935”},{“ID”:“c53ffdfd-b42a-5296-b5c0-6ddaa598573f”,“Path”:“C:\ProgramData\docker\windowsfilter\f8e2999ec07b43c1193c02d2b6c2d58190ddf7c8c5c50efbde82b4a9589701c5”},{“ID”:“6479a381-ce48-58f2-b5cc-d5f014820442”,“Path”:“C:\ProgramData\docker\windowsfilter\2e354cc7b7eeca5c2b8e115b9a6f570b542c5fab2d3c0a69ed227f65ddccad3b”},{“ID”:“f163fe2d-f540-5f87-9dad-d2305871706f”,“Path”:“C:\ProgramData\docker\windowsfilter\f99b594f2b9b0182145c83567e46f80d8a3cb097dfc01ce159b30597a6db428c”},{“ID”:“cc74a8b9-19f8-502b-aaf9-6736b2c8e46a”,“Path”:“C:\ProgramData\docker\windowsfilter\b72cc51cfdd7b89b011abe5ee0c1629af43c94f447cf1ab5513cdd5ade728b79”},{“ID”:“ce94355f-5ef1-5da3-bfbd-3f896b290876”,“Path”:“C:\ProgramData\docker\windowsfilter\bf7c5c4837de6c60a39241d52fd669a51e85d933c1a83984cb290b27eea58e07”},{“ID”:“2009e8ed-3d8f-51b9-bcf2-cd3e9260f4ef”,“Path”:“C:\ProgramData\docker\windowsfilter\87788f5b8d3fdabc18a31c30ce04013d3ce17a1efffcc5054e7a4a7e1eb42272”},{“ID”:“726249b6-96e9-56fb-ac9c-e2629e322b16”,“Path”:“C:\ProgramData\docker\windowsfilter\e6da7fa8ab24efc9b009975a751757554c6ff5ef4723b867f9ce521aafd2f20f”},{“ID”:“d7f85105-73aa-55f8-b9fb-2dcf5c2533cc”,“Path”:“C:\ProgramData\docker\windowsfilter\3ac78bc95f758a470ba777b2b7c302c30a923d2b1aa93a97094956ad5bff3b72”},{“ID”:“e2060c9d-e581-5ced-985f-074c654e1b53”,“Path”:“C:\ProgramData\docker\windowsfilter\42c2a2c581a88bfa9a0799523d4a0cfb900f83ef9fe6a911bd05c2359d7b9d77”},{“ID”:“fbcaa82c-4431-5dee-9d10-ca693ecd0b62”,“Path”:“C:\ProgramData\docker\windowsfilter\375051cfdfa541cd576c8309d33a59bd495ffe3ed992f1e3d74e1f1098166e3f”}],“HostName”:“c0bbece38727”,“MappedDirectories”:[],“HvPartition”:false,“EndpointList”:[“4c4942b9-ffc4-4856-926a-7a7ba1fd8aee”],“Servicing”:false,“AllowUnqualifiedDNSQuery”:true}"
time=“2018-03-27T16:05:06.984889200-05:00” level=debug msg=“HCSShim::CreateContainer succeeded id=c0bbece387279b5c068f5a9fe63fd177b81bdfb3c6755359093712d0b49cd752 handle=39975648”
time=“2018-03-27T16:05:06.984889200-05:00” level=debug msg=“libcontainerd: Create() id=c0bbece387279b5c068f5a9fe63fd177b81bdfb3c6755359093712d0b49cd752, Calling start()”
time=“2018-03-27T16:05:06.984889200-05:00” level=debug msg=“libcontainerd: starting container c0bbece387279b5c068f5a9fe63fd177b81bdfb3c6755359093712d0b49cd752”
time=“2018-03-27T16:05:06.984889200-05:00” level=debug msg=“HCSShim::Container::Start id=c0bbece387279b5c068f5a9fe63fd177b81bdfb3c6755359093712d0b49cd752”

Not sure the whole thing works for me straight up with no issues

PS C:\Users\admin> docker run --rm -p 8000:80 microsoft/dotnet-samples:aspnetapp
Unable to find image 'microsoft/dotnet-samples:aspnetapp' locally
aspnetapp: Pulling from microsoft/dotnet-samples
407ada6e90de: Already exists
5ebc0041967d: Already exists
733e1a195b04: Already exists
7cc5cd6886be: Already exists
51cdd78f16a6: Already exists
d10c87bd74b6: Already exists
f9c8e6bc8c06: Pull complete
bce25a786d7c: Pull complete
b962da1b779a: Pull complete
b152f9e2cbe4: Pull complete
Digest: sha256:fe6ca703cb4f1d7d494e663100048044c76f5be100008c25e810f97f49480629
Status: Downloaded newer image for microsoft/dotnet-samples:aspnetapp
Hosting environment: Production
Content root path: C:\app
Now listening on: http://[::]:80
Application started. Press Ctrl+C to shut down.

Docker ps

PS C:\Users\admin> docker ps
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS                  NAMES
7801402f12db        microsoft/dotnet-samples:aspnetapp   "dotnet aspnetapp.dll"   9 seconds ago       Up 3 seconds        0.0.0.0:8000->80/tcp   pedantic_nobel

Will you be able to share your docker info…details so I can cross verify

Containers: 3
 Running: 1
 Paused: 0
 Stopped: 2
Images: 59
Server Version: 18.03.0-ce
Storage Driver: lcow (linux) windowsfilter (windows)
 LCOW: 
 Windows: 
Logging Driver: json-file
Plugins:
 Volume: local
 Network: ics l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd gelf json-file logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 16299 (16299.15.amd64fre.rs3_release.170928-1534)
Operating System: Windows 10 Enterprise
OSType: windows
Architecture: x86_64
CPUs: 8
Total Memory: 63.92GiB
Name: CND70637PKZBook
ID: 3BGZ:RYAG:C7JG:MGHC:KPPR:A667:FJD7:SFGW:JLQ3:T5SC:ZYI4:QPV5
Docker Root Dir: D:\images
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Same works fine on EE too

Containers: 5
 Running: 3
 Paused: 0
 Stopped: 2
Images: 4
Server Version: 17.06.1-ee-1
Storage Driver: windowsfilter
 Windows: 
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: active
 NodeID: bsycxgv8eforeyrl0r2lfe8xi
 Is Manager: false
 Node Address: 10.44.129.133
 Manager Addresses:
  10.44.129.228:2377
  10.44.129.229:2377
  10.44.129.248:2377
Default Isolation: process
Kernel Version: 10.0 14393 (14393.2125.amd64fre.rs1_release.180301-2139)
Operating System: Windows Server 2016 Standard
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 8GiB
Name: DNVAPPCNTNR1
ID: CUKA:RU2K:5TMB:5PRZ:WEKJ:AGC3:GSNG:QUYB:AYK7:GICG:LFUK:LNPQ
Docker Root Dir: E:\images
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false