Docker Community Forums

Share and learn in the Docker community.

Container unresponsive/hang issue


(Srinidhim) #1

Dear Team,

One of the container hanged and became unresponsive with following error message in syslogs, any help in pinpointing the issue would be very much appreciated,

############### sys log error #############################################
Jul 20 12:58:26 B2BTestServer dockerd[1140]: time=“2017-07-20T12:58:26.879111489Z” level=error msg=“containerd: deleting container” error=“exit status 1: “container 5f090b50b3dd7840dd296bd3eede2c4db171b5787944317f0a0e52d71c368361 does not exist\none or more of the container deletions failed\n””

Jul 20 13:30:01 B2BTestServer CRON[42898]: (xxxx) CMD (docker exec xxApp_worker_1 ./artisan omni:key b2b_fwad login >/dev/null 2>&1)
Jul 20 13:30:01 B2BTestServer CRON[42899]: (xxxx) CMD (docker exec xxApp_app_1 ./artisan omni:key b2b_fwad login >/dev/null 2>&1)
Jul 20 13:30:01 B2BTestServer CRON[42901]: (xxxx) CMD (docker exec xxApp_worker_1 ./artisan omni:key b2b_yww login >/dev/null 2>&1)
Jul 20 13:30:01 B2BTestServer CRON[42900]: (xxxx) CMD (docker exec xxApp_app_1 ./artisan omni:key b2b_yww login >/dev/null 2>&1)
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.382668472Z” level=error msg="Error setting up exec command in container xxApp_app_1: Container f743904a3558b5475df8e574e9b27a5ca8b2c7256da59c356f4da1a9bda72e4b is not running"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.382714110Z” level=error msg="Handler for POST /v1.25/containers/xxApp_app_1/exec returned error: Container f743904a3558b5475df8e574e9b27a5ca8b2c7256da59c356f4da1a9bda72e4b is not running"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.392014905Z” level=error msg="Error setting up exec command in container xxApp_app_1: Container f743904a3558b5475df8e574e9b27a5ca8b2c7256da59c356f4da1a9bda72e4b is not running"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.392237089Z” level=error msg="Handler for POST /v1.25/containers/xxApp_app_1/exec returned error: Container f743904a3558b5475df8e574e9b27a5ca8b2c7256da59c356f4da1a9bda72e4b is not running"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.429292849Z” level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused “process_linux.go:83: executing setns process caused \“exit status 15\””\n"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.463663587Z” level=error msg=“Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused “process_linux.go:83: executing setns process caused \“exit status 15\””\n”

Jul 20 13:30:36 B2BTestServer kernel: [262136.359656] aufs au_opts_verify:1597:dockerd[2634]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer kernel: [262136.583654] aufs au_opts_verify:1597:dockerd[2634]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.120768757Z” level=error msg=“containerd: deleting container” error="exit status 1: “container f80c3f7d41bcbb2133036c795068ea2200b8043bce1a043f5fb5e09d09c2f720 does not exist\none or more of the container deletions failed\n”"
Jul 20 13:30:37 B2BTestServer kernel: [262136.651251] br-e80847589bfa: port 2(vethe12115e) entered disabled state
Jul 20 13:30:37 B2BTestServer kernel: [262136.668608] vethebfd8d9: renamed from eth0
Jul 20 13:30:37 B2BTestServer kernel: [262136.716289] aufs au_opts_verify:1597:dockerd[2634]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer kernel: [262136.738679] br-e80847589bfa: port 2(vethe12115e) entered disabled state
Jul 20 13:30:37 B2BTestServer kernel: [262136.753388] device vethe12115e left promiscuous mode
Jul 20 13:30:37 B2BTestServer kernel: [262136.762750] br-e80847589bfa: port 2(vethe12115e) entered disabled state
Jul 20 13:30:37 B2BTestServer kernel: [262136.804865] device vethdb0a6ac entered promiscuous mode
Jul 20 13:30:37 B2BTestServer kernel: [262136.814982] IPv6: ADDRCONF(NETDEV_UP): vethdb0a6ac: link is not ready
Jul 20 13:30:37 B2BTestServer kernel: [262136.827291] br-e80847589bfa: port 1(vethdb0a6ac) entered forwarding state
Jul 20 13:30:37 B2BTestServer kernel: [262136.840256] br-e80847589bfa: port 1(vethdb0a6ac) entered forwarding state
Jul 20 13:30:37 B2BTestServer systemd-udevd[43041]: Could not generate persistent MAC address for vethdb0a6ac: No such file or directory
Jul 20 13:30:37 B2BTestServer systemd-udevd[43040]: Could not generate persistent MAC address for vethf7a98cb: No such file or directory
Jul 20 13:30:37 B2BTestServer kernel: [262137.026630] aufs au_opts_verify:1597:dockerd[1209]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.579534632Z” level=warning msg="Your kernel does not support swap memory limit."
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.579622593Z” level=warning msg="Your kernel does not support cgroup rt period"
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.579632100Z” level=warning msg="Your kernel does not support cgroup rt runtime"
Jul 20 13:30:37 B2BTestServer kernel: [262137.101169] aufs au_opts_verify:1597:dockerd[1209]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37Z” level=info msg="Firewalld running: false"
Jul 20 13:30:37 B2BTestServer kernel: [262137.207004] aufs au_opts_verify:1597:dockerd[25633]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer kernel: [262137.248708] eth0: renamed from vethf7a98cb
Jul 20 13:30:37 B2BTestServer systemd-udevd[43171]: Could not generate persistent MAC address for veth91e8259: No such file or directory
Jul 20 13:30:37 B2BTestServer systemd-udevd[43172]: Could not generate persistent MAC address for veth3fff19d: No such file or directory
Jul 20 13:30:37 B2BTestServer kernel: [262137.273642] device veth3fff19d entered promiscuous mode
Jul 20 13:30:37 B2BTestServer kernel: [262137.287783] IPv6: ADDRCONF(NETDEV_UP): veth3fff19d: link is not ready
Jul 20 13:30:37 B2BTestServer kernel: [262137.307604] br-e80847589bfa: port 2(veth3fff19d) entered forwarding state
Jul 20 13:30:37 B2BTestServer kernel: [262137.320987] br-e80847589bfa: port 2(veth3fff19d) entered forwarding state
Jul 20 13:30:37 B2BTestServer kernel: [262137.335059] IPv6: ADDRCONF(NETDEV_CHANGE): vethdb0a6ac: link becomes ready
Jul 20 13:30:37 B2BTestServer kernel: [262137.349736] br-e80847589bfa: port 2(veth3fff19d) entered disabled state
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.883496197Z” level=warning msg="Your kernel does not support swap memory limit."
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.883793205Z” level=warning msg="Your kernel does not support cgroup rt period"
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.883937606Z” level=warning msg="Your kernel does not support cgroup rt runtime"
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37Z” level=info msg="Firewalld running: false"
Jul 20 13:30:38 B2BTestServer kernel: [262137.524751] eth0: renamed from veth91e8259
Jul 20 13:30:38 B2BTestServer kernel: [262137.544382] IPv6: ADDRCONF(NETDEV_CHANGE): veth3fff19d: link becomes ready
Jul 20 13:30:38 B2BTestServer kernel: [262137.556891] br-e80847589bfa: port 2(veth3fff19d) entered forwarding state
Jul 20 13:30:38 B2BTestServer kernel: [262137.569301] br-e80847589bfa: port 2(veth3fff19d) entered forwarding state
Jul 20 13:30:38 B2BTestServer kernel: [262137.704615] aufs au_opts_verify:1597:dockerd[3229]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:38 B2BTestServer kernel: [262137.803344] aufs au_opts_verify:1597:dockerd[3229]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:38 B2BTestServer kernel: [262137.902578] aufs au_opts_verify:1597:dockerd[1209]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:38 B2BTestServer kernel: [262137.926880] device veth4ff9aca entered promiscuous mode
Jul 20 13:30:38 B2BTestServer systemd-udevd[43302]: Could not generate persistent MAC address for veth4ff9aca: No such file or directory
Jul 20 13:30:38 B2BTestServer kernel: [262137.938135] IPv6: ADDRCONF(NETDEV_UP): veth4ff9aca: link is not ready
Jul 20 13:30:38 B2BTestServer systemd-udevd[43301]: Could not generate persistent MAC address for vetha4d818b: No such file or directory
Jul 20 13:30:38 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:38.495129396Z” level=warning msg="Your kernel does not support swap memory limit."
Jul 20 13:30:38 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:38.495215954Z” level=warning msg="Your kernel does not support cgroup rt period"
Jul 20 13:30:38 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:38.495226161Z” level=warning msg="Your kernel does not support cgroup rt runtime"
Jul 20 13:30:38 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:38Z” level=info msg="Firewalld running: false"
Jul 20 13:30:38 B2BTestServer kernel: [262138.116861] eth0: renamed from vetha4d818b
Jul 20 13:30:38 B2BTestServer kernel: [262138.136455] IPv6: ADDRCONF(NETDEV_CHANGE): veth4ff9aca: link becomes ready
Jul 20 13:30:38 B2BTestServer kernel: [262138.149540] br-e80847589bfa: port 4(veth4ff9aca) entered forwarding state
Jul 20 13:30:38 B2BTestServer kernel: [262138.162430] br-e80847589bfa: port 4(veth4ff9aca) entered forwarding state

Jul 20 13:35:31 B2BTestServer dockerd[1140]: time=“2017-07-20T13:35:31.489837293Z” level=error msg="Handler for POST /v1.25/containers/xxApp_app_1/start returned error: Container already started"
Jul 20 13:36:05 B2BTestServer dockerd[1140]: time=“2017-07-20T13:36:05.230362142Z” level=error msg=“Handler for POST /v1.25/containers/xxApp_worker_1/start returned error: Container already started”

Jul 20 13:50:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:50:01.606622410Z” level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused “process_linux.go:83: executing setns process caused \“exit status 15\””\n"
Jul 20 13:50:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:50:01.687809674Z” level=error msg=“Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused “process_linux.go:83: executing setns process caused \“exit status 15\””\n”

################################################################
sudo docker info
Containers: 8
Running: 3
Paused: 0
Stopped: 5
Images: 14
Server Version: 1.13.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 126
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-83-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.804 GiB
Name: TestServer

Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
############################################################


(Manasrb23) #2

We have same problem.

Please help.

Thanks,
Manas


(loanadda) #3

I also face same issue (which is home loan) and please help me