(thanks in advance for any help)
I have a dockerized application that processes data piped to it as follows:
cat data/dump.log | docker run -i private-docker-registry/images/haywire:0001-SNAPSHOT
However, when files of data greater than about 332K are piped to it, the container dies prematurely with this error message:
cat data/dump.log | docker run -i private-docker-registry/images/haywire:0001-SNAPSHOT
read unix @->/var/run/docker.sock: read: connection reset by peer
(Below 332K the container job completes)
I’ve monitored the container memory and cpu utilisation, however it doesn’t seem like the container is hitting resource limits:
(docker stats --no-stream , while running a job)
Wed Nov 22 22:54:39 SGT 2017 -
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
ad14acb12a57 178.01% 109.6MiB / 39.28GiB 0.27% 258B / 0B 0B / 65.5kB 24
Questions:
a) what is the meaning of this error message in the docker context?
b) is this likely a container related failure or an application related failure?
c) Is there some specific debugging I could use to troubleshoot this specific error message?
I’ve turned on debugging in docker-engine, but nothing specific in the logs seems to indicate why the container would exit this way:
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.221794353+08:00” level=debug msg=“Calling GET /_ping"
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.227806931+08:00” level=debug msg=“Calling POST /v1.29/containers/create"
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.228227054+08:00” level=debug msg=“form data: {“AttachStderr”:true,“AttachStdin”:true,“AttachStdout”:true,“Cmd”:[”–log-opt”,”-Dtopic=heatmap"],“Domainname”:"",“Entrypoint”:null,“Env”:[],“HostConfig”:{“AutoRemove”:false,“Binds”:null,“BlkioDeviceReadBps”:null,“BlkioDeviceReadIOps”:null,“BlkioDeviceWriteBps”:null,“BlkioDeviceWriteIOps”:null,“BlkioWeight”:0,“BlkioWeightDevice”:null,“CapAdd”:null,“CapDrop”:null,“Cgroup”:"",“CgroupParent”:"",“ConsoleSize”:[0,0],“ContainerIDFile”:"",“CpuCount”:0,“CpuPercent”:0,“CpuPeriod”:0,“CpuQuota”:0,“CpuRealtimePeriod”:0,“CpuRealtimeRuntime”:0,“CpuShares”:0,“CpusetCpus”:"",“CpusetMems”:"",“DeviceCgroupRules”:null,“Devices”:[],“DiskQuota”:0,“Dns”:[],“DnsOptions”:[],“DnsSearch”:[],“ExtraHosts”:null,“GroupAdd”:null,“IOMaximumBandwidth”:0,“IOMaximumIOps”:0,“IpcMode”:"",“Isolation”:"",“KernelMemory”:0,“Links”:null,“LogConfig”:{“Config”:{},“Type”:""},“Memory”:0,“MemoryReservation”:0,“MemorySwap”:0,“MemorySwappiness”:-1,“NanoCpus”:0,“NetworkMode”:“default”,“OomKillDisable”:false,“OomScoreAdj”:0,“PidMode”:"",“PidsLimit”:0,“PortBindings”:{},“Privileged”:false,“PublishAllPorts”:false,“ReadonlyRootfs”:false,“RestartPolicy”:{“MaximumRetryCount”:0,“Name”:“no”},“SecurityOpt”:null,“ShmSize”:0,“UTSMode”:"",“Ulimits”:null,“UsernsMode”:"",“VolumeDriver”:"",“VolumesFrom”:null},“Hostname”:"",“Image”:"lolco-docker-internal-docker.bintray.io/lolco/hyperwire-wifi-metrics:1.0.15-89-g980b483-SNAPSHOT",“Labels”:{},“NetworkingConfig”:{“EndpointsConfig”:{}},“OnBuild”:null,“OpenStdin”:true,“StdinOnce”:true,“Tty”:false,“User”:"",“Volumes”:{},“WorkingDir”:""}"
Nov 22 23:04:33 lol44 kernel: [9034251.803113] aufs au_opts_verify:1597:dockerd[800619]: dirperm1 breaks the protection by the permission bits on the lower branch
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.265562642+08:00” level=debug msg="container mounted via layerStore: /var/lib/docker/aufs/mnt/5ba0a31b856d9af84c3ebb4e578cb7f86b20d5e05183eb64903f96e4758b1e29"
Nov 22 23:04:33 lol44 kernel: [9034251.837020] aufs au_opts_verify:1597:dockerd[800619]: dirperm1 breaks the protection by the permission bits on the lower branch
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.299414410+08:00” level=debug msg="Calling POST /v1.29/containers/15a65d05b7ea4916bd6a60a57ca754635b5b203d618f8f1e47bd1436e5b49b3a/attach?stderr=1&stdin=1&stdout=1&stream=1"
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.299763275+08:00” level=debug msg="attach: stdin: begin"
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.299828000+08:00” level=debug msg="attach: stdout: begin"
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.299873851+08:00” level=debug msg="attach: stderr: begin"
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.300418724+08:00” level=debug msg="Calling GET /v1.29/events?filters=%7B%22container%22%3A%7B%2215a65d05b7ea4916bd6a60a57ca754635b5b203d618f8f1e47bd1436e5b49b3a%22%3Atrue%7D%2C%22type%22%3A%7B%22container%22%3Atrue%7D%7D"
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.301297708+08:00” level=debug msg="Calling POST /v1.29/containers/15a65d05b7ea4916bd6a60a57ca754635b5b203d618f8f1e47bd1436e5b49b3a/start"
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.302406006+08:00” level=debug msg="container mounted via layerStore: /var/lib/docker/aufs/mnt/5ba0a31b856d9af84c3ebb4e578cb7f86b20d5e05183eb64903f96e4758b1e29"
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.302880783+08:00” level=debug msg="Assigning addresses for endpoint condescending_leakey’s interface on network bridge"
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.302922254+08:00” level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, , map[])"
Nov 22 23:04:33 lol44 kernel: [9034251.873864] aufs au_opts_verify:1597:dockerd[800568]: dirperm1 breaks the protection by the permission bits on the lower branch
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.307329559+08:00” level=debug msg="Assigning addresses for endpoint condescending_leakey’s interface on network bridge"
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.313789381+08:00” level=debug msg="Programming external connectivity on endpoint condescending_leakey (4f1e4fc5f999697e34e4e95acc3cd868575a832911181d465bf05ef9516ab644)"
Nov 22 23:04:33 lol44 kernel: [9034252.043651] docker0: port 1(veth75c587c) entered forwarding state
Nov 22 23:04:33 lol44 kernel: [9034252.043667] docker0: port 1(veth75c587c) entered forwarding state
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.474099216+08:00” level=debug msg=“sandbox set key processing took 67.610038ms for container 15a65d05b7ea4916bd6a60a57ca754635b5b203d618f8f1e47bd1436e5b49b3a"
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.577914106+08:00” level=debug msg=“libcontainerd: received containerd event: &types.Event{Type:“start-container”, Id:“15a65d05b7ea4916bd6a60a57ca754635b5b203d618f8f1e47bd1436e5b49b3a”, Status:0x0, Pid:””, Timestamp:(*timestamp.Timestamp)(0xc42131c480)}"
Nov 22 23:04:33 lol44 dockerd[800568]: time=“2017-11-22T23:04:33.582638535+08:00” level=debug msg="libcontainerd: event unhandled: type:“start-container” id:“15a65d05b7ea4916bd6a60a57ca754635b5b203d618f8f1e47bd1436e5b49b3a” timestamp:<seconds:1511363073 nanos:577479466 > "
Nov 22 23:04:34 lol44 ntpd[48533]: Listen normally on 189 docker0 172.17.0.1:123
Nov 22 23:04:34 lol44 ntpd[48533]: Listen normally on 190 docker0 [fe80::42:41ff:fe4b:d69%3]:123
Nov 22 23:04:36 lol44 dockerd[800568]: time=“2017-11-22T23:04:36.55748669+08:00” level=debug msg=“containerd: process exited” id=15a65d05b7ea4916bd6a60a57ca754635b5b203d618f8f1e47bd1436e5b49b3a pid=init status=0 systemPid=887362
Nov 22 23:04:36 lol44 dockerd[800568]: time=“2017-11-22T23:04:36.564852714+08:00” level=debug msg="libcontainerd: received containerd event: &types.Event{Type:“exit”, Id:“15a65d05b7ea4916bd6a60a57ca754635b5b203d618f8f1e47bd1436e5b49b3a”, Status:0x0, Pid:“init”, Timestamp:(*timestamp.Timestamp)(0xc420c1f4d0)}"
Nov 22 23:04:36 lol44 dockerd[800568]: time=“2017-11-22T23:04:36.566082169+08:00” level=debug msg="attach: stdin: end"
Nov 22 23:04:36 lol44 dockerd[800568]: time=“2017-11-22T23:04:36.566112618+08:00” level=debug msg="attach: stderr: end"
Nov 22 23:04:36 lol44 dockerd[800568]: time=“2017-11-22T23:04:36.566099801+08:00” level=debug msg="attach: stdout: end"
Nov 22 23:04:36 lol44 dockerd[800568]: time=“2017-11-22T23:04:36.566138080+08:00” level=debug msg="Closing buffered stdin pipe"
Nov 22 23:04:36 lol44 dockerd[800568]: time=“2017-11-22T23:04:36.567583235+08:00” level=debug msg="Revoking external connectivity on endpoint condescending_leakey (4f1e4fc5f999697e34e4e95acc3cd868575a832911181d465bf05ef9516ab644)"
Nov 22 23:04:36 lol44 dockerd[800568]: time=“2017-11-22T23:04:36.571346269+08:00” level=debug msg="Client context cancelled, stop sending events"
Nov 22 23:04:36 lol44 dockerd[800568]: time=“2017-11-22T23:04:36.572212435+08:00” level=debug msg="DeleteConntrackEntries purged ipv4:0, ipv6:0"
Nov 22 23:04:36 lol44 kernel: [9034255.148140] docker0: port 1(veth75c587c) entered disabled state
Nov 22 23:04:36 lol44 kernel: [9034255.185100] docker0: por
— docker info output —
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 8
Server Version: 17.05.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 42
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-62-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 39.28GiB
Name: anx-dp02-044
ID: U3SB:IZIZ:3FQP:KZIZ:DD7V:LEPF:XSN4:BLE6:SC2N:2QVR:LQ44:WSRL
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 15
Goroutines: 22
System Time: 2017-11-23T00:01:37.224579759+08:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false