System Detail:
RAM: 128GB
CPU: 40 Core (Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz)
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.6 LTS"
$ uname -a
Linux pench 3.13.0-170-generic #220-Ubuntu SMP Thu May 9 12:40:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ docker info
Containers: 912
Running: 0
Paused: 0
Stopped: 912
Images: 85
Server Version: 18.06.3-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 2116
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: jj3klyl4me72g22wy8wc0hsva
Is Manager: true
ClusterID: tyq0ltx96xth062qpqlfrtpho
Managers: 1
Nodes: 1
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.129.2.55
Manager Addresses:
10.129.2.55:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: a592beb5bc4c4092b1b1bac971afed27687340c5
init version: fec3683
Security Options:
apparmor
Kernel Version: 3.13.0-170-generic
Operating System: Ubuntu 14.04.6 LTS
OSType: linux
Architecture: x86_64
CPUs: 40
Total Memory: 125.8GiB
Name: pench
ID: ANH5:H4BC:YBQJ:GQY3:YTU6:GGUC:MG6J:IT6Q:ZI7D:FZXC:2EVC:OIZL
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: ishtiyaq
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
$ docker-compose -v
docker-compose version 1.15.0, build e12f3b9
Issue
I am unable to deploy my docker stack service after the system update. Please see my docker log for more detail.
$ sudo tail -n 100 /var/log/upstart/docker.log
time="2021-02-12T01:23:13.779047025+05:30" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 35c0f8c8c636965f9ad2c0d5b1d71fab08fea012c7a93ea54f01bb4d5565f4da tw8fj35yp61v17tpf1mkw97rk err:cannot delete entry endpoint_table with network id tw8fj35yp61v17tpf1mkw97rk and key 35c0f8c8c636965f9ad2c0d5b1d71fab08fea012c7a93ea54f01bb4d5565f4da does not exist or is already being deleted"
time="2021-02-12T01:23:13.779170595+05:30" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster bodhi_flamingo_nginx 35c0f8c8c636965f9ad2c0d5b1d71fab08fea012c7a93ea54f01bb4d5565f4da aborted c.serviceBindings[skey] !ok"
time="2021-02-12T01:23:13.924493780+05:30" level=error msg="3e140ac9b382d1bded73885d010fdec7f0f45114c0f360754b16579deadd4a88 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:13.924606214+05:30" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:297: copying bootstrap data to pipe caused \\\"write init-p: broken pipe\\\"\": unknown" module=node/agent/taskmanager node.id=jj3klyl4me72g22wy8wc0hsva service.id=w1v95if41q0ugr89kaxehjbai task.id=oaxiboo8bli597m1xb037mujh
time="2021-02-12T01:23:13.947674219+05:30" level=error msg="f856e1a4cc6961b60f8ea39702c536fcf5aa8d6159fe47998eff0d2d9f21966e cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:13.947757596+05:30" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:297: copying bootstrap data to pipe caused \\\"write init-p: broken pipe\\\"\": unknown" module=node/agent/taskmanager node.id=jj3klyl4me72g22wy8wc0hsva service.id=qnyaubuvn2kz3scjvn49wihmt task.id=k89nxq0luf2r9rpxp0bsm0lnh
time="2021-02-12T01:23:13.975347705+05:30" level=error msg="23873380449c2c3e32216d759cd573d45fe94b3728f6a373c10e38de78d43a34 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:13.975410071+05:30" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:297: copying bootstrap data to pipe caused \\\"write init-p: broken pipe\\\"\": unknown" module=node/agent/taskmanager node.id=jj3klyl4me72g22wy8wc0hsva service.id=t79amno3wo2k432uutc6xauy5 task.id=udv1i2ny0od75k6zw35h9naa9
time="2021-02-12T01:23:13.994373817+05:30" level=error msg="b68c7325adaaee42c02550e1bbcea65efc9d9fdfd21e499bdaa522ca687663ac cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:13.994459089+05:30" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:297: copying bootstrap data to pipe caused \\\"write init-p: broken pipe\\\"\": unknown" module=node/agent/taskmanager node.id=jj3klyl4me72g22wy8wc0hsva service.id=l71fraoz07ewnzsez5fahjd6o task.id=20cz6xsxcops5mdugum35sc8c
time="2021-02-12T01:23:16.464340397+05:30" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint tw8fj35yp61v17tpf1mkw97rk 388e5a2dd770f45d72441059303ba0c7dc7f87c73e5f023b3496a4a7e7192918], retrying...."
time="2021-02-12T01:23:16.521409840+05:30" level=error msg="network bodhi_flamingo_bodhitree_network remove failed: unknown network bodhi_flamingo_bodhitree_network id tw8fj35yp61v17tpf1mkw97rk" module=node/agent node.id=jj3klyl4me72g22wy8wc0hsva
time="2021-02-12T01:23:16.521552414+05:30" level=error msg="network bodhi_flamingo_bodhitree_network remove failed: unknown network bodhi_flamingo_bodhitree_network id tw8fj35yp61v17tpf1mkw97rk" module=node/agent node.id=jj3klyl4me72g22wy8wc0hsva
time="2021-02-12T01:23:16.521641771+05:30" level=error msg="remove task failed" error="unknown network bodhi_flamingo_bodhitree_network id tw8fj35yp61v17tpf1mkw97rk" module=node/agent node.id=jj3klyl4me72g22wy8wc0hsva task.id=ifozcsfj9zgsnf54f03gy90ni
time="2021-02-12T01:23:16.521514675+05:30" level=error msg="remove task failed" error="unknown network bodhi_flamingo_bodhitree_network id tw8fj35yp61v17tpf1mkw97rk" module=node/agent node.id=jj3klyl4me72g22wy8wc0hsva task.id=9k1fi1pt6192gsi0539h968b6
time="2021-02-12T01:23:16.521739131+05:30" level=error msg="network bodhi_flamingo_bodhitree_network remove failed: unknown network bodhi_flamingo_bodhitree_network id tw8fj35yp61v17tpf1mkw97rk" module=node/agent node.id=jj3klyl4me72g22wy8wc0hsva
time="2021-02-12T01:23:16.521809326+05:30" level=error msg="remove task failed" error="unknown network bodhi_flamingo_bodhitree_network id tw8fj35yp61v17tpf1mkw97rk" module=node/agent node.id=jj3klyl4me72g22wy8wc0hsva task.id=rqy9vjyc4z1n33dby7cte6taa
time="2021-02-12T01:23:19.247341758+05:30" level=error msg="f9d717088d7c4aba22f2898818b54b378cade22a3c443e23e35c31adaa713b59 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:19.296460887+05:30" level=error msg="e67c7deb6b6145eafc9ea67fd371657238c8e82a5156cb12ca40604c4431e869 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:19.331897512+05:30" level=error msg="001651e19f1af72e1c2043fbfb3d3b09d98e6c02c78a5e5b6daa03fdbb235920 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:19.406731056+05:30" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodest_conn: no such file or directory"
INFO[1122] shim docker-containerd-shim started address="/containerd-shim/moby/4056a5fb0285e8996691dda3044982110ab470698e771c4329040754aeea6c19/shim.sock" debug=false pid=103586
INFO[1122] shim docker-containerd-shim started address="/containerd-shim/moby/e67c7deb6b6145eafc9ea67fd371657238c8e82a5156cb12ca40604c4431e869/shim.sock" debug=false pid=103598
INFO[1122] shim reaped id=4056a5fb0285e8996691dda3044982110ab470698e771c4329040754aeea6c19
INFO[1122] shim reaped id=e67c7deb6b6145eafc9ea67fd371657238c8e82a5156cb12ca40604c4431e869
time="2021-02-12T01:23:19.908074059+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:19.908092027+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:19.918753336+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:19.918805438+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:19.946143128+05:30" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for c2106e4a10b9403d0e1bdc44fbf8e55f3fee5cbc15e3dbc5582e343d9a72adda tw8fj35yp61v17tpf1mkw97rk err:cannot delete entry endpoint_table with network id tw8fj35yp61v17tpf1mkw97rk and key c2106e4a10b9403d0e1bdc44fbf8e55f3fee5cbc15e3dbc5582e343d9a72adda does not exist or is already being deleted"
time="2021-02-12T01:23:19.946189787+05:30" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster bodhi_flamingo_haproxy c2106e4a10b9403d0e1bdc44fbf8e55f3fee5cbc15e3dbc5582e343d9a72adda aborted c.serviceBindings[skey] !ok"
time="2021-02-12T01:23:19.962481936+05:30" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 88fd851199e07d3785ae2aa9506210957662ba9335d6781db9bcb6fe61207cbd tw8fj35yp61v17tpf1mkw97rk err:cannot delete entry endpoint_table with network id tw8fj35yp61v17tpf1mkw97rk and key 88fd851199e07d3785ae2aa9506210957662ba9335d6781db9bcb6fe61207cbd does not exist or is already being deleted"
time="2021-02-12T01:23:19.962533016+05:30" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster bodhi_flamingo_postgres 88fd851199e07d3785ae2aa9506210957662ba9335d6781db9bcb6fe61207cbd aborted c.serviceBindings[skey] !ok"
INFO[1122] shim docker-containerd-shim started address="/containerd-shim/moby/001651e19f1af72e1c2043fbfb3d3b09d98e6c02c78a5e5b6daa03fdbb235920/shim.sock" debug=false pid=103689
INFO[1122] shim docker-containerd-shim started address="/containerd-shim/moby/f9d717088d7c4aba22f2898818b54b378cade22a3c443e23e35c31adaa713b59/shim.sock" debug=false pid=103690
time="2021-02-12T01:23:19.974421035+05:30" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for c63d84890bb0f6848befbd12e262850d180d300cf70f53836490927dd0f1245b hwdtdhw7qhm2eu6xl0u7qt7n1 err:cannot delete entry endpoint_table with network id hwdtdhw7qhm2eu6xl0u7qt7n1 and key c63d84890bb0f6848befbd12e262850d180d300cf70f53836490927dd0f1245b does not exist or is already being deleted"
time="2021-02-12T01:23:19.974464895+05:30" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster bodhi_flamingo_haproxy c63d84890bb0f6848befbd12e262850d180d300cf70f53836490927dd0f1245b aborted c.serviceBindings[skey] !ok"
INFO[1122] shim reaped id=001651e19f1af72e1c2043fbfb3d3b09d98e6c02c78a5e5b6daa03fdbb235920
INFO[1122] shim reaped id=f9d717088d7c4aba22f2898818b54b378cade22a3c443e23e35c31adaa713b59
time="2021-02-12T01:23:20.001222626+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:20.001226449+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:20.001330511+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:20.001374819+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:20.065764231+05:30" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 337abd3921d7644309053f6d1a50d89e4d0375c996d35f768654bf6cc5435690 tw8fj35yp61v17tpf1mkw97rk err:cannot delete entry endpoint_table with network id tw8fj35yp61v17tpf1mkw97rk and key 337abd3921d7644309053f6d1a50d89e4d0375c996d35f768654bf6cc5435690 does not exist or is already being deleted"
time="2021-02-12T01:23:20.065854985+05:30" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster bodhi_flamingo_redis 337abd3921d7644309053f6d1a50d89e4d0375c996d35f768654bf6cc5435690 aborted c.serviceBindings[skey] !ok"
time="2021-02-12T01:23:20.066632235+05:30" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 3cb047b3e97886efb8a0469f708f8d0acdddc86fc87463ffbc435b0ce3d3ac67 tw8fj35yp61v17tpf1mkw97rk err:cannot delete entry endpoint_table with network id tw8fj35yp61v17tpf1mkw97rk and key 3cb047b3e97886efb8a0469f708f8d0acdddc86fc87463ffbc435b0ce3d3ac67 does not exist or is already being deleted"
time="2021-02-12T01:23:20.066685066+05:30" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster bodhi_flamingo_nginx 3cb047b3e97886efb8a0469f708f8d0acdddc86fc87463ffbc435b0ce3d3ac67 aborted c.serviceBindings[skey] !ok"
time="2021-02-12T01:23:20.111764528+05:30" level=error msg="e67c7deb6b6145eafc9ea67fd371657238c8e82a5156cb12ca40604c4431e869 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:20.111827388+05:30" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:297: copying bootstrap data to pipe caused \\\"write init-p: broken pipe\\\"\": unknown" module=node/agent/taskmanager node.id=jj3klyl4me72g22wy8wc0hsva service.id=w1v95if41q0ugr89kaxehjbai task.id=5bgsj0jg8ymutaeba7pq5nw90
time="2021-02-12T01:23:20.138829899+05:30" level=error msg="4056a5fb0285e8996691dda3044982110ab470698e771c4329040754aeea6c19 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:20.138894963+05:30" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:297: copying bootstrap data to pipe caused \\\"write init-p: broken pipe\\\"\": unknown" module=node/agent/taskmanager node.id=jj3klyl4me72g22wy8wc0hsva service.id=t79amno3wo2k432uutc6xauy5 task.id=xo0o4td9dpfo17kj8zildzzon
time="2021-02-12T01:23:20.282591869+05:30" level=error msg="f9d717088d7c4aba22f2898818b54b378cade22a3c443e23e35c31adaa713b59 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:20.282645611+05:30" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:297: copying bootstrap data to pipe caused \\\"write init-p: broken pipe\\\"\": unknown" module=node/agent/taskmanager node.id=jj3klyl4me72g22wy8wc0hsva service.id=qnyaubuvn2kz3scjvn49wihmt task.id=q6sf86epabu9eqaymmsoskpc2
time="2021-02-12T01:23:20.295662866+05:30" level=error msg="001651e19f1af72e1c2043fbfb3d3b09d98e6c02c78a5e5b6daa03fdbb235920 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:20.295776800+05:30" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:297: copying bootstrap data to pipe caused \\\"write init-p: broken pipe\\\"\": unknown" module=node/agent/taskmanager node.id=jj3klyl4me72g22wy8wc0hsva service.id=l71fraoz07ewnzsez5fahjd6o task.id=nhe1qyz78fkaiz3krjleq3k1f
time="2021-02-12T01:23:22.653955359+05:30" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint tw8fj35yp61v17tpf1mkw97rk b118bccf7b0dcb3fa89fab994a8445d4c75bebeb09738de13c70099239fb00c8], retrying...."
time="2021-02-12T01:23:22.708356396+05:30" level=error msg="network bodhi_flamingo_bodhitree_network remove failed: unknown network bodhi_flamingo_bodhitree_network id tw8fj35yp61v17tpf1mkw97rk" module=node/agent node.id=jj3klyl4me72g22wy8wc0hsva
time="2021-02-12T01:23:22.708467994+05:30" level=error msg="remove task failed" error="unknown network bodhi_flamingo_bodhitree_network id tw8fj35yp61v17tpf1mkw97rk" module=node/agent node.id=jj3klyl4me72g22wy8wc0hsva task.id=oe7c0yf6a6h1doeovfd9st8vj
time="2021-02-12T01:23:22.708491628+05:30" level=error msg="network bodhi_flamingo_bodhitree_network remove failed: unknown network bodhi_flamingo_bodhitree_network id tw8fj35yp61v17tpf1mkw97rk" module=node/agent node.id=jj3klyl4me72g22wy8wc0hsva
time="2021-02-12T01:23:22.708573701+05:30" level=error msg="network bodhi_flamingo_bodhitree_network remove failed: unknown network bodhi_flamingo_bodhitree_network id tw8fj35yp61v17tpf1mkw97rk" module=node/agent node.id=jj3klyl4me72g22wy8wc0hsva
time="2021-02-12T01:23:22.708667216+05:30" level=error msg="remove task failed" error="unknown network bodhi_flamingo_bodhitree_network id tw8fj35yp61v17tpf1mkw97rk" module=node/agent node.id=jj3klyl4me72g22wy8wc0hsva task.id=ncn4nur9cxnlhn6xegkukaknp
time="2021-02-12T01:23:22.708586122+05:30" level=error msg="remove task failed" error="unknown network bodhi_flamingo_bodhitree_network id tw8fj35yp61v17tpf1mkw97rk" module=node/agent node.id=jj3klyl4me72g22wy8wc0hsva task.id=jeujvkq8mp1p2oqmyocb39yk8
time="2021-02-12T01:23:26.117323201+05:30" level=error msg="05a87439cf529ab7fa38b52cbaf29f4b9a83ee21057bec96f1970fe975dc5986 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:26.139408079+05:30" level=error msg="26908ee9e083ccdb72093f48f1734fac769f4314a6e95b394947ef9639e6dc47 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:26.238453289+05:30" level=error msg="119d2b771ba9d6d445e478625fbe5a25ff313b9f15ff80b3b4accd7b384c84c2 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:26.288785392+05:30" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodest_conn: no such file or directory"
INFO[1129] shim docker-containerd-shim started address="/containerd-shim/moby/8d72bd36258631bb300511d68e2b3255ac4d3cc6f507ae076fe2d41f502807fc/shim.sock" debug=false pid=104169
INFO[1129] shim reaped id=8d72bd36258631bb300511d68e2b3255ac4d3cc6f507ae076fe2d41f502807fc
time="2021-02-12T01:23:27.037773952+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:27.037773913+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:27.119768785+05:30" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 0306de2a37382d50faf35f9ad639f4f955f2819e57b77044eea11f0d77d02914 tw8fj35yp61v17tpf1mkw97rk err:cannot delete entry endpoint_table with network id tw8fj35yp61v17tpf1mkw97rk and key 0306de2a37382d50faf35f9ad639f4f955f2819e57b77044eea11f0d77d02914 does not exist or is already being deleted"
time="2021-02-12T01:23:27.119815837+05:30" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster bodhi_flamingo_haproxy 0306de2a37382d50faf35f9ad639f4f955f2819e57b77044eea11f0d77d02914 aborted c.serviceBindings[skey] !ok"
INFO[1130] shim docker-containerd-shim started address="/containerd-shim/moby/119d2b771ba9d6d445e478625fbe5a25ff313b9f15ff80b3b4accd7b384c84c2/shim.sock" debug=false pid=104222
INFO[1130] shim reaped id=119d2b771ba9d6d445e478625fbe5a25ff313b9f15ff80b3b4accd7b384c84c2
time="2021-02-12T01:23:27.183146302+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:27.183191350+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:27.189096434+05:30" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 6a4efe4a8a8da862bd8d6197eb17f4042cb5c24f72629fded1cce303c6307eec hwdtdhw7qhm2eu6xl0u7qt7n1 err:cannot delete entry endpoint_table with network id hwdtdhw7qhm2eu6xl0u7qt7n1 and key 6a4efe4a8a8da862bd8d6197eb17f4042cb5c24f72629fded1cce303c6307eec does not exist or is already being deleted"
time="2021-02-12T01:23:27.189141495+05:30" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster bodhi_flamingo_haproxy 6a4efe4a8a8da862bd8d6197eb17f4042cb5c24f72629fded1cce303c6307eec aborted c.serviceBindings[skey] !ok"
time="2021-02-12T01:23:27.253038862+05:30" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for ab0a90687d0de6ba4ac82ae39149510066d0fe98b15052560413b716df0f8618 tw8fj35yp61v17tpf1mkw97rk err:cannot delete entry endpoint_table with network id tw8fj35yp61v17tpf1mkw97rk and key ab0a90687d0de6ba4ac82ae39149510066d0fe98b15052560413b716df0f8618 does not exist or is already being deleted"
time="2021-02-12T01:23:27.253081828+05:30" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster bodhi_flamingo_redis ab0a90687d0de6ba4ac82ae39149510066d0fe98b15052560413b716df0f8618 aborted c.serviceBindings[skey] !ok"
INFO[1130] shim docker-containerd-shim started address="/containerd-shim/moby/26908ee9e083ccdb72093f48f1734fac769f4314a6e95b394947ef9639e6dc47/shim.sock" debug=false pid=104273
INFO[1130] shim docker-containerd-shim started address="/containerd-shim/moby/05a87439cf529ab7fa38b52cbaf29f4b9a83ee21057bec96f1970fe975dc5986/shim.sock" debug=false pid=104274
INFO[1130] shim reaped id=05a87439cf529ab7fa38b52cbaf29f4b9a83ee21057bec96f1970fe975dc5986
INFO[1130] shim reaped id=26908ee9e083ccdb72093f48f1734fac769f4314a6e95b394947ef9639e6dc47
time="2021-02-12T01:23:27.306056178+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:27.306077712+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:27.311800205+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:27.311833602+05:30" level=error msg="stream copy error: reading from a closed fifo"
time="2021-02-12T01:23:27.331769760+05:30" level=error msg="8d72bd36258631bb300511d68e2b3255ac4d3cc6f507ae076fe2d41f502807fc cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:27.331883192+05:30" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:297: copying bootstrap data to pipe caused \\\"write init-p: broken pipe\\\"\": unknown" module=node/agent/taskmanager node.id=jj3klyl4me72g22wy8wc0hsva service.id=t79amno3wo2k432uutc6xauy5 task.id=k4qm8m6ht3ktvp3y0h9miw41x
time="2021-02-12T01:23:27.351421746+05:30" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 531498aeab45ce07715ca0caef66732c958d1599360a80c4523103c7eb0fba69 tw8fj35yp61v17tpf1mkw97rk err:cannot delete entry endpoint_table with network id tw8fj35yp61v17tpf1mkw97rk and key 531498aeab45ce07715ca0caef66732c958d1599360a80c4523103c7eb0fba69 does not exist or is already being deleted"
time="2021-02-12T01:23:27.351478624+05:30" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster bodhi_flamingo_postgres 531498aeab45ce07715ca0caef66732c958d1599360a80c4523103c7eb0fba69 aborted c.serviceBindings[skey] !ok"
time="2021-02-12T01:23:27.351571746+05:30" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 769b9a6146ccd6d84016fd8551237677eedac1750426e6b37d31d99986edd9ed tw8fj35yp61v17tpf1mkw97rk err:cannot delete entry endpoint_table with network id tw8fj35yp61v17tpf1mkw97rk and key 769b9a6146ccd6d84016fd8551237677eedac1750426e6b37d31d99986edd9ed does not exist or is already being deleted"
time="2021-02-12T01:23:27.351624981+05:30" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster bodhi_flamingo_nginx 769b9a6146ccd6d84016fd8551237677eedac1750426e6b37d31d99986edd9ed aborted c.serviceBindings[skey] !ok"
time="2021-02-12T01:23:27.388135082+05:30" level=error msg="119d2b771ba9d6d445e478625fbe5a25ff313b9f15ff80b3b4accd7b384c84c2 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:27.388186362+05:30" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:297: copying bootstrap data to pipe caused \\\"write init-p: broken pipe\\\"\": unknown" module=node/agent/taskmanager node.id=jj3klyl4me72g22wy8wc0hsva service.id=qnyaubuvn2kz3scjvn49wihmt task.id=bpz6s9h804vnthw5g5gqszd5e
time="2021-02-12T01:23:27.731364087+05:30" level=error msg="26908ee9e083ccdb72093f48f1734fac769f4314a6e95b394947ef9639e6dc47 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:27.731431802+05:30" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:297: copying bootstrap data to pipe caused \\\"write init-p: broken pipe\\\"\": unknown" module=node/agent/taskmanager node.id=jj3klyl4me72g22wy8wc0hsva service.id=w1v95if41q0ugr89kaxehjbai task.id=ikld3751f3g83284yxzi5mn4g
time="2021-02-12T01:23:27.755460322+05:30" level=error msg="05a87439cf529ab7fa38b52cbaf29f4b9a83ee21057bec96f1970fe975dc5986 cleanup: failed to delete container from containerd: no such container"
time="2021-02-12T01:23:27.755556974+05:30" level=error msg="fatal task error" error="starting container failed: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:297: copying bootstrap data to pipe caused \\\"write init-p: broken pipe\\\"\": unknown" module=node/agent/taskmanager node.id=jj3klyl4me72g22wy8wc0hsva service.id=l71fraoz07ewnzsez5fahjd6o task.id=f3uzgelxr77543wgq0gcp5kk4
I can not see any containers using docker ps command but docker ps -a give the result as shown below.
$ docker ps -a
4440f28c23a4 postgres:11 "docker-entrypoint.s…" 15 seconds ago Created 5432/tcp bodhi_flamingo_postgres.1.0x0smwegjiy18dc27qpbkt005
6ec5de5d27ad ishtiyaq/nginx-brotli:1.19.6 "/docker-entrypoint.…" 15 seconds ago Created 80/tcp bodhi_flamingo_nginx.1.xe1n7zhw2eopwpssj8j0o19v6
0c40c4d91d0b redis:6 "docker-entrypoint.s…" 15 seconds ago Created 6379/tcp bodhi_flamingo_redis.1.nk61hp6huma06qg6zz0kf9dar
ebfb7b34cf28 haproxy:2.3.2 "docker-entrypoint.s…" 15 seconds ago Created bodhi_flamingo_haproxy.1.pskzml57el0f4y3vi8vm8f86d
de66ade61b56 postgres:11 "docker-entrypoint.s…" 22 seconds ago Created 5432/tcp bodhi_flamingo_postgres.1.rtyrvm9r0iriwcmzi1h5dmnkg
b1021b90fdc8 redis:6 "docker-entrypoint.s…" 22 seconds ago Created 6379/tcp bodhi_flamingo_redis.1.bw624954ayc37quyyjjgeh77x
4590315cfc14 haproxy:2.3.2 "docker-entrypoint.s…" 22 seconds ago Created bodhi_flamingo_haproxy.1.juq35uv5vd9vbbv47qxfzzyka
9c347976939b ishtiyaq/nginx-brotli:1.19.6 "/docker-entrypoint.…" 22 seconds ago Created 80/tcp bodhi_flamingo_nginx.1.xkjrg6zazdj0ahz0x0mq62dsx
9c21fa7602f8 redis:6 "docker-entrypoint.s…" 28 seconds ago Created 6379/tcp bodhi_flamingo_redis.1.qz6xwr9jfwgcv1em0yaiueyno
8770fee86a21 haproxy:2.3.2 "docker-entrypoint.s…" 28 seconds ago Created bodhi_flamingo_haproxy.1.osmakuy1nm0d573c8ey3uqxcb
cd795f306b1c ishtiyaq/nginx-brotli:1.19.6 "/docker-entrypoint.…" 28 seconds ago Created 80/tcp bodhi_flamingo_nginx.1.ird9kjyacd4h1q1u9f3lby2le
b7d48792292b postgres:11 "docker-entrypoint.s…" 28 seconds ago Created 5432/tcp bodhi_flamingo_postgres.1.zmz9198llcb7gskvgoc2oitiz
b09d49d01236 redis:6 "docker-entrypoint.s…" 34 seconds ago Created 6379/tcp bodhi_flamingo_redis.1.aklhecrzzsvrt2r916xr6dphc
6141f45a80b1 postgres:11 "docker-entrypoint.s…" 34 seconds ago Created 5432/tcp bodhi_flamingo_postgres.1.oqs29wdayyazaj5zkgpqpd20t
b9f48a6898fc haproxy:2.3.2 "docker-entrypoint.s…" 34 seconds ago Created bodhi_flamingo_haproxy.1.pdb0pavo4p2cnnx0rpybiqdtl
08f042341368 ishtiyaq/nginx-brotli:1.19.6 "/docker-entrypoint.…" 34 seconds ago Created 80/tcp bodhi_flamingo_nginx.1.kwh6yzlnxgymoe9pvryft9r25
79e5be78b103 ishtiyaq/nginx-brotli:1.19.6 "/docker-entrypoint.…" 41 seconds ago Created 80/tcp bodhi_flamingo_nginx.1.xp588da3yu0q17c939mtvxn0j
ce137fab3e08 haproxy:2.3.2 "docker-entrypoint.s…" 41 seconds ago Created bodhi_flamingo_haproxy.1.r5tu3wj74p8wxbgpq2t5rfusk
82f72344fcb9 postgres:11 "docker-entrypoint.s…" 41 seconds ago Created 5432/tcp bodhi_flamingo_postgres.1.i9xpr4qo0i685c1r26ipg4p95
c38e909c4a14 haproxy:2.3.2 "docker-entrypoint.s…" 47 seconds ago Created bodhi_flamingo_haproxy.1.3ni57ywdj7nz9x2w1a16ljd35
44d3f26dae12 postgres:11 "docker-entrypoint.s…" 47 seconds ago Created 5432/tcp bodhi_flamingo_postgres.1.ybsjkx5jwtebvusamk9o11r25
75ea212fc262 redis:6 "docker-entrypoint.s…" 47 seconds ago Created 6379/tcp bodhi_flamingo_redis.1.xwm37o9x5m5rhj3tkimapz6qj
ad58fa1ec968 haproxy:2.3.2 "docker-entrypoint.s…" 53 seconds ago Created bodhi_flamingo_haproxy.1.ti5b5segqwhd7d8o51ug999v9
9e437443e12e redis:6 "docker-entrypoint.s…" 53 seconds ago Created 6379/tcp bodhi_flamingo_redis.1.iipx05o8xd43fpzp04bcdhg13
970f4b09db60 ishtiyaq/nginx-brotli:1.19.6 "/docker-entrypoint.…" 53 seconds ago Created 80/tcp bodhi_flamingo_nginx.1.tw1k38a1yxdpuuwemriu6lesi