Replying yo @meyay, the output of sudo ss -tlpn | grep ":80 " will return the process because now everything is fine now. As i said this is a prod environment and the reverse proxy uses this port, so when the problem happened, i only had few minutes to collect data before resolving the issue by restarting docker service.
My reverse proxy compose (the one with problems)
version: ‘3.8’
services:
proxy:
container_name: “devoz-proxy”
image: “haproxy:2.8.15”
user: root
ports:
- “80:80”
- “443:443”
- “1936:1936”
restart: always
networks:
- “ozmap_isolated_nw”
volumes:
- /opt/haproxy/:/usr/local/etc/haproxy/
- /opt/haproxy/errors/:/errors/
- /opt/ssl/:/ssl/
environment:
DEBUG: “*”
networks:
ozmap_isolated_nw:
external: true
So i tested with another compose:
version: “3.8”
services:
nginxproxy:
container_name: nginx
image: nginx:latest
ports:
- “90:80”
networks:
- “ozmap_isolated_nw”
networks:
ozmap_isolated_nw:
external: true
root@ip-xxxxxxxxx:~# docker kill nginx
root@ip-xxxxxxxxx:~# docker-compose ps -a
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTSnginx nginx:latest “/docker-entrypoint.…” nginxproxy 7 minutes ago Exited (137) 4 seconds ago
root@ip-xxxxxxxxx:~# docker-compose up -d
[+] Running 1/1
Container nginx Started 0.0s
No problems, i really dont know how to reproduce this behavior, sounds like random, or very specific scenario. The context of the error was, we had to restart the ec2 because it was very slow and had exhausted its resources.
Here is the journalctl
0 5f9ef26ca3fb4e65a3aa3784063e0ef0 Thu 2026-01-15 18:14:54 UTC—Mon 2026-01-19 13:17:21 → last reboot
Jan 15 18:10:28 ip-172-33-13-185 systemd[1]: Stopping Docker Application Container Engine…
– Subject: A stop job for unit docker.service has begun execution
– Defined-By: systemd
– Support: http://www.ubuntu.com/support
– A stop job for unit docker.service has begun execution.
– The job identifier is 147316.
Jan 15 18:10:51 ip-172-33-13-185 dockerd[714]: time=“2026-01-15T18:10:45.200692661Z” level=info msg=“Processing signal ‘terminated’”
Jan 15 18:11:58 ip-172-33-13-185 systemd[1]: docker.service: Killing process 714 (dockerd) with signal SIGKILL.
Jan 15 18:11:58 ip-172-33-13-185 systemd[1]: docker.service: Main process exited, code=killed, status=9/KILL
Jan 15 18:15:43 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:15:43.781491705Z” level=info msg=“Removing stale endpoint devoz-proxy (874034bfbc8726bb2ea779e669e19a362bac614e607ec44e4162319d2daf7efd)”
Jan 15 18:21:50 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:21:50.746692326Z” level=info msg=“No non-localhost DNS nameservers are left in resolv.conf. Using default external servers”
Jan 15 18:22:36 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:22:36.138480583Z” level=info msg=“No non-localhost DNS nameservers are left in resolv.conf. Using default external servers"Jan 15 18:22:36 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:22:36.153619568Z” level=warning msg=“Failed to allocate and map port: Bind for 0.0.0.0:80 failed: port is already allocated”
Jan 15 18:22:36 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:22:36.306307856Z” level=error msg=“2c320ec50cd7ca4523baf6aeb441d85a29230ce478bd725cabbbc0d6985212da cleanup: failed to delete container from containerd: NotFound: container "2c320ec50cd7ca4523baf6aeb441d85a29230ce478bd725cabbbc0d6985212da" in namespace "moby": not found”
Jan 15 18:22:36 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:22:36.374659318Z” level=error msg=“Handler for POST /v1.43/containers/2c320ec50cd7ca4523baf6aeb441d85a29230ce478bd725cabbbc0d6985212da/start returned error: driver failed programming external connectivity on endpoint devoz-proxy (d69497c01c8d632a87dd7702ab0fabd00006ce218fd9155992a6f01e464b2936): Bind for 0.0.0.0:80 failed: port is already allocated”
Jan 15 18:22:55 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:22:55.725811484Z” level=info msg=“ignoring event” container=990d9023d0b52e84608d421aed183e22b2c00c169ea27bf0b6413630bbcf3d4b module=libcontainerd namespace=mob
y topic=/tasks/delete type=”*events.TaskDelete"
Jan 15 18:22:56 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:22:56.224332340Z” level=info msg=“No non-localhost DNS nameservers are left in resolv.conf. Using default external servers"Jan 15 18:23:25 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:23:25.115320161Z” level=info msg=“No non-localhost DNS nameservers are left in resolv.conf. Using default external servers”
Jan 15 18:23:25 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:23:25.124166609Z” level=warning msg=“Failed to allocate and map port: Bind for 0.0.0.0:80 failed: port is already allocated"Jan 15 18:23:25 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:23:25.261664595Z” level=error msg=“Handler for POST /v1.43/containers/cd0fd7060baa72573ad6ddec766f1170bea03d30a7f19a4735b90d710f0bbff2/start returned error: dr
iver failed programming external connectivity on endpoint devoz-proxy (689de3d47529c354579fdb19a2cf687d95f509eb8c4fbeac1a0e3d2ef9b0cc0d): Bind for 0.0.0.0:80 failed: port is already allocated”
Jan 15 18:24:00 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:24:00.773598513Z” level=info msg=“ignoring event” container=990d9023d0b52e84608d421aed183e22b2c00c169ea27bf0b6413630bbcf3d4b module=libcontainerd namespace=mob
y topic=/tasks/delete type=”*events.TaskDelete"Jan 15 18:24:01 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:24:01.259127534Z” level=info msg=“No non-localhost DNS nameservers are left in resolv.conf. Using default external servers”
Jan 15 18:24:35 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:24:35.365226290Z” level=info msg=“No non-localhost DNS nameservers are left in resolv.conf. Using default external servers"Jan 15 18:24:35 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:24:35.380594537Z” level=warning msg=“Failed to allocate and map port: Bind for 0.0.0.0:80 failed: port is already allocated”
Jan 15 18:24:35 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:24:35.514454801Z” level=error msg=“Handler for POST /v1.43/containers/cd0fd7060baa72573ad6ddec766f1170bea03d30a7f19a4735b90d710f0bbff2/start returned error: dr
iver failed programming external connectivity on endpoint devoz-proxy (ed0161fb1018fa242e2a62995011a82214a04f5eefbcb87bc776d32a7448d83f): Bind for 0.0.0.0:80 failed: port is already allocated”
Jan 15 18:25:05 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:05.427204441Z” level=info msg=“ignoring event” container=990d9023d0b52e84608d421aed183e22b2c00c169ea27bf0b6413630bbcf3d4b module=libcontainerd namespace=moby topic=/tasks/delete type=”*events.TaskDelete”
Jan 15 18:25:05 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:05.866990172Z” level=info msg=“No non-localhost DNS nameservers are left in resolv.conf. Using default external servers”
Jan 15 18:25:17 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:17.520489278Z” level=info msg=“No non-localhost DNS nameservers are left in resolv.conf. Using default external servers”
Jan 15 18:25:17 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:17.529725109Z” level=warning msg=“Failed to allocate and map port: Bind for 0.0.0.0:80 failed: port is already allocated”
Jan 15 18:25:17 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:17.741545857Z” level=error msg=“Handler for POST /v1.43/containers/94c480f63bacf732ad7d0fe16c1b4cfa9a1667731982855899626c27e4b76b3e/start returned error: dr
iver failed programming external connectivity on endpoint devoz-proxy (89b70318eee9c7b3624d7160e21df120c4f6a1959e8d69f4d3534d3122b83193): Bind for 0.0.0.0:80 failed: port is already allocated”
Jan 15 18:25:33 ip-172-33-13-185 systemd[1]: Stopping Docker Application Container Engine… # manual restart docker
Jan 15 18:25:34 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:34.800383645Z” level=warning msg=“ShouldRestart failed, container will not be restarted” container=c9eb0b1a7e07afc46db72bf3f99ba0e2fea02988c44500dd22aeae9e86c4b13a daemonShuttingDown=true error=“restart canceled” execDuration=9m48.759099498s exitStatus=“{143 2026-01-15 18:25:34.598177725 +0000 UTC}” hasBeenManuallyStopped=false restartCount=0
Jan 15 18:25:34 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:34.956062563Z” level=info msg=“ignoring event” container=d38e755ff7f2ade5743709818840a5b5414bb75257fb3ff4890db747efd72521 module=libcontainerd namespace=moby topic=/tasks/delete type=“*events.TaskDelete”
Jan 15 18:25:35 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:35.057294771Z” level=warning msg=“ShouldRestart failed, container will not be restarted” container=d38e755ff7f2ade5743709818840a5b5414bb75257fb3ff4890db747efd72521 daemonShuttingDown=true error=“restart canceled” execDuration=9m49.606260329s exitStatus=“{0 2026-01-15 18:25:34.856535704 +0000 UTC}” hasBeenManuallyStopped=false restartCount=0
Jan 15 18:25:35 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:35.238302672Z” level=info msg=“ignoring event” container=5b9a50400ecff079c7841cf4efabc29d217fb353cd65a38cf097a8f70e46cc18 module=libcontainerd namespace=moby topic=/tasks/delete type=“*events.TaskDelete”
Jan 15 18:25:35 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:35.375621707Z” level=warning msg=“ShouldRestart failed, container will not be restarted” container=5b9a50400ecff079c7841cf4efabc29d217fb353cd65a38cf097a8f70e46cc18 daemonShuttingDown=true error=“restart canceled” execDuration=9m50.34974831s exitStatus=“{0 2026-01-15 18:25:35.086633594 +0000 UTC}” hasBeenManuallyStopped=false restartCount=0
Jan 15 18:25:35 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:35.762676976Z” level=info msg=“ignoring event” container=ef3088e8033f5b935ea659a5ab4751466d2bb7ed5111e42089b89445dd16878d module=libcontainerd namespace=moby topic=/tasks/delete type=“*events.TaskDelete”
Jan 15 18:25:35 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:35.839502235Z” level=warning msg=“ShouldRestart failed, container will not be restarted” container=ef3088e8033f5b935ea659a5ab4751466d2bb7ed5111e42089b89445dd16878d daemonShuttingDown=true error=“restart canceled” execDuration=9m50.435908582s exitStatus=“{0 2026-01-15 18:25:35.528535459 +0000 UTC}” hasBeenManuallyStopped=false restartCount=0
Jan 15 18:25:36 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:36.785015809Z” level=info msg=“ignoring event” container=de021ef52d7caeb6762d99ee09006a684c1176bd36d475da4a14ec092d392be1 module=libcontainerd namespace=moby topic=/tasks/delete type=“*events.TaskDelete”
Jan 15 18:25:36 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:36.832902752Z” level=warning msg=“ShouldRestart failed, container will not be restarted” container=de021ef52d7caeb6762d99ee09006a684c1176bd36d475da4a14ec092d392be1 daemonShuttingDown=true error=“restart canceled” execDuration=9m51.276333603s exitStatus=“{0 2026-01-15 18:25:36.726793761 +0000 UTC}” hasBeenManuallyStopped=false restartCount=0
Jan 15 18:25:36 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:36.948000755Z” level=info msg=“ignoring event” container=7e4de31c93e714fff55f58cfc6994fbe5d02503f82c12ab09eaf00aed8087123 module=libcontainerd namespace=moby topic=/tasks/delete type=“*events.TaskDelete”
Jan 15 18:25:37 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:37.036351042Z” level=warning msg=“ShouldRestart failed, container will not be restarted” container=7e4de31c93e714fff55f58cfc6994fbe5d02503f82c12ab09eaf00aed8087123 daemonShuttingDown=true error=“restart canceled” execDuration=9m51.088430839s exitStatus=“{1 2026-01-15 18:25:36.864220794 +0000 UTC}” hasBeenManuallyStopped=false restartCount=0
Jan 15 18:25:37 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:37.588878066Z” level=info msg=“ignoring event” container=36ae64323089a5a9ca33e58d2a39dfaa2a467255a0a576dd46480e3db23fdc19 module=libcontainerd namespace=moby topic=/tasks/delete type=“*events.TaskDelete”
Jan 15 18:25:37 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:37.616519337Z” level=warning msg=“ShouldRestart failed, container will not be restarted” container=36ae64323089a5a9ca33e58d2a39dfaa2a467255a0a576dd46480e3db23fdc19 daemonShuttingDown=true error=“restart canceled” execDuration=9m52.992341862s exitStatus=“{1 2026-01-15 18:25:37.563151054 +0000 UTC}” hasBeenManuallyStopped=false restartCount=0
Jan 15 18:25:40 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:40.026925047Z” level=info msg=“ignoring event” container=40b0e988e06145f0c21abacff1a0739af829b2d845ce4a90dfebe5939168f0f6 module=libcontainerd namespace=moby topic=/tasks/delete type=“*events.TaskDelete”
Jan 15 18:25:40 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:40.061800759Z” level=warning msg=“ShouldRestart failed, container will not be restarted” container=40b0e988e06145f0c21abacff1a0739af829b2d845ce4a90dfebe5939168f0f6 daemonShuttingDown=true error=“restart canceled” execDuration=9m54.154675997s exitStatus=“{0 2026-01-15 18:25:39.9986036 +0000 UTC}” hasBeenManuallyStopped=false restartCount=0
Jan 15 18:25:43 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:43.486459137Z” level=info msg=“Container failed to exit within 10s of signal 15 - using the force” container=990d9023d0b52e84608d421aed183e22b2c00c169ea27bf
0b6413630bbcf3d4b
Jan 15 18:25:43 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:43.534052469Z” level=info msg=“ignoring event” container=990d9023d0b52e84608d421aed183e22b2c00c169ea27bf0b6413630bbcf3d4b module=libcontainerd namespace=mob
y topic=/tasks/delete type=“*events.TaskDelete”
Jan 15 18:25:43 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:43.560975484Z” level=warning msg=“ShouldRestart failed, container will not be restarted” container=990d9023d0b52e84608d421aed183e22b2c00c169ea27bf0b6413630b
bcf3d4b daemonShuttingDown=true error=“restart canceled” execDuration=37.493817341s exitStatus=“{137 2026-01-15 18:25:43.51546101 +0000 UTC}” hasBeenManuallyStopped=false restartCount=8
Jan 15 18:25:43 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:43.575041568Z” level=info msg=“Container failed to exit within 10s of signal 15 - using the force” container=b453fc13c2703e9e0bbbc63c2687471998a75207878e641
4206b4d7caf9c7d4b
Jan 15 18:25:43 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:43.774468498Z” level=info msg=“ignoring event” container=b453fc13c2703e9e0bbbc63c2687471998a75207878e6414206b4d7caf9c7d4b module=libcontainerd namespace=mob
y topic=/tasks/delete type=“*events.TaskDelete”
Jan 15 18:25:43 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:43.826605136Z” level=warning msg=“ShouldRestart failed, container will not be restarted” container=b453fc13c2703e9e0bbbc63c2687471998a75207878e6414206b4d7ca
f9c7d4b daemonShuttingDown=true error=“restart canceled” execDuration=9m58.567299795s exitStatus=“{137 2026-01-15 18:25:43.702815126 +0000 UTC}” hasBeenManuallyStopped=false restartCount=0
Jan 15 18:25:44 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:44.056086664Z” level=info msg=“stopping event stream following graceful shutdown” error=“” module=libcontainerd namespace=moby
Jan 15 18:25:44 ip-172-33-13-185 dockerd[703]: time=“2026-01-15T18:25:44.064255019Z” level=info msg=“Daemon shutdown complete”
Jan 15 18:25:44 ip-172-33-13-185 systemd[1]: docker.service: Succeeded.
Jan 15 18:25:44 ip-172-33-13-185 dockerd[5662]: time=“2026-01-15T18:25:44.453349941Z” level=info msg=“Starting up”
Jan 15 18:25:44 ip-172-33-13-185 dockerd[5662]: time=“2026-01-15T18:25:44.453504110Z” level=warning msg=“Running experimental build”
Jan 15 18:25:44 ip-172-33-13-185 dockerd[5662]: time=“2026-01-15T18:25:44.454977062Z” level=info msg=“detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf”
Jan 15 18:25:44 ip-172-33-13-185 dockerd[5662]: time=“2026-01-15T18:25:44.554638832Z” level=info msg=“[graphdriver] using prior storage driver: overlay2”
Jan 15 18:25:45 ip-172-33-13-185 dockerd[5662]: time=“2026-01-15T18:25:45.271638005Z” level=info msg=“Loading containers: start.”
Jan 15 18:25:45 ip-172-33-13-185 dockerd[5662]: time=“2026-01-15T18:25:45.648439710Z” level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP
Jan 15 18:25:49 ip-172-33-13-185 dockerd[5662]: time=“2026-01-15T18:25:49.943881480Z” level=info msg=“Loading containers: done.”
Jan 15 18:25:50 ip-172-33-13-185 dockerd[5662]: time=“2026-01-15T18:25:50.026626428Z” level=info msg=“Docker daemon” commit=f9522e5 containerd-snapshotter=false storage-driver=overlay2 version=27.1.2
Jan 15 18:25:50 ip-172-33-13-185 dockerd[5662]: time=“2026-01-15T18:25:50.027422524Z” level=info msg=“metrics API listening on [::]:9323”
Jan 15 18:25:50 ip-172-33-13-185 dockerd[5662]: time=“2026-01-15T18:25:50.029200914Z” level=info msg=“Daemon has completed initialization”
Jan 15 18:25:50 ip-172-33-13-185 systemd[1]: Started Docker Application Container Engine.
– Subject: A start job for unit docker.service has finished successfully
– Defined-By: systemd
– Support: http://www.ubuntu.com/support
– A start job for unit docker.service has finished successfully.
– The job identifier is 776.
Jan 15 18:25:50 ip-172-33-13-185 dockerd[5662]: time=“2026-01-15T18:25:50.210238825Z” level=info msg=“API listen on /run/docker.sock”
Jan 15 18:26:33 ip-172-33-13-185 dockerd[5662]: time=“2026-01-15T18:26:33.387485544Z” level=info msg=“No non-localhost DNS nameservers are left in resolv.conf. Using default external servers”
Jan 15 18:27:03 ip-172-33-13-185 dockerd[5662]: time=“2026-01-15T18:27:03.408720179Z” level=info msg=“ignoring event” container=990d9023d0b52e84608d421aed183e22b2c00c169ea27bf0b6413630bbcf3d4b module=libcontainerd namespace=mo
by topic=/tasks/delete type=“*events.TaskDelete”
Jan 15 18:27:04 ip-172-33-13-185 dockerd[5662]: time=“2026-01-15T18:27:04.336235824Z” level=info msg=“No non-localhost DNS nameservers are left in resolv.conf. Using default external servers”
No errors beyond this point.