Docker engine panic on startup (Ubuntu 5:25.0.4-1~ubuntu.22.04~jammy)

Open Source Projects DockerEngine
1

My machine had a hard crash yesterday, and the Docker engine now no longer starts on my machine (though maybe that timing is unrelated) and gives a panic on startup. The full log is 2400+ lines long, but here are the first few lines:

Mar 7 10:59:56 myhost systemd[1]: Starting Docker Socket for the API…
Mar 7 10:59:56 myhost systemd[1]: Listening on Docker Socket for the API.
Mar 7 10:59:56 myhost systemd[1]: Starting Docker Application Container Engine…
Mar 7 10:59:56 myhost dockerd[305581]: time=“2024-03-07T10:59:56.307023210-07:00” level=info msg=“Starting up”
Mar 7 10:59:56 myhost dockerd[305581]: time=“2024-03-07T10:59:56.407815492-07:00” level=info msg=“[graphdriver] using prior storage driver: overlay2”
Mar 7 10:59:56 myhost dockerd[305581]: fatal error: runtime: out of memory
Mar 7 10:59:56 myhost dockerd[305581]: runtime stack:
Mar 7 10:59:56 myhost dockerd[305581]: runtime.throw({0x556af479ab4b?, 0x7fa51842b000?})
Mar 7 10:59:56 myhost dockerd[305581]: #011/usr/local/go/src/runtime/panic.go:1077 +0x5e fp=0x7fffdc215ac8 sp=0x7fffdc215a98 pc=0x556af2b6df5e
Mar 7 10:59:56 myhost dockerd[305581]: runtime.sysMapOS(0xc001400000, 0x1f000400000?)
Mar 7 10:59:56 myhost dockerd[305581]: #011/usr/local/go/src/runtime/mem_linux.go:167 +0x11b fp=0x7fffdc215b10 sp=0x7fffdc215ac8 pc=0x556af2b4b67b
Mar 7 10:59:56 myhost dockerd[305581]: runtime.sysMap(0x556af67d0100?, 0x556af2b61220?, 0x556af67e02c8?)
Mar 7 10:59:56 myhost dockerd[305581]: #011/usr/local/go/src/runtime/mem.go:155 +0x34 fp=0x7fffdc215b40 sp=0x7fffdc215b10 pc=0x556af2b4b0b4
Mar 7 10:59:56 myhost dockerd[305581]: runtime.(*mheap).grow(0x556af67d0100, 0xf800001?)

I don’t know why I would get “fatal error: runtime: out of memory” as an error. This machine still has 50GB of memory that’s untouched. It’s also worth noting that the it appears that the startup process attempts to start dockerd three different times, and the out-of-memory error doesn’t seem to show up each time.

A few things I tried:

  • Using dpkg to verify that no packages have corrupt files
  • Uninstalling and re-installing docker-ce and related dependencies (i.e., remove, autoremove, re-install)
  • Verifying that podman and/or docker-desktop can run containers, although with the issues/differences that accompany them.

Thoughts & suggestions?

Here’s a link to the logs: dockerd logs · GitHub

2

In investigating, I found that the out of memory error occurs when the following mmap call is made:

mmap(0xc001400000, 2130307973120, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)

I worked backwards to see what file it was trying to mmap and found it was the following:

/var/lib/docker/image/overlay2/layerdb/mounts/f7578af4440fd96d6a246410497d591d4274a231de8cc8b6586b 7edea8728165/init-id

Inspecting that file showed the following:

-rw-r--r-- 1 root root 2130303782912 Feb 16 08:45 /var/lib/docker/image/overlay2/layerdb/mounts/f7578af4440fd96d6a246410497d591d4274a231de8cc8b6586b7edea8728165/init-id

Looking at the file size (which is incorrect), the mmap’d space is approximately 4 MB larger, suggesting the two are related.

I went ahead and deleted the image layer and then docker started up perfectly.

As a note, here’s how I identified the failing mmap call:

strace -o /tmp/strace-docker -f -ff -s 512 start-stop-daemon --start --background --no-close --exec /usr/bin/dockerd --pidfile /var/run/docker-ssd.pid --make-pidfile -- -p /var/run/docker.pid >> /var/log/docker.log2>&1

I then grep’d /tmp/strace-docker* for “out of memory” and then traced back in the log to the prior open call.

I hope this helps someone!