Docker won't start

[sakos65 (Ákos Szommer)]

Hello,

I have a docker installed an OMV system. Recently docker stopped working. It won’t start.

root@nas:/# systemctl restart docker
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xeu docker.service" for details.

root@nas:/# systemctl status docker.service
× docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/docker.service.d
             └─waitAllMounts.conf
     Active: failed (Result: exit-code) since Wed 2024-10-23 20:25:40 CEST; 2min 15s ago
   Duration: 2min 12.243s
TriggeredBy: × docker.socket
       Docs: https://docs.docker.com
    Process: 66114 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=2)
   Main PID: 66114 (code=exited, status=2)
        CPU: 1.639s

okt 23 20:25:38 nas systemd[1]: docker.service: Consumed 1.639s CPU time.
okt 23 20:25:40 nas systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
okt 23 20:25:40 nas systemd[1]: Stopped docker.service - Docker Application Container Engine.
okt 23 20:25:40 nas systemd[1]: docker.service: Consumed 1.639s CPU time.
okt 23 20:25:40 nas systemd[1]: docker.service: Start request repeated too quickly.
okt 23 20:25:40 nas systemd[1]: docker.service: Failed with result 'exit-code'.
okt 23 20:25:40 nas systemd[1]: Failed to start docker.service - Docker Application Container Engine.
okt 23 20:25:46 nas systemd[1]: docker.service: Start request repeated too quickly.
okt 23 20:25:46 nas systemd[1]: docker.service: Failed with result 'exit-code'.
okt 23 20:25:46 nas systemd[1]: Failed to start docker.service - Docker Application Container Engine.

root@nas:/# journalctl -xeu docker.service
░░ Automatic restarting of the unit docker.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
okt 23 20:25:40 nas systemd[1]: Stopped docker.service - Docker Application Container Engine.
░░ Subject: A(z) docker.service egység befejezte a leállást
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A(z) docker.service egység befejezte a leállást.
okt 23 20:25:40 nas systemd[1]: docker.service: Consumed 1.639s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit docker.service completed and consumed the indicated resources.
okt 23 20:25:40 nas systemd[1]: docker.service: Start request repeated too quickly.
okt 23 20:25:40 nas systemd[1]: docker.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit docker.service has entered the 'failed' state with result 'exit-code'.
okt 23 20:25:40 nas systemd[1]: Failed to start docker.service - Docker Application Container Engine.
░░ Subject: A(z) docker.service egység hibát jelzett
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A(z) docker.service egység hibát jelzett.
░░ 
░░ Az eredmény: failed.
okt 23 20:25:46 nas systemd[1]: docker.service: Start request repeated too quickly.
okt 23 20:25:46 nas systemd[1]: docker.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit docker.service has entered the 'failed' state with result 'exit-code'.
okt 23 20:25:46 nas systemd[1]: Failed to start docker.service - Docker Application Container Engine.
░░ Subject: A(z) docker.service egység hibát jelzett
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A(z) docker.service egység hibát jelzett.
░░ 
░░ Az eredmény: failed.

If I configure a new empty docker storage folder then docker start hence I assume something in the original docker storage folder got corrupted. fdisk finds no error.
How could I recover docker with my existing containers?

[meyay (Metin Y.)]

The error logs are not really helpful. Though, we so see that OMV doesn’t seem to use vanilla docker-ce from the Docker repositories, but instead uses the Debian package from the Debian repositories, which as the output of journalctl -xeu docker.service indicates is supported by Debians support channels. We have no idea if, where and how a specific non-vanilla docker distribution deviates from vanilla docker-ce.

Personally, I would ask the OMV forum, cause it is high likely you are not the first OMV user to experience the problem.

[rimelek (Ákos Takács)]

What I can still say about the error message in general is that when you see “Start request repeated too quickly.” it means the systemd service already restarted so many times you don’t even see the original error message since the current issue is that it can’t start again yet. You should scroll up as much as you can in the journalctl output or use the --since option and possibly the --lines as well to get older logs.

What I saw usually is that when the “repeated too quickly” error message appeared, there was a syntax error in the daemon json, but unless your daemon.json is in the “storage folder”, that is probably not what happened in your case.

If you don’t find useful error messages, stop the docker daemon instead of restarting and run the command defined in the systemd service manually. Or if you are not sure what to run, wait enough so the systemd service can start again, start Docker and when it fails, it will probably not restart. But to be safe you can run systemctl stop docker docker.socket as well.

[sakos65 (Ákos Szommer)]

@sakos65

journalctl -xeu docker.service

Hi Ákos,
Thanks for your support. I see your point, however could not manage journalctl to show more lines neither with --since nor with --lines, -n switches. Anyway, probably this chunk below helps (?)

journalctl -xe --since "2024-10-24 08:30:00" -u  docker.service 
okt 24 08:44:02 nas dockerd[86805]:         /root/build-deb/engine/daemon/daemon_unix.go:850 +0x131
okt 24 08:44:02 nas dockerd[86805]: github.com/docker/docker/daemon.(*Daemon).restore(0xc00011db88, 0xc0001a8008)
okt 24 08:44:02 nas dockerd[86805]:         /root/build-deb/engine/daemon/daemon.go:581 +0x67b
okt 24 08:44:02 nas dockerd[86805]: github.com/docker/docker/daemon.NewDaemon({0x55de2679a998, 0xc000180500}, 0xc0001a9088, 0xc0007376e0, 0xc00072b380)
okt 24 08:44:02 nas dockerd[86805]:         /root/build-deb/engine/daemon/daemon.go:1246 +0x393a
okt 24 08:44:02 nas dockerd[86805]: main.(*DaemonCli).start(0xc000754000, 0xc000521f00)
okt 24 08:44:02 nas dockerd[86805]:         /root/build-deb/engine/cmd/dockerd/daemon.go:260 +0xe09
okt 24 08:44:02 nas dockerd[86805]: main.runDaemon(...)
okt 24 08:44:02 nas dockerd[86805]:         /root/build-deb/engine/cmd/dockerd/docker_unix.go:13
okt 24 08:44:02 nas dockerd[86805]: main.newDaemonCommand.func1(0xc000228f00?, {0xc0007360f0?, 0x7?, 0x55de25b5b438?})
okt 24 08:44:02 nas dockerd[86805]:         /root/build-deb/engine/cmd/dockerd/docker.go:37 +0x94
okt 24 08:44:02 nas dockerd[86805]: github.com/spf13/cobra.(*Command).execute(0xc0002efb08, {0xc000140010, 0x3, 0x3})
okt 24 08:44:02 nas dockerd[86805]:         /root/build-deb/engine/vendor/github.com/spf13/cobra/command.go:985 +0xaca
okt 24 08:44:02 nas dockerd[86805]: github.com/spf13/cobra.(*Command).ExecuteC(0xc0002efb08)
okt 24 08:44:02 nas dockerd[86805]:         /root/build-deb/engine/vendor/github.com/spf13/cobra/command.go:1117 +0x3ff
okt 24 08:44:02 nas dockerd[86805]: github.com/spf13/cobra.(*Command).Execute(...)
okt 24 08:44:02 nas dockerd[86805]:         /root/build-deb/engine/vendor/github.com/spf13/cobra/command.go:1041
okt 24 08:44:02 nas dockerd[86805]: main.main()
okt 24 08:44:02 nas dockerd[86805]:         /root/build-deb/engine/cmd/dockerd/docker.go:106 +0x17b
okt 24 08:44:02 nas systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ An ExecStart= process belonging to unit docker.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 2.
okt 24 08:44:02 nas systemd[1]: docker.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit docker.service has entered the 'failed' state with result 'exit-code'.
okt 24 08:44:02 nas systemd[1]: Failed to start docker.service - Docker Application Container Engine.
░░ Subject: A(z) docker.service egység hibát jelzett
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A(z) docker.service egység hibát jelzett.
░░ 
░░ Az eredmény: failed.
okt 24 08:44:02 nas systemd[1]: docker.service: Consumed 1.607s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit docker.service completed and consumed the indicated resources.

[rimelek (Ákos Takács)]

The error messages said nothing to me, but I searched for the following part which looked uniqe enough

@sakos65

/root/build-deb/engine/vendor/github.com/spf13/cobra/command.go

And found this (interestingly the same title):

Docker won't start.

Which suggests that there was a power failure and the volume db file was corrupted. It cold be something else in your case, but you cans till try to delete that db file. You will probably need to recreate the networks, but if you had compose files or you described the container deployment as code in any way, it will be easy… Before you remove the file, copy it to somewhere so you can copy it back if it was not the issue. I remember network issues when I recommended deleting that file, but I haven’t tried deleting it for a long time so at the moent I am not 100% sure what happens after. Here is where I wrote about that file:

Network is already using parent interface

It means you previously created tha network with the id “ce3efa4b0a15”. Now Docker thinks that network still exists. Here is what I would try in addition to what you have already tried: Restart the docker daemon Check if you have an other installation. For example a rootless docker. I am not sure you could create macvlan with that, but it is worth a try to check it. Since you wrote you have already deleted all networks, it shouldn’t be a problem, but maybe Docker still has the information abou…

[sakos65 (Ákos Szommer)]

@meyay

journalctl -xeu docker.service

Bingo! Deleting the local-kv.db file in the /var/lib/docker/network/files folder did the trick. Docker is running with the old containers. I did not mention that the issue was caused by a power outage in my case as well.

Thanks again for your quick and valuable help and your time.