All of a sudden docker.service Failed to start - Fedora 37

Docker Engine General
1

I rebooted my fedora 37 box, fired terminal and wanted to check my running containers
~]$ docker ps -a
I got

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

When I checked the status of docker.service, it has failed to start as follows

docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─docker.conf
Active: failed (Result: exit-code) since Wed 2023-10-11 16:44:38 CEST; 3min 33s ago
TriggeredBy: × docker.socket
Docs: https://docs.docker.com
Process: 1900 ExecStart=/usr/bin/dockerd (code=exited, status=1/FAILURE)
Main PID: 1900 (code=exited, status=1/FAILURE)
CPU: 128ms

Oct 11 16:44:36… systemd[1]: Failed to start docker.service - Docker Application Container Engine.
Oct 11 16:44:38 … systemd[1]: docker.service: Scheduled restart job, restart counter is at 4.
Oct 11 16:44:38 … systemd[1]: Stopped docker.service - Docker Application Container Engine.
Oct 11 16:44:38 … systemd[1]: docker.service: Start request repeated too quickly.
Oct 11 16:44:38… systemd[1]: docker.service: Failed with result ‘exit-code’.
Oct 11 16:44:38 … systemd[1]: Failed to start docker.service - Docker Application Container Engine.

After googling a little bit, I did
~]$ sudo dockerd --debug

INFO[2023-10-11T18:59:02.391622369+02:00] Starting up
DEBU[2023-10-11T18:59:02.393816461+02:00] Listener created for HTTP on unix (/var/run/docker.sock)
INFO[2023-10-11T18:59:02.394716857+02:00] detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf
DEBU[2023-10-11T18:59:02.395578854+02:00] Golang’s threads limit set to 41490
DEBU[2023-10-11T18:59:02.396351251+02:00] metrics API listening on /var/run/docker/metrics.sock
DEBU[2023-10-11T18:59:02.402030530+02:00] Using default logging driver json-file
DEBU[2023-10-11T18:59:02.402497928+02:00] No quota support for local volumes in /var/lib/docker/volumes: Filesystem does not support, or has not enabled quotas
DEBU[2023-10-11T18:59:02.403559024+02:00] processing event stream module=libcontainerd namespace=plugins.moby
INFO[2023-10-11T18:59:02.455617628+02:00] [graphdriver] trying configured driver: overlay2
DEBU[2023-10-11T18:59:02.455831628+02:00] daemon configured with a 15 seconds minimum shutdown timeout
DEBU[2023-10-11T18:59:02.455869727+02:00] start clean shutdown of all containers with a 15 seconds timeout…
DEBU[2023-10-11T18:59:02.457120623+02:00] Cleaning up old mountid : start.
DEBU[2023-10-11T18:59:02.457963020+02:00] Cleaning up old mountid : done.
failed to start daemon: error initializing graphdriver: overlay2: unknown option overlay2.override_kernel_check: overlay2

For last line, this is the 1st time I see this. I tried to search for log files but not sure where is the location

2

You indentified the most important line in the error message. That’s the one you need to search for in these cases. I searched for “unknown option: overlay2.override_kernel_check: overlay2” and found this:

It seems you have the same issue.

In short: Your docker version was probably updated and your existing daemon config is not compatible with it.

3

Thanks riemelek, I do regular update for my fedora boxes, last one was last week buty I dont remember that Docker was updated. That is why I am looking for Docker engine log to see if it was updated.

4

I don’t think Docker Engine log would tell you that. You could try to find a system log. For example if I’m not mistaken, Debian based distros has /var/log/apt/term.log for APT. I don’t know about Fedora.

5

Yes, it was the same issue, I removed

 "storage-opts": [
                "overlay2.override_kernel_check=true"
        ],

from daemon.json restsarted docker and was abl;e to issue docker commands as usual

So many thanks for your help