All of a sudden docker.service Failed to start - Fedora 37

I rebooted my fedora 37 box, fired terminal and wanted to check my running containers
~]$ docker ps -a
I got

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

When I checked the status of docker.service, it has failed to start as follows

docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
Active: failed (Result: exit-code) since Wed 2023-10-11 16:44:38 CEST; 3min 33s ago
TriggeredBy: × docker.socket
Process: 1900 ExecStart=/usr/bin/dockerd (code=exited, status=1/FAILURE)
Main PID: 1900 (code=exited, status=1/FAILURE)
CPU: 128ms

Oct 11 16:44:36… systemd[1]: Failed to start docker.service - Docker Application Container Engine.
Oct 11 16:44:38 … systemd[1]: docker.service: Scheduled restart job, restart counter is at 4.
Oct 11 16:44:38 … systemd[1]: Stopped docker.service - Docker Application Container Engine.
Oct 11 16:44:38 … systemd[1]: docker.service: Start request repeated too quickly.
Oct 11 16:44:38… systemd[1]: docker.service: Failed with result ‘exit-code’.
Oct 11 16:44:38 … systemd[1]: Failed to start docker.service - Docker Application Container Engine.

After googling a little bit, I did
~]$ sudo dockerd --debug

INFO[2023-10-11T18:59:02.391622369+02:00] Starting up
DEBU[2023-10-11T18:59:02.393816461+02:00] Listener created for HTTP on unix (/var/run/docker.sock)
INFO[2023-10-11T18:59:02.394716857+02:00] detected nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf
DEBU[2023-10-11T18:59:02.395578854+02:00] Golang’s threads limit set to 41490
DEBU[2023-10-11T18:59:02.396351251+02:00] metrics API listening on /var/run/docker/metrics.sock
DEBU[2023-10-11T18:59:02.402030530+02:00] Using default logging driver json-file
DEBU[2023-10-11T18:59:02.402497928+02:00] No quota support for local volumes in /var/lib/docker/volumes: Filesystem does not support, or has not enabled quotas
DEBU[2023-10-11T18:59:02.403559024+02:00] processing event stream module=libcontainerd namespace=plugins.moby
INFO[2023-10-11T18:59:02.455617628+02:00] [graphdriver] trying configured driver: overlay2
DEBU[2023-10-11T18:59:02.455831628+02:00] daemon configured with a 15 seconds minimum shutdown timeout
DEBU[2023-10-11T18:59:02.455869727+02:00] start clean shutdown of all containers with a 15 seconds timeout…
DEBU[2023-10-11T18:59:02.457120623+02:00] Cleaning up old mountid : start.
DEBU[2023-10-11T18:59:02.457963020+02:00] Cleaning up old mountid : done.
failed to start daemon: error initializing graphdriver: overlay2: unknown option overlay2.override_kernel_check: overlay2

For last line, this is the 1st time I see this. I tried to search for log files but not sure where is the location

You indentified the most important line in the error message. That’s the one you need to search for in these cases. I searched for “unknown option: overlay2.override_kernel_check: overlay2” and found this:

It seems you have the same issue.

In short: Your docker version was probably updated and your existing daemon config is not compatible with it.

Thanks riemelek, I do regular update for my fedora boxes, last one was last week buty I dont remember that Docker was updated. That is why I am looking for Docker engine log to see if it was updated.

I don’t think Docker Engine log would tell you that. You could try to find a system log. For example if I’m not mistaken, Debian based distros has /var/log/apt/term.log for APT. I don’t know about Fedora.

Yes, it was the same issue, I removed

 "storage-opts": [

from daemon.json restsarted docker and was abl;e to issue docker commands as usual

So many thanks for your help