I am unable to start docker daemon. When I try I get a bunch of errors saying
invalid image sha256:{some sha here}, failed to verify image: sha256{some sha here}
and then one line that says
Error starting daemon: layer does not exist
I suspect the images are corrupted or something—there was an issue the last time this server was shut down. I don’t mind removing the images, but I can’t find any way to do that without getting docker started. I was hoping there was some sort of --skip verify argument, but I can’t find anything like that.
I’ve also just experienced a similar problem after a server crash. When the docker daemon is started, I get an error about a non-existent layer. Is there any way to correct this problem without removing /var/lib/docker?
This is on a test server so I can recreate from scratch, but this would be a service issue if the problem happened on a live server.
I’m using Docker 1.12.6 on Oracle Enterprise Linux 6.8 with UEK4.1 kernel and BTRFS.
Client:
Version: 1.12.6
API version: 1.24
Go version: go1.6.4
Git commit: 1512168
Built: Wed Jan 11 09:49:56 2017
OS/Arch: linux/amd64
time=“2017-05-31T11:11:03.180783093Z” level=debug msg="Trusting 1 certs"
time=“2017-05-31T11:11:03.181916329Z” level=debug msg="docker group found. gid: 498"
time=“2017-05-31T11:11:03.181966855Z” level=debug msg="Listener created for HTTP on unix (/var/run/docker.sock)"
time=“2017-05-31T11:11:03.182174774Z” level=debug msg="Listener created for HTTP on tcp (0.0.0.0:2376)"
time=“2017-05-31T11:11:03.183313730Z” level=info msg="libcontainerd: new containerd process, pid: 49352"
time=“2017-05-31T11:11:03.185645104Z” level=debug msg="libcontainerd: containerd connection state change: CONNECTING"
time=“2017-05-31T11:11:03.185775916Z” level=debug msg="libcontainerd: containerd connection state change: TRANSIENT_FAILURE"
time=“2017-05-31T11:11:03.198742203Z” level=warning msg=“containerd: low RLIMIT_NOFILE changing to max” current=1024 max=4096
time=“2017-05-31T11:11:06.185847663Z” level=debug msg="libcontainerd: containerd connection state change: READY"
time=“2017-05-31T11:14:11.305892763Z” level=debug msg="Using default logging driver json-file"
time=“2017-05-31T11:14:11.306170373Z” level=debug msg=“Golang’s threads limit set to 925830"
time=“2017-05-31T11:14:11.306708939Z” level=debug msg=”[graphdriver] trying provided driver “btrfs”"
time=“2017-05-31T11:14:11.308093033Z” level=debug msg="Using graph driver btrfs"
time=“2017-05-31T11:14:11.569385342Z” level=debug msg="Max Concurrent Downloads: 3"
time=“2017-05-31T11:14:11.569435121Z” level=debug msg="Max Concurrent Uploads: 5"
time=“2017-05-31T11:14:11.838224203Z” level=debug msg="Cleaning up old mountid : start."
time=“2017-05-31T11:14:11.838592715Z” level=fatal msg=“Error starting daemon: layer does not exist”
Yeah, I don’t think there’s a way around this besides removing /var/lib/docker. I’ve had similar things happen even very recently. I would suggest architecting your deployments such that you are not relying on anything in /var/lib/docker (which I think should generally be the case anyway).