Docker fails to start after creating daemon.json with using the data-root configuration option

Docker Engine General
1

I have been trying to change the default configuration of the docker daemon by changing the data-root flag in the daemon.json file located in /etc/docker/.

Following the documentation on Custom Docker daemon options, I have added the data-root flag in the daemon.json file:

{
    "data-root": "/seq-data/docker-data",
}

where my new storage dir looks like:

drwx–x–x 14 root root 4096 Oct 1 17:57 docker-data

When trying to restart the daemon I get the following error:

sudo systemctl restart docker

Job for docker.service failed because the control process exited with error code. See “systemctl status docker.service” and “journalctl -xe” for details.

systemctl status docker.service

● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Tue 2019-10-01 17:49:39 CEST; 2min 28s ago
Docs: -
Process: 23755 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
Main PID: 23755 (code=exited, status=1/FAILURE)

journalctl -xe does not return any information but I can read from the /var/log/messages

Oct 1 17:57:24 master systemd: Stopping Docker Socket for the API.
Oct 1 17:57:24 master systemd: Starting Docker Socket for the API.
Oct 1 17:57:24 master systemd: Listening on Docker Socket for the API.
Oct 1 17:57:24 master systemd: Starting Docker Application Container Engine…
Oct 1 17:57:24 master dockerd: time=“2019-10-01T17:57:24.815570155+02:00” level=info msg=“Starting up”
Oct 1 17:57:24 master dockerd: time=“2019-10-01T17:57:24.822148476+02:00” level=info msg=“parsed scheme: "unix"” module=grpc
Oct 1 17:57:24 master dockerd: time=“2019-10-01T17:57:24.822197494+02:00” level=info msg=“scheme "unix" not registered, fallback to default scheme” module=grpc
Oct 1 17:57:24 master dockerd: time=“2019-10-01T17:57:24.822242119+02:00” level=info msg=“ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 }] }” module=grpc
Oct 1 17:57:24 master dockerd: time=“2019-10-01T17:57:24.822277440+02:00” level=info msg=“ClientConn switching balancer to "pick_first"” module=grpc
Oct 1 17:57:24 master dockerd: time=“2019-10-01T17:57:24.824072830+02:00” level=info msg=“parsed scheme: "unix"” module=grpc
Oct 1 17:57:24 master dockerd: time=“2019-10-01T17:57:24.824110334+02:00” level=info msg=“scheme "unix" not registered, fallback to default scheme” module=grpc
Oct 1 17:57:24 master dockerd: time=“2019-10-01T17:57:24.824150292+02:00” level=info msg=“ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 }] }” module=grpc
Oct 1 17:57:24 master dockerd: time=“2019-10-01T17:57:24.824183367+02:00” level=info msg=“ClientConn switching balancer to "pick_first"” module=grpc
Oct 1 17:57:24 master dockerd: time=“2019-10-01T17:57:24.839333096+02:00” level=warning msg=“Usage of loopback devices is strongly discouraged for production use. Please use --storage-opt dm.thinpooldev or use man dockerd to refer to dm.thinpooldev section.” storage-driver=devicemapper
Oct 1 17:57:24 master dockerd: time=“2019-10-01T17:57:24.868061534+02:00” level=warning msg=“Base device already exists and has filesystem xfs on it. User specified filesystem will be ignored.” storage-driver=devicemapper
Oct 1 17:57:24 master multipathd: dm-6: remove map (uevent)
Oct 1 17:57:24 master multipathd: dm-6: devmap not registered, can’t remove
Oct 1 17:57:24 master multipathd: dm-6: remove map (uevent)
Oct 1 17:57:24 master dockerd: time=“2019-10-01T17:57:24.899058627+02:00” level=info msg=“[graphdriver] using prior storage driver: devicemapper”
Oct 1 17:57:24 master dockerd: time=“2019-10-01T17:57:24.899105841+02:00” level=warning msg=“[graphdriver] WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release”
Oct 1 17:57:24 master dockerd: bolt.Close(): funlock error: function not implemented
Oct 1 17:57:24 master dockerd: failed to start daemon: error while opening volume store metadata database: function not implemented
Oct 1 17:57:24 master systemd: docker.service: main process exited, code=exited, status=1/FAILURE
Oct 1 17:57:24 master systemd: Failed to start Docker Application Container Engine.
Oct 1 17:57:24 master systemd: Unit docker.service entered failed state.
Oct 1 17:57:24 master systemd: docker.service failed.

My guess is that the key is in:

Oct 1 17:57:24 master dockerd: bolt.Close(): funlock error: function not implemented
Oct 1 17:57:24 master dockerd: failed to start daemon: error while opening volume store metadata database: function not implemented

I have tried the solutions suggested in this post with similar issues, but the error seems to be different.

Similarly, when deleting the daemon.json, docker restarts just fine.

 systemctl status docker.service

● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2019-10-01 18:13:20 CEST; 11s ago
Docs: -
Main PID: 24719 (dockerd)
Memory: 42.6M
CGroup: /system.slice/docker.service
└─24719 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

another important aspect might be the storage-driver that I’m using devicemapper

docker info

Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 19.03.3-beta2
Storage Driver: devicemapper
Pool Name: docker-8:3-201379873-pool
Pool Blocksize: 65.54kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data file: /dev/loop10
Metadata file: /dev/loop11
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 11.8MB
Data Space Total: 107.4GB
Data Space Available: 24.41GB
Metadata Space Used: 581.6kB
Metadata Space Total: 2.147GB
Metadata Space Available: 2.147GB
Thin Pool Minimum Free Space: 10.74GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.el7_lustre.x86_64
Operating System: Scientific Linux 7.3 (Nitrogen)
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 46.98GiB
Name: master
ID: -
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: -
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use --storage-opt dm.thinpooldev to specify a custom block storage device.

2

I would put my money on: xfs formated without d_type=1 flag, see: https://docs.docker.com/storage/storagedriver/overlayfs-driver/

Can you share the output of df /seq-data/docker-data?

3

Thanks for your response.

df /seq_data/docker-data

Filesystem                   1K-blocks        Used   Available Use% Mounted on
ip@tcp:/seqdata 32032928228 17566006760 12852089344  58% /seq_data
4

Hmm, instead of filesystem 32032928228, I would have expected to see xfs or a lvm volume group.
What type of filesystem is it?

Since the output of dfdidn’t provide for what I was looking for, can you provide the ouput of xfs_info /seq_data

5

I see what you mean, the problem must arise from the fact that we are using two different file systems.
df -Th

Filesystem                 Type      Size  Used Avail Use% Mounted on
/dev/sda3                  xfs        47G   24G   23G  52% /
devtmpfs                   devtmpfs   24G     0   24G   0% /dev
tmpfs                      tmpfs      24G     0   24G   0% /dev/shm
tmpfs                      tmpfs      24G   26M   24G   1% /run
tmpfs                      tmpfs      24G     0   24G   0% /sys/fs/cgroup
/dev/sda2                  xfs       269G   82G  188G  31% /home
/dev/sda1                  lustre    374G  583M  348G   1% /mdt-mds
/dev/mapper/seqdata        lustre     30T   17T   12T  58% /ost
ip@tcp:/seqdata            lustre     30T   17T   12T  58% /seq_data
tmpfs                      tmpfs     4.7G     0  4.7G   0% /run/user/1029

Currently, Docker is using /dev/sda3 which is of type xfs, and I’m trying to move it to ip@tcp:/seqdatalustre which is of type lustre.

6

This is definetely the issue I have tried a different location in the /dev/sda2 which uses the same xfs file system, e.g. I modified the daemon.json file to:

{
    "data-root": "/home/pmonteagudo/docker-data"
}

the docker daemon restarted just fine. I also confirmed that the data-root has been succesfully modified using the docker info command.

Now my new question: is there is any walk around to change the root-dir to a location with a different file system?

7

Only a hand full of backing filesystems are supported, see: https://docs.docker.com/storage/storagedriver/select-storage-driver/#supported-backing-filesystems