How to store docker system files in EFS?

General
1

Expected behavior

I expect the docker daemon to start without any error.

Actual behavior

This is the output from the journalctl.

-- Unit docker.service has begun starting up.
Dec 02 11:50:09 ip-10-2-1-250.ap-south-1.compute.internal dockerd[20041]: time="2021-12-02T11:50:09.530641246Z" level=info msg="Starting up"
Dec 02 11:50:09 ip-10-2-1-250.ap-south-1.compute.internal dockerd[20041]: time="2021-12-02T11:50:09.582308288Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Dec 02 11:50:09 ip-10-2-1-250.ap-south-1.compute.internal dockerd[20041]: time="2021-12-02T11:50:09.582906856Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Dec 02 11:50:09 ip-10-2-1-250.ap-south-1.compute.internal dockerd[20041]: time="2021-12-02T11:50:09.583139690Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Dec 02 11:50:09 ip-10-2-1-250.ap-south-1.compute.internal dockerd[20041]: time="2021-12-02T11:50:09.583196030Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Dec 02 11:50:09 ip-10-2-1-250.ap-south-1.compute.internal dockerd[20041]: time="2021-12-02T11:50:09.596461601Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Dec 02 11:50:09 ip-10-2-1-250.ap-south-1.compute.internal dockerd[20041]: time="2021-12-02T11:50:09.596509891Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Dec 02 11:50:09 ip-10-2-1-250.ap-south-1.compute.internal dockerd[20041]: time="2021-12-02T11:50:09.596540532Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Dec 02 11:50:09 ip-10-2-1-250.ap-south-1.compute.internal dockerd[20041]: time="2021-12-02T11:50:09.596557212Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Dec 02 11:50:09 ip-10-2-1-250.ap-south-1.compute.internal dockerd[20041]: time="2021-12-02T11:50:09.708418727Z" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2
Dec 02 11:50:09 ip-10-2-1-250.ap-south-1.compute.internal dockerd[20041]: time="2021-12-02T11:50:09.708480788Z" level=error msg="[graphdriver] prior storage driver overlay2 failed: driver not supported"
Dec 02 11:50:09 ip-10-2-1-250.ap-south-1.compute.internal dockerd[20041]: failed to start daemon: error initializing graphdriver: driver not supported
Dec 02 11:50:09 ip-10-2-1-250.ap-south-1.compute.internal systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Dec 02 11:50:09 ip-10-2-1-250.ap-south-1.compute.internal systemd[1]: Failed to start Docker Application Container Engine.

The last line:

[graphdriver] prior storage driver overlay2 failed: driver not supported

Additional Information

I would like to store my docker files, including volumes, images and container data in EFS mount.

To do that, I have this in my /etc/docker/daemon.json:

{
        "data-root": "/efs/system/docker/data-root",
}

The /efs/system/docker/data-root is located inside a EFS mount on my Amazon Linux, running on EC2.

Steps to reproduce the behavior

  1. Install docker on Amazon Linux 2. Add ec2-user to docker group.
  2. Mount an EFS drive on /efs. Use the command, sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <fs-efsid>.efs.<region>.amazonaws.com:/ /efs
  3. Edit /etc/docker/daemon.json to have the following:
{
        "data-root": "/efs/system/docker/data-root",
}
  1. Try to start the docker daemon.
2

The overlay2 driver that your dockerd is configured to use only supports xfs or ext4 as the backing filesystem type.

Here is the list of supported backing filesystems for the various drivers: Docker storage drivers | Docker Documentation

I don’t recommend using nfs for docker’s storage driver at all. I’d recommend you attach a block disk in AWS and use overlay2 with one of the supported backing systems. If you need docker data to stick around between instances like in the case of an AMI upgrade that replaces the whole EC2 instance, the block device can be detached from the old one and attached to the new one.

NFS can be utilized for persistent volume data since that does not utilize the layered storage driver that gets used for image and container layers.