Moving /var/lib/docker to Amazon EFS

General
1

Hi folks,

I’m running docker on an EC2 instance, and want to move my /var/lib/docker directory to a mounted AWS EFS directory. (don’t have to guess size needed like EBS, pay only for what’s used, backups easier, can share mount between instances, etc)

I know the trick about moving the directory and symlink’ing from old location to new, and that works fine when moving to another EBS device, but it doesn’t work for EFS, because of a single file: /var/lib/docker/overlay2/backingFsBlockDev. This is a special block device file, and apparently EFS doesn’t support those.

I tried moving everything except backingFsBlockDev (i.e. leave that on EBS and symlink from the EFS location to the EBS location; EFS does support symlinks), but Docker fails to start, something about an overlay test failing. Even tried using a bind-mount rather than a symlink, but same result.

Next thought was to find the setting that says where backingFsBlockDev is, again so I can point to that one file remaining somewhere on the EBS drive. But there doesn’t seem to be such a setting; at least not that’s easy to find.

Does anyone know where the constant “/var/lib/docker/overlay2/backingFsBlockDev” is stored? Or any other tricks that would allow /var/lib/docker to be moved over to a filesystem that doesn’t support block device files?

Thanks!

2

I’m also stuck with this trying to get CodeBuild to use EFS as the docker store. @kstillson did you ever solve this directly?

3

alas, no. I concluded it’s not currently possible and gave up. I don’t have the time / level-of-interest to really dig into the code and try to fix it with a patch… :-(…

1 Like
4

@kstillson thanks, I figured as much.

It would be so nice to have docker w/ EFS backed for CodeBuild. Using ECR as a cache is rather slow - relatively speaking and docker layer cache hits don’t happen 100% of the time even using --cache-from.

Not sure why on that last part, was hoping to avoid it by moving docker’s storage to EFS, but now it looks like I’ll have to dive in.

Thanks again.