Docker Community Forums

Share and learn in the Docker community.

Docker does not clean up layer files if overlay mount fails

I’m playing around with the Linux Kernel Security module, specifically the security_sb_mount() and security_sb_kern_mount() hooks which are both called when a mount() syscall is executed. In some cases I’ll send back a failure status from one of those hook points when the docker daemon tries to setup an overlay mount for a container instance. Docker seems to behave correctly when the failure status is returned and prints a corresponding message and the container is not launched and it appears that a container instance isn’t created. But the container instance layer is left in place and I would have expected that to be removed and cleaned up. I’d like to find out if this is incorrect behavior or not.

I have a test VM with Centos 7 installed and my 5.2 Linux kernel running with my LSM additions and I do something like:

$ docker -v
Docker version 20.10.1, build 831ebea

$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE

$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

$ sudo ls -Rla /var/lib/docker/overlay2
/var/lib/docker/overlay2:
total 0
drwx------. 3 root root 40 Jan 26 13:43 .
drwx–x--x. 13 root root 167 Jan 26 13:43 …
brw-------. 1 root root 253, 0 Jan 26 13:43 backingFsBlockDev
drwx------. 2 root root 6 Dec 16 18:19 l

/var/lib/docker/overlay2/l:
total 0
drwx------. 2 root root 6 Dec 16 18:19 .
drwx------. 3 root root 40 Jan 26 13:43 …

$ docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete
Digest: sha256:31b9c7d48790f0d8c50ab433d9c3b7e17666d6993084c002c2ff1ca09b96391d
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest

$ sudo ls -Rla /var/lib/docker/overlay2
/var/lib/docker/overlay2:
total 0
drwx------. 4 root root 112 Jan 26 13:44 .
drwx–x--x. 13 root root 167 Jan 26 13:43 …
drwx------. 3 root root 30 Jan 26 13:44 71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602
brw-------. 1 root root 253, 0 Jan 26 13:43 backingFsBlockDev
drwx------. 2 root root 40 Jan 26 13:44 l

/var/lib/docker/overlay2/71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602:
total 4
drwx------. 3 root root 30 Jan 26 13:44 .
drwx------. 4 root root 112 Jan 26 13:44 …
drwxr-xr-x. 2 root root 19 Jan 26 13:44 diff
-rw-r–r--. 1 root root 26 Jan 26 13:44 link

/var/lib/docker/overlay2/71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602/diff:
total 16
drwxr-xr-x. 2 root root 19 Jan 26 13:44 .
drwx------. 3 root root 30 Jan 26 13:44 …
-rwxrwxr-x. 1 root root 13336 Jan 2 2020 hello

/var/lib/docker/overlay2/l:
total 0
drwx------. 2 root root 40 Jan 26 13:44 .
drwx------. 4 root root 112 Jan 26 13:44 …
lrwxrwxrwx. 1 root root 72 Jan 26 13:44 PEZE2QJ6BBK2JLFXMLUEMWUD4N -> …/71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602/diff

$ docker run hello-world
docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/3423644a72818b2230b0f562f5457894d8cff6ea4a1ac9b92e15a0c6abe29c6a-init/merged: permission denied.
See ‘docker run --help’.

–> At this point my LSM has denied the container overlay instance mount and docker failed to start the container instance

$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest bf756fb1ae65 12 months ago 13.3kB

$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

–> It appears that no container instance is defined, but in the following output you can see the container specific instance layer (3423644a72818b2230b0f562f5457894d8cff6ea4a1ac9b92e15a0c6abe29c6a-init) is still present along with the soft link to it under the “l” directory.

$ sudo ls -Rla /var/lib/docker/overlay2
/var/lib/docker/overlay2:
total 0
drwx------. 5 root root 189 Jan 26 13:45 .
drwx–x--x. 13 root root 167 Jan 26 13:43 …
drwx------. 4 root root 55 Jan 26 13:45 3423644a72818b2230b0f562f5457894d8cff6ea4a1ac9b92e15a0c6abe29c6a-init
drwx------. 3 root root 47 Jan 26 13:45 71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602
brw-------. 1 root root 253, 0 Jan 26 13:43 backingFsBlockDev
drwx------. 2 root root 74 Jan 26 13:45 l

/var/lib/docker/overlay2/3423644a72818b2230b0f562f5457894d8cff6ea4a1ac9b92e15a0c6abe29c6a-init:
total 8
drwx------. 4 root root 55 Jan 26 13:45 .
drwx------. 5 root root 189 Jan 26 13:45 …
drwxr-xr-x. 2 root root 6 Jan 26 13:45 diff
-rw-r–r--. 1 root root 26 Jan 26 13:45 link
-rw-r–r--. 1 root root 28 Jan 26 13:45 lower
drwx------. 2 root root 6 Jan 26 13:45 work

/var/lib/docker/overlay2/3423644a72818b2230b0f562f5457894d8cff6ea4a1ac9b92e15a0c6abe29c6a-init/diff:
total 0
drwxr-xr-x. 2 root root 6 Jan 26 13:45 .
drwx------. 4 root root 55 Jan 26 13:45 …

/var/lib/docker/overlay2/3423644a72818b2230b0f562f5457894d8cff6ea4a1ac9b92e15a0c6abe29c6a-init/work:
total 0
drwx------. 2 root root 6 Jan 26 13:45 .
drwx------. 4 root root 55 Jan 26 13:45 …

/var/lib/docker/overlay2/71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602:
total 4
drwx------. 3 root root 47 Jan 26 13:45 .
drwx------. 5 root root 189 Jan 26 13:45 …
-rw-------. 1 root root 0 Jan 26 13:45 committed
drwxr-xr-x. 2 root root 19 Jan 26 13:44 diff
-rw-r–r--. 1 root root 26 Jan 26 13:44 link

/var/lib/docker/overlay2/71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602/diff:
total 16
drwxr-xr-x. 2 root root 19 Jan 26 13:44 .
drwx------. 3 root root 47 Jan 26 13:45 …
-rwxrwxr-x. 1 root root 13336 Jan 2 2020 hello

/var/lib/docker/overlay2/l:
total 0
drwx------. 2 root root 74 Jan 26 13:45 .
drwx------. 5 root root 189 Jan 26 13:45 …
lrwxrwxrwx. 1 root root 77 Jan 26 13:45 22LHILLBSDYQK3GVPXCF2SKGNM -> …/3423644a72818b2230b0f562f5457894d8cff6ea4a1ac9b92e15a0c6abe29c6a-init/diff
lrwxrwxrwx. 1 root root 72 Jan 26 13:44 PEZE2QJ6BBK2JLFXMLUEMWUD4N -> …/71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602/diff

This seems like incorrect behavior, especially since the “docker ps -a” command doesn’t show any containers. Am I just missing something here?