I’m playing around with the Linux Kernel Security module, specifically the security_sb_mount() and security_sb_kern_mount() hooks which are both called when a mount() syscall is executed. In some cases I’ll send back a failure status from one of those hook points when the docker daemon tries to setup an overlay mount for a container instance. Docker seems to behave correctly when the failure status is returned and prints a corresponding message and the container is not launched and it appears that a container instance isn’t created. But the container instance layer is left in place and I would have expected that to be removed and cleaned up. I’d like to find out if this is incorrect behavior or not.
I have a test VM with Centos 7 installed and my 5.2 Linux kernel running with my LSM additions and I do something like:
$ docker -v
Docker version 20.10.1, build 831ebea
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ sudo ls -Rla /var/lib/docker/overlay2
/var/lib/docker/overlay2:
total 0
drwx------. 3 root root 40 Jan 26 13:43 .
drwx–x--x. 13 root root 167 Jan 26 13:43 …
brw-------. 1 root root 253, 0 Jan 26 13:43 backingFsBlockDev
drwx------. 2 root root 6 Dec 16 18:19 l
/var/lib/docker/overlay2/l:
total 0
drwx------. 2 root root 6 Dec 16 18:19 .
drwx------. 3 root root 40 Jan 26 13:43 …
$ docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete
Digest: sha256:31b9c7d48790f0d8c50ab433d9c3b7e17666d6993084c002c2ff1ca09b96391d
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest
$ sudo ls -Rla /var/lib/docker/overlay2
/var/lib/docker/overlay2:
total 0
drwx------. 4 root root 112 Jan 26 13:44 .
drwx–x--x. 13 root root 167 Jan 26 13:43 …
drwx------. 3 root root 30 Jan 26 13:44 71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602
brw-------. 1 root root 253, 0 Jan 26 13:43 backingFsBlockDev
drwx------. 2 root root 40 Jan 26 13:44 l
/var/lib/docker/overlay2/71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602:
total 4
drwx------. 3 root root 30 Jan 26 13:44 .
drwx------. 4 root root 112 Jan 26 13:44 …
drwxr-xr-x. 2 root root 19 Jan 26 13:44 diff
-rw-r–r--. 1 root root 26 Jan 26 13:44 link
/var/lib/docker/overlay2/71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602/diff:
total 16
drwxr-xr-x. 2 root root 19 Jan 26 13:44 .
drwx------. 3 root root 30 Jan 26 13:44 …
-rwxrwxr-x. 1 root root 13336 Jan 2 2020 hello
/var/lib/docker/overlay2/l:
total 0
drwx------. 2 root root 40 Jan 26 13:44 .
drwx------. 4 root root 112 Jan 26 13:44 …
lrwxrwxrwx. 1 root root 72 Jan 26 13:44 PEZE2QJ6BBK2JLFXMLUEMWUD4N -> …/71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602/diff
$ docker run hello-world
docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/3423644a72818b2230b0f562f5457894d8cff6ea4a1ac9b92e15a0c6abe29c6a-init/merged: permission denied.
See ‘docker run --help’.
–> At this point my LSM has denied the container overlay instance mount and docker failed to start the container instance
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest bf756fb1ae65 12 months ago 13.3kB
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
–> It appears that no container instance is defined, but in the following output you can see the container specific instance layer (3423644a72818b2230b0f562f5457894d8cff6ea4a1ac9b92e15a0c6abe29c6a-init) is still present along with the soft link to it under the “l” directory.
$ sudo ls -Rla /var/lib/docker/overlay2
/var/lib/docker/overlay2:
total 0
drwx------. 5 root root 189 Jan 26 13:45 .
drwx–x--x. 13 root root 167 Jan 26 13:43 …
drwx------. 4 root root 55 Jan 26 13:45 3423644a72818b2230b0f562f5457894d8cff6ea4a1ac9b92e15a0c6abe29c6a-init
drwx------. 3 root root 47 Jan 26 13:45 71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602
brw-------. 1 root root 253, 0 Jan 26 13:43 backingFsBlockDev
drwx------. 2 root root 74 Jan 26 13:45 l
/var/lib/docker/overlay2/3423644a72818b2230b0f562f5457894d8cff6ea4a1ac9b92e15a0c6abe29c6a-init:
total 8
drwx------. 4 root root 55 Jan 26 13:45 .
drwx------. 5 root root 189 Jan 26 13:45 …
drwxr-xr-x. 2 root root 6 Jan 26 13:45 diff
-rw-r–r--. 1 root root 26 Jan 26 13:45 link
-rw-r–r--. 1 root root 28 Jan 26 13:45 lower
drwx------. 2 root root 6 Jan 26 13:45 work
/var/lib/docker/overlay2/3423644a72818b2230b0f562f5457894d8cff6ea4a1ac9b92e15a0c6abe29c6a-init/diff:
total 0
drwxr-xr-x. 2 root root 6 Jan 26 13:45 .
drwx------. 4 root root 55 Jan 26 13:45 …
/var/lib/docker/overlay2/3423644a72818b2230b0f562f5457894d8cff6ea4a1ac9b92e15a0c6abe29c6a-init/work:
total 0
drwx------. 2 root root 6 Jan 26 13:45 .
drwx------. 4 root root 55 Jan 26 13:45 …
/var/lib/docker/overlay2/71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602:
total 4
drwx------. 3 root root 47 Jan 26 13:45 .
drwx------. 5 root root 189 Jan 26 13:45 …
-rw-------. 1 root root 0 Jan 26 13:45 committed
drwxr-xr-x. 2 root root 19 Jan 26 13:44 diff
-rw-r–r--. 1 root root 26 Jan 26 13:44 link
/var/lib/docker/overlay2/71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602/diff:
total 16
drwxr-xr-x. 2 root root 19 Jan 26 13:44 .
drwx------. 3 root root 47 Jan 26 13:45 …
-rwxrwxr-x. 1 root root 13336 Jan 2 2020 hello
/var/lib/docker/overlay2/l:
total 0
drwx------. 2 root root 74 Jan 26 13:45 .
drwx------. 5 root root 189 Jan 26 13:45 …
lrwxrwxrwx. 1 root root 77 Jan 26 13:45 22LHILLBSDYQK3GVPXCF2SKGNM -> …/3423644a72818b2230b0f562f5457894d8cff6ea4a1ac9b92e15a0c6abe29c6a-init/diff
lrwxrwxrwx. 1 root root 72 Jan 26 13:44 PEZE2QJ6BBK2JLFXMLUEMWUD4N -> …/71e268189625956a4248ab09acc2f7ac54644bec05c550a0f5c8176b37c24602/diff
This seems like incorrect behavior, especially since the “docker ps -a” command doesn’t show any containers. Am I just missing something here?