How can I troubleshoot apparmor permissions issues from docker-default profile?

General
1

How can apparmor docker-default permission issues be debugged or futher investigated?

I have an ubuntu 16.04 vm running docker 17.06 ce that has been experiencing permissions issues while trying to interact with (stop, kill, exec) containers.

ex:

Error response from daemon: Cannot stop container XXX: Cannot kill container XXXXXXX: rpc error: code = 7 desc = permission denied
OR
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:1370: sending signal 0 to pid 1889 caused "permission denied"

I’ve already gone through the steps of trying to change the volume driver to overlay2 from aufs after seeing similar errors be found to be an issue with aufs volume driver to see if it resolves the problem but it does not.

This setup is a mac mini running vmware esxi and a vm running ubuntu 16.04. If I reboot the vm the containers startup and are running but when I try stop them or enter them I get these permissions errors. If I run dmesg I see numerous apparmor DENIED messages indicating they are coming from docker-default profile, but I don’t know how to investigate further to see what specific problem is causing this - i.e. what in the docker-default profile is telling apparmor that the events should be denied?

ex:

[2117079.944187] audit: type=1400 audit(1502636750.631:38181768): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=3287 comm="monit" requested_mask="tracedby" denied_mask="tracedby" peer="docker-default"
AND
[2117404.088531] audit: type=1400 audit(1502637074.772:38187445): apparmor="DENIED" operation="signal" profile="docker-default" pid=85049 comm="monit" requested_mask="receive" denied_mask="receive" signal=usr1 peer="docker-default"

Any clues or how to dig deeper here?

The problem is not limited to a single container.

Has anyone experienced similar problems on this setup with apple hardware, vmesxi 6.0, ubuntu 16 vm running docker 17.06 ce?

2

Hi! I am running into the same issues. Did you manage to get any solution to those errors?
Thanks in advance,
George.

3

I have not had any new discoveries on the ability to troubleshoot the apparmor permissions issues directly, but here are some notes about what I have done so far for what it is worth:

I created a new VM on my ESXi server (host hardware is Mac mini) but this time with Ubuntu 14.04 instead of 16.04 and haven’t come across the same issue (yet). I don’t know if the Ubuntu version is the relevant difference here or if it is something else.

The only other thing I can think of that I was doing with the other VM that I haven’t yet done in this new VM is to use the docker api and issue docker and docker-compose commands from a remote machine (development machine). I’ve thought perhaps something about the way I was using docker/docker-compose via the docker api remotely was the catalyst for the permissions problem but I haven’t yet found a way to troubleshoot the apparmor permissions further in order to make the determination one way or another what the source of the problem is. I’ve been a little gunshy to enable and start executing remote docker api docker/docker-compose commands in the new VM for fear that this may be what ends up “corrupting” the VM/docker permissions, but perhaps I will setup a new docker VM for just testing those scenarios to whittle down the problem.

Ultimately it would be helpful if someone can explain how to best troubleshoot the apparmor docker-default permissions issue so I can narrow down what the problem is that way.