We have some containers running with the standard Ubuntu image. We wanted to enable auditd to trace something going on INSIDE the container. The service installs, but can’t actually run it. It fails to start and when trying to list rules with auditctl, I get an error that the kernel doesn’t support it. Sure enough, the config shows that config_audit is not set to yes.
Before I keep searching for other images/kernels that might have that set to yes. Is this just something not supported inside a container ? ( since auditd is a kernel level auditing ) If so, any idea how?
( I know auditd can be enabled on the host itself, but I don’t think that will achieve what we need )
Unfortunately you can’t use the same tools inside a container that you did on a host machine or in a virtual machine. Don’t think of a container as a very small virtual machine. Think of it as a simple process that can’t see the whole host just because the kernel doesn’t let it see.
Everything that is “inside” a container is also outside in any other (not containerized) application’s point of view. Sometimes it does matter whether you check something inside the container or outside, but the solution is not running the “old” tools in the container but using something that was meant to use in a container or something that can run on the host but understands containers and can look inside if necessary.
I guess you could force auditd to run in the container with right privileges and capabilities assigned to the container but there is probably a better and easier solution. Since there is only one or maximum a few processes in a container, there must be an easier way
Can you tell us mor about what you want to achieve? Maybe we can suggest another way.
So it’s an Ubuntu image running appserver code. We’ve got a symbolic link that gets created in the dockerfile. And something is removing the symbolic link ( if I run the container just as a shell, I can see that the link IS there ). We’re not sure what and I’m not sure how to audit what is doing it.
An excellent question. I was led to believe that it happened intermittently during runtime. However, now I’m being told they aren’t sure… I’m waiting on actual confirmation
In the a meantime I share why I asked about that. If it disappears immediately, than it could be caused by a mounted folder over the folder of the symbolic link or something could delete it in the entrypoint. If none of those, then you at least know where to look for the deleting part in the app. Or if the symbolic link is in /tmp and it is on tmpfs, that could also hide the link.
If it disappears later, you could try to change the owner of the parent folder so the link can be deleted only by root. It would work only if you could make the parent be owned by root and the process that deletes it is not started by root.
Since I don’t know much about auditd, if you can tell me how that would help, that could give me an idea what else you could do.
So Auditd is a service that runs on the OS that audits resources according to set rules.
So, we can set a rule that checks for create/delete/modify of symbolic links and it would tells us if that happens and hopefully what process is doing it.
As you say though, might be a process that is too intensive and kernel-bound to run inside a docker container.
Thank you for the explanation. Since everything is available from the host, maybe you could run auditd on the host and try to watch what happens in the “Merged dir” of the container which is what you would see in the container as system root.
You could also try UpperDir and WorkDir ez well instead of MergedDir to check added and deleted files, but I am not sure how it would work since this is a special filesystem (most likely overlay2)…
I als realized I didn’t finish one of my previous suggestions:
I wanted to say that if you a process want’s to delete the link without permissions, it could throw an error that you can see in the logs.