Perplexing -v mount issue: /usr cannot be mounted?

General
,
1

So I believe my /usr/ is haunted.

On my host arm64 mac, I can do these commands:

$ ls -l /usr/local/go/bin/go 
-rwxr-xr-x    1 root   wheel  13761360 Feb 10 18:47 /usr/local/go/bin/go

$ ls -l /usr/local
total 0
drwxrwxr-x  988 johan  admin           31616 May 30 09:47 bin
drwxrwxr-x    2 johan  admin           64 Feb 11 16:07 Caskroom
drwxrwxr-x  119 johan  admin           3808 Apr  3 12:22 Cellar
drwxrwxr-x   21 johan  Domain Users    672 Feb 11 17:53 etc
drwxrwxr-x    3 johan  admin           96 Feb 11 16:25 Frameworks
drwxr-xr-x   18 root   wheel           576 Feb 10 18:44 go
drwxr-xr-x   23 johan  admin           736 May 20 11:27 Homebrew
drwxrwxr-x  145 johan  admin           4640 Mar 22 01:46 include
drwxr-xr-x    3 root   wheel           96 Dec 19 10:19 jamf
drwxrwxr-x  469 johan  admin           15008 Mar 22 01:46 lib
drwxr-xr-x    3 root   wheel           96 Aug 29  2024 libexec
drwxrwxr-x  149 johan  admin           4768 Apr  3 12:22 opt
drwxrwxr-x   10 johan  admin           320 Feb 11 16:25 sbin
drwxrwxr-x   47 johan  admin           1504 Mar 22 01:46 share
drwxrwxr-x   10 johan  Domain Users    320 Jun  4 17:21 var

If I mount /usr into a container and try to repeat the commands, we get errors. (we mounted /usr ā†’ /var/foo, so we update the command a bit)

$ docker run --rm -it -v /usr:/var/foo debian:bookworm ls -l /var/foo/local/go/bin/go 

ls: cannot access '/var/foo/local/go/bin/go': No such file or directory

So letā€™s grab a shell and investigate. /var/foo is there (bookwork does not have this directory normally)

$ docker run --rm -it -v /usr:/var/foo debian:bookworm
root@662d686adb6c:/# ls -l /var/foo/local
total 36
drwxr-xr-x 2 root root 169 Mar 26 21:18 bin
drwx--x--x 4 root root  80 Jun  5 16:59 containerd.opt
drwxr-xr-x 2 root root  27 Mar 17 00:00 etc
drwxr-xr-x 2 root root  27 Mar 17 00:00 games
drwxr-xr-x 2 root root  27 Mar 17 00:00 include
drwxr-xr-x 3 root root  49 Mar 26 21:18 lib
lrwxrwxrwx 1 root root   9 Mar 26 21:18 man -> share/man
drwxr-xr-x 2 root root  27 Mar 17 00:00 sbin
drwxr-xr-x 4 root root  69 Mar 26 21:18 share
drwxr-xr-x 2 root root  27 Mar 17 00:00 src
drwxr-xr-x 3 root root  60 Jun  5 17:37 var


root@662d686adb6c:/# ls -l /usr/local
total 32
drwxr-xr-x 2 root root 4096 May 20 00:00 bin
drwxr-xr-x 2 root root 4096 May 20 00:00 etc
drwxr-xr-x 2 root root 4096 May 20 00:00 games
drwxr-xr-x 2 root root 4096 May 20 00:00 include
drwxr-xr-x 2 root root 4096 May 20 00:00 lib
lrwxrwxrwx 1 root root    9 May 20 00:00 man -> share/man
drwxr-xr-x 2 root root 4096 May 20 00:00 sbin
drwxr-xr-x 3 root root 4096 May 20 00:00 share
drwxr-xr-x 2 root root 4096 May 20 00:00 src

So the /usr/local/go directory from my host machine is missing, and containerd.opt has appeared, among many changes. And it isnā€™t just a duplicate of the /usr/local from the same container.

Q: can you help me figure out what directory I /did/ mount, and more importantly: how I can mount my hostā€™s /usr directory into the container.

Thanks!

2

You cannot mount every folder into Docker Desktopā€™s virtual machine. You have to add those folders first in the Docker Desktop settings, but /usr is also a standard folder on Linux. So Docker actually mounted that folder from inside the virtual machine, inside the system container that runs the Docker daemon.

When you mount the Docker socket into a container in Docker Desktop, the same thing happens. The socket is actually mounted from the virtual machine.

3

Just out of curiosity: how much sense does it make to mount the arm64 Mac binaries into a arm64 Linux container?

Assuming the volume mapping works just fine: why would you expect binaries from one os to work in another?

There is not even a guarantee that binaries from a Linux host with a specific distribution version, work inside a container that uses a different distribution or distro version. One of the key aspects of a container is that itā€™s self-contained, as in: it should provide everything required to run the payload.

4

The binaries are just example files. Iā€™m actually after /usr/local/var/log/, but binaries are more commonly recognized and donā€™t need explanation. Both are affected the same way, so binaries work as a proxy for this discussion.

5

Rimelek,

Thanks for your reply but I donā€™t understand. I have two specific areas of ignorance:

1. I have certainly mounted application folders that have not been announced to docker beforehand.

e.g.

docker run --rm -it -v /Users/johan/data:/var/foo debian:bookworm

NVM I see now. All my tests have been mounting directories under /Users which is made available by default. Itā€™s remarkable Iā€™ve made it this long without finding this restriction.

  1. but out of curiosity: where is the /usr directory that ended up being mounted? It wasnā€™t from my host and it wasnā€™t from the bookworm image. And whatever modifications I make end up being persisted, so Iā€™m curious where my test files ended up living.

In any case. Thank you for the explanation!

6

This is how Docker Desktop works:

  • Docker Desktop has a virtual machine
  • There are system containers inside that virtual machine (I believe ran by containerd directly)
  • I donā€™t know how many containers are, but when I last checked at least the daemon ran in a system container
  • The docker daemon sees only the files inside that system container, including /usr.
  • When you mount a file, if it is not in a folder added to Docker Desktop, if it exists, it will be mounted from that system container in the VM.

You can try

docker run --rm -it -v /:/root -w /app bash

and look around inside

1 Like
7

sweet. Thanks for the lesson. I appreciate it.