"Error creating overlay mount" with SELinux enabled

Dear all,

Has this issue ever occured by anyone else? If so, would you be able to help me, please?

I installed Docker on a new machine and try to run a container. When trying to run a container I get the error message that there is an invalid argument when merging. When inspecting the /var/lib/docker/overlay2 folder, I noticed that the folder name of the downloaded image does not match the folder name which is shown in the output of the error message. In the example below the error message stated a folder starting with 8136b1 while with the ls command I see a folder name starting with bc91356 only.

Please notice, that I have SELinux running by adding the "selinux-enabled": true line to /etc/docker/daemon.json and that the /var/lib/docker is a mountpount to a logical volume which is separated from the lv of the kernel.

In the output below I used hello-world, but it happens to all ‘docker run’ commands.

root@server: which docker
/usr/bin/docker
root@server: 
root@server: docker run hello-world
Unable to  find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c1e2c31eb5944: Pull complete
Digest: sha256:d211f4...
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/8136b1...
See 'docker run --help'.
root@server: 
root@server: 
root@server: ls -la /var/lib/docker/overlay2
total 16
drwx--x---,  4 root root 4096 Oct 10 06:53 .
drwx--x---, 14 root root 4096 Oct 10 06:53 ..
drwx--x---,  3 root root 4096 Oct 10 06:53 bc91356...
drwx------,  2 root root 4096 Oct 10 06:53 l
root@server: 

There is no image renaming here. The overlay folder contains the filesĂ­stem layers of images and containers, not the image matadata. Try checking the docke rlogs using journalctl

journalctl -e -u docker

How did you install Docker by the way (share a link to the guide please) and on what operating system?

Hi Rimelek, thank you for your reply and explanation on the renaming.

I installed Docker on Debian 12 which is up to date (ran apt update/upgrade this morning before posting).

For the journal log:

Oct 10 11:09-08 server systemd[1]: Starting docker.service - Docker Application Container Engine...
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="Starting up"
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] parsed scheme: \"unix\"" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] scheme \"unix\": not registered, fallback to default scheme" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] ccResolverWrapper: sending update to cc: {[{unix://run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] ClientConn switching balancer to \"pick_first\"" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] Channel switched balancer to \"pick_first\"" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] Subchannel Connectivity change to CONNECTING" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] Subchannel picks a new address \"unix:///run/containerd/containerd.sock\" to connect" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] Subchannel Connectivity change to READY" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] Channel Connectivity change to READY" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] parsed scheme: \"unix\"" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] scheme \"unix\": not registered, fallback to default scheme" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] ccResolverWrapper: sending update to cc: {[{unix://run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] ClientConn switching balancer to \"pick_first\"" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] Channel switched balancer to \"pick_first\"" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] Subchannel Connectivity change to CONNECTING" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] Subchannel picks a new address \"unix:///run/containerd/containerd.sock\" to connect" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] Subchannel Connectivity change to READY" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[core] Channel Connectivity change to READY" module=grpc
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="[graphdriver] using prior storage driver: overlay2"
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="Loading containers: start."
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="Loading containers: done."
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="Docker daemon" commit=5d6db84 graphdriver(s)=overlay2 version=20.10.24+dfsg1"
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="Daemon has completed initalization"
Oct 10 11:09-08 server systemd[1]: Start docker.service - Docker Application Container Engine.
Oct 10 11:09-08 server dockerd[687]: time="2024-10-10T11:09:09..." level=info msg="API liston on /run/docker.sock"

My Docker install script used by Ansible:

- name: Set Docker's variable for system architecture
  changed_when: false
  no_log: true
  register: architecture_codename
  ansible.builtin.command: 'dpkg --print-architecture'

- name: Install Docker
  block:
    - name: Install Docker's dependencies
      ansible.builtin.apt:
        name: '{{ task_item }}'
        state: present
        update_cache: true
      with_items:
        - ca-certificates
        - curl
        - python3-pip
      loop_control:
        loop_var: task_item

    - name: Add Docker's GPG key
      ansible.builtin.apt_key:
        url: https://download.docker.com/linux/debian/gpg
        keyring: /etc/apt/keyrings/docker.gpg
        state: present

    - name: Add Docker's repository to apt
      no_log: true
      ansible.builtin.apt_repository:
        repo: >-
          deb [arch={{ architecture_codename.stdout }} signed-by=/etc/apt/keyrings/docker.gpg]
          https://download.docker.com/linux/debian {{ ansible_distribution_release | lower }} stable
        state: present

    - name: Install Docker
      ansible.builtin.apt:
        name: '{{ task_item }}'
        state: present
        update_cache: true
      with_items:
        - docker-ce
        - docker
        - docker-compose
      loop_control:
        loop_var: task_item

- name: Make sure the Ansible user can use Docker
  no_log: true
  block:
    - name: Ensure group 'docker' exists
      ansible.builtin.group:
        name: docker
        state: present

    - name: Adding the Ansible user to group 'docker'
      ansible.builtin.user:
        append: true
        groups: docker
        name: '{{ docker.user.ansible.name }}'

    - name: Change Docker's file ownership, group and permissions
      ansible.builtin.file:
        path: /usr/bin/docker
        group: '{{ docker.user.ansible.group }}'
        mode: '0755'
        owner: '{{ docker.user.ansible.name }}'

- name: Check if Docker is active
  ansible.builtin.service:
    name: docker
    state: started
    enabled: true

Before I run this script the SELinux & LV are created/applied. Script is used on multiple other bare-metal & VM’s with no errors.

@vinvar you can find the image hashes themselves under /var/lib/docker/image/overlay2/imagedb/**/sha256



I was diving into some images, looking for the layers within /var/lib/docker, but the hashes on overlay2 and images/** do not seem to match the layers

I tried looking for the layers anywhere
find / | grep c41cccbc36ff3c81
But it finds nothing

I’m guessing this means they’re not stored individually, but bundled together within the image itself
Then, what exactly do the /var/lib/overlay2 store?

I can’t share the exact file structure right now, but it is a complex one. Different kind of IDs, hashes, references in json files, checksums. I followed the traces once and figured it out maybe about 1.5 or 2 years ago. I think I talked about this too, but possibly not in English. Maybe I should.

The overlay2 folder is indeed the location of the image filesystem layers and container filesystem layers since these are the used folders for the “overlay filesystem” which is used by the overlay2 storage driver. Other storage drivers would have a different folder.

The “image” definition is just some metadata files. The tags if I remember correctly are in a separate yml containing all tags. The image metadata has the references to the filesystem layers. But it is a longer story than what I could quickly describe here.

Nothing I can relate together based on the files. But I do see error messages now in the journalctl.

File side:

root@server: docker system prune -a
...

root@server: docker run hello-world
Unable to  find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c1e2c31eb5944: Pull complete
Digest: sha256:d211f4...
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/6e0f9f.../merged: invalid argument.
See 'docker run --help'.

root@server: ls -la /var/lib/docker/overlay2
drwx--x---,  4 root root 4096 Oct 10 15:13 .
drwx---x--, 14 root root 4096 Oct 10 15:11 ..
drwx--x---, 3 root root 4096 Oct 10 15:13 23da8f...
drwx------,  2 root root 4096 Oct 10 15:13 l

root@server: ls -la /var/lib/docker/image/overlay2/imagedb/content/sha256
drwx------,  2 root root 4096 Oct 10 15:13 .
drwx------,  3 root root 4096 Oct 6 13:07 ..
-rw-------, 1 root root 581 Oct 10 15:13 d2c94e258...

root@server: ls -la /var/lib/docker/image/overlay2/imagedb/metadata/sha256
drwx------,  2 root root 4096 Oct 6 13:07 .
drwx------,  3 root root 4096 Oct 6 13:07 ..

Journalctl: (which seems to log errors on the prune command’s timestamp)

... same as in earlier post but with timestamp of start up.
Oct 10 15:11:13 server systemd[1]: Start docker.service - Docker Application Container Engine.
Oct 10 15:11:13 server dockerd[688]: time="2024-10-10T15:11:13..." level=info msg="API listen on on /run/docker.sock"
Oct 10 15:12:46 server dockerd[688]: time="2024-10-10T15:12:46..." level=warning msg="failed to prune image docker.io/ibrary/hello-world@sha:d111f485...: No such image: hello-world@sha:d211f485..."
Oct 10 15:13:01 server dockerd[688]: time="2024-10-10T15:13:01..." level=error msg="error unmounting /var/lib/docker/overlay2/6ef9f24.../merged: invalid argument" storage-driver=overlay2
Oct 10 15:13:01 server dockerd[688]: time="2024-10-10T15:13:01..." level=error msg="Handler for POST /v1.41/containers/create returned error: error creating overlay mount to /var/lib/docker/overlay2/6ef9f24.../merged: invalid argument"

Have you tried performing find /var/lib/docker | grep $img_hash?
(With hash being c1e2c31eb5944 and d211f4)

/var/lib/docker > docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
c1ec31eb5944: Pull complete 
Digest: sha256:d211f485f2dd1dee407a80973c8f129f00d54604d2c90732e8e320e5038a0348
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest

/var/lib/docker > find . | grep c1ec31eb5944
./image/overlay2/distribution/diffid-by-digest/sha256/c1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e

Now I did :slight_smile:

root@server:/var/lib/docker# find . | grep d211
root@server:/var/lib/docker#

root@server:/var/lib/docker# find . | grep c1e2
./image/overlay2/distribution/diffid-by-digest/sha256/c1e2...

Okay, figured out it is SELinux related


This morning, I did

  1. Complete re-install of the bare metal (Debian 12 as before), set up SELinux and the logical volume, ran the install script and the command docker run hello-world.
  2. Set SELinux to permissive mode, ran docker system prune -a' and docker run hello-world`.
  3. Unmounted the mount point /var/lib/docker (so it is on the same LV as the docker binaries), ran docker system prune -a' and docker run hello-world`.
  4. (Have SELinux on permissive mode still,) removed selinux-enabled: true from /etc/docker/daemon.json, ran docker system prune -a' and docker run hello-world`.

In the first three cases I got the .../merged: invalid argument. error. In the 4th case, it worked as expected.

So, I have troubleshot it. Now I need to figure out how to fix it. I shall investigate it myself but has anybody any ideas?

It seems we haven’t asked you yet, so how did you install Docker? Recently I saw someone using docker.io on Debian instead of Docker CE. What does the following command return?

dpkg -l | grep docker

Install is via ansible (see the script in the post above). It installs via apt

        - docker-ce
        - docker
        - docker-compose

Output of dpkg -l | grep docker

root@server: dpkg -l | grep docker
ii docker                    1.5-2                         ...
ii docker-buildx-plugin      0.17.1-1~debian.12~bookworm   ...
ii docker-ce                 5:27.3.1-1~debian.12~bookworm ...
ii docker-ce-cli             5:27.3.1-1~debian.12~bookworm ...
ii docker-ce-rootless-extras 5:27.3.1-1~debian.12~bookworm ...
ii docker-compose            1.29.2-3                      ...
ii docker-compose-plugin     2.29.7-1~debian.12~bookworm   ...
rc docker.io                 20.10.24+dfs1-1+b3            ...
ii python3-docker            5.0.3-1                       ...
ii python3-dockerpty         0.4.1-4                       ...
ii wmdocker                  1.5-2                         ...

So, yes. I see docker.io. However, my knowledge of Docker is not deep enough to understand the implications you are making. Should I install Docker another way?

Let me help you with that then :slight_smile: https://www.youtube.com/watch?v=ItSuWaxdHhA

So docker.io is not the official docker, but “rc” in the first column means you already uninstalled it and only some metadata was left behind. I see docker and wmdocker packages. I guess it was an attempt to install Docker before you found the official Docker. These packages has nothing to do with Docker as container engine. On the other hand, it should not matter that the packages are installed.

I don’t use SELinux, so I’m not sure what the problem is here. I mostly heard about using selinux labels when mounting volumes like described here:

https://docs.docker.com/engine/storage/bind-mounts/#configure-the-selinux-label

I haven’t find a documentation about what --selinux-enabled does, but checking the source code it seems it just makes using SELinux required. If the option is enabled, then Docker checks if SELinux is enabled and if not, it could fail.

Just in case it helps anyone, this is the other source code which throws the error message

It happens when the “mount” function fails, which is defined right before calling the function and uses the mountFrom function which is here

It doesn’t help me much to tell you what the problem is. I see only lowlevel filesystem operations.

You could try enabling the debug log in Docker. Passing --debug to the docker daemon would do it or "debug": true in the daemon.json.

I also noticed this in your previous log:

Althogh your package list indicates you uninstalled docker.io this log indicates the opposite.

Make sure you have only docker-ce installed. snap is used on Ubuntu mainly, but you cans till check if it is somehow installed on your Debian:

snap list | grep docker

You can also share the output of the following commands:

docker info
docker version
1 Like

Thank you for the elaborate answer and YouTube explanation. (In the HomeLab world you need to learn & remember a lot of different applications :slight_smile:)

I added an extra step to the install script. This first step is to remove docker, wmdocker, docker.io, and python3-dockerpty. Ran the script and I verified Docker using the Docker-ce (v27.3.1) now using the journalctl -eu docker.service, docker info, and docker version commands.

Issue still remains unfortunately. I found this blog post which points me in a direction of installing/using a container-selinux package. Maybe it is my understanding, but struggling with the instructions (or lack of) how to install & use it. Help is welcome and otherwise, I shall let you know if/once I figure it out.


EDIT: I was hoping to update the Ansible install script in my original post but it looks like the ‘timer expired’ on the edit button. So, here is the total install script I used.

- name: Remove earlier/unwanted Docker installations.
  block:
    - name: Remove earlier/unwanted Docker installations.
      ansible.builtin.apt:
        name: '{{ task_item }}'
        state: absent
      with_items:
        - docker
        - docker.io
        - wmdocker
        - python3-dockerpty # maybe this package is needed, just trying if it can do without.
      loop_control:
        loop_var: task_item

- name: Set Docker's variable for system architecture
  changed_when: false
  no_log: true
  register: architecture_codename
  ansible.builtin.command: 'dpkg --print-architecture'

- name: Install Docker
  block:
    - name: Install Docker's dependencies
      ansible.builtin.apt:
        name: '{{ task_item }}'
        state: present
        update_cache: true
      with_items:
        - ca-certificates
        - curl
        - python3-pip
      loop_control:
        loop_var: task_item

    - name: Add Docker's GPG key
      ansible.builtin.apt_key:
        url: https://download.docker.com/linux/debian/gpg
        keyring: /etc/apt/keyrings/docker.gpg
        state: present

    - name: Add Docker's repository to apt
      no_log: true
      ansible.builtin.apt_repository:
        repo: >-
          deb [arch={{ architecture_codename.stdout }} signed-by=/etc/apt/keyrings/docker.gpg]
          https://download.docker.com/linux/debian {{ ansible_distribution_release | lower }} stable
        state: present

    - name: Install Docker
      ansible.builtin.apt:
        name: '{{ task_item }}'
        state: present
        update_cache: true
      with_items:
        - docker-ce
        - docker
        - docker-compose
      loop_control:
        loop_var: task_item

- name: Make sure the Ansible user can use Docker
  no_log: true
  block:
    - name: Ensure group 'docker' exists
      ansible.builtin.group:
        name: docker
        state: present

    - name: Adding the Ansible user to group 'docker'
      ansible.builtin.user:
        append: true
        groups: docker
        name: '{{ docker.user.ansible.name }}'

    - name: Change Docker's file ownership, group and permissions
      ansible.builtin.file:
        path: /usr/bin/docker
        group: '{{ docker.user.ansible.group }}'
        mode: '0755'
        owner: '{{ docker.user.ansible.name }}'

- name: Check if Docker is active
  ansible.builtin.service:
    name: docker
    state: started
    enabled: true

I don’t even understand why those packages (especially docker.io) are installed on the machine in the first place. How is that machine created?

New users can edit posts only for 24 hours, but unless you find something in an oldr post you shouldn’t have shared, it is better to share a new code so the flow of the discussion remains clear.

Why do you install the docker package? Make sure you follow the official guide: Debian | Docker Docs

And you can implement those steps in Ansible. When it works, you can remove packages or add packages based on what you need

I also had a blogpost and video about installing Docker using Ansible following the official docs, but on Ubuntu: Install Docker and Portainer in a VM using Ansible - DEV Community

and without selinux. The docker pckage is not needed and not related to Docker CE

This is a transitional package for system tray docking application. It can safely be removed.

I don1t think it matters, but this is the only think I noticed which is not needed.

Maybe you could search for issues on GitHub as well:

Or open a new issue if you don’t find a related one. I couldn’t.

SELinux is usualy not enabled on Debian as far as I know, so maybe it was just not tested on that distribution

So, getting to the on-topic & original question. I agree with you, @rimelek, that what I want is not possible/supported in Debian. (At least, I did not get it to work.)

Off-topic: you are right that my Ansible script to install Docker was not following the official guide. After some digging, I found a mistake from my side as well. Docker.io is installed as I had not explicitely said install_recommends: false in the task running apt.

With that, hereby my latest version of the script, although off-topic.

- name: 'Remove earlier/unwanted Docker installations.'
  block:
    - name: 'Remove earlier/unwanted Docker installations.'
      ansible.builtin.apt:
        name: '{{ task_item }}'
        purge: true
        state: 'absent'
      with_items:
        - docker.io
        - docker-doc
        - docker-compose
        - podman-docker
        - containerd
        - runc
      loop_control:
        loop_var: task_item

- name: "Set Docker's variable for system architecture."
  changed_when: false
  no_log: true
  register: architecture_codename
  ansible.builtin.command: 'dpkg --print-architecture'

- name: Install Docker
  block:
    - name: Install Docker's dependencies
      ansible.builtin.apt:
        name: '{{ task_item }}'
        state: present
        update_cache: true
      with_items:
        - ca-certificates
        - curl
      loop_control:
        loop_var: task_item

    - name: Add Docker's GPG key
      ansible.builtin.apt_key:
        url: https://download.docker.com/linux/debian/gpg
        keyring: /etc/apt/keyrings/docker.gpg
        state: present

    - name: Add Docker's repository to apt
      no_log: true
      ansible.builtin.apt_repository:
        repo: >-
          deb [arch={{ architecture_codename.stdout }} signed-by=/etc/apt/keyrings/docker.gpg]
          https://download.docker.com/linux/debian {{ ansible_distribution_release | lower }} stable
        state: present

    - name: Install Docker
      ansible.builtin.apt:
        install_recommends: false
        name: '{{ task_item }}'
        state: present
        update_cache: true
      with_items:
        - docker-buildx-plugin
        - docker-ce
        - docker-ce-cli
        - containerd.io
        - docker-compose-plugin
      loop_control:
        loop_var: task_item

- name: Make sure the Ansible user can use Docker
  no_log: true
  block:
    - name: Ensure group 'docker' exists
      ansible.builtin.group:
        name: docker
        state: present

    - name: Adding the Ansible user to group 'docker'
      ansible.builtin.user:
        append: true
        groups: docker
        name: '{{ docker.users.ansible.name }}'

    - name: Change Docker's file ownership, group and permissions
      ansible.builtin.file:
        path: /usr/bin/docker
        group: '{{ docker.users.ansible.group }}'
        mode: '0755'
        owner: '{{ docker.users.ansible.name }}'

- name: Check if Docker is active
  ansible.builtin.service:
    name: docker
    state: started
    enabled: true

Also, to me this topic is closed (but I cannot seem to mark it as closed). Thank you for your support.