Merged directory not created under overlay2 for USB pass through

I have the following nested setup:

  1. Proxmox PVE on bare metal (Intel Nuc / 5.15.158-2-pve)
  2. Debian Stable as LXC container on 5.15.158-2-pve
  3. Home Assistant as unprivileged Docker container (Docker version 27.1.1, build 6312585)

This works well, except for USB pass-through, since I updated Docker this morning I get the error:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error creating device nodes: mount /dev/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2149131-if00:/var/lib/docker/overlay2/963a244fa0d220f872cc0e02714e6045b112c5db6404ce5a47903ec936b2e51e/merged/dev/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2149131-if00 (via /proc/self/fd/6), flags: 0x1000: no such file or directory: unknown

Does anyone know why this is happening and how I can solve this? I think it’s due to a Docker issue, but I’ve also asked the same question on the proxmox forum.

What doesn’t work:

  1. Restarting either the Docker container, the Debian LXC, or Proxmox PVE
  2. Running docker system prune -a
  3. Deleting /var/lib/docker/overlay2/ and re-pulling all images/containers
  4. Running docker in privileged mode

What does work:

  1. Running the container without the USB device

I’m using the following compose file:

tim@debian:~$ cat docker/home-assistant-compose.yml
# version: '3'
# https://www.home-assistant.io/installation/linux#docker-compose
# docker compose -f home-assistant-compose.yml up -d
services:
  homeassistant:
    container_name: homeassistant
    image: "ghcr.io/home-assistant/home-assistant:stable"
    volumes:
      - /var/lib/homeassistant:/config
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped
    network_mode: host
    devices:
      - /dev/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2149131-if00

When running this config, I get the following error:

tim@debian:~/docker$ sudo docker compose -f home-assistant-compose.yml up -d
WARN[0000] Found orphan containers ([docker-app-1 docker-db-1 pigallery2]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
[+] Running 0/1
 ⠼ Container homeassistant  Starting                                                                                                                                                         0.4s
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error creating device nodes: mount /dev/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2149131-if00:/var/lib/docker/overlay2/06f5efd809d035a93128edc19bb936c62b21b18f6e3e6a402a881c3852aa81c3/merged/dev/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2149131-if00 (via /proc/self/fd/6), flags: 0x1000: no such file or directory: unknown

Looking at the specifics, the source file exists:

tim@debian:~$ ls -l /dev/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2149131-if00
crw-rw---- 0 root dialout 166, 0 Aug  8 08:12 /dev/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2149131-if00

And the overlay source also exists:

tim@debian:~$ sudo ls -l /var/lib/docker/overlay2/06f5efd809d035a93128edc19bb936c62b21b18f6e3e6a402a881c3852aa81c3/
total 16
drwxr-xr-x 3 root root 4096 Aug  8 08:53 diff
-rw-r--r-- 1 root root   26 Aug  8 08:53 link
-rw-r--r-- 1 root root  927 Aug  8 08:53 lower
drwx------ 3 root root 4096 Aug  8 08:53 work

however there is no ‘merged’ folder under the overlay directory structure. It seems Docker doesn’t (re)create this?

Does anyone know why this could be the case? I’m at a loss :frowning: . It seems something changed on Docker as the problem started when I updated that. Thanks in advance! :slight_smile:

Did you update and reboot? Did the device path change on host?

Yes I updated and rebooted system. The device path didn’t change on either the proxmox or LXC host. Did something change in creation of the merged overlay2-subdirectory in Docker last year?

This is my scenario and problem as well. Any creative solution?

I didn’t solve it, instead I’m now running USB-related stuff outside docker unfortunately.

However, today I wanted to run another Docker image unrelated to all this (building VyOS from source to be precise), and I got the same error:

docker pull vyos/vyos-build:current
docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error creating device nodes: mount /dev/usb-FTDI_FT232R_USB_UART_AC2F17KR-if00-port0:/var/lib/docker/overlay2/58b81b89f41a9dbcdec50ab74ea57a5c0e7549b43b0c518e783441a0e81bac7d/merged/dev/usb-FTDI_FT232R_USB_UART_AC2F17KR-if00-port0 (via /proc/self/fd/7), flags: 0x1000: no such file or directory: unknown.

The strange this is, there is no reference to /dev/usb-FTDI_FT232R_USB_UART_AC2F17KR-if00-port0 in the directory I’m running this docker command, so not sure where it’s getting that from. Any ideas?

docker events shows:

2024-08-24T20:59:49.078413162+02:00 container create b8c7e4402b41ad8e85924fbebe56740fb5b31b9d9e70fab76fa86bfe26bd3482 (authors=VyOS Maintainers <maintainers@vyos.io>, image=vyos/vyos-build:current, name=reverent_wescoff, org.opencontainers.image.authors=VyOS Maintainers <maintainers@vyos.io>, org.opencontainers.image.base.name=docker.io/debian/debian:bookworm, org.opencontainers.image.description=Container to build VyOS ISO, org.opencontainers.image.documentation=https://docs.vyos.io/en/latest/contributing/build-vyos.html, org.opencontainers.image.licenses=GNU, org.opencontainers.image.source=https://github.com/vyos/vyos-build, org.opencontainers.image.title=vyos-build, org.opencontainers.image.url=https://github.com/vyos/vyos-build, org.opencontainers.image.vendor=Sentrium S.L.)
2024-08-24T20:59:49.083265286+02:00 container attach b8c7e4402b41ad8e85924fbebe56740fb5b31b9d9e70fab76fa86bfe26bd3482 (authors=VyOS Maintainers <maintainers@vyos.io>, image=vyos/vyos-build:current, name=reverent_wescoff, org.opencontainers.image.authors=VyOS Maintainers <maintainers@vyos.io>, org.opencontainers.image.base.name=docker.io/debian/debian:bookworm, org.opencontainers.image.description=Container to build VyOS ISO, org.opencontainers.image.documentation=https://docs.vyos.io/en/latest/contributing/build-vyos.html, org.opencontainers.image.licenses=GNU, org.opencontainers.image.source=https://github.com/vyos/vyos-build, org.opencontainers.image.title=vyos-build, org.opencontainers.image.url=https://github.com/vyos/vyos-build, org.opencontainers.image.vendor=Sentrium S.L.)
2024-08-24T20:59:49.129877373+02:00 network connect df4d28d31001b19e0b4feb83cc78c38c67604661bd3676f7b8eba0dec4f0983a (container=b8c7e4402b41ad8e85924fbebe56740fb5b31b9d9e70fab76fa86bfe26bd3482, name=bridge, type=bridge)
2024-08-24T20:59:49.393067009+02:00 network disconnect df4d28d31001b19e0b4feb83cc78c38c67604661bd3676f7b8eba0dec4f0983a (container=b8c7e4402b41ad8e85924fbebe56740fb5b31b9d9e70fab76fa86bfe26bd3482, name=bridge, type=bridge)
2024-08-24T20:59:49.412552072+02:00 container destroy b8c7e4402b41ad8e85924fbebe56740fb5b31b9d9e70fab76fa86bfe26bd3482 (authors=VyOS Maintainers <maintainers@vyos.io>, image=vyos/vyos-build:current, name=reverent_wescoff, org.opencontainers.image.authors=VyOS Maintainers <maintainers@vyos.io>, org.opencontainers.image.base.name=docker.io/debian/debian:bookworm, org.opencontainers.image.description=Container to build VyOS ISO, org.opencontainers.image.documentation=https://docs.vyos.io/en/latest/contributing/build-vyos.html, org.opencontainers.image.licenses=GNU, org.opencontainers.image.source=https://github.com/vyos/vyos-build, org.opencontainers.image.title=vyos-build, org.opencontainers.image.url=https://github.com/vyos/vyos-build, org.opencontainers.image.vendor=Sentrium S.L.)