Docker run failure - fuse: device not found

rozmanro · March 26, 2023, 9:41am

Hi all,

Using docker on Centos7, arm64.
Have an issue on one of our devices , cannot execute any “docker run” command getting a fuse related error:

[root@localhost docker]# docker run -it --device /dev/ttyMV1 modpoll
docker: Error response from daemon: using mount program fuse-overlayfs: **fuse: device not found,** try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such file or directory
: exit status 1.
See 'docker run --help'.
[root@localhost docker]#

modprobe fuse didn’t help (no errors on execution)

tried to run with all of the options, any combination:
–privileged --cap-add=SYS_ADMIN --device /dev/fuse

Do not help.same error.

Here is the docker info of 2 units, the working and the non-working one.

WORKING DEVICE:

[root@localhost internal]# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 5
 Server Version: 20.10.16
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
 runc version: v1.1.1-0-g52de29d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.128
 Operating System: CentOS Linux 7 (AltArch)
 OSType: linux
 Architecture: aarch64
 CPUs: 2
 Total Memory: 1.946GiB
 Name: localhost
 ID: XXXX
 Docker Root Dir: /USERFS/docker/dockerd
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.X.0.0/16, Size: 24

WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
[root@localhost internal]#

NON-WORKING DEVICE - FUSE ERROR:

[root@localhost /]# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 20.10.16
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
 runc version: v1.1.1-0-g52de29d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.128
 Operating System: CentOS Linux 7 (AltArch)
 OSType: linux
 Architecture: aarch64
 CPUs: 2
 Total Memory: 1.947GiB
 Name: localhost
 ID: XXXX
 Docker Root Dir: /USERFS/docker/dockerd
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.X.0.0/16, Size: 24

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
[root@localhost /]#

Same LINUX kernel on both of the devices.

nothing I have found so far helped out regarding the fuse.
Any suggestion ?
Thanks in advance!

rimelek · March 26, 2023, 10:31am

I don’t know how exactly Docker works on Centos 7, but I remember there was something about the storage driver. The current default storage driver is overlay2 every distribution I work with. Your Docker is using fuse-overlayfs and I can’t see the backing filesystem which should be under the storage driver information.

I know sometimes podman on Centos appears to be Docker, but I don’t think Docker info would how what you quoted in that case.

Another thing I noticed is warning at the bottom of the docker info output. This usually appears in case of rootless Docker and searching for the error message I found reports about rootless Docker.

The context usually shows if the Docker is rootless, but the report I mentioned shows the same “default” context.

github.com/moby/moby

CentOS/RHEL Kernel can use fuse-overlayfs but a kernel version check prohibits it

opened 02:09AM - 27 Oct 21 UTC

oe-hbk

area/storage area/storage/overlay kind/enhancement area/rootless

**Description** Running latest docker-ce-2020.10.9-3.el7 on fully patched Cen…tOS7/RHEL7 systems does not use the available fuse-overlayfs graph driver that can indeed work on this system. **Steps to reproduce the issue:** This can probably be done in root mode, but I have been testing rootless mode, where the performance difference between overlayfs2 in root mode, and vfs in rootless mode has been problematic for us. Following https://docs.docker.com/engine/security/rootless/ to setup docker in rootless mode. CentOS/RHEL7 does not have SystemD User services, so have to just run the dockerd-rootless.sh to start things up after all pre-reqs have been met. Once started, note that in the absence of a storage-driver config option, stderr shows: ``` ime="2021-10-26T15:35:59.012559946-05:00" level=error msg="failed to mount overlay: operation not permitted" storage-driver=overlay2 time="2021-10-26T15:35:59.017385380-05:00" level=error msg="AUFS cannot be used in non-init user namespace" storage-driver=aufs time="2021-10-26T15:35:59.017985810-05:00" level=error msg="failed to mount overlay: operation not permitted" storage-driver=overlay time="2021-10-26T15:35:59.044026058-05:00" level=info msg="Loading containers: start." time="2021-10-26T15:35:59.185399870-05:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" time="2021-10-26T15:35:59.279255747-05:00" level=info msg="Loading containers: done." time="2021-10-26T15:35:59.417946739-05:00" level=info msg="Docker daemon" commit=75249d8 graphdriver(s)=vfs version=20.10.9 ``` It did not try fuse-overlayfs driver, and went with vfs. The following fuse packages are installed: ``` $ rpm -q -a | grep fuse glusterfs-fuse-6.0-49.1.el7.x86_64 fuse-2.9.2-11.el7.x86_64 fuse3-libs-3.6.1-4.el7.x86_64 fuse-libs-2.9.2-11.el7.x86_64 gvfs-fuse-1.36.2-5.el7_9.x86_64 fuse-overlayfs-0.7.2-6.el7_8.x86_64 ``` The system is: ``` $ cat /etc/centos-release CentOS Linux release 7.9.2009 (Core) $ uname -r 3.10.0-1160.45.1.el7.x86_64 ``` **Describe the results you received:** In rootless docker on CentOS 7, with proper fuse-overlayfs installed, dockerd does not choose it. **Describe the results you expected:** The fuse-overlayfs graph driver be used n this system. **Additional information you deem important (e.g. issue happens only occasionally):** **Output of `docker version`:** ``` $ docker version Client: Docker Engine - Community Version: 20.10.10 API version: 1.41 Go version: go1.16.9 Git commit: b485636 Built: Mon Oct 25 07:44:50 2021 OS/Arch: linux/amd64 Context: default Experimental: true Server: Docker Engine - Community Engine: Version: 20.10.10 API version: 1.41 (minimum version 1.12) Go version: go1.16.9 Git commit: e2f740d Built: Mon Oct 25 07:43:13 2021 OS/Arch: linux/amd64 Experimental: true containerd: Version: 1.4.11 GitCommit: 5b46e404f6b9f661a205e28d59c982d3634148f8 runc: Version: 1.0.2 GitCommit: v1.0.2-0-g52b36a2 docker-init: Version: 0.19.0 GitCommit: de40ad0 ``` **Output of `docker info`:** ``` $ docker info Client: Context: default Debug Mode: false Plugins: app: Docker App (Docker Inc., v0.9.1-beta3) buildx: Build with BuildKit (Docker Inc., v0.6.3-docker) scan: Docker Scan (Docker Inc., v0.9.0) Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 20.10.10 Storage Driver: vfs Logging Driver: json-file Cgroup Driver: none Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8 runc version: v1.0.2-0-g52b36a2 init version: de40ad0 Security Options: seccomp Profile: default rootless Kernel Version: 3.10.0-1160.45.1.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 3.682GiB Name: vml-jharper ID: 6I2C:TIQQ:CVOE:SM54:IMT4:XC2Q:SX3U:XG4X:CC2J:7FDA:T6C5:SKNK Docker Root Dir: /data/docker/oeris Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: true Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false ``` **Additional environment details (AWS, VirtualBox, physical, etc.):** The Red Hat kernel notoriously "looks" old even though it has many features from upstream kernels merged into it. Running the latest kernel available for CentOS/RHEL7 ``` $ uname -r 3.10.0-1160.45.1.el7.x86_64 ``` But the check done at https://github.com/moby/moby/blob/9aa0b24dcfa8956e03644ec43cf09dadcd5b6223/daemon/graphdriver/fuse-overlayfs/fuseoverlayfs.go#L87 does not sufficiently capture the capabilities of this kernel. Fuse-overlayfs became part of RHEL7 in 2020-03 https://access.redhat.com/errata/RHEA-2020:1222, and presumably any kernel changes necessary were done around that time. I have tested successfully by removing that conditional in fuseoverlayfs.go#L87 and rebuilding. Now running dockerd in rootless mode. docker info now shows ``` $ docker info Client: Context: default Debug Mode: false Plugins: app: Docker App (Docker Inc., v0.9.1-beta3) buildx: Build with BuildKit (Docker Inc., v0.6.3-docker) scan: Docker Scan (Docker Inc., v0.9.0) Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: %{_origversion} Storage Driver: fuse-overlayfs Logging Driver: json-file Cgroup Driver: none Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8 runc version: v1.0.2-0-g52b36a2 init version: de40ad0 Security Options: rootless Kernel Version: 3.10.0-1160.45.1.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 3.682GiB Name: vml-jharper ID: 6I2C:TIQQ:CVOE:SM54:IMT4:XC2Q:SX3U:XG4X:CC2J:7FDA:T6C5:SKNK Docker Root Dir: /data/docker/oeris Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: true Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false ``` I have tested building and running my containers and builds that used to take 28 minutes in rootless mode, compared to about 8 minutes in root mode, now take about 13 minutes. More importantly, in rootless mode when using the vfs driver, one of my containers that had to do a lot of local IO when starting up, used to take about 45 seconds when run with a shell entrypoint, now takes about 5 seconds. In root mode, this startup was ~ 1 second. I don't know how to best check for whether fuse-overlayfs is really compatible, but if someone can give me any pointers, I'd be happy to submit a PR.

PS.: Since the question has nothing to do with Docker Hub, I am going to move the topic to another category (I know, many topics not related to Docker Hub in this category)