Some folders not showing on merged overlay directory

Docker Engine General
1

Hi,
Does anyone know why some of the folders are not showing on merged directory? Its present in diff folder but not on merged. Because of that, those files are not listed on container when we execute the ls -al command but cd to that folder works. This happens only randomly not always.
Any help is appreciated, thanks!

2

Please share more context about your environment. The output of the following commands first:

docker info
docker version

ā€œmergedā€ contains only files and folders that were not deleted. If something appears there randomly, that is a really strange behavior and it could be caused by an antivirus or any security software that deletes files, although I have no idea why it would not disappear permanently.

3

Hi Ɓkos,

Thanks for replying back. Here is the output of the commands:

docker info

Client:
 Version:    25.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.0.0+unknown
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 6
  Running: 5
  Paused: 0
  Stopped: 1
 Images: 7
 Server Version: 25.0.6
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: amazon-ecs-volume-plugin local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: 2c9f5602f0ba3d9da1c2596322dfc4e156844890
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 4.14.353-270.569.amzn2.x86_64
 Operating System: Amazon Linux 2
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.28GiB
 Name: ip-10-0-27-205.eu-central-1.compute.internal
 ID: 1228c8f4-c286-4e6c-b195-59c350fbe857
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  registry.ecs-cluster.internal
  127.0.0.0/8
 Registry Mirrors:
  https://registry.ecs-cluster.internal/
 Live Restore Enabled: false

WARNING: IPv4 forwarding is disabled

docker version

Client:
 Version:           25.0.5
 API version:       1.44
 Go version:        go1.22.5
 Git commit:        5dc9bcc
 Built:             Thu Aug 22 17:25:26 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          25.0.6
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.22.5
  Git commit:       b08a51f
  Built:            Thu Aug 22 17:26:01 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.20
  GitCommit:        8fc6bcff51318944179630522a095cc9dbf9f353
 runc:
  Version:          1.1.14
  GitCommit:        2c9f5602f0ba3d9da1c2596322dfc4e156844890
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Within the merged folder there is a folder called logs. And within the logs, different folders are there. Those folders within the logs are not visible using ls commands, but when we do cd foldername, it works.

When we start a new server all seems good, but after some hours this behaviour starts and also not always.
We have a folder within logs which is NFS mounted. Is it because of that some NFS and OverlayFS issues? Iā€™m sorry I not quite sure about it, so Iā€™m asking. And also this happens randomly cannot track it down.

4

Iā€™m not sure what causes this, so just some notes:

Iā€™m not familiar with Amazon Linux. I only used some amazonlinux images from Docker Hub, but never used it as a host. It is also not among the officially supported operating systems

remote filesystem mount would not be in the diff folder. At least I donā€™t see why it would be. But if files in diff not appearing in merged is a separate issue, then maybe.

You could try to upgrade Docker to a newer version. If there was a bug, it is possibly fixed by now. I guess it is also possible that the bug is in the Kernel.

I have no idea why some folders would disappear randomly from a fodler, but if you can cd into a disappeared fodler that is not really disappeared and maybe the issue is with the terminal or the operating system. And as you pointed out, the content of a mounted NFS folder could disappear too but that still doesnā€™t explain being able to cd into the folder thaty ou donā€™t see.

1 Like
5

I fully agree with @rimelek: remote share mounts wonā€™t be visible in the overlayfs diff folder. But it wonā€™t be visible in the merged folder either.

I just checked two containers that use remote shares, one that uses cifs, and one that uses nfsv4.1.
In both cases the volumes are mounted into the container and the expected data exists, while at the same time there is no trace in the overlay folder. The mounts are namespaced, so you will not see the actual container filesystem on the host that the container sees.

1 Like