When I run docker run command I am getting
docker: Error response from daemon: failed to start shim: fork/exec /opt/docker/19.03.8/containerd-shim: too many open files: unknown.
Docker run command was working well until I compress a lot of files on the server (Ubuntu 18.04), then all containers are crashed and I cannot create new ones.
Right now, no container is running yet when I run
lsof | awk ‘{print $2, $1}’ | sort | uniq -c | sort -r -n | head -n 5
77786 4085 container
2456 15309 memsqld
1072 15310 memsqld
714 4005 dockerd
200 12217 docker
I see a huge amount of descriptors on container.
The problem is that I have the same 4 serves to have the same docker configuration and I was faced with the same error on one of them and solved it by rebooting the machine. However, I have not sudo access to these machines, and rebooting them is a big mess. And interesting thing is that I run tar command on 3 of these servers and after immediately it’s finished I see this error on both of them.
I am looking for an answer without rebooting, if possible. I didn’t check file limits but my question is how is it possible for the container process is using a lot of files on lsof even when no container is working.
Any help or comments are appreciated.