Docker Community Forums

Share and learn in the Docker community.

Unable to find user root: no matching entries in passwd file

docker

(Vikram Khatri) #1

I have noticed that whenever I run docker cp command, the later invocations of docker exec do not work and I get the error “unable to find user root: no matching entries in passwd file

Here is an example where docker exec works fine but after I did docker cp, the docker exec refuse to work.

$ docker exec dashDB hostname -s
  node04

$ docker cp dashDB:/mnt/bludata0/options .

$ docker exec dashDB hostname
  unable to find user root: no matching entries in passwd file

$ docker version
Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        
 OS/Arch:      linux/amd64

(Bezzers) #2

Same issue here on 1.12.6. A docker cp followed by a docker exec fails to find the user. Running docker stop and then docker start fixes the problem but is hardly a long-term solution. Docker restart does not fix the problem. Is this related to docker cp not releasing a lock on the container?


(Vikram Khatri) #3

Ok - I am still seeing this error in Docker version 17.06.0-ce

The only solution is to restart docker service - which is an interruption.

Can this be fixed in docker and is becoming an issue for Docker being a SPOF?


(Leonhoffman) #4

Running 17.09.0-ce matches this exact pattern. This “unable to find user” message did not occur until after using docker cp. Is this being tracked still?

You can still issue “docker exec -ti -u 0 container command” but it would be nice if you didn’t have to use that.


(Slavik57) #5

After running it with -u 0 I got:

rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:75: starting setns process caused \"fork/exec /proc/self/exe: no such file or directory\""

This happened to me after using cp command as well.

Any news regarding this issue?


(Hudson) #6

After running the following command issue is fixed.

docker exec -it -u 0 /bin/bash


(Kingdon Barrett) #7

So I don’t know if this is the same issue, but this is the closest to a discussion that I’ve found explaining a problem I’m having on my managed Kubernetes cluster:

$ kubectl -n jekyll exec -it jekyll-web-75c8dbb69c-qr4zf echo
unable to find user slug: no matching entries in passwd file
command terminated with exit code 126

I found https://github.com/mikelorant/kubectl-exec-user which basically lets me get access to the docker sock even though I’m on a managed kubernetes cluster (that was surprising, but a welcome one…) and this happened:

$ kubectl -n jekyll plugin exec-user -u root jekyll-web-75c8dbb69c-qr4zf bash
unable to find user root: no matching entries in passwd file
pod "exec-user-jekyll-web-75c8dbb69c-qr4zf" deleted
pod jekyll/exec-user-jekyll-web-75c8dbb69c-qr4zf terminated (Error)
error: exit status 126

$ kubectl -n jekyll plugin exec-user -u 0 jekyll-web-75c8dbb69c-qr4zf bash
If you don't see a command prompt, try pressing enter.
root@jekyll-web-75c8dbb69c-qr4zf:~#
root@jekyll-web-75c8dbb69c-qr4zf:~#

So, does anyone know what causes unable to find user root: no matching entries in passwd file?

Now that I’m able to peek inside of the container, I can see that nothing looks especially unusual about /etc/passwd or /etc/shadow

/etc/passwd ...
root:x:0:0:root:/root:/bin/bash
...
slug:x:2000:2000::/app:/bin/bash
/etc/shadow ...
root:*:16667:0:99999:7:::
...
slug:!:17623:0:99999:7:::

Those both look like normal entries for root and slug, but neither username is recognized by docker exec -u or other more basic primitive ways to get processes exec’ed into the container by Kubernetes (health checking, or kubectl exec which cannot accept a user param)

My container has this for a readiness check,

readinessProbe:
        exec:
          command:
          - bash
          - -c
          - '[[ ''$(ps -p 1 -o args)'' != *''bash /runner/init''* ]]'
        failureThreshold: 1
        initialDelaySeconds: 30
        periodSeconds: 5
        successThreshold: 1
        timeoutSeconds: 5

and it is never marked ready because the container definition specified USER slug which as we’ve already seen is unknown, like in the other reports dating back to 2016. How I arrived at this is still a mystery (I think I can delete the pod and it will solve it, I had three of these and solved two that way, but as other reporters have said, this is not sustainable solution… “kubectl delete pod” is a manual step and if this issue can’t be resolved by health checks, it means we’re back to the stone ages of pets not cattle, and being woken in the middle of the night by a pager when the website goes down.)

I don’t have steps to reproduce, as I did not docker cp to create this failure. It sounds like vanilla docker users don’t have any trouble reproducing the issue though. So do we have any idea what causes it (and can it be fixed?)


(Kingdon Barrett) #8

I guess I could just not do health checking that way… the only pods that were affected had this readinessProbe (and it should be noted that, I’m pretty sure at least, the probe doesn’t actually guarantee that anything is ready to serve traffic…


(Kingdon Barrett) #9

https://success.docker.com/article/ucp-health-checks-fail-unable-to-find-user-nobody-no-matching-entries-in-passwd-file-observed

“Currently there is no resolution to this issue. It is being tracked via an internal bug.”

That could be a little more helpful, but at least it is acknowledged as a bug. The solution for me is probably to skip using container exec health checks (or make sure that it’s UID and not USER that gets set whenever I do use them) and wait for a fix to arrive upstream in docker, but meanwhile, when I find a node that encounters this problem, to resolve I can drain it, cordon and restart it. That’s really quite a bit less than optimal but that readinessProbe wasn’t really doing me any favors to be fair.