I’m new to this, setting up a private registry on premise, using htpasswd authentication for now and our digicert wildcard cert.
In testing I was able to get a self-signed cert working, but for real use I don’t want to hassle our devs with the need to add the cert to every workstation. But after a day or two of flailing, I’m stuck at a point where “docker login” attempts always return an error “Error response from daemon: Get https://d-------.org/v2/: x509: certificate signed by unknown authority”
In a browser I can enter the URL for the registry (https://d----.org) and while I get a blank page, the browser shows the site as secure and I can see our cert information.
However, from the workstation (where docker login fails), curl https://d----.org returns:
“curl: (60) SSL certificate problem: unable to get local issuer certificate”
and wget https://d----.org returns:
ERROR: cannot verify d—.org’s certificate, issued by ‘/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA’:
Unable to locally verify the issuer’s authority.
The certs are on the vm and mounted to the container at /certs, with
REGISTRY_HTTP_TLS_CERTIFICATE=/certs/—.org.crt’
and
REGISTRY_HTTP_TLS_KEY=/certs/—.key
passed to the container
Check you local ca-store on the failing client. Do you have ca-certificates installed?
As soon as curl-ing the repository works from that client also docker works.
Your client just cannot validate the certificate.
Thank you for a good suggestion. But ca-certificates is installed on all of the clients I’m trying from, including different linux distros and versions. Also, I can “curl” to other https URLS successfully, including real-world URLS like https://google.com and also our own domain and our registered cert.
The problem seems to be that docker is failing to find the needed CA information that curl and wget succeed for.
I’ve also tried copying our .pem file into /var/lib/ca-certificates/pem/ (openSUSE_leap) and running update-ca-certificates, then restarting docker, with no change. Also copied that pem file (and a copy renamed to .crt) into /etc/docker/certs.d and restarted docker. These changes were meant more to identify the problem, not as a real fix: if I have to touch every client to recognize a real cert I might as well use a self-signed one.
All of these changes still get “Error response from daemon: Get https://d-----.org/v1/users/: x509: certificate signed by unknown authority” when I attempt a docker login.
I’ve found some similar threads where folks could push and pull just fine but not login. No luck for me, however: attempting to push to my registry returns the “unknown authority” error.
I can pull the example you give. I can curl from other sites/services using our digicert-issued certificate. I can point chrome at my on-premise docker URL and while I get a blank page, the browser shows the site as secure and with a legit certificate. But curl, wget and docker all balk when I point them at my on-premise docker registry using the cert.
I’m leaning toward the theory that the docker registry container isn’t configured correctly for my cert pair, but don’t see what I could be missing if the only two settings involved are the two environment variables mentioned at https://docs.docker.com/registry/deploying/#run-an-externally-accessible-registry, REGISTRY_HTTP_TLS_CERTIFICATE and REGISTRY_HTTP_TLS_KEY.
Are there other settings for the docker registry container?
OK, if nothing else I’m making some newbie progress on learning how to troubleshoot a docker instance.
The container is running busybox/alpine 3.4 and I can get a shell with “docker exec -it registry /bin/ash”
From inside the container I can see that I am running registry with the correct config.yml file specified, that my docker VM’s version of the config.yml file is mounted inside the container. Also that my outside .htpasswd file, cert files and NFS mount to the registry data volume are present inside the container.
“docker inspect registry” also confirms this.
“docker logs registry” shows the process starting, complaining about needing to create a random secret (not a problem in this situation), and that it’s listening on port 443 as it should be. From within the container, netstat -lntp shows port 443 open and owned by the registry process.
“docker logs -f registry” shows two lines when I try to login to the registry:
2018/07/05 19:59:35 http: TLS handshake error from 10.6.107.217:42526: EOF
2018/07/05 19:59:35 http: TLS handshake error from 10.6.107.217:42528: EOF
A thought: should those lines say “https?” Am I not really listening via SSL?
Maybe you’re missing the intermediate certificate. Have you tried ssllabs?
On a related note, is there a way to get a shell session inside the container so I can poke around in there and see what it’s doing? I’ve tried docker exec with no luck…
By using docker exec. What do you mean with “no luck”.
2018/07/05 19:59:35 http: TLS handshake error from 10.6.107.217:42526: EOF
2018/07/05 19:59:35 http: TLS handshake error from 10.6.107.217:42528: EOF
A thought: should those lines say “https?” Am I not really listening via SSL?
Google would have solved that problem. SSL is incorrectly used for tls connections.
The timeline is linke so: SSL 1.0 (in 1994) through SSL 3.1 than switch to TLS 1.0 ( in the year 1999) than tls 1.1, tls 1.2 and released this year tls 1.3. If you’re using anything else than tls 1.2 and tls 1.3, you’re not up to date with security.
Bingo. Testing our SSL cert at SSLABS showed an intermediate cert from digikey in the chain. I found that crt file and used “cat” to join it to our own cert as documented at https://docs.docker.com/registry/deploying/, and loaded that.
I can now connect to our on-premise registry without errors using curl, wget and docker login. I’m guessing chrome must have been finding and adding the intermediate itself.