“Unable to load the service index for source https://api.nuget.org/v3/index.json.The SSL connection could not be established, see inner exception. The remote certificate is invalid because of errors in the certificate chain: UntrustedRoot”
Docker cannot do a dotnet restore, because the certificate chain are broken
This was due to installing CISCO VPN Anyconnect-win Umbrella
Umbrella stops DNS hijacking something that is common in docker.
So make sure your IT department does not roll out this new safety feature without testing it with the devs first.
I had to roll back to anyconnect-win-4.9.00086-core-vpn and not connect (otherwise it would upgrade) to get Docker back to normal.
Follow up -
It looks like the issue here is TLS interception: something (Umbrella) is intercepting HTTPS requests allowing it to read (and modify) any web traffic. Applications will see this as a “man-in-the-middle” attack and close the connection. In order to work around this Umbrella needs to be added as a trusted root certification authority in all Docker containers that make Internet bound HTTPS requests.
To do this, first export the Umbrella CA by loading certlm.msc, expanding “Trusted Root Certification Authority”, identifying the CA then right-clicking, “All tasks”, “Export”, “Next”, “Base-64 encoded X.509 (.CER)”, “Next”, then save it somewhere in the Docker context so likely the root of your git repo. In the examples below I’ve assumed it’s named “umbrella.crt” but update as necessary.
For Linux containers add this to each Dockerfile:
ADD umbrella.crt /usr/local/share/ca-certificates/umbrella.crt
RUN chmod 644 /usr/local/share/ca-certificates/umbrella.crt && update-ca-certificates
For Windows containers:
ADD umbrella.crt C:\umbrella.crt
RUN Import-Certificate -FilePath C:\umbrella.cert -CertStoreLocation Cert:\LocalMachine\Root\
It will mean having to create custom images for any standard images, e.g. ones that are referenced directly from Docker Hub. For example command like ( docker run alpine/git ) still won’t work as you’ll have to create a Dockerfile for it to add the certificate, build it then run it. This is obviously not ideal.
TLS interception will break developer workflows in unexpected ways. Allow lists might work but there could be a lot of hosts to add, including wildcards, and will likely change over time. Is a transparent proxy possible instead? That would still be able to sniff certificate data so can pick out the host names without breaking TLS connections (although I think this was purposefully designed out in TLS 1.3 for privacy reasons).
If an allow list is the route to go then the obvious ones are possibly:
- Docker Hub
- Microsoft Container Registry
- Google Container Registry
- Nuget repositories
- NPM
1 Like
Follow up:
Having implemented egress filtering for environments before I’ve seen the unhelpful “CANCELLED” message in the first screenshot, and gone down a rabbit hole of packet tracing to work out exactly what the issue was. The second screenshot has a slightly better message saying it was unable to verify the issuer certificate, which means the process was able to establish a network connection and send/receive data however failed on the certificate check. Usually when that happens it’s because the CA isn’t trusted, could be because it’s an internal service or MITM is going on.
Conceptually Docker containers can be thought of as completely isolated processes running on the host – it has no access to the file system, and therefore the host’s list of CA s. Similar to how a VM won’t inherit the CAs from the hypervisor.
I should point out that the Linux Dockerfile commands will work for Debian based systems. If you’re using an Alpine base image (as we do) or Red Hat base image then the RUN command will be different but the premise is the same and hopefully should be easy to find. And that’s just a workaround until your IT/NOC add the exceptions to the allow list.
Hey @dougthompson ,
I am getting the same issue in dotnet restore
Would appreciate if you can list the exact steps to be followed in order to fix this issue.
I don’t have cisco installed, may be IT is blocking something.
Thanks
Hello
For anyone who gets something similar in the future I had a problem with a different certificate, in my case it was due to an app installed by IT called zscaler. I basically exported all my certs, and tried them in groups of a time, to see which one I needed. Note: Be sure you check your image to make sure it’s actually importing the certs. Mine wasn’t at first (and no error appeared), so I assumed my certs were getting copied. I made a Dockerfile that went up to the steps of copying the cert files, and manually verified update-ca-certificates was installing them.
Also note: The normal .net Dockerfile uses a temporary image to build the project. It’s this image that needs the cert, so make sure the cert update is after the
FROM mcr.microsoft.com/dotnet/sdk:6.0-alpine AS build
command.
1 Like
Hi,
I’ve had the same problem with dotnet restore in a docker contaioner using my local nuget instance using a self signed certificate (created with OpenSSL using a self signedroot certificate). Having installed the root certificare on my Windows workstation I could build the app in Visual Studio, but the Docker container could not be built, getting me the aforementioned error message.
Having read this post reminded me that dotnet restore was running in the linux context of the container and this could not work since my root certificate was missing in that context.
Excerpt from my working docker file:
…
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
ADD path/to/myroot.crt /usr/local/share/ca-certificates/myroot.crt
RUN chmod 644 /usr/local/share/ca-certificates/myroot.crt && update-ca-certificates
COPY [“this”, “that”]
RUN dotnet restore …
Thank you @dougthompson for your post and your explanations.
Michael