Unable to pull docker images when routed over PIA

General
1

I’m in the process of migrating to a new TrueNAS SCALE server and have run into an issue that appears to be either Docker or pFSense related. Since this only effects Docker I figured this would be the best place to start.

I have a newly installed TrueNAS SCALE 25.10.4 server and that is totally stock with only a few TrueNAS apps installed (using Docker under the Truenas wrapper), totally basic network configuration using pFsense as a firewall/router. Everything worked fine and i was able to install/update apps without issue till I setup a firewall rule in pFsense to route all outbound traffic for the server over my Private Internet Access (PIA) VPN connection.

As soon as I activated the PIA connection I’m unable to install or update TrueNAS apps, but I’m also unable to create custom containers using Docker images. Going further I’m unable to install docker images via portainer/Dockage, and even pull images via docker on the command line.

Some of the errors I’ve received are:
Error response from daemon: Head "htts://registry-1.docker.io/v2/nardo86/nut-server/manifests/amd64-latest": Get "htts://auth.docker.io/token?scope=repository%3Anardo86%2Fnut-server%3Apull&service=registry.docker.io": context deadline exceeded

Get "htts://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Error response from daemon: Head "htts://registry-1.docker.io/v2/library/hello-world/manifests/latest": Get "htts://auth.docker.io/token?scope=repository%3Alibrary%2Fhello-world%3Apull&service=registry.docker.io": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

When I disable the firewall rule routing the server over the PIA VPN connection docker pulls immediately start working again.

Troubleshooting the issue some of the things i’ve done are:

  • Verified networking is was working correctly over the VPN for other apps including using curl to access sites and wget to download files.
  • Verified the DNS was working correctly getting the same results as other machines without the issue.
  • Switched the system and Docker DNS settings to 8.8.8.8 etc.
  • Adjusted the MTU in the daemon.json to 1400.
  • Restarted server.
  • Reset PIA VPN connection to get different IPs
  • Configured TrueNAS with optional PIA proxy server.
  • Configured Docker with optional PIA proxy server.

Are VPN IPs blocked from pulling Docker images? Or is there something else I’m missing? (note https in errors changed to htts cause of link limit in post)

2

Not in general, but it is not impossible that some IPs are blocke

Have you tried to use curl to access sites in the error messages?

curl https://registry-1.docker.io/v2/nardo86/nut-server/manifests/amd64-latest

and

curl 'https://auth.docker.io/token?scope=repository%3Anardo86%2Fnut-server%3Apull&service=registry.docker.io'

Don’t forget the apostrophes aroud the HRL in the second curl command.

I also noticed that your original error messages contained htts not https in the URLs. Did you change it to avoid the forum converting them to clickable links? You can use code blocks to avoid that. I changed your post to fix it. You can see what I changed by clicking on the pencil icon in the top right corner of your post.

3

curl https://registry-1.docker.io/v2/nardo86/nut-server/manifests/amd64-latest

Returns {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"nardo86/nut-server","Action":"pull"}]}]} which is the same message as on a machine with working docker.

and

curl 'https://auth.docker.io/token?scope=repository%3Anardo86%2Fnut-server%3Apull&serv

Returns curl: (28) Failed to connect to auth.docker.io port 443 after 270369 ms: Couldn't connect to server

As soon as i disable PIA it gives me a long what appears to be token key prefaced with {"token":

I compared the DNS returns on both the broken and working servers and they give the same results for auth.docker.io and both are able to ping auth.docker.io.

Also of note, I’ve tried multiple VPN locations and reloaded the connection numerous time to get different IPs including the “Streaming” endpoint which usually have IPs that aren’t yet on most VPN ban lists.

Yes i changed them to htts to avoid clikcable links, thanks for the fix.

4

I tried another VPN. Both URLs work. So it is either something blocked by PIA, or you can ask about it in the desktop feedback repo

I know that I can’t access some APT repositories thorugh the VPN I am using, but Docker URLs worked so far.