I’m in the process of migrating to a new TrueNAS SCALE server and have run into an issue that appears to be either Docker or pFSense related. Since this only effects Docker I figured this would be the best place to start.
I have a newly installed TrueNAS SCALE 25.10.4 server and that is totally stock with only a few TrueNAS apps installed (using Docker under the Truenas wrapper), totally basic network configuration using pFsense as a firewall/router. Everything worked fine and i was able to install/update apps without issue till I setup a firewall rule in pFsense to route all outbound traffic for the server over my Private Internet Access (PIA) VPN connection.
As soon as I activated the PIA connection I’m unable to install or update TrueNAS apps, but I’m also unable to create custom containers using Docker images. Going further I’m unable to install docker images via portainer/Dockage, and even pull images via docker on the command line.
Some of the errors I’ve received are: Error response from daemon: Head "htts://registry-1.docker.io/v2/nardo86/nut-server/manifests/amd64-latest": Get "htts://auth.docker.io/token?scope=repository%3Anardo86%2Fnut-server%3Apull&service=registry.docker.io": context deadline exceeded
Get "htts://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Error response from daemon: Head "htts://registry-1.docker.io/v2/library/hello-world/manifests/latest": Get "htts://auth.docker.io/token?scope=repository%3Alibrary%2Fhello-world%3Apull&service=registry.docker.io": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
When I disable the firewall rule routing the server over the PIA VPN connection docker pulls immediately start working again.
Troubleshooting the issue some of the things i’ve done are:
Verified networking is was working correctly over the VPN for other apps including using curl to access sites and wget to download files.
Verified the DNS was working correctly getting the same results as other machines without the issue.
Switched the system and Docker DNS settings to 8.8.8.8 etc.
Adjusted the MTU in the daemon.json to 1400.
Restarted server.
Reset PIA VPN connection to get different IPs
Configured TrueNAS with optional PIA proxy server.
Configured Docker with optional PIA proxy server.
Are VPN IPs blocked from pulling Docker images? Or is there something else I’m missing? (note https in errors changed to htts cause of link limit in post)
Don’t forget the apostrophes aroud the HRL in the second curl command.
I also noticed that your original error messages contained htts not https in the URLs. Did you change it to avoid the forum converting them to clickable links? You can use code blocks to avoid that. I changed your post to fix it. You can see what I changed by clicking on the pencil icon in the top right corner of your post.
Returns {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"nardo86/nut-server","Action":"pull"}]}]} which is the same message as on a machine with working docker.
Returns curl: (28) Failed to connect to auth.docker.io port 443 after 270369 ms: Couldn't connect to server
As soon as i disable PIA it gives me a long what appears to be token key prefaced with {"token":
I compared the DNS returns on both the broken and working servers and they give the same results for auth.docker.io and both are able to ping auth.docker.io.
Also of note, I’ve tried multiple VPN locations and reloaded the connection numerous time to get different IPs including the “Streaming” endpoint which usually have IPs that aren’t yet on most VPN ban lists.
Yes i changed them to htts to avoid clikcable links, thanks for the fix.
Just an update for everyone that finds this thread in the future, I was able to resolve this issue. Turns out i had two OpenVPN clients for PIA (to different zones) configured in pfSense, and while only one was assigned as a gateway to an interface having both running and connected created the issue. I wasn’t able to determine exactly the issue it caused or why it only effected my docker image pulls, but I can recreate the issue on demand by connecting the second PIA OpenVPN client.