Rootless docker cannot pull from registry-1.docker.io on an ipv6 only machine

I recently decided to try out an IPV6 only VPS running the latest Ubuntu. There are some quirks like not being able to ping ipv4 IPs, explanation by the company: " Outbound ICMP IPv4 IP pinging on VPS outside of the allocated port range, is not supported, due to a technical specific - the executed DNS64 implementation, at both the main server and the public service fallback. The packet header is changed on the process, and the DNS64 needs to translate the request to an adequate domain name first, meaning that it’s not recognizable IP-to-IP.

When I try to pull an image I get the error
Error response from daemon: failed to resolve reference "docker.io/library/hello-world:latest": failed to do request: Head "https://registry-1.docker.io/v2/library/hello-world/manifests/latest": dial tcp 54.x.x.x:443: connect: network is unreachable
(edited out the IP because I don’t know what it is)

The ipv4 displayed changes each time I try to pull and it won’t connect. I have tried the steps here Use IPv6 networking | Docker Docs about putting ipv6 in ~/.config/docker/daemon.json and putting Cloudflare ipv6 DNS in resolv.conf. I found some other posts too about removing the symlink on resolv.conf and that didn’t help either.

I have tried ping6 registry-1.docker.io and an IPv6 is displayed but I do not get a response from pinging it. I can curl https://registry-1.docker.io/v2/

I got fed up and installed Podman and somehow that worked fine. So I don’t know what is going on with rootless docker specifically on this strange network setup. If anyone could help me out, I’d really appreciate it.