Inconsistent container internet access behavior on IPv6 only server

I’m working on deploying a development environment to a server. I tested my docker compose application on win 10 machine with docker desktop installed and all worked well. All containers communicate with each other using a basic bridge network (my-network), some containers are API server, one container (my mail-handler) requires access to an external mail-server.

I ran into some inconsistencies when trying to deploy the application on a Debian 11 server that operates in IPv6 mode only. After running through the basic IPv6 hurdles to access the images things worked well with one exception; the containers can communicate, the API is available through the ports I configured to be exposed, however the mail-handler can not reach the external mail-server.

Several hours were spend trying to troubleshoot the issue. I found that when I use the direct IPv6 address of the mail-server and I manually connect the mail-handler to my-network it works again.

So I tried figuring out why this happens. After some simple tests this works:

$ docker run -it --rm registry.ipv6.docker.com/library/alpine ping6 -c 2 google.com
$ docker run -it --rm registry.ipv6.docker.com/library/alpine nslookup google.com

Now using compose without custom network:

version: "3.9"

services:
    demo_ping:
        container_name: ping_no_network
        image: registry.ipv6.docker.com/library/alpine 
        entrypoint: ping6 -c 2 google.com
    demo_ping_bridge:
        container_name: ping_no_network_with_bridgemode
        image: registry.ipv6.docker.com/library/alpine 
        network_mode: bridge
        entrypoint: ping6 -c 2 google.com
    demo_ns:
        container_name: ns_no_network
        image: registry.ipv6.docker.com/library/alpine 
        entrypoint: nslookup google.com
    demo_ns_bridge:
        container_name: ns_no_network_with_bridgemode
        image: registry.ipv6.docker.com/library/alpine 
        network_mode: bridge
        entrypoint: nslookup google.com

Interestingly now the container with network_mode: bridge pings and nslookups, the other fails

Now with a custom network:

version: "3.9"

services:
    demo_ping:
        container_name: ping_with_network
        image: registry.ipv6.docker.com/library/alpine 
        entrypoint: ping6 -c 2 google.com
        networks: 
            - dummy-net
    demo_ping_bridge:
        container_name: ping_with_network_and_bridgemode
        image: registry.ipv6.docker.com/library/alpine 
        network_mode: bridge
        networks: 
            - dummy-net
        entrypoint: ping6 -c 2 google.com
    demo_ns:
        container_name: ns_with_network
        image: registry.ipv6.docker.com/library/alpine 
        entrypoint: nslookup google.com
        networks: 
            - dummy-net
    demo_ns_bridge:
        container_name: ns_with_network_and_bridgemode
        image: registry.ipv6.docker.com/library/alpine 
        entrypoint: nslookup google.com
        network_mode: bridge
        networks: 
            - dummy-net
        

networks:
    dummy-net:
        name: dummy-net
        driver: bridge

Both pings and nslookups fail in this case. The workaround of manually connecting the container to the default bridge network did not resolve the issue.

So in summary.

  • Docker containers run directly ping6 and nslookup successfully on the IPv6 server.
  • Docker compose containers without custom network only ping and nslookup successfully if network_mode: bridge
  • Docker compose containers with a custom network never pings or nslookups successfully
  • In my application I have a workaround using docker network connect bridge mail-handler and restarting the mail-handler. This does not reproduce in my simple example.

It would be appreciated if anyone help me figure out what the problem is. Although I have a working workaround it feels like a solution that may fail in the future.

I’m still having this trouble. However connecting the container to the bridge network does not work anymore.

I posted a question on stack overflow where someone pointed out that fe00::0 ip6-localnet is supposed to be fe80::0 ip6-localnet (note the fe80)

Why does docker assign fe00 and not fe80?