[SOLUTION] Docker IPV6 and Docker compose woes

Docker Engine Compose
1

I started this document asking for help and while writing this I ended up finding a solution so i figured I’d share anyways.

I followed this this guide to setup ipv6 for docker.

In my case I have this in my daemon.json:

{
  "ipv6": true,
  "fixed-cidr-v6": "fd00::/80"
}

AND!!! needs custom iptables rules.

ip6tables -t nat -A POSTROUTING -s fd00::/80 ! -o docker0 -j MASQUERADE

It took some mucking around but eventually i got working and this line seems to work.

#!/usr/bin/env bash
docker run --rm -t busybox ping6 -c 4 google.com

output:

PING google.com (2607:f8b0:400a:808::200e): 56 data bytes
64 bytes from 2607:f8b0:400a:808::200e: seq=0 ttl=119 time=17.133 ms
64 bytes from 2607:f8b0:400a:808::200e: seq=1 ttl=119 time=17.119 ms
64 bytes from 2607:f8b0:400a:808::200e: seq=2 ttl=119 time=17.281 ms
64 bytes from 2607:f8b0:400a:808::200e: seq=3 ttl=119 time=17.430 ms

--- google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 17.119/17.240/17.430 ms

Okay great, So at least docker knows how to speak ipv6. Now docker-compose.

I ran into multiple references this and this one that point out that the latest compose format of compose doesn’t support IPv6 and you have to use the 2.1 format.

~/ $ docker -v                                                                                                                                                     
Docker version 19.03.12, build 48a66213fe
~/ $ docker-compose -v                                                                                                                                             
docker-compose version 1.26.2, build eefe0d31

So I believe that each time docker-compose runs unless i’m explicitly specifying an external network, it’s creating a new network for my stack. So even though docker itself works fine, compose needs more work.

The first attempt is this:

version: "2.1"
services:
  busy:
    image: busybox
    command: ping6 -c 4 google.com

out:

busy_1  | PING google.com (2607:f8b0:400a:808::200e): 56 data bytes
busy_1  | ping6: sendto: Network is unreachable

So, trying to define everything I did for docker in compose:

version: "2.1"
services:
  busy:
    image: busybox
    command: ping6 -c 4 google.com
    networks:
      - app_net


networks:
  app_net:
    enable_ipv6: true
    driver: bridge
    driver_opts:
      com.docker.network.enable_ipv6: "true"
    ipam:
      driver: default
      config:
       - subnet: 172.16.238.0/24
         gateway: 172.16.238.1
       - subnet: 2001:3984:3989::/64
         gateway: 2001:3984:3989::1

out:

busy_1  | PING google.com (2607:f8b0:400a:808::200e): 56 data bytes
busy_1  |
busy_1  | --- google.com ping statistics ---
busy_1  | 4 packets transmitted, 0 packets received, 100% packet loss

and last iteration explicitly choosing an IP

version: "2.1"
services:
  busy:
    image: busybox
    command: ping6 -c 4 google.com
    networks:
      app_net:
          ipv4_address: 172.16.238.10
          ipv6_address: 2001:3984:3989::10



networks:
  app_net:
    enable_ipv6: true
    driver: bridge
    driver_opts:
      com.docker.network.enable_ipv6: "true"
    ipam:
      driver: default
      config:
       - subnet: 172.16.238.0/24
         gateway: 172.16.238.1
       - subnet: 2001:3984:3989::/64
         gateway: 2001:3984:3989::1

out:

busy_1  | PING google.com (2607:f8b0:400a:808::200e): 56 data bytes
busy_1  |
busy_1  | --- google.com ping statistics ---
busy_1  | 4 packets transmitted, 0 packets received, 100% packet loss

Finally, the last thing I forgot is that since i’m defining a new network, I need to also create an iptables rule.

sudo ip6tables -t nat -A POSTROUTING -s  2001:3984:3989::/64 ! -o docker0 -j MASQUERADE

it works!!

busy_1  | PING google.com (2607:f8b0:400a:808::200e): 56 data bytes
busy_1  | 64 bytes from 2607:f8b0:400a:808::200e: seq=0 ttl=119 time=17.087 ms
busy_1  | 64 bytes from 2607:f8b0:400a:808::200e: seq=1 ttl=119 time=17.041 ms
busy_1  | 64 bytes from 2607:f8b0:400a:808::200e: seq=2 ttl=119 time=17.042 ms
busy_1  | 64 bytes from 2607:f8b0:400a:808::200e: seq=3 ttl=119 time=17.100 ms

So basically what I got out of this is this.

Summary

  • Docker IPv6 docs is mostly complete, requires an iptables rule to be in place.
  • Docker-compose works, but requires file format to be set to 2.1
  • For both docker and compose you will need to maintain your own routing rules via iptables in order for any traffic to be allowed to flow through.

Also for reference to remove the rule you’d replace the -A with a -D, example:

 ip6tables -t nat -D POSTROUTING -s  2001:3984:3989::/64 ! -o docker0 -j MASQUERADE

Have I missed something? This seems incredibly over complicated. The whole appeal of docker is that it hides much of complexities. You define the ports, the app create an image and then use compose and all the routing and such just works. It creates/destroys networks as needed.

I’m glad i finally figured out how to get IPv6 to work, but this is Not exactly trivial, not to mention since we’re changing firewall rules, it requires root access for each compose definition being deployed.

Any thoughts?

2

Thanks for sharing this. I am sure it will come handy in the near future. My only suggestion is to create an issue in docker github under compose. They can triage as a bug or as a feature request.

3

There has been several bugs open and closed regarding the topic, including the one from the original post I made there’s also these.



I’m not sure if this is still valid, but there was a bug that mentioned that swarm didn’t support IPV6 so compose files greater than 2.1 are not supporting IPV6. They’re not planning on fix it till that comes about.

Side note, this pattern relies on an a docker container managing your routing/iptables which has security implications but this works great. You don’t have to manage your own iptables rules and it behaves just like IPV4 does.

version: "2.1"
services:
  busy:
    image: busybox
    command: ping6 -c 4 google.com
    networks:
      - app_net


  ipv6:
    image: robbertkl/ipv6nat
    restart: unless-stopped
    network_mode: "host"
    privileged: true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /lib/modules:/lib/modules:ro

networks:
  beef:
    enable_ipv6: true
    driver: bridge
    driver_opts:
      com.docker.network.enable_ipv6: "true"
    ipam:
      driver: default
      config:
       - subnet: 2001:3984:3989::/64

I wrote a blog post about this that’s a bit more organized.

4

Why do you need the masquerade rule if you use a global ipv6 prefix? If your addresses aren’t routable and you need to use masquerade then why use global addresses instead of ULAs.

5

I need a masquerade rule because i want traffic to show as it comes from the host it’s running on, not some arbitrary IPv6 address assigned to the container.

6

I will add one more note, though not sure if it covers all edge cases. I have noticed that setting network_mode: host seems to fix a lot of the IPv6 issues. Though it does feel like the network equivalent of running everything as root.

Just a nugget of info for anyone struggling with this issue still.

1 Like
7

why not macvlan with it’s own ipv6 address. I’m trying to accomplish this for an adguard situation and can’t quite get it.