Hi, I’m trying to make sense of the IPv6 setup of docker.
TL;DR: I am trying to specify a range of IPv6 addresses as defaults for newly created networks but get the following error: Error response from daemon: could not find an available, non-overlapping IPv6 address pool among the defaults to assign to the network
I have tried a few things and ran into the following issues:
1) Changing the IPv6 subnet configuration in the daemon.json
It is mentioned in the docs, that in the deamon.json, fixed-cidr-v6 should be set to my desired v6 subnet.
I’ve thus tried two different things:
Setting it to my host’s address with a higher prefix (e.g. /80)
Using fc00:10:10:: with various prefixes
It is also stated that any changes made should come into effect after reloading the config file. However, I have found that the changes are not applied unless I restart docker, neither to existing networks (which does make sense) nor to newly created ones.
2) The bridge network hogs all prefixes
No matter what prefix I choose, the bridge network will always have the full size specified in fixed-cidr-v6. I’m not sure if that is intentional behavior. If so, the documentation should make that clear. However, them I’m unsure how to specify a pool of IPv6 addresses for the engine to choose from.
3) Restarting docker after changing the IPv6 prefix does not work
I’ve tested various prefixes and at some point started running into the following issue: After updating the daemon.json and reloading the config, docker would not restart. Instead, I got the following error:
failed to start daemon: Error initializing network controller: Error creating default "bridge" network: failed to add IPv6 address fc00:10:10::1/60 to bridge: file exists
This specific case was after I tried changing the prefix from /70 to /60. I’ve tried deleting the /var/lib/docker/network directory as advised in this post without success.
Any input on the above-mentioned points and especially my leading problem is greatly appreciated.
Alright, so I’ve gained some new insights, some of which seem obvious in hindsight:
About the original problem
I first specified the subnet explicitly when creating the network. After that, I looked into address pools a bit and found out how to specify them. I thus created a new pool of v6 addresses with a /96 prefix and size 112.
About 1) Choosing a v6 subnet base
For now, I’ve settled on using a subnet based on my machine’s address. I’m still unsure what the best practice here is.
About 2) The docker0 network
It is as I suspected in my post: The subnet specified in fixed-cidr-v6 defines what the docker0 network on the machine should look like. No more, no less. I still think this is poorly documented.
About 3) Restarting docker after a v6 prefix change
The issue I’ve identified here is the following: When restarting the docker service, the previously created networks are not removed. Thus, if you try to create another one with the same address (but another prefix), you get an error because that net is already taken.
The first confusing thing is that v4 and v6 work differently: you don’t have to give a fixed-cidr for v4 (it picks one from the pool itself), but you do for v6. That’s just a minor niggle, and the error message is clear if you omit fixed-cidr-v6.
The fundamental problem now is that I can only create a single network with IPv6. The first one works, the second one fails:
$ docker network create --ipv6 foo
877b4e2f39648a0fd47b2fee33f133d8c95ab94f08a070df39924fe18cec1586
$ docker network create --ipv6 bar
Error response from daemon: could not find an available, non-overlapping IPv6 address pool among the defaults to assign to the network
$
But the pool has a whole /49, so there are thousands of non-overlapping subnets for it to choose from (2001:db8:d0ca:1::/64 to 2001:db8:d0ca:7fff::/64)
I guess I need to try again with upstream binaries instead of the Ubuntu ones, to see if this is a fixed bug.
It’s the same with upstream docker-ce (5:20.10.21~3-0~ubuntu-jammy) and the same daemon.json I gave before:
nsrc@brian-kit:~$ docker network create --ipv6 foo
7f6ee1ac3ee7aaaa767469d51ebc1b40476ffe3d4dd8d6f1ce2ad21d52da9d83
nsrc@brian-kit:~$ docker network create --ipv6 bar
Error response from daemon: could not find an available, non-overlapping IPv6 address pool among the defaults to assign to the network
Digging on github, it looks like this is issue #41438 with underlying cause in #42801.
Workaround: create lots of separate /64 “pools” instead of an actual pool.
I think I shall revert to the Ubuntu version. No point having upstream repo enabled if it doesn’t actually fix bugs!
It is still recommended to use the repo provided by Docker. There are a lot of topics on the forum where one of our first questions is whether the user installed Docker from Docker’s repository. If not, we recommend to do that. Otherwise you can have different bugs or just different behaviors even if it didn’t fix your current issue. Following the official documentation is always the recommended way and it starts with a command that removes docker.io