Share and learn in the Docker community.
I’ve just realised that the default IP for the docker0 and docker_gwbridge bridge (172.18.0.0/16) conflicts with another network we already use, and can’t be changed.
This obviously prevents communication between docker and hosts on this network. Is there anyway to change the default IP range for this on the Azure hosts, as documented for a “normal” setup at Bridge network driver | Docker Docs
?
@markvr Can you please elaborate the specific connectivity issues you are having? When you say communication is being prevented between docker and hosts on the network, can you describe the topology a bit? Are the hosts that you refer to in the same virtual network in Azure (on the 10.0.0.0/8 network) as the Docker swarm nodes?
Note that the virtual network subnet hosting Docker swarm node VMs is configured on a 10.0.0.0/8 network and all traffic between the Docker nodes and any external hosts (within the Azure virtual network subnet) will, by default, be NATed. The external hosts won’t see the internal 172.18.0.0/16 addresses in the IP packets.
Sorry with hindsight there wasn’t anywhere near enough detail in this. We need the Docker Swarm to connect to our existing systems, and so have modified the ARM template slightly to deploy into an existing VNET and subnet, rather than provisioning it’s own. This vnet has routing (actually it’s a VPN) to an onsite network that coincidentally is on 172.18.0.0/16. This range conflicts with the range the “docker_gwbridge” is on.
I’ve actually just managed to half solve this, but it feels very hacky and brittle. For the “docker_gwbridge” (which for us happens to be the one that conflicts with our network) the docker docs say you can create the bridge in advance. So I tried adding 'docker network create --subnet 192.168.128.0/17 --opt com.docker.network.bridge.name=docker_gwbridge --opt com.docker.network.bridge.enable_icc=false docker_gwbridge' to the customData part of the template, and that has put the bridge onto the network specified. This seems to survive OK across rolling swarm updates as well, so I think we are going to go with this.
I don’t know how it to change the docker0 bridge, this is created earlier in the init process.
I’m not sure what most peoples use-cases are for this project. For greenfield deployments the current template is probably fine, but you would have to fit other non-docker systems around the docker configuration. For our use-case which is trying to move existing legacy systems to a more modern platform, the current template would benefit from being more customisable. E.g. by being able to specify network ranges etc so they can be compatible with existing networks, or by being able to specify existing storage shares to the “cloudstor” plugin.
Ah I see. Your approach of pre-configuring the bridge in advance certainly works since swarm init/joining happens at the end of custom data processing as part of init-azure .
The network customization/bring-your-own Virtual Network feature has not yet been implemented for Azure - it’s something that we will get to in a future release. So custom modifications to the template is the only way out right now.
Regarding configuration of cloudstor shares, we will introduce the ability to specify your own share name (within one of the storage accounts provisioned in the Docker4Azure resource group as well as other attributes like uid/gid for the mount) during volume creation shortly. However note that these are completely self contained within the cloudstor plugin and not something that affects the ARM template. Are you looking for an ability to also configure your own storage account as the backing store for shares used by cloudstor?
I couldn’t get the docker0 bridge to change, because I think that needs to be done by configuring DOCKER_OPTS before the docker engine itself starts up, and I couldn’t see anyway to hook into that. Luckily for us it was the “gwbridge” that conflicts.
This has gone somewhat off topic, but r.e. cloudstor I would like to be able to specify storage that is outside of the “docker-for-azure” resource group.
This is because if all data is stored outside the “docker-for-azure” resource group (i.e. databases, redis cache, file storage etc) it makes the resource group disposable. i.e. in a DR scenario, or if anything breaks (and I’ve broken it plenty in testing mainly by accidentally losing the raft quorum) then, it’s relatively easy to use version controlled templates and compose files to re-provision the entire stack and services in a new resource group, connect the services to the existing datastores, swap some DNS records, and delete the old resources. This could possibly even be done as a very safe way of doing blue/green swarm updates.
Not unlike having separate OS and data disks on a server, but on a slightly larger scale. The default could be like now, which is very easy to use, but just have some extra optional settings where needed. It would separate the “transient” data like OS disks etc from the business data.
You can also change the IP address and subnet of the bridge via the /etc/docker/daemon.json file (create it if needed). The “bip” entry defines the docker0 subnet. Here is an example of the /etc/docker/daemon.json file:
{
"bip": "172.17.1.1/24",
"ipv6": false,
"storage-driver": "devicemapper",
"storage-opts":
[
"dm.basesize = 41G",
"dm.thinpooldev=/dev/mapper/docker--vg-thinpool",
"dm.use_deferred_removal=true"
]
}thanks, the problem is specific to the docker-for-azure project though, because it isn’t possible to edit that file.
I recently had a client with a similar issue where 172.17.1.1 was actually the subnet of his company’s wifi. I found a workaround for him but it’s also quite hacky:
ip addr add dev docker0 172.18.0.1/16 – This adds a new IP range. You can use whatever range you want.
ip addr del dev docker0 172.17.0.1/16 – This deletes the old range
Here’s the problem, Docker is hard coded to look for 172.17.0.1. Changing this will require a rewrite of Docker’s code. If it can’t find 172.17.0.1/16, then any container using the default network will lose connectivity. Also, Docker recreates bridge0 every time it is loaded, so you will need to change the range again every time you reboot or restart the docker service.
Here’s the workaround to get your containers working after you change the bridge IP.
docker network create tempnet – This creates a new temporary network. You can name it what you want.
New containers that would normally use the default network will need to include “–net tempnet” in order.
Example:
docker run -it --net tempnet opensuse /bin/bash
I found the following post which solved the issue:
So in my case, the following modified the address permanently. Best of Luck.
sudo systemctl stop docker
touch daemon.json
echo ‘{’ > daemon.json
echo ’ “bip”: “192.168.1.5/24”,’ >> daemon.json
echo ’ “fixed-cidr”: “192.168.1.5/25”’ >> daemon.json
echo ‘}’ >> daemon.json
sudo cp daemon.json /etc/docker
sudo systemctl start docker
rm daemon.json
ip addr
This worked for me…! Thank you hank. I also had issues with cidr conflict issue with vpn network, which prevented SSH to the host server in AWS.
I shortened solution to the following 3 lines:
sudo printf '{\n "bip": "172.18.1.1/16",\n "ipv6": false,\n "fixed-cidr": "172.18.1.1/24"\n}\n' > /etc/docker/daemon.json
sudo systemctl restart docker
ip aI am noticing when I set the the bip to 192.168.0.0/16 subnet in the daemon.json file it is not working, Any other IP subnet is working. Any idea what could be the issue. When I set the 192.168.0.0/16 subnet and start the docker service, the container IPs are from the last configuration, in my case it was 172.19.x.x
For anyone’s future reference when setting bip in daemon.json you are setting the internal gateway for your docker containers. In @srikrame case it would need to be 192.168.0.1/16. It is not asking for the network name (ie 192.168.0.0/16)