Docker can't reach gateway after reinstall

Docker Hub
1

Hi,

I must have done something wrong during my uninstall/reinstall because now I can no longer use docker. I was wondering if anyone can help. I’m running docker 20.10.11 on Ubuntu 20.04

The other day I deleted a network in docker and ran into some issues, so I uninstalled/reinstall docker. However now I can’t even get ‘docker run hello-world’ to work.

I get the following error message:

x@x:~$ docker run hello-world
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: time="2021-12-02T09:42:45+01:00" level=fatal msg="failed to set gateway while updating gateway: route for the gateway 127.26.0.1 could not be found: invalid argument": unknown.
ERRO[0000] error waiting for container: context canceled

From what I understand the IP is correct, it shows as docker on ifconfig:

x@x:~$ ifconfig 
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 127.26.0.1  netmask 255.255.0.0  broadcast 127.26.255.255
        inet6 fe80::42:e7ff:fe02:c2d8  prefixlen 64  scopeid 0x20<link>
        ether 02:42:e7:02:c2:d8  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7  bytes 826 (826.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

I’ve tried an apt-get remove --prune to delete all the old config, rebooted and reinstalled but the issue persist. Maybe someone can help me figure out what I messed up?

Thanks!

2

It turns out there was a line in /etc/docker/daemon.json forcing the IP to be 127.26.0.1:
{ “bip”:“127.26.0.1/16”}. I don’t remember ever touching that file and particularly not in the last 5 days when I started having issues. So it’s a bit of a mystery how it got created, what I’m also not understanding is why this would persist even with a apt-get remove --purge… but that’s probably more an ubuntu question than a docker question. I’ve deleted the file and now it’s working.

For me what would be helpful to understand from the docker perspective is why this line succeeded in forcing the docker default bridge to be at 127.26.0.1 but failed to provide the pods a way to connect to the bridge. They knew the correct IP, but couldn’t reach it.
What was missing so that it would become a successful config allowing to move the default bridge to a different IPrange?

1 Like
3

When docker is installed, by default it does not create the file /etc/docker/daemon.json (at least not on ubuntu/centos).

Docker did, what you told it to do: it created the docker0 interface using the ip/cidrmask you configured as bip. That’s all this parameter is used for.

You need to configure the “fixed-cidr” as well, in order to let docker create its “bridge” network with this cidr.

See https://docs.docker.com/network/bridge/#connect-a-container-to-the-default-bridge-network for a full example. Try with the least amount of additional parameters, inspect the network and try again.

Add the additional key(s)/value(s)in /etc/docker/deamon.json and restart the docker service.
Inspect the “bridge” network and check if the new configuration applies to it (my expectation is that it doesn’t). Then delete the bridge network an either restart the docker service again.

4

Thanks, I don’t remember creating that file… but something or someone must have. I did see online that the file isn’t meant to exist by default.
For the fixed-cidr address do I use the first address in the range? (so 127.26.0.1) Or how do I know what address to put there?

5

As long as the fixed-cidr is within the subnet of the bip, it should work, I am not entrirely sure if "fixed-cidr": "127.26.0.1/16" works, but “fixed-cidr”: “172.26.0.1/17” or a usefull cidr netmask sufficient for the containers you plan to run in the default bridge - though, I would not use a cidr netmask higher than /26 (as this already is just ~64 ips).

1 Like