Docker creating routes which break networking for host

Docker Engine General
, , ,
1

Hello All,

I recently did an in place upgrade from Debian Bullseye to Bookworm. Once I did the transition, I noticed some networking issues and saw that Docker was creating 4 types of routes on my host’s network:
(Note: “host” in this post refers to the host device, NOT docker’s “host network”).

  1. 172.X.X.X dev docker0/br-* - The IP for each docker network gateway. This I have seen before on other systems with docker.
  2. 0.0.0.0 dev veth* - The default routes usually kept inside docker’s internal network management.
  3. 169.254.0.0 dev veth* - What I can only guess are null networks, usually kept inside docker network.
  4. default dev veth* - Only one of these is made, and it seemingly chooses one veth randomly. Should the container linked to that veth be stopped, another veth will randomly be chosen (unknown how or by what) to become the new default route.

When 4 appears, it stops the networking for my host system externally (e.g. ping 8.8.8.8 gives you host unreachable). You can still connect to to the host directly via IP, but not via hostname.

Points 2-4 make me believe that Docker is putting its internal networking iptables (or equivalent) on to the host system (which is then breaking the host system) rather than only “talking” to the host via 172.X.X.X.

This made me think something is screwy with the network, which makes sense given the changes made in networking services in different Debian releases.

First I checked daemon.json - empty.
Then I checked what network services I had running - systemd-networkd and connman. I also had installed netplan, which put together a basic networkd config. I thus checked connman’s main.conf and /{etc,lib}/systemd/network/* files vs another debian system I have running that was a fresh install. There was no difference.

I have checked my iptables on the two systems as well, and they also were the same.

The only difference is in ip routing.

I also scoured the internet (and used chatgpt) to see if I could debug this. The most common answer was to blacklist veth entries via connman’s main.conf. Unfortunately, when I do that, all connectivity to my docker containers dies.

The workaround I have currently is to search for and delete any route matching the format “default dev veth”. It’s a basic script that I have run every minute. In this configuration, everything (except intranet DNS resolution, which is likely a separate issue) works, although the network for my container can die up to every minute, which makes some of the service pretty unreliable.

Would really appreciate any pointers on how to fix this. More system info below. TIA.

docker version

Client: Docker Engine - Community
 Version:           28.0.1
 API version:       1.48
 Go version:        go1.23.6
 Git commit:        068a01e
 Built:             Wed Feb 26 10:40:57 2025
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.0.1
  API version:      1.48 (minimum version 1.24)
  Go version:       go1.23.6
  Git commit:       bbd0a17
  Built:            Wed Feb 26 10:40:57 2025
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.2.4
  GitCommit:        v1.2.4-0-g6c52b3f
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    28.0.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.21.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.33.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 14
  Running: 13
  Paused: 0
  Stopped: 1
 Images: 14
 Server Version: 28.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.2.4-0-g6c52b3f
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.21-v8+
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 3.705GiB
 Name: [redacted]
 ID: [redacted]
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
2

Please, format your post according to the following guide: How to format your forum posts
In short: please, use </> button to share codes, terminal outputs, error messages or anything that can contain special characters which would be interpreted by the MarkDown filter. Use the preview feature to make sure your text is formatted as you would expect it and check your post after you have sent it so you can still fix it.

Example code block:

```
echo "I am a code."
echo "An athletic one, and I wanna run."
```

After fixing your post, please send a new comment so people are notified about the fixed content.

3

And you didn’t, so you also didn’t get new replies because nobody knew about the change. “Fortunately” I came late.

That is indeed familiar from the other topic you linked and whre I commented. I can only say the same. Docker can add its networks and iptables rules, but it will not add a default route. It wouldn’t make sense. So it must be added by a network manager. I never had to configure conman, so I can’t tell you how it should be set. Is it a default software in Debian bookworm? I’m asking because Docker CE is officially supported on Bookworm, so no default software should break it. Or maybe it was because of the upgrade and the Docker CE installer would have recognized it otherwise. Was Docker also reinstalled to come from the right repository?

4

Thank you for catching it. New to the forums.

It is indeed a reinstalled Docker from the official instructions, as I thought the issues were caused by a stuck old version with different networking rules. I deleted the networking rules between uninstall and reinstall.

I hadn’t done anything custom to connman nor systemd-networkd before this started happening. If anything I’ve reverted everything to default and switched from the previous resolver to resolved (which is recommended for bookworm).

Any other suggestions for debugging? I appreciate the help.

5

Unfortunately, no. I only had problems with NetworkManager years ago and based on the other topic, it was similar to the issue with conman, but never had it with networkd and I never used conman, so you will need to wait for someone who did.

Update:

I just noticed the architecture in your docker info output. Is your host a standard Debian host or any special Debian based distribution that still shows it is Debian?

6

It’s a Raspberry Pi running standard Debian (see below).

Anything special for Pis? I would have guessed the networking stack is standard and this is somehow linked to how network modules changed from bullseye to bookworm.

lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm
7

Probably not as the official guide states the Debian guide should be followed for RPi arm64. I don’t know if the distribution installer itself could install different packages on a Rasberry Pi or configure anything differently. I don’t see why.

If you have time to try debian in a virtual machine not on Rasberry, you can check if you get the same issue, or if you think it is a bug in the current version of Docker Engine which somehow makes conman work differently, you can try reporting it on GitHub: GitHub · Where software is built

If the distro is not a modified one, Docker CE should work as an officially supported engine. If any special configuration is needed for the current conman version to work with the current Docker CE version, that could be included in the installation guide.