Unable to change subnet on docker network bridge

General Discussions
1

Hi all!

I have a situation at work. After some years of using Docker, some team realized that this subnet was causing errors with some packages.

Altough the docker network is not using (there are containers running but all of them are on the calico network), docker doesn’t allow to change the subnet.

All the info and documentation that I found to change the subnet is the same: change the “–bip” option and restart the docker service. Sometimes even you can remove the network database. However. everytime I did that, I have the same issue:

[root@server docker]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─50-CPUAccounting.conf, 50-MemoryAccounting.conf, http-proxy.conf
   Active: failed (Result: exit-code) since Tue 2023-08-29 05:57:53 CDT; 3s ago
     Docs: http://docs.docker.com
  Process: 14434 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=exited, status=1/FAILURE)
 Main PID: 14434 (code=exited, status=1/FAILURE)

Aug 29 05:57:52 server dockerd-current[14434]: time="2023-08-29T05:57:52.407937367-05:00" level=warning msg="could not change group /var/run/docker.sock to docker...not found"Aug 29 05:57:52 server dockerd-current[14434]: time="2023-08-29T05:57:52.424434973-05:00" level=info msg="libcontainerd: new containerd process, pid: 14446"
Aug 29 05:57:53 server dockerd-current[14434]: time="2023-08-29T05:57:53.442168839-05:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Aug 29 05:57:53 server dockerd-current[14434]: time="2023-08-29T05:57:53.443551745-05:00" level=info msg="Loading containers: start."
Aug 29 05:57:53 server dockerd-current[14434]: time="2023-08-29T05:57:53.451553970-05:00" level=info msg="Firewalld running: false"
Aug 29 05:57:53 server dockerd-current[14434]: Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to al...ady in useAug 29 05:57:53 server systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Aug 29 05:57:53 server systemd[1]: Failed to start Docker Application Container Engine.
Aug 29 05:57:53 server systemd[1]: Unit docker.service entered failed state.
Aug 29 05:57:53 server systemd[1]: docker.service failed.

Steps that I did:

  1. Change “–bip” configuration:
[root@server ~]# cat /etc/sysconfig/docker-network
# /etc/sysconfig/docker-network
#DOCKER_NETWORK_OPTIONS= "--bip=10.27.0.1/16" #old network
DOCKER_NETWORK_OPTIONS= "--bip=169.254.0.0/16" #new network

  1. (optional) Remove network database:
    [root@server ~]# rm /var/lib/docker/network/files/local-kv.db

  2. Restart service:
    [root@server ]# systemctl restart docker

The error after this steps is the one above.

Since this server is a Kubernetes worker node, I stop kubelet before too. I even remove all the containers, networks, volumes, etc from the node but same result.

What am I miss? Or what am I doing wrong?

More info:

[root@server ~]# docker info
Containers: 31
 Running: 23
 Paused: 0
 Stopped: 8
Images: 14
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 8891bca22c049cd2dcf13ba2438c0bac8d7f3343 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
Kernel Version: 3.10.0-1160.36.2.el7.x86_64
Operating System: Red Hat
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 8
Total Memory: 31.41 GiB
Name: server
ID: HAJC:NQT4:SVPH:HH6M:EAOA:7C6X:IOTM:X4KE:T4LR:HINI:6QIO:RW3D
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: XXXXXXXXXXXXX
Https Proxy: XXXXXXXXXXX
No Proxy:XXXXXXXXXXXXXX
Registry: https://registry.access.redhat.com/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: registry.access.redhat.com (secure), registry.redhat.io (secure), docker.io (secure), docker.io (secure)

Thanks in advance.

Edit: remove sensitive information.

2

More info.

As I said, there aren’t containers under the bridge network. All of them are on the calico network:

[root@server ~]# docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "46a3ea0236b06196307528e19fb9049ee2f17a63b1c7604f8681e9c3436150d4",
        "Created": "2023-08-30T06:31:39.289238774-05:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.27.0.1/16",
                    "Gateway": "10.27.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

I also tried to remove the docker0 interface but I had the same result all the time.