No connection to the bridge by default?

[pathsofglory31]

I am experiencing an issue where there is no connection to the internet from docker containers. I spent many hours looking for a solution, and I found one which is not ideal.

When the container is up, I run the following command to inspect the ethernet bridge configuration:

brctl show

which generates the following output:

bridge name     bridge id               STP enabled     interfaces
br-64c19992619a         8000.024253e6c1d2       no
br-c624d7c73543         8000.0242e068f9e8       no
docker0         8000.0242d03f9d44       no

As you can see there is no interface associated with docker0. To fix this, I need to run the following command:

ip link

to display my network interfaces (while the container is running):

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp34s0: <BROADCAST,MULTICAST,DYNAMIC,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:d8:61:a7:28:55 brd ff:ff:ff:ff:ff:ff
91: br-64c19992619a: <NO-CARRIER,BROADCAST,MULTICAST,DYNAMIC,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:53:e6:c1:d2 brd ff:ff:ff:ff:ff:ff
98: br-c624d7c73543: <NO-CARRIER,BROADCAST,MULTICAST,DYNAMIC,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:e0:68:f9:e8 brd ff:ff:ff:ff:ff:ff
193: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:d0:3f:9d:44 brd ff:ff:ff:ff:ff:ff
213: veth2241155@if212: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 4a:22:d9:c5:77:9e brd ff:ff:ff:ff:ff:ff link-netnsid 0

Then I have to copy the string that starts with “veth” and run the following command:

sudo ip link set veth2241155 master docker0

so that I can get internet connection in my container.

bridge name     bridge id               STP enabled     interfaces
br-64c19992619a         8000.024253e6c1d2       no
br-c624d7c73543         8000.0242e068f9e8       no
docker0         8000.0242d03f9d44       no              veth2241155

This solution is far from ideal. Any idea why it doesn’t happen automatically? The operating system is arch linux.

[meyay (Metin Y.)]

The bridge docker0 is only used by containers that are attached to the default bridge. If you run a container without providing the --network argument, it should be attached to docker0. The same is true for docker build.

Compose deployments create their own bridge and containers will be attached to it, instead to docker0. Swarm deployments will appear as attached to docker_gwbridge.

Back to the topic: it is indeed odd that the veth interface was not assigned to any of the bridges. So far I have no idea what causes this behavior.

Could you provide the output of docker version and docker info, so we can see which version you use?

You could check, whether missing kernel modules are responsible for this behavior.

curl -sL https://github.com/moby/moby/raw/master/contrib/check-config.sh | bash

And of course you can check the docker logs for errors:

sudo journalctl -u docker

[pathsofglory31]

docker version

Client:
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.20
 Git commit:        a5ee5b1dfc
 Built:             Sat Feb 11 13:58:04 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.20
  Git commit:       bc3805a0a0
  Built:            Sat Feb 11 13:58:04 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.6.16
  GitCommit:        31aa4358a36870b21a992d3ad2bef29e1d693bec.m
 runc:
  Version:          1.1.4
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc.)
    Version:  2.16.0
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 23.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.11-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.54GiB
 Name: pc-ms7b86
 ID: 2f90a560-0c54-44b3-873c-3307692499f1
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

curl -sL https://github.com/moby/moby/raw/master/contrib/check-config.sh | bash

Generally Necessary:
- cgroup hierarchy: cgroupv2
  Controllers:
  - cpu: available
  - cpuset: available
  - io: available
  - memory: available
  - pids: available
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_CGROUP_BPF: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: missing
    (cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_SECURITY_SELINUX: enabled
- CONFIG_SECURITY_APPARMOR: enabled
- CONFIG_EXT4_FS: enabled (as module)
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled (as module)
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled (as module)
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

sudo journalctl -u docker

lut 15 13:58:45 pc-ms7b86 systemd[1]: Starting Docker Application Container Engine...
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.541976419+01:00" level=info msg="Starting up"
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.545744139+01:00" level=info msg="[core] [Channel #1] Channel created" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.545772793+01:00" level=info msg="[core] [Channel #1] original dial target is: \"unix:///run/containerd/containerd.sock\"" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.545815569+01:00" level=info msg="[core] [Channel #1] parsed dial target is: {Scheme:unix Authority: Endpoint:run/containerd/containerd.sock URL:{Scheme:unix Opaque: User: Host: Path:/run/containerd/containerd.sock RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.545834272+01:00" level=info msg="[core] [Channel #1] Channel authority set to \"localhost\"" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.546287765+01:00" level=info msg="[core] [Channel #1] Resolver state updated: {\n  \"Addresses\": [\n    {\n      \"Addr\": \"/run/containerd/containerd.sock\",\n      \"ServerName\": \"\",\n      \"Attributes\": {},\n      \"BalancerAttributes\": null,\n      \"Type\": 0,\n      \"Metadata\": null\n    }\n  ],\n  \"ServiceConfig\": null,\n  \"Attributes\": null\n} (resolver returned new addresses)" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.546831512+01:00" level=info msg="[core] [Channel #1] Channel switches to new LB policy \"pick_first\"" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.548393041+01:00" level=info msg="[core] [Channel #1 SubChannel #2] Subchannel created" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.548438368+01:00" level=info msg="[core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.548470952+01:00" level=info msg="[core] [Channel #1 SubChannel #2] Subchannel picks a new address \"/run/containerd/containerd.sock\" to connect" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.548476783+01:00" level=info msg="[core] [Channel #1] Channel Connectivity change to CONNECTING" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.550807621+01:00" level=info msg="[core] [Channel #1 SubChannel #2] Subchannel Connectivity change to READY" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.550844446+01:00" level=info msg="[core] [Channel #1] Channel Connectivity change to READY" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.552493548+01:00" level=info msg="[core] [Channel #4] Channel created" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.552505179+01:00" level=info msg="[core] [Channel #4] original dial target is: \"unix:///run/containerd/containerd.sock\"" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.552521992+01:00" level=info msg="[core] [Channel #4] parsed dial target is: {Scheme:unix Authority: Endpoint:run/containerd/containerd.sock URL:{Scheme:unix Opaque: User: Host: Path:/run/containerd/containerd.sock RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.552531303+01:00" level=info msg="[core] [Channel #4] Channel authority set to \"localhost\"" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.552550636+01:00" level=info msg="[core] [Channel #4] Resolver state updated: {\n  \"Addresses\": [\n    {\n      \"Addr\": \"/run/containerd/containerd.sock\",\n      \"ServerName\": \"\",\n      \"Attributes\": {},\n      \"BalancerAttributes\": null,\n      \"Type\": 0,\n      \"Metadata\": null\n    }\n  ],\n  \"ServiceConfig\": null,\n  \"Attributes\": null\n} (resolver returned new addresses)" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.552569178+01:00" level=info msg="[core] [Channel #4] Channel switches to new LB policy \"pick_first\"" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.552586761+01:00" level=info msg="[core] [Channel #4 SubChannel #5] Subchannel created" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.552600523+01:00" level=info msg="[core] [Channel #4 SubChannel #5] Subchannel Connectivity change to CONNECTING" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.552614595+01:00" level=info msg="[core] [Channel #4 SubChannel #5] Subchannel picks a new address \"/run/containerd/containerd.sock\" to connect" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.552626317+01:00" level=info msg="[core] [Channel #4] Channel Connectivity change to CONNECTING" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.553888744+01:00" level=info msg="[core] [Channel #4 SubChannel #5] Subchannel Connectivity change to READY" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.553916508+01:00" level=info msg="[core] [Channel #4] Channel Connectivity change to READY" module=grpc
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.614565333+01:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.621038403+01:00" level=info msg="Loading containers: start."
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.768807766+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.790255111+01:00" level=info msg="Loading containers: done."
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.871223058+01:00" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" storage-driver=overlay2
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.871401183+01:00" level=info msg="Docker daemon" commit=bc3805a0a0 graphdriver=overlay2 version=23.0.1
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.872307046+01:00" level=info msg="Daemon has completed initialization"
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.882941994+01:00" level=info msg="[core] [Server #7] Server created" module=grpc
lut 15 13:58:45 pc-ms7b86 systemd[1]: Started Docker Application Container Engine.
lut 15 13:58:45 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:58:45.888747795+01:00" level=info msg="API listen on /run/docker.sock"
lut 15 13:59:00 pc-ms7b86 dockerd[1934]: time="2023-02-15T13:59:00.662529860+01:00" level=info msg="ignoring event" container=816570a5bee586a62badad33dc81662d691c7b794ab0077c54d45169c6dd5687 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
lut 15 14:02:17 pc-ms7b86 dockerd[1934]: time="2023-02-15T14:02:17.820464957+01:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=5df30894bd49fc12fdf57af8da1429447e6b432f9a5d8b6f3843a3174d3d18e9
lut 15 14:02:17 pc-ms7b86 dockerd[1934]: time="2023-02-15T14:02:17.862579336+01:00" level=info msg="ignoring event" container=5df30894bd49fc12fdf57af8da1429447e6b432f9a5d8b6f3843a3174d3d18e9 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
lut 15 14:02:17 pc-ms7b86 dockerd[1934]: time="2023-02-15T14:02:17.868020224+01:00" level=error msg="attach failed with error: error attaching stdout stream: write unix /run/docker.sock->@: write: broken pipe"

[pathsofglory31]

I fixed it. I had to run the following:

~$ sudo systemctl stop systemd-networkd.service                                                    
~$ sudo systemctl disable systemd-networkd.service                                                        
~$ sudo systemctl stop systemd-networkd.socket                                                            
~$ sudo systemctl disable systemd-networkd.socket
~$ sudo systemctl start NetworkManager
~$ sudo systemctl enable NetworkManager

[rimelek (Ákos Takács)]

NetworkManager usually manages network on desktop machines. There was a similar issue recently

Complete failure of Docker

After a long long trial and error, the issue has resolved by adjusting systemd-networkd config. I used it to configure DHCP for the ethernet adapter. However, I did it with this config file /etc/systemd/network/20-local-wired.network: [Match] Name=* [Network] DHCP=yes Although this was a heat-of-the-moment quickie to simplify the netplan/networkmanager mess, worked all right until docker’s fancy per-container interfaces appeared. I don’t know why, but systemd-networkd somehow silently and we…

Make sure you configure networkd properly so Docker network interfaces can be created by Docker.
If NetworkManager is good for you, you can keep that, but I wouldn’t use it on a server.

[wundo]

may i ask why using network manager is not a good idea for use on a sever??

[rimelek (Ákos Takács)]

As far as I know it is not the default on servers, at least not on servers I use (debian based). Also quote from NetworkManagers website:

NetworkManager for administrators

Server-like behavior

By default NetworkManager automatically creates a new in-memory connection for every Ethernet device that doesn’t have another candidate connection on disk. These new connections have name “Wired connection 1”, “Wired connection 2” and so on; they have DHCPv4 and IPv6 autoconfiguration enabled.

This behavior is usually not desirable on servers, where all interfaces should be configured explicitly. To disable the creation of such automatic connections, add no-auto-default=* to the [main] configuration section.

Also, NetworkManager requires carrier on an interface before a connection can be activated on it. If there are services that need to bind to the interface address at boot, they might fail if the interface has no carrier. The solution is to disable carrier detection with configuration option ignore-carrier=* in the [main] section.

Note that on Fedora and RHEL there is a NetworkManager-config-server package that install a configuration snippet with the two options described above.

It doesn’t mean you can’t use it, only that I wouldn’t as I know how to configure netplan with networkd properly On a desktop machine, it helps to detect wifi connections and connect to it when you click on a button on GUI.

update:

In my original post I accidentally wrote “netplan with NetworkManager”. I meant of course “netplan with networkd”.