Access to internet from containers connected to bridge

General
1

Hello all,
I’m now cracking my nuts trying to get containers connected to custom bridge network reach to internet.
Small background, I have internet facing server running the Docker daemon. As all my applications are web applications I need to have single point for maintaining SSL certificates and access rights to them.

As a solution I deployed Nginx reverse proxy (nginx-proxy-manager) that is handling all incoming traffic, serving the SSL certificates and deployed all web containers into the same custom network bridge.

It turned out, that in this way applications can communicate each other (so the reverse proxy can pass the traffic to them which is what I wanted), but they are unable to reach out the internet.

Is there any possible way how to with this setup allow containers speak to internet?

Many thanks for eventual hints.

lsb_release -a


No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

docker version

Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:20 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:26:12 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.15
  GitCommit:        5b842e528e99d4d4c1686467debf2bd4b88ecd86
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker network ls

NETWORK ID     NAME                       DRIVER    SCOPE
d95bb7471827   bridge                     bridge    local
2f660f266735   host                       host      local
36c3efe85b83   none                       null      local
d9eaf06a2693   reverseproxy-nw            bridge    local

docker network inspect d9eaf06a2693

[
    {
        "Name": "reverseproxy-nw",
        "Id": "d9eaf06a26936b155fac6568980d5f8fd685531bdc797911592e112a3b70a8a3",
        "Created": "2022-08-15T20:26:29.046077109+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
           }
        },
        "Options": {},
        "Labels": {}
    }
]
2

I edited your post to use code blocks. Please, format your post next time as described in the the following topic: How to format your forum posts

Containers should be able to communicate with the internet on bridge network. Each container has a network inside and the host is the gateway. Through that gateway the request goes to your LAN or WAN interface.

There was an issue before when someone noticed the routing was wrong and requests went to an other interface which was not connected to the internet. Do you have multiple networks on the same machine?

3

Many thanks for your answer and reformatting my text Akos!

No, I have only single network interface. I was initially thinking that this is problem with DNS resolution, but via daemon.json I’m inserting respective DNS servers. And even connection from the container directly to the IP is not working.

{
    "dns": ["8.8.4.4", "8.8.8.8"]
}

If my understanding is correct there seems to be problem in forwarding packets between gateway from bridge network (in my case 172.18.0.1) and the ethernet interface, is that correct?

4

It was only a guess. What is your host operating system? How did you install Docker?
Are you sure your environment has no conflicting LAN networks? For example WSL gets its IP address from the same IP range and I have seen some LAN networks with similar IP addresses.

5

It’s VPS so sitting directly on internet, therefore there is no IP collision possible.
The host operating system is Debian 11 and Docker is installed directly from docker.com repo.

# cat docker.list
deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian bullseye stable
6

Does the VPS has a firewall locally?

Your issue is actually related to something that I am working on for my next tutorial so I had to trace network packets and figure out why the internet connection didn’t work from containers. In my case it was because I played with network namespaces manually and tshark helped me to find the issue. You can also use tcpdump depending on which you prefer. Here is how I used tshark with very basic parameters, although it has more parameters for filtering packets.

List packets in the docker0 bridge

tshark -i docker0

You can also add multiple interfaces by using -i multiple times with different parameters or just trace packets on the WAN interface, but that will have much more traffic. You can try to send a ping request from the container so you can filter to lines that contain “ICMP”. I just used grep

tshark -i docker0 | grep 'ICMP'

If you can’t see ICMP packets on the bridge interface, it is possible that packets are not reaching that at all.

Each container connected to the bridge has an interface without IP address (veth*). This is where you should probably see something even if there is nothing on other interfaces.

In my case the problem was that I forgot to add the proper iptables rules to accept packets coming from docker0 to eth0 so I couldn’t see any ICMP reply on docker0 and no request on eth0. In case you see ICMP reply then the request goes out but the reply is blocked before it reaches the container that sent the request.

7

Hello Akos,
so small feedback. I was digging little bit more into this topic and it finished by downgrading docker-ce and docker-ce-cli packages to version 5:20.10.21 (from 5:20.10.23) and it started magically to work.
Not sure if there was something changed in this package, or it was just bug in 5:20.10.23, but it works.

Many thanks for your kind advices.