Docker containers behind Wireguard fail to connect to random servers (works for others)

Hi,

Just started recently to work with Docker and I struggle to fix one issue. I have provided the details below. I appreciate any help or guidance on how to make this work :slight_smile:

Had to add spaces to all URLs, otherwise the forum does not allow me to post

My setup

Servers

  1. Wireguard-Server
    root server (cloud VM with public IPv4 and IPv6)
    Configured as Wireguard server
    Also configured as web server for mydomain.tld

  2. Docker-Host
    Wireguard client connected to the Wireguard-Server via wg-quick (outside of docker)
    Connected via a home router with dynamic IP (Wireguard-Server – Home-Router – Docker-Host)
    Raspberry Pi 4 running Fedora Server 37 latest and Docker CE

What works

  • I can run Nextcloud on the Docker-Host using Docker
  • I can point a reverse proxy from Wireguard-Server to Docker-Host and access its Nextcloud instance via a subdomain (nextcloud.mydomain.tld)
  • I can download updates from ports .ubuntu .com from inside a docker container

What does not yet work

  • I cannot download updates from deb .debian .org
  • I cannot download Nextcloud apps from githubusercontent .com (timeout after 120 ms with 0 bytes downloaded)

I have tried around and noticed that the issue is gone when I use “–network host” instead of my normal bridge network
That’s why I believe that this is related to the iptables config and I need to add some rules to allow Debian downloads
I have tried myself, but did not get it to work

So my questions are :

  • Does anyone know why the connections to only those specific servers fail? Or why the others work fine? What is the difference between deb .debian .org and ports .ubuntu .com?
  • Can anyone guide me how to resolve this (without using host network)?

Detailed steps

# Create bridge network
docker network create --driver bridge alpine

# Connecting to ports .ubuntu .com WORKS
docker run --rm -it --network alpine --name alpine1 alpine /usr/bin/wget --timeout=5 httx://ports .ubuntu .com/ubuntu-ports
docker run --rm -it --dns 8.8.8.8 --network alpine --name alpine2 alpine /usr/bin/wget --timeout=5 httx://ports .ubuntu .com/ubuntu-ports
docker run --rm -it --network host --name alpine3 alpine /usr/bin/wget --timeout=5 httx://ports .ubuntu .com/ubuntu-ports

# Connecting to deb .debian .org FAILS with timeout
docker run --rm -it --network alpine --name alpine1 alpine /usr/bin/wget --timeout=5 httx://deb .debian .org/debian/dists/bullseye/InRelease

Changing to Google DNS does not make a difference

docker run --rm -it --dns 8.8.8.8 --network alpine --name alpine2 alpine /usr/bin/wget --timeout=5 httx://deb .debian .org/debian/dists/bullseye/InRelease

It only works with host network

docker run --rm -it --network host --name alpine3 alpine /usr/bin/wget --timeout=5 httx://deb .debian .org/debian/dists/bullseye/InRelease

# traceroute from docker container is successful and goes via Docker-Host and Wireguard-Server
[docker@Docker-Host ~]$ docker run --rm -it --network alpine --name alpine1 alpine /usr/bin/traceroute google .com
traceroute to google .com
1 Docker-Host.local (172.23.0.1) 0.053 ms 0.065 ms 0.055 ms
2 [WG IP of Wireguard-Server] (10.19.99.0) 9.129 ms 8.819 ms 8.736 ms

# curl --verbose from Docker container FAILS
docker run --rm -it --network alpine --name ubuntu1 ubuntu bash
[install curl]
root@bbdbefe7aebd:/# curl --verbose httx://deb .debian .org/debian/dists/bullseye/InRelease

  • Trying 146.75.122.132:80…
  • Connected to deb .debian .org (146.75.122.132) port 80 (#0)

GET /debian/dists/bullseye/InRelease httx/1.1
Host: deb .debian .org
User-Agent: curl/7.81.0
Accept: /

# also FAILS for httxs
root@0ee93537d57e:/# curl --verbose httxs://deb .debian .org/debian/dists/bullseye/InRelease

  • Trying 146.75.122.132:443…
  • Connected to deb .debian .org (146.75.122.132) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering httx/1.1
  • CAfile: /etc/ssl/certs/ca-certificates.crt
  • CApath: /etc/ssl/certs
  • TLSv1.0 (OUT), TLS header, Certificate Status (22):
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):

*# Wireguard config on Docker-Host
[root@Docker-Host ~]# wg
interface: wg0
public key: WHATEVER
private key: (hidden)
listening port: 34518
fwmark: 0xca6c
peer: WHATEVER
preshared key: (hidden)
endpoint: [IP OF WIREGUARD-SERVER]:123
allowed ips: 0.0.0.0/0, ::/0
latest handshake: 22 seconds ago
transfer: 2.22 GiB received, 1.10 GiB sent
persistent keepalive: every 25 seconds
[root@Docker-Host ~]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default _gateway 0.0.0.0 UG 100 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.23.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-65fa97586f90
192.168.178.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0

# iptables config

Generated by iptables-save v1.8.8 (nf_tables) on Sun Apr 16 13:00:04 2023

*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-65fa97586f90 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-65fa97586f90 -j DOCKER
-A FORWARD -i br-65fa97586f90 ! -o br-65fa97586f90 -j ACCEPT
-A FORWARD -i br-65fa97586f90 -o br-65fa97586f90 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-65fa97586f90 ! -o br-65fa97586f90 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-65fa97586f90 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT

Completed on Sun Apr 16 13:00:04 2023

Generated by iptables-save v1.8.8 (nf_tables) on Sun Apr 16 13:00:04 2023

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.23.0.0/16 ! -o br-65fa97586f90 -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i br-65fa97586f90 -j RETURN
-A DOCKER -i docker0 -j RETURN
COMMIT

Completed on Sun Apr 16 13:00:04 2023

Docker-Host version
dnf list docker*
Last metadata expiration check: 2:03:58 ago on Sun 16 Apr 2023 11:27:15 AM CEST.
Installed Packages
docker-buildx-plugin.aarch64 0.10.4-1.fc37 @docker-ce-stable
docker-ce.aarch64 3:23.0.3-1.fc37 @docker-ce-stable
docker-ce-cli.aarch64 1:23.0.3-1.fc37 @docker-ce-stable
docker-ce-rootless-extras.aarch64 23.0.3-1.fc37 @docker-ce-stable
docker-compose.noarch 1.29.2-6.fc37 @fedora
docker-compose-plugin.aarch64 2.17.2-1.fc37 @docker-ce-stable

I managed to fix it myself, but in case this helps anyone :

Wireguard has a MTU (maximum transmission unit / IP packet size) of 1420 by default
Docker bridge networks have a MTU of 1500
So the docker packets are too large to fit into the wireguard network and are discarded

Check if you are affected using command:
ip link | grep mtu
wg0 should have mtu=1420
Docker bridge networks probably have mtu=1500

To fix it :
Create the networks that need to establish outbound connections using MTU of 1420
(use ip link to confirm the MTU has changed)

docker
docker network create --opt com.docker.network.driver.mtu=1420 --driver bridge alpine

docker-compose
networks:
mynetwork:
driver: bridge
driver_opts:
com.docker.network.driver.mtu: 1420 # packet size must not exceed wg0 MTU

Credits go to