How to connect to reverse proxy directly when it's running in network_mode: service:wireguard

General
,
1

Hi,

docker noob here. I have a server running in my home network with wireguard as a client and nginx as a reverse proxy in network_mode: service:wireguard. This server is connected to vps with wireguard server.

When I’m connected to wireguard server I can connect to reverse proxy without any issue.

But when I try to connect to reverse proxy directly from local network, not via wireguard, connection times out.

This is my home server docker-compose.yml

version: "3.5"
services:
  wireguard:
    image: lscr.io/linuxserver/wireguard:latest
    container_name: wireguard
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    environment:
      ...
    volumes:
      - ./config:/config
      - /lib/modules:/lib/modules 
    ports:
      - 51820:51820/udp
      - 80:80
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
    restart: unless-stopped

  reverse-proxy:
    image: nginx
    container_name: reverse-proxy
    network_mode: service:wireguard
    volumes:
      - ./nginx-config/default.conf:/etc/nginx/conf.d/default.conf

networks:
  default:
    external: true
    name: 'my-network'

My guess is that I have to create some iptables rule because when i scan host with nmap I get

PORT     STATE    SERVICE
22/tcp   open     ssh
80/tcp   filtered http
...

Also there is a rule in iptables already but it isn’t enough:

iptables -S | grep 80
-A DOCKER -d 172.18.0.4/32 ! -i br-<id> -o br-<id> -p tcp -m tcp --dport 80 -j ACCEPT

172.18.0.4 is an ip addres of wireguard container in my-network

Can anyone enlighten me on docker networking.

2

Your wireguard container is the owner of the network namespace, as such it is correct that the wireguard container is the one that has to published ports. The nginx container uses the network namespace of the wireguard container, and acts network-wise as it would be a process that is running in the wireguard container…

If wireguard itself is not causing this, it should work without having to add any iptables rules.

You can easily test whether it’s docker or wireguard behavior, if you test it with an image that does not alter network state:

services:
  test:
    image: ubuntu
    tty: true
    ports:
      - 8080:80
  reverse-proxy:
    image: nginx
    network_mode: service:test
    volumes:
      - ./nginx-config/default.conf:/etc/nginx/conf.d/default.conf

networks:
  default:
    external: true
    name: 'my-network2'

You should be able to reach nginx from port 8080.

3

You are right it seem to be wireguard related issue. I can indeed access reverse proxy like this. I was able to find this post. But it didn’t help. It seems to be close. I need to somehow redirect traffic to 10.13.13.3 which is address of wireguard container in wireguard network (nothing to do with docker) and which I use to access reverse proxy via vps.

What I can do is somehow access reverse proxy from local network when address is resolved, but the proxy doesn’t redirect. Which is weird. (hostnames are resolved on router in local network)

curl -I http://jelly
HTTP/1.1 307 Temporary Redirect
Server: nginx/1.25.3
Date: Sun, 29 Oct 2023 16:41:17 GMT
Connection: keep-alive
X-Powered-By: Express
Location: /login

jelly resolves to 192.168.0.22

curl -I http://192.168.0.22
<timeout>

curl -I http://service1.jelly
<timeout>

curl -I http://service2.jelly
<timeout>
4

Are you sure the reverse proxy container is able to reach anything outside the wireguard network?

Have you tried asking in a wireguard forum? I assume it is more likely to find wireguard users in a wireguard forum that actually use wireguard in container.

5

I still don’t know what the issue is but using nginx-proxy-manager instead of vanilla nginx solves the issue. Thank you for help.

6

I found the cause while installing gitlab in a different environment.
The problem was that I installed a ubuntu distribution on wsl2 and reinstalled docker inside it.

I put gitlab on top of the docker I installed inside ubuntu and that’s why it was not visible.

I’m not sure, but the docker that is used immediately after connecting with wsl comes with a docker desktop installation, and this time I installed gitlab with docker from docker desktop and installed docker on ubuntu with the container running, and I couldn’t see the running container with docker ps -a like the previous problem.

I thought wsl2 provides an independent Linux kernel, but I don’t know why this docker is shared.

Translated with DeepL

7

If no OS is mentioned, we always assume that you run docker-ce on Linux baremetal or vm. WSL2 distributions are not vm’s, they are containers (but not docker containers) that run on a utility vm, which provides the kernel used by all distributions.

If you bind a port in a WSL2 distribution, it will automatically be reachable on the host’s localhost, but not on any other host ip. The port will not be accessible from lan, unless you create port forwarding for it.

Docker Desktop takes care of port forwarding, so that a published port can be reached on the host ip.

There is no such thing as a docker: it is a container. If you install Docker Desktop, you can access it from a WSL2 distribution by enabling the integration feature in the Docker Desktop settings. There is no need to additionally install docker-ce. Please check docker context ls to see which contexts are available and figure out in which context your containers are deployed.

Everything I wrote should have been findable by the forum search. So please use it if you encounter new challenges.

8

I have managed to find one of the issues. ping between containers was working but curl wasn’t because mtu was too small and wireguard was already using smaller mtu. But I haven’t managed to make reverse proxy accessible from local network. It is probably caused by wireguard creating extra interface.

From reverse proxy container which shares network stack with wireguard:

ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none 
21: eth0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:13:00:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0

My solution is to have two reverse proxies one for vpn and one for local network:

version: "3.5"
services:
  wireguard:
    image: lscr.io/linuxserver/wireguard:latest
    container_name: wireguard
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    environment:
    ...
    volumes:
      - ./config:/config
      - /lib/modules:/lib/modules 
    ports:
     #no need to expose port 80 to the host, it is occupied by other proxy anyway
      - 51820:51820/udp
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
    restart: unless-stopped

  reverse-proxy-vpn:
    image: nginx
    container_name: reverse-proxy-vpn
    network_mode: service:wireguard
    volumes:
      - ./nginx-config/default.conf:/etc/nginx/conf.d/default.conf
  
  reverse-proxy-local:
    image: nginx
    container_name: reverse-proxy-local
    ports:
      - 80:80
    volumes:
      - ./nginx-config/default.conf:/etc/nginx/conf.d/default.conf

networks:
  default:
    external: true
    name: 'my-network'