Routing network traffic from one service to another

Hi,

I’m this situation where I am using “docker stack deploy” to create my services using yaml files and noticed that things like network_mode that I once used to link one service to another to use the same network no longer works.

The purpose of this is I am creating an OPENVPN service and would like another service to go through that OpenVPN network and whatever I tried fails. I tested this by going in the OpenVPN container which returns me an IP that is different from my ISP’s while the other container returns the IP attributed by my ISP, hence it doesn’t go through the OpenVPN’s service network.

As an example, here is the OPENVPN service compose file :

---
version: "3.8"
services:
  service2:
    deploy:
      restart_policy:
        condition: on-failure
        max_attempts: 5
    image: someimage:latest
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/New York
    networks:
      - vpn-net
    volumes:
      - something-config:/config
    ports:
      - 8080:8080

volumes:
  something-config:
    external: true

networks:
  vpn-net:
    external: true

In the VPN service I have this in the docker compose file :

---
version: '3.3'
services:
  openvpn-client:
    deploy:
      restart_policy:
        condition: any
        max_attempts: 15
    image: haugene/transmission-openvpn:latest
    cap_add:
      - NET_ADMIN
    dns:
      - 8.8.8.8
      - 8.8.4.4
    networks:
      - vpn-net
    volumes:
      - openvpn_client-config:/data
    environment:
      - PGID=1000
      - PUID=1000
      - CREATE_TUN_DEVICE=true
      - OPENVPN_PROVIDER=PIA
      - OPENVPN_CONFIG=
      - OPENVPN_USERNAME=
      - OPENVPN_PASSWORD=
      - LOCAL_NETWORK=192.168.20.0/23
      - OPENVPN_OPTS=--inactive 3600 --ping 10 --ping-exit 60
    ports:
      - 9000:9000

networks:
  vpn-net:
    driver: overlay
    attachable: true

By doing this, each container get its own IP address, there isn’t any routing done through the openvpn network. Any idea as to what I am doing wrong?

Thanks

If you wan’t to the network traffic from one container to go through an other container’s network then using the same docker network won’t work. This is similar to a LAN network where you have a machine connecting to a VPN while the other machines on the same LAN won’t use that VPN. In that situation you would have to set up a VPN on your router. You probably know that already you need to set the openvpn client as a gateway for other containers. Normally the gateway is the IP address of the host on that docker network. This is why host network would work if you were connecting to the VPN from the host. I can’t tell you the proper way for sure to use a container as a gateway but you can actually run swarm services on host network.

You could find some solutions like this (not tested) Docker
or run a proxy server on the host or inside a container which connects to the vpn.

These are just quick ideas but there could be a much better way too.

What you explained make sense to me. What you are explaining to me is what I didn’t expect I was doing, I guess I don’t understand docker networking well enough. Basically, the container I called OPENVPN is running VPN and works great… I’m trying to get other containers created through another deploy to connect to the OPENVPN container that was deployed in another service. I’m not sure if it is possible to do that. I’m looked at the documentation, all I know is the old method which only works with “docker-compose up” through the network_mode where you tell a service to use the other service’s network, everything is routed through there hence you go through the network that uses a VPN.

So what I do have in my yaml files is probably showing you a picture that is different from what I am trying to do, hence my lack of knowledge. There is probably a way to do this with docker stack deploy (swarm) but I haven’t found it yet.

I looked into the HOST deployment and that will not work for me. With my scenario, I would have to deploy VPN on the entire VM and everything that I deploy on that node would have to use it.

It is too bad network_mode is not supported for swarms. For now I have no viable solution, at least none that is ideal.

Actualy it is possible to hook a service/container into the network namespace of another service or container.

The consuming service needs to use network_mode: service:${name of the vpn service in the same compose file}. Ports need to be published on the vpn service, not the consuming service. Since the consuming service now is hooked into the vpn services namespace, the consuming service can not have idividual additional networks. You need to make sure to not get port conflicts for the running applications inside the containers that use same network namespace (like it is within k8s pods).

I have no idea what specialities haugene/transmission-openvpn:latest provides. Though even if it provides a http_proxy ready to be used by other services, it is a tradeoff when it commes to security, but might allow more flexibility.

---
version: "3.8"
services:
  openvpn-client:
    deploy:
      restart_policy:
        condition: any
        max_attempts: 15
    image: haugene/transmission-openvpn:latest
    cap_add:
      - NET_ADMIN
    dns:
      - 8.8.8.8
      - 8.8.4.4
    volumes:
      - openvpn_client-config:/data
    environment:
      - PGID=1000
      - PUID=1000
      - CREATE_TUN_DEVICE=true
      - OPENVPN_PROVIDER=PIA
      - OPENVPN_CONFIG=
      - OPENVPN_USERNAME=
      - OPENVPN_PASSWORD=
      - LOCAL_NETWORK=192.168.20.0/23
      - OPENVPN_OPTS=--inactive 3600 --ping 10 --ping-exit 60
    ports:
      - 9000:9000
      - 8080:8080
  service2:
    deploy:
      restart_policy:
        condition: on-failure
        max_attempts: 5
    network_mode: service:openvpn-client
    image: someimage:latest
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/New York
    volumes:
      - something-config:/config

volumes:
  something-config:
    external: true

Though, I must admit: this solution has nothing todo with routing. It is just the standard way to share the connection of a vpn container with other services in the same compose file.

I knew you could use it with Docker Compose but I wasn’t sure you could do that with swarm if the host network doesn’t work with the docker stack deploy command. The other reason I did not mention it is the port collision. It would be hard to use it with many service but it is good you did mention it since it can be helpful for applications which don’t require listening on any port.

I am afraid, you are right. I ignored this is about swarm. I am generaly not a big fan of those cotainerized vpn setups - nothing you would ever encounter in enterprises.

At least swarm provides a way to use the host network, without having the nasty side effect network_mode: host would have.

networks:
  hostnet:
    external: true
    name: host

source: https://docs.docker.com/compose/compose-file/compose-file-v3/#host-or-none

1 Like

I didn’t know that. That could solve the problem @tessierp and could also provide a new solution for @mrmr2021

First, let me thank you for taking the time to reply and share solutions.

Probably they wouldn’t containerize a VPN but it is still a nice option if you consider that you may host other services on a node you may not want to go through a VPN. For me it all comes down to flexibility. We lost that flexibility with a swam and it is not like I want to go back and use “docker-composer up” neither.

I still think this is a functionality they should think of implementing. It was possible with docker-composer why not with “docker stack deploy”?

For me it is still not a viable solution to go have VPN on the host. I understand it is a solution but with what I am going or want to do it would be a patch which would mean I would use an entire node just for two services which I absolutely want to go through VPN.

Sorry, it looks like I have forgotten how you started the topic. This is what happens when I try to jump from topic to topic :slight_smile:

I think the reason is why network_mode is not supported on swarm that docker-compose run containers on one machine so you can share network namespaces. Docker swarm was designed to run containers on multiple machines which means running containers on the same machine is not guaranteed without additional rules. Yes, network_mode could have been one of those rules. Kubernetes support pods to run multiple containers in the same network namespace, so Docker Swarm could have done the same. I found a project to implement pods in Docker Swarm but the development stopped 3 years ago.

Akos nailed it. Sharing network namespaces can not working amongst process on different nodes.

Up to this day swarm services do not have feature parity with plain docker containers - and they never will.
Having potential replicas that might end up deployed on different nodes is a huge game changer.

Though, you might want to take a closer look if there isn’t a vpn image with build-in http_proxy (or even build your own?). With such an image, you could set the http_proxy as environment_variable and make containers use the vpn endpoint like that. Of course this solution is not as clean as using the vpn container as gateway, but it’s better then no solution at all.

I was afraid it would come to this… I guess my only solution is to move back to containers only If I really want this, at least for the time being unless they do implement a way to do something like this in the future…

Thanks for the help!