How to let traefik reverse-proxy services that are outside of docker that running traefik

Hi,I am new to docker and I find docker is easy to deploy services and I like it.I have use traefik to reverse-proxy containers on the same docker,and I can visit their webpages with domain.But I can’t apply the same thing to services that are outside of docker.
I have done some research,the answer may be related to docker’s nat.So I created a new VM to test it.The new VM only installed docker,and has only one container which is nginx.Let’s call the new VM’s nginx n2,because the old VM I used has a nginx too,let’s call that n1.I will use n1 as a reverse proxy later with traefik disabled.
Before test,I will tell you what I have done yet.I run VMware on a windows machine which IP address is 192.168.0.197,and use it to create 2 VMs.And there is a service running on windows has a webpage.Frist VM,VM1,is using Ubuntu server 20.04 as OS,with webmin and docker 20.10.17 installed,with 3 containers(portainer,traefik,nginx),IP 192.168.0.130.Second VM,VM2,also Ubuntu server 20.04,docker 20.10.17,nginx,IP 192.168.0.120.VM1 have are not enabled promiscuous mode yet.VM2 will not enable promiscuous mode.
All file related are show below:
traefik.yml

log:
  level: DEBUG
  filePath: /etc/traefik/logs/traefik.log
accessLog:
  filePath: /etc/traefik/logs/traefik-access.log
api:
  dashboard: true
  debug: true
entryPoints:
  web:
    address: :80
providers:
  docker:
    exposedByDefault: false
    watch: true
  file:
    filename: /config.yml
    directory: /etc/traefik
    watch: true

docker-compose.yml for traefik

version: "3"
 services:
   traefik:
     image: traefik:latest
     container_name: traefik
     restart: unless-stopped
     networks: 
       - "traefik_test"
     ports:
       - 80:80
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock
       - /etc/localtime:/etc/localtime:ro
       - /home/me/docker/traefik:/etc/traefik
       - /home/me/docker/traefik/config.yml:/config.yml
       - /home/me/docker/traefik/logs:/etc/traefik/logs
 networks:
   traefik_test:
     external: true

docker- compose.yml for n1

version: "3"
 services:
   nginx:
     image: nginx:latest
     container_name: nginx
     restart: unless-stopped
     networks: 
       - "traefik_test"
     ports:
       - 180:80
     volumes:
       - /home/me/docker/nginx/conf.d:/etc/nginx/conf.d

docker- compose.yml for n2

version: "3"
 services:
   nginx:
     image: nginx:latest
     container_name: nginx
     restart: unless-stopped
     ports:
       - 180:80

config.yml

http:
  routers:
    traefik:
      entryPoints:
        - web
      rule: "Host(`traefik.test.lan`)"
      service: api@internal
    n1:
      entryPoints:
        - web
      rule: "Host(`n1.test.lan`)"
      service: n1
    n2:
      entryPoints:
        - web
      rule: "Host(`n2.test.lan`)"
      service: n2
    app:
      entryPoints:
        - web
      rule: "Host(`app.test.lan`)"
      service: app
  services:
    n1:
      loadBalancer:
        servers:
          - url: "http://192.168.0.130:180"
    n2:
      loadBalancer:
        servers:
          - url: "http://192.168.0.120:180"
    app:
      loadBalancer:
        servers:
          - url: "http://192.168.0.197:1235"

nginx.conf(I actually use one at a time)

server {
  listen 80;

  server_name 192.168.0.130:9000;

  location / {
      proxy_pass http://portainer.test.lan/;
  }
}
server {
  listen 80;

  server_name 192.168.0.120:180;

  location / {
      proxy_pass http://n2.test.lan/;
  }
}
server {
  listen 80;

  server_name 192.168.0.197:1235;

  location / {
      proxy_pass http://app.test.lan/;
  }
}

This the line I created macvlan,Don’t worry they will not confilct with my other devices.

docker network create -d macvlan --subnet=192.168.0.0/24 --gateway=192.168.0.1 -o parent=ens32 macvlan

So let’s start the test.N1 was not configed yet.I first used traefik as reverse proxy provider,opened traefik dashboard and saw services running,then visiting webpages one by one,then changed network type one by one.Then I shuted down traefik,changed configraton in nginx.conf one by one.And changed n1’s port to 80.And repeat.Then enabled promiscuous mode on VM1 and repeat.Then deleted nginx.conf and changed n1’s port to 180 and started traefik.And repeat.
Test result:
test
N2 is showing 404.App is showing 504.
From start to end,traefik cannot reverse-proxy services outside of docker,not even by traefik’s ip.Last line is the most interested.With promiscuous mode enabled,and macvlan mode,although I can’t visit n2 and app by domain,but I can visit them by nginx’s ip.So it means container using macvlan with promiscuous mode enabled can communicate with services outside of docker.But why not by domain?
Messing up with promiscuous mode eventually caused VM1’s docker broke down.I can’t visit containers’ webpage on VM1 again.
So can anyone teach me how to let traefik reverse-proxy services outside of docker.

This is a rather complex setup, and without testing it probably will always remain guessing, but we can still try to untangle it.

Your config.yml looks fine, except that’s I am unsure if entryPoints support the sequence style you used and the key/value style. The documentation uses the later. Until the setup works as you intended, I would suggest to comment or remove the entrypoints from the config. Once overything is running, you can still re-add them.

The example you shared is for running the vm’s network interfaces in bridge mode (=getting a local lan ip) and running the containers on each docker engine in bridge mode as well. You should be able to reach each of those urls from a browser on your windows system - does it work for all of them? If this is not working then you already have a another problem to solve first.

Regardings your macvlan: I would higly recommend to use the --ip-range option and provide a range that is not handled by your networks dhcp server. Also: a mac vlan parent interface is not able to directly communicate with any of its child interfaces or vise versa (this is not a limitation introduced by docker, but there is a way to work arround that limitation). Afaik, If traefik is not using a macvlan ip itself, it will not be able to communicate with the other macvlan ips on the same node. Your windows host on the other side should be able to communicate with every macvlan ip - of course that is if promiscous mode is enabled for the interface.
Again: test if you can reach all target ip:ports from your Windows host to make sure the services themself work as they should.

One more thing: is you local dns able to resolve the domains you use?

Thank you for reply.

I would suggest to comment or remove the entrypoints from the config.

I have tried change web to “web” or comment the entrypoints from the config,and still the same,I can’t even see a change in traefik dashboard.

The example you shared is for running the vm’s network interfaces in bridge mode (=getting a local lan ip) and running the containers on each docker engine in bridge mode as well. You should be able to reach each of those urls from a browser on your windows system - does it work for all of them? If this is not working then you already have a another problem to solve first.

Yes the VMs’ network interfaces are in bridge mode.N2 was using bridge mode too.Treafik and n1 used bridge,host and macvlan mode one by one.App was on windows.Webmin was running on VM1.I can visit them with ip:port,except traefik,since it didn’t expose port.But I can visit traefik with domain.Those services were working fine.

a mac vlan parent interface is not able to directly communicate with any of its child interfaces or vise versa

Yes,I tested it today,and it returned 404.I must have joined traefik and nginx to traefik_test network thatday.

Afaik, If traefik is not using a macvlan ip itself, it will not be able to communicate with the other macvlan ips on the same node.

I’ve done a new test today.I only used VM1 this time.VM1’s promiscous mode was enabled.Traefik and jellyfin were both in macvlan mode.And I cannot use traefik to reverse proxy jellyfin.But traefik can ping jellyfin.

Your windows host on the other side should be able to communicate with every macvlan ip - of course that is if promiscous mode is enabled for the interface.

My windows can communicate with traefik in macvlan mode with or without promiscous mode enabled.

Again: test if you can reach all target ip:ports from your Windows host to make sure the services themself work as they should.

Yes, I can reach all target ip:ports.

One more thing: is you local dns able to resolve the domains you use?

I use pihole to resolve local dns.

There is one thing that realy is confusing in the example from above.

  • The routers have a rule per router to forward traffic to a service loadbalancer based by a full qualliifed domain name. The loadbalancer targets are called by ip.

  • The nginx config, has different server blocks which listen on ip and port, but use a proxy_pass directive, which uses the same full qualified domain name, the router rule in treafik used…

Something is not adding up here…

To which ip do the dns overrides resolve all the domains? If you want to use traefik, they must resolve to the ip of traefik.

Your nginx.conf looks weird: the server_name should be a fqdn or ip, but not fqdn:port or ip:port. The port is already handled by the listen directive. You must be running three different instances, which each need one of the server blocks you declared. Also the node where nginx is running must have the pihole as first nameserver in /etc/resolv.conf in order to leverage the overrides… and even if it doesn’t… don’t you create a recursive loop, as you point them to traefik again?

Oh,I just find out I put IP:ports in location and domain in server name.Sorry about that,I can’t find the edit button there,so I can’t edit the config to right.I pasted the wrong config instead.But don’t worry,I used the right configs when testing frist.And I changed to the wrong configs to test again when I could visit services by nginx’s ip and nothing crushed.Because I’m not familiar with nginx,I ran it 12 times,each time there is only one service in nginx.conf,I didn’t put 3 services all in.After nginx’s tests were over,I deleted nginx.conf,so they will not confilct with traefik.
And I don’t want to use nginx.I just learn it last minute trying to prove traefik is not alone useless when trying to reverse proxy services outside of docker.I just want to use traefik.So let’s ignore nginx.Just focus on traefik.I’m positive about my traefik’s config.They work flawlessly in bridge mode.
The name server is always pihole.As far as I know,neither traefik nor nginx is name server.

not sure what this is supposed to mean, but the underlying nodes need to be configured to have pihole as first nameserver configured if they (and containers running on them) should be able to resolve the domain names.

The big picture still feels ambigous, and always in flux for me, I won’t be able to help.

You might want to post your current state with all details relevant to the setup with actual configurations… Sometimes people tend to create abstract examples, where they leave out parts they think are irrelevant, but in fact cause the problem… I have seen this a lot in the past…

That line means I thought windows has pihole as name server,so it knows services’ ip,but doesn’t know the port,so reverse proxy sould tell the port,but traefik has it own ip,so I change windows’ name server to traefik,and everything become 404.
Yes that must be the reason.When traefik in brige mode,it has the same ip address with pihole.Windows can talk to pihole frist then traefik,so it gets the services’ ip and port.But when traefik in macvlan mode,isolated from docker,so windows will only get services’ ip.I will do some diging.Thanks for the enlightenment.
But in case I’m wrong,I wiil paste my full traefik configs.
traefik.yml

global:
  checkNewVersion: true
  sendAnonymousUsage: false
log:
  level: DEBUG
  filePath: /etc/traefik/logs/traefik.log
accessLog:
  filePath: /etc/traefik/logs/traefik-access.log
api:
  dashboard: true
  debug: true
serversTransport:
  insecureSkipVerify: true
entryPoints:
  web:
    address: :80
  websecure:
    address: :443
certificatesResolvers:
   staging:
     acme:
       email: myemail
       storage: /etc/traefik/certs/acme.json
       caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
       dnsChallenge:
         provider: duckdns
   production:
     acme:
       email: myemail
       storage: /etc/traefik/certs/acme.json
       caServer: "https://acme-v02.api.letsencrypt.org/directory"
       dnsChallenge:
        provider: duckdns
providers:
  docker:
    exposedByDefault: false
    network: traefik_proxy
    watch: true
  file:
    filename: /config.yml
    directory: /etc/traefik
    watch: true
  

docker-compose.yml for traefik

version: "3"
 services:
   traefik:
     image: traefik:latest
     container_name: traefik
     restart: unless-stopped
     networks: 
       - "traefik_proxy"
     ports:
       - 80:80
       - 443:443
     environment:
     - 'DUCKDNS_TOKEN=token'
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock
       - /etc/localtime:/etc/localtime:ro
       - /home/me/docker/traefik:/etc/traefik
       - /home/me/docker/traefik/config.yml:/config.yml
       - /home/me/docker/traefik/logs:/etc/traefik/logs
 networks:
   traefik_proxy:
     external: true

config.yml

http:
  routers:
    traefik:
      entryPoints:
        - "web"
      rule: "Host(`subdomain1.duckdns.org`)"
      middlewares:
        - https-redirectscheme
      service: api@internal
    traefik-secure:
      entryPoints:
        - "websecure"
      rule: "Host(`subdomain1.duckdns.org`)"
      tls: 
        certResolver: production
      service: api@internal
    n1:
      entryPoints:
        - web
      rule: "Host(`n1.test.lan`)"
      service: n1
    n2:
      entryPoints:
        - web
      rule: "Host(`n2.test.lan`)"
      service: n2
    app:
      entryPoints:
        - web
      rule: "Host(`app.test.lan`)"
      service: app
  services:
    n1:
      loadBalancer:
        servers:
          - url: "http://192.168.0.130:180"
    n2:
      loadBalancer:
        servers:
          - url: "http://192.168.0.120:180"
    app:
      loadBalancer:
        servers:
          - url: "http://192.168.0.197:1235"
  middlewares:
    https-redirectscheme:
      redirectScheme:
        scheme: https
        permanent: true

I am not entirely sure what you try to do there, but I wrote a couple of times that the first nameserver of the hosts (in your case the vm’s) need to have the nameserver entry for pihole as first entry in /etc/resolv.conf. Why would they (=vm’s) care what nameserver is configured in your windows host?!

update: This is only relevant if containers or the nodes should be able to resolve the domain names. For traefik itself, it doesn’t matter, as it reads the information from the http header (same is true for https, when tls-offloading is used).

Do you realy just want your traefik api to be reachable via https`? and the rest only via http?

Never the less, I feel there is not much I can do for you. I hope someone else is able to pitch in.

Ah, it turns out to be my misunderstanding about traefik’s enterpoint.I want to dig a hole and bury myself in. Everytime I visit the domain that traefik reverse-proxyed, in fact, I’m visiting traefik’s IP:port, then traefik pretending it’s the service original IP: port.All I need is binding services’ domains to traefik’s IP in pihole. Instead of binding to original services’ IP.Oh, I’m so silly. Can administrator delete this topic.
And I have not learnt nginx before.I only watch a short video, and copy paste and modified the config.
Sorry for wasting your time.

This actualy is how reverse proxies in general work. It is not a special behavior or requirement of traefik and the reason why I mentioned that all dns overrides (in pihole) must resolve to the traefik ip. like all reverse proxies, it uses either the http header for http/https or leverages SNI for tls passthrough traffic to determin the matching host rule.

I am not sure if an admin can delete the thread.

Don’t worry. Things happen. Enlightenment doesn’t care when or how it’s archived :wink:

1 Like

I could. But in general I don’t, at least not when the OP asks for that: the (your) responses may be valuable for future readers. I don’t know if that applies to this topic too, but surely you’ve spent time on answering and in general I don’t simply trash that.

1 Like