Reverse proxy connection to apache2 in container "bad request"

I am running two containers, one uses image dtjs48jkt/webtrees:v1.7.10 and the other image traefik:v2-5. The webtrees container is an apache2 app, and the traefik container is a reverse proxy.

They are in different directories, both have their own separate docker-compose.yml files.

The webtrees image accepts connections on port 443.
I configured traefik with a single entrypoint for port 443, hence no traffic in the traefik container except on port 443.

If I map ports 443:443 in the webtrees yml, docker-compose up fails because 443 is already mapped to traefik.
So I map ports 443 (just the container port), the webtrees container runs successfully and traefik sees it and creates the rule and service. Docker is mapping the container port 443 to a random high-numbered port on the host.

Connecting to webtrees via the internet produces apache2’s “Bad Request” error: “Your browser sent a request that this server could not understand. Reason: You’re speaking plain HTML to an SSL-enabled server port. Instead use the HTTPS scheme to access this URL, please.” – Apache/2.4.29 (Ubuntu) Server at webtrees Port 443

I’m using https: for the url, and traefik only handles network traffic on port 443. So there’s a mixup at the docker container for webtrees, where I have Ports 443 in the yml. It’s seeing the connect request on the host port and not the container port, because it’s sending out the “Bad Request” reply.

I can’t map actual host port 443 on both containers because docker won’t allow that. So how should I configure these two containers, so that incoming SSL traffic is handled by traefik, routed to webtrees, and webtrees receives it on port 443?

Hi,

Without seeing your configuration, my guess is that traefik is offloading SSL, meaning that the traffic comming from traefik to your container, is no longer encrypted, and then you try and connect the unencrypted data, to a encrypted endpoint.

In the webtrees docker hub page ( Docker Hub ) there is a point where you can start it HTTP only, can you try that?

Using http only
It is possible to use the image without https support. For that you have to start the container with the following environament variables set.

docker run -d -p 80:80 --name webtrees --link mysql:db -v /webtrees/data:/var/www/html/data -e DISABLE_SSL=TRUE -e PORT=80 --restart always dtjs48jkt/webtrees

Hi Martin,

I want to use SSL, HTTP is insecure. I know the webtrees container works, I can connect to it just fine without the reverse proxy.

Here’s my traefik config files:

# traefik.toml static config file created 23-jan-2022 by fgb
# For use with docker services on aws2020
# Mod 26-jan-2022, fgb: use only websecure entrypoint (443)

[entryPoints]
  [entryPoints.websecure]
    address = ":443"
    [entryPoints.websecure.http]
      [entryPoints.websecure.http.tls]
        certResolver = "lets-encrypt"
[api]
  dashboard = true
#  dynamic api config (for dashboard) in traefik's yaml file

[providers.docker]
  endpoint = "unix:///var/run/docker.sock"
  watch = true
  network = "dweb"

[certificatesResolvers.lets-encrypt.acme]
  email = "redacted@protonmail.ch"
  storage = "acme.json"
  [certificatesResolvers.lets-encrypt.acme.tlsChallenge]

[log]
  level = "ERROR"
  filePath = "/logs/traefik.log"

[accessLog]
  filePath = "/logs/traefik-access.log"

And here’s docker-compose.yml:

# docker-compose.yml for traefik v2.x reverse proxy container
# created 23-jan-2022 by fgb
# Mod 26-jan-2022, fgb: use port 443 only.
# NB: Host rule uses backticks to quote!  So nonintuitive!

version: '3'

services:
  traefik:
    image: traefik:v2.5
    restart: always
  # Here's the docker network we created: 
    networks:
      - dweb
    ports:
      - 443:443
    labels:
      - traefik.enable=true
      - traefik.docker.network=dweb
      - traefik.http.routers.api.rule=Host(`traefik.mrmr.pro`)
      - traefik.http.routers.api.service=api@internal
      - traefik.http.routers.api.middlewares=auth
      - traefik.http.routers.api.tls=true
      - traefik.http.routers.api.tls.certresolver=lets-encrypt
      - traefik.http.middlewares.auth.basicauth.users=redacted:$$apr1$$6sNEg4kB4$KnvnHdlDeUK.wIXw/
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /opt/traefik/traefik.toml:/traefik.toml
      - /opt/traefik/acme.json:/acme.json
      - /opt/traefik/logs:/logs
    container_name: traefik

networks:
  dweb:
    external: true

Thanks for your attention,
Frank

It was between your traefik and your webtrees i ment not to use ssl, for testing.

But im not an expert on traefik, on how you configure it to use secure internally.

I think its better to post in https://community.traefik.io/

Sure. I’m pretty sure that traefik and webtrees both are working fine. The issue is getting them both access to port 443 in separate docker containers. Hence, it seems to me to be a docker issue.

Sorry, but i still belive this is a misconfiguration of traefik, but we can wait and see if anyone else knows whats going on here.

Could you confirm it to me that you want to access the webtrees container through traefik and not the random port which is forwarded from the host to the webtrees container directly? This is why I am confused:

but I don’t see which port you are trying to use for the connection “via the internet”.

You probably meant HTTP not HTML

But you don’t do that because that is impossible and not because of Docker… Only one process can listen on one port on a specific IP address. This process is traefik in your case. Traefik has to forward the request to the backend container (webtrees).

Doesn’t matter which port traefik is listening on. This is just a port as any other. It is usually used for HTTPS protocol but any process could listen on port 443 and still expect HTTP traffic and not HTTPS.

Even if traefik forwards the request to the same port on the webtrees container, it can still forward the plain HTTP version. Traefik Routers Documentation - Traefik

When a TLS section is specified, it instructs Traefik that the current router is dedicated to TLS requests only (and that the router should ignore non-TLS requests).

By default, a router with a TLS section will terminate the TLS connections, meaning that it will send decrypted data to the services.

Do you really want to secure your internal traffic? Yes, HTTP is insecure when there is a chance that people will can catch that traffic beteen your machine and the server or between two servers. SSL termination is a common way to let a proxy to handle SSL so you can configure your certs in one place instead of configuring it for each container including the proxy

I don’t use traefik either so I can’t tell you the exact problem, although I tried to configure SSL with traefik now and I could not properly configure it, but I think @terpz is right.

Can you show how you started the webtrees container?

Thanks for your reply Ákos. Yes I want to access the webtrees container through traefik, because traefik handles the cert via lets-encrypt. You are right, I meant I want to use HTTPS to accept internet connections. It makes sense that only one process can use one port, however how do browsers allow multiple tab sessions with multiple HTTPS connections? The IP address doesn’t change for the computer hosting the browser as far as I know. I can even run multiple browsers at the same time connecting to different HTTPS sites…

Anyway your comments make sense, I am trying to achieve SSL termination. Thinking on your advice I went ahead and tried setting the traefik container port to 443:8081 (to catch incoming 443 traffic) and the webtrees container port to 8081:443, to get traffic forwarded internally from traefik. However I still end up with webtrees’s apache displaying the “Bad request” error.

Maybe you and @terpz are right and I should move this question to the traefik forum.
Screen Shot 2022-01-28 at 7.54.50 PM

Frank

Again, port doesn’t matter, protocol does. Docker will not magically convert your request from HTTP to HTTPS or HTTPS to HTTP. Traefik has to get the request on HTTPS and (optionally) decrypt it and forward to a backend container. You have to figure out the right configuration for traefik to do that regardless of what the ports are.

The port on your machine during the communication will not be 443. It is the port of the remote server where you want to send the request. Your machine will listen on a random port for the response. You can check it with netstat -nat. Currently on my machine I have multiple ports open between 53300 and 53360. These ports are in the “Local Address” column while all of the ports in the “Foreign Address” column are 443.

@mrmr1 can you please post the current compose files of your traefik and target service. And if you still use a toml file to configure traefik, the current state of it as well.

It is much easier to follow the flow when the full picture is visible. We need to see the container labels in case of a compose deployment and the service labels in case of a swarm deployment.

Before going to bed last night I went ahead and posted this issue on the traefik forum. No replies yet this morning.

After sleeping on it I believe the issue is with the webtrees app configuration. The app expects HTTPS traffic on port 443. It works fine when I configure docker container that way (443:443) and run it without traefik. The error gets thrown when traffic comes via a different port. The webtrees image docs describe how to configure the app to receive traffic on another port – using environment variables. However when I tried this I get a different error, “gateway not found”.

@meyay per your request, my config files:

# traefik.toml static config file created 23-jan-2022

[entryPoints]
  [entryPoints.websecure]
    address = ":443"
    [entryPoints.websecure.http]
      [entryPoints.websecure.http.tls]
        certResolver = "lets-encrypt"
  [entrypoints.webtrees]
    address = ":8081"

[api]
  dashboard = true
#  dynamic api config (for dashboard) in traefik's yaml file

[providers.docker]
  endpoint = "unix:///var/run/docker.sock"
  #domain = "mrmr.pro"
  watch = true
  network = "dweb"
  #exposedbydefault = false

[certificatesResolvers.lets-encrypt.acme]
  email = "redacted@protonmail.ch"
  storage = "acme.json"
  [certificatesResolvers.lets-encrypt.acme.tlsChallenge]

[log]
  level = "ERROR"
  filePath = "/logs/traefik.log"

[accessLog]
  filePath = "/logs/traefik-access.log"

# docker-compose.yml for traefik v2.x reverse proxy container
# created 23-jan-2022
# NB: Host rule uses backticks to quote!  So nonintuitive!

version: '3'

services:
  traefik:
    image: traefik:v2.5
    restart: always
  # Here's the docker network we created: 
    networks:
      - dweb
    ports:
      - 443:8081
    labels:
      - traefik.enable=true
      - traefik.docker.network=dweb
      - traefik.http.routers.api.rule=Host(`traefik.mydom.pro`)
      - traefik.http.routers.api.service=api@internal
      - traefik.http.routers.api.middlewares=auth
      - traefik.http.routers.api.tls=true
      - traefik.http.routers.api.tls.certresolver=lets-encrypt
      - traefik.http.middlewares.auth.basicauth.users=redacted:$$apr3$$6sNEg4kB$KnvnKzHdlDeUK.wIXw/
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /opt/traefik/traefik.toml:/traefik.toml
      - /opt/traefik/acme.json:/acme.json
      - /opt/traefik/logs:/logs
    container_name: traefik
networks:
  dweb:
    external: true

# docker-compose.yml for webtrees 
# written 05-jan-2022 by fgb
# NB: conflict w traefik for port 443
# Uses internal netwk to talk w db, dweb netwk to talk w traefik
# ** NB: do NOT indent version or services lines!
#
version: '2'
services:
             webtrees:
                     image: dtjs48jkt/webtrees:v1.7.10
                     ports:
                             - 8081:443
                     depends_on:
                             - db
                     volumes:
                             - /var/www/html/webtrees/data:/var/www/html/data
                             - /var/www/html/webtrees:/var/www/html
                     environment:
                             GROUP_ID: 501
                               #PORT: 8081
                             DISABLE_SSL: "TRUE"
                     networks:
                             - dweb
                             - internal
                     labels:
                             - traefik.enable=true
                             - traefik.http.routers.webtrees.rule=Host(`boohoo.mydom.pro`)
                             - traefik.http.routers.webtrees.tls=true
                             - traefik.http.routers.webtrees.tls.certresolver=lets-encrypt
                               #- traefik.http.services.webtrees.loadbalancer.server.port=443
                             - traefik.docker.network=dweb
             db:
                     image: mariadb:latest
                     restart: always
                     volumes:
                             - /var/lib/mysql:/var/lib/mysql
                     environment:
                             MYSQL_ROOT_PASSWORD: redacted
                     ports:
                             - 3306:3306
                     networks:
                             - internal
                     labels:
                             - traefik.enable=false
networks:
            dweb:
                    external: true
            internal:

Thanks for your patience with me on this. Here’s the error:
bad request

Usualy you would want to use the secure port on 443, handle LE certificates for you domains and use container labels to register reverse proxy rules. Treafik listens on docker events and fetches all container started/stopped events to register/unregister the reverse proxy rules by its own. There is no need to add them in your toml, except you want them staticly configured.

Thus said, traefik doesn’t need a publised port on the target service to access it.

Please remove [entrypoints.webtrees] in your tomly and replace the compose declaration for your webtrees service:

...
        webtrees:
        image: dtjs48jkt/webtrees:v1.7.10
        depends_on:
                - db
        volumes:
                - /var/www/html/webtrees/data:/var/www/html/data
                - /var/www/html/webtrees:/var/www/html
        environment:
                GROUP_ID: 501
                #PORT: 8081
                DISABLE_SSL: "TRUE"
                PORT: 80
        networks:
                - dweb
                - internal
        labels:
                - traefik.enable=true
                - traefik.http.routers.webtrees.rule=Host(`boohoo.mydom.pro`)
                - traefik.http.routers.webtrees.tls=true
                - traefik.http.routers.webtrees.tls.certresolver=lets-encrypt
                - traefik.docker.network=dweb
                - traefik.port=80

Note: the PORT: 80 environment is specific to this image, other images will not have such a setting. Then you will set “treafik.port” the whatever container port is required. The must be a more precise method, but I am not sure which one. I am still using traefik 1.7 myself, as 2.x removed the keystore to share letsencrypt certificates in a swarm cluster.

update: instead of traefik.port=80. it might be traefik.http.services.webtrees.loadbalancer.server.port=80

Finaly use https://boohoo.mydom.pro to access your webtrees container.

Thanks for your assistance @meyay. I made the changes you suggest – I had to add port 443:443 to traefik’s yml after removing the web trees entrypoint (not sure why but traefik dashboard was unreachable until I did this).

I modified webtrees yml per your advice – using label traefik.port=80 I got “bad gateway”, using traefik.http.services.webtrees.loadbalancer.server.port=80 I end up with the ‘bad request’ error I started with.

So back to square one… although it’s very good to know that I don’t need to map ports with docker, traefik does that by itself.
bad request

Actualy the traefik container uses the container network dweb to talk to the target container. That’s why only the traefik container needs to publish ports.

something is not adding up here. Apache in the webtrees container throws an error on port 443, because traefik tries to communicate http with a https port… almost as if you didn’t set PORT:80 and DISABLE_SSL and set the target port to 80. This is not a traefik error message.

“…almost as if you didn’t set PORT:80 and DISABLE_SSL and set the target port to 80. This is not a traefik error message.”

Yes. That’s why I didn’t post in the traefik forum until prompted to. I think it may be an issue with the webtrees container, I will troubleshoot later this afternoon and report back.

OK it’s working now. The issue turned out to be that the old version of the docker webtrees container I was using was published before the author implemented environment variable DISABLE_SSL. I chose to use an old version because I was migrating an even older instance, and webtrees recommends using as close a version as possible during server migrations. Which worked, but… led to my SSL problem. And my poor understanding of traefik’s communication made me waste a lot of time messing with the network config, since it seemed obviously a network problem.

Thanks for everyone’s help with this, especially @meyay for giving me confidence and a better understanding of traefik. I didn’t want to upgrade the webtrees app until I got traefik working, but it turned out that would have prevented the ‘bad request’ error if I’d done it BEFORE adding traefik.