Haproxy: backend 'app' has no server avaiable

Docker Hub
,
1

Hi everyone, I’m new on using haproxy and Docker and I’m having trubles setting up a web app with Docker-compose.
I am using an Node.js app listening on port 8001 and it connects on a local database mongodb on port 27017. I have created a Docker compose file for configuring a Docker environment with 3 serves of the app, the DB and the haproxy server.

version: “3”
services:
   app1:
     container_name: app1
     restart: always
     build: .
     ports:
       - “81:8001”
     links:
       - mongo
     networks:
       - my-net
   app2:
     container_name: app2
     restart: always
     build: .
     ports:
       - “82:8001”
     links:
       - mongo
     networks:
       - my-net
    app3:
      container_name: app3
      restart: always
      build: .
      ports:
        - “83:8001”
      links:
        - mongo
      networks:
        - my-net
    mongo:
      container_name: mongo
      image: mongo 
      ports:
        - “27017:27017”
      networks:
        - my-net
    haproxy:
      build: ./haproxy
      container_name: haproxy
      links:
        - app1
        - app2
        - app3
      ports:
        - “80:80”
        - “8404:8404”
      networks:
        - my-net
    networks: 
      my-net:
        name: my-net
        driver: bridge

the docker files for the haproxy is this:

FROM haproxytech/haproxy-alpine:2.4
COPY haproxy.conf /usr/local/etc/haproxy/haproxy.conf

and the docker file for the apps is the following:

FROM node:13-alpine

RUN mkdir -p /home/app

COPY ./nosql_practica4_bbdd-main /home/app

WORKDIR /home/app

RUN npm install

RUN npm run seed

CMD [“npm”, “start”]

EXPOSE 8001

The app is working if I type the ports on the browser but the haproxy isn’t working and on the logs it shows this error:
[ALERT] (8) : backend ‘app’ has no server available! docker

the haproxy.conf is the following:

global
  stats socket /var/run/api.sock user haproxy group haproxy mode 660 level admin expose-fd listeners
  log stdout format raw local0 info

defaults
  log global
  mode http
  option httplog
  option dontlognull
  timeout connect 5000
  timeout client 10000
  timeout server 10000
   
frontend balancer
  bind 0.0.0.0:80
  mode http
  default_backend apps

backend apps
  mode http
  balance roundrobin
  server app1 app1:8001 check
  server app2 app2:8001 check
  server app3 app3:8001 check

anyone can help me understand what I have done wrong?

2

Can you please reformat your post and wrap the content of the compose file and of the Dockerfiles in Preformatted text (</> icon) blocks. It is hard to read the way it is now. I must admit, I looked at the frist lines and gave up. Also adding the haproxy.conf would be helpfull to get a better understanding.

Note: I personaly don’t use haproxy, so it might be very well that I won’t able to help you, but still others might get a better chance to do so.

3

Thank you for the tips, I updated the post with the changes you suggested.

4

Observations:

  • The networks: declaration at the bottom should be a top level element. In your compsose file, it’s a sub element of the top level element services:. I asume this is a copy/paste error, as otherwise the containers wouldn’t start at all because of the invalid syntax and the resulting non existance of

  • The Dockerfile for the apps exposes port 8001 - did you create the application to actualy listen on port 8001? Expose has no effect by itself, it is just for documentation purposes. Since you wrote that the apps are accessible by their published host port, this must be the case. Though, it does not say, wether the application binds 0.0.0.0:8001 or 127.0.0.1:8001 - it must be the first one!

  • Port mapping of port 8404 on the haproxy service has no effect, as ha proxy only bind 0.0.0.0:80.

  • links: you can replace them by depends_on, as dns-based service discovery is already a part of user defined networks (docker-compose creates those!). Using links: Links are usefull If you want to inject environment variables and declare a

  • The haproxy error message doesn’t seem to match the name of the backend.

Note: it appears that app1, app2, app3 are identical stateless replicas/containers of the same image - you could use replicas: or scale: depending on the compose file schema and don’t use haproxy all along… The new merged compose file specification documentation says replicas:, but I am not sure if this works for docker-compose deployments. If it doesn’t you need to use schema version 2.4 with scale: if you want to use replicas.

I am going to research on haproxy configurations now.

5

I just google arround and I would try it with this configuration:

global
  stats socket /var/run/api.sock user haproxy group haproxy mode 660 level admin expose-fd listeners
  log stdout format raw local0 info

defaults
  log global
  mode http
  option httplog
  option dontlognull
  timeout connect 5s
  timeout client 10s
  timeout server 10s

resolvers docker
    nameserver dns1 127.0.0.11:53
    resolve_retries 3
    timeout resolve 1s
    timeout retry   1s
    hold other      10s
    hold refused    10s
    hold nx         10s
    hold timeout    10s
    hold valid      10s
    hold obsolete   10s
   
frontend balancer
  bind 0.0.0.0:80
  default_backend apps

backend apps
  server app1 app1:8001 check resolvers docker init-addr libc,none
  server app2 app2:8001 check resolvers docker init-addr libc,none
  server app3 app3:8001 check resolvers docker init-addr libc,none

It adds the build-in dns resolver for custom networks, and ignores if the service is not available on startup. Also the “mode: http” is removed from frontend and backend.

6

Thank you for your quick help.

Yes, the app is listening on port 8001 that is why I have linked the port 8001 in the docker compose file. The network my-net is just a copy paste error, it is not a sub element of services: .

I changed the links to depends_on: and also I changed the haproxy.conf as you suggested but still the balancer is not working. The app is still working if I type http://localhost:81, http://localhost:82 and http://localhost:83 but I still cannot acces to the app from the port of the balancer (80).

this are the logs of the container of the haproxy server:

[NOTICE]   (1) : New worker #1 (8) forked
[WARNING]  (8) : Server static/static1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (8) : Server static/static2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[NOTICE]   (8) : haproxy version is 2.4.18-1d80f18
[ALERT]    (8) : backend 'static' has no server available!
[WARNING]  (8) : Server app/app1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (8) : Server app/app2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (8) : Server app/app3 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (8) : Server app/app4 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT]    (8) : backend 'app' has no server available!
7

It looks like your application does bind 127.0.0.1.8001 instead of 0.0.0.0:8001. While the first address might work for published ports, it will not work for container network communication.

Also can you please post your updated haproxy.conf (the one you actualy use!), as the error message does not match the configuration you shared earlier. Something is not adding up here. Also the haproxy.conf does only work that way if the haproxy container does not use network_mode: host it must use a bridge network in order to reach the appX containers by service name.

8

I figured out why it was showing that weird error message.

I executed the shell (docker exec -it a4a /bin/sh) in the haproxy container and it showed this:

/usr/local/etc/haproxy # ls
dataplaneapi.hcl          errors            haproxy.cfg       haproxy.conf

As you can see I called the configuration file haproxy.conf instead of haproxy.cfg and doing so it was creating an other file in the haproxy folder (inside the haproxy container) and it was instead using the default configuration file.

This is the final haproxy.cfg file I used and it worked!

global
  stats socket /var/run/api.sock user haproxy group haproxy mode 660 level admin expose-fd listeners
  log stdout format raw local0 info

defaults
  log global
  mode http
  option httplog
  option dontlognull
  timeout connect 5000
  timeout client 10000
  timeout server 10000
   
frontend balancer
  bind *:80
  mode http
  default_backend apps

backend apps
  mode http
  balance roundrobin
  server app1 app1:8001 check
  server app2 app2:8001 check
  server app1 app3:8001 check