Docker Community Forums

Share and learn in the Docker community.

Docker Swarm Mode network and load balancing doesn't work for a service in worker node


(Appristas) #1

My setup

Two nodes (2GB RAM, 2 vCPU) running docker engine (v17.06.1-ce) – one swarm and one worker. Internal network bandwidth: 10Gbps. All files and databases are located outside this docker cluster (AWS S3 and different instances for database).

What I am trying to achieve?

I am trying to create a docker based “platform” where I push stateless services and docker handles load balancing, updates etc. Besides this, I am also trying to set up reverse proxy and allow specific services to have access to this proxy.

What I have done so far?

Firstly, I created an overlay network and called it “public.” (10.0.9.0/24) Then, I created an nginx service in “global” mode. The service by itself is attached to “public” network. I checked both my worker and swarm nodes and the service runs in both of them without a problem.

Secondly, I created docker compose file for fast deployment of multiple services. For the sake of my testings, I kept one service per compose file:

version: '3.3'
services:
  web:
    image: app1_image:latest
    networks:
      - public
networks:
  public:
    external:
      name: public

For the second service, I just changed the image name and kept everything else the same. Ran both “stacks”:

docker stack deploy --with-registry-auth --compose-file compose1.yml app1
docker stack deploy --with-registry-auth --compose-file compose2.yml app2

After inspecting both services, I see that both services are in “overlay” network with IPs such as 10.0.9.5 (app1_web) and 10.0.9.6 (app2_web). app1_web is created in swarm node and app2_web is created in worker node.

So, I create two nginx config files for both of my services in the following way:

server {
    listen 80;
    server_name app1.example.com;
    location / {
        proxy_pass http://app1_web; # This line is important
        # Other proxy parameters
    }
}

As you see I am passing the service name in nginx configuration. For easier config management I use docker configs:

docker config create nginx_app1.conf app1.conf
docker config create nginx_app1.conf app1.conf

docker service update --config-add source=nginx_app1.conf,target=/etc/nginx/conf.d/app1.conf nginx_proxy
docker service update --config-add source=nginx_app2.conf,target=/etc/nginx/conf.d/app2.conf nginx_proxy

Adding these configs automatically restarts nginx services and runs them. This is all. I wanted to give you a gist of my process before moving forward.

The problem

app1_web is created in swarm; so, when I go to app1.example.com, nginx proxies my request to the service and I get a proper output. This is what is expected and I am happy with the outcome.

However, because app2_web is created in worker node, nginx gives me an error that app2_web does not exist. So, I started troubleshooting.

From swarm, I found the docker instance ID and tried to run a command from nginx proxy:

docker exec nginx-proxy-id ping app2_web

This gave me an “Bad address” error. So, I went into compose2.yml and added ports:

ports:
  - 5380:80

When I went to swarm.example.com:5380, it basically gave me 404. However, opening the same port from worker.example.com:5380 opened app2.

I tested the same for app1. I replicated app1 using docker service scale app1=2 and the service got created in worker node. Then I paused the service in swarm using docker pause app1-id. When I went to app1.example.com, it would work half the time. I think it was still weird because I was expecting Docker to know that service is paused and only proxy the service to worker node but whatever. At least it was working. Replicating app2 did not help though. I still kept getting error that host name does not exist. After this, I went further and told the worker node to leave the swarm: docker swarm leave and coincidentally, everything worked normally…

After spending at least 10 hours on this, I am lost on what I am doing wrong here. For some reason, when service is created in worker first, Docker doesn’t like it.

Sorry for such a long wall of text. I wanted to share all the steps I took. I would really appreciate your help.


(Appristas) #2

Bump! Is this a normal behavior for Docker? I really don’t know what to do about this.


(Garylonbrown) #3

Should the following:
docker config create nginx_app1.conf app1.conf
docker config create nginx_app1.conf app1.conf
actually be:
docker config create nginx_app1.conf app1.conf
docker config create nginx_app2.conf app2.conf