Docker swarm overlay network not working on PORT

I’ll create so similiar seniario to my problem.

There is two ec2 nodes that I created in same vpc.

These are my inbound rules for both of them.

This command I run when ec2 instance run.

  • sudo docker swarm init --advertise-addr “MY-PUBLIC-IP”
  • docker node update “First node” --label-add service=websrv1
  • docker node update “Second node” --label-add service=websrv2
  • docker network create -d overlay tersinesat

Then I created services like below;

  • docker service create --name websrv1 --network tersinesat --publish 8081:80 --constraint ‘node.labels.service==websrv1’ ozgurozturknet/web:v1

  • docker service create --name websrv2 --network tersinesat --publish 8082:80 --constraint ‘node.labels.service==websrv2’ ozgurozturknet/web:v1

I can ping the container from inside of each other. But I can’t react of the port 80 or in other case which service port you publish, doesn’t matter.

This is just example case.

Doesn’t the command also require the node name which should be updated?

You want to connect from one container to the other on port 80? Is the app inside actually listening on port 80? Did you try curl or wget inside the container?

Doesn’t the command also require the node name which should be updated?

Yes, text editör doesn’t recognize my tag. Whatever I edited my text.

You want to connect from one container to the other on port 80? Is the app inside actually listening on port 80? Did you try curl or wget inside the container?

Basically I can sent ping with service-name like

  • ping websrv2
    and answer is coming. Dns resolver is working. But when I use telnet websrv2 8082 port. It’s not working


Ping

I don’t think I have seen any forum in IT where code blocks were not supported and required. Please, read the following guide to format your posts: How to format your forum posts

What do you expect? Is the connection established? If it’s a http port, then you would need to manually enter a request to see any response.

Basically these service works in play with docker applications. But I don’t know they handle network behind the scene.

What do you expect? Is the connection established? If it’s a http port, then you would need to manually enter a request to see any response

Yes there should be connection established. Bc port is correct 8082 and it should routering to my second container 80 all traffic port. I tried manuel response is same :confused:

Why would the host port 8082 be correct, if one container tries to reach a service of another container through a docker overlay network?

It should react to {service-name}:{container-port}, and not {service-name}:{host-port}. The later will work in no scenario. In a host or remote host to container scenario, you would use the {host-name-or-ip}:{host-port}

  • How do we know both hosts are using the same security group? If the instance has a public interface, you pretty much opened up your swarm control plane, esp encryption and overlay ports to the whole world. For those ports you want the source to be the very same security group id. Furthermore, you want to remove the all ports, all protocols rule.

I used to use swarm on aws years ago, and it worked like a charm when the security group allowed traffic on tcp ports 2377 + 7946, and udp ports 4789 + 7946. Those ports are allowed in your SG, so it should work.

I highly suggest simplifying your tests by deploying services using an image like the well known nginx image.

My docker swarm orchestration was working until 1 month ago. My load balancing nginx node server need to be close so I removed node and I added new node for load balancing then overlay network didn’t work with new kernel or new docker version. I couldn’t figure out what was the problem. But some how I tried everything, It did’nt work. Here I share the case simplifying also

Those are valuable information that should have been part of the first post.

Please share the output of docker info, so we get an understanding what os, kernel and docker version you are actually using.

When it comes to the Docker Engine: this forum is a support channel for the docker-ce packages from docker’s repository, running on supported operating systems: https://docs.docker.com/engine/install/#supported-platforms .

If for instance, your os happens to be Amazon Linux, then your best bet is to raise a support ticket @aws, as they are the support channel for their os and the packages of their os.

I previously had a Docker Swarm orchestration with 6 nodes. After adding a new Nginx to the orchestration, a connection issue arose between the ports of the nodes’ services. Below, I am sharing an exact example of my orchestration.

You can follow the steps below to install Docker on your Amazon Linux server:

  • Update Your Server:
  • First, update the existing packages.

sudo yum update -y

  • Install the Docker Package:
  • Install Docker from the repository for Amazon Linux 2.

sudo amazon-linux-extras install docker -y

  • Start Docker:

sudo service docker start

  • Set Up Automatic Start:
  • To start Docker automatically when the server starts:

sudo systemctl enable docker

  • Add User to Docker Group:
  • This step is necessary if you want to run Docker commands without sudo. For example, to add the “ec2-user” user to the Docker group:

sudo usermod -a -G docker ec2-user

newgrp docker

Create docker swarm from first aws node

sudo docker swarm init --advertise-addr <IP_ADDRESS>

elastic ip available elastic ip
this is the part that comes up in amazon, privilege required

After added second as worked to swarm, We name the nodes by service name

docker node update <1st node> --label-add service=hello

docker node update <2nd node> --label-add service=nginx

We create overlay network

docker network create -d overlay --opt com.docker.network.driver.mtu=1450 nginx

First node will be hello-service

Copy code

sudo nano

const express = require('express');
const app = express();
const port = 3000;

app.get('/hello', (req, res) => {
  res.send('Hello World!');
});

app.listen(port, () => {
  console.log(`App is running at http://localhost:${port}`);
});

save app.js

Copy code package.json

sudo nano

{
  "name": "express-app",
  "version": "1.0.0",
  "description": "A simple Express.js application",
  "main": "app.js",
  "scripts": {
    "start": "node app.js"
  },
  "dependencies": {
    "express": "^4.17.1"
  }
}

save package.json

sudo nano

# Get a Node.js-based image
FROM node:latest

# Create a working directory where we can run the application code
WORKDIR /usr/src/app

# Install application dependencies
COPY package*.json ./
RUN npm install

# Copy the application code
COPY . .

# Run the application
CMD [ "node", "app.js" ]

save Dockerfile

  • Now, build the Docker image using the Dockerfile. Run the following command in the terminal in the directory containing this file and your application files:

Copy code

docker build -t express-app .


docker service create --name hello-service --network nginx --constraint 'node.labels.service==hello' --publish 3000:3000 express-app:latest

After create hello-service we create nginx-service

docker service create --name hello-service --network nginx --constraint ‘node.labels.service==hello’ --publish 3000:3000 express-app:latest

sudo nano

worker_processes 1;

events { worker_connections 1024; }

http {
    resolver 127.0.0.11 valid=10s;

    server {
        listen 80;

	location / {
                default_type text/plain;
                return 200 'Hello World!';
        }

        location ~ ^/hello-service {
            rewrite ^/hello-service/(.*) /$1 break;
            proxy_pass http://hello-service:3000;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'Upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;
        }

    }
}

save nginx.conf

  • Then, create a file named Dockerfile and paste the following content into it:
    Dockerfile

Copy code

sudo nano

# Get an Nginx-based image
FROM nginx:latest

# Copy custom Nginx configuration
COPY nginx.conf /etc/nginx/nginx.conf

save Dockerfile

  • Build the Docker image using the Dockerfile. Run the following command in the terminal in the directory containing this file and the Nginx configuration file:
    bash

Copy code

docker build -t mynginx -f Dockerfile .
  • You can use the following command to start the Docker image:
    bash

Copy code

docker service create --name nginx-service --network nginx --constraint 'node.labels.service==nginx' --publish 80:80 mynginx:latest

Now problem is when we try to get request nginx node ip4 address, response is coming. When we ping inside nginx container to other node inside helle-service container by service-name, It’s working hostname resolver. But when we try to connect port, problem is starting :slight_smile: Do you have an any idea?

I already asked

but I do again


Please, format your post according to the following guide: How to format your forum posts
In short: please, use </> button to share codes, terminal outputs, error messages or anything that can contain special characters which would be interpreted by the MarkDown filter. Use the preview feature to make sure your text is formatted as you would expect it and check your post after you have sent it so you can still fix it.

Example code block:

```
echo "I am a code."
echo "An athletic one, and I wanna run."
```

It might be a MTU problem, why do you set it to 1450 and not 1400 (which usually works over VLAN/vSwitch/VPN)?

Try a simple ping with a payload of 1500 bytes to check. Simple ping usually works, with payload>MTU it shows potential issues.