Connection refused for NextJS inside Docker swarm only (Works for general run)

Open Source Projects Swarm
1

I have my private self-hosted docker image with nextjs inside. This image exposes 3000 port.

I tried to run this image via docker and docker swarm.

  1. Docker
docker run -p 3000:3000 path-to-my-image

Result:

docker ps:

0.0.0.0: 3000->3000/tcp, ::: 3000->3000/tcp

ps inside container:

PID   USER     TIME  COMMAND
    1 nextjs    0:00 next-router-wo
   22 nextjs    0:01 next-render-worker-app
   23 nextjs    0:01 next-render-worker-pages
   51 nextjs    0:00 sh
   57 nextjs    0:00 ps

netstat -tulp|grep 3000 inside container:

tcp        0      0 cf748210f6f3:3000       0.0.0.0:*               LISTEN      1/next-router-wo

curl http://127.0.0.1:3000 outputs correct response. So all is fine.

  1. Docker swarm

stack.yaml

version: "3.7"

services:
  app:
    image: path-to-my-image
    ports:
      - 3000:3000
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]

docker service ls shows service is online, 1 replica

ps inside container:

PID   USER     TIME  COMMAND
    1 nextjs    0:00 next-router-wo
   22 nextjs    0:01 next-render-worker-app
   23 nextjs    0:01 next-render-worker-pages
   50 nextjs    0:00 sh
   56 nextjs    0:00 ps

netstat -tulp|grep 3000 inside container:

tcp        0      0 3e53ca570e77:3000       0.0.0.0:*               LISTEN      1/next-router-wo

But curl http://127.0.0.1:3000 fails:

curl: (7) Failed to connect to 127.0.0.1 port 3000 after 0 ms: Connection refused

Tried another port, another machines, tried node and pm2. All is same.

I can’t understand the reason and where to dig next.

Thank you in advance.

2

Just tested this, we usually use Traefik without the Docker ingress network.

version: '3'

services:
  whoami:
    image: traefik/whoami:v1.10
    hostname: '{{.Node.Hostname}}'
    ports:
      - 3000:3000
    deploy:
      mode: global

When checking the ports I see port 3000 used on tcp6, but not on regular tcp4:

# netstat -tulpn | grep 3000
tcp6       0      0 :::3000                 :::*                    LISTEN      730/dockerd

That’s strange.

3

Just tested this on a brand new Debian 12 VM:

sudo apt update && sudo apt -y upgrade

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

cat <<EOF > whoami.yml
version: '3'
services:
  whoami:
    image: traefik/whoami:v1.10
    hostname: '{{.Node.Hostname}}'
    ports:
      - 3000:80
    deploy:
      mode: global
EOF

docker swarm init
docker stack deploy -c whoami.yml whoami

Leads to same result, no port 3000 on tcp4:

# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      862/sshd: /usr/sbin
tcp6       0      0 :::3000                 :::*                    LISTEN      8351/dockerd

UPDATE: Same on new VM with Ubuntu 22.04, no port 3000 on tcp4.

Maybe @meyay can have a look, I heard he is the Swarm expert :slight_smile:

UPDATE2:
Tested another image on Debian 12 VM

sudo apt update && sudo apt -y upgrade
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
reboot

cat <<EOF > python.yml
version: '3.9'
services:
  python:
    image: python:3.9-slim
    entrypoint: ["/bin/sh", "-c"]
    command: 
      - |
        echo Starting
        python -m http.server 80
    ports:
      - 3000:80
    deploy:
      mode: global

EOF

docker swarm init
docker stack deploy -c python.yml python

Same result:

# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      807/sshd: /usr/sbin
tcp6       0      0 :::3000                 :::*                    LISTEN      826/dockerd
4

I’m not him and definitely know less about Swarm, but have you tried curl as well or just checked the output of netstat? Apperantly when something is listening on all IPs including IPv6 IP addresses netstat will show you “:::PORT”. I’m not sure why.

5

I am not sure either, but I noticed it too! On some OSes applications that bind on ipv4/ipv6 dual stack, only the ipv6 binding is listed in netstat. I second what @rimelek suggested: try accessing the ipv4 address with wget, curl or nc,

6

Did two repositories and public docker images (see Makefile):

  1. GitHub - indapublic/docker-swarm-expose-test (Pure JS, Works)

  2. GitHub - indapublic/docker-swarm-expose-test-nextjs (NextJS, Issue)

So probably main reason is NextJS or something I didn’t notice. Do you have any idea?

7

Ok, another try. New Debian 12 VM.

sudo apt update && sudo apt -y upgrade
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
reboot

cat <<EOF > whoami.yml
version: '3'
services:
  whoami:
    image: traefik/whoami:v1.10
    hostname: '{{.Node.Hostname}}'
    ports:
      - 3000:80
    deploy:
      mode: global
EOF
docker swarm init
docker stack deploy -c whoami.yml whoami

Still netstats shows no no tcp (4):

# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      826/sshd: /usr/sbin
tcp6       0      0 :::22                   :::*                    LISTEN      826/sshd: /usr/sbin
tcp6       0      0 :::3000                 :::*                    LISTEN      840/dockerd

But a curl works successfully:

# curl http://127.0.0.1:3000
Hostname: debian-4gb-nbg1-1
...

# curl http://<external-IP>:3000
Hostname: debian-4gb-nbg1-1
...
8

I actually did build the image from docker-swarm-expose-test-nextjs, and even though the nodejs webapp is accessible inside the container on port 3001, it is not accessible from the host on port 3001.

Not sure if it’s a problem with how the webapp is configured, but then again I have no idea of nodejs.

9

I tried the nextjs project and worked. I tried with the original image and also building a new image from the Dockerfile.

Did you change the port? The app listens on port 3000 which worked for me on Ubuntu 22.04. So the app at least seems working.

10

This is what I did:

git clone https://github.com/indapublic/docker-swarm-expose-test-nextjs.git
cd docker-swarm-expose-test-nextjs/
docker build -t indapublic/docker-swarm-expose-test-nextjs:test .
docker stack deploy -c stack.yaml test

Then I looked on the host and inside the container with netstat -tlpn, tried curl http://localhost:3000 and curl http://localhost:3001.

When I tested it, the current commit id was 57d2b4d49482bfb4f33e92133c007d5410a2d0a3.

11

It seems I thought the dummy.js was responsible for running the service which sets port 3001, which is strange because I was sure it set port 3000, but I was wrong., On the other hand, the environment var in the Dockerfile sets port 3000. I indeed tried a different commit, but I don’t see any change in ports. port 3000 works, on which the server is listening and it works from the host and from the container.