Swarm networking weirdness

I hope this is the correct forum for my question. I’m trying to deploy a starman psgi service to my development 1 node swarm but am unable to make a successful connection to the service when run as a stack. The service is being proxied by a local apache server. I’ve successfully deployed similar stacks but cannot get this to work properly other than using “docker run”. I feel like I’m missing something fundamental here, but I’m at a loss as to what it might be. Below is my my compose file and some command output that should clarify my issue:

version: "3.7"
services:
  starman-devel:
    image: myregistry/starman_dev:latest
    hostname: starman-devel
    ports:
      - target: 3131
        published: 3129
        protocol: tcp
        mode: host

If I change the mode to “ingress” or simply define ports in the compose file to “3129:3131” with no difference.

$ docker ps 
CONTAINER ID        IMAGE                                                                          COMMAND                  CREATED             STATUS              PORTS                    NAMES
2576402a1314        myregistry/starman_dev:latest      "/bin/sh -c '/etc/in…"   3 minutes ago       Up 3 minutes        0.0.0.0:3129->3131/tcp   Starman_STK_starman-dev.1.b2vnp4i0ygioflcxujmlamouw

$  docker service ls
ID                  NAME                         MODE                REPLICAS            IMAGE                                                                          PORTS
x4lr5ttsc7cb        Starman_STK_starman-dev   replicated          1/1                 myregistry/starman_dev:latest

$ telnet localhost 3129
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.

$ netstat -an |grep 3129
tcp6       0      0 :::3129                 :::*                    LISTEN

When I attempt to connect, apache gives me a 502 error (“The proxy server received an invalid response from an upstream server.”) Attempting to telnet to the port from the swarm node yields an immediate closed connection. The application does not record any connections. I’ve verified that the application running in the container is actually listening on it’s defined port and it is working properly.

If I run the container "manually"with a docker run command, it works perfectly:

$ docker run --rm --name StarmanTest2  -p 3129:3131 --hostname=starman-dev -d     myregistry/starman_dev:latest

$ docker ps
CONTAINER ID        IMAGE                                                                          COMMAND                  CREATED             STATUS              PORTS                    NAMES
9166217a2671        myregistry/starman_dev:latest                                                        "/bin/sh -c '/etc/in…"   27 minutes ago      Up 27 minutes       0.0.0.0:3129->3131/tcp   StarmanTest2

$ telnet localhost 3129
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

In addition, I’ve opened up port 3129 on the host firewall to all traffic. port forwarding is enabled for both ipv4 and ipv6.
The swarm node is running docker-ce 19.03 on RHEL/CentOS 7.7

Where is apache reverse proxy located?

Apache is running natively (non-containerized) on the same host. We are working towards moving our entire environment to containers by migrating one app at a time.

Though so. What makes you think that localhost inside a container would be the same localhost of your docker host?

I don’t think that. Those telnet attempts and proxy connections are trying to be made from the docker host outside the container, not inside.

Sorry, I confused this thread with the one with the eclipse containers. My bad.

Did you try to deploy your compose-file with docker-compose to elimate general problems with overlay networks? As long as a process binds port 3131 inside the container. Access on you host port 3129 should be forwarded to the container port 3131.

For instance, on Synology the Docker-CE implementation provides everything necessary to use Swarm deployments, but it suffers from missing kernel modules for the overlay network. Maybe you are in a similar situation?

No worries. netstat and telnet within the container show that the process is listening on port 3131, the same as it does when launching with “docker run”. Outside the container port 3129 is open and listening according to netstat when the stack is deployed. I’m deploying the process with “docker stack deploy -c compose.yml StarmanMT_STK”, the name was changed in my last launch for my own mental clarity. It appears that the overlay network should be functioning properly given the output of network ls below and my other proxied stacks are functional. This app will need to be deployed to our production swarm so I’ve been focusing on getting the stack deploy to work properly.

$ docker network ls
NETWORK ID          NAME                     DRIVER              SCOPE
cqy4aztizlnm        RenderTron_STK_default   overlay             swarm
ow9uf8ko45e4        StarmanMT_STK_default    overlay             swarm
94c1263f8fdc        bridge                   bridge              local
s4kx88dvyox4        xyz-db-STK_mysql-net    overlay             swarm
f9dc92b6542f        docker_gwbridge          bridge              local
ff5d67928d95        drupaldbnet              bridge              local
0721426a6d51        host                     host                local
ry714h4mbkdz        ingress                  overlay             swarm
aaf5571697e2        none                     null                local

If other stacks do work, then everthing should be fine.
The whole situation does not realy add up. I can’t see anything wrong with it- espacialy if other containers, created as swarm tasks (within stack deployments) are reachable from outside.