Can't access docker container through port forwarded host machine from external hosts

I’m running docker container using below command and this container’s host is in the internal private network.

docker run --restart always --name srs2 -d -p 1935:1935 -p 1985:1985 -p 8080:8080 -p 8000:8000/udp -p 10080:10080/udp ossrs/srs:5
I need to access container’s 8080 port from the external hosts and i’ve set port forwarding in my router config page as 28080:8080. With this setup i expect i can access container using 28080 port but this not even allows me to start tcp handshake(SYN packet is not reaching).

I should mention that it is possible to access other ports(not in the container) of host without any issue. So there is no issue on port forwarding setup. Also it is possible that accessing container from other hosts in the same private network but not external.

The below is the output of docker inspect

"NetworkSettings": {
            "Bridge": "",
            "SandboxID": "59c04190dbc0ee6f24b8e48a3e3e4ac927995c35ebb00bdef057b4ab0887d025",
            "SandboxKey": "/var/run/docker/netns/59c04190dbc0",
            "Ports": {
                "10080/udp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "10080"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "10080"
                    }
                ],
                "1935/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "1935"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "1935"
                    }
                ],
                "1985/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "1985"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "1985"
                    }
                ],
                "5060/tcp": null,
                "8000/udp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "8000"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "8000"
                    }
                ],
                "8080/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "8080"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "8080"
                    }
                ],
                "9000/tcp": null
            }

This is the output of ss -ao sport :8080 which ensures the port is listening.

❯ ss -ao sport :8080
Netid   State    Recv-Q   Send-Q     Local Address:Port           Peer Address:Port   Process
tcp     LISTEN   0        4096             0.0.0.0:http-alt            0.0.0.0:*
tcp     LISTEN   0        4096                [::]:http-alt               [::]:*

And there is no firewall on my machine, The below one is the output of iptables --list-rules

-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p udp -m udp --dport 10080 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p udp -m udp --dport 8000 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 1985 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 1935 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN

Added docker version output

❯ docker version
Client: Docker Engine - Community
 Version:           26.1.2
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        211e74b
 Built:             Wed May  8 13:59:58 2024
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.2
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       ef1912d
  Built:            Wed May  8 13:59:58 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Can you connect locally on the node to the port of the container?

curl http://localhost:8080

Yes, both of localhost and its ip address.

Then something seems wrong with your router port forwarding. Try running whoami on the port and see if you can connect from external.

I had a similar problem, although not with Docker, but with LXD because of Docker. In my case Docker’s firewall rules disabled routing to the virtual machines.

In your case it could be a missing route or a route that overrides the one that is supposed to allow external access to your containers. Diy ou check the routing table as well?

ip route

For example, in my virtual machine I saw this:

default via 192.168.218.1 dev enp0s1 proto dhcp src 192.168.218.4 metric 100
172.18.0.0/16 dev br-a03940360fc8 proto kernel scope link src 172.18.0.1 linkdown
172.30.0.0/24 dev docker0 proto kernel scope link src 172.30.0.1
192.168.218.0/24 dev enp0s1 proto kernel scope link src 192.168.218.4 metric 100
192.168.218.1 dev enp0s1 proto dhcp scope link src 192.168.218.4 metric 100

Then I deleted the docker network routes

ip route del 172.18.0.0/16
ip route del 172.30.0.0/24

and the port stopped working.

I can connect to docker host from external. There is some ports that opened for all and these ports can be connected from external without any problem.

Below is the output of ip route on docker host.

default via 192.168.219.1 dev eth0 proto static metric 100
default via 192.168.0.1 dev eth0 proto static metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.0.0/16 dev eth0 proto kernel scope link src 192.168.219.202 metric 100
192.168.219.0/24 dev eth0 proto kernel scope link src 192.168.219.202 metric 100

As far as i know the forwarded ip is converted to internal device’s ip which is i set on the router’s port forwarding page and this internal ip is 192.168.219.202

What are these two networks?

192.168.0.0/16 dev eth0 proto kernel scope link src 192.168.219.202 metric 100
192.168.219.0/24 dev eth0 proto kernel scope link src 192.168.219.202 metric 100

The first includes the second. You should have distinct networks.

I’m not really sure… but i extended the default internal ip range of router. The default range was 192.168.0.0/24 and i extended it to 192.168.0.0/16 so that i can assign 192.168.219.x ip. Did i do something wrong?

How exactly did you do it? I did the same to once, but the network was not duplicated. Did you restart the machine after the cnahged network?

Well, I entered to router setting page and changed the default ip range and the 192.168.219.x is assigned manually. The router is restarted after i changed the range.

Can you share which kind of routr you are using? Changing the range is one thing, but you also cnanged the gateway and you got a second default route. Which one is the actual gateway (router IP)?

YES!! I just deleted the duplicated routers 192.168.219.1 after then finally it works!! Thank you so much it seems the connection from external couldn’t find which router should use.

You are welcome :slight_smile: I’m glad it was not complicated as I expected :slight_smile:

I think the internal device had its default router since it was in the other network. I borrowed this device from my friend.