Port mapping across several IPs results in error

Hi!

I have docker stack running on a Raspberry Pi with docker-compose. It’s been going nicely for about a year now, but now I want to bring it to the next level.

The issue is, the stack contains a few services with website, which I access from the network. Obviously, that means I always have to remember the ports of each of those services, which has become tiresome after a while. And I do this from several devices in the network and sometimes several browsers on each device (I have my reasons :slight_smile: ) so creating bookmarks isn’t an option.

So I thought I would solve this via DNS, as I already have pi-hole running in a container nicely as both a DNS and a DHCP server. For this to work I need, among others, the containers to listen to port 80.

Unfortunately, this is where the trouble shows.

The first thing I tried was to use ipvlan and macvlan networks. And it kind of works. But they still don’t solve the issue of remembering the ports, as the containers keep their current one. Port mapping doesn’t help here because of the reason below.

Another issue with this is that, as the containers also communicate to each other and there is a lot of existing configuration there, with IPs and ports, I have to connect them to 2 networks, a bridge for the inter-container communication, and an ipvlan for the outside communication. However, the port mapping is only one, and I don’t understand how it works when a container is connected to 2 networks. I thought this should have worked, but it didn’t.

So the next idea I had was to create several IPs on the host, then use the syntax “host_IP:port:container_port” to map as necessary.

So, for example, I have on the host

eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether dc:a6:32:07:f8:85 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.10/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 10.0.0.50/24 scope global secondary eth0
valid_lft forever preferred_lft forever
inet 10.0.0.51/24 scope global secondary eth0
valid_lft forever preferred_lft forever
inet 10.0.0.52/24 scope global secondary eth0
valid_lft forever preferred_lft forever
inet6 fe80::185a:8f0e:78b0:a110/64 scope link
valid_lft forever preferred_lft forever

and then in docker-compose.yml I have (only relevant part):

  grafana:
    ports:
    - "3000:3000"
    - "10.0.0.50:80:3000"

  nodered:
    ports:
    - "1880:1880"
    - "10.0.0.51:80:1880"

Unfortunately, this throws a bunch of errors when I run docker-compose up -d

ERROR: for grafana Cannot start service grafana: driver failed programming external connectivity on endpoint grafana (b70ca36d6f0953ca16e046c7af561a638ce2fe537417600e6c7f7796fbf228c0): Error starting userland proxy: listen tcp4 10.0.0.50Recreating 922d1912d260_nodered … error

ERROR: for 922d1912d260_nodered Cannot start service nodered: driver failed programming external connectivity on endpoint nodered (6676c921fc83a1a521db50bce6ef128ec7811b4dae4d69cf0597b9f118199bfc): Error starting userland proxy: listen tcp4 10.0.0.51:80: bind: address already in use

ERROR: for grafana Cannot start service grafana: driver failed programming external connectivity on endpoint grafana (b70ca36d6f0953ca16e046c7af561a638ce2fe537417600e6c7f7796fbf228c0): Error starting userland proxy: listen tcp4 10.0.0.50:80: bind: address already in use

ERROR: for nodered Cannot start service nodered: driver failed programming external connectivity on endpoint nodered (6676c921fc83a1a521db50bce6ef128ec7811b4dae4d69cf0597b9f118199bfc): Error starting userland proxy: listen tcp4 10.0.0.51:80: bind: address already in use
ERROR: Encountered errors while bringing up the project.

It sounds a bit like docker can not share the same port even when it belongs to several IPs on the host. I have a limited understanding of this, but from what I read, they shouldn’t conflict. A NIC with several IPs assigned can handle all ports on each IP, as the port belongs to TCP and UDP.

Any suggestions?

I assume the 10.0.0.x ips belong to the ipvlan? if so remove the published ports, the ipvlan is already accessible from outside, there is no port publishing involved.

No. Sorry I wasn’t more clear on that.

The multiple IPs are allocated to the eth0 of the host.

The Docker stack doesn’t have any ipvlan or macvlan network. There is a single bridge network.

In the meantime, I found this website which describes pretty much exactly what I want to achieve.

So I completely don’t understand why the error is there. I’m missing something.

So the 2nd part of this statement is incorrect and can be ignored.

Now that I know that it is just one of the host ips, the answer is simple:

The first published port binds host port 3000 on ALL network ips available to the system.
The 2nd published port binds host port 3000 on the ip 10.0.0.50. Just remove this, as it already is done by the above. You basically tried to bind the port 3000 on the ip 10.0.0.50 twice and caused a port conflict.

Thanks for the suggestion.

So I modified docker-compose.yml like this:

grafana:
ports:
- “10.0.0.50:80:3000”
nodered:
ports:
- “10.0.0.51:80:1880”

But when I run docker-compose up -d I still get the errors:

Recreating 1b5e2f442ad9_grafana … error

Recreating 922d1912d260_nodered … error
cp4 10.0.0.50:80: bind: address already in use

ERROR: for 922d1912d260_nodered Cannot start service nodered: driver failed programming external connectivity on endpoint nodered (9be1530fa5af3968a8cd3b7716ecad1980ef5745d89fdeef71baa0f9e77f951e): Error starting userland proxy: listen tcp4 10.0.0.51:80: bind: address already in use

ERROR: for grafana Cannot start service grafana: driver failed programming external connectivity on endpoint grafana (7f03775e5b9a085a8a1c37e029cfd8f3031437f4b520b6e795d882da66db34f4): Error starting userland proxy: listen tcp4 10.0.0.50:80: bind: address already in use

ERROR: for nodered Cannot start service nodered: driver failed programming external connectivity on endpoint nodered (9be1530fa5af3968a8cd3b7716ecad1980ef5745d89fdeef71baa0f9e77f951e): Error starting userland proxy: listen tcp4 10.0.0.51:80: bind: address already in use
ERROR: Encountered errors while bringing up the project.

Oh, my bad.

Update:
I removed the text I wrote earlier in this post as I got things all wrong. I should not respond to posts when I am already/still tired. I missed that you used a different host port for both published port on each service. It was already correct.

The only problem you have is that actually another host process or published container already binds that port. You need to figure out what actually binds the port and need to restrict it, to only bind to the other ips, or publish the container port to a different host port.

So reusing the same port of the host is not possible even if it belongs to a different IP of the host?

Binding the same port on the same ip more than once is not possible. Binding the same port on a different ip is possible. But whenever you only map {host port}:{container port} it will bind the host port to 0.0.0.0 and therefor to all ip’s the host has.

This is true for all processes that bind 0.0.0.0:{host port}, regardless whether it is a native process or a containerized process.

OK, I think understand. The details of what happens in the background with 0.0.0.0 are a bit out of reach for me, but that’s OK.

Is there any way to avoid binding to 0.0.0.0?

For containers: {ip to bind};{host port}:{container port}

For any other program running on the host: I am afraid you will have to read the documentation and find out how it’s done, as this is highly application specific.

A small typo :slight_smile:

That is actually:
{ip to bind}:{host port}:{container port}

I “cheated” in the quote to see the difference (semicolon)

Well spotted, thanks!