Container with dedicated internal IP accessible from entire network

Hello,

I am trying to set a static IP address inside a docker container (using a CentOS image with systemctl enabled).

The reason for this is, that I want to run a service inside a container which requires ssh and checks if a NIC with the IP it should listen on is available. The service communicates with other services in the same network so the IP needs to be accessible from outside the container as well.

It potentially could work with --net=host, but I could never figure how I would bind a range of ports to the container with that option. Using -p 10.2.0.253:22:22 does not work. When using e.g. SSH from any physical node on that network, I always end up on the docker host instead inside the container.

  1. The network is 10.2.0.0/24.
  2. The docker host has the IP 10.2.0.100
  3. Container should listen to all ports on 10.2.0.253 and most important, there must be an interface within the container with that IP
  4. The container should be reachable from any physical node or container on the 10.2.0.0/24 network

Is there any way make this work?

Greetings,
Bjoern

this is a challenging problem due to the way networking works in general.

the network adapter listens for traffic on its mac address.
a container on the same network would have a different mac address. and the host adapter would not be listening for traffic on the containers mac. you can (technically, but not practically) set the host adapter to promiscuous mode (is some environments) so that the host adapter can listen for traffic on ANY (also ALL) mac addresses… in almost all production environments this is a security violation

if you don’t want DHCP support for the container, its a ‘little’ easier…

we have been dicsussing this here
%How to set up a container just like a virtual machine in bridge mode? (meaning, the container gets its own external ip)

short net

  1. create a docker network in address range u want
  2. generate a mac for the container
  3. select an ip address for the container in the range of the network (using dhcp is a problem)
  4. on docker run set the network, mac and ip address for the container

but, at some point you will need promiscuous mode on the host to make this work.

it works on my physical box with ubuntu
it does NOT work on VMWARE instances on this same host with promiscuous mode on. (used to but haven’t figured it out)
it works on my virtual ubuntu under vmware on windows
it does NOT work on my virtual mac on the same vmware on windows
it does NOT work on docker on windows.

Thanks!

I don’t care about DHCP right now so I will give it a try and let you know if it worked.

Greetings,
Bjoern

Oh no…

Since my network is IP-over-IB it fails already by creating a macvlan :frowning:

[root@server40 docker]# ip link add dev ib1 link ib0 type macvlan
RTNETLINK answers: Invalid argument

Is there another way?

I’ll have a look at SRIOV…

Cheers,
Bjoern

the original poster (OP) uses a macvlan, mine just uses a Docker network

see post 3 (of 17 in the referenced topic)…

and u won’t need step 3 (busybox for dhcpclient)… as u know what address u want…

Thanks a lot.

I used this howto from Mellanox to use passthrough mode for now.
https://community.mellanox.com/docs/DOC-2899

It works fine, but still need to make GPFS start properly (mmstartup still fails)

Greetings,
Bjoern

Hej sdetweil,

I tried your way, and it partially works. I get a interface in my container with the right IP address. but I can’t ping any node on that network.

Host network:

[root@server40 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.2.50    0.0.0.0         UG    0      0        0 eno1
10.2.0.0        0.0.0.0         255.255.255.0   U     0      0        0 ib0
169.254.0.0     0.0.0.0         255.255.0.0     U     1004   0        0 eno1
169.254.0.0     0.0.0.0         255.255.0.0     U     1007   0        0 ib0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 eno1
192.168.110.0   0.0.0.0         255.255.255.0   U     0      0        0 enp131s0f1

Not sure what I shall use as “gateway” on the subnet 10.2.0.0/24 as there is none, so I used the hosts IP 10.2.0.100 (IP forwarding is enabled) or a unused IP address (didn’t make a difference):

docker network create -d macvlan --gateway=10.2.0.254 --subnet=10.2.0.0/24 mynet

Starting with:

docker run -it --privileged=true --cap-add=ALL --hostname=gsctos --net=mysnet --ip=10.2.0.253 --mac-address "02:68:b3:29:da:98" --name test gsctos

Inside the docker I see:

[root@gsctos /]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
19: eth0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 02:68:b3:29:da:98 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.2.0.253/24 scope global eth0
       valid_lft forever preferred_lft forever
[root@gsctos /]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.2.0.254      0.0.0.0         UG    0      0        0 eth0
10.2.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0

But I can’t ping any node on that network:

[root@gsctos /]# ping 10.2.0.1
PING 10.2.0.1 (10.2.0.1) 56(84) bytes of data.
From 10.2.0.253 icmp_seq=1 Destination Host Unreachable
From 10.2.0.253 icmp_seq=2 Destination Host Unreachable

Any suggestions?

Thanks a lot,
Bjoern

yeh, without a gateway to handle resolving mac address locations, its hard for me to understand how to configure this.

without a gateway, I think you have to manually add entries to the arp table of all the systems

for exmple

arp -s 157.55.85.212   00-aa-00-62-c6-09

in my situation i have a router on my network that provides this function.

I see…

Trying ARP but it fails:

arp -s 10.2.0.253 02:68:b3:29:da:98
SIOCSARP: Invalid argument

even when using “-i” to specify the interface.

Could I setup a router on the docker host?

Cheers,
Bjoern

notice the help for arp says the mac address uses ‘-’ to separate the parts, not ‘:’

Hej,

not in my case. “man arp”:

arp -s address hw_addr is used to set up a new table entry. The format of the hw_addr parameter is dependent on the hardware class, but for most classes one can assume that the usual presentation can be used. For the Ethernet class, this is 6 bytes in
hexadecimal, separated by colons. When adding proxy arp entries (that is those with the publish flag set) a netmask may be specified to proxy arp for entire subnets. This is not good practice, but is supported by older kernels because it can be useful. If
the temp flag is not supplied entries will be permanent stored into the ARP cache. To simplify setting up entries for one of your own network interfaces, you can use the arp -Ds address ifname form. In that case the hardware address is taken from the
interface with the specified name.

arp -s 10.2.0.253 02-68-b3-29-da-98
arp: invalid hardware address

Greetings,
Bjoern

Maybe I have to use infiniBand as type and an according address…

similar to

c06-ib (10.2.0.26) at a0:00:00:68:fe:80:00:00:00 [infiniband] on ib0

Indeed.

[root@server40 ~]# arp -v -H infiniband -s 10.2.0.253 a0:00:00:e0:db:55:ce:13:f1
arp: SIOCSARP()
[root@server40 ~]# arp -a -n | grep 253
? (10.2.0.253) at a0:00:00:e0:db:55:ce:13:f1 [infiniband] PERM on ib0

But this does not work with ‘–mac-address “a0:00:00:e0:db:55:ce:13:f1”’ when starting the container :frowning:

I think I have to go back and try SR-IOV