TCP connections from docker container reset when removing unrelated secondary IP address

General
1

We have the following setup: Docker 1.8.3 on CentOs 7.1.

Each Docker container has an associated Virtual IP address that is created via “ip addr add” command outside of the docker container. The incoming traffic into the containers is mapped for the few specific ports via each container’s VIP

Test:

  1. go inside a docker container via “docker exec -ti name bash”
  2. ssh out from inside the docker container to some other host, and start “top” command to get some traffic going
  3. Do "ip addr delete <other container’s secondary IP>/32 <if_name>

This should not have any affect on the ssh session since the IP address that is being removed is not used in that ssh connection.

However, the ssh session gets killed every time.

The packet capture shows something on the host sending a TCP RST packet.

Any ideas of why is this happening?

Here is the iptables config:

ip addr

…
4: eno16780032: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:50:56:31:61:d9 brd ff:ff:ff:ff:ff:ff
inet 10.1.9.95/24 brd 10.1.9.255 scope global eno16780032
valid_lft forever preferred_lft forever
inet 10.1.9.45/32 brd 10.1.9.255 scope global eno16780032
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fe31:61d9/64 scope link
valid_lft forever preferred_lft forever
5: docker0@NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 02:42:7b:48:7c:74 brd ff:ff:ff:ff:ff:ff
inet 172.17.42.1/16 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:7bff:fe48:7c74/64 scope link
valid_lft forever preferred_lft forever
7: vethfc1800a@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP
link/ether 1a:43:03:fe:39:02 brd ff:ff:ff:ff:ff:ff
inet6 fe80::1843:3ff:fefe:3902/64 scope link
valid_lft forever preferred_lft forever

iptables -n -L -t nat -w

Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all – 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all – 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all – 172.17.0.0/16 0.0.0.0/0

Chain DOCKER (2 references)
target prot opt source destination
DNAT tcp – 0.0.0.0/0 10.1.9.45 tcp dpt:5432 to:172.17.0.1:5432
DNAT tcp – 0.0.0.0/0 10.1.9.45 tcp dpt:24225 to:172.17.0.1:24225
DNAT tcp – 0.0.0.0/0 10.1.9.45 tcp dpt:9939 to:172.17.0.1:9939
DNAT udp – 0.0.0.0/0 10.1.9.45 udp dpt:24225 to:172.17.0.1:24225

The IP address that is being added/removed via the following command:
ip addr add 10.1.9.96/32 dev eno16780032
ip addr delete 10.1.9.96/32 dev eno16780032