Containers becoming increasingly slow under heavy network load

General Discussions
1

Issue Type:

I have ~20 containers running on RHEL 7.6. The run fine when doing initial tests but once moving content over the network or downloading content the containers start to freeze and my /var/log/messages log gets filled with these errors:

Mar  7 16:39:20 rheldocker NetworkManager[3312]: <info>  [1551994760.8911] manager: (vethbbeb797): new Veth device (/org/freedesktop/NetworkManager/Devices/30172)
Mar  7 16:39:20 rheldocker kernel: br-fdb1920a25f4: port 3(veth582d45d) entered disabled state
Mar  7 16:39:20 rheldocker kernel: device veth582d45d left promiscuous mode
Mar  7 16:39:20 rheldocker kernel: br-fdb1920a25f4: port 3(veth582d45d) entered disabled state
Mar  7 16:39:20 rheldocker NetworkManager[3312]: <info>  [1551994760.9130] device (veth582d45d): released from master device br-fdb1920a25f4
Mar  7 16:39:26 rheldocker systemd-logind: New session 175 of user root.
Mar  7 16:39:26 rheldocker systemd: Started Session 175 of user root.
Mar  7 16:40:20 rheldocker kernel: br-fdb1920a25f4: port 3(veth056b36a) entered blocking state
Mar  7 16:40:20 rheldocker kernel: br-fdb1920a25f4: port 3(veth056b36a) entered disabled state
Mar  7 16:40:20 rheldocker kernel: device veth056b36a entered promiscuous mode
Mar  7 16:40:20 rheldocker kernel: IPv6: ADDRCONF(NETDEV_UP): veth056b36a: 
link is not ready
Mar  7 16:40:20 rheldocker kernel: br-fdb1920a25f4: port 3(veth056b36a) entered blocking state
Mar  7 16:40:20 rheldocker kernel: br-fdb1920a25f4: port 3(veth056b36a) entered forwarding state
Mar  7 16:40:20 rheldocker kernel: br-fdb1920a25f4: port 3(veth056b36a) entered disabled state
Mar  7 16:40:20 rheldocker NetworkManager[3312]: <info>  [1551994820.8585] manager: (veth4c7b4c1): new Veth device (/org/freedesktop/NetworkManager/Devices/30173)
Mar  7 16:40:20 rheldocker NetworkManager[3312]: <info>  [1551994820.8606] manager: (veth056b36a): new Veth device (/org/freedesktop/NetworkManager/Devices/30174)
Mar  7 16:40:20 rheldocker containerd: time="2019-03-07T16:40:20.906144369-05:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/6215da439c5c246d82a4af5e9eb30e87cb90781b95f14ed49db174b037dada44/shim.sock" debug=false pid=4465
Mar  7 16:40:21 rheldocker kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Mar  7 16:40:21 rheldocker kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Mar  7 16:40:21 rheldocker kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth056b36a: link becomes ready
Mar  7 16:40:21 rheldocker kernel: br-fdb1920a25f4: port 3(veth056b36a) entered blocking state
Mar  7 16:40:21 rheldocker kernel: br-fdb1920a25f4: port 3(veth056b36a) entered forwarding state
Mar  7 16:40:21 rheldocker NetworkManager[3312]: <info>  [1551994821.4381] device (veth056b36a): carrier: link connected
Mar  7 16:40:22 rheldocker containerd: time="2019-03-07T16:40:22.071976690-05:00" level=info msg="shim reaped" id=6215da439c5c246d82a4af5e9eb30e87cb90781b95f14ed49db174b037dada44
Mar  7 16:40:22 rheldocker dockerd: time="2019-03-07T16:40:22.081980682-05:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Mar  7 16:40:22 rheldocker NetworkManager[3312]: <info>  [1551994822.1086] manager: (veth4c7b4c1): new Veth device (/org/freedesktop/NetworkManager/Devices/30175)
Mar  7 16:40:22 rheldocker kernel: br-fdb1920a25f4: port 3(veth056b36a) entered disabled state
Mar  7 16:40:22 rheldocker kernel: br-fdb1920a25f4: port 3(veth056b36a) entered disabled state
Mar  7 16:40:22 rheldocker kernel: device veth056b36a left promiscuous mode
Mar  7 16:40:22 rheldocker kernel: br-fdb1920a25f4: port 3(veth056b36a) entered disabled state
Mar  7 16:40:22 rheldocker NetworkManager[3312]: <info>  [1551994822.1321] device (veth056b36a): released from master device br-fdb1920a25f4

I’ve confirmed that the firewall is off but not sure what next steps to take…

2

Any ideas on what could be causing docker to stop networking like this? It is causing issues when moving files within my docker containers.

3

Did you check all your interfaces are working well… issue with br- ports are not stable

4

It’s a virtual interface. This is a VM, not seeing any issues on the OS level and am able to do normal file execution just fine.

5

Does anyone have any ideas on what I should be looking for here? This is still happening constantly, even though I have no use for IPv6…