Netstat shows thousands connections between bridge and container (looked like DoS attack)

General Discussions General
1

Issue Type: Potential docker network issue or potential loop
Host: Debian 10 (buster) VM on vSphere7
**Docker:**20.10.17
Containers:

  • tomcat:9.0.60-jdk17-openjdk-slim-buster
  • postgres:13.4 (debian:buster)

The Situation:
Recently ran into an issue where the network was reporting very high load (>5Gbps+)
While troubleshooting I found netstat was reporting several thousand HTTPS connections between the bridge and the container VIF - ESTABLISHED and CLOSE_WAIT.

There were NOT enough inbound connections to the host to account for more than ~1000 connections per minute.

Software stack is simple - webapp running in tomcat, data stored in postgres.
Port 443 is configured in Tomcat container and exposed on the host.

The Strange:

  1. Containers were destroyed and recreated multiple times using docker-compose, which claimed to remove the network.
  2. Host VM was rebooted to add more CPU to handle the load (under the assumption Tomcat servlet was falling behind due to processing incoming data)
  3. Issue continued to happen.
  4. Accidentally jacked up iptables while attempting to create rules to block all, but a single in-bound connection to the host. This effectively broke docker networking.
  5. Cleared all firewall rules.
  6. Restarted docker systemctl <stop/start> docker
  7. After docker daemon restart, network resumed normal operation. Inbound data <200Mbps

The Questions:

  1. Has anyone seen similar behavior before?
  2. What’s are the best resources and/or tools to troubleshoot docker network issues?