Docker breaks SSH inbound (already changed bridge network default)

Docker Engine General
1

Hi there, I spin up Ubuntu VMs as EC2 instances and like SSHing into them for dev. Commands for installation are below:

sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker ubuntu

I spin up docker containers typically for services like Supabase or Clickhouse/MinIO. Whenever I do, my SSH connection drops. I tried solving the problem first and found on other sources that it was due to conflicting subnets?

So I edited my daemon.json to the following, but to no avail:

{
  "default-address-pools": [
    {
      "base": "10.200.0.0/16",
      "size": 24
    }
  ],
  "bip": "10.200.1.1/24"
}

I’ve verified that the VM private IP is in a different subnet. Unfortunately a little lost when it comes to networking concepts like this, would appreciate help!

I can offer more info of environment if I can get instructions on how to find/look.

2

You need to restart the docker engine + delete old existing networks.

What’s the subnet range in the az’s of your vpc where docker is running?

3

Hey! I did restart with sudo systemctl restart docker, and did prune.

  • VPC CIDR: 172.31.0.0/16
  • Subnet CIDR: 172.31.0.0/20
  • Instance IP: 172.31.0.236
4

This is the defail address pool:

{
  "default-address-pools": [
    { "base": "172.17.0.0/16", "size": 16 },
    { "base": "172.18.0.0/16", "size": 16 },
    { "base": "172.19.0.0/16", "size": 16 },
    { "base": "172.20.0.0/14", "size": 16 },
    { "base": "172.24.0.0/14", "size": 16 },
    { "base": "172.28.0.0/14", "size": 16 },
    { "base": "192.168.0.0/16", "size": 20 }
  ]

}

You subnet cidr indeed collides with { "base": "172.28.0.0/14", "size": 16 }.
It shouldn’t have been a problem right away, but eventually docker would have used up all subnets from the other address pools, until it comes to a subnet that collides with your vpc cidr.

A prune doesn’t help for the default bridge, and doesn’t help if containers are attached to the network. Please run a docker inspect with all existing networks, and remove those that are not within your new address pool. If I am not mistaken you would also need to remove the docker0 interface before restarting the docker engine (or the compute node itself) .