Tcp connection to any port fails with "connection reset by peer"

Hey,
I’ve been trying to find a solution to this one for the past 3-4 hours and have failed in spite of all the efforts and research, so here we go:

after having several issues getting docker to work on a project I’m working on, I tried making a simple set up to test out a simple Go HTTP server:

package main

import "net/http"

func main() {
	println("Starting up...")
	http.ListenAndServe("0.0.0.0:8123", http.HandlerFunc(fn)) // tried also 127.0.0.1:8123, :8123.
}

func fn(w http.ResponseWriter, _ *http.Request) {
	w.Write([]byte("It works!"))
}

Similarly, the dockerfile is also straightforward:

FROM golang:latest

RUN apt update && apt install dumb-init
WORKDIR /opt/srv
COPY . .
RUN go build -o server .
EXPOSE 8123
CMD ["dumb-init", "./server"]

I then build the container with docker build -t test-docker ., and run it with docker run -p 8123:8123 test-docker. Attempting to connect to 8123 with curl, lynx always fails, and netcat shows how this fails following the first packet:

asciicast

To exclude any Go-specific issues, I attempted the same with a simple Python server:

docker run -p 8000:8000 -it python:3.7-slim python3 -m http.server --bind 0.0.0.0

The same exact issue still occurs.

Within the containers, the servers are also reachable. Outside, however, they are not.

I attempted many things, but essentially this issue persists in spite of an apt purge, rm -rf /var/lib/docker, removing $HOME/.docker/buildx, rm -rf /var/lib/containerd, removing the docker rules from iptables, removing the network interface docker0 and then finally restarting my entire machine and running this test with root, so I’m starting to think it may not actually be a configuration issue.

docker-proxy seems to be normally set up (I see this entry in ps aux):

/usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8123 -container-ip 172.17.0.2 -container-port 8123

The issue seems to persist also when connecting between containers. When I tried this on the larger project I was working on, this resulted in the connection just hanging indefinitely, though I haven’t confirmed this in a stripped-down setup.

Setting --network host works fine, and “solves” the issue, but I would hardly call that “solving it”.

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 23.10
Release:	23.10
Codename:	mantic
$ docker version
Client: Docker Engine - Community
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:14:22 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:22 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
$ sudo apt list --installed 'docker*'
Listing... Done
docker-buildx-plugin/mantic,now 0.12.1-1~ubuntu.23.10~mantic amd64 [installed]
docker-ce-cli/mantic,now 5:25.0.3-1~ubuntu.23.10~mantic amd64 [installed]
docker-ce-rootless-extras/mantic,now 5:25.0.3-1~ubuntu.23.10~mantic amd64 [installed,automatic]
docker-ce/mantic,now 5:25.0.3-1~ubuntu.23.10~mantic amd64 [installed]
docker-compose-plugin/mantic,now 2.24.5-1~ubuntu.23.10~mantic amd64 [installed]

I am using the apt repositories provided by docker (https://download.docker.com/linux/ubuntu mantic stable ).

Previous (unanswered) literature: Docker container connection reset by peer / Networking, Server, and Protection / Arch Linux Forums