Application in docker container cannot connect to other container in another server, have to restart docker service to restore the connection

Docker Engine General
,
1

I have a server A (application server) using docker service, and application inside running in docker container. I also have server B and C (database server primary and secondary) running in container as well.

Everything runs normally til one day, the application log says cannot connect to database, and both of the database (primary and secondary) cannot communicate with each other. I tried restart container/application, the connection still dead, so I have to restart the docker service of every server (A,B,C) so the connection restore back to normal.

I have checked with the network and firewall, and found that they are normal, nothing blocks the communication between them, so the problem should not be in this part.

I met this problem for several times in another environment, and all I have to do is restarting docker service.

I would like to know how to fix this and the root cause, or are there anyone met this problem and has the better work around than restarting docker service?

here is the version that I use:
Client: Docker Engine - Community
Version: 27.3.1
API version: 1.47
Go version: go1.22.7
Git commit: ce12230
Built: Fri Sep 20 11:42:37 2024
OS/Arch: linux/amd64
Context: default

Server: Docker Engine - Community
Engine:
Version: 27.3.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.7
Git commit: 41ca978
Built: Fri Sep 20 11:41:36 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.22
GitCommit: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
runc:
Version: 1.1.14
GitCommit: v1.1.14-0-g2c9f560
docker-init:
Version: 0.19.0
GitCommit: de40ad0

Thanks.

2

It’s a server with Docker and container each? Which OS? Are they virtualized?

How are containers connected? Using Swarm? Docker overlay networks? Publish ports?

Have you tried updating to Docker 27.5? (Maybe wait with latest 28.0 as there seems to be some networking issue)

3

  • Yes, its a redhat 8.9, 8.10 server with docker service and container running inside

  • Communication under the same docker service (same server) its Docker overlay network, but communicating across the server (container A in Server A to Container B in Server B) using Publish ports.

  • haven’t try that yet, my current version is 27.3.1

By the way, thanks for your reply and the version information

Thank you