I have a container stack that is started with docker compose. This stack communicates with a remote server to announce its availability to supply data. I’m having trouble with a network communication issue.
The following results in no communication to the remote server:
docker-compose up
– (good communication)
docker-compose down
docker-compose up
– (no communication)
The following reliably fixes it:
docker-compose up
– (good communication)
docker-compose down
Restart Docker Desktop via GUI
docker-compose up
– (good communication)
Then there’s this:
docker-compose up
– (good communication)
docker-compose restart
– (good communication)
docker-compose down
docker-compose up
– (no communication)
Restart docker app while containers are running
– (good communication)
Does anyone know what might cause the daemon restart requirement to reestablish communication to the remote server after the containers are stopped and started again in the same session?
There are apparently no external issues, such as firewalls, port forwarding, pass-through, etc. Machine is directly wired into the router with no other machines on the local network.
We have several developers across the country using Docker Desktop and they all report the same communication issue, which is reliably resolved by restarting the daemon.
I’ve attempted to clean the environment by removing dangling data, clearing networks, removing volumes, even pruning all docker data, etc. Nothing seems to work except a full app restart.
Is there a way to determine if the connection request from the containers is being blocked by docker versus docker simply not communicating over the network at all?
edit 1: Not sure if this is useful, but if I bash into the running container and kill the process, which forces the process to restart because of the restart rule in the docker-compose file, communication with the remote server is severed and the daemon must be restarted to regain communication. However, since the daemon restart is done with the containers running and the process unchanged, it would seem the issue has to exist in the docker process/environment, not the container (?).
What exactly do you mean by communication? How are you trying to access the port from the host?
I’m actually not sure, but I think the restart button restarts the whole virtual machine in which the Docker daemon is running. Since the daemon is inside the VM, Docker Desktop has to forward the host port into the virtual machine from there to the container. For me it seems to be more likely that the problem is caused by that special way of forwarding.
You could try to run a container on the host network which is the host network of the virtual machine and try curl in the container on the host network. Note that the host IP would not work there so you would either need to try the VM ip or localhost.
If it turns out to be a bug, you can report it on GitHub
Regarding the docker-compose vs docker compose question mentioned by @deanayalon, it is indeed usually true for Docker CE, but with Docker Desktop the docker-compose command is an alias to docker compose so you most likely have the Compose v2 already. It is still good to know about the difference so you use the right version on Linux as well. Which you will do if you install Docker CE from recent installation guides in the official documentation.
Thanks for your reply. You’re the second person in the last few hours to suggest it’s likely an issue at or in the WSL VM, so … time to start learning about that and running tests at that interface.
As for communication, when the local instance/container starts up, it uses 5121/udp to send a signal to the master server, which then communicates back on the same port looking for authentication. If the master server receives the correct authentication, the master server allows the local container to provide data to the server.
This setup works efficiently, effectively and quickly when the docker app instance is first started, just not after the local container is shut down, then started again without restarting the app. When I restart the app, both with or without live reload, the communication to the master server from the local container is re-established almost instantaneously. So I agree the issue has to be somewhere between the container and machine, and the WSL VM is the only interface left in between (I think).
I’ve attempted to to network_mode: host in the primary service in the docker-compose file, but that didn’t seem to have any effect. I’ll give you suggestion a shot about testing communication using the VM ip.
Thanks for the info about docker compose v docker-compose. I went ahead and changed our management scripts to docker compose just to be up to date, but as you suggested, there was no change as we were running the most current version of docker desktop.
Given that the few developers we have working on this project all report the same communication issue, you’d think this wouldn’t be an isolated incident and it’d be more widely reported/resolved, so I’m surprised there’s not more intel on this issue. Either that, or I’ve just massively screwed something up…
My suggestion was about running a new container on the host network so you can run curl in that container to test the port of the other container, assuming the local container uses an HTTP port. If not, you can test with netcat . The netshoot container probably contains everything you need and you can find examples for debugging in the description:
You mentioned a master server and communication in both ways. Docker Desktop is best for local development, when everything can be found locally except of course some repositories or APIs. So your use case could be something for whch Docker Desktop is not widely used. Maybe I just didn’t fully understand your description. It doesn’t matter, since it seems you had a working communication until you deleted the containers.
I emphesized “deleted” as you used the word “stopped”, but docker compsoe down deletes containers and all the networks that belong to the compose project. It just makes the issue weirder as you have completely new containers every time.
Have you tried for example a simple HTTP server like nginx with a forwarded port? You could test if you can access the port from the host when you delete the container and recreate it using the same port. You could test it with different docker networks. The default bridge without compose and a default compose network . Based on the result you could give a better description when you report it on GiHub if necessary.
Thanks for all those suggestions. I’ll be off work for the next few days, but will run through at least all of those and whatever other suggestions I can think of when I get back.