I’m using Docker Desktop for Mac and utilizing docker context to connect to remote servers via SSH. I then build and deploy services using Docker Compose. Everything seems to work normally, however, recently when I checked the server logs, I noticed that my machine was persistently making SSH connections to the remote server, connecting every few seconds.
root@aaaaaaaaa:~# tail -f /var/log/auth.log | grep Accepted
May 21 06:56:09 aaaaaaaa sshd[168869]: Accepted publickey for root from 111.111.111.111 port 57088 ssh2: ED25519 SHA256:xxxxxxxxxxxxxxxx
When I checked my local machine with pstree, I found that com.docker.build was constantly making SSH connections to the remote servers and executing the docker system dial-stdio command.
pstree -u $USER 06567
-+- 06567 tao com.docker.build
|--= 06609 tao ssh -- law-us docker system dial-stdio
|--= 06632 tao ssh -- shui-us docker system dial-stdio
|--= 06633 tao ssh -- law-us docker system dial-stdio
|--= 07561 tao ssh -- shui-us docker system dial-stdio
...
Interestingly, at this time, I am not even using the context of these servers. My current context is the local system. I’m wondering what could be causing this? How can I prevent these seemingly meaningless connection attempts?
This is the time when I have to say thank you just for asking a question It is starting to make sense. There is an old issue:
The last comment that I linked explains an error which is related to too many connections. dial-stdio is a non-documented argument which shouldn’t be called manually and it is proxying the standard io (stdin, stdout, stderr). It is probably required when you run a container and want to execcute a command in it remotely, but I agree, it shouldn’t be there when you don’t even use the context. If there is a good reason why it is running constantly, I don’t know that. My first thought was that it could be some kind of “keep-alived” mechanism, but there is nothing to keep alive in your case unles there is a process that is running and connecting to a remote containr even if you switched to another context in your current terminal session. Is it possible that you have a running process in the background and keeps the SSH connection to the remote context alive? Did you restart your machine or Docker Desktop recently?
I’ve just started to see those processes recently and my situation is almost the same as you. I have a few servers on my local network to which I have contexts created to connect to, but I’m not currently using any.
My environment is Win11 with Docker Desktop 4.19.0(106363), engine v23.0.5, compose v2.17.3 and I usually work through WSL2 and those servers are Ubuntu 20.04
I am seeing the same thing, observed when my system monitors flag very high process counts on the remote servers.
Configuration is Docker Desktop 4.20.0 on macOS 13.4. I have a couple of ssh-based contexts defined. Both of the remote machines are Ubuntu 22.04 running Docker 24.0.1 (one x86_64, the other aarch64).
On the remote machines, there are hundreds of these:
428883 ? Ssl 0:00 docker system dial-stdio
428942 ? Ss 0:00 sshd: root@notty
I’m talking about 250 of each of those processes, on each target machine.
This is observed this morning after I left Docker Desktop running overnight. There was one container running, but no builds have been performed since starting the application.
Terminating Docker Desktop causes all of the local (obviously) and remote processes to end; the remote process count on one machine dropped from 722 to 175.
Restarted Docker Desktop and it immediately started opening new ones:
It seems content with those 10 processes for now (although I don’t know why it would need to open even those) but I imagine something triggers additional ones being started later on.
Using Docker Desktop 4.20.1 (110738) on macOS 13.4, and I also have this issue. The passphrase for my SSH key is locked behind touchID, and it gets extremely spammy when Docker keeps trying to make SSH connections.
Same here with macOS 12.6 and Docker Desktop 4.21.1 (114176). Number of connections is steadily increasing. to some thousands on remote server. First I was thinking of a memory leak as pages get “eaten up” until oomkiller gets on duty.
From stat I assume first occurrence around end of june with older version of Docker Desktop. My key is protected by passphrase but added to ssh-agent once I commit a manual deployment. Overthinking that now.
Neither target host nor ssh account or ssh key are related to any running container or any docker build so I get no idea why those got picked and start opening connections.
Any hints? Anyone?
Edit:
After submitting I came up with an idea: The endpoint was configured as “DOCKER ENDPOINT” in one of my contexts docker context ls. That explains why those connections could get chosen, but not why docker is trying to connect (and keeping connections open).
I’m seeing the same thing on macOS 13.5, Docker Desktop 4.21.1. I only noticed it when I put the SSH keys for our remote system into 1Password’s SSH agent, as now I’m prompted to manually authorize it every time the key is accessed. Presumably this problem has been happening for a while as this key used to be loaded into a normal, non-touch-ID agent.
It started happening after I used the context requiring this key for a manual action, but the Docker process has been for access to this key every fifteen minutes or so ever since, without me running anything. I can see some SSH failures in Docker’s logs that for docker system dial-stdio but oddly nothing since I started getting these auth prompts.
I have a hunch as to what’s happening here. If I run docker buildx ls it takes ages and prints lots of errors, and triggers the prompt to get access to my SSH key. As far as I can tell from the output, it’s trying to connect to a buildx instance on every node I have a Docker Context set up to connect to. It errors on most of them because they’re old contexts I haven’t tidied up, or because it can’t get the relevant SSH key.
I think this is what’s been happening in the background - Docker has set up a remote buildx instance and is trying to check on it by making an SSH connection. I think this is technically documented here but I think it’s really surprising behaviour. I use Contexts to make it easier to run docker commands ad-hoc on remote hosts, I don’t expect Docker to start maintaining long-running processes on those hosts and I certainly don’t expect it to start connecting them in the background without me initiating any action.
Ooh. Interesting - thanks. I’ve been having issues with one of my remote machines running out of resources, and noticed all the docker system dial-stdio entries.
My local ssh on my Mac was configured with UseKeychain yes, which I’m guessing might have the same issues as your 1Password setup when it comes to passwords for the keys. Perhaps Docker understands ssh-agent but nothing else…
Have removed that option for now and will watch with interest…
I can confirm that this seems to have solved the problem. Not using the keychain is a workaround until they stop Docker from trying to make the unnecessary connections in the first place!
Am having a related problem - the constant (failed) attempts to the remote server is causing my account to be locked out as it thinks it is an attack. Only way to fix was to remove the context - or shutdown Docker Desktop totally - neither very good options.
Eight months later… is there any improvement on this? Kamal deploy is now built on this. It is killing the remote build host by making every developer machine that deploys create orphaned docker system dial-stdio processes every few minutes until the host goes down. Plus a corresponding ssh -o ConnectTimeout=30 -l [user] -- [IP] docker system dial-stdio process on the client.
It is starting to make sense. There is an old issue: