Docker Compose through SSH failing and referring to docker.example.com

Hello!
I’m done with configuring Dockerfile and Docker Compose for my deployment and whenever I test it on the local machine it is perform cool - everything loads properly.

But when I try to deploy using ssh, I’m getting this very strange error:

error during connect: Post "http://docker.example.com/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=project_web&target=&ulimits=null&version=1": write |1: broken pipe

Why does it try to post on docker.example.com? Where is my mistake?
Docker-compose and docker are running properly on remote machine… So how to get rid of this strange exception?

What command are you using for that?

I’m using this: docker-compose up -d
I’ve added DOCKER_HOST variable to my environment variables, so it uses needed machine as I can see.

Just to be 100% sure, for others who may be able to help: so you’re running docker-compose up -d on your local machine to start the services on the remote machine? And all you’ve configured is DOCKER_HOST? Which, I assume, does not refer to example.com, and the error message is really mentioning that domain?

Yus, exactly as you’ve said!

I am having a similar problem, but with setting a context. I have a Virtualbox VM (Ubuntu 20) running on my Mac. In the VM I have Docker installed rootlessly. The VM is called vmname and the user is username. Virtualbox exposes the VM port 22 on Mac port 3022. Docker client and server are both version 20.10.10.

I have this in my ~/.ssh/config file:

Host vmname
  HostName localhost
  User username
  Port 3022

So ssh works correctly:

mac$ ssh vmname

works correctly and gives me the vm console prompt.

vmname$ docker info

works correctly, giving me client and server info. Going back to the Mac,

mac$ unset DOCKER_HOST
mac$ docker context create vmname --docker "host=ssh://vmname"
mac$ docker context use vmname

seems to work correctly, because:

mac$ docker context ls
NAME                   TYPE          DOCKER ENDPOINT
vmname *             moby          ssh://vmname

But here’s the problem:

mac$ docker info
Client:
 Context:    vmname
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.6.3)
  compose: Docker Compose (Docker Inc., v2.1.1)
  scan: Docker Scan (Docker Inc., 0.9.0)

Server:
ERROR: Cannot connect to the Docker daemon at http://docker.example.com. Is the docker daemon running?
errors pretty printing info

There seems to be a clue on Github in docker/engine/client/errors.go; it looks as though some doRequest() invocation is confused about the host:

// Error returns a string representation of an errConnectionFailed
func (err errConnectionFailed) Error() string {
	if err.host == "" {
		return "Cannot connect to the Docker daemon. Is the docker daemon running on this host?"
	}
	return fmt.Sprintf("Cannot connect to the Docker daemon at %s. Is the docker daemon running?", err.host)
}

If I power down the VM, I get a different error, which seems to imply that when the VM is running the ssh connection is getting through, but the Docker server is getting confused?

mac$ docker info
Client:
  <same>
Server:
ERROR: error during connect: Get "http://docker.example.com/v1.24/info": command [ssh -- vmname docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ssh: connect to host localhost port 3022: Connection refused

I can’t find any other reports of this problem on Google, so I assume I am doing something very stupid. Any idea what?

1 Like

OK, as I suspected, I was doing something stupid. For rootless operation, I needed to set the DOCKER_HOST environment variable globally on the Docker server.

I added the following line to /etc/environment and restarted the server:

DOCKER_HOST=unix:///run/user/1000/docker.sock

and everything works great now.

** UPDATE ** see below - as Ákos Takács points out, you don’t need to set DOCKER_HOST on the server for rootless mode to work. You just have to docker context use rootless on the server if it is not the default context.

I didn’t set DOCKER_HOST and it worked for me switching to the rootless context:

docker context use rootless

If it doesn’t work check the available contexts first

docker context ls

* in the output means that context is used

But if DOCKER_HOST works for you, that is also good :slight_smile: Keep in mind though, when you create additional contexts you won’t be able to switch between them until you delete the variable.

OK, it is a confusion between the context on the server and the context on the client.

Thanks Ákos, you are right - I don’t need to set DOCKER_HOST on any machine when in rootless mode.

On my local development system I can use docker context use <name> to switch between target remote servers.

On every remote server I use docker context use rootless

The ‘rootless’ context is installed on the server by the dockerd-rootless-setuptool.sh script.

I was getting a similar error using a Mac with macOS Monterrey:

Server:
ERROR: error during connect: Get "http://docker.example.com/v1.24/info": command [ssh -l myremoteuser -- REMOTE_IP docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=Permission denied, please try again.

previously, I generated and installed the public RSA key via the commands:

ssh-keygen
ssh-copy-id myremoteuser@REMOTE_IP

and using the ssh myremoteuser@REMOTE_IP command, it asked for the passphrase and connected without problems, also my remote user was able to run docker commands.

The problem was when I tried to connect using a docker context created with:

docker context create \
    --docker host=ssh://myremoteuser@REMOTE_IP \
    --description="Remote engine at REMOTE_IP" \
    my-remote-engine

then I selected the context with docker context use my-remote-engine and when I tried to run docker info, I got the Permission denied error, but it did not ask for the passphrase.

So, it seems that docker does not ask for the passphrase and we need to use an ssh-agent to load the key and docker can use it. That will be done with the command:

ssh-add

just doing that let me run the docker info command without problems, additionally to persist this behavior we need to add the following lines to our ssh config file ~/.ssh/config

Host *
   AddKeysToAgent yes
   UseKeychain yes

More details at:

4 Likes

So, was anyone able to figure out where the name “docker.example.com” pops out from? Seems to be a mystery…

Do you mean where that default setting is coming from? It is part of the Docker CLI

Thank you for pointing this out.

It still does not make sense to me… Why would it try to connect to this non-existing URL?..

A stranger thing is that for me, the error happens intermittently when I try to do compose down on a remote host via docker context. And it happens in the middle of the process, so some containers are stopped and deleted, others remain.

If something was misconfigured, it would not work at all, right? But it works sometimes, and sometimes it does not.

Indeed it seems strange if only some of the containers are failing to stop so In your case it can also be a bug., but without knowing your configuration I can’t tell you more. I think I have seen similar messages when I tried Docker in Docker incorrectly so that was not about SSH and probably not the same domain. I don’t use ssh context with Docker Compose, so I have never really needed to think about it. I prefer to use Ansible or working from the server so I don’t remove a container on a remote server acidentally.

I don’t have all the answers yet. I would need to do a little bit more research, but I wouldn’t say that it “tries to connect to”. Since it is an “example” domain It is some kind of default domain, although I don’t know why it is needed for the SSH connection.

Reading the sourcecode of the Docker CLI it appears in one file two times but according to GoLand, multiple functions call refers to that connection helper and only ona is used in load.go. So it could be something inherited from an older version or it is called somewhere in a way that I can’t find it.

What is your Docker version and host operating system? Can you please share the output of the following commands?

docker info
docker version

and how exactly you installed Docker?

If it can be any use, here is the exact error message I get:

error during connect: Delete "http://docker.example.com/v1.42/containers/35b6f4271551a29ee4213fd8dbe9aa0239ae100bf9c74de2b780555918e134d8?force=1&v=1\": command [ssh -l xxxxxxxxx.xxx.xxx.xxx docker system dial-stdio] has exited with exit status 1, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=

(I masked user name and IP address with xxxxxxx)

Docker version on target machine:

Client: Docker Engine - Community
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        a5ee5b1
 Built:             Thu Feb  9 19:47:01 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       bc3805a
  Built:            Thu Feb  9 19:47:01 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.18
  GitCommit:        2456e983eb9e37e47538f59ea18f2043c9a73640
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info on target machine:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.16.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 8
  Running: 5
  Paused: 0
  Stopped: 3
 Images: 25
 Server Version: 23.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-41-generic
 Operating System: Ubuntu 22.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 15.41GiB
 Name: *xxxxxxxxxx*
 ID: a8964b45-099b-480a-88f8-f4c1292b2f7b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  *xxx.xxx.xxx.xxx:xxxx*
  *xxxxxxxxxx:xxxx*
  127.0.0.0/8
 Live Restore Enabled: false

Docker version on my workstation (where I run the command from):

Client:
 Cloud integration: v1.0.29
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:08:16 2022
 OS/Arch:           windows/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.15.0 (93002)
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 18:00:19 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.10
  GitCommit:        770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info on my workstation:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.9.1)
  compose: Docker Compose (Docker Inc., v2.13.0)
  dev: Docker Dev Environments (Docker Inc., v0.0.5)
  extension: Manages Docker extensions (Docker Inc., v0.2.16)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.22.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 42
 Server Version: 20.10.21
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.102.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 12.36GiB
 Name: docker-desktop
 ID: GMAT:ZHV5:2XGJ:WNFN:4D5L:HXBH:OKUT:GTEW:D7BU:5RPL:3IEW:7WAM
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  *xx.xx.xx.xx:xxxx*
  *xx.xx.xx.xx:xxxx*
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

My workstation is a Windows machine and has Docker Desktop installed. The target machine is a Linux VM, not sure how exactly Docker was installed there, but I suppose a standard way.

Thank you for the command outputs. Everything seems right. If I remember correctly in case someone installs Docker from the wrong repository, the version number has a suffix, but you can make sure that it is correct, by running the following commands too on the target machine:

dpkg -l | grep docker
snap list docker

I still don’t know what the reason of your error could be, but one difference is that I am using macOS, not Windows. Since the error message should always be shown or never, I thnk you should report it on GitHub.

I haven’t find any issues mentioning the domain name and it might have nothing to do with Docker Desktop, but you can start there.

This issue has not been solved and it is breaking all CI/CD pipelines and causing millions of dollars in losses to my company.

Steps to reproduce:

-Install docker on a remote machine
-Set up docker context: docker context create remote --docker “host=ssh://root@remote_ip”
-docker context use remote
-Check it is connected and working properly, say, by running docker info or docker container ls
-Now run docker compose build. Result:

error during connect: Get “http://docker.example.com/v1.24/containers/buildx_buildkit_fsr/json”: command [ssh -l root – 164.92.116.180 docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ssh: connect to host 164.92.116.180 port 22: Connection refused

Hola amigos,

Just like many people before me, I had this error popping up sometimes as well. What seemed really really strange to me is that the issue was non-reproducible, it occurred in ~60% of my pipelines, always at various different points in my build.

After thorough investigation, I found a line that drew my attention in the journals of my server:

May 19 12:11:37 xxxxxxxxx sshd[1995778]: error: beginning MaxStartups throttling

After some further investigation I found a related line:

May 19 12:12:19 xxxxxxxxx sshd[1995778]: exited MaxStartups throttling after 00:00:42, 2 connections dropped

Turns out that, instead of creating one ssh connection, docker-compose actually can create dozens of them, certainly when your compose file is getting bigger.

This in its turn triggered the ssh rate limiting that you can configure in your /etc/ssh/sshd_config file:

MaxStartups
     Specifies the maximum number of **concurrent   unauthenticated con-
     nections to the SSH daemon.**  Additional connections will be
     dropped until authentication succeeds or the LoginGraceTime
     expires for a connection.  The default is 10:30:100.

     Alternatively, random early drop can be enabled by specifying the
     three colon separated values "start:rate:full" (e.g.
     "10:30:60").  sshd(8) will refuse connection attempts with a
     probability of "rate/100" (30%) if there are currently
     "start" (10) unauthenticated connections.  The probability
     increases linearly and all connection attempts are refused if the
     number of unauthenticated connections reaches "full" (60).

Increasing my allowed connections (MaxStartups) in my sshd config file (/etc/ssh/sshd_config) fixed the issue for once and for all. I personallly opted to disable the unpredictable rate limiting as well, because I don’t like unpredictable systems:

  MaxStartups 100:0:100

Don’t forget to restart your ssh service afterwards:

systemctl restart ssh sshd

Since this was quite obscure to find, I thought I might as well share this finding to you guys. Hopefully it may help someone some day.

All the best
- P

5 Likes

After investigating, I found that it’s caused by my host not having the remote host as part of my known hosts.

When I added the host key fingerprint as part of my known hosts it started working properly

1 Like

You are a legend, thank you. Problem fixed.

One nuance in my case: setting MaxStartups to 100:0:100 was not allowed for whatever reason. I set it to 99:1:100 and it worked. There are rumours that setting this option to single number (e.g. 100) also works.

Thanks.

1 Like