Hola amigos,
Just like many people before me, I had this error popping up sometimes as well. What seemed really really strange to me is that the issue was non-reproducible, it occurred in ~60% of my pipelines, always at various different points in my build.
After thorough investigation, I found a line that drew my attention in the journals of my server:
May 19 12:11:37 xxxxxxxxx sshd[1995778]: error: beginning MaxStartups throttling
After some further investigation I found a related line:
May 19 12:12:19 xxxxxxxxx sshd[1995778]: exited MaxStartups throttling after 00:00:42, 2 connections dropped
Turns out that, instead of creating one ssh connection, docker-compose actually can create dozens of them, certainly when your compose file is getting bigger.
This in its turn triggered the ssh rate limiting that you can configure in your /etc/ssh/sshd_config file:
MaxStartups
Specifies the maximum number of **concurrent unauthenticated con-
nections to the SSH daemon.** Additional connections will be
dropped until authentication succeeds or the LoginGraceTime
expires for a connection. The default is 10:30:100.
Alternatively, random early drop can be enabled by specifying the
three colon separated values "start:rate:full" (e.g.
"10:30:60"). sshd(8) will refuse connection attempts with a
probability of "rate/100" (30%) if there are currently
"start" (10) unauthenticated connections. The probability
increases linearly and all connection attempts are refused if the
number of unauthenticated connections reaches "full" (60).
Increasing my allowed connections (MaxStartups) in my sshd config file (/etc/ssh/sshd_config) fixed the issue for once and for all. I personallly opted to disable the unpredictable rate limiting as well, because I don’t like unpredictable systems:
MaxStartups 100:0:100
Don’t forget to restart your ssh service afterwards:
systemctl restart ssh sshd
Since this was quite obscure to find, I thought I might as well share this finding to you guys. Hopefully it may help someone some day.
All the best
- P