Can't login and pull on hub.docker.com

Hi,

First of all, I have to mention that I do not claim this is a bug with the servers at hub.docker.com. But it happens only with them. I’m only searching for ways to investigate the issue.

For a few days now, I have one machine that cannot pull images from the hub. Also, I cannot login on the web interface from this machine either. To be more precise concerning the login part: I can display the home page, I can display the login page too, I can see that the first POST request to /login/ returns a 200 with the token, but then the POST request to /attempt-login/ fails with a 502 after approx. 15 seconds.

It happens with all tested browsers: Firefox, Chromium, Midori, Vivaldi, and I also tested with cURL which gave me the following trace (command is from the “Copy as cURL” feature in Chromium):

$ curl -v 'https://hub.docker.com/attempt-login/' -H 'Referer: https://hub.docker.com/login/' -H 'Origin: https://hub.docker.com' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36' -H 'Content-Type: application/json' --data-binary '{"jwt":"eyJ4NWMiOlsiTUlJQytEQ0NBcCtnQXdJQkFnSUJBREFLQmdncWhrak9QUVFEQWpCR01VUXdRZ1lEVlFRREV6dFJORm96T2tkWE4wazZXRlJRUkRwSVRUUlJPazlVVkZnNk5rRkZRenBTVFRORE9rRlNNa002VDBZM056cENWa1ZCT2tsR1JVazZRMWsxU3pBZUZ3MHhOVEEzTURjeU1EUTFNakZhRncweE5qQTNNRFl5TURRMU1qRmFNRVl4UkRCQ0JnTlZCQU1UTzBkUFJVczZSbEZhVURwR1MwY3pPa0pTU2xnNldGbEtXanBRUzFKRE9sZEhNMWc2VDBWS1dqcFBWMGd5T2tNMFRFYzZSVlJSTXpwSU5UZENNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXVSVU1kb2hzR0JJcHIydEpLblE0M2ZkcW1jY3ptTTM5cGdMTkQxeSt3R0g4Vk5lMTV1djZLMWE0eUFFaXVtdTF3S0FtSG5HVmQ4MFJFSklWTW1hdmt5ZmJNdFpBYUYxRkp1UUxPNDhoM2dhaVh6b1JlS0k0ME5qQ3F5RVh3Q0REdXowM2RXWFB0ZWJIYVZCRFhLa1MrUFJKQjkxOWdZaXZ3TDVXVmRlXC9tXC8zN3VkaEt6ZVRmWG1kajBPVUhcL3ZSNWEzd2w5UWtJaVkzQlJWR3NwWmZjRWdsSnhXbzhPdUl4RHhKQUsyb0hEQXp2anNnUis0RXR1Y05yT0NZNEY5dktpWlQ3STZ5MnF6RGhWYjBtWHh1OEpqNFVYTW82a2RsV1NqMVwvOEtjWEVzaTFMTWo2OXpObkFYY2h4N1BoczJoV2Q3UUYzMEltSkxwVkNwYWcxdmVGVXdJREFRQUJvNEd5TUlHdk1BNEdBMVVkRHdFQlwvd1FFQXdJQWdEQVBCZ05WSFNVRUNEQUdCZ1JWSFNVQU1FUUdBMVVkRGdROUJEdEhUMFZMT2taUldsQTZSa3RITXpwQ1VrcFlPbGhaU2xvNlVFdFNRenBYUnpOWU9rOUZTbG82VDFkSU1qcERORXhIT2tWVVVUTTZTRFUzUWpCR0JnTlZIU01FUHpBOWdEdFJORm96T2tkWE4wazZXRlJRUkRwSVRUUlJPazlVVkZnNk5rRkZRenBTVFRORE9rRlNNa002VDBZM056cENWa1ZCT2tsR1JVazZRMWsxU3pBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlBbVRrN083XC83SGZIcENjRW1ZSnFYMk84XC9ra0g1blVudm45eEhFTXkzczJRSWdLaFF1ak14NjZKUmVMYkI2U2l4MUxWRzZnQVM2RGJ4dktqKzdtRFZxa1o4PSJdLCJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzZXNzaW9uX2lkIjoiMEY1RTM2NjYwRUJDMzQzN0YyRkUyNTNCRDVBQURDNjUiLCJ1c2VyX2lkIjoiN2UzMTJhMmI0YjU0NGYyMGI1NGE3MGY5NTAyMDE4OTkiLCJlbWFpbCI6IiIsImV4cCI6MTQ1MDk3ODU4MCwidXNlcm5hbWUiOiJuYW5hd2VsIn0.LfthtHa4U43xIwDOcmtgLPGJzDklFQa3tBb5vWVNgqZzLFWJT9gWBx5pyNYnSUdjeolVLaiuNJYnzDSdEu_QdNAduIca-f2fG2CVz4ORgmDeSKzyQP3wE1n812kSqcrTBMm-ukVPN3MsJaF-XCmxY7ibTms_qoavhnuC7YXsZQdxkiTqrLwDfGDkEOWV0BtCd8w8LmV9x1k9t0qeWuWH2rGrcOh16tWF73ONcHuq8eKfYlxV0j9_XbwgqS5xZZ_PgYSUsvx1uvB9cunlgNhcGErpxpG1IyJrabSHGByyORU-b8KwcMtrcl-Jh04u9lGyMUr1NbSQVMu8Odp6pRgUWA"}' --compressed
*   Trying 52.20.109.58...
* Connected to hub.docker.com (52.20.109.58) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* 	 subject: OU=GT68254618; OU=See www.rapidssl.com/resources/cps (c)13; OU=Domain Control Validated - RapidSSL(R); CN=*.docker.com
* 	 start date: Oct 29 21:14:42 2014 GMT
* 	 expire date: Nov 13 14:19:33 2017 GMT
* 	 subjectAltName: hub.docker.com matched
* 	 issuer: C=US; O=GeoTrust Inc.; CN=RapidSSL SHA256 CA - G3
* 	 SSL certificate verify ok.
> POST /attempt-login/ HTTP/1.1
> Host: hub.docker.com
> Accept: */*
> Accept-Encoding: deflate, gzip
> Referer: https://hub.docker.com/login/
> Origin: https://hub.docker.com
> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
> Content-Type: application/json
> Content-Length: 1965
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
* HTTP 1.0, assume close after body
< HTTP/1.0 502 Bad Gateway
< Cache-Control: no-cache
< Connection: close
< Content-Type: text/html
< 
<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>

* TLSv1.2 (IN), TLS alert, Client hello (1):
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):

(there’s a 15 seconds delay after the line * We are completely uploaded and fine)

What I see when I trace the packets on the network with Wireshark is like some of them were lost, but always at the same time in the process. I’m not an expert at all but it seems my machine is waiting for a response from the server that never comes. Then HAProxy sends a 502 (I suppose it’s HAProxy by the 502 message it returns, and I assume it’s used as a frontend on hub.docker.com).

Here’s some more info about the situation:

  • I use an up-to-date Archlinux x64
  • I have no iptables rules (except the ones Docker adds automatically)
  • I connect directly to the Internet (no proxy, just a dumb router between)
  • All similar tests that I’ve done on different physical or logical machines on the same LAN succeeded (yes, even running the cURL command from a VirtualBox VM or Docker container on the problematic machine works!)

I usually do not ask for help when that kind of things happen on a single machine only, but I’ve been searching for days now without any progress and without experiencing a similar behavior on any other web server, so I’m guessing there must be something between the two, I’m just asking what I should check now. Maybe someone else got a similar issue once?

Thanks in advance

Sorry to bump, but the problem’s still here. Anyone?

I’m having the same issue from time to time with cURL. Maybe indication of your issue could be the line:

  • ALPN, server did not agree to a protocol

curl -svv could give you even more info.

In my case wget did work just fine so I used it.

Hi i4olin,
Thanks for your feedback, unfortunately I’m experiencing the problem with all HTTP client, not only cURL. It’s just that using this command gave me a more detailed output than on traditional browsers.
I tried adding the “more verbose” option as you said but I got nothing more.

I’ve tried the same command on another machine and it works perfectly, even if the warning message regarding ALPN is also displayed.

I’m kinda stuck.

Finally figured it out: I configured the physical network interface (eno1 in my case) with a MTU of 4000 instead of the standard 1500… It works with 99,99% of Internet servers, but it seems that some of them do not like it I don’t know why (Pinterest and Deezer were also unavailable).

Not sure anyway if this is really a problem with my machine or my LAN, router or modem, but it’s solved :slight_smile: