TLS handshake timeout pulling from Docker Hub on a Raspberry Pi 1 on docker 17.03

Docker Hub
1

Running into an issue, where we have some Raspberry Pi 1 devices, running Docker 17.03. Recently they started having issues pulling from Docker Hub. The issue seems to only happen when there are multiple layers in the image, and manifest itself in a net/http: TLS handshake timeout in ~10s:

HypriotOS/armv6: pirate@black-pearl in ~
$ docker pull resin/rpi-supervisor:v6.1.2
v6.1.2: Pulling from resin/rpi-supervisor
e68248c7f72c: Pulling fs layer 
0c4000169923: Pulling fs layer 
7df9349c9ba7: Pulling fs layer 
29c5739307d5: Waiting 
2831f22ab9ef: Waiting 
f16e318612a5: Waiting 
error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/7e/7ed5c4b78cbc94f313b3b2cc467ef14e5d0ae925f3397e5300be21c6adc2b65d/data?verify=1530810800-XldS9vFl%2Bm8NAw5Y%2Bdz%2B%2FTvUMWo%3D: net/http: TLS handshake timeout

Tested it on Hypriot v1.4.0 and on resinOS 2.2.0 for the RPi1, both coming with docker 17.03.x.

For cross-checking:

  • Pulling multilayer image from Docker Hub (eg. docker pull resin/rpi-supervisor:v6.1.2): does not work :x:
  • Pulling single layer image from Docker Hub (eg docker pull armv6l/hello-world): does work fine :heavy_check_mark:
  • Pulling the same multilayer image pushed to an alternate registry (e.g. docker pull registry.gitlab.com/imrehg/testing/resin/rpi-supervisor:v6.1.2): does work fine :heavy_check_mark:
  • Running on Raspberry Pi 3 or any X86 devices: seems to work fine :heavy_check_mark:
  • The devices can curl that failing URL: works fine :heavy_check_mark:
  • Running the docker daemon with --max-concurrent-downloads 1: seems to work around the problem :heavy_check_mark:
  • Running docker 17.06 or newer: works fine :heavy_check_mark:

This used to work before for sure, and seems to be related to the Cloudflare resolution of the registry.

Docker version:

$ docker version
Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 11:32:23 2017
 OS/Arch:      linux/arm

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 11:32:23 2017
 OS/Arch:      linux/arm
 Experimental: false

Unfortunately these devices / the docker on these devices cannot immediately be updated to a newer version.

Any idea why it might affect armv6 devices only? What in Docker Hub’s setup is that might cause an issue (since other registry works)?

2

Hi imrehg,

I do not have a reason why this is happening, but I have the same problem and worked around it.
My solution is to reduce the number of parallel downloads to 1 and then my PI Version 1 is able to download multilayer images without a TLS timeout.

/etc/docker/daemon.json
{
    ...
    "max-concurrent-downloads": 1
}

Hope that helpes.

1 Like
3

Hey, that’s exactly what I did as well (through daemon flags, or that config as you mentioned, I’ve listed it above in the things tried). Still would be good to know why is it happening, it seems like a regression to me.

Thanks for the note though, good to know it’s not just me who’s affected…