When pulling images from DockerHub, I’m seeing issues with some images having a seemingly random image layer being pulled at extremely slow rate, something at a little over 1MB/min perhaps. This problem has been happening on a certain org that I’m a part of, but it has been happening for every repo within that org.
It doesn’t appear to be an issue with the layer size, as we’ve seen the issue on a 20MB layer, and also a 600MB one, the 600MB took longer of course but both were pulling at the same rate.
The issue doesn’t happen on every new version of each image, I haven’t found and pattern that match the change set to when the issue will appear.
The issue is reproducible across machines. The weirdest thing is that it appears that the image pull is only ever as fast as the first machine to pull the image. For example:
- Machine 1 pulls Image A
- Image A is mostly pulled at an expected rate, but one image layer is pulling at 1MB/min
- Wait for 10 minutes, lets say the image is at 10MB done out of a 20MB layer
- Pull image A on Machine 2
- The image will pull at an expected rate, for all image layers, until it reaches 10MB on the problem layer, at which point it will slow down to the 1MB/min speed alongside Machine 1.
Is there something obvious I’m missing here? The behaviour seems very odd, I may be looking in the wrong places but I couldn’t see anyone else complaining about this online.