Speed up image downloading?

Open Source Projects Swarm
1

A suggestion / feature request.

Not sure how crazy this might be, but having struggled all day to pull images into a swarm cluster with poor connectivity, it struck me that it might be very, very cool if the swarm could implement bittorrent (or a bittorrent-like protocol) for image downloads.

Anyway, just a thought!

2

How many nodes in the Swarm cluster? And is the download much faster if downloaded separately on a single node? A comparison of speed could be made.

3

hi dvohra.

First, let met just say I don’t think this is a problem that “first world” users are likely to experience.

Our situation is as follows:

  1. Private docker registry hosted on site, behind SSL terminating haProxy
  2. docker swarm with overlay network running in two geographically separate data centers.
  3. Swarm is not large: 7 hosts at present count
  4. Attempt at a microservice architected app, 7 distinct containers at present count

We ARE currently having connectivity issues, but generally we are not considered one of the best broadband areas in any case.

What I could see by monitoring traffic on the private registry host is upload speeds of about 40kb. Which is rather sad and pathetic, but it is what it is.
So with 7 hosts trying to download the same image concurrently - you can imagine. It took me about 6 hours, with a whole bunch of restarts to get things running.

If I get the time, I will try an experiment with only one host downloading.

PS: related but separate - it might help if docker-compose could provide slightly more output when (for example) pulling images. Currently you get
xxxxxxxxxxx: Pulling repo.somehost.xx.aa/someapp-web:0.0.2...

which is fun for the first 5 minutes, but after 30 you start to wonder.

4

Did a test and speed of download was more than 16 MBPS.

40kb is too slow. Test with

5

Good day @dvohra.

I am not sure what to make of your reply.

Yes, our upload speed from our private registry is to slow. After initial post it is now back to any where from 100Kb to 400Kb. Which is better, but still to slow.
I am glad that you can get 16Mbps against whatever you tested, but that is unfortunately not relevant to us.

What I’m hoping/suggesting with this post is that maybe pull performance for docker hosts participating in a swarm can be improved, by allowing participating hosts to share pulled image data (for example by implementing a BitTorrent-like technology via a plugin).

This will most likely NOT be a problem for persons operating in a “good” broadband situation. And most likely also NOT for persons only pulling stock images from Docker Hub.
But where you need to host a private registry, and pull from that private registry over “poor” connections - this could potentially be very helpful.