Docker Community Forums

Share and learn in the Docker community.

A P2P Network for sharing Images & Volumes

Αt Arrikto, we are building the world’s first peer-to-peer network for syncing and sharing snapshots of VMs and Containers:

http://www.arrikto.com/

We are excited to announce that we have successfully integrated the network with Docker as the first step towards testing the technology in the container world. To do so, we’ve implemented our own Docker Storage Driver and Docker Volume Plugin. These drivers pair Docker with our distributed data services layer to allow instant snapshots of Images and Volumes to be distributed efficiently over the P2P network. Thus, the network can act as a decentralized registry for both Images and Volumes.

In a nutshell: You can take any local Image or snapshot of a Volume, and synchronize it efficiently to other locations, over P2P.

The problem

Image sharing and distribution: There has been a lot of discussion from early on in the Docker community regarding Image sizes, and how people struggle to distribute large Images. The problem seems to be present even today, after a lot of effort from the community and several companies. During the latest DockerCon2017 keynote speech, Solomon Hykes addressed this problem again straight from the beginning with the announcement of the Multi-Stage Builds feature. Second phrase on the feature’s official documentation page:

One of the most challenging things about building images is keeping the image size down.

We envision a world where the size of an Image will not matter any more and one will be able to package everything that makes their lives easier inside it, without worrying that the resulting Image will be large.

And we want to be able to treat Docker Volumes, where the critical application data reside, in the exact same way as Images; share Volume snapshots across locations, different deployments, even different platforms (VMware to Docker migration?) seamlessly. How can one do this in the most efficient and easy way?

Current approaches

Currently, the most advanced approaches try to address this problem by using BitTorrent technology inside the data center, tightly coupled with container image workflows. Then trying to replicate across multiple container registries around the globe. Approaches that are not so advanced try to do only the latter. Although P2P technology seems to be the only way forward, current solutions still do not address the problem for:

  • Image distribution on a global scale
  • Infrastructure that extends to end-user devices (laptops)
  • Dynamically changing content (pure BitTorrent only caters to static content)
  • Docker Volumes
  • Sharing Volumes from other, enterprise platforms with Docker
  • Exchanges that need to cross distinct administrative and security domains, e.g., between a corporate DC and a developer’s laptop

We are confident that solving the above problems in a way that is easy, uniform and preserves the current workflows, will enable completely new modes of collaboration among developers, allow enterprises to adopt faster, and simplify moving data from VM- to container-based environments, which until now seemed somewhat disconnected.

We are currently testing the product with initial, select customers and we would love to hear your comments, feedback, and questions.

The process works like this:
You run peer-to-peer file-sharing software (for example, a Gnutella program) on your computer and send out a request for the file you want to download.
To locate the file, the software queries other computers that are connected to the Internet and running the file-sharing software.