I’m using a service called Resin.io which runs a user-defined Docker image across large amount of IoT devices. Commonly, users of this service want to be able to run a Docker engine in the top-level container they are given, for the purpose of running multiple applications in containers on the device.
Users of this service don’t have access to the host OS, only the container that it builds and runs which the user has specified in a Dockerfile. For this reason, the standard docker-in-docker image (would link but I’m a new user and used my two links for this post) is not an option for users that want to run Docker inside of this container.
A successful hack was documented using Docker engine version 1.10.3, which mainly involved mounting the cgroup hierarchies to match those in the host OS; this allowed users to run Docker and docker-compose inside of a container. Recently, the service upgraded from Docker engine 1.10.3 to Docker CE 17.03.01, and the hack now fails to run. Some debugging of the break has been documented here.
The main issue seems to be that running the Docker daemon inside of the container causes the host’s Docker bridge (docker0, at 172.17.0.1) to shut down completely. They are also resolving DNS through this IP address, and when the bridge shuts down, internet access is lost inside the container.
I’ve tried making a separate network bridge to avoid stomping the docker0 bridge, but this is unsuccessful, and the docker0 bridge dies regardless.
What major differences are there between Docker engine 1.10.3 and Docker CE 17.03.01 that would cause this docker-in-docker hack to stop working, and are there any other options I can try to fix this issue?