Regd network config that gets copied to /var/lib/docker/containers/[container id] dir

I am having a weired issue, I have provisioned my cluster on Amazon EC2 with NetworkManager service fully switched off and removed. I am using Mesos for Orchestration and having mesos deploy docker containers.

For some reason with NetworkManager off, when I deploy containers thru mesos I dont see any of the Network config copied over to the /var/lib/docker/containers/[container id] dir. It was missing files like hosts, hostname, resolv.conf, mqueue, shm. Only files I see in there are config.json and hostconfig.json.

However on the same nodes if I manually deploy thru just ‘docker run’ cmd network configs are copied fine.

Question is how does docker knows where to pick the network config from, does it rely on specific network settings for it to copy? What causes docker not to copy those files to container dir?

Take a look at the docker inspect <container> output for both a working and non-working container? Are there any differences to the networking bits for those containers?

Yes. NetworkSettings are all sort of empy for the Mesos Launched container.

Mesos Luanched:

"NetworkSettings": {
    "Bridge": "",
    "SandboxID": "",
    "HairpinMode": false,
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "Ports": null,
    "SandboxKey": "",
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null,
    "EndpointID": "",
    "Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "IPAddress": "",
    "IPPrefixLen": 0,
    "IPv6Gateway": "",
    "MacAddress": "",
    "Networks": null
}

Direct Docker Launched:

"NetworkSettings": {
    "Bridge": "",
    "SandboxID": "dc79a56d094b5564685c84efabad7f3ee643eb44014b5195e6220ddc7cac0b8e",
    "HairpinMode": false,
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "Ports": {
        "6379/tcp": null
    },
    "SandboxKey": "/var/run/docker/netns/dc79a56d094b",
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null,
    "EndpointID": "ca18333d66949e4b8f29a14eca7e3af89c0ce0939f622346e548b17bf0159df6",
    "Gateway": "172.17.0.1",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "IPAddress": "172.17.0.2",
    "IPPrefixLen": 16,
    "IPv6Gateway": "",
    "MacAddress": "02:42:ac:11:00:02",
    "Networks": {
        "bridge": {
            "EndpointID": "ca18333d66949e4b8f29a14eca7e3af89c0ce0939f622346e548b17bf0159df6",
            "Gateway": "172.17.0.1",
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "02:42:ac:11:00:02"
        }
    }
}

This looks like a container that would have been started with the --net=none option. Whatever piece in your setup that is creating the containers appears to be intentionally turning off networking for some reason.

Ok, Thanks Jeff. Apparently the container definition I submitted to Marathon happens to have some Overlay Networking config with out the overlay library installed on the box. This caused Mesos apply NONE networking instead of HOST networking. As I got rid of those settings, it started to deploy fine.