Docker Community Forums

Share and learn in the Docker community.

Container Services works for few mins then stops responding to the host

docker

(Sam Thambu) #1

Docker for Mac Version: 17.09.1-ce-mac42 (21090)
OS X: 10.13.3

I have created a container using docker-compose (and also via dockerode). In both cases, the container gets created and started fine for sometime.

My Docker-Compose File

version: '3'

services:
 ns-xxxx:
    image: internalhost/db:master
    container_name: xxxx
  expose:
    - "27017"
ports:
  - "27017:27017"
restart: always
volumes:
  - ./shared:/data/shared
entrypoint: /data/shared/entrypoint.sh

I able to access the container-services (example: MongoDB based on the above compose-file) from the Host (MacOS). Say after sometime say 15mins or so, I am unable to access the container services from the Host(MacOS). Ie if i try to connect to mongodb via a mongo-client - I get error Network is Unreachable.

The interesting thing is this: the container is still active and healthy and responding if I do docker ps - a as well as I do

docker exec -it my-container-name /bin/bash
$mongo // inside the container - mongo-services works fine too as its able to connect to MongoDB

Also if I try to check if the host/port responding like the below command in the HOST (macOS):

nc -vz -w 1 localhost 27017

found 0 associations
        found 1 connections:
             1: flags=82<CONNECTED,PREFERRED>
            outif lo0
            src ::1 port 57605
            dst ::1 port 27017
            rank info not available
            TCP aux info available

        Connection to localhost port 27017 [tcp/*] succeeded!

Unsure what the issue is or where to look… as it works for the first few mins

PS: I have an office MAC with corp firewall (not sure if it has anything to do with the networking of the docker daemon)

My docker daemon logs:

&types.Event{Type:\"exit\", Id:\"cea655fd22c6531c22799e38e47be61a28b9967923893a823afc0bfa87a56213\", Status:0x0, Pid:\"3f767e2e969ce1a04a5148f3ece400f52cd3223c23f08c49948b11da69d1ca62\", Timestamp:(*timestamp.Timestamp)(0xc4215d96c0)}"  
Feb 15 21:54:09 moby root: time="2018-02-15T21:54:09.610503135Z" level=debug msg="attach: stderr: end"  
Feb 15 21:54:09 moby root: time="2018-02-15T21:54:09.610526005Z" level=debug msg="attach: stdout: end"  
Feb 15 21:54:09 moby root: time="2018-02-15T21:54:09.610580876Z" level=debug msg="Health check for container 84373b21f3ad4a915dc87303a0a3e356a770da01642495325014e46328778753 done (exitCode=0)"  
Feb 15 21:54:09 moby root: time="2018-02-15T21:54:09.610631303Z" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"exit\", Id:\"dca5fa7093590e9558053d8d3aae9430eb7bcb90042e7b6e5fabbc8660c73033\", Status:0x0, Pid:\"dd75386e692b5814cc88ef5a9070c24b51fbc57a1de8bfc3d3cb70402e5a533e\", Timestamp:(*timestamp.Timestamp)(0xc4215d9830)}"  
Feb 15 21:54:09 moby root: time="2018-02-15T21:54:09.610704721Z" level=debug msg="attach: stderr: end"  
Feb 15 21:54:09 moby root: time="2018-02-15T21:54:09.610714181Z" level=debug msg="attach: stdout: end"  
Feb 15 21:54:09 moby root: time="2018-02-15T21:54:09.610729097Z" level=debug msg="Health check for container cea655fd22c6531c22799e38e47be61a28b9967923893a823afc0bfa87a56213 done (exitCode=0)"  
Feb 15 21:54:09 moby root: time="2018-02-15T21:54:09.611490050Z" level=debug msg="attach: stderr: end"  
Feb 15 21:54:09 moby root: time="2018-02-15T21:54:09.611551629Z" level=debug msg="attach: stdout: end"  
Feb 15 21:54:09 moby root: time="2018-02-15T21:54:09.611566992Z" level=debug msg="Health check for container dca5fa7093590e9558053d8d3aae9430eb7bcb90042e7b6e5fabbc8660c73033 done (exitCode=0)"  
Feb 15 21:54:10 moby root: time="2018-02-15T21:54:10.364190087Z" level=debug msg="Name To resolve: docker.for.mac.localhost."  
Feb 15 21:54:10 moby root: time="2018-02-15T21:54:10.364367057Z" level=debug msg="[resolver] query docker.for.mac.localhost. (AAAA) from 172.18.0.12:45950, forwarding to udp:192.168.65.1"  
Feb 15 21:54:10 moby root: time="2018-02-15T21:54:10.370744338Z" level=debug msg="[resolver] received A record \"192.168.65.1\" for \"docker.for.mac.localhost.\" from udp:192.168.65.1"  
Feb 15 21:54:10 moby root: time="2018-02-15T21:54:10.579898294Z" level=debug msg="Name To resolve: x1-some-fake-host."  
Feb 15 21:54:10 moby root: time="2018-02-15T21:54:10.579988909Z" level=debug msg="[resolver] lookup name x1-some-fake-host. present without IPv6 address"  
Feb 15 21:54:10 moby root: time="2018-02-15T21:54:10.580626038Z" level=debug msg="Name To resolve: x1-some-fake-host."  
Feb 15 21:54:10 moby root: time="2018-02-15T21:54:10.580724092Z" level=debug msg="[resolver] lookup for x1-some-fake-host.: IP [172.18.0.3]"  
Feb 15 21:54:11 moby root: time="2018-02-15T21:54:11.366577040Z" level=debug msg="Name To resolve: docker.for.mac.localhost."  
Feb 15 21:54:11 moby root: time="2018-02-15T21:54:11.366774851Z" level=debug msg="[resolver] query docker.for.mac.localhost. (AAAA) from 172.18.0.12:49930, forwarding to udp:192.168.65.1"  
Feb 15 21:54:11 moby root: time="2018-02-15T21:54:11.367222223Z" level=debug msg="[resolver] received A record \"192.168.65.1\" for \"docker.for.mac.localhost.\" from udp:192.168.65.1"  
Feb 15 21:54:12 moby root: time="2018-02-15T21:54:12.367918835Z" level=debug msg="Name To resolve: docker.for.mac.localhost."  
Feb 15 21:54:12 moby root: time="2018-02-15T21:54:12.368099776Z" level=debug msg="[resolver] query docker.for.mac.localhost. (AAAA) from 172.18.0.12:38466, forwarding to udp:192.168.65.1"  
Feb 15 21:54:12 moby root: time="2018-02-15T21:54:12.368604729Z" level=debug msg="[resolver] received A record \"192.168.65.1\" for \"docker.for.mac.localhost.\" from udp:192.168.65.1"  
Feb 15 21:54:13 moby root: time="2018-02-15T21:54:13.369450214Z" level=debug msg="Name To resolve: docker.for.mac.localhost."  
Feb 15 21:54:13 moby root: time="2018-02-15T21:54:13.369619231Z" level=debug msg="[resolver] query docker.for.mac.localhost. (AAAA) from 172.18.0.12:51783, forwarding to udp:192.168.65.1"  
Feb 15 21:54:13 moby root: time="2018-02-15T21:54:13.370095101Z" level=debug msg="[resolver] received A record \"192.168.65.1\" for \"docker.for.mac.localhost.\" from udp:192.168.65.1"  
Feb 15 21:54:13 moby root: time="2018-02-15T21:54:13.586119843Z" level=debug msg="Name To resolve: x2-some-fake-host."  
Feb 15 21:54:13 moby root: time="2018-02-15T21:54:13.586285820Z" level=debug msg="[resolver] lookup name x2-some-fake-host. present without IPv6 address"  
Feb 15 21:54:13 moby root: time="2018-02-15T21:54:13.586171325Z" level=debug msg="Name To resolve: x5-some-fake-host."  
Feb 15 21:54:13 moby root: time="2018-02-15T21:54:13.586459758Z" level=debug msg="[resolver] lookup name x5-some-fake-host. present without IPv6 address"  
Feb 15 21:54:13 moby root: time="2018-02-15T21:54:13.586528186Z" level=debug msg="Name To resolve: x2-some-fake-host."  
Feb 15 21:54:13 moby root: time="2018-02-15T21:54:13.586724674Z" level=debug msg="[resolver] lookup for x2-some-fake-host.: IP [172.18.0.9]"  
Feb 15 21:54:13 moby root: time="2018-02-15T21:54:13.587005819Z" level=debug msg="Name To resolve: x5-some-fake-host."  
Feb 15 21:54:13 moby root: time="2018-02-15T21:54:13.587072990Z" level=debug msg="[resolver] lookup for x5-some-fake-host.: IP [172.18.0.6]"  
Feb 15 21:54:14 moby root: time="2018-02-15T21:54:14.370378559Z" level=debug msg="Name To resolve: docker.for.mac.localhost."  
Feb 15 21:54:14 moby root: time="2018-02-15T21:54:14.370592525Z" level=debug msg="[resolver] query docker.for.mac.localhost. (AAAA) from 172.18.0.12:32948, forwarding to udp:192.168.65.1"  
Feb 15 21:54:14 moby root: time="2018-02-15T21:54:14.371058859Z" level=debug msg="[resolver] received A record \"192.168.65.1\" for \"docker.for.mac.localhost.\" from udp:192.168.65.1"  
Feb 15 21:54:14 moby root: time="2018-02-15T21:54:14.589264739Z" level=debug msg="Name To resolve: x3-some-fake-host."  
Feb 15 21:54:14 moby root: time="2018-02-15T21:54:14.589400743Z" level=debug msg="[resolver] lookup name x3-some-fake-host. present without IPv6 address"  
Feb 15 21:54:14 moby root: time="2018-02-15T21:54:14.589860972Z" level=debug msg="Name To resolve: x3-some-fake-host."  
Feb 15 21:54:14 moby root: time="2018-02-15T21:54:14.589950607Z" level=debug msg="[resolver] lookup for x3-some-fake-host.: IP [172.18.0.11]"  
Feb 15 21:54:15 moby root: time="2018-02-15T21:54:15.372011095Z" level=debug msg="Name To resolve: docker.for.mac.localhost."  
Feb 15 21:54:15 moby root: time="2018-02-15T21:54:15.372596986Z" level=debug msg="[resolver] query docker.for.mac.localhost. (AAAA) from 172.18.0.12:53653, forwarding to udp:192.168.65.1"  
Feb 15 21:54:15 moby root: time="2018-02-15T21:54:15.373880782Z" level=debug msg="[resolver] received A record \"192.168.65.1\" for \"docker.for.mac.localhost.\" from udp:192.168.65.1"  
Feb 15 21:54:15 moby root: time="2018-02-15T21:54:15.590203435Z" level=debug msg="Name To resolve: x4-some-fake-host."  
Feb 15 21:54:15 moby root: time="2018-02-15T21:54:15.590376543Z" level=debug msg="[resolver] lookup name x4-some-fake-host. present without IPv6 address"  
Feb 15 21:54:15 moby root: time="2018-02-15T21:54:15.590320742Z" level=debug msg="Name To resolve: x6-some-fake-host."  
Feb 15 21:54:15 moby root: time="2018-02-15T21:54:15.590487756Z" level=debug msg="[

docker inspect container-name

[
    {
        "Id": "133f77e6d00f5afc1031928d1e0468204b1829ab10975285e4f69eb1241d74d0",
        "Created": "2018-02-15T20:51:39.046278673Z",
        "Path": "/data/shared/entrypoint.sh",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 2483,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2018-02-15T20:51:39.841657732Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:9f9ba1d05947e0f95435e03520ef6124e1bdba44cae8fac9811ae66ee7fbf5d5",
        "ResolvConfPath": "/var/docker/containers/133f77e6d00f5afc1031928d1e0468204b1829ab10975285e4f69eb1241d74d0/resolv.conf",
        "HostnamePath": "/var/docker/containers/133f77e6d00f5afc1031928d1e0468204b1829ab10975285e4f69eb1241d74d0/hostname",
        "HostsPath": "/var/docker/containers/133f77e6d00f5afc1031928d1e0468204b1829ab10975285e4f69eb1241d74d0/hosts",
        "LogPath": "/var/docker/containers/133f77e6d00f5afc1031928d1e0468204b1829ab10975285e4f69eb1241d74d0/133f77e6d00f5afc1031928d1e0468204b1829ab10975285e4f69eb1241d74d0-json.log",
        "Name": "/build-slaveqadb",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/xxxx/yyyyy/mongodb/shared:/data/shared:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "containername_default",
            "PortBindings": {
                "27017/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "27017"
                    }
                ]
                ]
            },
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/docker/overlay2/9911ec636156cdd8d1c9e3256587c914ea875b9ba3643faee91f6cea35824698-init/diff:/var/docker/overlay2/21753ab186c2f5087f5e1b390cf23ea23aaa209b62a5373922f44b6e74e4c8bc/diff:/var/docker/overlay2/32643bdeb1ffb425ef3c93b17752d0615b02060ef239a9191891e52c6ff45063/diff:/var/docker/overlay2/072ede27bcb7c3d8882e5dccb85f9794b3c30b693946037a35fb97fc64a2f356/diff:/var/docker/overlay2/8dbfa3c7c93e23fffa8a26a4bb89ba154b496b9ebc85cb488bd7746652338b32/diff:/var/docker/overlay2/1a0d89c74b0ec24a00333b6ddf4faf6a29af63d02d1de1c1a29de0abc88ce7c2/diff:/var/docker/overlay2/43c7c2f763e08c2735996d529a3b125fae54c9a95c7b10badabd953b9d70d354/diff:/var/docker/overlay2/00fc9994414b848a3ab3b2b0062b20a7b7f13c97fe35e4531ed7ca740f1ea24e/diff:/var/docker/overlay2/10cd148ce7245bbf0a2f1dfeb298f8e4fdef40490511cbeb48e3e40a038515c3/diff:/var/docker/overlay2/4f717b7605262c7b462a136d08712b264b855da697bb54047738a229d23f9091/diff:/var/docker/overlay2/12d845ec364c1faab03f8c653924c1f0fa9d3870370ba4d748651ad73d189758/diff:/var/docker/overlay2/4a613bd647be114d175e319d4d989083c63d2a3c0dd8a4ac8ff137ef66e8dc7d/diff:/var/docker/overlay2/5c6f650aaeda667bde37505ad586bd0c61316033050edde15eeb9a43339172df/diff:/var/docker/overlay2/a4fa7024bc7713dda5ef7feac520043b1d66a70287030eec104bb3f2317959d8/diff:/var/docker/overlay2/659e88c4869c3e37abbad3678509a75b1b63ba510775867ec07942cd180d8f7b/diff:/var/docker/overlay2/b93e4f4175021869322c9d7aa6ad86103a5e044d98a895f91a59d34b107aa812/diff:/var/docker/overlay2/6cd58d86fb7980f0b5d29f859b2efcec9d8b1f747640cbb473fe5eaf47bab110/diff:/var/docker/overlay2/c66f2cf20025340f253d57e53b12b5df87ce2ba766fe928b5b76d3655025041a/diff:/var/docker/overlay2/909053671b5e9f4d95a73b10e08b40b6fcc1581be0998c30207312b2ffbf15af/diff:/var/docker/overlay2/fff6d1761131c032e6bad383753d1d3021eed7fe60b7c9eadb762b3e45c33b58/diff:/var/docker/overlay2/db6c99767c383eb5e64bd77651c95df53b9eb6c40c1c5ac055ee1c07fdcf1486/diff:/var/docker/overlay2/22dcb5f0a3b5ff1527e6bc427fd0ad4d3f8f65642562ea9a3a1ac995fd128498/diff:/var/docker/overlay2/c5273b7e4f5f361eeb7932a1376798a9db484e9cd3e04ec44871252350f5faa3/diff:/var/docker/overlay2/99670a813929b63d21de5b4d07f9796cbcacd103e9e110f1a9824f0cd60c4003/diff:/var/docker/overlay2/12e457bd1264395ac4158bb1e89e62daafec55ee4a4554da4c83c263188f8009/diff:/var/docker/overlay2/0ef440acc9a9c55770633fc79369dacbb3fc20c91e1051d0d7d2ed9583726ab7/diff:/var/docker/overlay2/a90a94e4489d89e2e6cd616b8c44681d236ea202ab1dd01e29666f66d3ca7ef5/diff:/var/docker/overlay2/28108308dfc503368ae5f6759545fd3f1551310638baae3b2b74c8b843f69ba5/diff:/var/docker/overlay2/615a27792fa2c9d84c3e4e5e8a8d1355b9a979ee12e485350b6f2b2af5dc3918/diff:/var/docker/overlay2/e2c19ea5f841fd1e965e41a3728d9f4e9a40a44931d03f48e901616548afb47e/diff:/var/docker/overlay2/668347bea5b89f07ba47a62234b7a3b3f56f17329c4448613727961042744256/diff",
                "MergedDir": "/var/docker/overlay2/9911ec636156cdd8d1c9e3256587c914ea875b9ba3643faee91f6cea35824698/merged",
                "UpperDir": "/var/docker/overlay2/9911ec636156cdd8d1c9e3256587c914ea875b9ba3643faee91f6cea35824698/diff",
                "WorkDir": "/var/docker/overlay2/9911ec636156cdd8d1c9e3256587c914ea875b9ba3643faee91f6cea35824698/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/xxxx/yyyy/mongodb/shared",
                "Destination": "/data/shared",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "133f77e6d00f",
            "Domainname": "",
            "User": "root",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "27017/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "no_proxy=*.local, 169.254/16",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "http_proxy=http://proxy:xxxx",
                "https_proxy=http://proxy:xxxx",
                "JETTY_VERSION=9.2.23.v20171218"
            ],
            "Cmd": null,
            "Image": "masked-dockerimage-host:tag",
            "Volumes": {
                "/data/shared": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "/data/shared/entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "3f7d146d57d71823a7c0af805f0e50fed6081339e883674abbad2127980cd1a5",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "xxxxx",
                "com.docker.compose.service": "xxxxx",
                "com.docker.compose.version": "1.17.1"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "9caa7a0e2e1da9e64d40005b530af40f6d7e8877bec9eea06c48e209c80079e6",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "27017/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "27017"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/9caa7a0e2e1d",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "my-custom-network": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": [
                        "133f77e6d00f"
                    ],
                    "NetworkID": "26d427d5e938a54546e0e5661868fec112e3a9ce5c3779eb4079ce48c78638ce",
                    "EndpointID": "7d60c0be429f89125428a1f4c2f6dc7191faad2c5c643ff4bc2413974374bf6b",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:12:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

Not in VPN but I do have a VPN client in the MAC.

Any pointers on why it works for sometime and then not? Networking Issue?