Container fails with error 137, but no OOM flag set (and there's plenty of RAM)

General Discussions
1

Issue type: Containers being killed unexpectedly after updating to latest version of Docker
OS Version/build: Ubuntu 18.04.2 LTS
App version: Docker version 18.09.2, build 6247962
Steps to reproduce: I upgraded to the latest version of Docker, and after that point I’m getting a container dying every 2-3 days. A docker inspect on the downed container reveals exit code 137 (which I believe is associated with running out of memory), yet the OOMKilled flag on docker inspect is set to false, and the host machine has plenty of RAM (using less than 50% of what’s available).

The syslog of the hosting machine is as follows:

Feb 17 07:21:15 vm-name dockerd[19076]: time="2019-02-17T07:21:15.675878118Z" level=info msg="NetworkDB stats vm-name(9084bcf5cb15) - netID:snxdatdr3bu8ev74u7d4iktt1 leaving:false netPeers:3 entries:6 Queue qLen:0 $
Feb 17 07:21:15 vm-name dockerd[19076]: time="2019-02-17T07:21:15.676486061Z" level=info msg="NetworkDB stats vm-name(9084bcf5cb15) - netID:65qxed2d27vqusgl724k5umov leaving:false netPeers:3 entries:69 Queue qLen:0$
Feb 17 07:21:20 vm-name containerd[9959]: time="2019-02-17T07:21:20.789853273Z" level=info msg="shim reaped" id=ef442c28b0c7ee66c23f64c1c8d08a13a88b52118f487d599672f7612e694030
Feb 17 07:21:20 vm-name containerd[9959]: time="2019-02-17T07:21:20.790303433Z" level=warning msg="cleaning up after killed shim" id=ef442c28b0c7ee66c23f64c1c8d08a13a88b52118f487d599672f7612e694030 namespace=moby
Feb 17 07:21:20 vm-name dockerd[19076]: time="2019-02-17T07:21:20.928539426Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 17 07:21:20 vm-name kernel: [12420727.853095] br0: port 11(veth18) entered disabled state
Feb 17 07:21:20 vm-name kernel: [12420727.853261] veth4746493: renamed from eth0
Feb 17 07:21:20 vm-name systemd-udevd[27533]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 17 07:21:20 vm-name kernel: [12420727.899744] docker_gwbridge: port 10(vethe07fe12) entered disabled state
Feb 17 07:21:20 vm-name systemd-networkd[2544]: vethe07fe12: Lost carrier
Feb 17 07:21:20 vm-name systemd-timesyncd[2614]: Network configuration changed, trying to establish connection.
Feb 17 07:21:20 vm-name kernel: [12420727.910597] veth35e9c17: renamed from eth1
Feb 17 07:21:21 vm-name systemd-udevd[27534]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 17 07:21:21 vm-name systemd[1]: Starting resolvconf-pull-resolved.service...
Feb 17 07:21:21 vm-name systemd-timesyncd[2614]: Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com).
Feb 17 07:21:21 vm-name systemd[1]: Started resolvconf-pull-resolved.service.
Feb 17 07:21:21 vm-name systemd-timesyncd[2614]: Network configuration changed, trying to establish connection.
Feb 17 07:21:21 vm-name networkd-dispatcher[20508]: WARNING:Unknown index 1013 seen, reloading interface list
Feb 17 07:21:21 vm-name systemd[1]: Starting resolvconf-pull-resolved.service...
Feb 17 07:21:21 vm-name systemd[1]: Started resolvconf-pull-resolved.service.
Feb 17 07:21:21 vm-name systemd-timesyncd[2614]: Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com).
Feb 17 07:21:21 vm-name systemd-timesyncd[2614]: Network configuration changed, trying to establish connection.
Feb 17 07:21:21 vm-name kernel: [12420728.108966] docker_gwbridge: port 10(vethe07fe12) entered disabled state
Feb 17 07:21:21 vm-name kernel: [12420728.118328] device vethe07fe12 left promiscuous mode
Feb 17 07:21:21 vm-name kernel: [12420728.118330] docker_gwbridge: port 10(vethe07fe12) entered disabled state
Feb 17 07:21:21 vm-name systemd[1]: Starting resolvconf-pull-resolved.service...
Feb 17 07:21:21 vm-name kernel: [12420728.139447] br0: port 11(veth18) entered disabled state
Feb 17 07:21:21 vm-name kernel: [12420728.149272] device veth18 left promiscuous mode
Feb 17 07:21:21 vm-name kernel: [12420728.149276] br0: port 11(veth18) entered disabled state
Feb 17 07:21:21 vm-name systemd[1]: Started resolvconf-pull-resolved.service.
Feb 17 07:21:21 vm-name systemd-timesyncd[2614]: Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com).
Feb 17 07:21:35 vm-name dockerd[19076]: time="2019-02-17T07:21:35.441873889Z" level=error msg="Error setting up exec command in container downed_container_name: Container ef442c28b0c7ee66c23f64c1c8d08a13a88b52118f487d5996$
Feb 17 07:21:50 vm-name dockerd[19076]: time="2019-02-17T07:21:50.420833479Z" level=error msg="Error setting up exec command in container downed_container_name: Container ef442c28b0c7ee66c23f64c1c8d08a13a88b52118f487d5996$
Feb 17 07:22:05 vm-name dockerd[19076]: time="2019-02-17T07:22:05.563774509Z" level=error msg="Error setting up exec command in container downed_container_name: Container ef442c28b0c7ee66c23f64c1c8d08a13a88b52118f487d5996$
Feb 17 07:22:20 vm-name dockerd[19076]: time="2019-02-17T07:22:20.346961696Z" level=error msg="Error setting up exec command in container downed_container_name: Container ef442c28b0c7ee66c23f64c1c8d08a13a88b52118f487d5996$

Whilst Docker inspect for the downed container is as follows:

[
{
    "Id": "ef442c28b0c7ee66c23f64c1c8d08a13a88b52118f487d599672f7612e694030",
    "Created": "2019-02-14T08:02:30.713269654Z",
    "Path": "postfix-entrypoint.sh",
    "Args": [
        "/bin/sh",
        "-c",
        "/usr/sbin/apache2ctl -D FOREGROUND"
    ],
    "State": {
        "Status": "exited",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 137,
        "Error": "",
        "StartedAt": "2019-02-14T08:02:31.877483084Z",
        "FinishedAt": "2019-02-17T07:21:20.927979789Z"
    },
    "Image": "sha256:ca7e0f9e43fa72a5b56fb4ba44ce242a66cd5fc79a5fb80d3e28a6289c49e1e8",
    "ResolvConfPath": "/path/to/imageef442c28b0c7ee66c23f64c1c8d08a13a88b52118f487d599672f7612e694030/resolv.conf",
    "HostnamePath": "/path/to/imageef442c28b0c7ee66c23f64c1c8d08a13a88b52118f487d599672f7612e694030/hostname",
    "HostsPath": "/path/to/imageef442c28b0c7ee66c23f64c1c8d08a13a88b52118f487d599672f7612e694030/hosts",
    "LogPath": "/path/to/imageef442c28b0c7ee66c23f64c1c8d08a13a88b52118f487d599672f7612e694030/ef442c28b0c7ee66c23f64c1c8d08a13a88b52118f487d599672f7612e694030-json.log",
    "Name": "/downed_container_name",
    "RestartCount": 0,
    "Driver": "overlay2",
    "Platform": "linux",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "docker-default",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": null,
        "ContainerIDFile": "",
        "LogConfig": {
            "Type": "json-file",
            "Config": {}
        },
        "NetworkMode": "cross-docker-network",
        "PortBindings": {},
        "RestartPolicy": {
            "Name": "no",
            "MaximumRetryCount": 0
        },
        "AutoRemove": false,
        "VolumeDriver": "",
        "VolumesFrom": null,
        "CapAdd": null,
        "CapDrop": null,
        "Dns": [],
        "DnsOptions": [],
        "DnsSearch": [],
        "ExtraHosts": null,
        "GroupAdd": null,
        "IpcMode": "shareable",
        "Cgroup": "",
        "Links": null,
        "OomScoreAdj": 0,
        "PidMode": "",
        "Privileged": false,
        "PublishAllPorts": false,
        "ReadonlyRootfs": false,
        "SecurityOpt": null,
        "UTSMode": "",
        "UsernsMode": "",
        "ShmSize": 67108864,
        "Runtime": "runc",
        "ConsoleSize": [
            0,
            0
        ],
        "Isolation": "",
        "CpuShares": 0,
        "Memory": 0,
        "NanoCpus": 0,
        "CgroupParent": "",
        "BlkioWeight": 0,
        "BlkioWeightDevice": [],
        "BlkioDeviceReadBps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteIOps": null,
        "CpuPeriod": 0,
        "CpuQuota": 0,
        "CpuRealtimePeriod": 0,
        "CpuRealtimeRuntime": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "Devices": [],
        "DeviceCgroupRules": null,
        "DiskQuota": 0,
        "KernelMemory": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "MemorySwappiness": null,
        "OomKillDisable": false,
        "PidsLimit": 0,
        "Ulimits": null,
        "CpuCount": 0,
        "CpuPercent": 0,
        "IOMaximumIOps": 0,
        "IOMaximumBandwidth": 0,
        "MaskedPaths": [
            "/proc/asound",
            "/proc/acpi",
            "/proc/kcore",
            "/proc/keys",
            "/proc/latency_stats",
            "/proc/timer_list",
            "/proc/timer_stats",
            "/proc/sched_debug",
            "/proc/scsi",
            "/sys/firmware"
        ],
        "ReadonlyPaths": [
            "/proc/bus",
            "/proc/fs",
            "/proc/irq",
            "/proc/sys",
            "/proc/sysrq-trigger"
        ]
    },
    "GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/docker/overlay2/58f18ce368873d8624ded491cd1da83b3b9d32b5335e3cfa4c7ae79ecf6c809a-init/diff:/var/lib/docker/overlay2/b3b091d267b345880d7c826f5289abbe58d2755618571deb023d2df89b6c74e2/diff:/var/lib/docker/overlay2/b685584d9c6631df72e5d459a3ec4c55ea58f6eb7cca8aa9c9e4b6487bf9c19f/diff:/var/lib/docker/overlay2/06234cd3652e2244a24c229b3f42ac596b80402e6962bc0ab21ee444eba39405/diff:/var/lib/docker/overlay2/0ec485049a653d80bdfd77b4a288ef35399f05700a38c4b7b63b0225894f62a0/diff:/var/lib/docker/overlay2/fdc4828d7bee80cc5a1f06ca6b974ce50726a70f94f5a1e3ce9bf0dcebf47a71/diff:/var/lib/docker/overlay2/ae0b03256e694bb71eebf8842a31da3ce3250482d0939a64fd74f9c228bbb1da/diff:/var/lib/docker/overlay2/fd72210cfe43bd8055b0194dc1be221faded7264bc3b1ba73b2060674f747a67/diff:/var/lib/docker/overlay2/a87e16923231a743becab201f256ef1e226dcbffd07feaa4e9f320a49f8b0c6f/diff:/var/lib/docker/overlay2/87d5f60633c2f9e196a1b622b4ad05b35182207a1ffc44c62b061219d68ac544/diff:/var/lib/docker/overlay2/183d8043572110e74300cb105e3bf41ae464056a9ac69898899cf0c383841cd1/diff:/var/lib/docker/overlay2/5d385dc9857bcd19e2715f339404d89db02e5e68b7e87b5464031cd7ad8deaff/diff:/var/lib/docker/overlay2/6bb1b21e87cb81e721c197417b2dfbe808850911614d2b2504266fe126c99a23/diff:/var/lib/docker/overlay2/92105a7b16f4444614c2cc0bb29a03201c6bef200c77a3251978b0da49a49930/diff:/var/lib/docker/overlay2/a88361a49941042c8ed3f346ff562dd462bcdad18419699bcfae64483e0e5c05/diff:/var/lib/docker/overlay2/91aec0444281fd868e2430983b77fe2df8fb3a28008f93b6f4c2e13e8344b310/diff:/var/lib/docker/overlay2/27c0c6eee07bb00dc780bd51fc020e05202070043262015371bc3a70894b8bc5/diff:/var/lib/docker/overlay2/8c4ecafe0c5d8c1ff9a52007582d0937c3130ca295551bf89353bb0a0a84b756/diff:/var/lib/docker/overlay2/e444976dda23fc606b8d3a62b393613c80c31eaec1b1243b77e1cee570909baf/diff:/var/lib/docker/overlay2/636f258ad86f78f9f7a6a6ba8f392de4a943bc4257e0d8997de7cd4bc02ccbd7/diff:/var/lib/docker/overlay2/5556aef835e759c84576269ece6ba7fc7be1b43fe673babafb1dd5253e22f221/diff:/var/lib/docker/overlay2/9ef4e28f2a6cc2bdd0874275fca085394f002d7e8fdfd6712e563afe8502f503/diff:/var/lib/docker/overlay2/132109c604355ea09022578657a413be8dcc5e3046b53d04e4ab8e999924ef73/diff:/var/lib/docker/overlay2/4f8b78a60155d76b80f03bd3baaa197d675b9988c5fa6fac31fedeafb030d158/diff:/var/lib/docker/overlay2/c8e6505d95c9f5f7d0da12641219691365d0414186fdb40bae7ffc8816557654/diff:/var/lib/docker/overlay2/f3491b845bf78d8d8eddccb9fd08b7ef7cac4010c4bbafe0f62eab853ceebd21/diff:/var/lib/docker/overlay2/8fbc0b3a678c67a9a163d0d5bb9efc9bbbc4c5609732ae9834b9d3c2c53af607/diff:/var/lib/docker/overlay2/f9cde724374ff453746b1a1280447c2eb824d3b2c529cad7604d9cb64c2789fd/diff:/var/lib/docker/overlay2/7e8830910c825a3902c002ab7a2feb38e0f0bbcb780ebbd5c2a6623e1203af89/diff:/var/lib/docker/overlay2/e9844c8b88a464557fba3874a6d12b81d4e25329d779e6e08956e09932cb893d/diff:/var/lib/docker/overlay2/f8806b447d2c8f71bd0399ed48e28455cb0e9e857a1fb5fc3c475ca041dbe093/diff:/var/lib/docker/overlay2/6caacd8fa592e94b19284f335098940406cbcd1458da15b57eabcfa7f32eb89a/diff:/var/lib/docker/overlay2/6076613a18ff9a6557a9f9f8e8f190e8fafaa7ad3fb3fbdea30b6dea52375e86/diff:/var/lib/docker/overlay2/f81565367638089fb3349cc3e99c8e9d3805bc11d647529797700466cf5fed84/diff:/var/lib/docker/overlay2/bec1a40e917f0a6c46d95f279f5df90d64bb9b7670197d7963fdbfd453a5dda1/diff:/var/lib/docker/overlay2/ce824efd6cd33deeff6e70ca10e616231f8b7b3249dc475116780282f1106d80/diff:/var/lib/docker/overlay2/1b1cabfc142ea68dac112e71d937e1ce54fe1ef59da2b49b72809baee24de9e7/diff:/var/lib/docker/overlay2/606d7060f5a2949b968e752ad68409a63c8792904c0df42cd6ca5dd165abc809/diff:/var/lib/docker/overlay2/d02e18cc54422db65607bb99579d2ca5992f303198718e9de2809545cfac1c80/diff:/var/lib/docker/overlay2/82e51082017aad03bc1e3edb1069f4907de07955a9e76c39ed57101ff8fc8049/diff",
            "MergedDir": "/var/lib/docker/overlay2/58f18ce368873d8624ded491cd1da83b3b9d32b5335e3cfa4c7ae79ecf6c809a/merged",
            "UpperDir": "/var/lib/docker/overlay2/58f18ce368873d8624ded491cd1da83b3b9d32b5335e3cfa4c7ae79ecf6c809a/diff",
            "WorkDir": "/var/lib/docker/overlay2/58f18ce368873d8624ded491cd1da83b3b9d32b5335e3cfa4c7ae79ecf6c809a/work"
        },
        "Name": "overlay2"
    },
    "Mounts": [],
    "Config": {
        "Hostname": "ef442c28b0c7",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "ExposedPorts": {
            "80/tcp": {}
        },
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "TZ=Europe/London",
            "APACHE_RUN_USER=www-data",
            "APACHE_RUN_GROUP=www-data",
            "APACHE_LOG_DIR=/var/log/apache2",
            "APACHE_LOCK_DIR=/var/lock/apache2",
            "APACHE_PID_FILE=/var/run/apache2.pid"
        ],
        "Cmd": [
            "/bin/sh",
            "-c",
            "/usr/sbin/apache2ctl -D FOREGROUND"
        ],
        "ArgsEscaped": true,
        "Image": "image_name",
        "Volumes": null,
        "WorkingDir": "",
        "Entrypoint": [
            "postfix-entrypoint.sh"
        ],
        "OnBuild": null,
        "Labels": {
            "Taggedversion": "version",
            "color": "green"
        }
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "10832deb1af71f250fe9d93d8e96f197ee924f8e0037b3d4d92f7383f0b30465",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {},
        "SandboxKey": "/var/run/docker/netns/10832deb1af7",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "MacAddress": "",
        "Networks": {
            "cross-docker-network": {
                "IPAMConfig": {
                    "IPv4Address": "LOCAL_IP_ADDRESS"
                },
                "Links": [
                    "other-container:name"
                ],
                "Aliases": [
                    "ef442c28b0c7"
                ],
                "NetworkID": "65qxed2d27vqusgl724k5umov",
                "EndpointID": "",
                "Gateway": "",
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "",
                "DriverOpts": null
            }
        }
    }
}
]

Thank you in advance for any assistance anybody can offer!

2

Java Application?

The application exceeded the jvm memory limitation before it exceeded the container restriction.

3

Nope, no java running in the container. Just Apache, PHP, Postfix, cURL?

4

This has now happened a couple more times, and it seems that the only thing consistent each time is the following entries in the host’s syslog talking of networking disconnects and reconnects, anybody have any idea what might be causing this?

Feb 24 20:32:42 vm-name kernel: [13073042.049912] br0: port 8(veth38) entered disabled state
Feb 24 20:32:42 vm-name kernel: [13073042.054083] vethe151fdb: renamed from eth0
Feb 24 20:32:42 vm-name systemd-udevd[30082]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 24 20:32:42 vm-name systemd-networkd[32602]: veth3fcaaa7: Lost carrier
Feb 24 20:32:42 vm-name kernel: [13073042.107760] docker_gwbridge: port 7(veth3fcaaa7) entered disabled state
Feb 24 20:32:42 vm-name systemd-timesyncd[32718]: Network configuration changed, trying to establish connection.
Feb 24 20:32:42 vm-name kernel: [13073042.110738] veth7635164: renamed from eth1
Feb 24 20:32:42 vm-name systemd-timesyncd[32718]: Synchronized to time server 123.456.789.0:123 (ntp.ubuntu.com).
Feb 24 20:32:42 vm-name networkd-dispatcher[20508]: WARNING:Unknown index 1091 seen, reloading interface list
Feb 24 20:32:42 vm-name systemd-timesyncd[32718]: Network configuration changed, trying to establish connection.
Feb 24 20:32:42 vm-name systemd-udevd[30104]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb 24 20:32:42 vm-name systemd-timesyncd[32718]: Synchronized to time server 123.456.789.0:123 (ntp.ubuntu.com).
Feb 24 20:32:42 vm-name systemd-timesyncd[32718]: Network configuration changed, trying to establish connection.
Feb 24 20:32:42 vm-name systemd-timesyncd[32718]: Synchronized to time server 123.456.789.0:123 (ntp.ubuntu.com).
Feb 24 20:32:42 vm-name systemd[1]: Starting resolvconf-pull-resolved.service...
Feb 24 20:32:42 vm-name systemd[1]: Started resolvconf-pull-resolved.service.
Feb 24 20:32:42 vm-name systemd-timesyncd[32718]: Network configuration changed, trying to establish connection.
Feb 24 20:32:42 vm-name kernel: [13073042.335850] docker_gwbridge: port 7(veth3fcaaa7) entered disabled state
Feb 24 20:32:42 vm-name kernel: [13073042.339579] device veth3fcaaa7 left promiscuous mode
Feb 24 20:32:42 vm-name kernel: [13073042.339582] docker_gwbridge: port 7(veth3fcaaa7) entered disabled state
Feb 24 20:32:42 vm-name systemd-timesyncd[32718]: Synchronized to time server 123.456.789.0:123 (ntp.ubuntu.com).
Feb 24 20:32:42 vm-name systemd[1]: Starting resolvconf-pull-resolved.service...
Feb 24 20:32:42 vm-name kernel: [13073042.365143] br0: port 8(veth38) entered disabled state
Feb 24 20:32:42 vm-name systemd-timesyncd[32718]: Network configuration changed, trying to establish connection.
Feb 24 20:32:42 vm-name kernel: [13073042.370782] device veth38 left promiscuous mode
Feb 24 20:32:42 vm-name kernel: [13073042.370785] br0: port 8(veth38) entered disabled state
Feb 24 20:32:42 vm-name systemd-timesyncd[32718]: Synchronized to time server 123.456.789.0:123 (ntp.ubuntu.com).
Feb 24 20:32:42 vm-name systemd[1]: Started resolvconf-pull-resolved.service.
5

We’re seeing the same issue: after upgrading to Docker CE version 18.09.2, build 6247962, containers randomly die with exit code 137, OOMKilled flag set to false. No error messages in our application logs, just these entries in /var/log/messages:

containerd: time="2019-03-05T06:14:26.774849684-05:00" level=warning msg="cleaning up after killed shim" id=0a9bc92a6a0e4ec3cb0d3dd535d922f6512f9d22407ed068ec98e496643ec126 namespace=moby
containerd: time="2019-03-05T08:52:07.895620591-05:00" level=warning msg="cleaning up after killed shim" id=778937aeebdc78aacef0cabecc28fb24272e77bc37359e7b5fb8fe120d403b24 namespace=moby
6

Well I’m glad to hear it’s not just affecting me… I may try upgrading to 18.09.3 to see if anything has been fixed (though there’s nothing in the changelog that explicitly references this issue), however as I haven’t had any drops in a few days I don’t think it’ll be quick to find out whether it’s resolved…

7

cross referencing a matching github issue: https://github.com/moby/moby/issues/38768

8

Same issue on 18.09.3 and swarm service create. I assume that it has something to do with this:

In Docker versions prior to 18.09, containerd was managed by the Docker engine daemon. In Docker Engine 18.09, containerd is managed by systemd. Since containerd is managed by systemd, any custom configuration to the docker.service systemd configuration which changes mount settings (for example, MountFlags=slave ) breaks interactions between the Docker Engine daemon and containerd, and you will not be able to start containers.

9

OMG I am so glad I finally found this thread and that’s not only me having this issue. I have exactly the same problem running mysql 5.7. There is minimum traffic, docker stats does not exceeds 260M and there is 12 GB of memory so no way OOM is reached.

Version: 18.09.2
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 6247962

10

Bump

Is there anything new or does someone has a solution to this peculiar problem ?

11

Hello bobnautic,
just had same result with “OOM Killed: false” and container used more then 90% of memory before exit.

“Created”: “2021-01-26T13:18:58.298270949Z”,
“Path”: “application”,
“Args”: ,
“State”: {
“Status”: “exited”,
“Running”: false,
“Paused”: false,
“Restarting”: false,
“OOMKilled”: false,
“Dead”: false,
“Pid”: 0,
“ExitCode”: 137,
“Error”: “”,
“StartedAt”: “2021-01-26T13:18:58.660261846Z”,
“FinishedAt”: “2021-01-26T13:20:51.838960906Z”

Some details from my env:
Server:
Server Version: 20.10.2
Swarm: inactive
Kernel Version: 4.19.121-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 3
Total Memory: 1.942GiB
Name: docker-desktop

Issue was resolved when I increase RAM from 2Gb to 4Gb.

Basically check memory usage e.g. via “docker stats” command and if it’s high before container exit, try to increase memory for it.

12

I am having the exact same issue… But get this…
Its alpine 3.14 and all its running is aria2c. Thats it. The aria2c gets killed every 5min or so. restarts…
“OOMKilled” is FALSE., return code 137.
I have tried different options in aria2 to see if it effect it no…

But even more interesting is that. 3 systems, all Dell R710 running ubuntu 20.04 native, with docker from ubuntu and from docker directly. All show the exact same problem.

BUT, on a Dell R720, running vmware 6.7, with a VM running ubuntu20.04 (i.e. a test environment) The problem goes away!!

I have been working on this for a week now.

aria2server_1 | /start_aria2_and_monitor.sh: line 15: 8 Killed aria2c --conf-path=/etc/aria2.conf
Aug 10 03:30:04 cm-forbes dockerd[1322]: time=“2021-08-10T03:30:04.758277598Z” level=info msg=“ignoring event” container=2f76ce49faa3423da707711575a0e57c6729fdb25394f35b1b194228ffb35b78 module=libcontainerd namespace=moby topic=/tasks/delete type=“*events.TaskDelete”
Aug 10 03:30:04 cm-forbes containerd[1163]: time=“2021-08-10T03:30:04.758416346Z” level=info msg=“shim disconnected” id=2f76ce49faa3423da707711575a0e57c6729fdb25394f35b1b194228ffb35b78
Aug 10 03:30:04 cm-forbes containerd[1163]: time=“2021-08-10T03:30:04.758523052Z” level=error msg=“copy shim log” error=“read /proc/self/fd/15: file already closed”
cinema-catcher-app_aria2server_1 exited with code 137

This is a mis of the log file for docker and syslog with containerd.

Any help would be great…
James