Team,
A running container got deleted automatically.
We had a large container running on an Ubutu server and all of a sudden the container disappeared
O.S : VERSION="16.04.5 LTS (Xenial Xerus
Docker: version 18.06.1-ce, build e68fc7a
Storage Driver: aufs
Observation: We noticed high memory utilization on the host on that particular day, almost 100%, and the utilization started just 30 minutes before the container disappear incident
The logs captured during the analysis from various sources are given below
apport.log
ERROR: apport (pid 195722) Thu Jan 10 01:26:24 2019: 2 18446744073709551615, dump mode 1
ERROR: apport (pid 195722) Thu Jan 10 01:26:24 2019: ignoring implausibly big core limit, treating as unlimited
ERROR: apport (pid 195722) Thu Jan 10 01:26:25 2019: executable: /sbin/auplink (command line “auplink /var/lib/docker/aufs/mnt/ad9ab9225c557c10acf97aab3ae2f57fe56f0ad2fbae1798485b48b627a1f198 flush”)
journel.log
Jan 10 01:26:23 xxxxxx dockerd[5506]: time=“2019-01-10T02:26:23-05:00” level=info msg=“shim reaped” id=8bdd4adeda1b84cc59cbd474d58a19bd2a8e97e63978e0d58f4bc7f3ce3a37e3
Jan 10 01:26:23 xxxxxxx dockerd[5506]: time=“2019-01-10T02:26:23.683906749-05:00” level=info msg=“ignoring event” module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jan 10 01:26:25 xxxxxxx dockerd[5506]: time=“2019-01-10T02:26:25.020765635-05:00” level=warning msg=“Couldn’t run auplink before unmount /var/lib/docker/aufs/mnt/ad9ab9225c557c10acf97aab3ae2f57fe56f0ad2fbae1798485b48b627a1f198” error=“signal: segmentation fault (core dumped)” storage-driver=aufs
Kernel.log
Jan 10 02:26:23 xxxxxx kernel: [2834598.543878] docker0: port 1(veth0c49618) entered disabled state
Jan 10 02:26:23 xxxxx kernel: [2834598.544926] docker0: port 1(veth0c49618) entered disabled state
Jan 10 02:26:23 xxxxxx kernel: [2834598.547121] device veth0c49618 left promiscuous mode
Jan 10 02:26:23 xxxxxx kernel: [2834598.547124] docker0: port 1(veth0c49618) entered disabled state
Jan 10 02:26:24 xxxxxx kernel: [2834599.394252] auplink[195721]: segfault at 7ffd0d1a1108 ip 00007f06362775b9 sp 00007ffd0d1a1110 error 6 in libc-2.23.so[7f063617e000+1c0000]
Syslog.log
Jan 10 02:26:23 xxxxxx dockerd[5506]: time=“2019-01-10T02:26:23-05:00” level=info msg=“shim reaped” id=8bdd4adeda1b84cc59cbd474d58a19bd2a8e97e63978e0d58f4bc7f3ce3a37e3
Jan 10 02:26:23 xxxxxx dockerd[5506]: time=“2019-01-10T02:26:23.683906749-05:00” level=info msg=“ignoring event” module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jan 10 02:26:23 xxxxxxx kernel: [2834598.509967] veth5b6df3a: renamed from eth0
Jan 10 02:26:23 xxxxxxx kernel: [2834598.543878] docker0: port 1(veth0c49618) entered disabled state
Jan 10 02:26:23 xxxxxxxx kernel: [2834598.544926] docker0: port 1(veth0c49618) entered disabled state
Jan 10 02:26:23 xxxxxx kernel: [2834598.547121] device veth0c49618 left promiscuous mode
Jan 10 02:26:23 xxxxxx kernel: [2834598.547124] docker0: port 1(veth0c49618) entered disabled state
Jan 10 02:26:24 xxxxxxx kernel: [2834599.394252] auplink[195721]: segfault at 7ffd0d1a1108 ip 00007f06362775b9 sp 00007ffd0d1a1110 error 6 in libc-2.23.so[7f063617e000+1c0000]
Jan 10 02:26:25 xxxxxxx dockerd[5506]: time=“2019-01-10T02:26:25.020765635-05:00” level=warning msg=“Couldn’t run auplink before unmount /var/lib/docker/aufs/mnt/ad9ab9225c557c10acf97aab3ae2f57fe56f0ad2fbae1798485b48b627a1f198” error=“signal: segmentation fault (core dumped)” storage-driver=aufs
We have noticed below error in the kernel log for a week prior to the incident
kernel: [2492882.553551] SLUB: Unable to allocate memory on node -1 (gfp=0x2080020)
kernel: [2492882.553556] cache: kmalloc-128(1480:8bdd4adeda1b84cc59cbd474d58a19bd2a8e97e63978e0d58f4bc7f3ce3a37e3), object size