Hi,
Containers on ubuntu(Ubuntu 20.04.1 LTS x86_64) with custom data-root stop and disapper repeatly.
> cat /etc/docker/daemon.json
{
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
},
"registry-mirrors": ["https://9ivn8uoz.mirror.aliyuncs.com"],
"data-root":"/disk2/docker",
"cgroup-parent": "/docker_limit.slice"
}
> df -h
/dev/sda1 2.7T 444G 2.2T 18% /disk2
I have change the docker data root to a big disk /dev/sda1(mount manually by mount /dev/sda1 /disk2), it works fine for serval months. Two days ago, when I run docker ps -a as usally, all container disapper. I check the data-root, all files are there.
After I run systemctl restart docker, containers come back. But it disapper again and again.
Here is the latest case:
I restart docker service 7 hour ago, when I run docker ps -a just now, no containers there.
> systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-08-30 10:28:08 CST; 7h ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 3722 (dockerd)
Tasks: 67
Memory: 117.6M
CGroup: /system.slice/docker.service
└─3722 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
I come here to seek help, below are infoamation i can provide.
> docker version
Client:
Version: 19.03.8
API version: 1.40
Go version: go1.13.8
Git commit: afacb8b7f0
Built: Wed Oct 14 19:43:43 2020
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.13
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: bd33bbf
Built: Fri Feb 5 15:58:24 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.3.7
GitCommit: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc:
Version: 1.0.0-rc10
GitCommit:
docker-init:
Version: 0.18.0
GitCommit: fec3683
> journalctl -u docker.service
-- Reboot --
8月 30 10:15:15 host3 systemd[1]: Starting Docker Application Container Engine...
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.278020564+08:00" level=info msg="Starting up"
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.280407920+08:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.322278539+08:00" level=info msg="parsed scheme: \"unix\"" module=grpc
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.322347605+08:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.322394899+08:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.322421805+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.326195188+08:00" level=info msg="parsed scheme: \"unix\"" module=grpc
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.326231631+08:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.326251148+08:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.326270628+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.345319476+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.354598503+08:00" level=warning msg="Your kernel does not support cgroup rt period"
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.354627329+08:00" level=warning msg="Your kernel does not support cgroup rt runtime"
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.354637108+08:00" level=warning msg="Your kernel does not support cgroup blkio weight"
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.354645920+08:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.354898276+08:00" level=info msg="Loading containers: start."
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.548578001+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a p>
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.594800699+08:00" level=info msg="Loading containers: done."
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.628490658+08:00" level=info msg="Docker daemon" commit=afacb8b7f0 graphdriver(s)=overlay2 version=19.03.8
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.628979532+08:00" level=info msg="Daemon has completed initialization"
8月 30 10:15:15 host3 systemd[1]: Started Docker Application Container Engine.
8月 30 10:15:15 host3 dockerd[2064]: time="2021-08-30T10:15:15.666250674+08:00" level=info msg="API listen on /run/docker.sock"
8月 30 10:27:32 host3 systemd[1]: Stopping Docker Application Container Engine...
8月 30 10:27:32 host3 dockerd[2064]: time="2021-08-30T10:27:32.580377052+08:00" level=info msg="Processing signal 'terminated'"
8月 30 10:27:32 host3 dockerd[2064]: time="2021-08-30T10:27:32.582012192+08:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
8月 30 10:27:32 host3 dockerd[2064]: time="2021-08-30T10:27:32.582846061+08:00" level=info msg="Daemon shutdown complete"
8月 30 10:27:32 host3 dockerd[2064]: time="2021-08-30T10:27:32.582949988+08:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugin>
8月 30 10:27:32 host3 systemd[1]: docker.service: Succeeded.
8月 30 10:27:32 host3 systemd[1]: Stopped Docker Application Container Engine.
8月 30 10:27:50 host3 systemd[1]: Starting Docker Application Container Engine...
8月 30 10:27:50 host3 dockerd[3722]: time="2021-08-30T10:27:50.317660523+08:00" level=info msg="Starting up"
8月 30 10:27:50 host3 dockerd[3722]: time="2021-08-30T10:27:50.320766573+08:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
8月 30 10:27:50 host3 dockerd[3722]: time="2021-08-30T10:27:50.613001178+08:00" level=info msg="parsed scheme: \"unix\"" module=grpc
8月 30 10:27:50 host3 dockerd[3722]: time="2021-08-30T10:27:50.613059435+08:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
8月 30 10:27:50 host3 dockerd[3722]: time="2021-08-30T10:27:50.613108394+08:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
8月 30 10:27:50 host3 dockerd[3722]: time="2021-08-30T10:27:50.613140603+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
8月 30 10:27:50 host3 dockerd[3722]: time="2021-08-30T10:27:50.617631571+08:00" level=info msg="parsed scheme: \"unix\"" module=grpc
8月 30 10:27:50 host3 dockerd[3722]: time="2021-08-30T10:27:50.617676411+08:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
8月 30 10:27:50 host3 dockerd[3722]: time="2021-08-30T10:27:50.617708255+08:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
8月 30 10:27:50 host3 dockerd[3722]: time="2021-08-30T10:27:50.617733273+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
8月 30 10:27:53 host3 dockerd[3722]: time="2021-08-30T10:27:53.251900501+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
8月 30 10:27:59 host3 dockerd[3722]: time="2021-08-30T10:27:59.503537788+08:00" level=warning msg="Your kernel does not support cgroup rt period"
8月 30 10:27:59 host3 dockerd[3722]: time="2021-08-30T10:27:59.503585206+08:00" level=warning msg="Your kernel does not support cgroup rt runtime"
8月 30 10:27:59 host3 dockerd[3722]: time="2021-08-30T10:27:59.503593716+08:00" level=warning msg="Your kernel does not support cgroup blkio weight"
8月 30 10:27:59 host3 dockerd[3722]: time="2021-08-30T10:27:59.503600893+08:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"
8月 30 10:27:59 host3 dockerd[3722]: time="2021-08-30T10:27:59.503805811+08:00" level=info msg="Loading containers: start."
8月 30 10:28:07 host3 dockerd[3722]: time="2021-08-30T10:28:07.001600829+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a p>
8月 30 10:28:07 host3 dockerd[3722]: time="2021-08-30T10:28:07.924567219+08:00" level=info msg="Loading containers: done."
8月 30 10:28:08 host3 dockerd[3722]: time="2021-08-30T10:28:08.438749711+08:00" level=info msg="Docker daemon" commit=afacb8b7f0 graphdriver(s)=overlay2 version=19.03.8
8月 30 10:28:08 host3 dockerd[3722]: time="2021-08-30T10:28:08.439708145+08:00" level=info msg="Daemon has completed initialization"
8月 30 10:28:08 host3 dockerd[3722]: time="2021-08-30T10:28:08.923334130+08:00" level=info msg="API listen on /run/docker.sock"
8月 30 10:28:08 host3 systemd[1]: Started Docker Application Container Engine.
Thank you!!!