Unable to restart the container when the data of data-root is lost

General
1

I am running docker on Ubuntu 16.04TLS on cloud environment. Because the os disk of the vm is too small, I reconfigured the docker daemon’s configuration and set the data-root to another disk. But this disk is a TEMPORARY DISK, so the data on this disk may be lost.
Today, all of my data on the temporary disk is lost. And after the vm is rebooted, docker daemon is unable to restart the container which is assigned with the flag --restart=always.
I have following Questions:

  1. I known that it’s due to the lost of the docker’s data in the data-root path. But I want to know is it possible to restart the container by docker daemon if all of the data in data-root is lost or deleted?

  2. I find both container data and image data is stored in data-root. Is it possible to separate them into different path?

I think I could recover the container through following approach if the docker’s data-root is lost.

  1. Start the container which should be aways restarted through systemd. So even all the data of data-root is lost, systemd could try to restart the container.

However this solution is too tricky. And I hope to achieve it within the mechanism provided by docker. Is it possible?

Docker Info on my vm

Containers: 19
Running: 4
Paused: 0
Stopped: 15
Images: 8
Server Version: 18.09.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-1036-azure
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 220.4GiB
Name: myhost
ID: J7PZ:SWPE:KT7U:SNOP:ENF4:PPRS:PALX:NXG6:NRHL:WOIE:CQBU:MIGC
Docker Root Dir: /tmp/disk
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

2

data-root gone == everything Docker related (images, containers, configs, secrets, volume declarations, network declarations and meta-data) is gone.

So you basicly scrub Docker’s “brain” and still expect it to remember how to remedy particular things on it’s own… There is nothing left to restart…

What you want is ti recreate the containers, which should work just fine, that is of course if you stored state in volumes outside your temporary filesystem… Go ahead and create a systemd configuration for your needs.