Running systemctl daemon-reload and systemctl restart docker resulted in the loss of container metadata

The mounted disk didn’t show any issues, and I only entered two commands. Has anyone encountered a similar problem? Please help me. :persevering_face:

Please, use </> button to share codes, terminal outputs, error messages instead of sharing screenshots. That helps others to search for similar issues and us to read it more easily and quote parts of the message so we can help you more quickly. You can find a complete guide in the following post: How to format your forum posts


Regarding the issue, if you mount a new disk to where the original data was, you will indeed lose the original data. If I misunderstand it and it is not what happened, please, explain what you did exactly.

Reloading the systemd daemon is neccessary only when you change the systemd service config and if you lose data after that, you either changed the docker data root or any other config like the storage driver that affects where Docker stores data.

The good news is that unless you deleted the original data, it is not likely that you actually lost data. Docker just doesn’t find it and uses a new folder.

First of all, thank you for your reply. I’m sorry, this is my first time posting in the Docker community, and I’m not very familiar with the rules. Let me clarify the issue I encountered and the steps I took. The problem occurred after performing the following steps:

  1. Modified the daemon.json file to:
{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    },
    "registry-mirrors": [
        "https://hub.fast360.xyz",
        "https://hub.rat.dev",
        "https://hub.littlediary.cn",
        "https://docker.kejilion.pro",
        "https://dockerpull.cn",
        "https://docker-0.unsee.tech",
        "https://docker.tbedu.top",
        "https://docker.1panelproxy.com"
    ]
}
  1. Used systemctl daemon-reload and systemctl restart docker to restart, but found that docker ps -a and docker images showed no information.

After repeatedly trying, I remounted the disk but still couldn’t see anything. I then rebooted the server, and docker ps -a still showed no information.

The Docker directory is mounted on the disk /dev/nvme0n1, which has 3.5T capacity, 2.5T used, and 1.1T available (72% usage) at /var/lib/docker. I’ve been using the same one, and I’m sure I could see the container information before the issue occurred.

If this situation isn’t data loss, how can Docker recognize the container information?

Please, don’t create new topics for the same question. I assume it was not intentional, but I deleted one topic and merged another to this topic because you shared more information in it. The spamfilter can block posts and remains hidden until a moderator accepts it, but if you send the same post in multiple topics all of your posts could be automatically blocked.


Thank you for the daemon json. One thing I notice in it is that nvidia-container-runtime is an archived and deprecated project for about one and a half years.

https://github.com/NVIDIA/nvidia-container-runtime

You should follow the recent recommndations:

But it is not likely to cause disappearing containers.

Changing the daemon json does not require reloading the systemd daemon. So if the issue started then, it could be that you changed something long before the systemd daemon reload which you don’t even remember and it was applied then.

Please run the following commands:

docker info
docker version

Review the output before sharing and remove confidential data if any appears (public IP for example)

Also run the following too:

dpkg -l 'docker*' | grep '^ii'
snap list docker

Don’t forget to use code blocks as in your last posts

Explanation why I ask for these outputs:

If you somehow installed another docker daemon accidentaly as it happened to some users already, it is posisble that you just see missing containers because you don’t connect to the right daemon. Although I don’t think you have the Docker snap package as it was not on your first screenshot among snap mounts.

Thank you again for your response. Here is the information I have gathered: I also believe the issue occurred before I restarted the Docker service, but I couldn’t find any logs to support this view.

root@dbcloud:~# docker info
Client: Docker Engine - Community
 Version:    26.1.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 26.1.0
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: nvidia runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.4.0-214-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 128
 Total Memory: 1008GiB
 Name: dbcloud
 ID: 8b8be401-9936-4bb6-b6ad-303e68b90e1c
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://hub.fast360.xyz/
  https://hub.rat.dev/
  https://hub.littlediary.cn/
  https://docker.kejilion.pro/
  https://dockerpull.cn/
  https://docker-0.unsee.tech/
  https://docker.tbedu.top/
  https://docker.1panelproxy.com/
 Live Restore Enabled: false

[DEPRECATION NOTICE]: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
In future versions this will be a hard failure preventing the daemon from starting! Learn more at: https://docs.docker.com/go/api-security/
WARNING: No swap limit support

root@dbcloud:~# docker version
Client: Docker Engine - Community
 Version:           26.1.0
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        9714adc
 Built:             Mon Apr 22 17:07:12 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       c8af8eb
  Built:            Mon Apr 22 17:07:12 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

root@dbcloud:~# dpkg -l 'docker*' | grep '^ii'
ii  docker-buildx-plugin      0.13.1-1~ubuntu.20.04~focal   amd64        Docker Buildx cli plugin.
ii  docker-ce                 5:26.1.0-1~ubuntu.20.04~focal amd64        Docker: the open-source application container engine
ii  docker-ce-cli             5:26.1.0-1~ubuntu.20.04~focal amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras 5:26.0.1-1~ubuntu.20.04~focal amd64        Rootless support for Docker.
ii  docker-compose            1.25.0-1                      all          Punctual, lightweight development environments using Docker
ii  docker-compose-plugin     2.26.1-1~ubuntu.20.04~focal   amd64        Docker Compose (V2) plugin for the Docker CLI.


root@dbcloud:~# snap list docker
error: no matching snaps installed

So nothing seems to be wrong, but I start to understand your previous posts like:

So you meant you always used this disk that you are using now, but after changing the daemon json, you could not find the conainers and images when running docker ps -a and docker images, so that is why you tried remounting the disk hoping it would help.

I see one explanation here. Since the storage driver is overlay2 and you have that folder and only that folder that could be a storage driver, since Docker uses /var/lib/docker as data root, but doesn’t show anything even though t looks like there is a large amount of Data on the disk, the filesystem must be damaged. Either in the entire data root, or at least the container and image metadata folders are corrupted.

Container metadata is under /var/lib/docker/containers/ and image metadata is under ls -lahs /var/lib/docker/image/overlay2/. It would be difficult to restore if this is the case, but you can try to check some files. For example:

file /var/lib/docker/image/overlay2/repositories.json

It should return JSON text data. If it is a json, you can check if it contains images. If you have jq installed on the machine, you can try this, but you don’t have to share the output. You will recognize if it is shows familiar image repository names or an empty list or error message

cat /var/lib/docker/image/overlay2/repositories.json | jq '.Repositories | keys'

You could check other files as well, but it would be long to guide you through all the files here and I don’t remember where I wrote about it, so let’s just test that repositories json first.

I don’t hink it will reveal anything, but please, check if you se multiple contexts here:

docker context ls

Thank you for your response. The repositories.json file is empty, and it seems likely that the data cannot be recovered. Fortunately, the data in the overlay2 directory is still intact, so I can transfer the important files, though it might be a bit troublesome. Thank you very much!

I tested removing the content of the repository json and restarting Docker. It would not even start. Even if I add {} as content, the images are still listed in the output of docker image ls, except there are no tags. Which is actually understandable as that file just maps the tags to IDs. So if you don’t see images either even if without tags, your “images/overlay2” folder is probably broken as well and yes, it could be pretty hard to restore that.