Question about implications of host shut-down

General
1

Dear all,

our thingsboard/tb-postgres container is running on an Linux server non-stop since more than a year. In the near future, the server will be completely shut down for maintenance for a few hours. How can I identify, whether the data written into the PostgreSQL database will be available afterwards? Is this something I can verify with a docker command, e.g. docker inspect… ?

Thank you

2

Hi @lysandrosmpi.

To confirm if your PostgreSQL container has data, you can use several methods:

  1. Access the Container’s Shell and Use psql:

Connect to the container.

    docker exec -it <container_name_or_id> psql -U <username> <database_name>

(Replace <container_name_or_id>, <username>, and <database_name> with your specific values.)

  • List tables: Once connected to psql, use \dt to list tables in the current database.
  • Query data: Execute SELECT * FROM <table_name>; to view data from a specific table.
  1. Inspect the Docker Volume:

If your PostgreSQL container uses a Docker volume for data persistence, you can inspect the volume’s contents:

  • Identify the volume:

Use docker volume ls to list all volumes and find the one associated with your PostgreSQL container (often named something like postgres_data).

  • Inspect the volume:

Use docker volume inspect <volume_name> to get details about the volume, including its mount point on the host.

  • Browse the host directory:

Navigate to the mount point on your host system to directly view the PostgreSQL data files. Be cautious when directly manipulating these files.

  1. Use a Database Management Tool (e.g., pgAdmin):
  • Connect pgAdmin to your container:

Configure pgAdmin (or another similar tool) to connect to your PostgreSQL container using its IP address and port (e.g., localhost:5432 if port-mapped).

  • Browse the database:

Once connected, you can visually inspect databases, schemas, tables, and their contents within the tool’s interface.

Hope the above answers helped!

3

@tazkysfin’s answer is great for checking the data after rebooting the machine, but I assume you want to know how you can make sure the data will not be lost, so I would like to add some notes.

If you stop a container, the data will still be there after restarting it, even if you wrote everything on the container’s filesystem, which is usually a bad idea. When you shut down a host, it should only stop containers, not delete them. Data is lost only when you save data on the container’s filesystem without using a volume or bind mounting data from a specific folder on the host. So that is what you need to check.

Another important thing is that even if you have the data on the filesystem, if a database server suddenly shuts down without having a chance to finish writing, that could also cause some lost transactions or inconsistent data. Normally, a container gets a signal to shut down, and if it times out, it will be forcefully killed. It could time out because it has too many things to do before shutting down, or because it could not get the stop signal. But the default value is 10 seconds, so that should be enough to stop if it gets the signal. It normally gets it, unless you overwrote the entrypoint or made your image for yourself and had an incorrect entrypoint or command. For example, running the database process in a shell script without using “exec” in that script to replace the shell script in the process list and get PID 1.

I would stop the container first before shutting down the host. Then if it shuts down faster than 10 seconds, it should be fine. Then you can stop the host so you can avoid any kind of additional timeout if there is any at the system level. I am not sure. But if you have just a few containers, I would not expect any issues with timeouts.

Checking the volumes is still a good idea before shutting down the machine, but checking it inside the container would not help. The database would not work if the data was not there, so the question is whether you are using a volume / bind mount or just writing to the container’s filesystem. The previously mentioned docker volume ls can show you if there is any volume, but docker inspect containername can show you if that specific container has that volume and mounted to the right folder inside the container. If there is any automation to delete stopped containers or unused volumes (in case the container is deleted for any reason), that could also delete data, so make sure the restart policy is correct and/or there is no automation deleting containers and volumes.

1 Like