Store Data to local disk

I need some suggestions as I am new in Docker World and this is my first post in this forum. I am trying to develop an image with the combination of 5-6 software and tools and I have already successfully build it. It combines something like following

  1. A Database (Postgresql)
  2. A Node Server
  3. An API server
  4. A Go Server
  5. Others are the data processor tool

The image will run through a few Arguments such as process, start, stop, backup, restore and update commands. Here is the utilization of those commands separately

process - When the image will run with process command few data will be automatically processed from different online file sources and will be saved on a database and a few others will be save as a file on disk.
start - This will start all servers along with postgres DB
stop - This will stop all servers along with postgres DB
backup - All database tables and processed files will be back up to a local directory
restore - Search for backup data in a given location and restore all the data then start the servers with those data
update - Delete the existing data and process data again

I need suggestions on the following topics:

  1. After processing data the data will not be removed if the container is even closed. How can I do that?
  2. I have already written the backup code. But I want to make the backup in a local directory, How?
  3. Is it a good practice to merge 5-6 or more tools together in a container or should I use docker-compose?
  4. Is the Argument/Command (process, start, stop, backup, restore and update) architecture a good practice? if not what will be the best concept?

Thanks in Advance…

After the container is running… I am not certain you can just copy/move data to the host machine for persistent storage.
I have a similar question about best practices in regard to configuration of the container during and after run time.
Topics:
1.) docker run -v; allows your container to mount a host volume or directory. -v : ; docker run -v /tmp/hostdir/:/home/container/data/ myimage:tag
now anything that gets put during run time into /home/container/data/ will show up in /tmp/hostdir/
Docker volume create can create a volume in a docker directory where data will persist as well…

2.) see 1… but how will you tell the container after it is running to do the back up…I am not aware of a way to run through docker an entry point script call in your case “backup” after the container is started…

3.)No t really… each container should be a single thing, separation of concerns stuff… but you can if you think that is best to wrap you head around what is happening.

4.) I had this question also and no one answered… we are thinking like applications and not containers…
The entry point script is for setting up run time configurations… After the container is running, sending it more commands cannot happen through docker… unless you have a script that you want to access on the container through docker exec … docker exec /home/user/backupscript.sh
but you may as well expose an API call or something to trigger the event through a simple REST app…

Cheers,
Hope this helps…
Check out my work it might help…
https://hub.docker.com/r/harperdb/hdb

1 Like

Volumes are not limited to store data in /var/lib/docker/volumes/{volumename}. They can do more than that: they can be used to mount remote fileshares or bind any local path (this is not the same as a bind-mount). Also -v /host/path:/container/path is not realy a volume, it is a bind-mount, which does nothing else than the shell command mount --bind {source path} {target path}. A bind-mount gets mounted on top of the target folder, which makes the previous content of the target folder unavailable to the container…

Like zacharyhdb wrote: Seperating the services by concern is the way to go. Use docker-compose.yml to orchestrate your container zoo.

When it commes to your commands:
Make sure to understand the difference of entrypoint scripts and commands in docker. Make sure you understand that your commands can only be provided when the container is created. Due to that your stop command doesn’t make sense.

Take a look at how sameersbn solved it for his gitlab image: https://github.com/sameersbn/docker-gitlab/blob/master/entrypoint.sh. He provides examples that create a oneshot container to execute a command and get removed again when finished: docker run --name gitlab -it --rm [OPTIONS] sameersbn/gitlab:13.0.3 app:rake gitlab:backup:create

I realy like that you think about the operation part of the process as well. Seeing someone anticipates backup/restore in the image design is realy nice.

2 Likes

Thanks. I will try to make a github repo and provide the link here after a few days. Actually there is a few more tools I am going to use. Most of the software dependencies are almost the same. So if I build those softwares separately with docker-compose then the image will cost a huge image on the local disk. Trying to do something experimental.

You can build your own base image that covers a common base for all images. Then use it as the base image of your final images.

In case you missed in the sameersbn/docker-gitlab example, that the real functionality is actualy sourced from this file https://github.com/sameersbn/docker-gitlab/blob/master/assets/runtime/functions. The entrypoint script and this make up the command handling for the container.