Docker-compose scheduled temporary tar file in /tmp

General Discussions
1

I’m new to Docker. I’ve also been unable to find much in the way of documentation around this automated activity.

In reviewing the activity of a server running 4 Docker containers, I’ve found that docker-compose automatically creates a temporary tar file in /tmp at random intervals. I’d like to determine the purpose of the creation of this temp tar file in /tmp and why it’s needed.

I’ve also found that .dockerignore file can be used to control / exclude files from this tar file, but I don’t know where the .dockerignore file should live in each container?

Additionally, if there is more than one Docker container running on a single server, can docker-compose processes run concurrently (i.e., can more than one of these tar files be created simultaneously in /tmp)?

Finally, is there a way to control the frequency with which this file is created? I don’t see a cron job for it, so I’d like to know how this automated activity is controlled and what controls it.

I’m attempting to plan for the disk space needed on the aforementioned server. In that goal, I need to better understand the purpose of this automated docker-compose activity and the possible growth size of these tar files to best support the disk space needed for that growth.

Any information is helpful.

Thanks.

2
  1. Docker Compose will not create anything randomly since it doesn’t run as a service.
  2. tar files could be filesystem contexts for docker build. When a client run docker build (without buildkit) it sends the whole context to the server even if the server and the client are the same, so the server can build the docker image.
  3. .dockerignore is not something that you copy inside a container. It is on the host usually next to the Dockerfile tand contains patterns to ignore some files and not to send it to the server.

As I mentioned above, it is not a service, just a client. You can run multiple instances to start different compose stacks but it is not what usually happens.

You need to understand what Docker Compose is (a client, not a service) and how docker build works.

I am sure those tar files created when you run docker build directly or use run docker-compose for which docker-compose.yml defines parameters for docker build.

If you really experience automated and continuous creation of those tar files, then something else is responsible not docker-compose alone.

3

Hi Rimelek,

What you’re suggesting doesn’t mirror the behavior I’m seeing from docker-compose, but there’s more below regarding my supposition. The docker-compose command most certainly seems to run in an automated way, either because of the way the Docker container was originally defined or because of some internal synchronization / retry system.

As I said, I’m new to Docker. Thus, I don’t know what setups are possible within containers. I also had no hand in building the container nor the deployment of it. I’m stepping in much later to review an already built and operating container (4 of them, actually).

What I’ve come to find is a large tar file is being built in the /tmp directory at random, but regular intervals. I stepped through standard SA file determination steps (using ‘file’ and ‘lsof’) on that temporary file and have uncovered docker-compose at the heart this tar file build. I understand that the docker-compose command can be used for backup and restoration manually, but that’s not what seems to be occurring on this system. I’ve spoken to the developer of the container and he has not mentioned launching such backup / restore activity. However, I might not have all of the necessary information here after-the-fact.

It seems that whatever way each container is built is somehow causing docker-compose to produce temporary tar files in /tmp… as I said, regularly. Though, I believe this process may also be failing because /tmp is too small and it’s running the volume out of space.

With that said, it’s possible that a docker-compose command was originally issued manually at some point in the past and because /tmp is being run out of space, it is continuing to retry this request infinitely and failing every time. To me and because I’m coming into this late, it looks like an automated process and, in a way, is automated if it is retrying infinitely. I also have no idea how to cancel it, if so.

Though, I am certain that it is docker-compose creating (and possibly failing to create) this tar file in an automated way. Whether that “automation” is because it’s failing and then retrying, I have no way to easily know this.

The whole reason this has become a problem is that when this docker-compose process runs, it also runs the /tmp volume out of space. This, in turn, affects another Docker container’s software which doesn’t recover from this out-of-space condition without restarting (a separate bug report for another product).

Yes, there are other issues surrounding the volumes built in the operating system that also need to be addressed to prevent such future issues, but docker-compose is compounding this issue because I’ve no way to know the reason why it continually keeps running this tar file build or how to cancel the operation… or even better, how to redirect this build operation to a separate volume with more space.

If you have any other ideas, please let me know.

Thanks, Rimelek.

4

@rimelek nailed it. Something different must be responsible for this.

docker-compose and the docker cli are both just “frontend applications”. One uses a compose file yaml as input, the other command line arguments. Both translate the input to api commands and sends them/it to the docker engine to be processed. Both allow to start containers in attached or detached mode. In attached mode the process will remain in the foreground and show the log output of the container(s) on the console. Neither docker, nor docker-compose will run backed-processes or run as service. The docker engine (and containerd) are the only services that are running in the background.

Can you elaborate on this? I have never heard that claim before :slight_smile:

I am afraid it is not possible to use a sperate volume for the builds only. Though, it is possible to change docker’s root dir to a different volume. When docker builds images (regarless if triggered by docker build or docker-compose build, inside the the build container /tmp is always an ephemeral tmpfs drive.

5

I don’t have. The official docker-compose works as we described. If you have some kind of modified docker-compose, we don’t know how that works.

This was not the first you mentioned building or configuring containers. Is your docker-compose inside a Docker container? Because if you have a container with a docker compose command which is failing while the container is restarting that could look like as if docker-compose were running in the background.

But again, the official docker-compose without additional tools would not do what you described.

6

Hi Meyay,

I can’t explain what it’s doing either, but here are the commands I found in ‘ps’:

root      87516      1  0 16:01 ?        00:00:00 /usr/local/bin/docker-compose -f osquery-server.yml up -d
root      87517  87516 99 16:01 ?        00:00:22 /usr/local/bin/docker-compose -f osquery-server.yml up -d

As you can see, the owner of the launching command is 1 (no parent). Meaning, it’s a dead end for allowing me to link it to a specific container. Something is launching this command on the system. It’s definitely not cron. I’ve looked through every crontab I can find and nothing. That doesn’t mean there’s not a cron system running inside of a container.

Can you elaborate on this? I have never heard that claim before

You don’t create tarballs for any purpose other than for backup and restoration. It might be called ‘building’ in the Docker world, but tar files are created for the expressed intent of backing up data to allow for later restoring it.

I’ve grabbed a copy of the truncated 13G /tmp tarball for review using tar. After review, I’ve found that docker-compose command (shown above) is creating a tar copy of the all 4 containers running on the server. Again, I’ve no idea how the command is being executed or why it’s performing this command at intervals… nor, most importantly, how to stop it.

It’s possible that Wazuh is responsible as it is a security monitoring tool and is running in one of the Docker containers on the server.

7

Each OCI container image consists of a manifest with a list of image layers and the layers themself (either .tar or .tar.gz, I can’t remember). The common use case for images is to encapsualate a single application, its dependencies and a more or less clever entrypoint script that allows to configure a container created based on the image using environment variables.

A container is not a vm, as it has no bios, no kernel, no boot process and usually commes with the minimal set of binaries and configuration files that make up the os - without beeing a full os. Container’s are ment to be ephemaral and not supposed to store state inside the container…

Thus said, I am not sure how Docker’s use case alligns with the view that it would be a general backup/restore mechanism. You could say each image-layer is backed up and restored, but those are partial diffs. It kind of sounds like you are trypting to use it in a scenario it is not designed for.

What you experience is not docker or docker-compose default behavior. This must be an intentional action triggered by something. You might want to inspect each of your containers and see if one or more bind the host’s /tmp folder and/or the docker.sock as “volumes”.

8

If the parent is PID 1 then I suspect it is started by systemd or a similar process manager. Try this in case of systemd on the host:

systemctl status | grep compose

and

sudo grep -r "compose" /etc/systemd

and

sudo grep -r "compose" /etc/init.d
1 Like
9

Hi Rimelek,

This is the key. Thanks. It is a systemd service that’s running this command regularly, apparently. It’s part of the osquery-server.service. In case you’re wondering, Osquery is an abstraction system that turns disparate operating system files and various system information into SQL-like query-able entities. It’s likely Wazuh is consuming Osquery. How creating this huge tar file of Docker’s containers ties into the Osquery system, I’m at a loss. Apparently, it is the Osquery team who is using docker-compose for this purpose. I’ll head over to the Osquery project to find out what the heck they are doing and if there is some way to tame it.

Thanks for your help in helping get to the bottom of this.

10

As a follow up explanation…

The situation ended up as a somewhat complicated series of misconfigurations which began because a Docker image was removed from the system, leaving behind its associated YAML docker-compose file. Because the docker-compose YAML file defined an image build location for when it can’t find its image #, docker-compose begins a build. Because the build: line in the YAML inadvertently defines a location at the root of a large partition (incorrect for building this image), the docker-compose tar process began tarring up that large directory of data into /tmp.

As for why docker-compose continually launches, it’s because of a systemd service definition set as Restart: always. Which means that other than a small cooldown timer between restart attempts, the systemd service “always” attempts to (re)launch docker-compose infinitely (in an attempt to start that “missing” container) resulting in the above build situation in the first paragraph.

Of course, I came into this situation with no history at all and had to back-build my knowledge from various people all giving up tiny fragments of this server’s history over a couple of days.

The appropriate team is now attempting to recover the missing Docker image onto the server.

11

Its definitly far away from beeing the ideal situation to get first impressions with docker.

The systemd approach is rarely used like that and using the filesystems root as build-context is right away nasty…

If you want to dig deeper into docker, I can highly recommend this excellent free self-paced training. It should provide a good foundation about concepts of docker and its usage. I wasted time two classroom trainings, both can not messure up with the self-paced training.

12

Agreed. However, it may be worth calling out a bug + enhancement related to docker-compose image building.

I might suggest the following…

Any Build: line placed in a YAML should require specific files to exist within the specified build directory so that docker-compose can perform a sanity check prior to attempting to build an image. This sanity check should occur prior to launching the creation of a tar file. If such required “preparation” files don’t exist, docker-compose should exit with an error stating that the directory wasn’t properly prepared for building.

Basically, the directory specified under Build: should contain prep files, such as a .dockerbuild or .dockercompose or similar. This sanity check reveals that the person who specified the Build: directory within the YAML intended to specify this directory by properly preparing the build directory in advance.

To allow for this, I’d suggest that the docker-compose command should add a ‘prepare’ command to update the specified directory with the required files to define that directory as a build location. Example: docker-compose prepare /path/to/build/location

Such a required preparation process would have prevented the misconfiguration that led to docker-compose attempting to tar up an unprepared directory. In fact, such lack of sanity checking could be used for malicious intent.

13

It was not Docker Compose. Docker builds the image and you can use Docker Compose to define what you want Docker to do. For me it’s like running rm -rf . in the linux system root. We have to be careful where we run certain commands. Other example could be a backup. There is probably no specific file like “.backupthisdirectory”. :slight_smile: