Error when trying to commit changes to a container image

To whom it may concern,

I have been working with docker for about a year.

I have multiple small applications running in Docker containers.

I have never had trouble committing changes to my docker images in the past.

But, just recently, when I attempted to commit my latest updates to ANY of my containers, I am getting an error message:

“Error response from daemon: Error processing tar file(exit status 1): chmod /opt: permission denied”

Any ideas?

Has anyone seen this before?

To the best of my knowledge, I am not making any changes to /opt.

None of my containers use or reside in the host OS /opt filesystem.

Remember, I am now getting this same error when I try to commit any of my existing docker images.

Any help would be greatly appreciated.

Joe F.

I am running CentOS 7 Linux as the HostOS.

I am running docker-ce 18.03.1.ce-1.el7.centos.x86_64

I am running containerd.io-1.2.0-3.el7.x85_64

When I attempt to commit changes to any of my current docker images, I get the exact same error message:

Error response from daemon: Error processing tar file(exit status 1): chmod /opt: permission denied

What tar file?

What “chmod /opt”?

Are there any specific Docker error logs that I can look at?

I already have logging set to debug: true, but I don’t see any log entries that help so far.

I do see the following errors in /var/log/messages:

Apr 14 15:39:28 dockerd: time=“2020-04-14T15:39:28.613511857-04:00” level=debug msg=“Calling GET /v1.37/services?filters=%7B%22id%22%3A%7B%22docker.service%22%3Atrue%7D%7D”
Apr 14 15:39:28 dockerd: time=“2020-04-14T15:39:28.613603857-04:00” level=error msg=“Error getting services: This node is not a swarm manager. Use “docker swarm init” or “docker swarm join” to connect this node to swarm and try again.”
Apr 14 15:39:28 dockerd: time=“2020-04-14T15:39:28.613633858-04:00” level=error msg=“Handler for GET /v1.37/services returned error: This node is not a swarm manager. Use “docker swarm init” or “docker swarm join” to connect this node to swarm and try again.”

The server that I’m trying to run these commands on is a standalone CentOS, Docker, Development server.

I am not part of, or joined to any Docker Swarm.

I ran the command “docker swarm init”, and I am no longer getting the “GET” error.

But, now I am getting some more detailed error messages inside of the logs:

dockerd: time=“2020-04-15T11:12:46.645302649-04:00” level=error msg=“Can’t add file /var/lib/docker/overlay2/14e88abe242eab3444fc115732c884329cb03b06fa80d2e81d9e27e188a11058/diff/var/lib/samba/private/msg.sock/8204 to tar: archive/tar: sockets not supported”

Note that there are several sockets in this “msg.sock” subdirectory.

According to the error log messages, we are creating sockets, that can not be included with the commit image and commit command.

Is there a way to omit these sockets, when committing the container image?

I am still having the issue, and here are some more logfile messages:

Apr 16 13:06:05 ServerName dockerd: time=“2020-04-16T13:06:05.825686036-04:00” level=debug msg=“form data: null”
Apr 16 13:06:05 ServerName dockerd: time=“2020-04-16T13:06:05-04:00” level=debug msg=“event published” module=“containerd/tasks” ns=moby topic="/tasks/paused" type=containerd.events.TaskPaused
Apr 16 13:06:05 ServerName dockerd: time=“2020-04-16T13:06:05.850886207-04:00” level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/paused
Apr 16 13:06:05 ServerName dockerd: time=“2020-04-16T13:06:05.873969163-04:00” level=debug msg=“Tar with options on /var/lib/docker/overlay2/14e88abe242eab3444fc115732c884329cb03b06fa80d2e81d9e27e188a11058/diff”
Apr 16 13:06:05 ServerName dockerd: time=“2020-04-16T13:06:05.874671868-04:00” level=debug msg=“Applying tar in /var/lib/docker/overlay2/144f6ac3bbfeac08d102be6655da3e92e9db42637b3a1c722d5acdf4a70e427a/diff”
Apr 16 13:06:54 ServerName dockerd: time=“2020-04-16T13:06:54.386242343-04:00” level=debug msg=“Cleaning up layer 144f6ac3bbfeac08d102be6655da3e92e9db42637b3a1c722d5acdf4a70e427a: Error processing tar file(exit status 1): chmod /opt: permission denied”
Apr 16 13:06:54 ServerName dockerd: time=“2020-04-16T13:06:54-04:00” level=debug msg=“event published” module=“containerd/tasks” ns=moby topic="/tasks/resumed" type=containerd.events.TaskResumed
Apr 16 13:06:54 ServerName dockerd: time=“2020-04-16T13:06:54.583054577-04:00” level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/resumed
Apr 16 13:06:54 ServerName dockerd: time=“2020-04-16T13:06:54.614233089-04:00” level=debug msg=“FIXME: Got an API for which error does not match any expected type!!!: Error processing tar file(exit status 1): chmod /opt: permission denied” error_type="*errors.errorString" module=api
Apr 16 13:06:54 ServerName dockerd: time=“2020-04-16T13:06:54.614270089-04:00” level=error msg=“Handler for POST /v1.37/commit returned error: Error processing tar file(exit status 1): chmod /opt: permission denied”
Apr 16 13:06:54 ServerName dockerd: time=“2020-04-16T13:06:54.614286989-04:00” level=debug msg=“FIXME: Got an API for which error does not match any expected type!!!: Error processing tar file(exit status 1): chmod /opt: permission denied” error_type="*errors.errorString" module=api
Apr 16 13:07:00 ServerName dockerd: time=“2020-04-16T13:07:00-04:00” level=debug msg=“event published” module=“containerd/events” ns=moby topic="/tasks/exit" type=containerd.events.TaskExit

I manually removed the 4 sockets that were causing previous errors, after verifying that these same 4 sockets still existed on the Host, after rebooting the host itself, before even starting Docker, or any other applications.

At this point, I am no longer getting the socket errors.

But, I still can not commit any of my images on this server.

Again, any help would be much appreciated.

Since nobody on these threads seems to have a clue of how to help me.

I will ask for help in a different way.

I have made significant advances / changes to my container image, since being able to save the image, the last time.

I believe I need to reinstall Docker, to clear up the current issue.

But, I am afraid that if I try to remove Docker, it will also remove my current images, to include the one that has not been saved / committed.

Is there an alternate method of saving the current image, aside from using the commit command?

Is there a way to save all of my current images, so that I will not lose any of my images, if / when I reinstall Docker?

Are there any specific directory structures that I need to save?

Do I need to completely uninstall Docker, before reinstalling it?

Is there something I can do to troubleshoot the current Docker installation?

I put too much time and effort into the development of this particular container, to lose all of my work.

Any help would be greatly appreciated.

Does anyone know if there is a way to save the current running image, if you can not commit it?

Does the current running image exist in any kind of temporary directory structure, where I could copy it off somewhere?

Since nobody on this thread seems to have a clue, I will share a little bit about what I found so far.

Docker has the ability to save the entire container, to include what it needs from the underlying host OS, in a tar file.

The command to save the entire container, if you can not commit your current changes is this:

  • docker export Container_Name > Container_Name.tar

Example: docker export myimage > myimage.tar

To verify that it saved your modified files correctly, you can execute the following commands:

From the HostOS:

  • tar -tvf MyImage.tar | grep MyFile

From inside of the container:

  • ls -l Path_2_MyFile

Example: ls -l /var/www/html/MyFile

Note that the returned file size and last modified date should match.

This does not resolve my issue, but at least it gives me a hard backup copy of my currently running environment.