MacOS 12.4, Docker fatal error...no space left

SITUATION
• Docker for Mac stopped running…error below.
• Docker related terminal command lines give ‘connection refused’ messages.
_
_

TROUBLESHOOTING
• Tried various tutorials, blogs most of which appear to instruct system prune commands.
• System prune commands (and the like) get “connection refused” errors.
• Out of desperation, tried Reset Docker to factory defaults which seems to time out…no change
• Located related files inside #home > Library/Containers/Docker/Data
• Not locating #home/Containers/com.docker.docker ← where is this? all com. directories are com.Apple
_
_

ERROR
Fatal Error

Docker daemon failed to start

[199:14:10:53.189][I] failed to start daemon: Unable to get the TempDir under /var/lib/docker: mkdir /var/lib/docker/tmp: no space left on device.


/Library/Containers/com.docker.docker/Data/docker.raw.sock: connect: connection refused

(post deleted by author)

UPDATE
Removing #user/Library/Containers/Docker/Data/vms/0/data/Docker.raw allowed Docker to at least open up and run. That said everything is back to factory default and would require a rebuild. Swap the Docker.raw file back an original problem is back…looking for a way to At this point, open to rebuilding as that might be a quick way back to normal - HOWEVER - how would I prevent this from happening again in the future?

UPDATE 2
Moved the Docker.raw file to another drive…started a fresh Docker.raw file. This allowed Docker to at least run where I then went to preferences to increase the disk image size. Quick docker, replace the original Docker.raw file and everything has returned to normal (yea!) - then the prune commands worked as expected.

OPEN QUESTION
Unclear how the raw file is filling up or how to better manage. Any source video files, for example, are stored externally. Looking at the app in question, CPU and memory usage can run high at times and the images themselves clock in collectively at around 1.7gig…certainly not the allocated 64gig (which is now 72) so what is happening here? Docker Desktop has a “Clean Up’ function which then states 'Space to be reclaimed 0 Bytes.”

1 Like

I had a similar problem before, but in my case Docker Desktop didn’t stop, so I could uninstall the Docker extension which required about 40GB disk space.

When Docker Desktop doesn’t work, you can (what you did) try using the docker command from terminal. If that doesn’t work either, you can try to log in to the virtual machine:

and see if you can delete at least some megabytes to get Docker Desktop back. The above gist mentions multiple solutions that requires Docker, so that would not work, but using netcat might work. Bret Fisher mentioned one way to use netcat to get a debug shell, but if you try that, you will get some extra characters in the prompt which makes it harder to it. If you scroll down to the comments, there is an other solution which requires socat. (brew install socat)

Use socat to get a debug shell

You will not have write access to the root disk so you can’t install Docker client on that, but you can use nsenter to get inside a container if you know which container saved a large amount of data to its filesystem or you can go to /var/lib/docker/volumes and see if you can delete the content of a volume safely without loosing important data. /var/lib/docker is not on the root disk of the debug shell, so that shell should work, but I am not sure about that. If you are in the VM, you could also try to send raw queries somehow to the Docker socket, but I could not do that.

To prevent this from happening in the first place, you should regurarly delete dangling volumes, dangling images and also images that you don’t really need in the near future and build cache.

An other idea is to search for a tool which which can run in a container and detect when you have for example 10 gigabytes left. Unfortunately even if you do that, in some cases the data could grow faster than you could handle the situation when something is constantly writing the disk.

If Docker desktop can stop when there is not enough space on /var/lib/docker and I was just lucky, it should be fixed in Docker Desktop. It should not depend on Docker’s filesystem. It should be just a graphical interface where you can handle situations like this, since you don’t have full control over the VM, at least not easily. It would be nice for example if you could turn on some kind of emergency mode where you could use Docker Desktop, but the Docker engine would not start and you could increase the size of the virtual disk without the terminal. I will try to reproduce this issue to see if it really stops.

On the other hand, Docker Desktop is a great tool, but we should remember it is NOT Docker running on the host so it would probably be smart to use it only as a development tool and have configurations files and example data with which we can reset everything easily when something like this happens.

if I understood you correctly, you did exactly that since it was not a problem to reset Docker.

I am glad to read that :slight_smile: and thanks for sharing your solution.

Update:

There is actually a Docker client in the VM of Docker Desktop, it is just in an other namespace and you could use “ctr” command to exec into the container that contains the docker client. I don’t have time to go into details, but you can use ctr ns ls to see namespaces then run ctr -n services.linuxkit container ls to see linuxkit containers, then use ctr -n services.linuxkit task exec -t --exec-id docker docker sh to get a terminal. Then you can use the docker client.