During pulling a docker image from docker hub, my device accidentally went off. After I turned it on again, the docker image was shown as downloaded. When I ran my application, it kept failing. So I logged into the container and found all the files inside a folder are 0 byte (see attached screenshot). After I deleted the image and pulled it again, everything is back to normal.
Why the docker engine think the image is downloaded successfully even though there is obvious file corruption? How can I check this before running my application?
A container is made of filesystem layers including the image filesystem layers. So it is possible that some base layers were downloaded but the last layer was just partially so you can run a shell, but other files donāt work. Delete the image completely. Delete all the containers using that image first (check stopped containers), delete the image (the delete command must show ādeletedā not just āuntaggedā and try to pull the image again.
There are some things that cannot be automated or hard to do and may not be worth it. If Docker canāt recognize incomplete downloads, you can run a container from the pulled image and execute some test command in it before replacing the image with the new one. Since you canāt think of evrything, you could have problems like now and manually fix the issue. These things can happen even in highly available systems, but when you have multiple replicas on different nodes, you can disable one node until you fix the issue and your application continues to work.
I just noticed that your original post actually contained what I suggested (deleting and pulling the image again), so I think you did what you could. I never had issue slike this, although I heard about some in this forum too. Iām a little surprised the docker command lets you start a container without running some integrity check.
I would search for existing issues in the Moby repository and open a new issue if there is none.
I have a similar situation. Sometimes my docker images are pulled with random files with 0 bytes size. I have ~20 nodes in k8s and happens randomly on different nodes.
I thought that it was a problem related to Linux kernel Very specific set of circumstances leads to zero-byte (empty) file being created Ā· Issue #1015 Ā· docker/for-linux Ā· GitHub but the affected versions donāt match so I stuck with searching for some resolutions and extra monitoring to catch the case asap and manually fix itā¦
And you use dockerd as container runtime for Kubernetes? If not (and the current default is containerd, not dockerd), it is not just a Docker issue. Where you should report it, could depend on what actually causes it.
Yeah sure I have containerd, I think is some Linux/kernel/dependency problem but with the same result so just mentioning it.
It is not reproducible in any predictable way so no good way to report
I am even more surprised that incomplete layers passed the integrity check for the layer before its content was extracted, and even more surprised this lead to a registered image in the local image cache.
May I ask if there is any new findings on mitigations / automatic detection for your issue? I am running into a very similar issue with containerd as well, but am puzzled on how to even detect it, let alone what is going wrong. So far, I am suspecting the issue is due to pulling images on the same node in parallel, and am trying to avoid that as the next step.
Thank you so much for your reply. I think this definitely could be the root cause for our issue as we need to reboot our VMs at some point as part of our process.
