I noticed something curious happening as of today. The image in question filipfilmar/ebook-buildenv:2.1, does not appear to have some binaries in it but only when run on github.
For example, /usr/bin/dot does not seem to be present when run on github, but if I run the container locally as docker run -it filipfilmar/ebook-buildenv:2.1 ls /usr/bin/dot, it is obviously there.
Which error message indicates that dot was not found in the container? Are you sure you are looking for the same file locally that was missing on GitHub?
File would not just disappear from an image. They cannot unless an instruction deletes those files or you mount a folder over an existing folder in the container in build time.
Another option would be that some files are created when an entrypoint executes some code and it will not run when you build an image based the one that you could run locally.
I understand your point, but this is not what I’m observing.
To wit, I have been using this exact image in GitHub actions for years now, and it worked without any issues. I install graphviz into it at build time, which installs /usr/bin/dot, /usr/bin/neato etc. There have been no changes to the image in years. I have not mounted anything over anything.
However, starting yesterday, the image behaves as if graphviz was never installed. If I have an action run dpgk -L graphviz, the response is graphviz is not installed. Very weird.
What you observed is that you had an image that didn’t have the binaries you expected. I was not questioning that observation, but I was not completely right either and I did not have a lot of time to think about the issue more deeply. So Docker itself would not remove or hide files, but something could change files in the docker data root and remove files. I’m not sure why it would only remove some of the files like graphviz, unless it is an antivirus detecting possible threats, but since it is on GitHub, I don’t think it is likely.
So if you did not mount anything in any way and GitHub did not do anything new, and nothing removed files in the Docker data root, then the image on GitHub must be another image just with the same tag. Why it happened, I don’t know.
Without ruling out anything, I would try getting some information about the image on GitHub and about Docker. What version of Docker is running there and what is running on your local machine. What image ID you see on GitHub and what on your local machine.
I also wonder, do you always pull the base image from the registry or is that loaded from a cache? I have seen cases when someone didn’t understand why their image was broken and it turned out something went wrong with the cache. After building the image with --no-cache, it worked again.
Just to sure: you run a github action pipeline, use a bazel action to setup bazel(?), then clone the repo and use bazel to build something. Do I get this right? If so, how would this be a docker related issue?
The error occurs during bezel build and logs Runner: sandbox-fallback in the step that works, and in the step that fails. bezel abstracts away the usage of docker. Thus the solution must be found within bezel.
Isn’t this issue more suited for a github actions forum, or even better a bezel forum?
Well, maybe.I had to start my investigation from somewhere. In my case, bazel does not abstract away the use of docker, although this use is a bit special. In the meantime I think I did limit the investigation to the effect of bazel on the docker image, though the onset of the issue was very sudden. It basically started a few days ago, after many months of no issues at all. I’ll go look elsewhere for the time being.
As for the other questions about image integrity, it seems that the image is pulled from the docker hub each time the action is run, and that the SHA256 of the image is the same as I’m observing locally.