I’m attempting to build an image containing Vivado. This requires copying Vivado’s installer into the image. The install is quite large (~90GB).
It seems others have had success with this:
When I attempt to copy the file, it is only partially copied. During the build, I see output showing transferring context: #.#GB. The highest I have seen this get is 62GB. That’s 30GB shy of what I’d expect. Attempting to extract the file in the container results in tar: Unexpected EOF in archive. The file size is significantly smaller in the image indicating that it has not been completely copied over.
I’m quite certain I’m not running out of storage space. df -h shows I’ve got >200G of available space on the drive.
Is there some setting I need to change to allow for building these disgustingly large images?
Don’t copy such a large file, just mount it into the build container:
If you copy the file you would never be able to delete it from the image even after you finished the installation. You could only hide it making the image 90GB larger than it should be.
I don’t know Vivado, but if something is so large, I’m not sure that should be run in a Docker container.
If I go the mount route, it will require me to install multiple versions of Vivado on multiple build nodes. Not out of the question, but it’d be nice if I could avoid it.
I’m not concerned about image size being large. The image is just a build environment that artifacts will be copied out of.
Looks like I underestimated the amount of storage required to make this image. After clearing an additional 200GB of space, things seem to be working.
I’ve also utilized multi-stage build to remove the installer from the image size. I know I said I wasn’t too worried about it above, but a smaller image never hurt anyone.
I’m not sure what you mean. I suggested mounting the installer only. Once the image is built, you could even copy it to other nodes or push it to a registry if you have one. You wouldn’t have to install anything on the nodes, just have the installer file there.
Yes, if you don’t mount, that is a good idea too, but you will still have the installer on the host and in Docker build cache as well
Sorry @rimelek, you’re 100% correct. I didn’t quite understand your reply before, but I do now. Thought you were telling me to mount it as a volume. Helps to read the documentation you so helpfully linked!
This is a much better solution than throwing more hard drive space at it. Thanks!