I using Linux based containers to build my projects.
For most project, I download the file on my host and I mount the source folder using the -v option to build them using the tools contained in the container.
The build process is using at some point “sed -i” to to perform in place replacement to some files generated during the build process. These files are located on the mounted value. When the build process run the sed command it output the following error:
`
sed: couldn't open temporary file ./sed7hc859: Permission denied
The file is visible from both the host and the container, but it as no permissions.
smartd@14eaf668b949:/work$ ls -la
ls: sed7hc859: No such file or directory
total 24
drwxrwxrwx 11 smartd hostusers 352 Jun 9 22:09 .
drwxr-xr-x 1 root root 4096 Jun 9 21:34 ..
drwxr-xr-x 17 smartd hostusers 544 Jun 9 21:33 .git
-rw-r--r-- 1 smartd hostusers 23 Jun 9 20:12 .gitignore
-rw-r--r-- 1 smartd hostusers 778 Jun 9 20:12 .gitmodules
drwxr-xr-x 6 smartd hostusers 192 Jun 7 14:02 .vscode
-rw-r--r-- 1 smartd hostusers 3584 Jun 9 20:12 Makefile
-rw-r--r-- 1 smartd hostusers 499 Jun 9 20:12 README.md
-rw-r--r-- 1 smartd hostusers 7 Jun 9 21:36 pat.txt
---------- 1 smartd hostusers 0 Jun 9 22:04 sed7hc859
drwxr-xr-x 3 smartd hostusers 96 Jun 4 16:53 sources
smartd@14eaf668b949:/work$
Note that I don’t have a user permission issue because I am running as the same user in the container and my host computer which is running Mac OSX 12.3.1 for X86_64.
This command works perfectly when I disable VirtioFS and restart the docker desktop for Mac.
Obviously, I don’t like to disable VirtioFS because the build is a lot slower without it.
Do you have any idea why I can’t run sed -i my files?
VirtioFS is still experimental, so I guess we will see more of similar issues like this. I tried what you did and got the same result. Using the same user in the container as on the host might not be enough, since VirtioFS does not show you the file directly from the host. I guess you could change the file in the container, but you can’t do that on the host, therefore it will not allow you to change it inside the container either.
sed creates files without any permissions. VirtioFS allows you to create it because you have permission to write the folder, but when you remove all the permissions of from a file, you will not be able to change it back either since you are no longer have permission to do anything with that file which would not be true if that file was on the container’s filesystem.
Honestly I never use inplace replacements. I like to control the temporary files myself. It you can change the code without too much work, I recommend you to do that. It will not solve your original issue, but at least you could work on what you want.
An other solution could be that you keep how you change the files with sed, but do it in a temporary folder on the container’s filesystem. When you are done, copy the content of the temporary folder back to the mounted folder.
I got the same issue when compiling a large project. It is a little difficult to replace all sed commands.
Finally, I found a solution:
Replace sed with older version, like 4.1.X
why this works?
Possible reasons:
in 4.1.X, sed uses mkstemp to create temporary file, which was replaced with mkostemp
test result as below
root@19e97b91ce9c:/home/tmp/test-sed# echo 'abc' > test.txt
root@19e97b91ce9c:/home/tmp/test-sed# sed-4.2.2 --version
sed-4.2.2 (GNU sed) 4.2.2
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Jay Fenlason, Tom Lord, Ken Pizzini,
and Paolo Bonzini.
GNU sed home page: <http://www.gnu.org/software/sed/>.
General help using GNU software: <http://www.gnu.org/gethelp/>.
E-mail bug reports to: <bug-sed@gnu.org>.
Be sure to include the word ``sed'' somewhere in the ``Subject:'' field.
root@19e97b91ce9c:/home/tmp/test-sed# sed-4.2.2 -i '1 a0' test.txt
sed-4.2.2: couldn't open temporary file ./sed7MHt9t: Permission denied
root@19e97b91ce9c:/home/tmp/test-sed# sed-4.1.5 --version
GNU sed version 4.1.5
Copyright (C) 2003 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE,
to the extent permitted by law.
root@19e97b91ce9c:/home/tmp/test-sed# sed-4.1.5 -i '1 a1' test.txt
root@19e97b91ce9c:/home/tmp/test-sed# cat test.txt
abc
1
root@19e97b91ce9c:/home/tmp/test-sed#
“sed -i” now creates temporary files with correct umask (limited to u=rwx).
Previously sed would incorrectly set umask on temporary files, resulting
in problems under certain fuse-like file systems.
[bug introduced in sed 4.2.1]
sed: set correct umask on temporary files
"sed -i" now creates temporary files with correct umask (limited to u=rwx).
Previously sed would incorrectly set umask, and combined with mkostemp
creating file with mode 0600, the result would be a file with
permission mode 0.
Reported by Dr N.W. Filardo <nwf20@cam.ac.uk>:
https://lists.gnu.org/r/sed-devel/2019-08/msg00000.html
"The net effect is that this patch does not do what it says on the tin:
it does not improve the security story at all. Things continue to
function because the subsequent operations are via f*() APIs, which
take the open file handle, and in particular fchmod() will put the
bits back to something sensible.
However, when running atop, for example, fuse-style filesystems which do
not keep open descriptors to underlying files, this is catastrophic:
the underlying file will have I_SRWXU of zero, and so the filesystem
server will be unable to open the file for the fchmod() and that's
the end of that."
"fuse-overlayfs" is an example of a filesystem with such issues.
This change was made in commit [5156c19b23c41f438bf8658e1b9a43a5ff136835](http://git.savannah.gnu.org/gitweb/?p=sed.git;a=object;h=5156c19b23c41f438bf8658e1b9a43a5ff136835)
and was released in sed 4.2.1.
I was facing the same issue and I fixed it by choosing “gRPC FUSE” file sharing system from the settings instead of VirtioFS. I did not have any issue before updating to the version 4.23.0 (120376). Everything was running good in 4.21.0 version. I did not change anything in the settings before so I do not know if VirtioFS was selected before or it has been selected automatically after upgrading to version 4.23.0. But now it is working fine after changing the file sharing system to “gRPC FUSE” from the settings page.
Can confirm the same behavior for me. Updated to 4.23.0 and inline sed commands started breaking. Can also confirm that switching to gRPC fixed it.
On MacOS 13.5.2 and docker desktop 4.24.0