Bug on apt install (permission denied)

esteffin · October 30, 2020, 6:27pm

Hi everyone,
I’m experiencing a failure (PermissionError: [Errno 13] Permission denied) when installing python3 with apt in a docker container (debian based, e,g, ubuntu:20.04).

Has anyone experienced the same problem?
If so, is there a workaround to permanently fix it?

Environment:

Host: Ubuntu 20.04 on x86-64
Docker version: 19.03.13, build 4484c46d9d
Guest OSs: ubuntu:18.04, ubuntu:20.04, debian:9
Issue-failure: apt install -y python3 fails with permission denied during both build and run phases.
Docker info: attached below

To reproduce it is enough to build the following Dockerfile

FROM ubuntu:20.04
RUN apt update && apt install -y python3

When doing docker build . docker fails giving the output attached below.

Instead, if I build the same Dockerfile on Docker for Mac it works successfully.
Similarly python2 can be installed with no problems using apt on the same setting.

docker info output:

Client:
 Debug Mode: false

Server:
 Containers: 15
  Running: 14
  Paused: 0
  Stopped: 1
 Images: 112
 Server Version: 19.03.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: 
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-48-generic
 Operating System: Ubuntu Core 16
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 62.86GiB
 Name: *******
 ID: NIDK:WUWH:D7QW:MKEW:A6CO:5RPM:PEDB:WKNW:OGUB:JM2G:ZYIF:JFQF
 Docker Root Dir: /var/snap/docker/common/var-lib-docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

docker build . output:

Sending build context to Docker daemon  359.4MB
Step 1/2 : FROM ubuntu:20.04
 ---> d70eaf7277ea
Step 2/2 : RUN apt update &&     apt install -y python3
 ---> Running in d1653a1b18be

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [107 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [111 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [98.3 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [1170 B]
Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [438 kB]
Get:7 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [85.3 kB]
Get:8 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [639 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:12 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [857 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [21.6 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [808 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [107 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [4277 B]
Fetched 16.4 MB in 2s (6728 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
1 package can be upgraded. Run 'apt list --upgradable' to see it.

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  file libexpat1 libmagic-mgc libmagic1 libmpdec2 libpython3-stdlib
  libpython3.8-minimal libpython3.8-stdlib libreadline8 libsqlite3-0 libssl1.1
  mime-support python3-minimal python3.8 python3.8-minimal readline-common
  xz-utils
Suggested packages:
  python3-doc python3-tk python3-venv python3.8-venv python3.8-doc binutils
  binfmt-support readline-doc
The following NEW packages will be installed:
  file libexpat1 libmagic-mgc libmagic1 libmpdec2 libpython3-stdlib
  libpython3.8-minimal libpython3.8-stdlib libreadline8 libsqlite3-0 libssl1.1
  mime-support python3 python3-minimal python3.8 python3.8-minimal
  readline-common xz-utils
0 upgraded, 18 newly installed, 0 to remove and 1 not upgraded.
Need to get 7372 kB of archives.
After this operation, 32.7 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2 [1318 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-minimal amd64 3.8.5-1~20.04 [714 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libexpat1 amd64 2.2.9-1build1 [73.3 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-minimal amd64 3.8.5-1~20.04 [1898 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-minimal amd64 3.8.2-0ubuntu2 [23.6 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 mime-support all 3.64ubuntu1 [30.6 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libmpdec2 amd64 2.4.2-3 [81.1 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 readline-common all 8.0-4 [53.5 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libreadline8 amd64 8.0-4 [131 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsqlite3-0 amd64 3.31.1-4ubuntu0.2 [549 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-stdlib amd64 3.8.5-1~20.04 [1671 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8 amd64 3.8.5-1~20.04 [373 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 libpython3-stdlib amd64 3.8.2-0ubuntu2 [7068 B]
Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3 amd64 3.8.2-0ubuntu2 [47.6 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 libmagic-mgc amd64 1:5.38-4 [218 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 libmagic1 amd64 1:5.38-4 [75.9 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 file amd64 1:5.38-4 [23.3 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 xz-utils amd64 5.2.4-1ubuntu1 [82.5 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 7372 kB in 1s (9815 kB/s)
Selecting previously unselected package libssl1.1:amd64.
(Reading database ... 4121 files and directories currently installed.)
Preparing to unpack .../libssl1.1_1.1.1f-1ubuntu2_amd64.deb ...
Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2) ...
Selecting previously unselected package libpython3.8-minimal:amd64.
Preparing to unpack .../libpython3.8-minimal_3.8.5-1~20.04_amd64.deb ...
Unpacking libpython3.8-minimal:amd64 (3.8.5-1~20.04) ...
Selecting previously unselected package libexpat1:amd64.
Preparing to unpack .../libexpat1_2.2.9-1build1_amd64.deb ...
Unpacking libexpat1:amd64 (2.2.9-1build1) ...
Selecting previously unselected package python3.8-minimal.
Preparing to unpack .../python3.8-minimal_3.8.5-1~20.04_amd64.deb ...
Unpacking python3.8-minimal (3.8.5-1~20.04) ...
Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.30.0 /usr/local/share/perl/5.30.0 /usr/lib/x86_64-linux-gnu/perl5/5.30 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.30 /usr/share/perl/5.30 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Setting up libpython3.8-minimal:amd64 (3.8.5-1~20.04) ...
Setting up libexpat1:amd64 (2.2.9-1build1) ...
Setting up python3.8-minimal (3.8.5-1~20.04) ...
Traceback (most recent call last):
  File "/usr/lib/python3.8/py_compile.py", line 218, in <module>
    sys.exit(main())
  File "/usr/lib/python3.8/py_compile.py", line 209, in main
    compile(filename, doraise=True)
  File "/usr/lib/python3.8/py_compile.py", line 172, in compile
    importlib._bootstrap_external._write_atomic(cfile, bytecode, mode)
  File "<frozen importlib._bootstrap_external>", line 126, in _write_atomic
PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.8/__pycache__/__future__.cpython-38.pyc.140554240003856'
dpkg: error processing package python3.8-minimal (--configure):
 installed python3.8-minimal package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 python3.8-minimal
E: Sub-process /usr/bin/dpkg returned an error code (1)
The command '/bin/sh -c apt update && apt install -y python3' returned a non-zero code: 100

tekki · October 31, 2020, 7:39am

This Dockerfile builds fine on my machine (Debian Buster). When did you pull the ubuntu:20.04 image? Maybe you should update it?
To get rid of the “apt does not have a stable CLI interface.” warnings you could use apt-get, but I don’t think this is related to your error.

sebcmp · September 22, 2021, 3:05pm

Tracked this down:

dpkg --configure / post-install runs py_compile
py_compile tries an atomic write
Permissions errors happens when os.open() is called with given flags

>>> import os
>>> os.open("/usr/lib/python3.8/__pycache__/__future__.cpython-38.pyc.139823044469568",
os.O_EXCL | os.O_CREAT | os.O_WRONLY, 0o666)... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.8/__pycache__/__future__.cpython-38.pyc.139823044469568'

Apparently this is a limitation of the Storage Driver: overlay2 ?

Workaround:

cat /etc/docker/daemon.json
{
  "storage-driver": "vfs"
}

github.com

python/cpython/blob/main/Lib/importlib/_bootstrap_external.py#L191

    
      
                  """Replacement for os.path.isabs."""
                  return path.startswith(path_separators)
          
          

          
def _write_atomic(path, data, mode=0o666):
              """Best-effort function to write data to a path atomically.
              Be prepared to handle a FileExistsError if concurrent writing of the
              temporary file is attempted."""
              # id() is used to generate a pseudo-random filename.
              path_tmp = '{}.{}'.format(path, id(path))
              fd = _os.open(path_tmp,
                            _os.O_EXCL | _os.O_CREAT | _os.O_WRONLY, mode & 0o666)
              try:
                  # We first write data to a temporary file, and then use os.replace() to
                  # perform an atomic rename.
                  with _io.FileIO(fd, 'wb') as file:
                      file.write(data)
                  _os.replace(path_tmp, path)
              except OSError:
                  try:
                      _os.unlink(path_tmp)

meyay · September 22, 2021, 7:21pm

The root cause was that the OP did use the Snap package.
It would not and will not happend with the packages from Docker repositories.

The Snap docker package is customized to follow Snap’s philosphy and by doing so breaks Docker. Seems they tried for tame the beast called Docker, but failed misserably in the attempt…

samdporter · June 10, 2022, 2:47pm

Hi,
I’m struggling with this same issue, but don’t understand the answer given above.
Dockerfile:

FROM ubuntu:20.04

RUN apt-get update && apt-get upgrade -y

RUN apt-get install python3-dev -y

This returns the same error given in the original question.
I have created a file such that

~$ cat ../../etc/docker/daemon.json
{
  "storage-driver": "vfs"
}

Is this what was suggested?

meyay · June 10, 2022, 3:06pm

Do not use the docker package from snap. Use snap list docker to check if you installed the snap package. If it is the case remove it and install docker following the instructions from Install Docker Engine | Docker Documentation.

samdporter · June 10, 2022, 5:21pm

snap installation was the issue. Thanks

plars · February 27, 2023, 11:07pm

As mentioned by others, the problem seems to happen when using the overlay2 storage backend in docker. This isn’t limited to the snap, but if you are using the snap then it’s the default configuration. To change this in the snap-installed version of docker:

edit /var/snap/docker/current/config/daemon.json and replace
“overlay2” with “vfs” for the storage-driver
run sudo snap restart docker

rimelek · February 27, 2023, 11:24pm

overlay2 is the default in probably every installation when Docker is installed from the official, supported repository. Snap is also based on containers, so installing Docker as a Snap package was always strange to me. Ubuntu started to move many applications to snap but it is not good for everything. So using Docker as a snap package could be the worst choice of all. It is better to install it from Ubuntu’s APT repository, and the best is to install it from Docker’s APT repository.

It wouldn’t have happened without snap (please, share if you experienced it without snap). I don’t recommend to use it, but thank you for sharing where the daemon.json is in case of snap.

dembal1990 · March 8, 2023, 3:20pm

I have the same problem as you. Any solution ?

Topic		Replies	Views
Error 13 permission denied while building a Dockerfile General	4	23488	June 22, 2023
Docker fail in container when install pyhton, permission denied General docker	1	1171	May 13, 2023
Python error occurs when running docker-compose in ubuntu 22.04 General docker , docker-compose	2	33434	April 19, 2023
Docker giving permission issues in django Docker Hub docker	1	4539	June 26, 2020
SPI permission denied in Raspberry Pi5 Docker - Must sudo Python3 General docker	1	290	May 11, 2024

Bug on apt install (permission denied)

Related Topics