Multiple projects stopped building on Docker Hub, "Operation not permitted"

Docker Hub
, ,
1

I have several projects linked to Github repositories that build via a Dockerfile in each project. This arrangement has been working fine, but all my builds started failing 100% of the time a while back. I’ve been trying to trace it and realized I didn’t really do anything and the issue it fails on is quite basic.

I’m using the archlinux base image and copying the source of my app to the container using something like:

COPY ./ /src

Next I move into that directory:

WORKDIR /src

Then I try to build the software. Normally this would involve running ./configure, but that dies with a weird error saying “Operation not permitted”. It turn out this isn’t just an issue for GNU Make, even ls can’t read the directory! Specifically the directory, not the stuff inside it. I can run RUN ls -l and get a list of files in the source that got copied over, but I can’t run RUN ls -ld to show the directory properties, it dies like this:

Step 17/28 : RUN ls -ald
---> Running in ec8f9f6c3604
ls: cannot access '.': Operation not permitted

Removing intermediate container ec8f9f6c3604
The command '/bin/sh -c ls -ald' returned a non-zero code: 2

I can run various other commands, but anything that tries to look at the directory itself dies like this. I can even create files in the directory.

Note these same Dockerfiles build just fine on my local system using docker build. What gives? What is different about building them on Docker Hub?

2

YES, thank god somebody else posted an issue with operation not permitted, i have been banging my head against a brick wall for 2 days now with this issue, i get issues during makepkg, if i build locally it builds fine!, started approx 4 days ago for me, snippet from the console output on docker hub:-

e[91m==> WARNING: Using existing $srcdir/ tree
e[0m
==> Starting build()...
e[91mls: cannot access '.': Operation not permitted
configure: error: working directory cannot be determined
e[0m
e[91m==> ERROR: A failure occurred in build().
Error making: makemkv

Of note is that i am also using Arch Linux here as my base OS, so its not improbable for this to be a Arch Linux issue and not a Docker hub issue - still if that were the case then why can i build the image locally?!.

3

I’ve opened an issue report upstream on the Arch Linux docker repository which supposedly holds the tooling used to generate the image in the first place.

4

Thanks for the reply, i will watch the issue with interest.

5

We’ve traced this as far as being related to the latest GNU coreutils package. Everything is fine unless you load the latest release of that package into the container, then directories become un-stat-able. Again this only surfaces when building on Docker Hub, building anywhere else seems fine.

Does anybody have this problem on other distros? Are there even any that have brand new coreutils?

Docker folks does this ring any bells for you?

6

Just a follow up for anybody hitting the same issue:

  1. This is happening because the Docker Hub hosts run old kernels that are not compatible with the latest GNU coreutils, specifically the statx system calls.

  2. The Docker host kernels don’t properly announce their version and hence alert coreutils that they can’t make statx calls, something they would normally gracefully work around.

  3. This is not just Arch Linux, all distributions with new packages are hitting the same issue. Recent builds of Fedora have the same problem, SUSE Tumbleweed, etc.

  4. This isn’t just Docker Hub, Travis is also causing a similar issue.

1 Like
7

I am still getting these errors for OpenSuSE Tumbleweed and Fedora Rawhide. Does anyone know when this is going to b e fixed? Any workaround that I can use? I need containers from bleeding edge distributions.

8

No clue. Docker themselves haven’t so much as acknowledged that there is an issue that I know of. Fedora and Arch Linux have been throwing around hacks to keep coreutils held back. The GNU folks didn’t do anything wrong, their implementation and backwards compatibility shim seem to be okay, this is an issue –as I understand it– with the hypervisor reporting functionality but not delivering, especially when paired with old host kernels and/or security options.

9

Hi all,
I see that the issue mentioned above was deleted and isn’t available anymore.

Is it going to be handled?

10

did anyone found any decent workaround so far?

11

I was experiencing the same issue as described in this post. My builds were working good locally but on our CI (Jenkins) they were failing.
After checking that the possible cause was a new version of coreutils, I downgraded the base image of debian from bulleye to buster. Bullseye is using 8.32 and buster 8.30.
Now the CI is building correctly.

Hope it helps anyone having the same issue.

12

I ran into this issue on Travis CI using Docker with debian:testing, ubuntu:20.10, and fedora:33 base images. Setting the Travis CI dist to bionic resolved the issue.

14

I am having the same issue but with base image alpine:3. Locally building the following works just fine.

# Dockerfile
FROM alpine:3
LABEL maintainer="Arne Ludwig <ludwig@mpi-cbg.de>"

# Install dependencies (build & runtime) via apk
RUN echo 'http://dl-cdn.alpinelinux.org/alpine/edge/main' >> /etc/apk/repositories && \
    echo 'http://dl-cdn.alpinelinux.org/alpine/edge/community' >> /etc/apk/repositories && \
    apk update && \
    apk add build-base git strace libcap zlib-dev && \
    git clone https://github.com/thegenemyers/DAZZ_DB.git && \
    cd DAZZ_DB && \
    git checkout d22ae58d32a663d09325699f17373ccf8c6f93a0 && \
    capsh --print && \
    make

But the same Dockerfile causes “Operation not permitted” on DockerHub:

Cloning into '.'...
Warning: Permanently added the RSA host key for IP address '140.82.112.3' to the list of known hosts.
Submodule 'darg' (https://github.com/a-ludi/darg.git) registered for path 'darg'
Cloning into 'darg'...
Submodule path 'darg': checked out '3ffea1a4f263f1e9f3a4366f5bf08878d99ee3bd'
Switched to a new branch 'feature/singularity'
KernelVersion: 4.4.0-1060-aws
Components: [{u'Version': u'19.03.8', u'Name': u'Engine', u'Details': {u'KernelVersion': u'4.4.0-1060-aws', u'Os': u'linux', u'BuildTime': u'2020-03-11T01:24:30.000000000+00:00', u'ApiVersion': u'1.40', u'MinAPIVersion': u'1.12', u'GitCommit': u'afacb8b7f0', u'Arch': u'amd64', u'Experimental': u'false', u'GoVersion': u'go1.12.17'}}, {u'Version': u'1.2.13', u'Name': u'containerd', u'Details': {u'GitCommit': u'7ad184331fa3e55e52b890ea95e65ba581ae3429'}}, {u'Version': u'1.0.0-rc10', u'Name': u'runc', u'Details': {u'GitCommit': u'dc9208a3303feef5b3839f4323d9beb36df0a9dd'}}, {u'Version': u'0.18.0', u'Name': u'docker-init', u'Details': {u'GitCommit': u'fec3683'}}]
Arch: amd64
BuildTime: 2020-03-11T01:24:30.000000000+00:00
ApiVersion: 1.40
Platform: {u'Name': u'Docker Engine - Community'}
Version: 19.03.8
MinAPIVersion: 1.12
GitCommit: afacb8b7f0
Os: linux
GoVersion: go1.12.17
Starting build of index.docker.io/aludi/dentist:staging...
Step 1/3 : FROM alpine:3
---> 7731472c3f2a
Step 2/3 : LABEL maintainer="Arne Ludwig <ludwig@mpi-cbg.de>"
---> Running in 8a36a64aef78
Removing intermediate container 8a36a64aef78
---> 7dea54777733
Step 3/3 : RUN echo 'http://dl-cdn.alpinelinux.org/alpine/edge/main' >> /etc/apk/repositories && echo 'http://dl-cdn.alpinelinux.org/alpine/edge/community' >> /etc/apk/repositories && apk update && apk add build-base git strace libcap zlib-dev && git clone https://github.com/thegenemyers/DAZZ_DB.git && cd DAZZ_DB && git checkout d22ae58d32a663d09325699f17373ccf8c6f93a0 && capsh --print && make
---> Running in 86a06ad46206
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
v3.13.0-127-g9cfb48d2a1 [https://dl-cdn.alpinelinux.org/alpine/v3.13/main]
v3.13.0-127-g9cfb48d2a1 [https://dl-cdn.alpinelinux.org/alpine/v3.13/community]
v3.13.0-904-g983b3ab5a2 [http://dl-cdn.alpinelinux.org/alpine/edge/main]
v3.13.0-903-g66f163558e [http://dl-cdn.alpinelinux.org/alpine/edge/community]
OK: 24928 distinct packages available
(1/32) Upgrading musl (1.2.2_pre7-r0 -> 1.2.2-r1)
(2/32) Installing libgcc (10.2.1_pre1-r3)
(3/32) Installing libstdc++ (10.2.1_pre1-r3)
(4/32) Installing binutils (2.35.1-r1)
(5/32) Installing libmagic (5.39-r0)
(6/32) Installing file (5.39-r0)
(7/32) Installing libgomp (10.2.1_pre1-r3)
(8/32) Installing libatomic (10.2.1_pre1-r3)
(9/32) Installing libgphobos (10.2.1_pre1-r3)
(10/32) Installing gmp (6.2.1-r0)
(11/32) Installing isl22 (0.22-r0)
(12/32) Installing mpfr4 (4.1.0-r0)
(13/32) Installing mpc1 (1.2.1-r0)
(14/32) Installing gcc (10.2.1_pre1-r3)
(15/32) Installing musl-dev (1.2.2-r1)
(16/32) Installing libc-dev (0.7.2-r3)
(17/32) Installing g++ (10.2.1_pre1-r3)
(18/32) Installing make (4.3-r0)
(19/32) Installing fortify-headers (1.1-r0)
(20/32) Installing patch (2.7.6-r6)
(21/32) Installing build-base (0.5-r2)
(22/32) Installing ca-certificates (20191127-r5)
(23/32) Installing brotli-libs (1.0.9-r3)
(24/32) Installing nghttp2-libs (1.42.0-r1)
(25/32) Installing libcurl (7.74.0-r0)
(26/32) Installing expat (2.2.10-r1)
(27/32) Installing pcre2 (10.36-r0)
(28/32) Installing git (2.30.0-r0)
(29/32) Installing libcap (2.47-r0)
(30/32) Installing strace (5.10-r0)
(31/32) Installing pkgconf (1.7.3-r0)
(32/32) Installing zlib-dev (1.2.11-r3)
Executing busybox-1.32.1-r0.trigger
Executing ca-certificates-20191127-r5.trigger
OK: 210 MiB in 45 packages
e[91mCloning into 'DAZZ_DB'...
e[0m
e[91mNote: switching to 'd22ae58d32a663d09325699f17373ccf8c6f93a0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
e[0m
e[91mHEAD is now at d22ae58 Update Catrack.c
e[0m
Current: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=eip
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Ambient set =
Current IAB: cap_chown,cap_dac_override,!cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,!cap_linux_immutable,cap_net_bind_service,!cap_net_broadcast,!cap_net_admin,cap_net_raw,!cap_ipc_lock,!cap_ipc_owner,!cap_sys_module,!cap_sys_rawio,cap_sys_chroot,!cap_sys_ptrace,!cap_sys_pacct,!cap_sys_admin,!cap_sys_boot,!cap_sys_nice,!cap_sys_resource,!cap_sys_time,!cap_sys_tty_config,cap_mknod,!cap_lease,cap_audit_write,!cap_audit_control,cap_setfcap,!cap_mac_override,!cap_mac_admin,!cap_syslog,!cap_wake_alarm,!cap_block_suspend,!cap_audit_read
Securebits: 00/0x0/1'b0 (no-new-privs=0)
secure-noroot: no (unlocked)
secure-no-suid-fixup: no (unlocked)
secure-keep-caps: no (unlocked)
secure-no-ambient-raise: no (unlocked)
uid=0(root) euid=0(root)
gid=0(root)
groups=0(root),1(bin),
2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
Guessed mode: UNCERTAIN (0)
gcc -O3 -Wall -Wextra -Wno-unused-result -fno-strict-aliasing -o fasta2DB fasta2DB.c DB.c QV.c -lm
e[91mmake: gcc: Operation not permitted
e[0m
e[91mmake: *** [Makefile:11: fasta2DB] Error 127
e[0m
Removing intermediate container 86a06ad46206
The command '/bin/sh -c echo 'http://dl-cdn.alpinelinux.org/alpine/edge/main' >> /etc/apk/repositories && echo 'http://dl-cdn.alpinelinux.org/alpine/edge/community' >> /etc/apk/repositories && apk update && apk add build-base git strace libcap zlib-dev && git clone https://github.com/thegenemyers/DAZZ_DB.git && cd DAZZ_DB && git checkout d22ae58d32a663d09325699f17373ccf8c6f93a0 && capsh --print && make' returned a non-zero code: 2

Furthermore, if I copy the cap as shown in the log to my local machine. I do not get the error.

I read the issue with Arch Linux and used the oldest Alpine that would resolve my packages. But that does not help.

At some point I also tried to use a current Ubuntu image but it failed with the same error.