Service Webserver failed to build : Temporary failure/Failed to Fetch

HI @rimelek.Have you check the rootf’s older ? Do you still want me to upload and share the files despite it being the same as the one I shared with the link ?

@rimelek Is there anything else required for me to publish

Only patience. :slight_smile: I don’t have much time recently working on several projects so it takes time to get back to your questions so I can read other questions too. I will check the repo, but since you mentioned you haven’t touched anything in it, do you really need to build your own image when all images are on Docker Hub? Did you find a bug in the image on Docker Hub?

Hi again. When I said I didn’t touch anything , I meant those rootfs folders. They are as is. It’s an open source project so we can make UI changes and copy our changes on top of the base image. But the RUN command itself doesn’t work strangely. My user interface changes starts with 2 COPY commands that comes after the RUN command but the build isn’t reaching to that point at all.

Also ,This is the same dockerFile the repo owners use to build theirs. It is quite strange it isn’t working properly here on my side. I have changed to Linux/Debian 10 and to Ubuntu 20.04 from my earlier 22.04.

There are many above been using this repo to make their own changes in top.

Also for your reference , I posted the same issue on stack. Lots of answers but none worked.

https://stackoverflow.com/questions/75888174/unable-to-build-docker-file-for-jitsi-web

It worked for me

docker build . --build-arg BASE_TAG=stable  -t localhost/test

It worked on macOS using Docker Desktop and I also tried it inside an Ubuntu 22.04 virtual machine successfully.

By the way you don’t have to modify the Dockerfile to use the stable base image. You can just pass build args as I did.

So to summarize

  • The build works for me, so the repo itself is probably okay
  • I tried it on Ubuntu 22.04 so that can’t be the reason either
  • Domain name resolution works perfectly for you in a simple container so the nameserver in your environment must be good.
  • Docker build works for you when you remove everything from the Dockerfile except the FROM keyword, so the DNS server is okay during the build too.
  • I haven’t find anything DNS or APT related in the rootfs folder
  • Besides the rootfs there is an acme.sh script copied into the image, but that can’t really change anything before running apt.

So even though I can’t explain why removing the copy helped still that is my best guess.
Now you try to

  • use the original Dockefile, but replace the rootfs folder with an empty folder.
  • If that works, then try to remove only one file at a time and build again.
  • If you can find the file that somehow affects apt, we can try find out why.
  • If removing the content of the rootfs doesn’t help, then I was wrong and there is something else in the Dockerfile that I didn’t think of.
  • Or maybe something is happenning while you are running the build and by the time you reach the “apt” command, the DNS server doesn’t work. In that case, you can also change that line and use the following Dockerfile.
ARG JITSI_REPO=jitsi
ARG BASE_TAG=latest
FROM ${JITSI_REPO}/base:${BASE_TAG}

RUN cat /etc/resolv.conf

LABEL org.opencontainers.image.title="Jitsi Meet"
LABEL org.opencontainers.image.description="WebRTC compatible JavaScript application that uses Jitsi Videobridge to provide high quality, scalable video conferences."
LABEL org.opencontainers.image.url="https://jitsi.org/jitsi-meet/"
LABEL org.opencontainers.image.source="https://github.com/jitsi/docker-jitsi-meet"
LABEL org.opencontainers.image.documentation="https://jitsi.github.io/handbook/"

ADD https://raw.githubusercontent.com/acmesh-official/acme.sh/2.8.8/acme.sh /opt
COPY rootfs/ /

RUN cat /etc/resolv.conf && \
    apt-dpkg-wrap apt-get update && \
    apt-dpkg-wrap apt-get install -y cron nginx-extras jitsi-meet-web socat curl jq && \
    mv /usr/share/jitsi-meet/interface_config.js /defaults && \
    rm -f /etc/nginx/conf.d/default.conf && \
    apt-cleanup

EXPOSE 80 443

VOLUME ["/config", "/usr/share/jitsi-meet/transcripts"]

I added cat /etc/resolv.conf to the beginning and before the apt command to see if that changes for some reason.

You can also add an apt update command to the beginning and see if that changes anything and if at least the first update works.

ARG JITSI_REPO=jitsi
ARG BASE_TAG=latest
FROM ${JITSI_REPO}/base:${BASE_TAG}

RUN apt-get update

LABEL org.opencontainers.image.title="Jitsi Meet"
LABEL org.opencontainers.image.description="WebRTC compatible JavaScript application that uses Jitsi Videobridge to provide high quality, scalable video conferences."
LABEL org.opencontainers.image.url="https://jitsi.org/jitsi-meet/"
LABEL org.opencontainers.image.source="https://github.com/jitsi/docker-jitsi-meet"
LABEL org.opencontainers.image.documentation="https://jitsi.github.io/handbook/"

ADD https://raw.githubusercontent.com/acmesh-official/acme.sh/2.8.8/acme.sh /opt
COPY rootfs/ /

RUN apt-dpkg-wrap apt-get update && \
    apt-dpkg-wrap apt-get install -y cron nginx-extras jitsi-meet-web socat curl jq && \
    mv /usr/share/jitsi-meet/interface_config.js /defaults && \
    rm -f /etc/nginx/conf.d/default.conf && \
    apt-cleanup

EXPOSE 80 443

VOLUME ["/config", "/usr/share/jitsi-meet/transcripts"]

If the first apt-get update works, but the second doesn’t, you can also install nslookup at the beginning and replace the second apt-get update with an nslookup test command.

ARG JITSI_REPO=jitsi
ARG BASE_TAG=latest
FROM ${JITSI_REPO}/base:${BASE_TAG}

RUN apt-get update &&\
  apt-get install -y --no-install-recommends dnsutils

LABEL org.opencontainers.image.title="Jitsi Meet"
LABEL org.opencontainers.image.description="WebRTC compatible JavaScript application that uses Jitsi Videobridge to provide high quality, scalable video conferences."
LABEL org.opencontainers.image.url="https://jitsi.org/jitsi-meet/"
LABEL org.opencontainers.image.source="https://github.com/jitsi/docker-jitsi-meet"
LABEL org.opencontainers.image.documentation="https://jitsi.github.io/handbook/"

ADD https://raw.githubusercontent.com/acmesh-official/acme.sh/2.8.8/acme.sh /opt
COPY rootfs/ /

RUN nslookup deb.debian.org

EXPOSE 80 443

VOLUME ["/config", "/usr/share/jitsi-meet/transcripts"]

So I don’t see other way then debugging.and trying to find out what file, config makes the build fail.

Hi. Got the culprit. It is the rootfs folder itself. I Commented the line and now At least the RUN commands work but I’m facing a problem with

COPY myapp_web/interface_config.js /defaults/interface_config.js

I’ll post a separate question for the above, but meanwhile it is the rootfs folder for the original error.

I will have to now figure which file and which line is causing the issue. Any help from your end would be great.

EDIT :
I placed the COPY command after the RUN command just to check the problem with the rootfs folder and this is the error that comes up. Hope changing the sequence of COPY command shouldn’t be a problem ?

 => [internal] load build definition from Dockerfile                                                                                                                                       0.0s
 => => transferring dockerfile: 1.15kB                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                                            0.0s
 => [internal] load metadata for docker.io/jitsi/base:stable                                                                                                                               0.3s
 => https://raw.githubusercontent.com/acmesh-official/acme.sh/2.8.8/acme.sh                                                                                                                0.2s
 => [internal] load build context                                                                                                                                                          0.0s
 => => transferring context: 63.66kB                                                                                                                                                       0.0s
 => [1/6] FROM docker.io/jitsi/base:stable@sha256:1834b73753760e91a2775e87680665078ac59f50333c678233abaced7ad81dd3                                                                         0.0s
 => CACHED [2/6] ADD https://raw.githubusercontent.com/acmesh-official/acme.sh/2.8.8/acme.sh /opt                                                                                          0.0s
 => CACHED [3/6] RUN apt-dpkg-wrap apt-get update &&     apt-dpkg-wrap apt-get install -y cron nginx-extras jitsi-meet-web socat curl jq &&     mv /usr/share/jitsi-meet/interface_config  0.0s
 => ERROR [4/6] COPY rootfs/ /                                                                                                                                                             0.0s
------
 > [4/6] COPY rootfs/ /:
------
failed to solve: cannot copy to non-directory: /var/lib/docker/overlay2/hfwnv7o1y4ufuc5lemivsz5q2/merged/defaults

EDIT:
The culprit file is this one : 10-config inside rootfs/etc/cont-init.d/10-config
This is the file that was causing issues.

The following thing I did was:

  • Delete the 10-config file and upload it again later

But after re-uploading 10-config , the the web container builds however while spawning fails to continue with an error: which is supposed to execute the same 10-config bash script returns

Cannot execute bash script --> No such file or directory.

I even tried using dos2unix to see if its a file format problem but the issue is with the file content itself.

It always depends on what the commands do. If those commands use the files that should have been copied, those commands could either fail or do something so the copy fails because you want to copy it over another file/symbolic link. If that is not supported, then the copy will fail.

I think that is an information which could help the developers to find the cause so I wouldn’t ask on stackoverflow but on GitHub directly in an issue

I am not saying that it is a bug of their software, but they know it better. That config file should not cause any DNS error unless it runs during the build (because it is a shell script), which shouldn’t happen or apt itself reads it, but that file doesn’t seem related to APT. It refers to the resolv.conf but only reads it

I posted it on their issues section and the reply I get is

I have no idea how those 2 can be related.

We do daily builds without a problem.We don’t touch resolv.conf, we just read it for using it with Nginx.Also, our nightly unstable builds are built on Ubuntu using a GH action.Sorry but I don’t know how to help you.

Hi Again. I found one more thing. Whatever nameserver I enter for /etc/docker/daemon.json for eg

 { "dns" : [ "114.114.114.114" , "8.8.8.8" ] } 

When I enter the container and open the /etc/resolv.conf . All I see three lines : search,nameserver 127.0.0.11, options edns trustad ndots:0. Seems the changes arent taking place. Seems the file isnt getting updated.

The newly added dns addresses aren’t showing there.

Where do you see that? It can’t be inside the container as it would mean the name server has to be inside the container. I don’t remember when I checked last time so I did now, when you build an image, the container will not use the host network. Without host network, loopback ip addressess can’t work unless the service is running inside the container.

When you build an image, Docker will check the available DNS server on the host and create a new file containing that IP name server address. In case of the old way (without buildkit) it will be copied into the temporary folder of the container which builds the layer created by the RUN instruction in the Dockerfile.
You could actually check it yourself (example later).

In case of buildkit, the resolv.conf will be at /lib/docker/buildkit/executor/resolv.conf.
It will use the same IP address which you could find with this command running on your host:

cat /run/systemd/resolve/resolv.conf

except it will not be a full copy of that file but the ip address will be the same. I can imagine that it can be different too in different Docker versions or different installations, but this is what I see in the official Docker Engine 23.0.1.

Now find the resolv.conf without buildkit

Dockerfile

FROM alpine

# can be smaller number if that number of seconds is enough
RUN sleep 120 

Run

DOCKER_BUILDKIT=0 docker build . -t localhost/test 

Open another terminal and run docker ps to find the latest container started seconds ago. It will be something random like fervent_taussig.

Run

docker container inspect heuristic_pare  --format '{{ .ResolvConfPath }}'

Find the resolv.conf with buildkit

In case of buildkit, the builder containers ran by runc directly without Docker or containerd. So you can find the process by using the ps command:

ps auxf | less

Press /sleep to search for “sleep”.
You will find something like this:

root        4530  0.0  0.2 1311756 9864 ?        Sl   21:12   0:00  \_ runc --log /var/lib/docker/buildkit/executor/runc-log.json --log-format json run --bundle /var/lib/docker/buildk
it/executor/numojawlmo7f2ej1hhjbgrveh numojawlmo7f2ej1hhjbgrveh
root        4541  0.0  0.0   1676     4 ?        Ss   21:12   0:00      \_ sleep 120

I know, hard to see, but it is two lines.
The path of the runc-log.json is interesting too, but there is a folder after --bundle that contains a config.json with all the metadata of the container. You can also filter to that line using grep or here is a ful command (requires jq) to cat the resolc.conf path on the host:

jq -r '.mounts[] | select(.destination == "/etc/resolv.conf") | .source' $(ps aux | grep 'runc-log\.json' | awk '{print $(NF-1)}')/config.json

Or get the content immediately

cat $(jq -r '.mounts[] | select(.destination == "/etc/resolv.conf") | .source' $(ps aux | grep 'runc-log\.json' | awk '{print $(NF-1)}')/config.json)

It won’t unless you restart the docker daemon.

As you said , I looked at the wrong place for the addresses to be updated (Not inside the containers of course).

/etc/resolv.conf

I commented out all the lines except the 1st 3 lines in the dockerfile and just ran & checked if the nameservers which I updated on my host reflected there after I did a restart by

sudo systemctl restart docker.

and what I did see is what I have posted in my previous comment. I will now do as you asked me too now in your previous reply.

Yes. As I checked, the addresses at dameon.json are the same as at the location you specified
lib/docker/buildkit/executor/resolv.conf

@rimelek The guys at Jitsi have given me a cold shoulder with the below comment.

I have no idea how those 2 can be related.
We do daily builds without a problem.We don’t touch resolv.conf, we just read it for using it with Nginx.Also, our nightly unstable builds are built on Ubuntu using a GH action.Sorry but I don’t know how to help you.

I would need to replicate the exact process that you are employing on Ubuntu to make it work.

The only 2 steps before I run docker-compose are:

  • source myEnv/bin/activate
  • sudo su username which is my ubuntu username added to the group docker.

Rest is being done as per the usual docker compose process of the newest verion of docker and docker compose.

@rimelek I ran the script manually outside the docker container on my Ubuntu console. Here is the detailed log.

mkdir: cannot create directory ‘/config’: Permission denied
mkdir: cannot create directory ‘/config’: Permission denied
mkdir: cannot create directory ‘/var/lib/nginx/tmp’: Permission denied
generating self-signed keys in /config/keys, you can replace these with your own keys if required
Generating a RSA private key
......................................................................+++++
................................................+++++
writing new private key to '/config/keys/cert.key'
req: Can't open "/config/keys/cert.key" for writing, No such file or directory
Using Nginx resolver: =127.0.0.53=
10-config: line 92: /config/nginx/nginx.conf: No such file or directory
10-config: line 94: /config/nginx/meet.conf: No such file or directory
10-config: line 99: /config/nginx/ssl.conf: No such file or directory
10-config: line 101: /config/nginx/site-confs/default: No such file or directory
10-config: line 103: /config/config.js: No such file or directory
10-config: line 104: /config/config.js: No such file or directory
cp: cannot stat '/defaults/interface_config.js': No such file or directory```

That script was designed to run in a container as root. I wouldn’t run it on the host.

I would like to help, but I can’t reproduce the issue and out of ideas. I totally understand the Jutsi developers too. They responded the same as I predicted . That file just reads resolv.conf, but I hoped they have an idea how that file could be relevant.

I think all you can do is continue the investigation and test it in different environments. You mentioned Windows before. If you coned the project on Windows, clone it directly on a Linux system and see if that makes a difference.

Hi again. The last statement of yours : cloning it in a Linux system is what I must try. But need to ask : Is it because of the character structure being different in both ? I did use dos2unix ‘filename’ to convert first gen-passwords.sh(was giving problem running it) and then this file itself.

Hi @rimelek Problem solved. In the instructions, it was mentioned “DO NOT CLONE” but download the repo. It now works. I would like to thank you for your time and help all his while.

1 Like

The new line character is different and if the git client on is Windows not configured properly, it changes the line breaks after cloning the project. I knew you tried dos2unix, but this was my only idea. I am glad that you found that “DO NOT clone” statement. It was a really strange error and I still don’t know why that single config file made any difference. It doesn’t matter now, the point is that you solved it :slight_smile:

1 Like

I Still cant figure it why it worked on a single file which is the same. One possible reason I could think of over the past 2 days is that I downloaded the stable release as a .tar.gz package and there wouldnt have been any scope for any file to be in non-unix format?