Changing the order of RUN execution in the Dockerfile

General
1

Hi there,

I am having a problem with changing the order of RUN execution in the Dockerfile, which is not reflected in the results of the execution.

Please teach me what documents can be used as references and what keywords should be looked up?

ref: GitHub - hytdsh/my-open-interpreter: My Open Interpreter

The following writing works as expected.

...snip...
USER openint
WORKDIR /home/openint

RUN mkdir bin && bash -c '(echo && echo "PATH=\"$HOME/bin:$PATH\"")' >>${HOME}/.bashrc
COPY 3.5-turbo.py 3.5-turbo-16k.py 4.py 4-32k.py tokenizer.py bin/

RUN python -m venv python \
    && . python/bin/activate \
    && pip install \
        google-search \
        matplotlib \
        numpy \
        open-interpreter \
        pandas \
        tiktoken
RUN bash -c '(echo && echo "source ~/python/bin/activate")' >>${HOME}/.bashrc

~/work/open-interpreter$ docker compose build
~/work/open-interpreter$ docker compose up -d
~/work/open-interpreter$ docker compose exec openint bash
(python) openint@a18424c997a4:~$

(python) openint@a18424c997a4:~$ which python
/home/openint/python/bin/python

(python) openint@a18424c997a4:~$ which pip
/home/openint/python/bin/pip

(python) openint@a18424c997a4:~$ which interpreter
/home/openint/python/bin/interpreter

(python) openint@a18424c997a4:~$ pip list
Package            Version
------------------ ------------
...snip...
open-interpreter   0.1.4
...snip...

(python) openint@a18424c997a4:~$ ls -la ~/python/bin
total 100
drwxr-xr-x 2 openint openint 4096 Sep 21 12:45 .
drwxr-xr-x 6 openint openint 4096 Sep 21 12:45 ..
-rw-r--r-- 1 openint openint 9033 Sep 21 12:44 Activate.ps1
-rw-r--r-- 1 openint openint 1991 Sep 21 12:44 activate
-rw-r--r-- 1 openint openint  917 Sep 21 12:44 activate.csh
-rw-r--r-- 1 openint openint 2197 Sep 21 12:44 activate.fish
-rwxr-xr-x 1 openint openint  227 Sep 21 12:45 dotenv
-rwxr-xr-x 1 openint openint  231 Sep 21 12:45 f2py
-rwxr-xr-x 1 openint openint  232 Sep 21 12:45 fonttools
-rwxr-xr-x 1 openint openint  228 Sep 21 12:45 gp
-rwxr-xr-x 1 openint openint  254 Sep 21 12:45 huggingface-cli
-rwxr-xr-x 1 openint openint  223 Sep 21 12:45 interpreter
-rwxr-xr-x 1 openint openint  235 Sep 21 12:45 markdown-it
-rwxr-xr-x 1 openint openint  259 Sep 21 12:45 normalizer
-rwxr-xr-x 1 openint openint  236 Sep 21 12:45 openai
-rwxr-xr-x 1 openint openint  236 Sep 21 12:44 pip
-rwxr-xr-x 1 openint openint  236 Sep 21 12:44 pip3
-rwxr-xr-x 1 openint openint  236 Sep 21 12:44 pip3.11
-rwxr-xr-x 1 openint openint  229 Sep 21 12:45 pyftmerge
-rwxr-xr-x 1 openint openint  230 Sep 21 12:45 pyftsubset
-rwxr-xr-x 1 openint openint  230 Sep 21 12:45 pygmentize
lrwxrwxrwx 1 openint openint   15 Sep 21 12:44 python -> /usr/bin/python
lrwxrwxrwx 1 openint openint    6 Sep 21 12:44 python3 -> python
lrwxrwxrwx 1 openint openint    6 Sep 21 12:44 python3.11 -> python
-rwxr-xr-x 1 openint openint  222 Sep 21 12:45 tqdm
-rwxr-xr-x 1 openint openint  227 Sep 21 12:45 ttx

The order of execution of RUN was swapped.

...snip...
USER openint
WORKDIR /home/openint

RUN python -m venv python \
    && . python/bin/activate \
    && pip install \
        google-search \
        matplotlib \
        numpy \
        open-interpreter \
        pandas \
        tiktoken
RUN bash -c '(echo && echo "source ~/python/bin/activate")' >>${HOME}/.bashrc

RUN mkdir bin && bash -c '(echo && echo "PATH=\"$HOME/bin:$PATH\"")' >>${HOME}/.bashrc
COPY 3.5-turbo.py 3.5-turbo-16k.py 4.py 4-32k.py tokenizer.py bin/

~/work/open-interpreter$ docker compose build
~/work/open-interpreter$ docker compose up -d
~/work/open-interpreter$ docker compose exec openint bash
(python) openint@99c95f53739a:~$

(python) openint@99c95f53739a:~$ which python
/usr/bin/python

(python) openint@99c95f53739a:~$ which pip
/usr/bin/pip

(python) openint@99c95f53739a:~$ which interpreter
(python) openint@99c95f53739a:~$

(python) openint@99c95f53739a:~$ pip list
Package    Version
---------- -------
pip        23.0.1
setuptools 66.1.1
wheel      0.38.4

(python) openint@99c95f53739a:~$ ls -la ~/python/bin
total 100
drwxr-xr-x 2 openint openint 4096 Sep 21 09:32 .
drwxr-xr-x 6 openint openint 4096 Sep 21 09:32 ..
-rw-r--r-- 1 openint openint 9033 Sep 21 09:31 Activate.ps1
-rw-r--r-- 1 openint openint 1991 Sep 21 09:31 activate
-rw-r--r-- 1 openint openint  917 Sep 21 09:31 activate.csh
-rw-r--r-- 1 openint openint 2197 Sep 21 09:31 activate.fish
-rwxr-xr-x 1 openint openint  227 Sep 21 09:32 dotenv
-rwxr-xr-x 1 openint openint  231 Sep 21 09:32 f2py
-rwxr-xr-x 1 openint openint  232 Sep 21 09:32 fonttools
-rwxr-xr-x 1 openint openint  228 Sep 21 09:32 gp
-rwxr-xr-x 1 openint openint  254 Sep 21 09:32 huggingface-cli
-rwxr-xr-x 1 openint openint  223 Sep 21 09:32 interpreter
-rwxr-xr-x 1 openint openint  235 Sep 21 09:32 markdown-it
-rwxr-xr-x 1 openint openint  259 Sep 21 09:32 normalizer
-rwxr-xr-x 1 openint openint  236 Sep 21 09:32 openai
-rwxr-xr-x 1 openint openint  236 Sep 21 09:31 pip
-rwxr-xr-x 1 openint openint  236 Sep 21 09:31 pip3
-rwxr-xr-x 1 openint openint  236 Sep 21 09:31 pip3.11
-rwxr-xr-x 1 openint openint  229 Sep 21 09:32 pyftmerge
-rwxr-xr-x 1 openint openint  230 Sep 21 09:32 pyftsubset
-rwxr-xr-x 1 openint openint  230 Sep 21 09:32 pygmentize
lrwxrwxrwx 1 openint openint   15 Sep 21 09:31 python -> /usr/bin/python
lrwxrwxrwx 1 openint openint    6 Sep 21 09:31 python3 -> python
lrwxrwxrwx 1 openint openint    6 Sep 21 09:31 python3.11 -> python
-rwxr-xr-x 1 openint openint  222 Sep 21 09:32 tqdm
-rwxr-xr-x 1 openint openint  227 Sep 21 09:32 ttx

RUN that were deemed necessary were added at the end.

...snip...
USER openint
WORKDIR /home/openint

RUN python -m venv python \
    && . python/bin/activate \
    && pip install \
        google-search \
        matplotlib \
        numpy \
        open-interpreter \
        pandas \
        tiktoken
RUN bash -c '(echo && echo "source ~/python/bin/activate")' >>${HOME}/.bashrc

RUN mkdir bin && bash -c '(echo && echo "PATH=\"$HOME/bin:$PATH\"")' >>${HOME}/.bashrc
COPY 3.5-turbo.py 3.5-turbo-16k.py 4.py 4-32k.py tokenizer.py bin/

RUN . python/bin/activate

~/work/open-interpreter$ docker compose build
~/work/open-interpreter$ docker compose up -d
~/work/open-interpreter$ docker compose exec openint bash
(python) openint@3e8f95436d8a:~$

(python) openint@3e8f95436d8a:~$ which python
/usr/bin/python

(python) openint@3e8f95436d8a:~$ which pip
/usr/bin/pip

(python) openint@3e8f95436d8a:~$ which interprter
(python) openint@3e8f95436d8a:~$

(python) openint@3e8f95436d8a:~$ pip list
Package    Version
---------- -------
pip        23.0.1
setuptools 66.1.1
wheel      0.38.4

(python) openint@3e8f95436d8a:~$ ls -la ~/python/bin
total 100
drwxr-xr-x 2 openint openint 4096 Sep 21 09:32 .
drwxr-xr-x 6 openint openint 4096 Sep 21 09:32 ..
-rw-r--r-- 1 openint openint 9033 Sep 21 09:31 Activate.ps1
-rw-r--r-- 1 openint openint 1991 Sep 21 09:31 activate
-rw-r--r-- 1 openint openint  917 Sep 21 09:31 activate.csh
-rw-r--r-- 1 openint openint 2197 Sep 21 09:31 activate.fish
-rwxr-xr-x 1 openint openint  227 Sep 21 09:32 dotenv
-rwxr-xr-x 1 openint openint  231 Sep 21 09:32 f2py
-rwxr-xr-x 1 openint openint  232 Sep 21 09:32 fonttools
-rwxr-xr-x 1 openint openint  228 Sep 21 09:32 gp
-rwxr-xr-x 1 openint openint  254 Sep 21 09:32 huggingface-cli
-rwxr-xr-x 1 openint openint  223 Sep 21 09:32 interpreter
-rwxr-xr-x 1 openint openint  235 Sep 21 09:32 markdown-it
-rwxr-xr-x 1 openint openint  259 Sep 21 09:32 normalizer
-rwxr-xr-x 1 openint openint  236 Sep 21 09:32 openai
-rwxr-xr-x 1 openint openint  236 Sep 21 09:31 pip
-rwxr-xr-x 1 openint openint  236 Sep 21 09:31 pip3
-rwxr-xr-x 1 openint openint  236 Sep 21 09:31 pip3.11
-rwxr-xr-x 1 openint openint  229 Sep 21 09:32 pyftmerge
-rwxr-xr-x 1 openint openint  230 Sep 21 09:32 pyftsubset
-rwxr-xr-x 1 openint openint  230 Sep 21 09:32 pygmentize
lrwxrwxrwx 1 openint openint   15 Sep 21 09:31 python -> /usr/bin/python
lrwxrwxrwx 1 openint openint    6 Sep 21 09:31 python3 -> python
lrwxrwxrwx 1 openint openint    6 Sep 21 09:31 python3.11 -> python
-rwxr-xr-x 1 openint openint  222 Sep 21 09:32 tqdm
-rwxr-xr-x 1 openint openint  227 Sep 21 09:32 ttx

Regards,

2

Sorry, I could not perfectly follow your description. Please, point out what you expected and what you got instead. Sharing the list of files multiple times means we have to figure out what is wrong like when we try to find the differences in two almost identical pictures.

What I can tell you is that for changing the PATH variable you should set the new PATH in an ENV instruction like:

ENV PATH=$HOME/bin:$PATH

Don’t change variables in a bashrc in containers.

Activating a python environment in a RUN instruction and doing nothing in that layer also makes no sense. Activating an environment is basically settting the correct paths in PATH so the python interpreter can be found. Every RUN instruction is a new container so nothing will remain for the container unless you save it to the filesystem.

3

@rimelek Thank you for your reply!

What I expected (and what I have achieved once)

  1. I can attach to openint container using bash.
  2. I can use interpreter command in the container.

At this time, the situation was as follows

Dockerfile: The order of execution is RUN mkdir block, then RUN python -m venv block.

RUN mkdir bin && add PATH line to ${HOME}/.bashrc
COPY script files bin/

RUN python -m venv python ...
RUN add `source python-venv-activate-script` to ${HOME}/.bashrc

And python pip interpreter commands are

(python) openint@a18424c997a4:~$ which python
/home/openint/python/bin/python (venv local python)

(python) openint@a18424c997a4:~$ which pip
/home/openint/python/bin/pip (venv local pip)

(python) openint@a18424c997a4:~$ which interpreter
/home/openint/python/bin/interpreter (interpreter in venv)

(python) openint@a18424c997a4:~$ echo $PATH
/home/openint/python/bin:/home/openint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

What I got instead when I changed the order of execution

  1. I can attach to openint container using bash.
  2. I can NOT use interpreter command in the container.

Dockerfile: The order of execution is RUN python -m venv block, then RUN mkdir block.

RUN python -m venv python ...
RUN add `source python-venv-activate-script` to ${HOME}/.bashrc

RUN mkdir bin && add PATH line to ${HOME}/.bashrc
COPY script files bin/

And python pip interpreter commands are

(python) openint@99c95f53739a:~$ which python
/usr/bin/python (system wide python)

(python) openint@99c95f53739a:~$ which pip
/usr/bin/pip (system wide pip)

(python) openint@99c95f53739a:~$ which interpreter
(python) openint@99c95f53739a:~$ (not found in PATH)

(python) openint@99c95f53739a:~$ echo $PATH
/home/openint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

However, files related to the interpreter command are placed.

(python) openint@99c95f53739a:~$ ls -la ~/python/bin
...snip
-rwxr-xr-x 1 openint openint  223 Sep 21 09:32 interpreter
...snip...

Why I edit .bashrc

I want to use command /home/openint/bin/* when I attach to openint container using bash.
My understanding is that when attaching to a container with the docker compose exec openint bash command, .profile is not executed, but .bashrc is.

With ENV,

ENV PATH=$HOME/bin:$PATH
RUN mkdir bin
COPY script files bin/

RUN python -m venv python ...
RUN add `source python-venv-activate-script` to ${HOME}/.bashrc

I got as follows.

(python) openint@87b7b85f1863:~$ echo $PATH
/home/openint/python/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

The /home/openint/python/bin is supposed to be added by source python-venv-activate-script.
The /bin to the right of /home/openint/python/bin is probably due to ENV, but ${HOME} is missing.

so I added one line echo "PATH=$HOME/bin:$PATH" to the .bashrc.

Why my third attempt did not work

Activating a python environment in a RUN instruction and doing nothing in that layer also makes no sense.

I think I know why my third attempt did not work.

RUN python -m venv python ...
RUN add `source python-venv-activate-script` to ${HOME}/.bashrc

RUN mkdir bin && add PATH line to ${HOME}/.bashrc
COPY script files bin/

RUN . python/bin/activate <<<MAKES NO SENSE

and what I found

When I reviewed .bashrc again, I found the following in .bashrc.

in 1st attempt

PATH="/home/openint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

source ~/python/bin/activate

in 2nd attempt

source ~/python/bin/activate

PATH="/home/openint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

PATH in .bashrc is FIXED string. Not PATH=$HOME/bin:$PATH.
The one that didn’t work (2nd attempt) overwrote the PATH variable set in source ~/python/bin/activate with the fixed string in the next line.

Escaping $ now works as intended.

- echo && echo "PATH=\"$HOME/bin:$PATH\""
+ echo && echo "PATH=\"\$HOME/bin:\$PATH\""

Regards,

4

Did you change the actual code into a pseudocode?
Please, don’t. The order of the RUN instructions does nothing just changes the order of the commands executed. It has nothing to do with the fact that you use Docker. If an command is not found or another command was found that means you either not installed the command even if the folder exists or you overwrote the bashrc file which shouldn’t even be used, instead of appending a new value.

If you change the original commands, I have to look at your original commands and your new post as well to understand what you are doing.

None of those files should be changed at all.

Indeed, my previous advice was wrong as I sugested using the $HOME variable without a shell. You can use variables existing variables in an ENV instruction, but it seems HOME is not available. It actually makes sense as Docker would need to read it from the user properties during build to make it available during the interpretation of the Dockerrfile. So if you want to use HOME, you need to define that as well as you defined or redefined PATH.

If you check your paths you can see you had “/bin” in there two times. Once at the end and once at the beginning. The one at the beginning was the result of ENV PATH=$HOME/bin:$PATH

1 Like