Docker-daemon-version vs. docker-container-linux-version

Hi all

I’ve a small question:

→ Can I run a very new docker-linux-container with an very old docker-daemon?

Vise-Versa: running a very old docker-linux-container with an very new docker-daemon is working, I was able to check it.

Many thanks for your feedback!

Regards,
Marco

Can you elaborate on what you mean by new/old docker-linux-container?

Are you refering to images, and asking wether it makes a difference if an image was build using an old docker version works with a recent docker version?

Container runtimes use images according the OCI container image specs.
Practicaly, you can build container images and run container based on them without docker. You could use buildkit directly, kaniko or podman to build your image. You could run your containers on kubernetes, podman, crio, containerd …

Which translates to “Can I run a container based on a very new linux container image with a very old docker-daemon?”
I don’t see a reson why not, as long as you are talling about vanila docker. I can imaging this is not always the case with modifed forks.

Hi Metin

Thank you for your answer.

What is vanilla docker?

I’ve this question, because I wanna provide some docker-container to some peoples.
My docker-containers have inside a very new linux, but these people which are running my docker-container don’t want to upgrade their docker-daemon, because it’s only allowed for they to install stuff from the official redhat-repository, and there is only a very old docker-daemon in the official redhat-repository.

Best regards,
Marco

From my point of view, the docker distributions you get from docker’s repositories are vanila docker.

From my point ov view, this is not a vanila docker distribution, as redhat forked and customized the docker sources to align them with their own operating system and philosophy.

This doesn’t necessarily mean it will not work, but it means that there is no guaranty that redhat’s docker distribution behaves like a vanila docker distribution.

Are they still on 1.13 something? Even if, this version should already work with OCI complient container images, regardless if build with docker, buildkit, buildah, kaniko or whatever tool used that creats OCI complient container images…

Okay, it does not work!
I’ve a very new docker-linux-container (debian-11) with an very old docker-daemon (18.06.1-ce, build e68fc7a).
If I’m trying to run this new docker-linux-container with this old docker-daemon, a normal file-system-listing in the container shows me only question marks and I got permissions issue with the root user:

BASH> docker exec -it test-container bash
root@4492f78adefb:/# ls -la
ls: cannot access '.': Operation not permitted
ls: cannot access '..': Operation not permitted
ls: cannot access 'bin': Operation not permitted
ls: cannot access 'boot': Operation not permitted
ls: cannot access 'dev': Operation not permitted
ls: cannot access 'etc': Operation not permitted
ls: cannot access 'home': Operation not permitted
ls: cannot access 'lib': Operation not permitted
ls: cannot access 'lib64': Operation not permitted
ls: cannot access 'media': Operation not permitted
ls: cannot access 'mnt': Operation not permitted
ls: cannot access 'opt': Operation not permitted
ls: cannot access 'proc': Operation not permitted
ls: cannot access 'root': Operation not permitted
ls: cannot access 'run': Operation not permitted
ls: cannot access 'sbin': Operation not permitted
ls: cannot access 'srv': Operation not permitted
ls: cannot access 'sys': Operation not permitted
ls: cannot access 'tmp': Operation not permitted
ls: cannot access 'usr': Operation not permitted
ls: cannot access 'var': Operation not permitted
ls: cannot access 'transfer': Operation not permitted
ls: cannot access 'supervisord.log': Operation not permitted
ls: cannot access 'supervisord.pid': Operation not permitted
ls: cannot access '.dockerenv': Operation not permitted
ls: cannot access 'docker-entrypoint.sh': Operation not permitted
ls: cannot access 'docker-entrypoint.d': Operation not permitted
total 0
d????????? ? ? ? ?            ? .
d????????? ? ? ? ?            ? ..
-????????? ? ? ? ?            ? .dockerenv
d????????? ? ? ? ?            ? bin
d????????? ? ? ? ?            ? boot
d????????? ? ? ? ?            ? dev
d????????? ? ? ? ?            ? docker-entrypoint.d
l????????? ? ? ? ?            ? docker-entrypoint.sh
d????????? ? ? ? ?            ? etc
d????????? ? ? ? ?            ? home
d????????? ? ? ? ?            ? lib
d????????? ? ? ? ?            ? lib64
d????????? ? ? ? ?            ? media
d????????? ? ? ? ?            ? mnt
d????????? ? ? ? ?            ? opt
d????????? ? ? ? ?            ? proc
d????????? ? ? ? ?            ? root
d????????? ? ? ? ?            ? run
d????????? ? ? ? ?            ? sbin
d????????? ? ? ? ?            ? srv
-????????? ? ? ? ?            ? supervisord.log
-????????? ? ? ? ?            ? supervisord.pid
d????????? ? ? ? ?            ? sys
d????????? ? ? ? ?            ? tmp
d????????? ? ? ? ?            ? transfer
d????????? ? ? ? ?            ? usr
d????????? ? ? ? ?            ? var

Please share how you started the container. Something is off here.

I’ve started the container just with docker-compose up -d <<container-name>>

… the command itself is meaningless without the compose file that is used to configure the container(s).

We’ve nothing special in the docker-compse yml-config:

  docker-container:
    image: www.example.com/example/docker-container:5.1.61
    container_name: docker-container
    ports:
      - "42300:42300"
    environment:
      TZ: Europe/Zurich
      DB_TYPE: oracle
      DB_HOST: ${ENV_DB_HOST}
      DB_PORT: ${ENV_DB_PORT}
      DB_NAME: ${ENV_DB_SERVICE}
      DB_USER: ${ENV_DB_USER_COMMON}
      DB_PASS: ${ENV_DB_PASS_COMMON}
    volumes:
      - /something/something/abc:/something/something/xyz
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 4096M
  1. Can you run any container using docker directly without docker-compose? For example:
    docker run --rm -it bash ls -la /
    
  2. Can you try an old container on the old docker daemon? Does that work? For example:
    docker run --rm -it bash:4.4.0 ls -la /
    
    which is from 2018.
  3. What is your storage driver? If you could share the output of
    docker info
    
    I think that could also help to identify the cause.
  4. What is the filesystem under your docker root dir? (/var/lib/docker by default) Is it a network filesystem?
  5. What happens when you try to list files of a running container on the host?
    docker run -d --name test httpd:2.4
    docker container inspect test
    sudo ls -la $(docker container inspect test --format '{{ .GraphDriver.Data.MergedDir }}')
    

I guess it is just a similar, changed config, not what you used. Metin probably asked for the original (if that can be shared), because the “special” part is often not what keywords you use, but what values those have. Although I think if you can answer my questions, that can also help us to help you.

Hi Rimelek

Many thanks for your feedback.

Following are my answers:

  1. Yes, it works with this 6 weeks old docker-image.

  2. Yes, it works with this 5 years old docker-image.

  3. Storage Driver: overlay2

  4. No, it’s not a network fileystem/share. It’s normal XFS.

  5. I got the following output:

$ docker run -d --name test httpd:2.4
Unable to find image 'httpd:2.4' locally
2.4: Pulling from library/httpd
f7a1c6dad281: Pull complete
f18d7c6e023b: Pull complete
bf06bcf4b8a8: Pull complete
4566427976c4: Pull complete
70a943c2d5bb: Pull complete
Digest: sha256:b7907df5e39a98a087dec5e191e6624854844bc8d0202307428dd90b38c10140
Status: Downloaded newer image for httpd:2.4
6e60d51715f112990e90e9316f4c63d2a5652eb956c417a8f9ccd2774ac80da4


$ docker image ls | grep httpd
httpd                                                          2.4                 faed93b28859        7 days ago          144MB


$ sudo ls -la $(docker container inspect test --format '{{ .GraphDriver.Data.MergedDir }}')
total 4
drwxr-xr-x. 1 root root   17 Mar  9 09:21 .
drwx------. 5 root root   69 Mar  9 09:21 ..
drwxr-xr-x. 1 root root    6 Mar  1 14:20 bin
drwxr-xr-x. 2 root root    6 Dec 11 18:25 boot
drwxr-xr-x. 1 root root   43 Mar  9 09:21 dev
-rwxr-xr-x. 1 root root    0 Mar  9 09:21 .dockerenv
drwxr-xr-x. 1 root root   66 Mar  9 09:21 etc
drwxr-xr-x. 2 root root    6 Dec 11 18:25 home
drwxr-xr-x. 1 root root   30 Mar  1 14:20 lib
drwxr-xr-x. 2 root root   34 Feb 28 01:00 lib64
drwxr-xr-x. 2 root root    6 Feb 28 01:00 media
drwxr-xr-x. 2 root root    6 Feb 28 01:00 mnt
drwxr-xr-x. 2 root root    6 Feb 28 01:00 opt
drwxr-xr-x. 2 root root    6 Dec 11 18:25 proc
drwx------. 2 root root   37 Feb 28 01:00 root
drwxr-xr-x. 3 root root   30 Feb 28 01:00 run
drwxr-xr-x. 2 root root 4096 Feb 28 01:00 sbin
drwxr-xr-x. 2 root root    6 Feb 28 01:00 srv
drwxr-xr-x. 2 root root    6 Dec 11 18:25 sys
drwxrwxrwt. 1 root root    6 Mar  1 14:19 tmp
drwxr-xr-x. 1 root root   19 Feb 28 01:00 usr
drwxr-xr-x. 1 root root   41 Feb 28 01:00 var
$

So the age of the image does not really matter. Do you have some security feature enabled?
I see that dot after the file permissions.

No, I don’t have security features enabled, I don’t know about security features.
It’s just very new docker-linux-container (debian-11) with an very old docker-daemon (18.06.1-ce, build e68fc7a).
Nothing more, nothing special :wink:

In addition, I think this Operation not permitted comes, because everything has only ??? as file-permissions/user/group/size!?

You can run getenforce to see wether selinux is set to enforced, permissive or disabled.

Running container with enforced selinux is a different beast, you might want to check this blog post Selinux with Containers. I had people taking care of creating selinux polices for containers whenever something didn’t run and I never asked how they actualy created them.

Docker-CE on CentOS should have come with a selinux policy package. As far as I remember installing the CentOS packages on RHEL lacked a dependency in that area. I never liked docker on RHEL based systems…

Hi Meyay

Yes, I got the output "“Enforcing”.
So, you mean if I deactivate “selinux” it should work?

Many thanks for your feedback.

Best regards,
Marco

For testing purposes this should be fine. But I would advise against making it a permanent solution, as deactivating selinux will have a bad impact on the security.

You might want to wait for someone with current experience in running Docker on a RHEL-based system. Last time I used Docker EE on Rhel was 3-4 years ago…