Why aren't base layer images listed in `docker image ls -a`?

General
1

I’m a newbie trying to gain a deeper understanding on docker’s behavior.

Suppose we start with a simple Dockerfile:

FROM ubuntu:latest
CMD [ "echo", "Hello world" ]

And then run:

$ docker build -t mybuild:v1 .

Which gives the following output:

[+] Building 7.8s (5/5) FINISHED                                                                                                                                              docker:default
 => [internal] load .dockerignore                                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                                         0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                    0.0s
 => => transferring dockerfile: 84B                                                                                                                                                     0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                                                        3.9s
 => [1/1] FROM docker.io/library/ubuntu:latest@sha256:6042500cf4b44023ea1894effe7890666b0c5c7871ed83a97c36c76ae560bb9b                                                                  3.9s
 => => resolve docker.io/library/ubuntu:latest@sha256:6042500cf4b44023ea1894effe7890666b0c5c7871ed83a97c36c76ae560bb9b                                                                  0.0s
 => => sha256:6042500cf4b44023ea1894effe7890666b0c5c7871ed83a97c36c76ae560bb9b 1.13kB / 1.13kB                                                                                          0.0s
 => => sha256:bbf3d1baa208b7649d1d0264ef7d522e1dc0deeeaaf6085bf8e4618867f03494 424B / 424B                                                                                              0.0s
 => => sha256:174c8c134b2a94b5bb0b37d9a2b6ba0663d82d23ebf62bd51f74a2fd457333da 2.30kB / 2.30kB                                                                                          0.0s
 => => sha256:a486411936734b0d1d201c8a0ed8e9d449a64d5033fdc33411ec95bc26460efb 29.55MB / 29.55MB                                                                                        2.9s
 => => extracting sha256:a486411936734b0d1d201c8a0ed8e9d449a64d5033fdc33411ec95bc26460efb                                                                                               0.9s
 => exporting to image                                                                                                                                                                  0.0s
 => => exporting layers                                                                                                                                                                 0.0s
 => => writing image sha256:f3a903e727fafcb1683f74f251e50860b1385c86d998a9f861dda1bbd6bab49c                                                                                            0.0s
 => => naming to docker.io/library/mybuild:v1

Running docker image ls -a afterwards yielded:

REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
mybuild      v1        f3a903e727fa   12 days ago   77.8MB

What’s surprising to me was that the image of the base layer (ubuntu:latest) wasn’t included in the list. I would expect the ubuntu:latest image was cached locally somewhere, but why isn’t it included in the list of all the images in the docker host?

To verify that the ubuntu:latest image does indeed exist locally, I removed the mybuild:v1 image and then try to pull the ubuntu:latest image directly.

$ docker image rm mybuild:v1
Untagged: mybuild:v1
Deleted: sha256:f3a903e727fafcb1683f74f251e50860b1385c86d998a9f861dda1bbd6bab49c

At this point, docker image ls -a yields an empty list.

$ docker image ls -a
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

Pulling ubuntu:latest once again,

$ docker image pull ubuntu:latest
latest: Pulling from library/ubuntu
a48641193673: Already exists
Digest: sha256:6042500cf4b44023ea1894effe7890666b0c5c7871ed83a97c36c76ae560bb9b
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest

Clearly, the image for ubuntu:latest already exists on the docker host due to it being used in the previous build’s base layer. If that’s the case, why isn’t it listed out by docker image ls even with the --all flag specified?

Client:
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.21.3
 Git commit:        afdd53b4e3
 Built:             Sun Oct 29 15:42:02 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.21.3
  Git commit:       311b9ff0aa
  Built:            Sun Oct 29 15:42:02 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.11
  GitCommit:        64b8a811b07ba6288238eefc14d898ee0b5b99ba.m
 runc:
  Version:          1.1.10
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
2

What you were looking for is the image tag which will not be added to the image list until you run a container specifically from it or at least pull that image. Otherwise only the image layers will be downloaded. Notice the “Already exists” part in the output of your docker pull

So the layer was there. Docker just added the image tag to the layer so you can see it in the image list.

The exact solution how Docker stores these layers and show it in the output could depend on the docker version. I tried it with a Docker Desktop on macOS where I have Docker 24.0.7 and it gave me the output you had. Then I tried it with a Docker 20.10 and I didn’t see the “already exits” line in the output, and when I deleted the ubuntu image, it didn’t show “deleted” only “untagged”.

Note that when you deleted the image, the ID that you saw was the ID of the Ubuntu image which doesn’t mean you lose the filesystem layers or the necessary data couldn’t be saved in the cache.

One thing is sure. You can’t delete the filesystem layers when another image uses those layers. So even if you don’t see the image without a tag in the image list and Docker reports it deleted the image, the required filesystem layers are saved and when you pull the image directly, it can be reused.

1 Like
3

Thank you for taking the time to answer my question.

Indeed, that’s what I thought as well. That’s why I got a little confused when I didn’t see the output for ubuntu:latest in the image list.

So even though the image layer is pulled down, it won’t appear in the image list until a container is run from it? Is this behavior particular to recent versions of docker? The reason I got confused is because I came across various resources (this article, for instance) where the image of the base layer actually appears in the image list after a successful build.

The article I mentioned uses the exact same Dockerfile as the one in my original post.

FROM ubuntu:latest
CMD [ "echo", "Hello world" ]

But the output of docker images, according to the author, included ubuntu:latest.

4

You quoted the answer too

I mentioned running the container only because it automatically pulls the image if it doesn’t exist locally

I’m not sure how you came to that conclusion. Please, quote the part from the article that indicated that the image wasn’t pulled before running the build command.

5

This behaviour was changed at some point. In the previous versions of Docker, you could see the pulled parent image in the image list immediately after the image was built.
Now it behaves differently, which confuses me a bit.

6

I do remember older versions indeed showing the tags of the base images as well. Though, it is not like it hides crucial information, because as the maintainer of the Dockerfile you already know which image it will use as base image.

An image tag points to a manifest, which knows which sequence of image layers make up a particular image. When a tag is pulled, it just downloads and caches the manifest it points to, and then pulls and caches all layers that don’t exist in the local cache. The image layers have no knowledge about which manifest references them.

7

Docker uses Buildkit as a build engine instead of the legacy builder since docker v23.
By design choice, Buildkit does not tag anymore the base images, hence the base images are not listed by docker images nor docker image ls -a.