Why does "docker pull ubuntu:focal" give a different SHA256SUM than what's published by ubuntu?

General
, ,
1

If I understand how the docker pull works it’s supposed to check the SHA256SUM to see if there’s a discrepancy between what’s the official published image and what’s downloaded.

So before doing the download I went to
(1) https://hub.docker.com/_/ubuntu?tab=tags&page=1&name=focal
and to
(2) https://partner-images.canonical.com/core/focal/current/SHA256SUMS

to get the SHA256 tags for ubuntu:focal

According to (1) the SHA256 on amd64 for ubuntu:focal is

Digest:sha256:60f560e52264ed1cb7829a0d59b1ee7740d7580e0eb293aca2d722136edb1e24

According to (2) the SHA256SUM is

4f3f0de8799c8cbda1990d78182aaca0f5eb9b66315704b229374f2c62099ef7 *ubuntu-focal-core-cloudimg-amd64-root.tar.gz

and when I do “docker pull” I get a completely different SHA256SUM from both.

$ sudo docker pull ubuntu:focal
focal: Pulling from library/ubuntu
3ff22d22a855: Pull complete 
e7cb79d19722: Pull complete 
323d0d660b6a: Pull complete 
b7f616834fd0: Pull complete 
Digest: sha256:5d1d5407f353843ecf8b16524bc5565aa332e9e6a1297c73a92d3e754b8a636d
Status: Downloaded newer image for ubuntu:focal
docker.io/library/ubuntu:focal

$ export DOCKER_CONTENT_TRUST=1 && sudo docker pull ubuntu:focal
focal: Pulling from library/ubuntu
Digest: sha256:5d1d5407f353843ecf8b16524bc5565aa332e9e6a1297c73a92d3e754b8a636d
Status: Image is up to date for ubuntu:focal
docker.io/library/ubuntu:focal

$ sudo docker images --digests
REPOSITORY            TAG                 DIGEST                                                                    IMAGE ID            CREATED             SIZE
ubuntu                focal               sha256:5d1d5407f353843ecf8b16524bc5565aa332e9e6a1297c73a92d3e754b8a636d   1e4467b07108        33 hours ago        73.9MB

I did a search across all the recent images at (1) and NONE of them match the digest reported by docker pull.

So where is this SHA256 for ubuntu:focal coming from in docker pull and why does it not match any SHA256 I can find on (1) or (2)?

2

I get the same SHA sum as you and have no idea why there is a difference on 1). The sums on 2) are those of the tar files of the OS. This file is added at the beginning of the Dockerfile and its checksum is therefore different to the one of the resulting image.

3

The reason why the diggests differ is due to images beeing stored in remote repos are compressed before push and therefore lead to a different sha256 checksum. When you pull an image, it will be uncompressed before the checksum is calculated.

You can find a beautiful in-depth analysis here: https://windsock.io/explaining-docker-image-ids/

1 Like
4

Thanks @meyay for the link to the detailed analysis. I looked more deeply at all the layers of the pulled image

$ sudo docker history ubuntu:focal
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
1e4467b07108        2 days ago          /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B                  
<missing>           2 days ago          /bin/sh -c mkdir -p /run/systemd && echo 'do…   7B                  
<missing>           2 days ago          /bin/sh -c set -xe   && echo '#!/bin/sh' > /…   811B                
<missing>           2 days ago          /bin/sh -c [ -z "$(apt-get indextargets)" ]     1.01MB              
<missing>           2 days ago          /bin/sh -c #(nop) ADD file:65a1cc50a9867c153…   72.9MB

Which I found here: https://hub.docker.com/layers/ubuntu/library/ubuntu/focal/images/sha256-60f560e52264ed1cb7829a0d59b1ee7740d7580e0eb293aca2d722136edb1e24?context=explore (3)

$ sudo docker inspect ubuntu:focal
[
    {
        "Id": "sha256:1e4467b07108685c38297025797890f0492c4ec509212e2e4b4822d367fe6bc8",
        "RepoTags": [
            "ubuntu:focal"
        ],
        "RepoDigests": [
            "ubuntu@sha256:5d1d5407f353843ecf8b16524bc5565aa332e9e6a1297c73a92d3e754b8a636d"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2020-07-24T14:38:35.464294608Z",
        "Container": "9d62e8d762827123636cb28eacfed9975890fd659cac66adee63fc3a969bb8a9",
        "ContainerConfig": {
            "Hostname": "9d62e8d76282",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "CMD [\"/bin/bash\"]"
            ],
            "ArgsEscaped": true,
            "Image": "sha256:905a090e9b85447aff61cda51518fc1ab45af28e185981b70bc73dc9c03abf13",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "DockerVersion": "18.09.7",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/bash"
            ],
            "ArgsEscaped": true,
            "Image": "sha256:905a090e9b85447aff61cda51518fc1ab45af28e185981b70bc73dc9c03abf13",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 73859057,
        "VirtualSize": 73859057,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/e2ed3f902006cf267ea7004c8fed820f6871bdcd858ded3ff15bc750d7911fa2/diff:/var/lib/docker/overlay2/d9651e9965038eff4339389194d164c977560a53e822eac5be245d98542da20e/diff:/var/lib/docker/overlay2/45369f361b9da46cd455d330c3cc00d4e3a2ffa8120720d57656f74319de8a71/diff",
                "MergedDir": "/var/lib/docker/overlay2/bb8b5256e42a9f1e64de8fd6385a0f28cd7e4d4d558657c83acd55c09c4f0f4e/merged",
                "UpperDir": "/var/lib/docker/overlay2/bb8b5256e42a9f1e64de8fd6385a0f28cd7e4d4d558657c83acd55c09c4f0f4e/diff",
                "WorkDir": "/var/lib/docker/overlay2/bb8b5256e42a9f1e64de8fd6385a0f28cd7e4d4d558657c83acd55c09c4f0f4e/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:ce30112909569cead47eac188789d0cf95924b166405aa4b71fb500d6e4ae08d",
                "sha256:8eeb4a14bcb4379021c215017c94800a848a8203a8ce76aa1bd211d4c995f792",
                "sha256:a37e74863e723df4ddd599ef1b7d9a68e2301794a8c37c2370f8c2c8993ef72c",
                "sha256:095624243293a7dfdb582f8471d6e2d9d7772dd621bc57906b034c59f388ebac"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

But none of the SHA256SUMs matched what Ubuntu says the base image hashes are. The article did go into hashing in great detail but did not detail how one can independently confirm that the base image provided by Ubuntu is uncompromised.

What’s more - in “docker inspect” it says

"Architecture": "amd64",
        "Os": "linux",
        "Size": 73859057,

but at (3) the amd64 compressed image is only 27.23MB, which is about 1/3 the size.

5

As docker already does verify the integration of each downloaded image layer, I never felt the urge to perform additional checks. If this mechanism is to “weak” for your test, you might want to check “docker content trust”, which introduces digital signatures to images.

After pulling the mutable tag ubuntu:focal yesterday, today I fetched the digest and it updated the image?!:

docker pull ubuntu@sha256:60f560e52264ed1cb7829a0d59b1ee7740                                                                                        d7580e0eb293aca2d722136edb1e24
sha256:60f560e52264ed1cb7829a0d59b1ee7740d7580e0eb293aca2d722136edb1e24: Pulling                                                                                         from library/ubuntu
Digest: sha256:60f560e52264ed1cb7829a0d59b1ee7740d7580e0eb293aca2d722136edb1e24
Status: Downloaded newer image for ubuntu@sha256:60f560e52264ed1cb7829a0d59b1ee7                                                                                        740d7580e0eb293aca2d722136edb1e24
docker.io/library/ubuntu@sha256:60f560e52264ed1cb7829a0d59b1ee7740d7580e0eb293ac                                                                                        a2d722136edb1e24

Me beeing perplexed about the status, retried the pull command which led to a “Image is up to date” status.

If you perform an inspect, you can see that the expected Digist is now listed under RepoDigestes. Though, it is unclear why there are two repo digests now and why the later one was not there in the first place.

docker image inspect ubuntu:focal --format '{{json .RepoDigests}}'
["ubuntu@sha256:5d1d5407f353843ecf8b16524bc5565aa332e9e6a1297c73a92d3e754b8a636d","ubuntu@sha256:60f560e52264ed1cb7829a0d59b1ee7740d7580e0eb293aca2d722136edb1e24"]

This remains a mistery.

6

Looking more deeply. I found the pull from Ubuntu to /docker-library/official-images here: https://github.com/docker-library/official-images/pull/8429 ( commit: 5c0b84978d9fe2ab9fa80a477885ffb33a537bd1 )

but no sha256 documentation or matches.

So it seems to be a mystery to me for how one can independently verify checksums of docker images.

7

Not manualy. Docker does this for you like I wrote in my previous post.

8

But the problem is that without independent confirmation you can’t be sure the pulled image is not corrupted between upload by the original provider and preparation by Docker or if there’s something going on unseeen in the process of running “docker pull.” Why should one “just trust docker?”

As a counter example, in debian distros one can cross-check in any number of ways. You can download an ISO of the full system and compare to a published hash. For an installed system (among many other ways), you can run apt and look at the downloaded file(s) and compare to published hashes, or even download the debian package by itself and install via dpkg after verifying the .deb package. There are published keys on separate key server, published hashes for .deb packages, etc., etc.

I’m looking for some way to verify that what’s checked in by the original base image creator is what’s downloaded with “docker pull” outside of “just trusting docker” and not finding anything.

9

Actualy the ISO example is a different usecase. Neither does the ISO include the information about the checksum, nor does the client automaticly checks it after the download.

Though, the comparision with apt is correct. Apt does check the integrity after the download and before the installation. So does the docker-cli.

Though, I haven’t seen that people actualy not trust Apt and download single deb packages and compare integrity for them before installing them (b/c apt already does that for us)

You can always open an issue on git hub for docker regarding the obscurity of the checksum information. I agree that it’s odd that there are different checksum values for the very same image and that some of the values can not be found within the image metadata.

10

Did some more research:

It looks like the only way to do a comparison to the core image is to do a “docker save” to get the docker distributed tarball and compare it to the Canonical tarfile ( https://partner-images.canonical.com/core/focal/20200720/ubuntu-focal-core-cloudimg-amd64-root.tar.gz )

They do differ though. Haven’t yet figured out if the differences are significant or not.

13

The core issue: https://github.com/tianon/docker-brew-ubuntu-core/issues/185

There was an unsigned binary file added by the docker process.