Can't find why my docker images digest are different when building from the exact same repo version

Docker Engine General
,
1

I’m running in a hell since I try to avoid deploying some unchanged container, and want to use the digest to detect an image is the same … but I can’t have those digest to be the same.

I’m building a dotnet app, out of the container, then I COPY this app and dependencies in the container in order to ship & deploy it.

This is the dockerfile, obviously the ARGuments sent are exactly the same

ARG RUNTIME_IMAGE

FROM $RUNTIME_IMAGE AS final
COPY . /app/.
EXPOSE 80
ARG PROJECT_NAME
ENV RUN_PROJECT_NAME=$PROJECT_NAME
ENTRYPOINT dotnet $RUN_PROJECT_NAME.dll

Already verified :

  • the binaries are exactly the same (used container-diff to check this topic, then compared files from extracted folders in a diff/merged tool, i’m 100% sure)
  • Also set always same the creation/access/write time of the files and directories copied
  • Also set always same the creation/access/write time of the dockerfile itself

The image

   TAG                 IMAGE ID                                                                  CREATED              SIZE   
   test11              sha256:74b1558ea77c84bf980281082719973a6d14b1eed24c23d434e589bb5b77db4e   2 seconds ago        279MB  
   test10              sha256:2e8de287b201c415aa87fb4c06de95727d5e4474bc54c5ecfbf8e1697635e795   About a minute ago   279MB

This is how I build it :

docker build --build-arg RUNTIME_IMAGE=mcr.microsoft.com/dotnet/core/runtime:3.1 --build-arg PROJECT_NAME=My.Project.Name --file C:\_git\PROJECT\_.artifacts\app.dockerfile --force-rm --no-cache --tag my.project.name:test17 C:\_git\PROJECT\_.artifacts\My.Project.Name

Please help …

1 Like
2

Each time you build an image, at least the metadata that stores the create data for each image layer will differ. Seems like the digiest is not only calculated based on the payload itself…

See for yourself:
– docker save test10 > test10.tar
– docker save test11 > test11.tar
Extract both tar files to a distinct folder to unwrapp the image layer. Take a look at those files.

3

@meyay thank you.
So the only difference I can find really is a .tar datetime in the layer create on the COPY command

image
image1936×658 68 KB

4

Buy using a multistage build and tricking the /app folder time, I finally have the layer.tar file exactly identicall between two builds. Now the layer manisfest json file is different because it brings a creation date with itself … does it use this date to compute the digest hash ?

image
image1874×399 40.6 KB

5

The changes in the manifest.json were the only differences i could spot when building the same image two times in a row. Without taking a look at the sources, this will only remain guessing…

6

This is the json file comparison

image
image1808×201 18.9 KB

7

And now once pushed on the registry, digets (and size! WTF !!) are differents while all the layers digests are same !

image
image1866×455 30 KB

8

You might want to read this blog post: How to Digest a Docker Image. How to calculate and manipulate Docker… | by Graham Jenson | Maori Geek

After reading the blog post, I realized I used the wrog filename here. cough

9

Ok, by some tricks I finally managed to have the layers digests to be the same :

  • create all the files in an app folder
  • then touch all files in the host to have the exact always same dates
  • then COPY . .

I can now compare the layers digests and detect them to be the same, while the full image manifest is still differente, it’s enough for my needs today.

thanks a lot for your help @meyay

10

The files checked in should be pulled with the same time on the other side.
The culprit is likely the time resolution.

Linux has a time resolution in milliseconds, while on Mac OS X the resolution is only down to the second, which causes discrepancies in file times.