Docker daemon image layer reuse

Docker Engine General
1

We are trying to maximize layer reuse / minimize layer pushes and pulls in our fleet of docker daemons.
We have a private registry version 2.6 and docker daemons version 1.13
I’m seeing an inconsistent docker daemon behavior regarding image layer reuse:

  • sometimes pulling layers it already pulled before
  • sometimes pushing layers that already exist in the target registry
    Is the docker daemon behavior documented somewhere?
    What are the guidelines and best practices to achieve maximum layer reuse?
2

Check here(https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#examples-for-official-repositories) - build cache section.

3

I’m asking about a different thing.
In my case a custom image is built on top of a base image. Other than the top most layer, the custom image shares all layers with the base image. Yet, when pushing the custom image to our registry, sometimes all layers are pushed, even though the registry should have all layers since I pulled the base image from it.

Here’s an example log of a pull - build - push sequence, plus the relevant docker registry log:

If there’s other info that can be useful I’ll be happy to share.

4

ok, I see what you are saying now.
What version of Docker registry are you using? can you pls check if thats the latest. I suspect that this has something to do with registry version as I saw some bugs with older registry versions w.r.to caching. Another thing to try would be to do the same with Docker hub just to eliminate registry dependencies.

5

Registry v2.6.0, which is quite recent
Which bugs with caching do you refer to? Can you share links?

We have a HA setup with 2 registry servers running with the following configuration:

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  s3:
    accesskey: ***
    secretkey: ***
    region: us-east-1
    bucket: ***
    v4auth: true
  delete:
    enabled: true
http:
  addr: :5000
  debug:
    addr: :5001
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

I tried the same pull-build-push sequence against docker hub registry and didn’t experience any issue. All shared layers were skipped (“Layer already exists”) during push.

6

Also, I came across this docker issue: sometimes already existing layer (e.g., layer for ubuntu:14.04) pushed again to dockerhub · Issue #1464 · distribution/distribution · GitHub
It mentions a docker engine v2 metadata cache:

Docker maintains a cache of layer digests that remote registries already have, to avoid unnecessary layer uploads. The size of this cache is limited, so it’s not guaranteed to prevent these uploads.

What is the size of this cache and what is the policy that evacuates entries from it?

How does this relate to the registry v2 API where docker daemon is able to check for a layer existence via a HEAD request (Registry | Docker Docs) before uploading?
Will the daemon always make the layer existence check?

7

My understanding is that the cache is used only during building the Docker image so that new layers dont get created in localhost. For pushing to remote registry, layerid and digest is used as indicated in this link(https://docs.docker.com/registry/spec/api/#pushing-an-image). Its better to open issue directly under here(https://github.com/moby/moby/issues) with registry tag and all logs.