[Issue] Metadata for targets expired

Docker Engine General
1

Hello,

I have an issue pulling a certain image:

$ docker pull discourse/base:2.0.20191219-2109

ERRO[0001] Metadata for targets expired
ERRO[0002] Metadata for targets expired
Error: remote repository docker.io/discourse/base out-of-date: targets expired at Mon Nov 18 12:52:09 -0500 2019
Same command with debug output: 
DEBU[0000] reading certificate directory: /root/.docker/tls/notary.docker.io
DEBU[0000] No yubikey found, using alternative key storage: no library found
DEBU[0000] Making dir path: /root/.docker/trust/tuf/docker.io/discourse/base/changelist
DEBU[0000] entered ValidateRoot with dns: docker.io/discourse/base
DEBU[0000] found the following root keys: [5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8]
DEBU[0000] found 1 valid leaf certificates for docker.io/discourse/base: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for docker.io/discourse/base
DEBU[0000] checking root against trust_pinning config for docker.io/discourse/base
DEBU[0000] checking trust-pinning for cert: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0000]  role has key IDs: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0000] verifying signature for key ID: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0000] root validation succeeded for docker.io/discourse/base
DEBU[0000] entered ValidateRoot with dns: docker.io/discourse/base
DEBU[0000] found the following root keys: [5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8]
DEBU[0000] found 1 valid leaf certificates for docker.io/discourse/base: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for docker.io/discourse/base
DEBU[0000] checking root against trust_pinning config for docker.io/discourse/base
DEBU[0000] checking trust-pinning for cert: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0000]  role has key IDs: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0000] verifying signature for key ID: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0000] root validation succeeded for docker.io/discourse/base
DEBU[0000] updating TUF client
DEBU[0000] Loading timestamp...
DEBU[0001] 200 when retrieving metadata for timestamp
DEBU[0001] timestamp role has key IDs: db679e1f17a2a2bdd9952b6f37c9048635611f162bc6a0957ec30fbb1412d366
DEBU[0001] verifying signature for key ID: db679e1f17a2a2bdd9952b6f37c9048635611f162bc6a0957ec30fbb1412d366
DEBU[0001] timestamp role has key IDs: db679e1f17a2a2bdd9952b6f37c9048635611f162bc6a0957ec30fbb1412d366
DEBU[0001] verifying signature for key ID: db679e1f17a2a2bdd9952b6f37c9048635611f162bc6a0957ec30fbb1412d366
DEBU[0001] successfully verified downloaded timestamp
DEBU[0001] Loading snapshot...
DEBU[0001] snapshot role has key IDs: e88a99d5b23301178976a4a103df4716f7943301d9972db0541bfa17c94cb75b
DEBU[0001] verifying signature for key ID: e88a99d5b23301178976a4a103df4716f7943301d9972db0541bfa17c94cb75b
DEBU[0001] successfully verified cached snapshot
DEBU[0001] Loading targets...
DEBU[0001] no targets in cache, must download
DEBU[0001] 200 when retrieving metadata for targets.09b50021560effa664eb8271cd16e2212ed55b7b63b2127cf9a0e69cfe2b47b7
DEBU[0001] targets role has key IDs: c016860a9293a48bc98f27b9c36d81515f4bcc3dd2ec93bb2d6d8af8f37a39d2
DEBU[0001] verifying signature for key ID: c016860a9293a48bc98f27b9c36d81515f4bcc3dd2ec93bb2d6d8af8f37a39d2
ERRO[0001] Metadata for targets expired
DEBU[0001] downloaded targets.09b50021560effa664eb8271cd16e2212ed55b7b63b2127cf9a0e69cfe2b47b7 is invalid: targets expired at Mon Nov 18 12:52:09 -0500 2019
DEBU[0001] Client Update (Targets): targets expired at Mon Nov 18 12:52:09 -0500 2019
DEBU[0001] Error occurred. Root will be downloaded and another update attempted
DEBU[0001] Resetting the TUF builder...
DEBU[0001] entered ValidateRoot with dns: docker.io/discourse/base
DEBU[0001] found the following root keys: [5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8]
DEBU[0001] found 1 valid leaf certificates for docker.io/discourse/base: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0001] found 1 leaf certs, of which 1 are valid leaf certs for docker.io/discourse/base
DEBU[0001] found the following root keys: [5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8]
DEBU[0001] found 1 valid leaf certificates for docker.io/discourse/base: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0001] found 1 valid root leaf certificates for docker.io/discourse/base: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0001]  role has key IDs: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0001] verifying signature for key ID: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0001] checking root against trust_pinning config for docker.io/discourse/base
DEBU[0001] checking trust-pinning for cert: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0001]  role has key IDs: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0001] verifying signature for key ID: 5f5ba22b20768b8f2d49e9035400a1922ae761aa70b5bb0d930fe91dd06bd5d8
DEBU[0001] root validation succeeded for docker.io/discourse/base
DEBU[0001] successfully verified cached root
DEBU[0001] retrying TUF client update
DEBU[0001] Loading timestamp...
DEBU[0002] 200 when retrieving metadata for timestamp
DEBU[0002] timestamp role has key IDs: db679e1f17a2a2bdd9952b6f37c9048635611f162bc6a0957ec30fbb1412d366
DEBU[0002] verifying signature for key ID: db679e1f17a2a2bdd9952b6f37c9048635611f162bc6a0957ec30fbb1412d366
DEBU[0002] successfully verified downloaded timestamp
DEBU[0002] Loading snapshot...
DEBU[0002] snapshot role has key IDs: e88a99d5b23301178976a4a103df4716f7943301d9972db0541bfa17c94cb75b
DEBU[0002] verifying signature for key ID: e88a99d5b23301178976a4a103df4716f7943301d9972db0541bfa17c94cb75b
DEBU[0002] successfully verified cached snapshot
DEBU[0002] Loading targets...
DEBU[0002] no targets in cache, must download
DEBU[0002] 200 when retrieving metadata for targets.09b50021560effa664eb8271cd16e2212ed55b7b63b2127cf9a0e69cfe2b47b7
DEBU[0002] targets role has key IDs: c016860a9293a48bc98f27b9c36d81515f4bcc3dd2ec93bb2d6d8af8f37a39d2
DEBU[0002] verifying signature for key ID: c016860a9293a48bc98f27b9c36d81515f4bcc3dd2ec93bb2d6d8af8f37a39d2
ERRO[0002] Metadata for targets expired
DEBU[0002] downloaded targets.09b50021560effa664eb8271cd16e2212ed55b7b63b2127cf9a0e69cfe2b47b7 is invalid: targets expired at Mon Nov 18 12:52:09 -0500 2019
DEBU[0002] Client Update (Targets): targets expired at Mon Nov 18 12:52:09 -0500 2019
Error: remote repository docker.io/discourse/base out-of-date: targets expired at Mon Nov 18 12:52:09 -0500 2019

About my setup:

OS: Ubuntu 18.04 LTS
Docker Version: Docker version 19.03.5, build 633a0ea838

Other images can be pulled and run just fine, but I also think that the image itself is ok, because I appear to be the first user to have this problem.

My best guess is that I have misconfigured my Docker daemon and would be glad about any hints.

Furthermore, I want to note that my servers time in synced and up to date.

2

You are right, this is not a general problem with the OS and Docker version combination. I just pulled the image without any issues.

Since you already point out you suspect a misconfigured Docker engine, would you mind sharing your modifications? I would put my money on activated content trust in combination with an image that is not trusted.

Update: I can confirm this is your problem!

meyay@swarm1:~$ export DOCKER_CONTENT_TRUST=1
meyay@swarm1:~$ docker pull discourse/base:2.0.20191219-2109
ERRO[0002] Metadata for targets expired
ERRO[0003] Metadata for targets expired
Error: remote repository docker.io/discourse/base out-of-date: targets expired at Mon Nov 18 12:52:09 -0500 2019

My guess is that the certificate of the signee is expired and thus results in an invalid validation.

3

@meyay Thank you for your help! I indeed use DOCKER_CONTENT_TRUST=1.

Can you explain the issue a bit more? I would suspect that this is something that the image owner would have to fix? Can you elaborate on the “how”?

4

Honesty, DCT is not my strong suite.

It is clear that the problem needs to be remedied by the “image owner”.

You migh find this (old, but hopefully still valid) blog post helpfull to dig deeper into the topic: