Can't get Notary / Content Trust delegation to work

Docker Engine
1

I’m trying to use Content Trust delegation following the instructions here: Docker Docs

I’ve got as far as publishing the delegation to Notary (apparently successfully); however, the Docker CLI fails to pull or push successfully after that.

I’m using:

(I’ve tried various other combinations of Docker and Notary version, but this is the farthest I’ve managed to get).

This is the output from a push to a brand-new repository, followed by Notary content delegation, and subsequent attempted Docker pull of the image that was just pushed :

> export NOTARY_ROOT_PASSPHRASE=root
> export NOTARY_TARGETS_PASSPHRASE=repository
> export NOTARY_SNAPSHOT_PASSPHRASE=repository
> export DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE=root
> export DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE=repository
> export DOCKER_CONTENT_TRUST=1
> export NOTARY_SERVER=https://notary-server:4443
> export DOCKER_CONTENT_TRUST_SERVER=$NOTARY_SERVER

> docker tag -f good_image registry-server:5000/good_image:latest

> docker push registry-server:5000/good_image:latest
The push refers to a repository [registry-server:5000/good_image] (len: 1)
634a8be49a18: Preparing
634a8be49a18: Pushing
634a8be49a18: Pushed
51f9dd12da79: Preparing
51f9dd12da79: Pushing
51f9dd12da79: Pushed
73e8d4f6bf84: Preparing
73e8d4f6bf84: Pushing
73e8d4f6bf84: Pushed
latest: digest: sha256:f434bfc771b78180862207bafaf52f4fc8f067dd9b3544556f94fdffd8b67f34 size: 5947
Signing and pushing trust metadata
Finished initializing “registry-server:5000/good_image”

> /root/notary/notary -s $NOTARY_SERVER -d /root/.docker/trust key rotate registry-server:5000/good_image -t snapshot -r

> /root/notary/notary -s $NOTARY_SERVER -d /root/.docker/trust delegation add registry-server:5000/good_image targets/releases /root/delegation.crt --all-paths

Addition of delegation role targets/releases with keys [74c5cd5e53f0b2839325350ee761f703dbbb9a18c42bde107dd63fad1a98095d], with paths [“” ], to repository “registry-server:5000/good_image” staged for next publish.

> /root/notary/notary -s $NOTARY_SERVER -d /root/.docker/trust publish registry-server:5000/good_image
Pushing changes to registry-server:5000/good_image

> docker pull registry-server:5000/good_image:latest
time=“2016-04-13T15:36:07Z” level=error msg=“Client Update (Targets): json: cannot unmarshal object into Go value of type data.PublicKey”
json: cannot unmarshal object into Go value of type data.PublicKey

> echo $?
1

I tried pulling with debug output enabled:

> docker -D pull registry-server:5000/good_image:latest | tail -n 14
time=“2016-04-13T15:34:17Z” level=debug msg=“successfully verified timestamp”
time=“2016-04-13T15:34:17Z” level=debug msg=“verifying signature for key ID: ceb219b20eaca18979e96d3c3c729c72dc62f91ef41d2e4936c52da5e7d30673”
time=“2016-04-13T15:34:17Z” level=debug msg=“continuing b/c keyid was invalid: ceb219b20eaca18979e96d3c3c729c72dc62f91ef41d2e4936c52da5e7d30673 for roledata &{{[aeccaa04556c95b62387ff005ed9a1f09a319d3e9a8fda0ac417eed4ba952845] %!s(int=1)} snapshot }\n”
time=“2016-04-13T15:34:17Z” level=debug msg=“successfully verified snapshot”
time=“2016-04-13T15:34:17Z” level=debug msg=“200 when retrieving metadata for targets”
time=“2016-04-13T15:34:17Z” level=debug msg=“targets role has key IDs: 292a62f9a405904fc886ce6fa40cd5f18d9036d3eb9df724d62a9ce3e858503b”
time=“2016-04-13T15:34:17Z” level=debug msg=“verifying signature for key ID: 292a62f9a405904fc886ce6fa40cd5f18d9036d3eb9df724d62a9ce3e858503b”
time=“2016-04-13T15:34:17Z” level=debug msg=“successfully verified targets”
time=“2016-04-13T15:34:17Z” level=error msg=“Client Update (Targets): json: cannot unmarshal object into Go value of type data.PublicKey”
json: cannot unmarshal object into Go value of type data.PublicKey

Any help would be appreciated – thanks! I can add more debug output wherever it would be useful, but didn’t want to make this post too unreadable.

2

Apologies, but unfortunately Docker versions < 1.10.0 will not work with delegations: https://docs.docker.com/engine/breaking_changes/#docker-content-trust.

The cannot unmarshal object into Go value of type data.PublicKey is definitely the error you’d get on Docker 1.9.x - could you let us know what error you see on Docker 1.10?

3

I actually get a worse error with Docker 1.10 – the publish itself fails:

This is with:

Running the exact same commands as above, the publish command (with additional debugging) fails as follows:

/root/notary/notary -v -D -s https://notary-server:4443 -d /root/.docker/trust publish registry-server:5000/good_image | tail -n 11

time=“2016-04-13T21:15:19Z” level=debug msg=“successfully verified snapshot”
time=“2016-04-13T21:15:19Z” level=debug msg=“Downloading Targets…”
time=“2016-04-13T21:15:19Z” level=debug msg=“using cached targets”
time=“2016-04-13T21:15:19Z” level=debug msg=“targets role has key IDs: eda4c2790fd9e4abea7fff20ecab740f0e4bc8ee6bc74d3d601117fd4821f364”
time=“2016-04-13T21:15:19Z” level=debug msg=“verifying signature for key ID: eda4c2790fd9e4abea7fff20ecab740f0e4bc8ee6bc74d3d601117fd4821f364”
time=“2016-04-13T21:15:19Z” level=debug msg=“successfully verified targets”

  • fatal: targets/releases.c1ace40d15fde936ae078e375966a854b96b6a6b4ee9f40760de6f9afdcd3f7c trust data unavailable. Has a notary repository been initialized?

time=“2016-04-13T21:15:19Z” level=debug msg=“received HTTP status 404 when requesting targets/releases.c1ace40d15fde936ae078e375966a854b96b6a6b4ee9f40760de6f9afdcd3f7c.”
time=“2016-04-13T21:15:19Z” level=error msg=“Error getting targets file:targets/releases.c1ace40d15fde936ae078e375966a854b96b6a6b4ee9f40760de6f9afdcd3f7c trust data unavailable. Has a notary repository been initialized?”
time=“2016-04-13T21:15:19Z” level=debug msg=“Client Update (Targets): targets/releases.c1ace40d15fde936ae078e375966a854b96b6a6b4ee9f40760de6f9afdcd3f7c trust data unavailable. Has a notary repository been initialized?”
time=“2016-04-13T21:15:19Z” level=error msg=“Could not publish Repository: targets/releases.c1ace40d15fde936ae078e375966a854b96b6a6b4ee9f40760de6f9afdcd3f7c trust data unavailable. Has a notary repository been initialized?”

4

Just sanity checking: what is the context around the publish? Are you starting with a notary server (with no data) and running through the instructions all over again? If so, was the /root/.docker/trust directory cleared of metadata?

5

It’s a completely clean Notary server, Docker registry and /root/.docker directory. I’m recreating everything from scratch everytime we run this test-case.

6

Sorry for all this trouble! Can you try running with notary server v0.2?

It looks like the version you have (commit docker-v1.10.2 (13de903e)) doesn’t have the database migrations yet for consistent downloads; notary version docker-v1.10.2 was about a week before notary 0.2, and didn’t support consistent downloads. Notary (server and client) 0.2 do.

7

Thanks.

When I use version v0.2.0 of Notary and try to bring up Notary instances, the Notary server fails to connect to the Mysql instance: do I need to do some extra configuration to make it work?

git checkout v0.2.0
…
docker-compose build
…
docker-compose up -d
…
docker logs notary_server_1 2>&1 | tail
waiting for notarymysql to come up.
waiting for notarymysql to come up.
waiting for notarymysql to come up.
waiting for notarymysql to come up.
waiting for notarymysql to come up.
waiting for notarymysql to come up.
waiting for notarymysql to come up.
waiting for notarymysql to come up.
waiting for notarymysql to come up.
notaryserver database failed to come up within 30 seconds

The Mysql instance seems to be up:

docker logs notary_mysql_1 2>&1 | tail
2016-04-14 1:51:25 140059817031616 [Note] InnoDB: Highest supported file format is Barracuda.
2016-04-14 1:51:25 140059817031616 [Note] InnoDB: 128 rollback segment(s) are active.
2016-04-14 1:51:25 140059817031616 [Note] InnoDB: Waiting for purge to start
2016-04-14 1:51:25 140059817031616 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-76.0 started; log sequence number 1616829
2016-04-14 1:51:26 140059030292224 [Note] InnoDB: Dumping buffer pool(s) not yet started
2016-04-14 1:51:26 140059817031616 [Note] Plugin ‘FEEDBACK’ is disabled.
2016-04-14 1:51:26 140059817031616 [Note] Server socket created on IP: ‘::’.
2016-04-14 1:51:26 140059817031616 [Warning] ‘proxies_priv’ entry ‘@% root@e8f823f55aeb’ ignored in --skip-name-resolve mode.
2016-04-14 1:51:26 140059817031616 [Note] mysqld: ready for connections.
Version: ‘10.1.10-MariaDB-1~jessie’ socket: ‘/var/run/mysqld/mysqld.sock’ port: 3306 mariadb.org binary distribution

8

Hmm… shouldn’t be. docker-compose build && docker-compose up works for me on v0.2.0. The only thing I can think of is that maybe there was some issue re-creating the containers? I sometimes have issues with containers re-linking to each other. Were the old ones stopped and removed first? Or alternately maybe remove the mysql data volumes first?

9

OK – that did it. I had to abandon “docker-compose up” and replace it with separate “docker run” commands, but then content trust delegation seemed to work. Thanks for your help.

10

Thank you for your patience - sorry for all the trouble you ran into! We’ll add a note in the documentation that the client should probably match the notary server version, in case the client supports more features than the server.

1 Like