Docker 1.11 push to private 2.4.0 registry with token authentication

Docker Engine
1

Dear Docker folks,

I’m running a Docker Registry 2.4.0 with token authentication and self-signed certificate.
When I’m trying to push anything to the registry, it starts retrying and finally fails:

docker push localhost:5000/hello-world/hello-world:latest
The push refers to a repository [localhost:5000/hello-world/hello-world]
5f70bf18a086: Retrying in 1 second
33e7801ac047: Retrying in 1 second
unable to decode token response: json: cannot unmarshal string into Go value of type int

In daemon log, these lines appear:
Apr 27 18:07:50 rsuvorov docker[28661]: time=“2016-04-27T18:07:50.050923877+03:00” level=error msg=“Upload failed, retrying: unable to decode token response: json: cannot unmarshal string into Go value of type int”
Apr 27 18:08:10 rsuvorov docker[28661]: time=“2016-04-27T18:08:10.047221876+03:00” level=error msg=“Upload failed: unable to decode token response: json: cannot unmarshal string into Go value of type int”
Apr 27 18:08:10 rsuvorov docker[28661]: time=“2016-04-27T18:08:10.068000535+03:00” level=error msg=“Upload failed: unable to decode token response: json: cannot unmarshal string into Go value of type int”
Apr 27 18:08:10 rsuvorov docker[28661]: time=“2016-04-27T18:08:10.068110884+03:00” level=error msg=“Attempting next endpoint for push after error: unable to decode token response: json: cannot unmarshal string into Go value of type int”

Token server returns a valid token that can be decoded using python: http://pastebin.com/tQfYBqMn
Here is the decoded payload: {“access”:[{“type”:“repository”,“name”:“hello-world/hello-world”,“actions”:[“push”,“pull”]}],“jti”:“5f40bf413b6f42569ab17385005ce375”,“sub”:“test”,“exp”:1461772266,“iss”:“my_test_repo”,“iat”:1461768666,“nbf”:1461768666,“aud”:“npinfo-registry”}

This setup used to work in December 2015 with docker 1.9.

$ docker info
Containers: 25
Running: 4
Paused: 0
Stopped: 21
Images: 719
Server Version: 1.11.1
Storage Driver: aufs
Root Dir: /opt/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 621
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host bridge null
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 23.55 GiB
Name: rsuvorov
ID: VONR:B6MM:7AIE:JM4G:TRRO:PLQC:HVAG:6HAQ:EWSS:5CQG:NO22:7SJF
Docker Root Dir: /opt/var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

2

What token server are you running, and what does your config.yml look like for distribution 2.4?

3

I’m running my own token server based on Django. I’ve pasted the token that it returns, should I provide more info?

Here is the registry config:

version: 0.1
log:
  level: debug
  fields:
    service: registry
storage:
    cache:
        blobdescriptor: inmemory
    filesystem:
        rootdirectory: /var/lib/registry
http:
    addr: :5000
    headers:
        X-Content-Type-Options: [nosniff]
    tls:
        certificate: /etc/docker/certs/cert.pem
        key: /etc/docker/certs/key.pem
    clientcas:
        - /etc/docker/certs/ca.pem
auth:
  token:
    service: npinfo-registry
    issuer: distrib.netpoliceinfo.com
    rootcertbundle: /etc/docker/certs/ca.pem

Certificates are self-signed. The registry is hosted in a docker container, based on registry:2.4.0.

4

What does the payload returned by your token service look like? It definitely sounds like that token service isn’t generating something quite the way that distribution expects.

5

I pasted the payload in the first message of the topic, here it is (decoded):
{“access”:[{“type”:“repository”,“name”:“hello-world/hello-world”,“actions”:[“push”,“pull”]}],“jti”:“5f40bf413b6f42569ab17385005ce375”,“sub”:“test”,“exp”:1461772266,“iss”:“my_test_repo”,“iat”:1461768666,“nbf”:1461768666,“aud”:“npinfo-registry”}

The whole token with headers and signature is on the pastebin: http://pastebin.com/tQfYBqMn

The token is generated using PyJWT library (https://github.com/jpadilla/pyjwt). Let me remind that exactly this token server without any changes worked fine in December 2015 with Registry 2.2.0 and Docker Engine 1.9.

6

I have created a minimal environment to reproduce the problem: https://github.com/windj007/docker-token-auth-test

Should I post a ticket?