Docker 1.11 push to private 2.4.0 registry with token authentication

Dear Docker folks,

I’m running a Docker Registry 2.4.0 with token authentication and self-signed certificate.
When I’m trying to push anything to the registry, it starts retrying and finally fails:

docker push localhost:5000/hello-world/hello-world:latest
The push refers to a repository [localhost:5000/hello-world/hello-world]
5f70bf18a086: Retrying in 1 second
33e7801ac047: Retrying in 1 second
unable to decode token response: json: cannot unmarshal string into Go value of type int

In daemon log, these lines appear:
Apr 27 18:07:50 rsuvorov docker[28661]: time=“2016-04-27T18:07:50.050923877+03:00” level=error msg="Upload failed, retrying: unable to decode token response: json: cannot unmarshal string into Go value of type int"
Apr 27 18:08:10 rsuvorov docker[28661]: time=“2016-04-27T18:08:10.047221876+03:00” level=error msg="Upload failed: unable to decode token response: json: cannot unmarshal string into Go value of type int"
Apr 27 18:08:10 rsuvorov docker[28661]: time=“2016-04-27T18:08:10.068000535+03:00” level=error msg="Upload failed: unable to decode token response: json: cannot unmarshal string into Go value of type int"
Apr 27 18:08:10 rsuvorov docker[28661]: time=“2016-04-27T18:08:10.068110884+03:00” level=error msg=“Attempting next endpoint for push after error: unable to decode token response: json: cannot unmarshal string into Go value of type int”

Token server returns a valid token that can be decoded using python: http://pastebin.com/tQfYBqMn
Here is the decoded payload: {“access”:[{“type”:“repository”,“name”:“hello-world/hello-world”,“actions”:[“push”,“pull”]}],“jti”:“5f40bf413b6f42569ab17385005ce375”,“sub”:“test”,“exp”:1461772266,“iss”:“my_test_repo”,“iat”:1461768666,“nbf”:1461768666,“aud”:“npinfo-registry”}

This setup used to work in December 2015 with docker 1.9.

$ docker info
Containers: 25
Running: 4
Paused: 0
Stopped: 21
Images: 719
Server Version: 1.11.1
Storage Driver: aufs
Root Dir: /opt/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 621
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host bridge null
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 23.55 GiB
Name: rsuvorov
ID: VONR:B6MM:7AIE:JM4G:TRRO:PLQC:HVAG:6HAQ:EWSS:5CQG:NO22:7SJF
Docker Root Dir: /opt/var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

What token server are you running, and what does your config.yml look like for distribution 2.4?

I’m running my own token server based on Django. I’ve pasted the token that it returns, should I provide more info?

Here is the registry config:

version: 0.1
log:
  level: debug
  fields:
    service: registry
storage:
    cache:
        blobdescriptor: inmemory
    filesystem:
        rootdirectory: /var/lib/registry
http:
    addr: :5000
    headers:
        X-Content-Type-Options: [nosniff]
    tls:
        certificate: /etc/docker/certs/cert.pem
        key: /etc/docker/certs/key.pem
    clientcas:
        - /etc/docker/certs/ca.pem
auth:
  token:
    service: npinfo-registry
    issuer: distrib.netpoliceinfo.com
    rootcertbundle: /etc/docker/certs/ca.pem

Certificates are self-signed. The registry is hosted in a docker container, based on registry:2.4.0.

What does the payload returned by your token service look like? It definitely sounds like that token service isn’t generating something quite the way that distribution expects.

I pasted the payload in the first message of the topic, here it is (decoded):
{“access”:[{“type”:“repository”,“name”:“hello-world/hello-world”,“actions”:[“push”,“pull”]}],“jti”:“5f40bf413b6f42569ab17385005ce375”,“sub”:“test”,“exp”:1461772266,“iss”:“my_test_repo”,“iat”:1461768666,“nbf”:1461768666,“aud”:“npinfo-registry”}

The whole token with headers and signature is on the pastebin: http://pastebin.com/tQfYBqMn

The token is generated using PyJWT library (https://github.com/jpadilla/pyjwt). Let me remind that exactly this token server without any changes worked fine in December 2015 with Registry 2.2.0 and Docker Engine 1.9.

I have created a minimal environment to reproduce the problem: https://github.com/windj007/docker-token-auth-test

Should I post a ticket?