you are talking with two different services, which have two different API’s. They look similar but are not.
registry-1.docker.io is the docker-registry used for storing images, this API is public, but requires authorization.
index.docker.io is the “index” now called RegistryHub, this tags endpoint is not public, and might change at any time. I would not use this one, since it is for internal use only, and may change without warning.
Answers to your questions.
It fails because you need authorization when talking with the registry.
You need an auth token, which comes from the index/ registryHub, if you auth against the registryHub it will give you a token which you can use against the docker-registry.
You would need to do what you have, but with an valid token to the docker-registry.
We hope to offer an official API in the future that will make all of this much easier. I don’t have an ETA on when this might be available.
Ok after getting the token I am stuck. I have tried every possible way of supplying the token to the registry but it just keeps replying with 401, auth required. Here is the header which I am supplying to the registry server https://registry-1.docker.io/v1/repositories/_/ubuntu/tags:
As you can see I am supplying the token every possible way I could think of, including supplying it as cookie as returned by the Hub. No luck. I tried to packet-sniff how the docker cli does it but had no luck either.