Error when migrating image from v1 registry to v2 registry

Docker Engine
1

Hi everyone,

We are currently trying to migrate (i.e. copy) images from a private v1 registry to a private v2 registry.
Our migration procedure consists of the following steps:

docker pull <image_from_v1_registry>
docker tag <image_from_v1_registry> <image_for_v2_registry>
docker push <image_for_v2_registry>

This procedure worked for some of our images, but failed for the majority.

The error is always the same and happens when the image is pushed (pulling and tagging is rarely a problem).
However, the error occurs at different stages during the push. For some images a considerable amount of layers can be pushed successfully before the error occurs while others fail already at the start.

The error looks as follows:

Stdout at the client (from which the image is pushed):

f1c16dcdd4bf: Image already exists 
7a20706ab657: Pushing [==================================================>] 101.5 MB/101.5 MB
Received unexpected HTTP status: 500 Internal Server Error

Docker Logs of the registry container:

level=warning msg="the ResponseWriter does not implement CloseNotifier" http.request.method=PATCH http.request.uri="/v2/marketplace/runnable-centos6/blobs/uploads/42777140-70c9-48cc-a561-e131212793f2?_state=lwM74ER_UadZz64kem0aKvzKAiRPgxwOCJ0AJIXCufJ7Ik5hbWUiOiJtYXJrZXRwbGFjZS9ydW5uYWJsZS1jZW50b3M2IiwiVVVJRCI6IjQyNzc3MTQwLTcwYzktNDhjYy1hNTYxLWUxMzEyMTI3OTNmMiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNS0wOS0yOFQxMDo0ODoxMi40Mjg4NDY1NFoifQ%3D%3D" 
level=error msg="unable to delete layer upload resources "/docker/registry/v2/repositories/marketplace/runnable-centos6/_uploads/42777140-70c9-48cc-a561-e131212793f2": remove /var/lib/registry/docker/registry/v2/repositories/marketplace/runnable-centos6/_uploads/42777140-70c9-48cc-a561-e131212793f2: directory not empty"  http.request.id=fe2d1445-1d20-416c-9bd6-666800a310a6 http.request.method=PUT
level=error msg="unknown error completing upload: &os.PathError{Op:\"remove\", Path:"/var/lib/registry/docker/registry/v2/repositories/marketplace/runnable-centos6/_uploads/42777140-70c9-48cc-a561-e131212793f2"  Err:0x27}" http.request.method=PUT http.request.useragent="docker/1.8.2 go/go1.4.2 git-commit/0a8c2e3 kernel/3.16.0-49-generic os/linux arch/amd64" version=v2.1.1
level=error msg="unable to delete layer upload resources \"/docker/registry/v2/repositories/marketplace/runnable-centos6/_uploads/42777140-70c9-48cc-a561-e131212793f2\": remove /var/lib/registry/docker/registry/v2/repositories/marketplace/runnable-centos6/_uploads/42777140-70c9-48cc-a561-e131212793f2/.nfs000000000009f583000001ea: device or resource busy"
level=error msg="error canceling upload after error: remove /var/lib/registry/docker/registry/v2/repositories/marketplace/runnable-centos6/_uploads/42777140-70c9-48cc-a561-e131212793f2/.nfs000000000009f583000001ea: device or resource busy" http.request.method=PUT

Someone with the same setup reported the exact same error here: Docker 1.8.1 push issue to Registry 2.1.1 with nfs filesystem · Issue #15833 · moby/moby · GitHub, except that in my case it doesn’t seem to work with older docker clients either.

Setup

Local Client

  • Docker v1.8.2, API v1.20

Private Remote Registry

  • v2
  • NFS backend
  • Docker Client and Server v1.7.1, API v1.19