Docker pull resulting in 'unknown blob error' on some nodes

Setup:

Ours is a cluster of AWS nodes running docker (no swarm) and a local insecure docker registry on one of the nodes. All the docker containers/apps and the registry are discoverable via consul.

Issue:

We are facing the ‘unknown blob’ error while pulling a docker image from local registry. This is not happening on all the nodes in the cluster, the same image can be pulled successfully on some other node. Also this is happening for some of the images:

$ docker pull docker-registry.service.local:5040/iotsp/iot-elasticsearch-sanity:master_17
master_17: Pulling from iotsp/iot-elasticsearch-sanity
2219af950598: Already exists
ceb4033df908: Already exists
445def1dbd86: Already exists
0e42df4d81f1: Downloading
25ae6a6fc1d0: Download complete
e1dd602ba1ef: Download complete
861884f7de6c: Download complete
642d407ead82: Download complete
48868290f9f9: Download complete
139fd3739a4e: Download complete
364a4802d436: Download complete
6e8d74529bcf: Download complete
6e7bab496e0b: Download complete
16c8a9a82187: Download complete
unknown blob

Docker Info:

$ docker info
Containers: 11
 Running: 11
 Paused: 0
 Stopped: 0
Images: 12
Server Version: 17.12.0-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: syslog
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.16.1-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 30.41GiB
Name: tme-eu-8-worker-7
ID: ZEXJ:3QFJ:A65D:BCHU:FQLM:F2ST:QZ2U:W67N:YFGY:UY7B:CAYG:W5R3
Docker Root Dir: /mnt/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 docker-registry.service.local:5040
 127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

As a temporary workaround we are again pulling the image from dockerhub and then pushing it to local docker registry. After this the ‘affected’ image can be pulled on all the nodes.

@rsjethani I am facing the same issue since morning on for some of the images in my private repository. So, I’m looking for a solution. Were you able to find a good solution to this issue?

1 Like