I want to create my private registry,
Using rados backend for store private images. but now I cant start up my registry docker image.
My ceph cluster works fine.
[root@sh-storage-128204 ~]# ceph -s
cluster 63c85f43-43a2-4d3e-b5a1-9933a6105438`
health HEALTH_OK
monmap e2: 3 mons at {sh-storage-128200=10.198.128.200:6789/0,sh-storage-128202=10.198.128.202:6789/0,sh-storage-128203=10.198.128.203:6789/0}, election epoch 10, quorum 0,1,2 sh-storage-128200,sh-storage-128202,sh-storage-128203
osdmap e172: 32 osds: 32 up, 32 in
pgmap v6809: 7672 pgs, 12 pools, 3613 bytes data, 53 objects
34029 MB used, 8905 GB / 8938 GB avail
7672 active+clean
And I can connect raods on the host where plan to use for private docker registry
[root@sh-storage-128204 ~]# rados lspools
rbd
.rgw.root
.rgw.control
.rgw
.rgw.gc
.users.uid
.users
.users.swift
.users.email
.rgw.buckets.index
cephpools
dockerpools
[root@sh-storage-128204 ~]# rados -p dockerpools ls
obj1
there is my ceph.conf
[root@sh-storage-128204 ~]# cat /etc/ceph/ceph.conf
[global]
fsid = 63c85f43-43a2-4d3e-b5a1-9933a6105438
mon_initial_members = sh-storage-128200, sh-storage-128202, sh-storage-128203
mon_host = 10.198.128.200,10.198.128.202,10.198.128.203
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
public_network = 10.198.128.0/23
cluster_network = 240.40.128.0/21
osd_journal_size = 1024
osd_pool_default_size = 3
osd_pool_default_min_size = 1
osd_pool_default_pg_num = 512
osd_pool_default_pgp_num = 512
osd_crush_chooseleaf_type = 1
max_open_files = 13104765
my docker host info
[root@sh-storage-128204 ~]# docker info
Containers: 3
Images: 15
Storage Driver: devicemapper
Pool Name: docker-8:17-1075308946-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 1.113 GB
Data Space Total: 107.4 GB
Data Space Available: 106.3 GB
Metadata Space Used: 1.642 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.146 GB
Udev Sync Supported: true
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 24
Total Memory: 125.7 GiB
Name: sh-storage-128204.sh.vclound.com
ID: QV4X:VHQE:EMOI:4TBJ:FZ6K:3N3C:A64Y:PRYR:X2QZ:HHUB:OTND:ZSFF
there is my raods.yml
version: 0.1
log:
level: debug
formatter: text
fields:
service: registry
loglevel: debug
storage:
rados:
poolname: rbd
username: admin
I get the registry image from docker.io
[root@sh-storage-128204 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
test 7.1.1503 4398cb5aed12 2 days ago 227.6 MB
registry 2 b2c682ed7e4c 4 weeks ago 224.5 MB
10.198.128.204:5000/centos 7.1.1503 fab4b1df8eb1 12 weeks ago 212.1 MB
centos 7.1.1503 fab4b1df8eb1 12 weeks ago 212.1 MB
But I can not startup my private registry
[root@sh-storage-128204 ~]# docker -D run -d -p 5000:5000 -e CEPH_CONF=/etc/ceph/ceph.conf -v /tmp/registry:/registry -v /root/docker/rados.yml:/etc/docker/registry/config.yml -v /etc/docker/certs.d:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/docker.crt -e REGISTRY_HTTP_TLS_KEY=/certs/ca.key -d registry:2
b5d4ef596ba26ff624351ec08709eb21b6d9cc443c98cdad5c815657f97ffc22
DEBU[0000] End of CmdRun(), Waiting for hijack to finish.
there is some debug info
[root@sh-storage-128204 ~]# docker logs b5d4ef596ba26ff624351ec08709eb21b6d9cc443c98cdad5c815657f97ffc22
time="2016-01-07T02:34:51.833833628Z" level=debug msg="using \"text\" logging formatter"
time="2016-01-07T02:34:51.836710537Z" level=info msg="Opening connection to pool rbd using user admin"
panic: rados: ret=-22
goroutine 1 [running]:
github.com/docker/distribution/registry/handlers.NewApp(0x7f088c0b2280, 0xc82033f6e0, 0xc820382c80, 0x7f088c0b2280)
/go/src/github.com/docker/distribution/registry/handlers/app.go:105 +0x3b6
github.com/docker/distribution/registry.NewRegistry(0x7f088c0b22c0, 0xc82033f6e0, 0xc820382c80, 0xc820382c80, 0x0, 0x0)
/go/src/github.com/docker/distribution/registry/registry.go:94 +0x2b0
github.com/docker/distribution/registry.glob.func1(0x134e2e0, 0xc820303ba0, 0x1, 0x1)
/go/src/github.com/docker/distribution/registry/registry.go:57 +0x2b3
github.com/spf13/cobra.(*Command).execute(0x134e2e0, 0xc82000a090, 0x1, 0x1, 0x0, 0x0)
/go/src/github.com/docker/distribution/Godeps/_workspace/src/github.com/spf13/cobra/command.go:495 +0x6e3
github.com/spf13/cobra.(*Command).Execute(0x134e2e0, 0x0, 0x0)
/go/src/github.com/docker/distribution/Godeps/_workspace/src/github.com/spf13/cobra/command.go:560 +0x180
main.main()
/go/src/github.com/docker/distribution/cmd/registry/main.go:22 +0x23
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1721 +0x1
goroutine 6 [syscall]:
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x37
there is something wrong with [ panic: rados: ret=-22 ]
reference the link:
https://github.com/yp-engineering/rbd-docker-plugin/blob/master/driver.go
it show up
panic: Unable to create ceph connection to cluster=ceph with user=admin: rados: ret=-22
Can anyone help me resolve this problem ?
Thank you very much.