Docker Community Forums

Share and learn in the Docker community.

Private registry connect to rados false, please help [resolve]

(Signmem) #1

I want to create my private registry,
Using rados backend for store private images. but now I cant start up my registry docker image.

My ceph cluster works fine.

[root@sh-storage-128204 ~]# ceph -s
cluster 63c85f43-43a2-4d3e-b5a1-9933a6105438`
 health HEALTH_OK
 monmap e2: 3 mons at {sh-storage-128200=,sh-storage-128202=,sh-storage-128203=}, election epoch 10, quorum 0,1,2 sh-storage-128200,sh-storage-128202,sh-storage-128203
 osdmap e172: 32 osds: 32 up, 32 in
  pgmap v6809: 7672 pgs, 12 pools, 3613 bytes data, 53 objects
        34029 MB used, 8905 GB / 8938 GB avail
        7672 active+clean

And I can connect raods on the host where plan to use for private docker registry

[root@sh-storage-128204 ~]# rados lspools
[root@sh-storage-128204 ~]# rados -p dockerpools ls

there is my ceph.conf 

    [root@sh-storage-128204 ~]# cat /etc/ceph/ceph.conf
    fsid = 63c85f43-43a2-4d3e-b5a1-9933a6105438
    mon_initial_members = sh-storage-128200, sh-storage-128202, sh-storage-128203
    mon_host =,,
    auth_cluster_required = cephx
    auth_service_required = cephx
    auth_client_required = cephx
    filestore_xattr_use_omap = true
    public_network =
    cluster_network =
    osd_journal_size = 1024
    osd_pool_default_size = 3
    osd_pool_default_min_size = 1
    osd_pool_default_pg_num = 512
    osd_pool_default_pgp_num = 512
    osd_crush_chooseleaf_type = 1
    max_open_files = 13104765

my  docker host info

    [root@sh-storage-128204 ~]# docker info
    Containers: 3
    Images: 15
    Storage Driver: devicemapper
     Pool Name: docker-8:17-1075308946-pool
     Pool Blocksize: 65.54 kB
     Backing Filesystem: xfs
     Data file:
     Metadata file:
     Data Space Used: 1.113 GB
     Data Space Total: 107.4 GB
     Data Space Available: 106.3 GB
     Metadata Space Used: 1.642 MB
     Metadata Space Total: 2.147 GB
     Metadata Space Available: 2.146 GB
     Udev Sync Supported: true
     Library Version: 1.02.93-RHEL7 (2015-01-28)
    Execution Driver: native-0.2
    Kernel Version: 3.10.0-229.el7.x86_64
    Operating System: CentOS Linux 7 (Core)
    CPUs: 24
    Total Memory: 125.7 GiB

there is my raods.yml 

    version: 0.1
      level: debug
      formatter: text
        service: registry
    loglevel: debug
        poolname: rbd
        username: admin

I get the registry image from

    [root@sh-storage-128204 ~]# docker images
    REPOSITORY                   TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
    test                         7.1.1503            4398cb5aed12        2 days ago          227.6 MB
    registry                     2                   b2c682ed7e4c        4 weeks ago         224.5 MB   7.1.1503            fab4b1df8eb1        12 weeks ago        212.1 MB
    centos                       7.1.1503            fab4b1df8eb1        12 weeks ago        212.1 MB

But I can not startup my private registry

    [root@sh-storage-128204 ~]# docker -D run -d -p 5000:5000 -e CEPH_CONF=/etc/ceph/ceph.conf -v /tmp/registry:/registry -v /root/docker/rados.yml:/etc/docker/registry/config.yml -v /etc/docker/certs.d:/certs  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/docker.crt -e REGISTRY_HTTP_TLS_KEY=/certs/ca.key -d registry:2
    DEBU[0000] End of CmdRun(), Waiting for hijack to finish.

there is some debug info

    [root@sh-storage-128204 ~]# docker logs b5d4ef596ba26ff624351ec08709eb21b6d9cc443c98cdad5c815657f97ffc22
    time="2016-01-07T02:34:51.833833628Z" level=debug msg="using \"text\" logging formatter"
    time="2016-01-07T02:34:51.836710537Z" level=info msg="Opening connection to pool rbd using user admin"
    panic: rados: ret=-22
    goroutine 1 [running]:, 0xc82033f6e0, 0xc820382c80, 0x7f088c0b2280)
            /go/src/ +0x3b6, 0xc82033f6e0, 0xc820382c80, 0xc820382c80, 0x0, 0x0)
            /go/src/ +0x2b0, 0xc820303ba0, 0x1, 0x1)
            /go/src/ +0x2b3*Command).execute(0x134e2e0, 0xc82000a090, 0x1, 0x1, 0x0, 0x0)
        /go/src/ +0x6e3*Command).Execute(0x134e2e0, 0x0, 0x0)
        /go/src/ +0x180
        /go/src/ +0x23

    goroutine 17 [syscall, locked to thread]:
        /usr/local/go/src/runtime/asm_amd64.s:1721 +0x1

    goroutine 6 [syscall]:
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
    created by os/signal.init.1
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

there is something wrong with [ panic: rados: ret=-22 ]

reference the link:

it show up 

    panic: Unable to create ceph connection to cluster=ceph with user=admin: rados: ret=-22

Can anyone help me resolve this problem ?
Thank you very much.

(Signmem) #2

modify startup script

docker -D run -d -p 5000:5000 -e CEPH_CONF=/etc/ceph/ceph.conf -v /var/lib/ceph:/var/lib/ceph  -v /etc/ceph:/etc/ceph -v /root/docker/rados.yml:/etc/docker/registry/config.yml -v /etc/docker/certs.d:/certs  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/docker.crt -e REGISTRY_HTTP_TLS_KEY=/certs/ca.key -d registry:2

then problem resolve.