Docker Community Forums

Share and learn in the Docker community.

Private registry connect to rados false, please help [resolve]


(Signmem) #1

I want to create my private registry,
Using rados backend for store private images. but now I cant start up my registry docker image.

My ceph cluster works fine.

[root@sh-storage-128204 ~]# ceph -s
cluster 63c85f43-43a2-4d3e-b5a1-9933a6105438`
 health HEALTH_OK
 monmap e2: 3 mons at {sh-storage-128200=10.198.128.200:6789/0,sh-storage-128202=10.198.128.202:6789/0,sh-storage-128203=10.198.128.203:6789/0}, election epoch 10, quorum 0,1,2 sh-storage-128200,sh-storage-128202,sh-storage-128203
 osdmap e172: 32 osds: 32 up, 32 in
  pgmap v6809: 7672 pgs, 12 pools, 3613 bytes data, 53 objects
        34029 MB used, 8905 GB / 8938 GB avail
        7672 active+clean

And I can connect raods on the host where plan to use for private docker registry

[root@sh-storage-128204 ~]# rados lspools
rbd
.rgw.root
.rgw.control
.rgw
.rgw.gc
.users.uid
.users
.users.swift
.users.email
.rgw.buckets.index
cephpools
dockerpools
[root@sh-storage-128204 ~]# rados -p dockerpools ls
obj1

there is my ceph.conf 


    [root@sh-storage-128204 ~]# cat /etc/ceph/ceph.conf
    [global]
    fsid = 63c85f43-43a2-4d3e-b5a1-9933a6105438
    mon_initial_members = sh-storage-128200, sh-storage-128202, sh-storage-128203
    mon_host = 10.198.128.200,10.198.128.202,10.198.128.203
    auth_cluster_required = cephx
    auth_service_required = cephx
    auth_client_required = cephx
    filestore_xattr_use_omap = true
    public_network = 10.198.128.0/23
    cluster_network = 240.40.128.0/21
    osd_journal_size = 1024
    osd_pool_default_size = 3
    osd_pool_default_min_size = 1
    osd_pool_default_pg_num = 512
    osd_pool_default_pgp_num = 512
    osd_crush_chooseleaf_type = 1
    max_open_files = 13104765


my  docker host info


    [root@sh-storage-128204 ~]# docker info
    Containers: 3
    Images: 15
    Storage Driver: devicemapper
     Pool Name: docker-8:17-1075308946-pool
     Pool Blocksize: 65.54 kB
     Backing Filesystem: xfs
     Data file:
     Metadata file:
     Data Space Used: 1.113 GB
     Data Space Total: 107.4 GB
     Data Space Available: 106.3 GB
     Metadata Space Used: 1.642 MB
     Metadata Space Total: 2.147 GB
     Metadata Space Available: 2.146 GB
     Udev Sync Supported: true
     Library Version: 1.02.93-RHEL7 (2015-01-28)
    Execution Driver: native-0.2
    Kernel Version: 3.10.0-229.el7.x86_64
    Operating System: CentOS Linux 7 (Core)
    CPUs: 24
    Total Memory: 125.7 GiB
    Name: sh-storage-128204.sh.vclound.com
    ID: QV4X:VHQE:EMOI:4TBJ:FZ6K:3N3C:A64Y:PRYR:X2QZ:HHUB:OTND:ZSFF


there is my raods.yml 

    version: 0.1
    log:
      level: debug
      formatter: text
      fields:
        service: registry
    loglevel: debug
    storage:
      rados:
        poolname: rbd
        username: admin

I get the registry image from docker.io


    [root@sh-storage-128204 ~]# docker images
    REPOSITORY                   TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
    test                         7.1.1503            4398cb5aed12        2 days ago          227.6 MB
    registry                     2                   b2c682ed7e4c        4 weeks ago         224.5 MB
    10.198.128.204:5000/centos   7.1.1503            fab4b1df8eb1        12 weeks ago        212.1 MB
    centos                       7.1.1503            fab4b1df8eb1        12 weeks ago        212.1 MB


But I can not startup my private registry


    [root@sh-storage-128204 ~]# docker -D run -d -p 5000:5000 -e CEPH_CONF=/etc/ceph/ceph.conf -v /tmp/registry:/registry -v /root/docker/rados.yml:/etc/docker/registry/config.yml -v /etc/docker/certs.d:/certs  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/docker.crt -e REGISTRY_HTTP_TLS_KEY=/certs/ca.key -d registry:2
    b5d4ef596ba26ff624351ec08709eb21b6d9cc443c98cdad5c815657f97ffc22
    DEBU[0000] End of CmdRun(), Waiting for hijack to finish.


there is some debug info

    [root@sh-storage-128204 ~]# docker logs b5d4ef596ba26ff624351ec08709eb21b6d9cc443c98cdad5c815657f97ffc22
    time="2016-01-07T02:34:51.833833628Z" level=debug msg="using \"text\" logging formatter"
    time="2016-01-07T02:34:51.836710537Z" level=info msg="Opening connection to pool rbd using user admin"
    panic: rados: ret=-22
    
    goroutine 1 [running]:
    github.com/docker/distribution/registry/handlers.NewApp(0x7f088c0b2280, 0xc82033f6e0, 0xc820382c80, 0x7f088c0b2280)
            /go/src/github.com/docker/distribution/registry/handlers/app.go:105 +0x3b6
    github.com/docker/distribution/registry.NewRegistry(0x7f088c0b22c0, 0xc82033f6e0, 0xc820382c80, 0xc820382c80, 0x0, 0x0)
            /go/src/github.com/docker/distribution/registry/registry.go:94 +0x2b0
    github.com/docker/distribution/registry.glob.func1(0x134e2e0, 0xc820303ba0, 0x1, 0x1)
            /go/src/github.com/docker/distribution/registry/registry.go:57 +0x2b3
    github.com/spf13/cobra.(*Command).execute(0x134e2e0, 0xc82000a090, 0x1, 0x1, 0x0, 0x0)
        /go/src/github.com/docker/distribution/Godeps/_workspace/src/github.com/spf13/cobra/command.go:495 +0x6e3
    github.com/spf13/cobra.(*Command).Execute(0x134e2e0, 0x0, 0x0)
        /go/src/github.com/docker/distribution/Godeps/_workspace/src/github.com/spf13/cobra/command.go:560 +0x180
    main.main()
        /go/src/github.com/docker/distribution/cmd/registry/main.go:22 +0x23

    goroutine 17 [syscall, locked to thread]:
    runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1721 +0x1

    goroutine 6 [syscall]:
    os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
    created by os/signal.init.1
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x37


there is something wrong with [ panic: rados: ret=-22 ]


reference the link:

    https://github.com/yp-engineering/rbd-docker-plugin/blob/master/driver.go


it show up 

    panic: Unable to create ceph connection to cluster=ceph with user=admin: rados: ret=-22

Can anyone help me resolve this problem ?
Thank you very much.

(Signmem) #2

modify startup script

docker -D run -d -p 5000:5000 -e CEPH_CONF=/etc/ceph/ceph.conf -v /var/lib/ceph:/var/lib/ceph  -v /etc/ceph:/etc/ceph -v /root/docker/rados.yml:/etc/docker/registry/config.yml -v /etc/docker/certs.d:/certs  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/docker.crt -e REGISTRY_HTTP_TLS_KEY=/certs/ca.key -d registry:2

then problem resolve.