Ceph Research / Overview


#1

The following in a high level overview of the Ceph Distributed Object Store for reference & exploring the possibility of adding support for it to MiQ.

Ceph - Distributed Object Store

  • monitoring : ceph-mon

  • metadata : ceph-mds

  • object store devices : ceph-osd

  • rest gateways : used to provide API access & adapt the interface

    Multiple filesystem backends are supported, recommended ext4/xfs/btrfs/zfs

    Object Store: Provides snapshots, replication, consistency checking

    Primarily runs through httpd / fastcgi & configurable gateway
    (provided by the radosgw-agent package)

    librados - library used to interface w/ osd interfaces to get/set
    objects. Wrappers exist for many major languages

    rados_create2(&cluster, cluster_name, user_name, flags);
    rados_connect(cluster);
    rados_aio_read(io, “hw”, comp, read_res, 12, 0);
    rados_wait_for_complete(comp);
    rados_aio_release(comp);
    rados_ioctx_destroy(io);
    rados_shutdown(cluster);

    Admin REST API:

  • GET /{admin}/usage?format=json HTTP/1.1

  • DELETE /{admin}/usage?format=json HTTP/1.1

  • GET /{admin}/user?format=json HTTP/1.1

  • PUT /{admin}/user?format=json HTTP/1.1

  • POST /{admin}/user?format=json HTTP/1.1

  • DELETE /{admin}/user?format=json HTTP/1.1

  • PUT /{admin}/user?subuser&format=json HTTP/1.1

  • POST /{admin}/user?subuser&format=json HTTP/1.1

  • DELETE /{admin}/user?subuser&format=json HTTP/1.1

  • PUT /{admin}/user?key&format=json HTTP/1.1

  • DELETE /{admin}/user?key&format=json HTTP/1.1

  • GET /{admin}/bucket?format=json HTTP/1.1

  • GET /{admin}/bucket?index&format=json HTTP/1.1

  • DELETE /{admin}/bucket?format=json HTTP/1.1

  • POST /{admin}/bucket?format=json HTTP/1.1

  • PUT /{admin}/bucket?format=json HTTP/1.1

  • DELETE /{admin}/bucket?object&format=json HTTP/1.1

  • GET /{admin}/bucket?policy&format=json HTTP/1.1

  • PUT /{admin}/user?caps&format=json HTTP/1.1

  • DELETE /{admin}/user?caps&format=json HTTP/1.1

  • GET /admin/user?quota&uid=&quota-type=user

  • PUT /admin/user?quota&uid=&quota-type=user

  • GET /admin/user?quota&uid=&quota-type=bucket

  • PUT /admin/user?quota&uid=&quota-type=bucket

    S3 Support:

  • List Buckets Supported

  • Delete Bucket Supported

  • Create Bucket Supported

  • Bucket Lifecycle Not Supported

  • Policy (Buckets, Objects) Not Supported

  • Bucket Website Not Supported

  • Bucket ACLs (Get, Put) Supported

  • Bucket Location Supported

  • Bucket Notification Not Supported

  • Bucket Object Versions Not Supported

  • Get Bucket Info (HEAD) Supported

  • Bucket Request Payment Not Supported

  • Put Object Supported

  • Delete Object Supported

  • Get Object Supported

  • Object ACLs (Get, Put) Supported

  • Get Object Info (HEAD) Supported

  • POST Object Supported

  • Copy Object Supported

  • Multipart Uploads Supported

    Swift Support:

  • Authentication Supported

  • Get Account Metadata Supported

  • Swift ACLs Supported

  • List Containers Supported

  • Delete Container Supported

  • Create Container Supported

  • Get Container Metadata Supported

  • Update Container Metadata Supported

  • Delete Container Metadata Supported

  • List Objects Supported

  • Static Website Not Supported

  • Create Object Supported

  • Create Large Object Supported

  • Delete Object Supported

  • Get Object Supported

  • Copy Object Supported

  • Get Object Metadata Supported

  • Update Object Metadata Supported

  • Expiring Objects Not Supported

  • Object Versioning Not Supported

  • CORS Not Supported

    CLI Utils:

  • ceph : storage cluster client

  • ceph-authtool : manages authentication keys/mechanisms

  • ceph-deploy : deploys config (/etc/ceph) updates across all clusters

  • rados : rados service client

  • radosgw-admin : manages rados-gw service devices.
    Subcommands:

    • user create
    • user modify
    • user info
    • user rm
    • user suspend
    • user enable
    • user check
    • user stats
    • caps add
    • caps rm
    • subuser create
    • subuser modify
    • subuser rm
    • key create
    • key rm
    • bucket list
    • bucket link
    • bucket unlink
    • bucket stats
    • bucket rm
    • bucket check
    • object rm
    • object unlink
    • quota set
    • quota enable
    • quota disable
    • region get
    • regions list
    • region set
    • region default
    • region-map get
    • region-map set
    • zone get
    • zone set
    • zone list
    • pool add
    • pool rm
    • pools list
    • policy
    • log list
    • log show
    • log rm
    • usage show
    • usage trim
    • temp remove
    • gc list
    • gc process
    • metadata get
    • metadata put
    • metadata rm
    • metadata list
    • mdlog list
    • mdlog trim
    • bilog list
    • bilog trim
    • datalog list
    • datalog trim
    • opstate list
    • opstate set
    • opstate renew
    • opstate rm
    • replicalog get
    • replicalog delete

    Creating pools:

    ceph osd pool create {poolname} {pg-num} {pgp-num} {replicated | erasure} [{erasure-code-profile}] {ruleset-name} {ruleset-number}
    rados lspools

    Updating config:

    ceph-deploy --overwrite-conf config pull {hostname}

    Standard apache configuration also applies to setup and secure the access
    to Ceph.

    Block Store: manages block devices which are thinly provisioned across OSDs

    Rados Block Device (RBD) - loaded via kernel modules, librbd, cli utils

    Provides snapshots, caching, kvm/libvirt integration

    librdb (python):

    with rados.Rados(conffile=‘my_ceph.conf’) as cluster:
    with cluster.open_ioctx(‘mypool’) as ioctx:
    rbd_inst = rbd.RBD()
    size = 4 * 1024**3 # 4 GiB
    rbd_inst.create(ioctx, ‘myimage’, size)
    with rbd.Image(ioctx, ‘myimage’) as image:
    data = ‘foo’ * 200
    image.write(data, 0)

    FS: posix compliant filesystem utilizing the block store & metadata server,
    provides journaling and standard fs configuration mechanisms

    libcephfs can be used to interface w/ fs

    Linkage: