ceph rbd block device as persistent storage for openshift

After Fedora 23 – ceph installation and Fedora 23 – Openshift installation setup it is now time to hook openshift environment to use CEPH storage backend

Openshift pods will be using CEPH rados block devices as persistent storage and to achieve this one option is to follow below steps

  • create ceph pool and desired number of images on top of it, this can be done manually or using ceph-pool-setup.sh script. If ceph-pool-setup.sh is used,read README before running it.
  • create ceph-secret file. As an example is possible to use ceph-secret
  • define persistent volume and persistent volume claim. Yaml example files ceph-pv and ceph-pv-claim
  • create pod file adapting it to use ceph pv and ceph pv claims pod-pv-pvc-ceph
  • In above examples is necessary to change variables to suit different environments ( ceph pool name, ceph monitor(s) ip addresses … )

    Once all is in place, then running at below on Ceph cluster and after Openshift master will create pod which will in return start using rbd as persistent

    Ceph side

    # ./ceph-pool-setup -a c -p mypool -i 1 -r 3 -s 1

    This will create three way replicated ceph poll with name mypool, with one image on top of it with size of 1 GB

    Openshift side

    # oc create -f ceph-secrets.yaml
    # oc create -f ceph-pv.yaml 
    # oc create -f ceph-pv-claim.yaml
    # oc create -f pod-pv-pvc-ceph.json

    If all is fine pod should start and mount rbd inside pod with ext4 file system preformanted

    # oc rsh pod 
    # mount | grep rbd 
    /dev/rbd0 on /mnt/ceph type ext4 (rw,relatime,seclabel,stripe=1024,data=ordered)

    This setup will enable openshift pods to use ceph rbd device as persistent storage, and in case pod is removed, and started at some other openshift node it will get same data in case it has access to rbd device which was used before pod was deleted. As name said, this is persistent volume and it should persist across pod re-creation.


    #ceph, #ceph-rbd, #ceph-storage, #kubenetes, #openshift