~evilnick/clouddocs/maas-tags

Title: Appendix - Ceph and OpenStack
Status: Done

# Appendix: Ceph and OpenStack

Ceph stripes block device images as objects across a cluster. This way it provides
a better performance than standalone server. OpenStack is able to use Ceph Block Devices
through `libvirt`, which configures the QEMU interface to `librbd`. 

To use Ceph Block Devices with OpenStack, you must install QEMU, `libvirt`, and OpenStack 
first. It's recommended to use a separate physical node for your OpenStack installation. 
OpenStack recommends a minimum of 8GB of RAM and a quad-core processor. 

Three parts of OpenStack integrate with Ceph’s block devices:

- Images: OpenStack Glance manages images for VMs. Images are immutable. OpenStack 
  treats images as binary blobs and downloads them accordingly.
- Volumes: Volumes are block devices. OpenStack uses volumes to boot VMs, or to 
  attach volumes to running VMs. OpenStack manages volumes using Cinder services.
- Guest Disks: Guest disks are guest operating system disks. By default, when you 
  boot a virtual machine, its disk appears as a file on the filesystem of the 
  hypervisor (usually under /var/lib/nova/instances/<uuid>/). Prior OpenStack Havana, 
  the only way to boot a VM in Ceph was to use the boot from volume functionality 
  from Cinder. However, now it is possible to directly boot every virtual machine 
  inside Ceph without using Cinder. This is really handy because it allows us to 
  easily perform maintenance operation with the live-migration process. On the other 
  hand, if your hypervisor dies it is also really convenient to trigger Nova evacuate 
  and almost seamlessly run the virtual machine somewhere else.

You can use OpenStack Glance to store images in a Ceph Block Device, and you can 
use Cinder to boot a VM using a copy-on-write clone of an image.

## Create a pool

By default, Ceph block devices use the `rbd` pool. You may use any available pool. 
We recommend creating a pool for Cinder and a pool for Glance. Ensure your Ceph 
cluster is running, then create the pools.

````
ceph osd pool create volumes 128
ceph osd pool create images 128
ceph osd pool create backups 128
````

## Configure OpenStack Ceph Clients

The nodes running `glance-api`, `cinder-volume`, `nova-compute` and `cinder-backup` act 
as Ceph clients. Each requires the `ceph.conf` file

````
ssh {your-openstack-server} sudo tee /etc/ceph/ceph.conf </etc/ceph/ceph.conf
````

On the `glance-api` node, you’ll need the Python bindings for `librbd`

````
sudo apt-get install python-ceph
sudo yum install python-ceph
````

On the `nova-compute`, `cinder-backup` and on the `cinder-volume` node, use both the 
Python bindings and the client command line tools

````
sudo apt-get install ceph-common
sudo yum install ceph
````

If you have cephx authentication enabled, create a new user for Nova/Cinder and 
Glance. Execute the following

````
ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images'
ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'
````

Add the keyrings for `client.cinder`, `client.glance`, and `client.cinder-backup` 
to the appropriate nodes and change their ownership

````
ceph auth get-or-create client.glance | ssh {your-glance-api-server} sudo tee /etc/ceph/ceph.client.glance.keyring
ssh {your-glance-api-server} sudo chown glance:glance /etc/ceph/ceph.client.glance.keyring
ceph auth get-or-create client.cinder | ssh {your-volume-server} sudo tee /etc/ceph/ceph.client.cinder.keyring
ssh {your-cinder-volume-server} sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring
ceph auth get-or-create client.cinder-backup | ssh {your-cinder-backup-server} sudo tee /etc/ceph/ceph.client.cinder-backup.keyring
ssh {your-cinder-backup-server} sudo chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring
````

Nodes running `nova-compute` need the keyring file for the `nova-compute` process. 
They also need to store the secret key of the `client.cinder` user in `libvirt`. The 
`libvirt` process needs it to access the cluster while attaching a block device 
from Cinder.

Create a temporary copy of the secret key on the nodes running `nova-compute`

````
ceph auth get-key client.cinder | ssh {your-compute-node} tee client.cinder.key
````

Then, on the compute nodes, add the secret key to `libvirt` and remove the 
temporary copy of the key

````
uuidgen
457eb676-33da-42ec-9a8c-9293d545c337

cat > secret.xml <<EOF
<secret ephemeral='no' private='no'>
  <uuid>457eb676-33da-42ec-9a8c-9293d545c337</uuid>
  <usage type='ceph'>
    <name>client.cinder secret</name>
  </usage>
</secret>
EOF
sudo virsh secret-define --file secret.xml
Secret 457eb676-33da-42ec-9a8c-9293d545c337 created
sudo virsh secret-set-value --secret 457eb676-33da-42ec-9a8c-9293d545c337 --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml
````

Save the uuid of the secret for configuring `nova-compute` later.

**Important** You don’t necessarily need the UUID on all the compute nodes. 
However from a platform consistency perspective it’s better to keep the 
same UUID.

## Configure OpenStack to use Ceph

### Glance

Glance can use multiple back ends to store images. To use Ceph block devices 
by default, edit `/etc/glance/glance-api.conf` and add

````
default_store=rbd
rbd_store_user=glance
rbd_store_pool=images
````

If want to enable copy-on-write cloning of images into volumes, also add:

````
show_image_direct_url=True
````

Note that this exposes the back end location via Glance’s API, so 
the endpoint with this option enabled should not be publicly 
accessible.

### Cinder

OpenStack requires a driver to interact with Ceph block devices. You 
must also specify the pool name for the block device. On your 
OpenStack node, edit `/etc/cinder/cinder.conf` by adding

````
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_pool=volumes
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot=false
rbd_max_clone_depth=5
glance_api_version=2
````

If you’re using cephx authentication, also configure the user and 
uuid of the secret you added to `libvirt` as documented earlier

````
rbd_user=cinder
rbd_secret_uuid=457eb676-33da-42ec-9a8c-9293d545c337
````

## Cinder Backup

OpenStack Cinder Backup requires a specific daemon so don’t 
forget to install it. On your Cinder Backup node, 
edit `/etc/cinder/cinder.conf` and add:

````
backup_driver=cinder.backup.drivers.ceph
backup_ceph_conf=/etc/ceph/ceph.conf
backup_ceph_user=cinder-backup
backup_ceph_chunk_size=134217728
backup_ceph_pool=backups
backup_ceph_stripe_unit=0
backup_ceph_stripe_count=0
restore_discard_excess_bytes=true
````

### Nova

In order to boot all the virtual machines directly into Ceph Nova must be 
configured. On every Compute nodes, edit `/etc/nova/nova.conf` and add

````
libvirt_images_type=rbd
libvirt_images_rbd_pool=volumes
libvirt_images_rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=cinder
rbd_secret_uuid=457eb676-33da-42ec-9a8c-9293d545c337
````

It is also a good practice to disable any file injection. Usually, while 
booting an instance Nova attempts to open the rootfs of the virtual machine. 
Then, it injects directly into the filesystem things like: password, ssh 
keys etc... At this point, it is better to rely on the metadata service 
and cloud-init. On every Compute nodes, edit `/etc/nova/nova.conf` and add

````
libvirt_inject_password=false
libvirt_inject_key=false
libvirt_inject_partition=-2
````

## Restart OpenStack

To activate the Ceph block device driver and load the block device pool name 
into the configuration, you must restart OpenStack. 

````
sudo glance-control api restart
sudo service nova-compute restart
sudo service cinder-volume restart
sudo service cinder-backup restart
````

Once OpenStack is up and running, you should be able to create a volume 
and boot from it.