nova

Adventures in OpenStack – Nova Limits

vancluevertech,

This morning I was setting up some test instances on my home OpenStack deployment and ran into some issues. The first instance launched okay, but the second failed, telling me that no suitable host was found.

I have not really oversubscribed, or so I thought. Turns out I was oversubscribed on both RAM and Disk.

OpenStack has defaults surrounding this:
* Disk is not oversubscribed by default. There is a 1-1 subscription per host. In a production setup, this should never be changed – an instance launching on a host that could run out of space and take down all the instances on that node is definitely a bad thing.
* Memory is oversubscribed 1.5 times.
* CPU is oversubscribed 16 times (cores).

Disk and memory allocation are not so much as critical on standalone setups. These will inevitably need changing as more instances are stood up as OpenStack still counts shut down instances towards the limit, even for RAM and CPU.

Changing the defaults

The defaults be changed: just edit your Nova configuration (ie: /etc/nova/nova.conf) and look for:
* disk_allocation_ratio for disk
* ram_allocation_ratio for Memory
* cpu_allocation_ratio for CPU.

Change these limits as you see fit and then just restart Nova (ie: openstack-service restart nova).

Reference: http://docs.openstack.org/developer/nova/devref/filter_scheduler.html

Advertisement

Adventures in OpenStack – On-Disk Anatomy of an Instance

vancluevertech, ,

When an instance is set up and started on OpenStack, the base image will be copied over and stored the local compute node from Glance. Here is what it looks like on disk, from a test instance of mine:

/var/lib/nova/instances/646d10cb-a5f5-4669-a155-cc78b5d59406
# ls -al 
total 65628
drwxr-xr-x. 2 nova nova       69 Feb 12 23:30 .
drwxr-xr-x. 5 nova nova       93 Feb 12 23:30 ..
-rw-rw----. 1 root root    14796 Feb 13 00:17 console.log
-rw-r--r--. 1 root root 67174400 Feb 13 00:17 disk
-rw-r--r--. 1 nova nova       79 Feb 12 23:29 disk.info
-rw-r--r--. 1 nova nova     2629 Feb 12 23:43 libvirt.xml

And checking the disk file, we see the following:

# qemu-img info disk
image: disk
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 64M
cluster_size: 65536
backing file: /var/lib/nova/instances/_base/acc45e1bf0ad2336505dcad48418ce2564b701c4
Format specific information:
    compat: 1.1
    lazy refcounts: false

Also, while the user instance disk is qcow2, the backing file is *not*:

image: /var/lib/nova/instances/_base/acc45e1bf0ad2336505dcad48418ce2564b701c4
file format: raw
virtual size: 2.0G (2147483648 bytes)
disk size: 1.1G

Remember that instance storage for Nova needs to be engineered properly: /var/lib/nova/instances could be local, but should be shared or replicated storage to ensure proper availability levels.