Configuration Management Series Part 1.5 of 3 – Ansible Follow-Up

Before we get started on working with our next configuration management system, Chef, I thought I’d post a follow-up to last week’s article covering Ansible. In this article, I will be touching up on some of the conclusions I drew last week, and my opinion on them now, and also some of the things I have learned about working with playbooks.


First off, some things on programmability:

Advantages of sequential execution

I really do enjoy the top-down nature of Ansible. You don’t really have to set up any sort of dependency chain for classes or resources, which can really work to your advantage.

Consider the scenario where you have a file that needs to be written to or a command that needs to be run only when a specific package is installed.

In Puppet, this needs to be written:

file { '/etc/example/example.conf':
  content => template('example_mod/example.conf.erb'),
  require => Package['example']

package { 'example':
  ensure => installed

This is not so bad, and in fact, the declarative nature of Puppet makes it pretty easy to manage this stuff.

But what if you try to require a different resource, like a class? Or multiple packages?

Your child resource could end up with dependencies like this, which would then need to be repeated:

file { '/etc/example/example.conf':
  content => template('example_mod/example.conf.erb'),
  require => [ Package['example1'], Package['example2'], Package['example3'] ]

You may also try to chain off a class or service, however when I have attempted these kinds of things, I have just ran into dependency loops.

Then you get so far down the rabbit hole you need a grapher. Consider the following article regarding dependency issues. When I need a whole article to try to figure out how to properly structure dependencies, I honestly tune out.

In Ansible, you don’t really need all of this. You just make sure your package tasks are set up before the tasks that depend on them:

- name: Install packages
  yum: name='{{ item }}' state=present
    - example1
    - example2
    - example3

-name: Put the file where it needs to go now
  template: src=example.conf.j2 dest=/etc/example/example.conf mode=0644

In this instance, if the first task fails, the other ones will not execute.

Another easy check that you can stick at the start of a playbook is an OS check or what not, such as:

- name: check for supported OS
  fail: msg=Unsupported distribution (Ubuntu 14.04 supported only)
  when: ansible_distribution_release != 'trusty'

This will fail if the playbook is not running against anything but Ubuntu 14.04, allowing you to have the rest of the playbook run without worrying about it being run against an unsupported OS.

Frustrations with bad error messages

A few things I have noticed so far with Ansible is the Python-ish error messages you get out of it, sometimes, unfortunately, out of context. I have gotten error messages with stack traces in them (ie: when processing Jinja2 stuff and also when a module fails thru some sort of untrapped Python method). I actually don’t mind this, because at least I know where to start looking.

It’s error messages like this that drive me up the wall.

msg: Error in creating instance: 'str' object has no attribute 'get' 
FATAL: all hosts have already failed -- aborting

This happened to me while trying to mess with the nova_compute module. Where exactly am I supposed to start looking for something wrong here? I have even attempted to use pdb to try and chase it down, but I had to give up because I had other stuff to do. I may try again, but I really am on my own here, barring outside help. This error message does nothing to tell me where my problem is or how to correct it.

It almost would have been better if the error was left un-handled, so I could see the Python stack trace.


And also, some thoughts and bits of knowledge on playbooks.

The playbook as a site configuration

If you looked at the previous article, I had a bit of trouble trying to figure out what a playbook really was in relation to how Puppet is laid out. At first glance, it seemed to me that a playbook was close to what a module was, but that’s not really the case at all.

Playbooks are modular, sure, but they are more like individual site configurations more than anything else. Think about them as individual Puppet or Chef site installs.

According to the Ansible playbook best practices guide there is a very specific layout. This is reproduced below:

production                # inventory file for production servers
stage                     # inventory file for stage environment

   group1                 # here we assign variables to particular groups
   group2                 # ""
   hostname1              # if systems need specific variables, put them here
   hostname2              # ""

library/                  # if any custom modules, put them here (optional)
filter_plugins/           # if any custom filter plugins, put them here (optional)

site.yml                  # master playbook
webservers.yml            # playbook for webserver tier
dbservers.yml             # playbook for dbserver tier

    common/               # this hierarchy represents a "role"
        tasks/            #
            main.yml      #  <-- tasks file can include smaller files if warranted
        handlers/         #
            main.yml      #  <-- handlers file
        templates/        #  <-- files for use with the template resource
            ntp.conf.j2   #  <------- templates end in .j2
        files/            #
            bar.txt       #  <-- files for use with the copy resource
          #  <-- script files for use with the script resource
        vars/             #
            main.yml      #  <-- variables associated with this role
        defaults/         #
            main.yml      #  <-- default lower priority variables for this role
        meta/             #
            main.yml      #  <-- role dependencies

    webtier/              # same kind of structure as "common" was above, done for the webtier role
    monitoring/           # ""
    fooapp/               # ""

Note how everything is accounted for here. For someone like me who comes from Puppet, there are some easy analogs here that have helped me:

  • site.yml and the other playbooks are pretty similar to Puppet site manifest files. From them entire configurations can be laid out without a need for anything else, or different roles can be referenced.
  • group_vars and host_vars behave similar to Hiera for storing variable overrides.
  • Finally, the roles structure is the closest thing that Ansible has to Puppet modules.

Honourable mentions to the library and filter_plugins directories. Ansible modules are actually closer to the core than Puppet’s, and normally do not need to be written, but you may find yourself doing so if your roles require functionality that cannot be reproduced in any existing module and you want that recycled across modules. Also, jinja2 filter plugins can be written (we give a small example below).


As mentioned, roles are probably the closest thing that Ansible has to a puppet module. These are recyclable “playbooks” that are self-contained and are then included within a proper playbook via the following means:

- hosts: webservers
  sudo: yes
    - common
    - webtier
    - monitoring

This would ensure that webservers got any tasks related to the common, webtier, and monitoring roles, running under sudo. Imagine that this means that some basic common tasks are done (ie: maybe push out a set of SSH keys and make sure that DNS servers are set correctly), set up a webserver, and install a monitoring agent.

Similar to Puppet or Chef, there is a file structure to roles that allows you to split out various parts of the roles to different directories:

  • tasks for role tasks, the main workflow of a role and Ansible itself.
  • handlers for server actions and what not (ie: restart webserver)
  • templates to store your jinja2 templates for template module tasks
  • files for your static content
  • defaults to define default variables
  • vars to define local variables (NOTE: I have decided to stay away from these and I recommend you do too. Use defaults!)

main.yml is the universal entry point for all of these. from here, you can include all you wish. Example:

- name: check for supported OS
  fail: msg=Unsupported distribution (Ubuntu 14.04 supported only)
  when: ansible_distribution_release != 'trusty'

- include: install_pkgs.yml

This would be a tasks/main.yml file and demonstrates how you can mix includes and actual tasks in the role.

Group and host variables

This is where I would recommend you store variables if you need them. The obvious place would be group first, then host if you need specific host variables.

For example, this would apply foo=bar to all webservers for all plays:

# group_vars/webservers.yml
foo: bar

You can also include them in the playbook directly, ie: like so:

- hosts: webservers
  sudo: yes
    - common
    - webtier
    - monitoring
    foo: bar

Secure data

ansible-vault is a very basic encryption system that encrypts an entire file (does not need to be YAML) off the command line. It hooks into ansible and ansible-playbook via the --ask-vault-pass switch that will take any encrypted files it encounters and decrypt them.

It is recommended that anything with sensitive data be encrypted. The one main difference here is that as opposed to Hiera-eyaml, for example, the entire file is encrypted, so you may wish to separate encrypted and unencrypted data through separate files, if it can be helped. This ensures non-sensitive data can still be published to source control and be reviewed.

Unfortunately, it does not look like there is a certificate or private/public key system in use here, so remembering the password is very important, as is picking a secure one. Passwords can also be stored in a file (make sure it’s set to mode 0600), and passed using the --vault-password-file option under all Ansible utilities, assumedly.

Filter Plugins for jinja2

And finally, I wrote that Jinja2 plugin I was hitting the wall with. ūüėČ It’s very simple and can be found below. This would go into the filter_plugins directory.

# Add string split filter functionality
def split(value, sep, max):
    return value.split(sep, max)

class FilterModule(object):
    def filters(self):
        return {
            'split': split

This now gives you the split filter that can be invoked like so:

{{ (example|split('.'))[0] }}

Assuming, this would give example.

“Final” Thoughts

To be honest, I think that Ansible is more of an orchestration tool rather than a configuration management system. The agentless push nature of it, combined with its top-down workflow, roll-your-own inventories, and loosely enforced site conventions give it a chaotic feel that, even after giving it a week, still does not feel like configuration management to me.

This is not to say it’s a bad tool in any right, in fact, quite the opposite. Ansible is a great addition to my toolbox, and I am looking forward to applying it in everyday engineering. The above points that may be “weaknesses” when considering configuration management can equally be strengths when you need to get something done quickly and with as little of a footprint as possible.

Further to that, Ansible can be combined with something like Chef or Puppet, with each tool playing on its strengths. Ansible can be used to bootstrap either tool, with Chef or Puppet doing things like managing SSH keys (such as one for Ansible), enforcing configuration, etc. Ansible can be used in place of something like knife or puppet kick to force updates as well.

Some tools need work: Vault, for example, could do better to be more than just a wrapper for encryption, and set up some sort of keypair structure to reduce the amount of typing you have to do. And error messaging could do with a bit of work so that engineers are not scratching their heads trying to figure out what variable caused Python to barf when errors are trapped.

All in all though, I really like Ansible. There are some other projects that I want to try it with, namely working with the nova_compute and vsphere_guest modules to help automate orchestration even further.

I know that I mentioned that Chef was going to be this Sunday. Unfortunately (or maybe fortunately) I spent a bit more time with Ansible than I thought I was going to, and the research has not been done to properly get a Chef article up yet – ie: I have not yet started. ūüėõ in any case, that will be happening next – so stay tuned for the it next weekend!

Configuration Management Series Part 1 of 3: Ansible

This is the first of a three-part series that I am doing regarding reviewing 3 major configuration management tools: Ansible, Chef, and Puppet.

You may have seen that I have written here about Puppet before – indeed, it is the configuration management tool that I have the most experience with and probably the one that I will be sticking with personally (famous last words :p). However, I think that it’s always in an engineer’s best interest to be able to understand, support, and thrive with technologies other than the ones that he or she may personally favour as part of their own toolkit – that is what teamwork is built on, after all. So with that, take the opinions in these next few articles with a grain of Salt (which, incidentally, is the name of another tool that I will not be reviewing at this point in time, heh). I hope that you can value my opinion while still understanding that is it subjective, and your experience will vary. Honestly, I think that all 3 of these tools have value, and in the hands of competent engineers, you would be hard-pressed to find shortcomings in any of them.

One other note: I have removed the ratings from this article and the future articles will not have them either. I thought it was a good idea at first, but as time went on, and my experience with the tools increase, they seem more and more pretentious and quite misinformed.

Now that I have that disclaimer out of the way, let’s get started shall we?

As mentioned in the first part of this series, we are covering Ansible.

About Ansible

A long time ago in a technology scene not so far away (but definitely quite different than the landscape today, I used to work at NetNation (you may now know them as Hostway Canada). On our management station we had a very simple shell script that performed a very valuable task: batch administration of our shared hosting server farm over SSH.

It has now been nearly 10 years since I left Hostway, yet some concepts never die, they just get better. Ridiculously better. Magnitudes of evolution better.

Maybe about 5 or so years ago now I actually wrote a better version of this script in Python – doing some research I fond a very nifty SSH library called Paramiko ( and added a few things to make the experience better: namely, the ability to upload files and run shell scripts on the remote machine.

Admittedly, I haven’t really looked into Ansible recently, but imagine my surprise to find out that someone has basically taken that concept and made it leaps and bounds more awesome.

Ansible is basically this concept on – even though I loathe to use this analogy – steroids. Its main command transport is SSH, and does not require an agent on managed nodes to run. It allows for complex configuration management thru a YAML-based pseudo-DSL, one-off command execution, file upload, and also remote execution of complete shell scripts. A very powerful swiss-army knife for both one-off and ongoing configuration management.


Installation of Ansible is probably the most minimal out of the three that we will be covering. The three major methods are either: 1) git repo (, 2) package, or 3) pip.

With either method you may notice something that you may not be used to if you have used a more traditional configuration management tool such as Chef or Puppet: no management server.

This is almost true. Ansible nodes normally do not use a check-in method to get their configuration data, rather it is pushed out to various nodes, so there is no need for a server with this capability. Nonetheless, there is a central configuration structure that can be found in a directory such as /etc/ansible and needs to be edited accordingly. More on that in brief later.

You can find the installation guide at

Windows Support

This was going to detail the support of both major operating systems, but I felt the need to mention Linux support was redundant, as all 3 were obviously built to manage Linux first.

Ansible was written with Linux and other open source server systems in mind. With that said, they have had made a very recent push to supporting Windows, and it is looking pretty promising. Going with their agentless model again, they have decided to use WinRM instead. Some of the Windows specific features look amazing as well and definitely on par with something like Puppet.

Check out the Windows section at The list of Windows-specific modules can be found at


Since Ansible does not rely on agents, there is no real host discovery in the sense that hosts would be checking in with your management sever. From what I’ve seen, there is not even any real language to refer to the “master” server.

Inventory is managed in /etc/ansible/hosts in a pretty simple configuration file that is hard to mess up. Here is my sample hosts file:

# Our lab webserver

# Our lab database server

Hosts can be referred to by IP address, hostname, and can be further abstracted with aliases. They can also be grouped together using primitive expressions if your hosts all follow a certain convention. Various plugins exist to gather inventory dynamically too – see

The only thing that I really see missing here is some native sort of host registration and grouping. This may not be something that, by design, would be Ansible’s domain though, considering it’s push-based architecture. Also, depending on the environment you are using it in, this may be something that you can handle with dynamic inventory.

Best practice for storing host parameters (variables) actually follows very closely with how Puppet handles things with Hiera. See here: and you will see that data is generally stored in YAML files in a very similar fashion. There is also Ansible Vault: which allows for the storage of encrypted data.

There is also an enterprise console, of course. Check out Ansible tower here:

Execution Model

Several options exist for you to use with Ansible’s powerful command-line structure. The ansible command line tool can be used to execute one-off commands, module runs, and even push shell scripts and files over to remote nodes. And since it’s all intended to be run over SSH via standard user accounts, existing access control schemes can be extended to configuration management as well.

Even though I don’t believe it’s a strength of the tool, there is a pull option available as well via ansible-pull


My opinion here might be a bit polarizing or just completely out there, but I have not had the best experience writing for Ansible so far, although mind you I have only been doing it for an evening, and obviously still have a lot to learn.

Modules are actually bit more of an advanced topic, with Playbooks taking the place of general modules like you would be used to in Puppet, or cookbooks in Chef. These are collections of YAML files that dictate orchestration and management of configuration files. Playbook documentation can be found at

Templating for Playbooks is done using jinja2 – a powerful template language written for Python. You can find the template reference at Ansible also extends this a bit and its worth reading to find out how.

Unfortunately, this is where I got a little hung up on a learning curve. My impressions were:

  • The YAML structure for Playbooks was open to a bit of interpretation, and the ambiguity of it was throwing me off at points. I found that I was trying to figure out the best way to nest, if I should be at all, how much code to recycle, etc. Ironically, there seems to be several ways to do it, which kind of flies in the face of one of the core philosophies of what Ansible is written in, Python (see
  • I hit a straight up wall when I found out that jinja2 does not have a split filter – you need to write one of your own. An easy task in Python, but annoying nonetheless.

Lastly, when it comes to config management, I prefer to not have to write any of my own modules if I can help it. Unfortunately, briefly looking at Ansible Galaxy left me unfulfilled. Standard modules that I was used to having in Puppet, such as mainline designed modules like Apache and MySQL, were not up to par or were just flat out missing. Other highly-rated modules that purported to do one thing did others (ie: one Apache module was trying to manage SSH keys as well). Several user accounts (even one named “ihateansible”) exist with zero contributions. I hate to say it but my at least first impression is that there are some quality control issues here.


  • Strengths:
    • Agentless push-style management
    • Powerful inventory management
  • Weaknesses
    • Core programmability features will take some getting used to if you are coming from Chef/Puppet.
    • Ansible Galaxy possibly has some quality control issues.

Ansible is great, no doubt about that. It has taken a tried and reliable concept and turned it into something so much more than I could have imagined.

Installation is dead simple, and hence it excels in environments that need to stand up some sort of configuration management fast and with as little impact to existing infrastructure as possible. Its simple yet powerful approach to host management compliments this process even more.

Windows support is now available and looks very promising, and personally I am impressed on how fast they have been able to stand this side of it up.

Programmability could be worked on a bit, but again this could be my inexperience talking. At the very least, I would prefer to see more mainline created and managed Playbooks, which would then raise the overall quality of the modules seen on Galaxy.

Stay tuned for part 2: Chef, either later this week or on next Sunday. There will probably be another article or two regarding Ansible as well before we wrap up!

Adventures in OpenStack – Nova Limits

This morning I was setting up some test instances on my home OpenStack deployment and ran into some issues. The first instance launched okay, but the second failed, telling me that no suitable host was found.

I have not really oversubscribed, or so I thought. Turns out I was oversubscribed on both RAM and Disk.

OpenStack has defaults surrounding this:
* Disk is not oversubscribed by default. There is a 1-1 subscription per host. In a production setup, this should never be changed – an instance launching on a host that could run out of space and take down all the instances on that node is definitely a bad thing.
* Memory is oversubscribed 1.5 times.
* CPU is oversubscribed 16 times (cores).

Disk and memory allocation are not so much as critical on standalone setups. These will inevitably need changing as more instances are stood up as OpenStack still counts shut down instances towards the limit, even for RAM and CPU.

Changing the defaults

The defaults be changed: just edit your Nova configuration (ie: /etc/nova/nova.conf) and look for:
* disk_allocation_ratio for disk
* ram_allocation_ratio for Memory
* cpu_allocation_ratio for CPU.

Change these limits as you see fit and then just restart Nova (ie: openstack-service restart nova).


TL;DR Tutorials: Hiera

I thought this would be a neat idea for a series, but we will see how it goes. The idea being of course, that I will aim to give you, the reader, a moderately quick crash course in a specific technology or product, basically giving you enough knowledge to go out in the world and and have a good working knowledge or whatever topic is covered, from which you can build on.

And with that, I present my first topic: Hiera.

About Hiera

A configuration management system is all about reducing work, and keeping the data points that a person needs to manually touch to a minimum.¬†Puppet can go a long way in helping you out with this, but as time goes on and you set up more and more features, with modules that accept parameters, you might find that things will get a little messy. Speaking from my own experience,¬†the place that I used to¬†stick unique parameter data would be the node’s manifest:

node '' {
    include module1
    class { 'module2':
        param1 => 'value1',
        param2 => 'value2'
    class { 'module3':
        param_list => [ 'value1', 'value2', 'value3' ]

You can imagine that after you repeat this several hundred times, try to apply different settings with say node inheritance (which, in fact, is now deprecated), it can become possibly as annoying as managing the node directly.

Worse, what if you have to store secure data, such as passwords?

Hiera solves these problems for you. It offers a central place to store all of your configuration data, allowing node manifests to be greatly simplified, and hopefully eliminating the need for inheritance completely. Better yet, multiple options exist for not only encryption, but data storage as well.

In this article we will cover the basics of using Hiera with a YAML backend, and even better, we will use hiera-eyaml to demonstrate how data can be encrypted as well.

Before we start with examples, it is important to mention the Hiera documentation – which will help you with anything missed here. Namely, we don’t cover direct or merge lookups – if you want to know more about those, check the docs. Be sure to read the README on the hiera-eyaml project page too, so you know how to properly use eYAML.

The Example

Check out the following project on my GitHub, which I will be referencing for the rest of this article.

If you download this, and run it according to the instructions in the, using, say, the following command:

FACTER_tldrnode=dataset1 puppet apply --hiera_config hiera.yaml \
 --modulepath modules/ -e 'include tldr_hiera'

The generated /tmp/ should look something like this:


In the example, you can change FACTER_tldrnode=dataset1 to FACTER_tldrnode=dataset2 to get different output:


Different output happens as well if you omit or change the value to something outside of these two, but we are going to skip that for now (this is supposed to be short and sweet, after all).

The Module

If you look in the included module, you might find that it does not look too much different than a standard puppet module:

class tldr_hiera (
  $tldr_string = 'Not found!',
  $tldr_array = [ 'Not found!' ],
  $tldr_hash = { 'tldr_key' => 'Not found!' },
  $tldr_encstring = 'Not found!'
  ) {

  # File resource where Hiera values get written to
  file { '/tmp/':
    content => template('tldr_hiera/')

Parameters are still declared and passed along as per normal, so where is the data coming from?


This is where Hiera comes in. Note that we including --hiera_config hiera.yaml when we invoke puppet. Let’s take a look at an abridged version of that file:

  - eyaml
  - yaml
  :datadir: hieradata/
  :datadir: hieradata/
  :pkcs7_private_key: hieradata/keys/private_key.pkcs7.pem
  :pkcs7_public_key: hieradata/keys/public_key.pkcs7.pem
# We use a custom fact for hiearchy, just for easy demonstration
  - "%{::tldrnode}"
  - common

What’s going on here:

  • We are using both the eYAML and the YAML backends for Hiera data sources (it’s important that eYAML gets included first to ensure encrypted values are queried first, if you are separating content).
  • Data for both backends are stored in the hieradata/ subdirectory
  • Private and public keys for eYAML are configured. These are generated with eyaml createkeys, and should be stored appropriately (the private key especially should have restricted permissions. Never distribute a private key like we have here for anything else other than demonstration purposes!)
  • The data hierarchy is defined in the hierarchy section. We first reference a custom fact, tldrnode, and then fall back to a common YAML if that cannot be found.

hiera.yaml is just a configuration file for Hiera. After Hiera knows where to look and what to look for, the next step is getting the data.

A Hiera Data File

Here is the directory listing of the hieradata directory on our Vagrant instance.

-rw-r--r-- 1 vagrant vagrant 702 Mar  9 04:59 common.eyaml
-rw-r--r-- 1 vagrant vagrant 267 Mar  9 04:37 common.yaml
-rw-r--r-- 1 vagrant vagrant 283 Mar  9 03:50 dataset1.yaml
-rw-r--r-- 1 vagrant vagrant 284 Mar  9 03:50 dataset2.yaml
drwxr-xr-x 1 vagrant vagrant 136 Mar  9 03:17 keys

In the spirt of keeping things short, we will give an abridged version of one of the data files, dataset1.yaml:

# String
tldr_hiera::tldr_string: string_value_dataset1
# Array
 - element1_dataset1
 - element2_dataset1
# Hash
    tldr_key: tldr_value_dataset1

And again, what’s going on:

  • Puppet has an auto-lookup function that will look up class parameters within Hiera within its own scope. So, if you want to define settings for the tldr_hiera module, you ensure that all of your values are set up within that¬†appropriate namespace.
  • Strings, arrays, and hashes are all set up with an appropriate YAML syntax, seeing as this is the YAML backend.

Lookup Order

There is also a common.yaml file, which acts as a fallback file. This was the common entry in our hierarchy definition.

The lookup order that is defined in that section basically dictates the workflow. So in this instance:

  • The custom fact tldrnode is looked up, and a match is attempted on its value (ie: dataset1.yaml)
  • If that cannot be found, the static value common is used. Note that this is just a convention, you can use whatever name you want.

Values are then looked up according to class scope, as mentioned in the last section.


Finally, there is encryption. common.eyaml looks like so:

tldr_hiera::tldr_encstring: >

However, as long as you have the private key, you can go in and edit the file by using eyaml edit common.eyaml and you will get this instead:

tldr_hiera::tldr_encstring: >

You can then edit the value. When you are using eyaml edit and an appropriate text editor, you will get additional instructions on how to add new values as well.

Another note: if you are using vim to edit the eYAML, you will probably need to change the default tab behaviour or it will break the YAML when adding a multi-line hash. Add something like the following to your .vimrc:

filetype plugin indent on
set tabstop=2
set shiftwidth=2
set expandtab

set expandtab is the key directive here, this is the one that turns tabs into spaces.

Use With a Puppet Master

To use this appropriately with a Puppet master, keep in mind:

  • Open source Puppet expects the master Hiera config to be at /etc/puppet/hiera.yaml
  • Again, be sure to lock down the keys. It is recommended that you keep them out of the data dir – although as long as the permission are tight there should be no issue, but it is probably better to store them somewhere else, such as a location like /etc/puppet/secure/keys. Do not give the private key to anyone else!

Adventures in OpenStack – Neutron Setup Errata and Multiple External Networks

So, a bit of a confession. The article¬†Adventures in OpenStack ‚Äď Networking¬†has a¬†an issue in how to set up external networks. The section that mentioned what to enter in the [ovs] section actually does nothing.

Why? Because by default, the Layer 3 (L3) agent only allows one external network, hardcoded in the config and defaulted to br-ex:

# Name of bridge used for external network traffic. This should be set to
# empty value for the linux bridge. when this parameter is set, each L3 agent
# can be associated with no more than one external network.
# external_network_bridge = br-ex

As such, the first network flagged as “external”, as long as the name is available as a valid network name in the ML2 plugin config, will function and be mapped to the external bridge listed above.

This came up for me when I tried to add a second external network. Before a few versions of OpenStack ago, this needed to be done by running a second Layer 3 agent. But now with the ML2 plugin it is possible to run them out of the same agent.

The right way to do this is:

  • Make sure that have all the external networks listed in the ML2 config. If this is the flat network type, it would look like so:
flat_networks = external,external2
  • Then, head over to l3_agent.ini and ensure that you have cleared the following as¬†blank: 1) the external gateway ID, and the external network bridge name:
gateway_external_network_id =
external_network_bridge =

I make sure these are explicitly defined (uncommented and set correctly), to avoid issues.

  • Finally, add the OVS bits to the¬†right config which should be in¬†/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini:
network_vlan_ranges = external,external2
bridge_mappings = external:br-ex,external2:br-ex2

Even though network_vlan_ranges might seem redundant or misleading for flat networks, the definition is still required.

You are now ready to restart Neutron. There is a shortcut for this, by the way:

openstack-service restart neutron

One Last Thing

If you are having issues with your existing external networks after performing the above, check for this in your Neutron logs:

openvswitch-agent.log:2015-02-22 18:27:39.369 22377 WARNING neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2fb3e41a-f068-4737-97f8-580af5ddad27 None] Device 598f6434-06ff-4280-af5c-6ce7c312ca5b not defined on plugin
server.log:2015-02-22 18:27:37.393 22455 WARNING neutron.plugins.ml2.rpc [req-2fb3e41a-f068-4737-97f8-580af5ddad27 None] Device 598f6434-06ff-4280-af5c-6ce7c312ca5b requested by agent ovs-agent-beefhost.vcts.local on network d8f7930b-d4e9-422d-b789-7c2866ab81e8 not bound, vif_type: binding_failed

If you see this, you may need to re-create your external networks. There may be another way to fix this, seeing as it may have to do with a bad external network/bridge association, but if you are in a (small enough) lab, it should be trivial to just tear down and rebuild the network and router gateway bindings (only the external networks, subnets, and gateway associations need to be fixed).

Adventures in OpenStack – On-Disk Anatomy of an Instance

When an instance is set up and started on OpenStack, the base image will be copied over and stored the local compute node from Glance. Here is what it looks like on disk, from a test instance of mine:

# ls -al 
total 65628
drwxr-xr-x. 2 nova nova       69 Feb 12 23:30 .
drwxr-xr-x. 5 nova nova       93 Feb 12 23:30 ..
-rw-rw----. 1 root root    14796 Feb 13 00:17 console.log
-rw-r--r--. 1 root root 67174400 Feb 13 00:17 disk
-rw-r--r--. 1 nova nova       79 Feb 12 23:29
-rw-r--r--. 1 nova nova     2629 Feb 12 23:43 libvirt.xml

And checking the disk file, we see the following:

# qemu-img info disk
image: disk
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 64M
cluster_size: 65536
backing file: /var/lib/nova/instances/_base/acc45e1bf0ad2336505dcad48418ce2564b701c4
Format specific information:
    compat: 1.1
    lazy refcounts: false

Also, while the user instance disk is qcow2, the backing file is *not*:

image: /var/lib/nova/instances/_base/acc45e1bf0ad2336505dcad48418ce2564b701c4
file format: raw
virtual size: 2.0G (2147483648 bytes)
disk size: 1.1G

Remember that instance storage for Nova needs to be engineered properly: /var/lib/nova/instances could be local, but should be shared or replicated storage to ensure proper availability levels.

From the “Holy Crap!” Department: Ruby Backup Gem!

TIL there is actually a ruby gem devoted to streamlining backups!

See for more details.

I haven’t explored it at all yet other than looking at the webpage, but it looks extremely promising. There are tons of features!

Of course, automation is what makes this even cooler – and there is a puppet module for that – see evenup/backup at Puppet Forge.

I am really looking forward to trying this out!

packstack – Cinder and Nagios Options, and Advanced Operation

After getting a grasp on all of the concepts that I have discussed regarding OpenStack, I have decided to give my dev server a re-deploy, namely to set up LVM for proper use with Cinder.

I noticed a few things that should probably be noted when using packstack to configure an OpenStack host.

First off, packstack is not necessarily just for setting up lab environments. By running packstack –help,¬†the program will output a plethora of options that can be used for controlling how packstack runs. This can be used to limit deployment to just a few options, so that, for example, it only deploys a compute node, or a storage node, etc. It also allows for answer file use. With fine-tuning of the options, packstack can be used to configure entire clouds, or at the very least more sophisticated multi-server labs.

Another thing to note is that there are several options that are set by default that may not be desired. For example, ultimately my second run at at an OpenStack all-in-one is looking like this:

packstack --allinone --provision-all-in-one-ovs-bridge=n --cinder-volumes-create=n --nagios-install=n

This excludes the following:

  • The Open vSwitch bridge setup. This is because I want to re-configure the Open vSwitch bridges again as per this article.
  • Creation of the test Cinder volume group – I already have one created this time around that I want to use with Cinder. This is named¬†cinder-volumes as per the default volume group that Cinder looks for, and is also the volume group that packstack will create with raw file on the file system, which is not suitable for production use. If you have this volume group set up already and do not select this flag, packstack will ultimately fail.
  • Disabling the installation of a full Nagios monitoring suite on the host, as I plan to set up monitoring later – and not with Nagios, mind you!

Remember that you can check out a brief rundown on how to install and use packstack at the quickstart page on RDO.

iOS Syslog Stuff!

The iPhone and iOS in general have never been really well known for their ability to get too much under the hood, unless you are jailbreaking. That does not necessarily mean that there are not, albeit limited, options.

Tonight I was poking around to see if I could get at the system log on my iPhone. This article at the iPhone Wiki gives many options, but after doing a bit more looking, I found a bit more of a direct method in the macports tree (this is of course, OS X only).

The MobileDevice tool is available off macports and can be installed as simple as running

sudo port install MobileDevice

After this is done, you have access to the MobileDevice API off the command line with the mdf tool.

The current syslog can be gathered by running

mdf -d 0 syslog

This will gather the current syslog buffer and also start tailing the device. You will need to make sure that the device is authorized on the computer first. mdf list will list your connected devices (-d 0 is the device specification).

There are a few other neat things that you can do with the mdf script and the MobileDevice modules themselves, check out the markdown for more info.

Adventures in OpenStack – Launching an Instance, and Attaching a Floating IP

Now that I have all the basic building blocks set up, I can proceed to setting up an instance.

Incidentally, packstack set most of this up already, but I actually ripped out most of the networking stuff created by packstack, and also removed the demo tenant and set all that stuff up from scratch. I’m glad I did, as I have a lot better understanding as to what is going on with that part now. Understanding of Cinder and Swift is not necessarily nearly as important on a single server cloud¬†as far as I’m concerned, and can be looked into later.

In any case, time to get started.

Make sure that the correct tenant is selected before doing any of this – admin is not the correct one! I may address this in a later article, but you either want to make sure that admin has correct permission for the project, or log in as the project’s admin user.

Adding a Key Pair

Instances cannot be logged into over SSH if there is no key pair.

Go to Projects -> Compute -> Access and Security. Under Key Pairs, a key pair can either be created, or imported. Importing brings up this window which you can put in your own OpenSSH public key:

OpenStack - Import Key Pair

Creating a key pair will create a key pair, and then automatically download it.

Adding SSH Access Rules

Under Projects -> Compute -> Access and Security -> Security Groups, a firewall rule needs to be added for SSH. Click Add Rule after selecting the default security group and enter the following:

OpenStack - Security Group Rule

This allows SSH globally. Rules can also be made to link security groups, allowing you to create more sophisticated access schemes across different networks.

Instance Creation

Time to create the instance.

Head to Project -> Compute -> Instances and click Launch Instance:

OpenStack - Create Instance 1

OpenStack - Create Instance 2

OpenStack - Create Instance 3

Note the options. I am using preconfigured flavours with the Debian Jessie image I uploaded. I am also using a test key pair, and the network I created.

Post-Creation gives you a place to add any post-installation scripts and what not, and Advanced Options allows you to set up custom partitioning if you want. I don’t need either of those, so I did not include them or modify them.

After clicking Launch, and waiting a bit of time, the instance should be visible:

OpenStack - Running Instance

I can now proceed to attaching a floating IP, so that the instance is publicly accessible.

Adding a Floating IP Address

Click the drop down box at the far right of the instance listing and select Associate Floating IP:

OpenStack - Floating IP 1

Since I do not have any IPs allocated, I have to request one, by clicking the + (plus) button:

OpenStack - Floating IP 2

I can then use the allocation given to me:

OpenStack - Floating IP 3

And now the IP address will be visible in the instance, after clicking Associate of course:

OpenStack - Floating IP 4This instance is now ready to log into! By connecting using the SSH key I was given, I can connect to over SSH and start working on the instance.

The Debian image’s default login at this time is debian. Most instances will NOT let you log in as root, so consult the image documentation for any specific default user that is created. Sometimes, logging in as root will give you the correct user to log in as and then disconnect you (such is the case with the Debian image).