vancluevertech.com

I thought this would be a neat idea for a series, but we will see how it goes. The idea being of course, that I will aim to give you, the reader, a moderately quick crash course in a specific technology or product, basically giving you enough knowledge to go out in the world and and have a good working knowledge or whatever topic is covered, from which you can build on.

And with that, I present my first topic: Hiera.

About Hiera

A configuration management system is all about reducing work, and keeping the data points that a person needs to manually touch to a minimum. Puppet can go a long way in helping you out with this, but as time goes on and you set up more and more features, with modules that accept parameters, you might find that things will get a little messy. Speaking from my own experience, the place that I used to stick unique parameter data would be the node’s manifest:

node 'web01.example.com' {
    include module1
    class { 'module2':
        param1 => 'value1',
        param2 => 'value2'
    }
    class { 'module3':
        param_list => [ 'value1', 'value2', 'value3' ]
    }
}

You can imagine that after you repeat this several hundred times, try to apply different settings with say node inheritance (which, in fact, is now deprecated), it can become possibly as annoying as managing the node directly.

Worse, what if you have to store secure data, such as passwords?

Hiera solves these problems for you. It offers a central place to store all of your configuration data, allowing node manifests to be greatly simplified, and hopefully eliminating the need for inheritance completely. Better yet, multiple options exist for not only encryption, but data storage as well.

In this article we will cover the basics of using Hiera with a YAML backend, and even better, we will use hiera-eyaml to demonstrate how data can be encrypted as well.

Before we start with examples, it is important to mention the Hiera documentation – which will help you with anything missed here. Namely, we don’t cover direct or merge lookups – if you want to know more about those, check the docs. Be sure to read the README on the hiera-eyaml project page too, so you know how to properly use eYAML.

The Example

Check out the following project on my GitHub, which I will be referencing for the rest of this article.

https://github.com/vancluever/tldr_hiera

If you download this, and run it according to the instructions in the README.md, using, say, the following command:

FACTER_tldrnode=dataset1 puppet apply --hiera_config hiera.yaml \
 --modulepath modules/ -e 'include tldr_hiera'

The generated /tmp/tldr_hiera_out.md should look something like this:

In the example, you can change FACTER_tldrnode=dataset1 to FACTER_tldrnode=dataset2 to get different output:

Different output happens as well if you omit or change the value to something outside of these two, but we are going to skip that for now (this is supposed to be short and sweet, after all).

The Module

If you look in the included module, you might find that it does not look too much different than a standard puppet module:

class tldr_hiera (
  $tldr_string = 'Not found!',
  $tldr_array = [ 'Not found!' ],
  $tldr_hash = { 'tldr_key' => 'Not found!' },
  $tldr_encstring = 'Not found!'
  ) {

  # File resource where Hiera values get written to
  file { '/tmp/tldr_hiera_out.md':
    content => template('tldr_hiera/tldr_hiera_out.md.erb')
   }
}

Parameters are still declared and passed along as per normal, so where is the data coming from?

Hiera

This is where Hiera comes in. Note that we including --hiera_config hiera.yaml when we invoke puppet. Let’s take a look at an abridged version of that file:

---
:backends:
  - eyaml
  - yaml
:yaml:
  :datadir: hieradata/
:eyaml:
  :datadir: hieradata/
  :pkcs7_private_key: hieradata/keys/private_key.pkcs7.pem
  :pkcs7_public_key: hieradata/keys/public_key.pkcs7.pem
# We use a custom fact for hiearchy, just for easy demonstration
:hierarchy:
  - "%{::tldrnode}"
  - common

What’s going on here:

• We are using both the eYAML and the YAML backends for Hiera data sources (it’s important that eYAML gets included first to ensure encrypted values are queried first, if you are separating content).
• Data for both backends are stored in the hieradata/ subdirectory
• Private and public keys for eYAML are configured. These are generated with eyaml createkeys, and should be stored appropriately (the private key especially should have restricted permissions. Never distribute a private key like we have here for anything else other than demonstration purposes!)
• The data hierarchy is defined in the hierarchy section. We first reference a custom fact, tldrnode, and then fall back to a common YAML if that cannot be found.

hiera.yaml is just a configuration file for Hiera. After Hiera knows where to look and what to look for, the next step is getting the data.

A Hiera Data File

Here is the directory listing of the hieradata directory on our Vagrant instance.

-rw-r--r-- 1 vagrant vagrant 702 Mar  9 04:59 common.eyaml
-rw-r--r-- 1 vagrant vagrant 267 Mar  9 04:37 common.yaml
-rw-r--r-- 1 vagrant vagrant 283 Mar  9 03:50 dataset1.yaml
-rw-r--r-- 1 vagrant vagrant 284 Mar  9 03:50 dataset2.yaml
drwxr-xr-x 1 vagrant vagrant 136 Mar  9 03:17 keys

In the spirt of keeping things short, we will give an abridged version of one of the data files, dataset1.yaml:

---
# String
tldr_hiera::tldr_string: string_value_dataset1
# Array
tldr_hiera::tldr_array:
 - element1_dataset1
 - element2_dataset1
# Hash
tldr_hiera::tldr_hash:
    tldr_key: tldr_value_dataset1

And again, what’s going on:

• Puppet has an auto-lookup function that will look up class parameters within Hiera within its own scope. So, if you want to define settings for the tldr_hiera module, you ensure that all of your values are set up within that appropriate namespace.
• Strings, arrays, and hashes are all set up with an appropriate YAML syntax, seeing as this is the YAML backend.

Lookup Order

There is also a common.yaml file, which acts as a fallback file. This was the common entry in our hierarchy definition.

The lookup order that is defined in that section basically dictates the workflow. So in this instance:

• The custom fact tldrnode is looked up, and a match is attempted on its value (ie: dataset1.yaml)
• If that cannot be found, the static value common is used. Note that this is just a convention, you can use whatever name you want.

Values are then looked up according to class scope, as mentioned in the last section.

Encryption

Finally, there is encryption. common.eyaml looks like so:

---
tldr_hiera::tldr_encstring: >
  ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEw
  DQYJKoZIhvcNAQEBBQAEggEAp40PiGZXvJWZzEAdjife+lpE9T0P5QwUfKB6
  pIpRkMM1BKNbBze1Sak7A0Xfr9F2vemJ2BwRI2GYDgBARcgj2pMJ4zMvNVXE
  kk4AFSyBeey7BgKKrhCo8AhYGbdVhpGByJ11oAHL0cPFUKZJSjGF8OgFfwgF
  Vp1bptgM0WwRFLo9gY7WOTPXRo1IGngm0UuGoVwPc7u5NLElOpZ0ueUhAlgP
  pYjZKWMzg3Aem78oC+ORYpRVYR/P0dB706xI2RsBhk7Nf4g6kMw1pmkSWK6p
  v83+HdwoV76qi9oC/0N2QXugcV1F7A9/X4MFKVH8ai6juoJtKtG1NHU+6EuE
  4JUOZDBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBlxLHlf/BJx17oHWWb
  h4jXgCAG1uuIthUD7hd6LSeCCG1YDN2D6zckGzDqAAZg5eGhpQ==]

However, as long as you have the private key, you can go in and edit the file by using eyaml edit common.eyaml and you will get this instead:

---
tldr_hiera::tldr_encstring: >
  DEC(1)::PKCS7[string_value_decrypted]!

You can then edit the value. When you are using eyaml edit and an appropriate text editor, you will get additional instructions on how to add new values as well.

Another note: if you are using vim to edit the eYAML, you will probably need to change the default tab behaviour or it will break the YAML when adding a multi-line hash. Add something like the following to your .vimrc:

filetype plugin indent on
set tabstop=2
set shiftwidth=2
set expandtab

set expandtab is the key directive here, this is the one that turns tabs into spaces.

Use With a Puppet Master

To use this appropriately with a Puppet master, keep in mind:

• Open source Puppet expects the master Hiera config to be at /etc/puppet/hiera.yaml
• Again, be sure to lock down the keys. It is recommended that you keep them out of the data dir – although as long as the permission are tight there should be no issue, but it is probably better to store them somewhere else, such as a location like /etc/puppet/secure/keys. Do not give the private key to anyone else!