Using X11 Forwarding Through sudo

X11 forwarding comes in handy from time to time. For example, say you want to use virt-manager to work with KVM VMs on your lab machine but want to do it from your Mac (ahem).

Yeah, if you didn’t gather it already, by “you”, I mean “me”. 😉

The one main issue with this very specific above scenario is that virt-manager will more than likely require some sort of root-level privileges, and neither the ~/.Xauthority file, nor the DISPLAY or XAUTHORITY environment variables survive sudo escalation.

The manual fix is pretty easy though. Before escalation, run xauth list to get the session data.

The output looks like:

$ xauth list
localhost.localdomain/unix:99  MIT-MAGIC-COOKIE-1  aabbccddeeffgghh00112233445566

Take the second line (which is the session data). Then, after getting root (sudo su - works great), run xauth add with the session data:

xauth add localhost.localdomain/unix:99  MIT-MAGIC-COOKIE-1  aabbccddeeffgghh00112233445566

This will create the ~/.Xauthority file and aforementioned environment variables.

We are now able to run X11 apps as root!

Shout out to Backdrift for the source.


nginx: More FastCGI Stuff

nginx’s a pretty powerful server, no doubt. But depending on your situation, a simple caching solution as discussed in my FastCGI Caching Basics article might not be enough, or may outright break your setup. You may also want nginx to co-exist with any existing application cache as well.

In this article, I’ll explain a few techniques that can be used to help alter nginx’s caching and proxy behaviour to get the configuration you need. 

Advanced use of fastcgi_param

fastcgi_param is an important part of setting up the environment for proper operation. It can also be used to manipulate the HTTP request as it is being passed to the backend, performing an ad-hoc mangling of the request.

Example: clearing the Accept-Encoding request header

fastcgi_param HTTP_ACCEPT_ENCODING "";

This is useful when an application is configured to send a response deflated, ie: with gzip compression. This is something that should be handled in nginx, not the backend, and caching a gzipped response with a key that may cause it to be sent to a client that does not support compression will obviously cause problems. Of course, this should probably be turned off in the application, but there may be situations where that is not possible. This effectively clears whatever was set here in the first place, possibly by something earlier in the config. 

Ignoring and Excluding Headers

Via fastcgi_ignore_headers and fastcgi_hide_header, one can manipulate the headers passed to the client and even control nginx’s behaviour itself. 

Example: Ignoring Cache-Control

Imagine a scenario where the application you are using has a caching feature that can be used to compliment nginx’s own cache by speeding up updates, further reducing load. However, this feature sends Cache-Control and Expires headers. nginx honours these headers, which may manipulate nginx’s cache in a way you do not want. 

This can be corrected via:

fastcgi_ignore_headers Cache-Control Expires;
fastcgi_hide_header Cache-Control;
fastcgi_hide_header Expires;

This does the following:

  • The fastcgi_ignore_headers line will ensure that nginx will not honor any data in the Cache-Control and Expires header. Hence, if caching is enabled, nginx will cache the page even if there is a Cache-Control: no-cache header. Additionally, Expires from the response is ignored as well.
  • However, with this on, the response will still be passed to the client with the bad headers. That is probably not what is desired, so fastcgi_hide_header is used to ensure that those headers are not passed to the client as well.

Gotcha #1: Ensuring an Admin Area is not Cached

If the above techniques are employed to further optimize the cache, then it is probably a good idea to ensure that the admin area is not cached, explicitly, if Cache-Control was relied on to do that job before.

Below is an example on how to structure that in a config file, employing all of the other items that I have discussed already. Note that static entry of the SCRIPT_FILENAME and SCRIPT_NAME is probably necessary for the admin area, as the dynamic location block for general PHP files will have the caching credentials in it.

# fastcgi for admin area (no cache)
location ~ ^/admin.* {
  include fastcgi_params;

  # Clear the Accept-Encoding header to ensure response is not zipped
  fastcgi_param HTTP_ACCEPT_ENCODING "";

  fastcgi_pass unix:/var/run/php5-fpm.sock;
  fastcgi_index index.php;
  fastcgi_param SCRIPT_FILENAME $document_root/index.php;
  fastcgi_param SCRIPT_NAME /index.php;

location / {
  try_files $uri $uri/ /index.php$is_args$args;

# Standard fastcgi (caching)
location ~ \.php$ {
  include fastcgi_params;

  # Clear the Accept-Encoding header to ensure response is not zipped
  fastcgi_param HTTP_ACCEPT_ENCODING "";

  fastcgi_ignore_headers Cache-Control Expires;
  fastcgi_hide_header Cache-Control;
  fastcgi_hide_header Expires;

  fastcgi_pass unix:/var/run/php5-fpm.sock;
  fastcgi_index index.php;
  fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;

  fastcgi_cache fcgi_one;
  fastcgi_cache_valid 200 1m;
  fastcgi_cache_key $request_method-$http_host$fastcgi_script_name$request_uri;
  fastcgi_cache_lock on;
  fastcgi_cache_use_stale error timeout invalid_header updating http_500 http_503;

This setup will route all requests for /admin to FastCGI with a statically defined page and no cache options. This will ensure that no caching and header manipulation happens, while regular FastCGI requests will be sent to the optimized block with the caching and header bypass features.

Note the fastcgi_cache_key. This is the last thing I want to talk about:

Gotcha #2: Caching and HEAD Requests

nginx’s fastcgi_cache_key option does not have a default value, or at least the docs do not give one. The example may also be insufficient for everyday needs, or at least I thought it was. However, the proxy_cache_key has the following default value:


This was pretty close to what I needed initially, and I just altered it to $http_host$fastcgi_script_name$request_uri. However, when I put this config into production, we started seeing the occasional blank page, and our monitors were reporting timeouts. It took quite a bit of digging to find out that some of our cache entires were blank, and a little googling later ended up revealing the obvious, which was confirmed in logs: HEAD requests were coming thru when there was no cache data, and nginx was caching this to a key that could not discern between a HEAD and a GET request.

There are a couple of options here: use fastcgi_cache_methods to restrict caching to only GET requests, or add the request method into the cache key. I chose the latter, as HEAD requests in my instance caused just as much load as GET requests, and I would rather not run into a situation where we encounter load issues because the servers are hammered with HEAD requests.

The new cache key is below:

fastcgi_cache_key $request_method-$http_host$fastcgi_script_name$request_uri;

iTerm2 Tweaks for Fun and Privacy

For any sort of system administrator working with Linux systems, a good terminal emulator is an essential part of the toolbox.

Using – the terminal that comes with OS X for default – has started to wear on me, due to the lack of shared pasteboard (leading to more mispastes this year than I have probably had in the last decade), and the lack of basic reconfiguration of certain keys like the tab macros (although these can be changed using system preferences).

Recently I have been using iTerm2. It’s extremely powerful, but some of its options needed a bit of tweaking to get the privacy to a level that I’m more comfortable with. I have also made a couple of other look and feel tweaks that are worth noting as well.

Privacy Tweaks

These tweaks can help secure iTerm2 a bit, especially if these options are not used often.

Disable pasteboard history

Controlling the number of items in the pasteboard history is a hidden option documented here.

This can be used to turn off the history altogether. Run this, and restart iTerm, and now pressing Command-Shift-H will return an empty table no matter what:

defaults write com.googlecode.iterm2 MaxPasteHistoryEntries -int 0

Disable VT100 printer code handling

I don’t know why this is not a default, but it can be can turned off easy enough.

Head to Preferences -> Profiles, select the profile to update (ie: the default one), then select Terminal and ensure Disable session-initiated printing is enabled.

Disable semantic history

Semantic history is a feature that facilitates a kind of Finder-like behaviour in iTerm. By Command-Click-ing something, that file can be loaded like it was run through Finder. It could possibly be useful, but I can see myself fat-fingering something I shouldn’t have and then having to deal with the consequences.

I haven’t found a way to disable the functionality completely, but the aforementioned macro can be disabled by going to Preferences -> Pointer and ensuring Command-Click Opens Filename/URL is not checked.

Disable application clipboard control (default)

This is thankfully a default, but it might be worth double-checking: go to Preferences -> General and ensure that Allow clipboard access to terminal apps is disabled. This ensures that the proprietary iTerm clipboard control code is not handled. See here for a full list of other iTerm-specific codes.

Fun Tweaks

These tweaks will generally improve the experience.

Keybinds (Global)

Head to Preferences -> Keys. These are the global keybinds.

Here, keys can be bound for general application-use macros. I like Control-PgUp/PgDn for Previous Tab/Next Tab respective as it’s a pretty well-accepted standard. Also, Control-Shift-T for new tab. This one needs to be selected as Select Menu Item… in the Action combo box, after which New Tab will be selectable.

Note that terminal-use keybinds can be edited on a per-profile basis in Preferences -> Profiles -> Keys.

Unlimited scrollback

Head over to Preferences -> Profiles, select the profile to update (ie: the default one), then select Terminal and ensure Unlimited scrollback is enabled. This limits the scrollback buffer to available memory.


No terminal is complete without a transparency feature. 😉 Head over to Preferences -> Profiles, select the profile to update (ie: the default one), and select Window. The transparency slider is there. The results show immediately, so it’s easy to see how much is needed.


I’m going to get tired of copying/pasting this text, honest. 😉

Head over to Preferences -> Profiles, select the profile to update (ie: the default one), and select Text. Fonts can be selected there.

For even more awesome fonts, check out Beautiful fixed-width fonts for OS X. This page has a tarball of the various misc-fixed fonts that are default for X11, including terminals like xterm. This is really great for that classic look-and-feel.


Last one. For now. 😉

Head over to Preferences -> Profiles, select the profile to update (ie: the default one), and select Colors.

Colours can be changed here, but even better, they can be saved as schemes. There is even a gallery with a pretty impressive collection of schemes to choose from.

Honorable Mention – X11

For terminal alternatives, this is worth mentioning. There are a few options to install X11 on a Mac, it can be downloaded here or through MacPorts. MacPorts can be used to install other terminals as well, such as rxvt-unicode or mrxvt, two extremely fast terminals that can be pretty well customized in their own right. The latter does not have unicode support, but is a personal favourite of mine, and if it wasn’t a bit of a pain to have to adjust locales on all the systems I may touch, I would probably be using it.

There are a few things to note if when using X11 terminals on Mac:

Launch proper terminal on X11 start

The that MacPorts installs is actually simply a wrapper to xterm (and by proxy, startx). The following command below will change the startup app to mrxvt

defaults write org.macports.X11 app_to_run /opt/local/bin/mrxvt

Enabling pasteboard text selection update

The gremlin of a split clipboard rears its head again. 😉

Luckily, this time it’s fixable. Simply open X11’s preferences, and select Pasteboard -> Update Pasteboard immediately when new text is selected.

nginx: FastCGI Caching Basics

I’m back!

Today, I am going to share some things regarding how to do caching in nginx, with a bit of a write up and history first.

The LAMP Stack

An older acronym these days, LAMP stands for:

  • (L)inux,
  • (A)pache,
  • (M)ySQL,
  • (P)HP.

I really am not too sure when the term was coined, but it definitely was a long time, probably about 15 years ago. The age definitely shows: there are definitely plenty of alternatives to this stack now, giving way to several other acronyms which I am not going to attempt to catalog. The only technology that has remained constant in this evolution has been the operating system: Linux.

MySQL (and Postgres, for that matter) have seen less use in favour of alternatives after people found out that not everything is best suited to go into a relational database. PHP has plenty of other alternatives, be it Node, Ruby, Python, or others, all of which have their own middleware to facilitate working with the web.

Apache can serve the aforementioned purpose, but really is not necessarily well-fated for the task. That’s not to say it can’t be. Apache is extremely well featured, a product of it being one of the oldest actively developed HTTP servers currently available, and can definitely act as a gateway for several of the software systems mentioned above. It is still probably the most popular web server on the internet, serving a little over 50% of the web’s content.

Apache’s Dated Performance

However, as far as performance goes, Apache has not been a contender for a while now. More minimal alternatives, such as the subject of this article, nginx, offer fewer features, but much better performance. Some numbers put nginx at around twice the speed – or faster – of some Apache MPMs, even on current versions. Out of the box, I recently clocked the memory footprint of a nginx and PHP-FPM stack at roughly half of the memory footprint of an Apache and mod_php5 server, a configuration that is still in popular use, mainly due to the issues the PHP project has historically had with threading.

Gateway vs. Middleware

PHP running as a CGI has always had some advantages: from a hosting background, it allows hosters to ensure that scripts and software get executed with a segregated set of privileges, usually the owner of the site. The big benefit to this was that any security problems with that in particular site didn’t leak over to other sites.

Due to the lack of threading, this is where PHP has gotten most of the love. Aside from FastCGI, there are a couple of other popular, high-performance options to use for running PHP middleware:

  • PHP-FPM, which is mainline in PHP
  • HipHopVM, Facebook’s next generation PHP JIT VM, that supports both PHP and Facebook’s own Hack derivative.

These of course, connect to a webserver, and when all the webserver is doing now is serving static content and directing connections, the best course of action is to pick a lightweight server, such as nginx.

Dynamic Language for Static Content?

Regardless of the option chosen, one annoying fact may always remain – the fact that there is a very good chance that the content being served by PHP is ultimately static during a very large majority of its lifetime. A great example of this is a CMS system, such as WordPress, running a site that may see little to no regular updates. In this event, the constant re-generation of content will place unnecessary load on a system, wasting system resources and increasing page load times.

The CMS in use may have caching options, which may be useful in varying capacities. Depending on how they run their cache, however, this could still mean unnecessary CPU used to run PHP logic, or database traffic if the cache is stored in the database.

Enter nginx’s Caching

nginx has some very powerful options for serving as a proxy server, and is perfectly capable of running as a layer 7 load balancer, complete with caching. The latter is what I am covering in this article.

nginx has 2 specific caching modules: the cache options stored in ngx_http_proxy_module and ngx_http_fastcgi_module. These control their respective areas: proxy_cache_* options are used in conjunction with standard requests and proxy options, and fastcgi_cache_* options are used with the FastCGI options (locations generally used with fastcgi_pass proxied namespace).

Setting up the Middleware

I am not covering setting up the middleware in this article, but it is very easy to get started with PHP-FPM. Usually, installing it is as easy as installing it through the respective distro (ie: apt-get install php5-fpm in modern versions of Debian or Ubuntu).

Ubuntu 14.04 sets up PHP-FPM to listen on /var/run/php5-fpm.sock, but it can, of course, be configured to listen on TCP as well.

Setting up nginx for FastCGI

Before jumping into the config below, keep in mind that the FastCGI cache needs to be defined in the core nginx http config, like so:

http {
  # Several omitted options here...
  fastcgi_cache_path /var/cache/nginx levels=1:2 keys_zone=fcgizone:10m max_size=200m;

This option dictates several things:

  • The path of the cache, in this case /var/cache/nginx
  • The structure of the path hierarchy. 1:2 constructs a directory structure that takes the last three characters of the MD5 hash computed file name and creates a structure like /var/cache/nginx/c/ab/ if your last 3 characters were abc.
  • keys_zone is the name and the size of the key cache. This is not the actual cache size, but memory used for cache key entries and metadata. One megabyte can hold approximately 8000 keys or cache entries.
  • max_size is what defines the size of the cache.

This can also be included and dropped into /etc/nginx/conf.d/ in most default setups.

The following details a basic server section in nginx. This will lie in somewhere like /etc/nginx/conf.d/, or /etc/nginx/sites-available/ if Debian or Ubuntu convention is being followed.

server {
  listen 80;

  root /var/www/;
  index index.php index.html;

  server_name _;

  # Logs
  access_log /var/log/nginx/;
  error_log /var/log/nginx/;

  location / {
          try_files $uri $uri/ /index.php$is_args$args;

  # FastCGI stuff
  location ~ \.php$ {
          include fastcgi_params;

          fastcgi_pass unix:/var/run/php5-fpm.sock;
          fastcgi_index index.php;
          fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;

          fastcgi_cache fcgizone;
          fastcgi_cache_valid 200 1m;
          fastcgi_cache_key $http_host$fastcgi_script_name$request_uri;
          fastcgi_cache_lock on;
          fastcgi_cache_use_stale error timeout invalid_header updating http_500 http_503;
  # deny access to .htaccess files, if Apache's document root
  # concurs with nginx's one
  location ~ /\.ht {
          deny all;

A note before I move on to the FastCGI stuff: the location block is set up to try a few options before 404ing: the direct URI, the URI as a sub directory (ie: to see if there is a default file here), and then, as a fallback, to request /index.php itself. This is mainly designed for sites that have a CMS system that uses permalinks (again, like WordPress).

Now, on to the FastCGI bits in the location block (which, if it wasn’t evident, passes all PHP content):

  • First off, /etc/nginx/fastcgi_params is included (shorthanded to a relative path). This file sets a number of environment variables that are essential to a functional FastCGI environment, and is included with most bundled versions of nginx.
  • fastcgi_pass is the option that passes the request to – in this case – PHP-FPM.
  • fastcgi_param as it’s shown here overrides an option that was set in fastcgi_params, and serves as an example of how to set environment. Basically I am building the SCRIPT_FILENAME environment variable for FastCGI by combining the document root and the path to the running script (evaluating to something like /var/www/ This is needed by some CMS systems.

Now, on to the main attraction:

  • fastcgi_cache references the cache zone that was defined earlier. This effectively turns on the cache, and is in theory the only option needed.
  • fastcgi_cache_valid sets cache entries for 200 (OK) code responses for 1 minute, in this instance.
  • fastcgi_cache_key is building a cache key. The object is to get a unique enough key generated here so that there are no cache conflicts that could lead to broken content. The one listed here gives one such as, which should be plenty. Of course, cache entries can be built off a number of things, including the many list of variables that nginx has.
  • fastcgi_cache_lock ensures that only one request for a new cache entry is sent at the same time. This has a default timeout of 5 seconds by default (controlled with fastcgi_cache_lock_timeout), after which the request will be passed through to avoid errors. However:
  • fastcgi_cache_use_stale, the last option, has a option named updating that allows a stale entry to be passed to any other requests in the case that there is currently a lock. This enables a simple yet effective throttling mechanism to back end resources. In this configuration, approximately one request per URL would come through every minute. There are also other flags here that allow stale entries to be used in the event of several kinds of errors. Depending on how the application is set up, your mileage may vary.

Lastly, not a caching option, but I block .htaccess files just in case there are any left over in the content since moving from Apache, if that change was made.

This above is really the tip of the iceberg when it comes to nginx caching. There are several other cache manipulation options, allowing for finer grained cache control, such as fastcgi_cache_bypass to bypass the cache (ie: honouring Cache-Control headers inbound or manually exempting admin areas), or even more sophisticated scenarios such as setting up the cache to be purged or revalidated via special requests. Definitely take a look at the documentation mentioned above if you are interested. Keep in mind that some require later versions of nginx (the one bundled in Ubuntu 14.04 for example is only 1.4.6), and cache purging actually requires nginx+, nginx’s premium server.

One last thing about the cache that should be noted: Cache-Control response headers from FastCGI are honoured. This means that if the application in question has an admin area that passes these headers, it is not necessary to set up any exceptions using fastcgi_cache_bypass.

Adventures in OpenStack: Intro to OpenFlow, and Network Namespaces

I’m digging into the backlog today! I’ve had these thoughts jotted down since trying to solve a problem on another OpenStack all-in-one box a few weeks ago, and I’m glad to finally get it finished. So without further ado, let’s jump in!

The Questions

I have already covered Open vSwitch and OpenStack networking in the following two articles:

There have been some unanswered questions for me however:

Why VLAN trunking anomalies seem to be present on patch ports

If one looks at the output of ovs-vsctl show, some confusion may ensue. For example, there are several VLAN tags there, but if all of them are trunked across (as is the behaviour of a patch port), which VLAN wins? Do any? How is this even working?

    Bridge br-int
        fail_mode: secure
        Port "foo"
            tag: 4
            Interface "foo"
                type: internal
        Port "bar"
            tag: 3
            Interface "bar"
                type: internal
        Port "jack"
            tag: 1
            Interface "jack"
        Port "jill"
            tag: 2
            Interface "jill"
                type: internal
        Port br-int
            Interface br-int
                type: internal
        Port "int-br-ex"
            Interface "int-br-ex"
                type: patch
                options: {peer="phy-br-ex"}
    Bridge br-ex
        Port "enp2s0"
            Interface "enp2s0"
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.1.3"

How does OpenStack handle several layer 3 networks over the same router

My other question was – observing that OpenStack does not create any sort of VM for routing or what not – how does routing even work? I mean, managing ultimately what could be thousands of tenant networks and possibly dozens or even hundreds of external networks can get pretty messy, I would imagine.

The answers were pretty clear, once I dug a bit deeper.

Open vSwitch, and Integration to External Bridge Mapping

The OpenStack integration bridge will maps to two kinds of bridges, depending on where in the architecture is looked at:

  • The external bridge (as shown above) – this is generally done on network nodes and my all-in-one setup
  • The tunnel bridge (not shown above to save space) – this is done on regular compute nodes, for example

This is specifically denoted by the two patch ports in each bridge:

        # br-int
        Port "int-br-ex"
            Interface "int-br-ex"
                type: patch
                options: {peer="phy-br-ex"}
        # br-ex
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}

As mentioned, all VLANs are passed over a bridge. Think of it as a trunk port on a physical switch that is set to match all VLANs. The vSwitch layer of OVS does not perform any sort of selective VLAN mapping.

So if all VLANs going over this port are tagged, then how do we make sense of what we see in the external bridge, which has no tags at all? All ports are either untagged or are trunks, so just looking at this at face value, it would seem that like a bad configuration.

Not necessarily.

OpenFlow Magic on External Bridge

The switch layer is only half the story when deal with Open vSwitch. The second part is what happens with OpenFlow on the external bridge:

# ovs-ofctl dump-flows br-ex
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=3776.846s, table=0, n_packets=5106, n_bytes=1142456, idle_age=0, priority=1 actions=NORMAL
 cookie=0x0, duration=3654.201s, table=0, n_packets=0, n_bytes=0, idle_age=3654, priority=4,in_port=2,dl_vlan=1 actions=strip_vlan,NORMAL
 cookie=0x0, duration=3776.341s, table=0, n_packets=132, n_bytes=10608, idle_age=3703, priority=2,in_port=2 actions=drop

The second rule is the specific one we want to pay attention to. This rule contains the strip_vlan action, which actually removes any tags outgoing on this port, matching off VLAN 1. So any traffic coming into port 2 on the external bridge (basically the peer port to the integration bridge) off of VLAN 1 (which one would assume is the external network), will have its VLAN stripped before being forwarded.

And hence, mystery solved! Now moving on to the other issue – routing.

Network Namespaces

As previously mentioned, one would imagine that networking would get pretty messy when implementing the routing of several several tenant networks over a single router – consider the amount of networks, interfaces, and routes (including default routes) that these nodes would have to manage, and the head may spin pretty quickly.

So how to manage all of these routes in a sane fashion? Enter network namespaces.

Network namespaces are a fairly recent addition to the Linux kernel. Introduced in version 2.6.24, I have found the easiest way to think about the feature is to think about it in the context of the work that has been done on containers in the last few years (to support things like LXC, CoreOS, and Docker). Each network namespace is its own individual pseudo-container, an island of networking, pretty much its own individual virtual router.

These map to OpenStack pretty visibly. For example:

# neutron router-list -F id -F name
| id                                   | name                      |
| f44651e2-0aab-435b-ad11-7ad4255825c7 | r.lab.vcts.local          |

Above is the router ID for my current lab network. Perhaps, in the name of good convention, this has a matching namespace?

# ip netns show | grep f44651e2-0aab-435b-ad11-7ad4255825c7

Why yes, yes it does!

Now, there are tons of things that can be done within a network namespace, but I’m not going to cover them all, as they are not necessarily relevant within the context of a fully working OpenStack implementation, as everything is already going to be set up.

One of the best ways to troubleshoot a namespace is to enter it using ip netns exec. Note that this is not a fully separate container. Instead, commands are just executed within the context of that specific network namespace, the idea being that commands can be run that are not necessarily namespace aware.

Commands can be ran individually, but it may just be easier to run a shell within the target context, like so:

# ip netns exec qrouter-f44651e2-0aab-435b-ad11-7ad4255825c7 /bin/bash
# ip route show
default via dev qg-0c4c9d04-f0 dev qr-da3efe6d-a2  proto kernel  scope link  src dev qg-0c4c9d04-f0  proto kernel  scope link  src 

When the above is looked at, some pieces may start fitting together. And even though I haven’t covered it here, it will make sense from the above: There is the internal interface qr-da3efe6d-a2, which has the internal network The external interface has been bound thru OpenStack controls to on qg-0c4c9d04-f0, which then allows general outbound through the default route, and 1-1 nat for floating IP addresses.

Plenty of other commands can be run within this bash shell to get useful information, such as ip addr, ifconfig, and iptables, to get information on which IP addresses are bound to the router and how the firewall and NAT is set up.

Additional Reading

Hopefully the above gives you lots of insight into how networking works on OpenStack. For further reading, check out Networking in too Much Detail, the page that served as a starting point for a lot of this research. also has a pretty awesome article explaining namespaces here.

Configuration Management Series Part 2.5 – Chef Follow-Up

As luck would have it, I have been working a lot more with Chef in the last couple of weeks, so a supplemental article is in order.

There have been a number of things that I have found out in this period, and there is a good amount that I probably will not cover, but I am going to try and cover some of the key challenges that I have encountered upon my journey. This should give you, the reader, some insight into why I have chosen to continue with Chef, and maybe help you make some good decisions on data management and code design that will save you time on your own ramp-up.

Provisioning with Knife Plugins

First off, the reason why I have decided to keep going with Chef in the first place. Incidentally, it would seem that Chef has the best support for provisioning out of the three tools that I have looked at, especially as far as vSphere is concerned.

As it currently stands, I am in a situation where I need a tool that is going to handle the lifecycle of an instance, end-to-end. Ansible’s vsphere-guest module, unfortunately, was not satisfactory for this job at all, and in fact, working with this in particular piece of code was a lesson in frustration. The code is highly inconsistent across operations – specifically, very few features that are available in instance creation are available in template clone operations, and the available functionality in the former was not suitably portable. Puppet seemingly lacks such a feature altogether, outside of Puppet Enterprise. Both of these are show stoppers for me.

Enter knife-vsphere. An amazing tool, and 100% free. Incidentially, cloning from template is the only creation operation supported, which makes sense, considering that a brand new instance would more than likely be useless for provisioning on. Linux guest customization is fully supported, and programmable support for Windows customization is as well (see examples in the README). Finally, it can even bootstrap Chef onto the newly created node. The tool will even destroy instances and remove their entries from the Chef server if the instance label has a matching node.

Having these features in Chef, it was pretty much a no-brainer to see what wins out here.

Of course, vSphere is by far not the only knife plugin that supports stuff like this. Check out knife-ec2 and knife-openstack, both of which support similar behaviour.

Data Bags and Vault, Oh My!

I think my assessment that data bags were close to Hiera was premature. In the time that I wrote that, I have learned a few key takeaways.

First off, data bags are global. And even though I used global lookups directly in my first article about the subject, Hiera objects are designed to be used in a scoped auto-lookup mechanism, as per their documentation. Incidentally, data bags are the ones that are intended to be accessed in a fashion that is more in line with what I described in that article. Live and learn eh?

Conventions can be created that can help with this. For example, one can set up a bag/item pair like foo/bar if the cookbook’s name is foo. Also, cookbooks can still be parameterized elsewhere, and in fact Chef features like roles and environments are better suited for this, by design. (Note: Please read the last section of this article before investing too much into roles and environments, as roles in particular are frowned upon in parts of the Chef community).

Considering the latter part of the last paragraph, one would wonder: what exactly are data bags ultimately good for then? What indeed. A little bit of research for me revealed this good commentary on the subject by awesome community Chef 2015 winner Noah Kantrowitz. Incidentally, Noah recommends creating resources for this, versus using roles or environments.

As it currently stands, my general rule of thumb for using data bags is encryption. Does it need to be encrypted? Then it goes into a bag. Not a regular encrypted bag though – a regular encrypted bag requires a pre-shared key that needs to be distributed to the nodes somehow. There is no structure around controlling access to the data. How does this problem get solved then? Enter chef-vault.

Vault allows encryption via the public keys of the nodes that need the data, effectively ensuring only these nodes have access. This can explicitly be controlled by node or through a search string that allows searching on a key:data pair (ie: os:linux). This addresses both concerns mentioned in the previous paragraph. The only major issue with this setup is that the data needs to be encrypted across the key that will ultimately be supplied to the node, creating a bit of a chicken-before-the-egg problem. Luckily, it looks like Chef is catching up to this, and recently vault options for knife bootstrap were added to get past this. Now, when nodes are created, vault items can be updated by the host doing the provisioning, allowing a node access to vault items even during the initial chef-client run. This is not supported on validator setup, as the ability to do this anonymously (well, with the validator key) could possibly mean a compromise of the data.

Down the Rabbit Hole

Lastly, I thought I would share some next steps for me. The really fascinating (and frustrating) part of Chef for me is how the community has adopted its own style for using it. There are several years of social coding practices, none of which have been in particularly well documented by mainline.

First off, I am currently working on re-structuring my work to be supported by Berkshelf. This is now a mainline part of the Chef DK, and is used to manage cookbooks and their dependencies.

There are also a number of best practices for writing cookbooks, usually referred to as cookbook patterns in the community, which definitely reflects the developer-centric nature of Chef. Based on some very light and non-scientific observation, one of the more popular documents for this seems to be The Environment Cookbook Pattern by Jamie Winsor, the author of Berkshelf.

2 other articles that have helped me so far are found below as well:

You can probably expect to hear more out of me when it comes to Chef, stay tuned!

Configuration Management Series Part 3 of 3: Puppet

At the start of 2013, the MSP that I was working for at the time landed its largest client in the history of the company. The issue: this meant that I needed to set up approximately 40 servers in a relatively small period of time. This aside, the prospect of manually installing and configuring 40 physical nodes did not entertain me at all. Network installs with Kickstart were fine, sure, but I needed more this time.

I had been entertaining the thought of implementing some sort of configuration management for our systems for a while. I had always hit a wall as to exactly what to use it for. Implementing it for the simple management of just our SSH keys or monitoring system configs seemed like a bit of a waste. I had lightly touched Chef before, but as mentioned in the last article, did not have much success in getting it set up fully, and did not have much time to pursue it further at that point.

I now had a great opportunity on my hands. I decided to take another look at Puppet, which I had looked at only briefly, and found that it seemed to have a robust community version, a simple quickstart workflow, and quite functional Windows support. Call it a matter of circumstance, but it left a much better impression on me than Chef did.

I set up a Puppet master, set up our Kickstart server to perform the installs and bootstrap Puppet in the post-install, and never looked back.

For this client, I set up Puppet to manage several things, such as SSH keys, monitoring system agents (Nagios and Munin), and even network settings, with excellent results. I went on to implement it for the rest of our network and administration became magnitudes easier.

Since then I have used Puppet to manage web servers, set up databases, manage routing tables, push out new monitoring system agents and configure them, and even implement custom backup servers using rsnapshot. One of my crowning achievements is using Puppet to fully implement the setup of private VPN software appliances, including OpenVPN instances complete with custom Windows installers.

Ladies and gentlemen, I present to you: Puppet.


I am used to this process by now, but I’d probably have to say that standing up a Puppet master is still an extremely easy process. There are plenty of options available – if it’s not possible to use the official repository (although it is recommended), most modern distros do carry relatively up to date versions of the Puppet master and agent (3.4.3 in Ubuntu 14.04 and 3.6.2 in CentOS 7). Installation of both the WEBrick and Passenger versions are both very straightforward and not many (if any) configuration settings need to be changed to get started.

There is also the emergent Puppet Server, which is a new Puppet master that is intended to replace the old implementations. This is a Clojure-based server (meaning it runs in a JVM), but don’t let that necessarily dissuade you. If the official apt repositories are being used, installing puppetserver well install everything else needed to run it, including the Java runtime.

Funny enough, in contrast to the beefy 1GB requirements of Chef, I was able to get Puppet Server up and running with a little over 300 MB of RAM used. Even though the installation instructions recommend a minimum of 512MB, I was able to run the server with JAVA_ARGS="-Xms256m -Xmx256m" (a 256MB heap size basically) and perform the testing I needed to do for this article, without any crashes.

After installing the server, things were as easy as:

  • apt-get installing the puppet agent,
  • Configuring the agent to contact the master,
  • Starting the agent (ie: service puppet start),
  • And finally, using puppet cert list and puppet cert sign to locate and sign the agent’s certificate request.

The full installation instructions can be found here:

Windows Support

As mentioned last article, one or the reasons that I actually did choose Puppet over Chef was that its Windows support seemed to be further along.

One of the things that I do inparticularly like about Puppet’s Windows support was the way they made the package resource work for Windows: it can be used to manage Windows Installer packages. This made deployment of Nagios agents and other software utilities to Windows infrastructure extremely easy.

Other than that, several of the basic Puppet resources (such as file, user, group, and service) all work on Windows.

Head on over to the Windows page for more information.


In the very basic model, Puppet agent nodes are managed through the main site manifest located in /etc/puppet/manifests/site.pp. This could look like:

node "srv2.lab.vcts.local" {
  include sample_module

Where sample_module is a module, which is a unit of actions to run against a server (just like cookbooks in Chef).

Agents connect to the Puppet master, generally over port 8140 and over an SSL connection. Nodes are authenticated via certificates which Puppet keeps in a central CA:

root@srv1:~# puppet cert list --all
+ "srv1.lab.vcts.local" (SHA256) 00:11:22... (alt names: "DNS:puppet", "DNS:srv1.lab.vcts.local")
+ "srv2.lab.vcts.local" (SHA256) 00:11:22...

This makes for extremely easy node management. All that needs to be done to authorize an agent on the master is puppet cert sign the pending request. puppet cert revoke is used to add a agent’s certificate to the CRL, removing their access from the server.

As far as data storage goes, I have already written an article on Hiera, Puppet’s data storage mechanism. Check it out here. Again, Hiera is a versatile data storage backend that respects module scope as well, making it an extremely straightforward way to store node parameter data. Better yet, encryption is supported, as are additional storage backends other than the basic JSON and YAML setups.

Execution Model

Natively, right now I would only say that Puppet natively supports a pull model. This is because its push mechanism, puppet kick, seems to be in limbo, as illustrated by the redmine and JIRA issues. The alternative is to apparently use MCollective, which I have never touched.

By default, Puppet runs every 30 minutes on nodes, and this can be tuned by playing with settings in /etc/puppet/puppet.conf.

Further to that, one-off runs of the puppet agent are easy enough, just run puppet agent -t from the command line, which will perform a single run with some other options (ie: slightly higher verbosity). This can easily be set up to run out of something like Ansible (and Ansible’s SSH keys can even be managed through Puppet!)

Puppet also supports direct, master and agentless runs of manifests thru the puppet apply method. Incidentally, this is used by some pretty well-known tools, notably packstack.


This was neat to come back to during my evaluation yesterday. The following manifest took me maybe about 5 minutes to write, and supplies a good deal of information on how the declarative nature of Puppet works. Here, srv2.lab.vcts.local is set up with a MySQL server, a database, and backups, via Puppet’s upstream-supported MySQL module.

node "srv2.lab.vcts.local" {
  class { '::mysql::server':
    root_password => 'serverpass',
  mysql::db { 'vcts':
    user => 'vcts',
    password => 'dbpass',
  file { '/data':
    ensure => directory
  class { '::mysql::server::backup':
    backupuser => 'sqlbackup',
    backuppassword => 'backuppass',
    backupdir => '/data/mysqlbackup',
    backupcompress => true,
    backuprotate => 7,
    file_per_database => true,
    time => [ '22', '00' ],

The DSL is Ruby-based, kind of like Chef, but unlike Chef, it’s not really Ruby anymore. The declarative nature of the DSL means that it’s strongly-typed. There is also no top-down workflow – dependencies need to be strung together with require directives that point to services or packages that would need to be installed. This is a strength, and a weakness, as it is possible to get caught in a complicated string of dependencies and even end up with some circular ones. But when kept simple, it’s a nice way to ensure that things get installed when you want them to be.

Templates are handled by ERB, like Chef. The templates documentation can help out here.

The coolest part about developing for Puppet though has to be its high-quality module community at Puppet Forge. I have used several modules from here over the years and all of the ones that I have used have been of excellent quality – two that come to mind are razorsedge/vmwaretools and pdxcat/nrpe. Not just this, but Puppet has an official process for module approval and support with Puppet Enterprise. And to ice the cake, Puppet Labs themselves have 91 modules on the Forge, with several being of excellent quality and documentation, as can be seen by looking at the MySQL module above. It’s this kind of commitment to professionalism that really makes me feel good about Puppet’s extensibility.


A good middle ground that has stood the test of time

Puppet is probably the first of a new wave of configuration management tools that followed things the likes of CFEngine. I really wish I knew about it when it first came out – it definitely would have helped me solve some challenges surrounding configuration management much earlier than 2013. And it’s as much as usable today, if not more so. With the backing of companies like Google, Cisco, and VMware, Puppet is not going away any time soon. If you are looking for a configuration management system that balances simplicity and utility well, then Puppet is for you. I also can’t close this off without mentioning my love for Puppet Forge, which I personally think is the best module community for its respective product, out of the three that I have reviewed.


In the circles that I’m a part of, and out of the three tools that I have reviewed over this last month, Puppet doesn’t nearly get the love that I think it should. Some people love the Ansible push model. Other developers love Chef because that’s what they are used to and what they have chosen to embrace. Puppet – for no fault of its own, to be honest – seems to be the black sheep here.

Maybe I just need to get out more. A little over two years ago, RedMonk published this article comparing usage of major configuration management tools. If you look at this, at least 2 years ago, there would be no reason to say that Puppet should be put out to pasture yet.

The End

I hope you enjoyed this last month of articles about configuration management.

I thought I’d put forward some thoughts I have had while working with and reviewing these tools for the last month. I came into this looking to do a “shootout” of sorts – basically, standing up each tool and comparing their strengths and weaknesses based off a very simple set of tests and a checklist of features.

I soon abandoned that approach. Why? First off – time. I found that had very limited time to do this, sometimes mainly a Sunday afternoon of an otherwise busy week, to set up, review, and write about each tool (and even at that the articles sometimes came out a few days late).

But more importantly for me I didn’t want to hold on to some stubborn opinion about one tool being better than the other. So I decided to throw all of that stuff out and approach it with an open mind, and actually immerse myself in the experience of using each tool.

I think that I have come out of it better, with an actually objective and educated opinion now about the environment that suits each tool best. Ultimately, this has made me a better engineer.

If you have read this far, and have read all three major articles, first off, thank you. Second off, if you are having trouble choosing between one of the three reviewed in this series, or another altogether, you might want to ask yourself the following few questions:

  • What does my team have knowledge on already?
  • Will I be able to ramp up on the solution in a timely manner with the least impact?
  • Will the solution be equally (or better supported) by future infrastructure platforms?

Most of all, keep looking forward and keep an open mind. Don’t let comfort in using one tool keep you closed off to using others.

Thanks again!

Configuration Management Series Part 2 of 3: Chef

Be sure to read the follow up to this article as well!

Back when I was looking at configuration management about 4 years ago, I explored the possibility of using Chef over Puppet. A few things threw me off at that point in time: the heavy reliance on Ruby for the setup and configuration (at least via the main documented methods), the myth that you needed to know Ruby to work with it, and ultimately, time: the ramp-up time to get started seemed so long that it was actually time prohibitive for me at a period in my life where I had very little to spare. A couple of years later, when it came to pick one to work with, I ultimately chose Puppet.

Over the years, I have heard talk every now and then about use of configuration management tools. It saddened me to hear not so much talk about Puppet, but interestingly, a lot of developers that I spoke with really like Chef, and there’s no doubt that some consider it an essential part of their toolbox. Read on, and you will probably see why.

About Chef

Chef seems to have come about as a matter of necessity, and ultimately was the manifestation of the end result of DevOps – infrastructure automation. Creator Adam Jacob probably tells it way better than I could in this video, where he explains the name, his journey in making the software, and OpsCode’s beginnings.

What I actually got from the video was the dispelling of a misconception I had – that Chef actually came from former employees of Puppet that wanted to create something better. If that was the case, it may have just been a natural evolution of the philosophies that Puppet was built on, and how Adam and company thought they could make it better to suit OpsCode’s needs.


I’m not going to lie – this was probably the most frustrating part of my teardown on Sunday.

Installation is pretty straightforward. Actually, very straightforward now, as opposed to when it tried it out so many years ago. The server install documentation now gives a much better path to installation, versus how it used to be when it involved gems and what not.

Unfortunately, the base install of Chef server is extremely resource intensive. The installer package for the server on Ubuntu is about 460 MB in size, which takes sizeable amount of time to download and install. You need about 2 GB of RAM to run the whole stack – after installation the total footprint is a little over 1 GB in RAM. I actually had to rebuild my test instance after the initial install with 512 MB failed.

From here, what to do next was a little confusing for me. This ultimately is due to the very modular nature of the Chef management lifecycle – which is great for developers (see below), but is a bit intimidating for first-time admin users or people looking to stand up with little time and minimal knowledge – only having a Sunday afternoon for review is a great example of this.

Ultimately, I pushed through and figured it out. The full path to getting a node up is basically:

This will get you set up with the server, a workstation to do your work, and a node to test with. What this does not do, apparently, is get chef-client to run on boot on the bootstrapped node. I’m not too sure why this is, but the installer package does not include any init scripts, unfortunately. With me being overdue for press time as it is, I chose not to investigate for now, and just ran chef-client manually to test the cookbook that I wrote.

Windows Support

Chef has pretty mature Windows support. Actually, one of the reasons that I chose Puppet over Chef in the first place was that Puppet’s Windows support was further along. I would imagine both are in the same place now.

Chef has also been working on support for PowerShell DSC, MS’s own approach to configuration management through PowerShell.

Head over to the Windows page to see the full feature set for Chef on Windows.


Chef takes a very modular approach to management. Ideally, one has the server, and changes are made from workstations thru knife, save possibly organization creation and user management. This includes cookbook design, cookbook uploading, node bootstrapping, run list editing (the list of items that get run on a node during a run), and pretty much everything else about the Chef development lifecycle.

Again, hosts are bootstrapped usually via knife bootstrap – but there are other deployment options as well. See the bootstrap page for more options.

Data storage for nodes is done thru data bags – JSON-formatted data with support for encryption, similar to Hiera and hiera-eyaml for Puppet. I haven’t had much of a chance to look yet, but it looks like the features to not only automate this process and have encrypted and unencrypted data co-exist within the same data bag is a lot more automated and developer-friendly, probably better than Puppet and much better than Ansible.

Finally, there are the premium features. Check the Chef server page for more details. These features include a web interface, extra support for push jobs, analytics, and high availability. It should be noted that these features are free for up to 25 nodes.

Execution Model

Some more time needs to be taken by myself to evaluate all of these methods, but generally, chef-client is your go-to for all execution. Take a look at the run model here. This can be run as a daemon, periodically thru cron, direct on the server, or via knife ssh.

Speaking of which, there is also knife, where most of your Chef management needs will be taken care of thru.

There is also chef-solo – basically, a standalone Chef that does not require a server to be run. This kind of supplies an agentless push-execution model for Chef that can be useful for orchestration.


Chef really is a developer’s dream come true.

The Chef DK installation model encourages a developer-centric development and deployment cycle, allowing all management and development to be done from a single developer’s workstation, with changes checked into central source control.

The DSL is pretty much pure Ruby. When Chef first started to become a thing, this was yet another excuse thrown out to not use it – ie: you would need to learn Ruby to learn Chef. But really, nearly all configuration management systems these days use some sort of language for their DSL, be it Ruby, YAML, JSON, or whatever. I would even submit that one can begin teaching themselves a specific language by taking up a configuration management system – learning Puppet actually helped me understand Ruby and ERB a bit. In addition to that, Chef has a great doc that can help you out – Just Enough Ruby for Chef.

Templating is done in ERB, just like in Puppet. See the template documentation for more details.

Terminology and actual concepts are more in line with Puppet. Cookbooks are pretty much the analog to modules or classes in Puppet, with recipes translating to individual manifests (the run data). You can use knife cookbook to create, verify, and upload cookbooks.

Speaking of development – this is actually a functional recipe:

file '/etc/motd' do
  content 'hello world!'

Set up within a cookbook – this would write “hello world!” to /etc/motd. Pretty simple!

I think the killer app for development for me here though has to be the testing support that Chef has built in:

  • kitchen is Chef’s built-in test suite that allows you to orchestrate a test scenario from scratch using a wide range or virtualization technologies (including Vagrant – so you don’t even need to have code leave your workstation)!
  • is a great resource for learning Chef, versus testing Chef. It is a hosted training platform that will set up temporary instances for you to use, along with a guided tutorial.

For me, when people say that Chef is more developer-friendly – it’s not necessarily what you can do or what you can use, but the fact that the toolset enables developers to get code out all that faster.


Great for developers

Chef is awesome for developers, and it shows. The Ruby DSL is extremely easy to work with – that fact that they have made it more programmatic than other configuration management systems would seem to allow it to do more without having to extend it too much or look to third parties for extensions. Also, the Chef DK and Kitchen provide for a very pleasant development experience.

A little tough to get ramped up on

If you are in a pinch and don’t have a lot of time to set up and do not know Chef, you might want to figure something else out until you can block a bit of time to teach yourself. There are a lot of options for you to use Chef, which can make it overwhelming. Setting up the Chef server as well is a bit of a commitment of resources that you may want to consider carefully before you undertake it.

Next up in the series – an oldie but a goodie, the tool that got me started down the configuration management rabbit hole – Puppet.

Ansible: Handling Different Operating Systems with Variables

A fundamental part of working with configuration management tools is the ability to handle different operating systems within the same logic stream, such as a module.

I’ve found a few challenges addressing this topic within Ansible, and hence I think it’s worth noting certain things that you can’t do, and I think is the most logical way (so far).

Things That do not Work

As with most things I learn in life, I figured these out via trial and error, via an approach something along the lines of “I should be able to do this, shouldn’t I?”

Variables and facts cannot be directly referenced in a handler

I tried this:

# handlers/main.yml
- name: (Ubuntu) restart ssh
  service: name=ssh state=restarted

- name: (CentOS) restart ssh
  service: name=sshd state=restarted

# tasks/main.yml
- name: test SSH restart
  lineinfile: dest=/etc/ssh/sshd_config line=#testing
  notify: ({{ ansible_distribution }}) restart ssh

This gives you this:

ERROR: change handler (({{ ansible_distribution }}) restart ssh) is not defined


Logic does not work in handler

You also can’t do something like this:

# handlers/main.yml
- name: restart ssh
  service: name=ssh state=restarted
  when: ansible_distribution == 'Ubuntu'

- name: restart ssh
  service: name=sshd state=restarted
  when: ansible_distribution == 'CentOS'

In this instance, only the first handler is looked at, and if the logic in when: does not evaluate to true, the handler will be skipped. Completely.

Defining variables within a task or handler

You cannot use a vars directive within a task or handler: you simply get this:

ERROR: vars is not a legal parameter in an Ansible task or handler

Note that facts can be set using the set_fact module but this is better used when there is a need to define host-specific variables for use later, in more of a host scope versus a role scope.

The Right Way

I mentioned that variables within a role should not be used, but that’s only half the story. Specifically, variables should not be used for parameters – if you are going to define those variables later within a playbook or via group_vars/ or host_vars/, it is better to make them defaults.

But on the other hand, they are better suited as variables if they are not intended for assignment outside of the scope of the role. This is perfect for what we are planning on using them for here.

Variables can be defined in a role pretty easy – see roles. Note that in here there is a vars/ directory, from which vars/main.yml is loaded with high priority.

This is not the file I am looking for though, as variable files are not very flexible: no additional files or logic can go into it, just values.

So how can variables be included logically then?

include_vars and with_first_found

include_vars is a module that allows you to include variables from within a task. Perfect for the kind of logic flow we are looking for, and allows you to specify a file based off either a jinja2 template handler or the special {{ item }} token.

That’s only part of it though. Logic is needed to handle different operating systems, and can get pretty messy if all that is being used is a bunch of when: clauses. Enter with_first_found:

# tasks/main.yml
- name: set distro-specific variables
  include_vars: '{{ item }}'
    - '{{ ansible_os_family }}.yml'
    - default.yml

Using this, files can now be placed like so:

# vars/Debian.yml
ssh_svc: ssh

# vars/RedHat.yml
ssh_svc: sshd

# vars/default.yml
ssh_svc: ssh

Now, if you have a handler such as:

# handlers/main.yml
- name: restart ssh
  service: name={{ ssh_svc }} state=restarted

Either ssh or sshd would be used, depending on distro.

Note that with_first_found can be used to set up logic for other things, not just include_vars, it can be especially useful within your task workflow, to avoid including logic that may artificially take up log space or time in a playbook. These includes could be used in the place of when: clauses, for example.

More detail on logic and templating below. Most with_ clauses can be found in the Loops page, but there are a couple of more within the other pages. I wish Ansible was better at documenting this!

From the “Holy Crap” department: RegExr!

Trial-and-error testing of regular expressions has always been a hassle for me – usually involving a decent amount of debugging, command-line testing and otherwise messing with code in a way that I ultimately have to revert before closing off.

Today while helping out a colleague, I was like “man, it would be cool if there was a regex fiddle”.

Turns out there are several, but the first real functional one I stumbled on – RegExr – seems pretty amazing.

You can check it out at

The “killer app” features that I personally like about RegExr in particular are the real-time expression compilation and match highlighting, and the highly informative tooltips – including listing backreference data when you mouse over matches. There is also a nicely set up left sidebar, with documentation, and links to community expressions (oh yeah, you can share the expressions you create with the world, of course).

The codebase is MIT Licensed as well, and be found on the GitHub repo.

This fine work seems to be the doing of one Edmontonian named Grant Skinner ( and his associates/employees. Thanks dude, if you’re ever in Vancouver I’ll buy you a beer. 🙂