Adventures in OpenStack – Neutron Setup Errata and Multiple External Networks

So, a bit of a confession. The article Adventures in OpenStack – Networking has a an issue in how to set up external networks. The section that mentioned what to enter in the [ovs] section actually does nothing.

Why? Because by default, the Layer 3 (L3) agent only allows one external network, hardcoded in the config and defaulted to br-ex:

# Name of bridge used for external network traffic. This should be set to
# empty value for the linux bridge. when this parameter is set, each L3 agent
# can be associated with no more than one external network.
# external_network_bridge = br-ex

As such, the first network flagged as “external”, as long as the name is available as a valid network name in the ML2 plugin config, will function and be mapped to the external bridge listed above.

This came up for me when I tried to add a second external network. Before a few versions of OpenStack ago, this needed to be done by running a second Layer 3 agent. But now with the ML2 plugin it is possible to run them out of the same agent.

The right way to do this is:

  • Make sure that have all the external networks listed in the ML2 config. If this is the flat network type, it would look like so:
flat_networks = external,external2
  • Then, head over to l3_agent.ini and ensure that you have cleared the following as blank: 1) the external gateway ID, and the external network bridge name:
gateway_external_network_id =
external_network_bridge =

I make sure these are explicitly defined (uncommented and set correctly), to avoid issues.

  • Finally, add the OVS bits to the right config which should be in /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini:
network_vlan_ranges = external,external2
bridge_mappings = external:br-ex,external2:br-ex2

Even though network_vlan_ranges might seem redundant or misleading for flat networks, the definition is still required.

You are now ready to restart Neutron. There is a shortcut for this, by the way:

openstack-service restart neutron

One Last Thing

If you are having issues with your existing external networks after performing the above, check for this in your Neutron logs:

openvswitch-agent.log:2015-02-22 18:27:39.369 22377 WARNING neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2fb3e41a-f068-4737-97f8-580af5ddad27 None] Device 598f6434-06ff-4280-af5c-6ce7c312ca5b not defined on plugin
server.log:2015-02-22 18:27:37.393 22455 WARNING neutron.plugins.ml2.rpc [req-2fb3e41a-f068-4737-97f8-580af5ddad27 None] Device 598f6434-06ff-4280-af5c-6ce7c312ca5b requested by agent ovs-agent-beefhost.vcts.local on network d8f7930b-d4e9-422d-b789-7c2866ab81e8 not bound, vif_type: binding_failed

If you see this, you may need to re-create your external networks. There may be another way to fix this, seeing as it may have to do with a bad external network/bridge association, but if you are in a (small enough) lab, it should be trivial to just tear down and rebuild the network and router gateway bindings (only the external networks, subnets, and gateway associations need to be fixed).

Adventures in OpenStack – Launching an Instance, and Attaching a Floating IP

Now that I have all the basic building blocks set up, I can proceed to setting up an instance.

Incidentally, packstack set most of this up already, but I actually ripped out most of the networking stuff created by packstack, and also removed the demo tenant and set all that stuff up from scratch. I’m glad I did, as I have a lot better understanding as to what is going on with that part now. Understanding of Cinder and Swift is not necessarily nearly as important on a single server cloud as far as I’m concerned, and can be looked into later.

In any case, time to get started.

Make sure that the correct tenant is selected before doing any of this – admin is not the correct one! I may address this in a later article, but you either want to make sure that admin has correct permission for the project, or log in as the project’s admin user.

Adding a Key Pair

Instances cannot be logged into over SSH if there is no key pair.

Go to Projects -> Compute -> Access and Security. Under Key Pairs, a key pair can either be created, or imported. Importing brings up this window which you can put in your own OpenSSH public key:

OpenStack - Import Key Pair

Creating a key pair will create a key pair, and then automatically download it.

Adding SSH Access Rules

Under Projects -> Compute -> Access and Security -> Security Groups, a firewall rule needs to be added for SSH. Click Add Rule after selecting the default security group and enter the following:

OpenStack - Security Group Rule

This allows SSH globally. Rules can also be made to link security groups, allowing you to create more sophisticated access schemes across different networks.

Instance Creation

Time to create the instance.

Head to Project -> Compute -> Instances and click Launch Instance:

OpenStack - Create Instance 1

OpenStack - Create Instance 2

OpenStack - Create Instance 3

Note the options. I am using preconfigured flavours with the Debian Jessie image I uploaded. I am also using a test key pair, and the network I created.

Post-Creation gives you a place to add any post-installation scripts and what not, and Advanced Options allows you to set up custom partitioning if you want. I don’t need either of those, so I did not include them or modify them.

After clicking Launch, and waiting a bit of time, the instance should be visible:

OpenStack - Running Instance

I can now proceed to attaching a floating IP, so that the instance is publicly accessible.

Adding a Floating IP Address

Click the drop down box at the far right of the instance listing and select Associate Floating IP:

OpenStack - Floating IP 1

Since I do not have any IPs allocated, I have to request one, by clicking the + (plus) button:

OpenStack - Floating IP 2

I can then use the allocation given to me:

OpenStack - Floating IP 3

And now the IP address will be visible in the instance, after clicking Associate of course:

OpenStack - Floating IP 4This instance is now ready to log into! By connecting using the SSH key I was given, I can connect to over SSH and start working on the instance.

The Debian image’s default login at this time is debian. Most instances will NOT let you log in as root, so consult the image documentation for any specific default user that is created. Sometimes, logging in as root will give you the correct user to log in as and then disconnect you (such is the case with the Debian image).

Adventures in OpenStack – External Networks

When I left off yesterday, I just finished discussing the basics of setting up Neutron on a single-server OpenStack deployment. I set up a flat network – external – and mapped it to my external bridge – br-ex.

I am now going to discuss how to set up an external network for use by OpenStack tenants. Note that these are not usually used directly, but are taken by tenants as floating IP addresses, that are NATed from specific hosts in tenant networks. How floating IP addresses are specifically allocated will be discussed in a later article when I start to launch instances.

Planning the Network

There’s only a few things to consider here. Remember that the general best practice for OpenStack right now assumes that external traffic is sent to a router that is independent of the OpenStack deployment.

Hence, in a production setup, there are a couple of real-world scenarios:

  • IP space is fully managed and routed by the cloud administrator and realistically a full range will be available to assign to the external network, which can then be given out as floating IPs.
  • IP space is managed by a hosting provider that provides services to the cloud administrator, and a specific, probably smaller, range will be able to be assigned to the external network.

In both of these scenarios the setup is the same, the only thing that is really different is how the traffic is handled after it leaves the cloud, which is out of the scope of this article.

If a development server is being set up, on the other hand, sometimes other considerations need to be taken into account, such as any existing DHCP ranges that will affect the range that I give to OpenStack.

Assuming a network of

Network address		-
Network mask		-
Broadcast address	-
Addresses in network	- 256
Network range		- -
Usable range		- -

Generally, the low addresses (less than or possibly equal to 10) are reserved for network devices. Say this is an existing network as well with a DHCP range of, which cannot be encroached on. I need a significant range of addresses for floating IP addresses, and router IPs for tenant networks.

Based off of this, may be a good start.

External Network Setup in the OpenStack Web Interface

This can be done in both the CLI and also the web interface. The manuals discuss how to do it in the CLI (see here), but I will discuss the web interface, as it is perfectly capable of doing everything that the CLI can do for this task.

Head over to Admin -> System -> Networks and click Create Network;

OpenStack - External Net Creation

Make sure the admin project owns this network as it is crucial for routing. Mark the network as external, and ensure the physical network reads external as well as this is the physical network that was mapped to the external bridge in the Neutron configuration. If another flat network name was chosen for the mapping, use that name. Admin state should be up, unless you need it disabled at creation time for admin purposes.

After the network is created, I can proceed to subnet creation. Click on the newly created network and then click Create Subnet:OpenStack External Network - Subnets 1OpenStack External Networks - Subnets 2

Note that DHCP is not enabled. This is not needed for external networks and is generally kept off. Otherwise, the rest of the setup is pretty straightforward. One thing to note is how the ranges are entered: ranges are entered on separate lines, with a comma separating the first and the last IP in the range.

Name servers and host routes are generally entered on the tenant network, as those settings are added to instances.

After the subnet is created, network deployment can proceed to creation of the internal network.

Adventures in OpenStack – Networking

Note: Some inaccurate information was corrected in this article – see here for the details.

The past articles regarding Open vSwitch have kind of been a precursor to this one, because to understand how OpenStack networking worked, the concepts regarding some of the underlaying components needed to be understood for me first.

When I started looking into this last week, I really had no idea where to start. As I dug deeper, I found that this guide was probably the best in explaining the basics on how Neutron worked: Neutron in the RHEL/CentOS/Fedora deployment guide.

Neutron Network Example (Courtesy OpenStack Foundation - Apache 2.0 License)
Neutron Network Example (Courtesy OpenStack Foundation – Apache 2.0 License)

The diagram above was probably one of the tools that helped me out the most. You can see how Neutron works on both the compute and network nodes, and the role that Open vSwitch plays in the deployment at large.

Note that both GRE and VXLAN are supported for tunnels, and in fact packstack will configure your setup with VXLAN. Some features are still being developed with VXLAN, and because I haven’t delved into it too much I’m not too sure what is still missing (although one feature seems to be VLAN pruning). I really don’t have the experience to say which one is the currently the better choice as of Juno.

For now, I am focusing on the basics – what I needed to do to get my dev server set up. This entailed a few things:

  • Re-configuring my external bridge so that I could run my management interface and the “external” network on the same physical interface – see this previous article
  • Setting up neutron to map the external network to the external bridge, explicitly
  • Setting up my external and internal networks

Network Types

There are currently five network types that you can set up in OpenStack.

  • Local: I equated this to “non-routed”, but it can be used on single server setups for tenant networking. However, it cannot scale past one host.
  • Flat: A untagged direct network-to-physical mapping. This was ultimately the best choice for my external network since my requirements are not that complicated at this point in time.
  • VLAN: This is like Flat with VLAN tagging. This would, of course, allow you to run multiple segregated external networks over a single interface.
  • GRE/VXLAN: Your tunneling options. Generally used on the integration bridge to pass traffic between nodes. Best used for tenant networks.

For my setup, as I mentioned, I ultimately settled on using a flat network for our external bridge, and I haven’t touched the internal network setups (it really is not necessary at this point in time, seeing as I only have one host).

Neutron Configuration

Keep in mind that I don’t cover how to do the Open vSwitch stuff here. If you need that info see this previous article – An Intro to Open vSwitch.

With that in mind, if you are using a separate interface you can simply add it to the Open vSwitch database without much in the way of extra configuration – just run the following:

ovs-vsctl add port br-ex eth1

Assuming that eth1 is your extra interface.

On to the Neutron configuration. Generally, this is stored in /etc/neutron/plugin.ini. Note that we are using the ML2 (Modular Layer 2) plugin here, which has to be symlinked appropriately:

lrwxrwxrwx.   1 root root       37 Jan 29 23:24 plugin.ini -> /etc/neutron/plugins/ml2/ml2_conf.ini

Make sure you define the network types you will allow:

type_drivers = flat,vxlan

Pick a network type for your tenant networks, generally one is fine:

tenant_network_types = vxlan

Mechanism drivers – using Open vSwitch for now, of course. This will be set up for you by default if you are using packstack.

mechanism_drivers = openvswitch

From here I am going to skip to the changes I needed to make in my packstack setup to get the external bridge working. Most of the config that I had I left at the defaults, so if you are using packstack as well it does not need to be changed much.

The only thing left is to define your external network as a flat network:

flat_networks = external

Restarting Services

Once this is all done, you can save and restart nova and neutron services. Restarted the services below based on the node that is being updated.

# Controller Node
systemctl restart openstack-nova-api.service openstack-nova-scheduler.service \
systemctl restart neutron-server.service
# Network Node
systemctl restart neutron-openvswitch-agent.service neutron-l3-agent.service \
  neutron-dhcp-agent.service neutron-metadata-agent.service
# Compute Node
systemctl restart openstack-nova-compute.service
systemctl restart neutron-openvswitch-agent.service
# All-in-One Node
systemctl restart openstack-nova-api.service openstack-nova-scheduler.service \
  openstack-nova-conductor.service openstack-nova-compute.service
systemctl restart neutron-openvswitch-agent.service neutron-l3-agent.service \
  neutron-dhcp-agent.service neutron-metadata-agent.service neutron-server.service

Now, we are ready to set up our external and internal networks. I will cover this tomorrow in a couple of other articles!

An Intro to Open vSwitch

I’ve spent the last few days getting my bearings around Open vSwitch. It’s pretty amazing, and IMO if you are virtualizing under Linux these days, it’s pretty much a must.

So what is Open vSwitch? It’s basically the Open Source answer to other proprietary technologies, such as VMware’s distributed vSwitch (which you have probably used if you have ever used multiple servers within vSphere). It allows you to build a distributed layer-2 network entirely in software. It also performs very well under a single server setup, allowing you to build sophisticated switch fabrics under a single physical interface. It solves some frustrations surrounding network bridging under a KVM setup as well that you may have encountered, such as having a bridge that shares the same physical interface as your host’s management address.

Check out for some documentation and tutorials.

I will be explaining some basic concepts regarding Open vSwitch here, namely how to set up a bridge, attach an interface to it, and also how to automate the process using ifcfg-* files under CentOS (and by proxy, probably RHEL and Fedora as well). Also, we will discuss how to set up a VM on the bridge using libvirt.

Some Terms

bridge is a network fabric under Open vSwitch. For the purpose of this tutorial, a bridge represents a broadcast domain within the fabric at large, ie: a VLAN. Note that this does not have to be the case all the time, as it is possible to have a bridge that has ports on different VLANs, just like a physical switch.

port is a virtual switch port within the bridge. These ports are attached to interfaces, such as physical ones, virtual machine interfaces, or other bridges.

Open vSwitch Basic Bridge

Above is a very simple diagram that depicts the bridge br-ex, with ports connected to a VM’s eth0, and the host machine’s eth0.

Creating a Bridge

Run the following command to create a bridge:

ovs-vsctl add-br br-ex

This command would create a new empty bridge, br-ex. This bridge can then be addressed just like a regular interface on the system, but of course, would not do much at this point in time since we do not have any ports attached to it.

Adding a Port

ovs-vsctl add-port br-ex eth0

This would add eth0 to the bridge br-ex.

CentOS/RedHat/Fedora Interface Configuration Files

You can also have the OS set up bridges for you upon system startup – this is especially useful if you are binding IP addresses to a specific bridge. Note that any bridges that you create like this will get destroyed/re-created upon restart of the network (ie: system network restart or systemctl restart network.service).

Change your ifcfg-eth0 file to look something like this:


And create a ifcfg-br-ex interface configuration file:


Sub in your values for MAC addresses, physical interface names, and IP addresses, obviously.

Another note here that is extremely important is that you need to make sure that you use the DEVICE directive instead of the NAME directive. The latter may be left over in your physical interface configuration file from installation, so make a note to change it. I will address the exact reason why in a different article.

Setting Up a Libvirt VM to use a Bridge

Now that you have set up the above, you can add a VM to the bridge with Libvirt. Edit your domain’s (VM’s) XML file and add a block like this for every NIC you want to create:

 <interface type='bridge'>
  <mac address='52:54:00:71:b1:b6'/>
  <source bridge='br-ex'/>
  <virtualport type='openvswitch'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

This was taken directly from the Open vSwitch Libvrirt HOWTO.

Make sure of course that you assign a correct PCI ID. You may wish to create the domain first via other means, add the devices you will need, and just elect to not use network at first. Unfortunately, it does not seem that a lot of Libvirt admin tools have specific Open vSwitch support just yet (at least it does not seem that the version of virt-manager that comes with most distributions does, anyway).

Edit Feb 12 2015 – Slight correction to physical interface config file – duplicate device type.