When is it not cool to add a new OpenStack configuration option?

Adding new configuration options has a cost, and makes already complex projects (Hi Neutron!) even more so. Double so when we speak of architecture choices, it means that we have to test and document all permutations. Of course, we don’t always do that, nor do we test all interactions between deployment options and other advanced features, leaving users with fun surprises. With some projects seeing an increased rotation of contributors, we’re seeing wastelands of unmaintained code behind left behind, increasing the importance of being strategic about introducing new complexity.

I categorize the introduction of new OpenStack configuration options to two:

  1. There’s two or more classes of operators that have legitimate use cases and a configuration option would enable those use cases without hurting cloud interoperability
    • For example: Neutron DVR, is essentially a driver for router implementations, that changes your L3 architecture substantially while abstracting details via an API. DVR has various costs and benefits and letting operators make that choice, especially as the feature matures, makes sense to me.
  2. Developers that don’t fully understand the choice they are making and pass the complexity down to the operators to figure out. This results in options that are often never changed from their defaults because operators don’t have access to sufficient documentation that explains the rationale for choosing a specific value, as well as a misuse of time and energy because developers sometimes focus on use cases that are off center or don’t exist.
    • For example: neutron.conf:DEFAULT:send_events_interval: Number of seconds between sending events to nova if there are any events to send. Unless you grep through the code, how is an operator supposed to know if they should increase or decrease the value? Even then, shouldn’t developers take responsibility of that choice, and test for the best value? If problems are found under load, shouldn’t the value be calculated as a function of some variable? Instead of distributing the work to thousands of operators, wouldn’t we as a community like to to do the work in one place?

When contemplating adding a new option, ask yourself:

  1. Is it possible that you don’t fully understand the use case, and in lieu of making a choice, you’re letting the operator bear the burden?
  2. Are you, and your replacement, prepared to own the cost of the new option indefinitely?
OpenStack, Talks

Upstream Contribution – Give Up or Double Down? The Boston Edition

In continuation to a previous blog post, I presented a version with new content at the recent OpenStack Boston summit. The session had a fair amount of participation and discussions, where we talked about the journey of a new contributor in OpenStack through the lens of data gathered from Gerrit and Stackalytics. We even had members of the Technical Committee that were interested in concrete action items they could take on their part – How do we make it easier for new contributors? What behaviors work in the quest to get your patches merged quicker?

The video, slides and code used to generate the data are below:


Python Appreciation Day

Every once in a while I have to recognize the beauty that is a high level language.

Consider this code snippet:

def square(x):
 return x**2

l = [1, 2, 3, 4]
for x in map(square, l):

And this one:

from multiprocessing import Pool

def square(x):
 return x**2

l = [1, 2, 3, 4]
for x in Pool().map(square, l):

Discounting the import line (Because I’m trying to make a point here, dammit!), the diff is a total of 7 characters. Of course, if you recall your Computer Science operating systems 101, the complexity that is hidden here is breath taking. And yet, Python multiprocessing exposes an API that is identical to Python’s good old ‘map’. It does so and manages to keep it boring, without exposing the caller to messy internals and endless complications that are usually involved whenever we dive in to the world of multi processing.

Even more impressive is the fact that I exploited this abstraction in a real world app that does honest to Zeus useful things, and it “just worked”. The processing time was cut by half and I didn’t need to worry about forking, sockets, PIDs, creating and managing processes or any of that mess. All I had to do was focus on the subject matter and not the underlying code. That is the beauty of a high level language. You don’t have to use up cognitive load thinking about memory management, the syntax of templates meta programming, or the depth of pointer dereferencing required, instead you can focus on getting something useful done.

* Herein lies a vim vs. Emacs type disclaimer: Every tool has its place. I would never dream of using Python for a demanding mobile game (Or a cloud operating system, for that matter), but wow is Python fun when you’re parsing and processing tens of thousands of JSON files.


The OpenStack PTL Elections are Happening!

Tomorrow morning, February 1st, OpenStack contributors will receive emails to elect their new PTLs. OpenStack Networking is having an interesting election this cycle. Please, have some fun with your morning coffee, spend a few minutes and read Ihar and Kevin’s nomination emails.



Both are incredible developers and some of the finest Engineers I’ve had the pleasure of working with. I have immense respect for both, which if anything makes this a fascinating spectator sport.

Best of luck to both candidates, and as Gandalf so eloquently put in those X-Men movies of his: “May the force live long and prosper, Harry.”


Do you want to work on OpenStack networking?

The position has been filled, thank you for all of the applications!

I’m looking for a Software Engineer to join the Red Hat OpenStack Networking team. I am presently looking to hire in Europe and Israel. The candidate may work from home or from one of the offices listed here. The team is globally distributed and comprised of talented, autonomous, empowered and passionate individuals with a healthy work/life balance. The candidate will work on OpenStack networking projects such as Neutron and LBaaS. The candidate will write and review code while working with upstream community members and fellow Red Hatters. If you want to do open source, Red Hat is objectively where it’s at. We have an institutional culture of open source at all levels and this has a ripple effect on your day to day and your career at the company.

Please email me CVs at

The ideal candidate is familiar with some or all of the following subjects:

  • Python
  • Networking knowledge and terms such as L2, L3, ARP and IP
  • Cloud networking knowledge. For example VXLAN tunneling, network namespaces and OVS
  • Familiarity with virtualization technology, cloud and infrastructure as a service and OpenStack in particular


  • Write a bunch o’ code
  • Review code
  • Resolve bugs
  • Draft design documents
  • Implement features
  • Lead and participate in design discussions
  • Attend conferences
  • Improve our testing infrastructure
  • Assist in RPM packaging
  • Resolve customer issues

Required skills:

  • Bachelors Degree in Computer Science or equivalent
  • 3 years of significant software development experience
  • Excellent English verbal and written communication skills
  • Comfortable communicating and collaborating with upstream community members outside of the team and company



Is OpenStack Neutron ML2/OVS Production Ready for Large Scale Deployments?

One of my personal highlights of the recent Barcelona Summit was a session by Mirantis engineers Elena and Oleg titled “Is OpenStack Neutron Production Ready for Large Scale Deployments?”. In the session they outline a comprehensive control and data plane testing effort, run on two labs, one with 200 nodes and run of the mill hardware, and the other with 378 and top of the line hardware, all running the Mirantis distribution based off Mitaka with standard ML2/OVS, DVR, L2POP and VXLAN. In the session they show near line-rate speed for east/west and north/south routing with jumbo frames and VXLAN offload enabled. They were also able to spawn 24,500 VMs across 125 networks without errors and low CPU consumption.

Slides on SlideShare

Turning our eyes to adoption, the OpenStack Foundation conducts a usage survey every 6 months. Looking at the April 2016 user survey, we can see that ML2 with Open vSwitch and Linux Bridge dwarf other solutions.

neutron-plugins-breakdown, slide 41, taking out “ML2” and “nova-network”.

Examining the openstack/neutron project via Stackalytics we see that ML2/OVS has a rich and robust community with 20 companies contributing over 5 patches in the Newton time frame. 779 people have contributed at least 1 patch to Neutron since its inception, 215 of which during the Newton timeframe. Some of the effort targeted the base Neutron platform, e.g. configuration options, database work, versioned objects, quotas or other in-tree ML2 drivers such as SRIOV. Looking at the contribution of everyone who has committed at least 5 patches in the Newton cycle, we are left with 50 authors, 42 of which contributed at least 1 patch to ML2/OVS.


Taken from after filtering authors with less than 5 patches

Looking at the interactive version of the OpenStack user survey we can see that ML2/OVS is the most popular choice by an order of magnitude regardless of deployment size. And so to answer the question: “Is OpenStack Neutron ML2/OVS Production Ready for Large Scale Deployments?”. Yes, it is, of course it is. It has been for some time now.

OpenStack, Talks

Upstream Contribution – Give Up or Double Down?

Ever since I’ve been involved with OpenStack people have been complaining that upstream is hard. The number one complaint is that it takes forever to get your patches merged. I thought I’d take a look at some data and attempt to visualize it. I wrote some code that accepts an OpenStack project and a list of contributors and spits out a bunch of graphs. For example:

  • How long does it take to merge patches in a given project over time? Looking back, did governance changes affect anything?
  • Is there a correlation between the size of a patch and the length of time it takes to merge it? (Spoiler: The answer is… Kind of)
  • Looking at an author: Does the time to merge patches trend down over time?
  • Looking at the average length of time it takes to merge patches per author, how does it look like when we graph it as a function of the author’s number of patches? Reviews? Emails? Bug reports? Blueprints implemented?

The data suggestes answers for many of those questions and more.

Here’s the code for you to play with:

And some conclusions in the slides embedded below:

Here’s a few resources about effective upstream contribution. It’s all content written by and for the Neutron community but it’s applicable to any OpenStack project.