OpenStack, Talks

Upstream Contribution – Give Up or Double Down? The Boston Edition

In continuation to a previous blog post, I presented a version with new content at the recent OpenStack Boston summit. The session had a fair amount of participation and discussions, where we talked about the journey of a new contributor in OpenStack through the lens of data gathered from Gerrit and Stackalytics. We even had members of the Technical Committee that were interested in concrete action items they could take on their part – How do we make it easier for new contributors? What behaviors work in the quest to get your patches merged quicker?

The video, slides and code used to generate the data are below:

OpenStack, Talks

Upstream Contribution – Give Up or Double Down?

Ever since I’ve been involved with OpenStack people have been complaining that upstream is hard. The number one complaint is that it takes forever to get your patches merged. I thought I’d take a look at some data and attempt to visualize it. I wrote some code that accepts an OpenStack project and a list of contributors and spits out a bunch of graphs. For example:

  • How long does it take to merge patches in a given project over time? Looking back, did governance changes affect anything?
  • Is there a correlation between the size of a patch and the length of time it takes to merge it? (Spoiler: The answer is… Kind of)
  • Looking at an author: Does the time to merge patches trend down over time?
  • Looking at the average length of time it takes to merge patches per author, how does it look like when we graph it as a function of the author’s number of patches? Reviews? Emails? Bug reports? Blueprints implemented?

The data suggestes answers for many of those questions and more.

Here’s the code for you to play with:

And some conclusions in the slides embedded below:

Here’s a few resources about effective upstream contribution. It’s all content written by and for the Neutron community but it’s applicable to any OpenStack project.


Vote for OpenStack Paris sessions!

The window to vote for the Paris summit sessions has opened. I’ve submitted two talks:

Neutron is Impossible!

“In this talk, we’ll demystify Neutron by explaining the what and the how of each major component. Additionally, we’ll describe what happens when you boot a VM in a visual way – How Nova interacts with Neutron, and the role of each Neutron component. We’ll talk about how Neutron accomplishes networking segmentation via VLANs and overlay networks. Just what are VLANs and tunneling, and how do they both solve the same problem? Finally, we’ll talk about what happens on the compute and network nodes by examining the different Open vSwitch and Linux bridges, patch ports and veth pairs, namespaces and tap devices and hopefully understand how it all comes together.

Following the talk, operators and developers should have a solid understanding of Neutron’s more puzzling concepts.”

Neutron Network Node High Availability

I’ve been working with Sylvain Afchain of Enovance on layer 3 high availability throughout the Juno cycle, and we’ve submitted a talk about it:

“Today, you can configure multiple network nodes to host DHCP and L3 agents. In the Icehouse release, load sharing is accomplished by scheduling virtual routers on L3 agents. However, if a network node or L3 agent goes down, all routers scheduled on that node will go down and connectivity will be lost. The Juno release will introduce L3 high availability as well as distributed routing.

L3 high availability schedules routers on more than a single network node. Stand-by routers use keepalive (And the VRRP protocol internally) to monitor the active router and pick up the slack in case of a failure. We’ll explain what is VRRP and how will the feature affect the network node, as well as give a demonstration.

Distributed routing, or DVR, moves routing to the compute nodes, while keeping SNAT on the network nodes.

We’ll talk about DHCP HA, go into a conceptual overview of both L3 HA and DVR, give demonstrations, and explain how to integrate them together to get the best of both worlds.”

OpenStack, Talks

Introduction to Neutron

I recently gave an internal Red Hat talk entitled: Introduction to Neutron. It is a high-level, concepts oriented talk.

In it I talk about:

  • Why Neutron?
  • An example of network virtualization
  • Ports, networks and subnets
  • External, provider and tenant networks
  • L3 model – Internal and external subnets, routers, NAT and floating IPs
  • An overview of the different Neutron components
  • Nova <–> Neutron interaction when creating a VM
  • Explanation of the core plugin concept
  • Brief rundown of the service plugins (VPN, Load balancing and Firewalls)

Here’s the PDF.

And the video:

OpenStack, oVirt, Talks

What Does Open Source Mean to Me?

I gained some development experience in various freelance projects and figured I’d apply for a development position during my last semester of Computer Science studies. I sought a student role in a large corporation so that I wouldn’t be relied upon too heavily, as I wanted to prioritize my studies (Please see ‘You have your entire life left to work’ and similar cliches). I applied to a bunch of places, including Red Hat – My would be boss gave a talk in my school about open source culture and internship positions, otherwise I would have never heard about a Linux company in a Microsoft dominated nation. Microsoft has solid contracts with the Israeli Defense Force, and with the Israeli high tech being lead mostly by ex- IDF officers, CTOs tend to go with Microsoft technology. In any case, Red Hat had an internship position in the networking team of a virtualization product (I had networking experience from my army service), paid generously, their offices were close by, it all lined up.

At this point, open source meant nothing to me.

At Red Hat, I started working on a project called oVirt. While it has an impressive user base, and its Q&A mailing list gets a healthy amount of traffic, it does not have a significant development community outside of Red Hat. Here I started experiencing the efforts that go into building an expansive open source community. Open source is not free contrary to popular belief – It is, in fact, quite costly, for a project in oVirt’s stage. For example, when working in a closed source company and designing a new feature, normally you would write a specification down, discuss it with your team members, and get going. In oVirt, you’d share the specification first with the rest of the community. The resulting discussion can take weeks, and with a time based release schedule that inherent delay must be factored in during planning. All communication must be performed on public (and archived) medias such as mailing lists and IRC channels. Hallway discussions are natural but frowned upon when it comes to feature design and other aspects of the development process that should be shared with the community. Then comes the review process. I’m a big believer in peer reviews, regardless if the project is open or closed, but surely in an open source project the review process is much better felt. One of the key elements to building a community is taking the time to review code submitted by non-Red Hatters. You could never hope to get an active development community going if code sits in the repository for weeks, attracting no attention. To this end, code review becomes part of your job description. Some people do it quite well, some people like me have a lot of room to improve. I find reviewing code infinitely harder than writing it. In fact, I find it so hard that I must force myself to do it, double so when the code is written by a faceless community member that cannot knock a basketball over my head if I don’t review his code (Dear mankind: Please don’t ever invent that technology).

At this point, open source was a burden for me.

Six months back I was moved to another project called OpenStack. Still in the same team, under the same boss, just working on another project. OpenStack, while comparable to oVirt technologically,  is very different from oVirt, in the sense that it has a huge development community. By huge, I mean thousands strong. OpenStack is composed of sub projects – The networking project alone has hundreds of developers working on it regularly. At the time I was moved I was the only Israeli developer working on it. The rest of the Red Hat OpenStack team was located in the Czech Republic and in the US. As you can imagine, a lot of self learning was to be had. Conveniently, the (community maintained) OpenStack documentation is excellent. My team mates were no longer working for the same company I was, nor were they down the hall. I did most of my work with individuals spread all over the world. I met some in FOSDEM this past February (Probably the highlight of the event for me), at which point I began to understand the importance of building personal relationships and I will expand on this below.

The beauty of open source and the basis of a meritocracy is that the strongest idea wins. You might stumble upon an infuriating bug which might seem like the most important issue facing the project (And, in fact, humanity). You start working on it, submit a patch, and quickly discover that nobody gives a shit about your bug. Instead of being frustrated by the difficulty of moving forward, I learned two lessons:

  1. Building personal relationships is the only way to drive change
  2. ‘The community’ can realign your understanding of what is important

Maybe there is good reason nobody cares about that bug. Maybe it was a waste of time working on it, not because the patch was not accepted (In time, or at all), maybe it was a waste of time because it was just a waste of time. Maybe that bug was just not important, and you should have invested your time working on anything else. There is a larger amount of issues than resources available and your choice of what to tackle is more important than the urgency of what’s in front of you.

In addition to navigating between the perceived urgency of issues, the community can help you reflect and choose the better solution. I always love hearing people’s ideas, and this concept is expressed beautifully in the review system. Getting criticism from strangers and collaborators alike always constitutes to a learning experience. Luckily OpenStack is being developed by very smart individuals that can help you understand if your solution is terrible, or simply realign your trajectory. I find that it’s sometimes even helpful to get feedback from people with opposing interests – Perhaps together you can form a solution that will answer all use cases in a generalized manner. Such a solution might just end up to be of higher quality than one that would have dealt only with your own customer’s needs.

At this point, open source is obvious to me.