A Roundtable with Three Release Engineers

FOCUS: GUEST EDITORS’ INTRODUCTION

The Practice and Future of Release Engineering A Roundtable with Three Release Engineers

Bram Adams, Polytechnique Montréal // Stephany Bellomo, Software Engineering Institute // Christian Bird, Microsoft Research // Tamara Marshall-Keim, Software Engineering Institute // Foutse Khomh, Polytechnique Montréal // Kim Moir, Mozilla

RELEASE ENGINEERING focuses manager; and Kim Moir, a release us to gauge the overall release health on building a pipeline that transforms engineer. We asked each of them of a service; the third measure can source code into an integrated, com- the same questions covering topics uncover issues in the testing pipe- piled, packaged, tested, and signed such as release-engineering metrics, line or growing code complexity. We product that’s ready for release. The continuous delivery’s bene ts and track these metrics and make com- pipeline’s input is the source code de- limitations, the required job skills, parisons from quarter to quarter. velopers write to create a product or the required changes in education, Related to testing, another metric modify an existing one. Enterprises and recommendations for future is the greenness of the testing pipe- running large-scale websites and de- research. line. Many tests, from code to perfor- livering mobile applications with mil- mance tests, are run daily in a con- lions of users must rely on a robust Every product release must meet an tinuous fashion. Stability of tests is a release pipeline to ensure they can expected level of quality, and release signal of product maturity and good deliver and update their products to processes undergo continual ne- engineering practices. Despite some new and existing customers, at the tuning. What metrics do you use to arguments to the contrary, this mea- required release cadence. monitor a release’s quality? Do you sure effectively increases the velocity This special issue provides an roll back broken releases after de- of product development and release. overview of research and practi- ployment? If so, how? We also track a host of more ne- tioner experience, and this article in grained metrics. Every step—with particular aims to give you insight Debic: Our main measures are its duration, outcome, operation, into the state of the practice and the threefold: the number of open bugs logs, arguments, and other relevant challenges release engineers face. It ranked by priority, the number and details in execution and setup—is features highlights from interviews percentage of successful releases, and logged for every release that runs with Boris Debic, a privacy engi- the number and percentage of re- at our company. Re nements in the neer (and former release engineer); leases that are abandoned late in the release system are direct results of Chuck Rossi, a release-engineering game. The rst two measures allow observing patterns and quantifying

s2gei.indd 42 2/4/15 6:33 PM different processes, tools, and ap- amount of tagging. We compare the that the front end is rendering so proaches using this dataset. growth and interaction metrics from slowly because I’ve lost half the To gain another perspective, the canary to those from production. back-end machines that are provid- we have systems that interact with A release engineer and the develop- ing data for this service. I wouldn’t our users, either by providing them ers look at them with more detailed have found that internally, but I will a way to give direct feedback or dashboards. For example, the ad nd it in canary because it is mil- by going through logs and look- teams have dashboards on ad dis- lions of people. ing for different types of failures. plays and ad click-throughs. Mobile deployments are more This data is distilled and presented Our alerting system works on challenging than Web deployments to product teams as a collection of either absolute numbers or the per- because we don’t own the ecosystem, signals that speak of product ro- centage rate. The biggest alert for so we can’t do all the things that we bustness and of complaints that users mention most often. For Web services and servers, For Web services and servers, “canarying” is another key component of successful releases. Canary “canarying” is a key component of rollout strategies depend on the type successful releases. of service, user expectations, and contractual obligations. In this type of rollout, we gradually increase the exposure of the new binary and the Web is the log data for each new would normally do. And the canar- at all times monitor the critical pa- build. In a canary, we collect that log ies are huge. We watch cold start, rameters. Canaries are the bread and data separately from the regular pro- warm start, the app size, and the butter of the nal stages of a well- duction traf c. A website has thou- numbers of photos uploaded, com- designed release process. sands of ring errors and warnings, ments, and ads being displayed or and we look for changes in those. An clicked. Growth and engagement Rossi: I’ll talk about Web deploy- analysis of errors in the log data that numbers and the crash rate are im- ment rst and then contrast it with differ from production is the rst portant to the company. If the crash mobile deployment. For Web de- part of the canary. That’s easy, and rate uctuates, we immediately take ployment, we use the metrics of the it’s universal—it doesn’t matter what action to understand why. code going into the master branch, your app is doing. Concerning rollback, we’ve never the test results, and performance lab Another big alert is when the ca- had a canary that bad. Generally, results. The next level includes met- nary TTI is much higher than the it’s always rolling forward. We’ll rics for products being released, such production TTI. Is it because we just promote the release candidate to as core tests, unit tests, and perfor- increased login calls by 10 times? Or the production binary in our store, mance experiments like time to in- because a database call to render the roll it to 5 percent of users, and get teraction (TTI), fatal-error rates, the rst page is not going through cache data back from that. If that 5 per- number of errors per page, and any and it’s trying to do a lookup ev- cent looks good, we’ll roll it out to new errors that we hadn’t seen in the ery time? TTI helps us esh out the the rest of the population. I always production logs. problem. We pay particular atten- make the analogy that it’s like a bul- Then comes the canary step. The tion to how long it takes the main let from a gun. It just keeps going. set of binaries for a release sit in the page to render. The mobile ecosystem is so broken canary state for 30 minutes to an I have graphs for the back-end when it comes to software man- hour. I look at the logging and ag machines, but I look for effects on agement that I don’t want to force new errors, error rate changes, and the front end. When I see those ef- people to re-download. Every time I fatal or elevated error rates for an fects, I’ll start digging down. I might have to ask them to re-download, I existing error. Core metrics include see, for example, that only Internet lose a certain percentage of people TTI; the number of likes, photos Explorer 7 on Windows boxes is who just never do it. So that’s the uploaded, and comments; and the showing a bad TTI. Or I’ll realize challenge.

MARCH/APRIL 2015 | IEEE SOFTWARE 43

s2gei.indd 43 2/4/15 6:33 PM FOCUS: GUEST EDITORS’ INTRODUCTION

Moir: At Mozilla, release engineers Concerning rollback, we don’t Continuous deployment obvi- don’t monitor the quality of the re- really roll releases back. If there ously shines in the Web area, where lease; we have a team called Release were a serious problem, like a huge you own the ecosystem. You can Management to perform that func- number of crashes on a certain re- publish effortlessly to your Web tion. We use a “train model” for lease, we would block it so that no fleet, and your users get the fixes managing releases. When develop- updates would occur and then do a and features instantly without notic- ers have a new feature, they’ll land point release. For example, if there ing it. In minutes or hours, you will it on a certain branch and make were a security issue causing prob- know whether something is wrong sure the test suites run green. If so, lems, we would do a point release with the release. the change set will be uplifted to so that users wouldn’t get the last As I understand continuous deliv- another branch to ensure that the release and would be automati- ery, it will not happen on mobile in patch integrates with other changes cally updated to the newer release the near future. The current app dis- on that branch and tests run green. with the security fix. We call this a tribution system is based on an an- Eventually, the new feature reaches “zero-day fix.” cient model that’s not even as good the Aurora branch, which is an al- as shrink-wrapped software. In this pha branch, where it will sit for six Amid all the hype and buzz about model, you build an artifact, you weeks to bake; then it goes to the continuous delivery, what’s cur- put it out to a third party that has beta branch. Finally, six weeks later rently possible, and what are the total control over when and how it it goes to the release branch. This is limitations? How far should you go gets out, then the end users consti- one way of ensuring stability. with continuous delivery? tute a completely disparate map of if, We limit the number of people when, and how it gets updated. And who get a release. On a given re- Rossi: I’ve never worked in a true there is no way for you to influence lease day, we might let 5 percent of continuous-deployment environ- that ecosystem. the population running the desktop ment. We have a pseudo-continuous So, you need user interaction version of our browser get the new deployment here—it’s twice a day. for every single update, and that’s release. We have automatic crash Size is the limiting factor. All the insane. Why should I have to take reporting in the browser that re- continuous-deployment places I’ve time out of every day for the rest of my life to push a button and have my phone update its apps? But that’s the model that we’ve had Mobile deployments are more with iOS. iOS 7 has an easy way challenging than Web deployments to turn on automatic app updates. because we don’t own the ecosystem. Then you’re not seeing that double- digit red number on your App Store icon every day. Unfortunately, though, this feature is not on by ports to databases here. How many visited had engineering teams of 20 default. Android will put up road- crashes occurred? Are certain op- to 50, even 100 people, pushing to blocks even if you have auto-update erating systems, platforms, or add- a website with a number of users turned on. Of course, the owners ons having problems? We’ll analyze at best in the double-digit millions of the platforms have valid reasons answers to those questions to deter- per day. The same processes don’t for trying to maintain this control, mine whether we can roll the release scale above a few hundred develop- such as preventing malicious apps out to the rest of the population. ers working on a common code base from auto-installing. Other metrics come from users who or to a website that has either more Our company, which has both give us feedback during the beta, complexity or users into the hun- good infrastructure and complex support requests on our support dreds of millions. It doesn’t scale apps, can do automatic updates. In website, sentiment analysis on Twit- at our company’s size. Continuous fact, any mobile developer could pro- ter, and the top 10 crashes across our deployment works for small teams, vide users the infrastructure to use continuous integration every week. with 20 to 30 changes per day. their own channels to update apps.

44 IEEE SOFTWARE | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE

s2gei.indd 44 2/4/15 6:33 PM Software deployment profession- getting the product in place. The re- transforming code into a form that’s als need a solution that addresses the lease engineer doesn’t get upset if you tested, deployed, signed, and repro- security concerns but lets us update say, “You can’t put that feature in be- ducible. How do you educate others our valid, legitimate apps seamlessly cause it’s going to break everything, about the value that release engi- with no pain to the user. and we need to ship tomorrow.” As a neering brings to a team? release engineer, your focus is getting Moir: I think the continuous-delivery a stable release out the door. Moir: In my current environment, model for desktop software works release engineering is definitely well well if the updates are silent and us- Debic: The possibilities and limita- received. Because if you can’t build, ers don’t get constantly notified about tions of continuous delivery depend we can’t ship. And if we can’t ship, them. Otherwise, they get annoyed. on the type of deliverable. Is it a we don’t get paid. In other compa- As Chuck said, the mobile model obviously is different because Google and Apple own the distribution, and A company needs to get the default behaviors require users to update as they feel like it. a release engineer on board At our company, we are focused in the early stage, not the later stage. on relentlessly automating everything. We’re automating the uplift of all the changes from beta to release or from Aurora to beta, to have Web service, a mobile application, nies, I’ve seen that release engineers fewer manual steps. We’ve come a or software for a medical device or are second-class citizens, or they are long way from when we first started aircraft autopilot? In the high-tech expected to perform miracles with releasing software, and we had a business, the Holy Grail of release no advance warning or resources. big page of instructions to follow by engineering is something called In those cases, obviously some hand, which was not very efficient. “push-on-green.” As soon as a de- education is necessary. And maybe it Now there’s a great deal of deep veloper has committed a change list stems from the fact that release en- knowledge about how everything to the code base, it automatically gineering is not taught in school as works, so that when something goes gets into a pipeline that tests, exe- a discipline, so people aren’t exposed wrong, we can fix it. This lets us fo- cutes, and canaries the change list. to it. I like to help spread the word cus on writing tools to improve our If all of the elements pass the change by writing about release engineering continuous-integration farm and our list, it goes into production. Push- on my blog. And I’ve helped orga- release automation. on-green does not always make nize workshops for release engineer- If a company is thinking about sense: a change list may be depen- ing to try to bring the community moving to continuous integration, dent on a set of change lists. There together. it needs to get a release engineer on may be dependencies between func- In his book about remote work, board in the early stage, not the later tional parts of the product or be- A Year without Pants, Scott Berkun stage. Sometimes, product teams tween services. It may be impossible writes about his time as a manager at work on a product almost in secret to immediately deploy the change— Automattic, which developed Word- and throw it over the fence when think of mobile devices that have a Press.com. At Automattic, all new they’re done. Then, release engineers wholly different model than a ser- hires spend a few weeks in a support want to run away screaming when vice in a datacenter. Users may not role, which gives them a better un- they see that the product is built on want to interrupt their days, or the derstanding of customer issues and a hacky pile of spaghetti, and they change may not be compatible with the overall process to get software have to fix it. all devices. out the door. It’s also good to have someone DevOps (development and op- who’s not emotionally attached to Often, people unfamiliar with re- erations) is another practice that the code and who is focused on get- lease engineering don’t under- breaks down the walls between op- ting the pipeline in place as well as stand the inherent complexity of erations and development. If you

MARCH/APRIL 2015 | IEEE SOFTWARE 45

s2gei.indd 45 2/4/15 6:33 PM FOCUS: GUEST EDITORS’ INTRODUCTION

give developers the responsibility not in the world. My wife makes fun of as a discipline. Given this limita- only to land code but also to make me because I come home and tell her, tion, how do you find good release sure that it actually works in produc- “Oh, you know, Sylvia, three recruit- engineers to hire? Should curricula tion and that the customer is happy, ers contacted me today. I’m thinking change to include these skills? If so, they become more aware of the I’m hot stuff.” And she says, “Yes, what courses would be essential? whole pipeline of moving software I told you no one wants to do what from development to the customer. you do.” Debic: This is a very good question. And if something does not work, de- There’s a conception that release Release engineering is not taught; it’s velopers are involved with backing it engineering is nasty work. When I often not even mentioned in courses out and writing patches to fix it. started doing this, my first job was where it should be mentioned. I think with IBM in 1988. I worked on the the main reason is that the release- Debic: If I need to describe release release of an integration of two op- engineering practice itself has been engineering to colleagues who do not erating systems. This is really com- hard to define. As you see from the know my work from firsthand inter- plex—a huge software project with answers of your other guests, the ap- actions, I tell them that release en- two massive things that intersect, proaches are quite diverse in nature gineering is the difference between and it has to be reproducible, re- and scope. manufacturing software in small peatable, and testable. No one gets But perhaps we should not have teams or startups and manufacturing this exposure until they’re dropped skills-based curricula for release en- software in an industrial way that is in the middle of it and have to react gineering anyway. At Google, release repeatable, gives predictable results, to it. And you’ll find many good re- engineers are software engineers; and scales well. These industrial- lease engineers who are release engi- there is no difference. The complex- style practices not only contribute to neers now because they started in a ity of work they do and the tools the growth of a company but also are company where no one would do it. they use are the same as for product key factors in enabling growth. That’s traditionally how people have engineers. Certain establishments A release engineer has a special fallen into this role. treat release engineers and quality mind-set. We look at everything that I don’t think you have to make the assurance engineers differently, but is going on in a tech company, and we value proposition to any company of this is a short-sighted strategy, and try to industrialize the process. Where why you want someone doing release my company is proof that the oppo- others see features, we see release chal- engineering, especially since there has site works better. lenges. Where others count change been movement in the continuous- lists, we count how long it took for a integration and continuous-delivery Rossi: I’ve spent some time trying to work through this both at the university level to get the curriculum lined up and at a personal level. If you can’t build, we can’t ship. One of my biggest hiring concerns is that I need to hire release engineers. And if we can’t ship, we don’t get paid. It’s like finding unicorns. I look for a strong technical background and experience with programming on either the product side or the infra- change from the time it was submit- worlds in the past. I’ve talked to structure side. I want utilitarian pro- ted until the time it was in front of the small startups, and generally it’s not grammers and people who get stuff customer. While others add people to one of the first things they’re worried done in the realm of system admin- a project, we look at how the added about. But once they start to grow, istration or tool writers. I don’t need complexity will affect it. they begin to look for a person to do top-notch C++ programmers or peo- release work. ple concerned with the delicacies of Rossi: I’ve always maintained that if optimizing C algorithms. you’re a good release engineer, you Universities and colleges don’t ex- The next thing I want is archi- can work for any software company plicitly teach release engineering tecture knowledge. I want people to

46 IEEE SOFTWARE | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE

s2gei.indd 46 2/4/15 6:33 PM understand large-scale, multitiered, infrastructure. I think the expecta- would love to know—given x number distributed systems well enough to tion is that students will learn the of platforms, y number of branches, debug or get into a system and see practical aspects later. and the matrix of tests and builds we where it is falling over. If you’re re- It would be great if schools taught run on each of these platforms—if lease engineering for the full stack, release-engineering skills, but what we increase our number of commits, you’re pushing everything from the classes would you remove from a how much additional capacity will databases, the back-end systems, the computer science curriculum to ac- we need within the next year? caching layer, the front end, the Web commodate this? Still, some top- We could also model large servers, and everything in between. ics that I would like to see in a continuous-integration farms for the You need to know how it all works if you’re the one rolling it out. Then I look for release engineering proper. Release engineers un- Release engineers are hard to find, derstand where there’s risk in mak- and one problem is that they ing things reproducible, repeatable, don’t teach the skill set in school. and able to go back to any state of what was built. Experience tells them when you don’t make this kind of change at this point in the cycle course are version control systems, possible effects of reducing the num- because it’s too great a risk—that’s like cloning, branching, and merg- ber of tests run. We could use an al- hard to teach. Release engineers are ing; bug-tracking systems; writing gorithmic model that you can plug familiar with the source control sys- patches and testing them against in and enter parameters such as the tems, and they can do surgery on existing code bases; and interacting type of machines running, the envi- trees and branches; flatten conflicts; with people. Other skills would be ronment, the number of builds, and and safely deploy a new binary, how to maintain continuous-integra- other constraints. And the model drain existing connections, and tion deployment and infrastructure would show where your limits on ca- bring up new services seamlessly. and how to set up a release pipeline. pacity might be. If people come to me from a job Case studies of how large compa- Another issue is high pending where they had maintenance win- nies do release engineering would counts. We have a lot of jobs wait- dows for rollouts, that’s a joke. I’m be useful. Continuous Delivery, by ing for machines, and it seems like not going to take you seriously if Jez Humble and David Harley, could we’re always playing Whac-A-Mole you’re from a context where you be an excellent textbook for such a on the bottleneck. We run almost all can’t use your bank between mid- class. tests on all commits, but do we re- night and 3 a.m. because it is down ally need to go to that effort and ex- for maintenance. That’s just not ac- What should researchers focus on pense? How can we break out only ceptable from a release-engineering regarding release engineering, con- the relevant tests that need to be run standpoint. tinuous delivery, and related topics? on a given commit so that we use Where can research contribute to our capacity more efficiently? Some Moir: Release engineers are hard to problems you see with release engi- dependence analysis tools would be find, and one problem is that they neering or continuous delivery? useful, to trace through the code, don’t teach the skill set in school. map code changes, test coverage, I recently looked at the undergrad Moir: One thing I struggle with is and thus invoke only the relevant classes required to graduate with how to model the capacity I need tests for that code. a computer science degree from a for our continuous-integration farm. major university, and I was struck For instance, yesterday we ran 3,200 Rossi: This doesn’t really apply in by how much of it was theory and build jobs and 74,000 test jobs, and a pure continuous-delivery situ- not much was practice and deploy- each test job ran performance tests ation, but in a near-continuous- ing code. In most computer science or correctness tests. It’s an active deployment system, I would like to programs, there is little emphasis on and complicated environment, and I know the velocity of code change

MARCH/APRIL 2015 | IEEE SOFTWARE 47

s2gei.indd 47 2/4/15 6:33 PM FOCUS: GUEST EDITORS’ INTRODUCTION

over time. Does the change rate nar- and the number of qualified profes- a predictable schedule. Company row down to a point, or does it ramp sionals is growing. What can we do culture regarding release engineer- up as the date gets closer? Does the about this gap? People are trying dif- ing’s importance, infrastructure and defect rate increase or decrease as we ferent approaches. My colleague Pe- tooling investment, and commit- get to that end point? Because, if I do ter Norvig is working on expanding ment to continuous delivery varies analysis on my cycles, and I see that the workforce through online educa- widely among enterprises. Similarly, two or three days before the final re- tion. MOOCs (massive open online the scope of a release engineer’s role lease I’m getting a twofold increase courses) are available, and people depends on where she or he works, in the number of changes, this indi- are taking advantage of this new, un- the number of products to build, the operating systems and platforms on which they’re deployed, and the release cadence. This roundtable raises How can we break out only many interesting areas for research the relevant tests that need and for improving education to en- to be run on a given commit? sure that future software developers better appreciate the scope and challenge of release engineering. The seven articles in this special cates risk. And risk has increased at precedented channel. Ray Kurzweil issue benefit developers in two ways. the worst possible time, at the end of is more pragmatic. He is building a The first group of articles reports on the cycle. As a release engineer, you computer—an AI, really—that pro- the experience of companies who mi- always feel like you’re cramming in grams itself. And my colleague Sinisa grated toward rapid or even continu- stuff at the last minute, when you’re Srbljic thinks that the best way for- ous release schedules. In “Continuous trying to have time to settle, let the ward is to build a platform that con- Delivery: Huge Benefits, but Chal- metrics come in, and get what we sumers can use to customize applica- lenges Too,” Lianping Chen discusses call our “soak period.” But we of- tions by themselves, without formal the benefits and challenges of contin- ten can’t do it because we’re taking knowledge of computer science. uous delivery at Paddy Power. Martin changes right up to the moment that If a system is well engineered, it Michlmayr and his colleagues inves- we release. Is it really always this should be adaptable to its environ- tigate release planning’s importance mad dash at the end? ment and perhaps even learn from for open source systems in “Why The development cycle would it. Right now, too much software and How Should Open Source Proj- make an excellent subject for anal- change happens as a result of hu- ects Adopt Time-Based Releases?” In ysis too. What is the effect on code mans banging on keyboards, and “The Highways and Country Roads delivery and the defect rate of two-, then we have to release all of that. to Continuous Deployment,” Marko four-, or six-week cycles? This is very We are running out of programmers, Leppänen and his colleagues exam- relevant to mobile. Does a quicker so software in the long term will ine Finnish industry’s adoption of release cycle in mobile produce bet- have to be either more adaptable by continuous deployment. “Achieving ter and less buggy products? design or written in such a way that Reliable High-Frequency Releases consumers can change and adapt in Cloud Environments,” by Liming Debic: An escalating number of com- it. This would change our model of Zhu and his colleagues, discusses puter software applications, systems, computing to include consumers as reliability issues related to high- and home-electronics products are also modifiers, creators, contribu- frequency releases in the cloud. permeating all industries. This re- tors, and editors of software. The second group of articles fo- sults in an exponential growth of cuses on release engineering’s specific programmable entities. On the other challenges. “Release Stabilization on side, we have the output of computer elease engineering is a com- Linux and Chrome,” by Md Tajmilur science schools, which is growing lin- plex field with many ap- Rahman and Peter Rigby, reports on early and slowly. The gap between R proaches to ensuring that an empirical study of the time and ef- the work to support this growth quality software can be released on fort involved in release stabilization

48 IEEE SOFTWARE | WWW.COMPUTER.ORG/SOFTWARE | @IEEESOFTWARE

s2gei.indd 48 2/4/15 6:33 PM on Linux and Chrome, whereas “Rapid Releases and Patch Backouts: A Software Analytics Approach,” by BRAM ADAMS is an assistant professor at Polytechnique Rodrigo Souza and his colleagues, Montréal, where he heads the Lab on Maintenance, Construc- examines how the release process tion, and Intelligence of Software. His research interests include software release engineering in general, as well as software changed when Mozilla transitioned integration and software build systems in particular. Adams to rapid releases. Finally, Jonathan received a PhD in computer science engineering from Ghent Bell and his colleagues propose ap- University. Contact him at [email protected]. proaches to speed up testing of Java projects in “Vroom: Faster Build Pro- STEPHANY BELLOMO is a senior member of the technical cesses for Java.” We hope these ar- staff at the Carnegie Mellon University Software Engineering ticles convey an idea about the state Institute. Her research interests include incremental software development, the architectural implications of DevOps, and of the practice and the challenges of continuous integration and delivery. Bellomo received an MS in release engineering today. software engineering from George Mason University. Contact her at [email protected]. ABOUT THE AUTHORS THE ABOUT

CHRISTIAN BIRD is a researcher at Microsoft Research in Redmond, Washington. His main research interest is empirical software engineering, predominantly examining collaboration See www.computer.org/ and coordination in large software teams in both industrial and software-multimedia open source contexts. Bird received a PhD in computer science for multimedia content from the University of California at Davis under advisor Prem related to this article. Devanbu. Contact him at [email protected].

TAMARA MARSHALL-KEIM is a senior editor at the Carnegie Mellon University Software Engineering Institute. Her research interests are applied linguistics, theory of rhetoric, and computer studies in language and literature. Marshall-Keim received an MA in English from the University of Florida. Contact her at [email protected].

FOUTSE KHOMH is an assistant professor at Polytechnique Montréal, where he leads the SWAT (Software Analytics and Technologies Lab) team on software analytics and cloud- engineering research. His research interests include software maintenance and evolution, cloud engineering, service-centric software engineering, empirical software engineering, and software analytics. Khomh received a PhD in computer science from the University of Montreal. Contact him at foutse.khomh@ FIND US ON polymtl.ca. FACEBOOK KIM MOIR is a release engineer at a Mozilla. Her research & TWITTER! interests are build optimization, scaling large infrastructure, and optimizing build and release pipelines. Moir received a Bachelor facebook.com/ of Business Administration from Acadia University. Contact her ieeesoftware at [email protected].

twitter.com/ ieeesoftware

MARCH/APRIL 2015 | IEEE SOFTWARE 49

s2gei.indd 49 2/4/15 6:34 PM