Exploring Extreme Programming in Context: An Industrial Case Study Lucas Layman1, Laurie Williams1, Lynn Cunningham2 1North Carolina State University, Department of Computer Science, {lmlayma2,lawilli3}@ncsu.edu 2Clarke College, [email protected] Abstract ment for external customers to develop customized end user and business software. A longitudinal case study evaluating the effects of XP originators aimed at developing a methodology adopting the Extreme Programming (XP) methodology suitable for “object-oriented projects using teams of a was performed at Sabre Airline Solutions™. The Sabre dozen or fewer programmers in one location” [10]. Abra- team was a characteristically agile team in that they had hamsson et al. [2] contend that the XP methodology is no need to scale or re-scope XP for their project parame- “situation appropriate” in that the methodology can be ters and organizational environment. The case study adjusted to different situations. The characteristics of the compares two releases of the same product. One release Sabre project placed it in the agile home ground [5] and was completed just prior to the team’s adoption of the XP allowed the use of XP in a nearly “pure” form. methodology, and the other was completed after ap- As discussed in Section 3, several XP case studies proximately two years of XP use. Comparisons of the were performed at Sabre. To differentiate this case study new release project results to the old release project re- from other case studies performed at Sabre, we heretofore sults show a 50% increase in productivity, a 65% im- refer to the team in this study as Sabre-A (Agile). The provement in pre-release quality, and a 35% improve- Sabre-A team was among the first to use XP at Sabre Air- ment in post-release quality. These findings suggest that, line Solutions. The perceived success of this and the over time, adopting the XP process can result in in- other early XP projects led to the use of XP with over 30 creased productivity and quality. teams with more than 200 people throughout the organi- zation. 1. Introduction In our case study, we examined five null hypotheses regarding XP’s effectiveness. Because we are reporting a The introduction of Extreme Programming (XP) [4] single case study, we cannot conclusively reject or accept into mainstream software development has met with both these hypotheses. Our results add to the weight of evi- enthusiasm and skepticism. Reports both extol the virtues dence in support or in refutation of these propositions. and question the shortcomings of XP. Most often, these We triangulate upon this support or refutation via objec- reports take the form of anecdotal success stories or les- tive and subjective quantitative methods and via qualita- sons-learned from organizations that have adapted XP for tive data collection and analysis. The null hypotheses a project [15, 16, 24]. However, many organizations re- were as follows: main skeptical regarding XP’s value. For these decision- When used by teams operating within the specified con- makers, an empirical, quantitative investigation is benefi- text, the use of XP practices leads to no change in: cial for demonstrating XP’s efficacy. We increase the H10: pre-release quality (as measured by defects existing evidentiary base of empirical XP knowledge with found before product release) a detailed study of an industrial team. We present this H20: post-release quality (as measured by defects case study within the context of the XP Evaluation found by the customer after release) Framework [25, 26]. Our findings are useful for organi- H30: programmer productivity (as measured by both zations seeking scientific investigation into the real-world user stories and lines of code per person-month) impacts of utilizing XP practices. H40: customer satisfaction (measured via interview In this single, longitudinal, holistic [28] case study, we and customer feedback) examine a product created by an XP software develop- H50: team morale (assessed via a survey) ment team at Sabre Airline Solutions™ in the United The remainder of this paper is organized as follows. States. We evaluated and compared two releases of the Section 2 provides background information, and Section 3 Sabre team’s product. One release was completed just describes the context of the case study. Section 4 presents prior to the team’s initial adoption of XP; the other re- the results of the case study. Section 5 discusses the case lease was completed after two years of stabilized XP use. study limitations. Finally, Section 6 summarizes our find- This ten-person team develops a scriptable GUI environ- ings and future work. 2. Background and related work of this adoption. The necessity for common ontologies emerges from the need to exchange knowledge [19]. The In this section, we discuss the advantages and limita- XP-EF is composed of three parts: XP Context Factors tions of case study and qualitative research in software (XP-cf); XP Adherence Metrics (XP-am); and XP Out- engineering. We also discuss the Extreme Programming come Measures (XP-om). Evaluation Framework created by the authors and provide In the XP-EF, researchers and practitioners record es- a brief survey of other XP research. sential context information of their project via the XP Context Factors (XP-cf). Recording context factors such 2.1. Case study research as team size, project size, criticality, and staff experience can help explain differences in the results of applying the Case studies can be viewed as “research in the typical” methodology. The second part of the XP-EF is the XP [7, 12]. As opposed to formal experiments, which often Adherence Metrics (XP-am). The XP-am use objective have a narrow focus and an emphasis on controlling con- and subjective metrics to express concretely and compara- text variables, case studies test theories and collect data tively the extent to which a team utilizes the XP practices. through observation of a project in an unmodified setting When researchers examine multiple XP-EF case studies, [29]. However, because corporate, team, and project the XP-am also allow researchers to investigate the inter- characteristics are unique to each case study, comparisons actions and dependencies between the XP practices and and generalizations of results are difficult and are subject the extent to which the practices can be separated or to questions of internal validity [13]. Nonetheless, case eliminated. Part three of the XP-EF is the XP Outcome studies are valuable because they involve factors that Measures (XP-om), which enable one to assess the busi- staged experiments generally do not exhibit, such as ness-related results (productivity, quality, etc.) of using a scale, complexity, unpredictability, and dynamism [20]. full or partial set of XP practices. Case studies are particularly important for industrial A more detailed discussion of the XP-EF, its creation, evaluation of software engineering methods and tools rationale, and shortcomings may be found in [26]. An [12]. Researchers become more confident in a theory initial validation of the XP-EF may be found in [26]; fur- when similar findings emerge in different contexts [12]. ther work in this area is ongoing. Instructions and tem- By recording the context variables of multiple case stud- plates for measuring and reporting an XP case study data ies and/or experiments, researchers can build evidence via XP-EF Version 1.3 have been documented by the au- through a family of experiments. Replication of case thors of this paper [25, 26]. studies addresses threats to experimental validity [3]. 2.4. XP studies 2.2. Qualitative research Practitioners and researchers have reported numerous, Qualitative methods can be used to enrich quantitative predominantly anecdotal and favorable studies of XP. A findings with explanatory information that helps to under- number of these reports discuss the use of XP with small, stand “why” and to handle the complexities of issues in- co-located teams. Wood and Kleb [27] formed a two- volving human behavior. Seaman [23] discusses methods person XP team and analyzed the productivity of their for collecting qualitative data for software engineering project as part of a pilot study at NASA to assess XP in a studies. One such method is interviewing. Interviews are mission-critical environment. When the project results used to collect historical data from the memories of inter- were normalized with past comparable projects, the XP viewees, to collect opinions or impressions, or to under- approach was approximately twice as productive. stand specific terminology. Interviews can be structured, Abrahamsson [1] conducted a controlled case study of unstructured, or semi-structured [23]. Semi-structured four software engineers using XP on a data management interviews, as were conducted in this case study, are a project at a Finnish research institute. The project lasted mixture of open-ended and specific questions designed to eight weeks with a fixed development schedule and fixed elicit unexpected types of information. resources. Comparison between the first and second re- leases yielded the following results: planning estimation 2.3. Extreme Programming Evaluation Frame- accuracy improved by 26%, productivity increased by 12 work lines of code (LOC)/hour, and the defect rate remained constant at 2.1 defects/thousand lines of code. Similarly, Maurer and Martel [17] reported a case study of a nine- The Extreme Programming Evaluation Framework programmer web application project. The team showed (XP-EF) is an ontology-based benchmark for expressing strong productivity gains after switching from a docu- case study information [25]. The XP-EF records the con- ment-centric development process to XP. text of the case study, the extent to which an organization Reifer reported the results of a survey of 14 firms has adopted and/or modified XP practices, and the result spanning 31 projects [21].