Exploring the Relationship between Project Selection and Requirements Analysis: An Empirical Study of the New Millennium Program Mark Bergman Gloria Mark Information and Computer Science Dept. Information and Computer Science Dept. University of California, Irvine University of California, Irvine [email protected] [email protected] Abstract quantitative economic perspective, i.e. determining the most important decision factors and using them in a The relationship between project selection and model to inform the decision, e.g. [3-5]. Although the requirements analysis is important, yet has not received needs and the shareholders are somewhat addressed in much attention. The decisions made during project these models, the details of their requirements are often selection directly affect and frame a project’s lost in aggregation. In addition, the specific issues of requirements analysis. In current practice, it is expected technology design are usually not a factor in the decision that requirements analysis begins after project selection model construction. has occurred. Yet, we know little empirically about the Requirements analysis is commonly viewed as a first procedural relationship between project selection and step in transforming a given problem into a new solution requirements analysis. We performed a field study to [6, 7]. The details of the stakeholders’ requirements are examine in detail how project selection is performed and fundamental to the process of creating a requirements what procedural relationship, if any, exists between specification and are used to help define system design. project selection and requirements analysis. We found, During requirements analysis, economic factors are often contrary to the common view of requirements engineering used as nonfunctional requirements to inform design, practice, that requirements analysis occurred in multiple rather than solely used to decide which design to parallel streams. We observed that requirements analysis implement. helped define the initial project choices. In addition, as Hence, there has been little crossover between the the project selection process progressed, each candidate study of project selection and of requirements engineering project’s requirements were further refined. We call this (RE). Also, another difficulty in studying this relationship process multiple parallel competitive requirements is that those who are responsible for performing project analysis (MPCRA). We argue that MPCRA should be selection are in a sensitive economic and political considered as a viable alternative to current requirements situation. As the project choice increases in economic engineering practices, especially for determining and impact, i.e. costing in the millions of dollars, the scrutiny designing large-scale complex projects. of the selection process and its influences become very high in an organization. Sites we contacted to study were 1. Introduction uniformly reluctant to be observed because of the fear of 1) sensitive business information being “leaked” to those The relationship between project selection and who should not have access it and 2) the fear of bias being requirements analysis is important, but is not well introduced into the process by having a third party watch. understood. It is important because the decisions made Altogether, these “barriers to entry” reduce the number of when selecting a project frame the project’s requirements detailed studies of the relationship between project analysis. These early decisions arguably impact the selection and requirements analysis. ultimate success or failure of the project. And, as We found a group, the National Air and Space commonly asserted, requirements errors are the most Administration’s (NASA) New Millennium Program costly to fix [1, 2]. (NMP), who allowed us to observe their project selection Yet, there have been few studies which examine the and requirements analysis processes. Two important relationship between project selection and requirements. factors allowed this to occur. First. The NMP’s selection Project selection studies tend to be modelled from a process is mandated by NASA, by a United States congressional policy, to be an “open and fair” process. process used for project selection, and to observe the Since their process was “open,” it was generally relationship between project selection and requirements observable (at least the parts that were not highly analysis. This approach has been used in similar research sensitive) and well documented. Details about this process to understand how complex organizations deal with are discussed later in this paper. Second, they repeat their designing, maintaining or repairing technology [16] and selection process at least once a year. This allowed the to understand new system design [17]. NMP and the investigators to plan for and conduct the The field site we observed was the Jet Propulsion investigation, and to observe a few selection cycles. The Laboratory (JPL), which is a NASA (National repeated processes enabled us to compare current results Aeronautics and Space Administration) research with the program’s past selections. This paper reports laboratory located in southern California. The group we results of the empirical investigation. studied was the New Millennium Program (NMP). Field site data collection consisted of: 1) participant observation 1.1. Background over five months of the NMP space flight validation process in action, 2) 46 semi-structured interviews with There is a small, but growing, body of research that NMP members (one NASA administrator) and 34 group deals directly with the issue of conflict and negotiation meetings, 3) informal and semi-formal discussions with during RE. This includes the application of formal small groups of NMP program, and other JPL, members, techniques to resolve stakeholders’ goal conflicts [8], 4) attending five detailed technical presentations and 5) analyzing how modeling of goals can help deal with goal studying hundreds of related documents, slide sets and inconsistency [9], a cost-value framework for carrying out papers that described the NMP process. requirements analysis negotiation [10], the use of a goal- All interviews and many of the small group oriented model view of requirements to reveal discussions were digitally audio recorded and transcribed. stakeholders’ interests and concerns [11], and the The data was analyzed using open and axial coding [15]. “WinWin” framework [12]. Although these are good The open coding was used to identify the important approaches to revealing conflicting requirements, they do components in the NMP process. The axial coding was not deal with the details in the process of selecting used to organize and relate the components in a way to projects that underlies and generates requirements and faithfully reproduce and represent what was observed. goal conflicts. The analysis focused on comparing the process as We assert that a careful empirical examination of the professed, usually via documents and slide presentations, process of selecting projects may reveal some of the and the process as enacted, i.e. “invisible work.” [18]. procedural intricacies that relate the determination of Using an autodriving method of data review [19], the projects to requirements analysis. Hence, in this field “correctness” of the observed process was validated by study, we explored the following questions: does the the NMP members themselves. It should be noted that the order of first determining project choices, making project NMP members, in general, were highly insistent on being selection and then performing requirements analysis have sure the details were correctly identified and learned by an empirical basis? Are these processes really conducted the researchers, as they repeatedly quizzed and corrected this way in practice? If not, what are possible the researchers on the process details during many onsite relationships between project choice construction, project visits. selection and requirements analysis? How are they similar The findings in this paper focus on the differences or different to current RE views? between the theoretical view of project selection and requirements analysis and the process as observed. When 2. Research Methodology a notable difference was discovered, follow-up semi- structured interviews were conducted to examine and Although econometric and statistical techniques have better understand the difference in detail. been employed widely for analyzing decision-making in projects, we feel that they do not provide insight on 3. The Field Site process details for how project selection actually is performed in the field and what is, if any, the application The JPL has been in existence for over 40 years. It of requirements analysis to this process. In addition, had been involved in the design and development of although there has been work on applying ethnography to technologies used in nearly all of the NASA space (and the requirements process, especially requirements Earth) based missions during that time, including landing elicitation [13, 14], there has been little empirical work on on the moon and the Mars rover. In general, JPL’s main observing the requirements process itself. mission is to research, invent and develop new We decided to use ethnographic field work methods technologies to promote and enable (mainly) space based [15] to capture a detailed understanding of a complex scientific research. The members of the lab participate