Success in Acquisition: Using Archetypes to Beat the Odds William E. Novak Linda Levine September 2010 TECHNICAL REPORT CMU/SEI-2010-TR-016 ESC-TR-2010-016 Acquisition Support Program Unlimited distribution subject to the copyright. http://www.sei.cmu.edu This report was prepared for the SEI Administrative Agent ESC/XPK 5 Eglin Street Hanscom AFB, MA 01731-2100 The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. This work is sponsored by the U.S. Department of Defense. The Software Engineering Institute is a federally funded research and development center sponsored by the U.S. Department of Defense. Copyright 2010 Carnegie Mellon University. NO WARRANTY THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. Use of any trademarks in this report is not intended in any way to infringe on the rights of the trademark holder. Internal use. Permission to reproduce this document and to prepare derivative works from this document for inter- nal use is granted, provided the copyright and “No Warranty” statements are included with all reproductions and derivative works. External use. This document may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other external and/or commercial use. Requests for permission should be directed to the Software Engineering Institute at [email protected]. This work was created in the performance of Federal Government Contract Number FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. The Government of the United States has a royalty-free government-purpose license to use, duplicate, or disclose the work, in whole or in part and in any manner, and to have or permit others to do so, for government purposes pursuant to the copyright license under the clause at 252.227-7013. For information about SEI publications, please visit the library on the SEI website (www.sei.cmu.edu/library). Table of Contents Acknowledgments v Abstract vii 1 Introduction 1 1.1 The Problem 1 1.2 Background and Rationale 2 2 Systems Thinking 5 2.1 Feedback Loops and Open and Closed Systems 5 2.2 Causal Loop Diagrams 6 3 The Systems Archetypes 9 3.1 Fixes That Fail 10 3.2 Shifting the Burden 11 3.3 Accidental Adversaries 13 3.4 Escalation 15 3.5 Drifting Goals 16 3.6 Growth and Underinvestment 18 3.7 Success to the Successful 20 3.8 Limits to Growth 22 3.9 Tragedy of the Commons 23 3.10 Balancing Loop with Delay 25 3.11 Some Observations on Systems Thinking and the Systems Archetypes 26 4 Applying the Systems Archetypes to Software Acquisition 29 5 The Acquisition Archetypes 31 5.1 The Bow Wave Effect 31 5.2 Firefighting 35 5.3 Everything for Everybody 38 5.4 Feeding the Sacred Cow 41 5.5 PMO Versus Contractor Hostility 45 5.6 Staff Burnout and Turnover 49 5.7 Underbidding the Contract 52 5.8 Longer Begets Bigger 55 5.9 Robbing Peter to Pay Paul 58 5.10 “Happy Path” Testing 61 5.11 Brooks’ Law 64 5.12 Shooting the Messenger 68 6 Challenges, Implications, and Future Directions 71 6.1 Short-Term Thinking 72 6.2 Misaligned Goals 74 6.3 Future Directions 75 References 79 CMU/SEI-2010-TR-016 | i CMU/SEI-2010-TR-016 | ii List of Figures Figure 1: Causal Loop Diagrams of Reinforcing and Balancing Loops 7 Figure 2: Causal Loop Diagram of “Fixes That Fail” 10 Figure 3: Causal Loop Diagram of “Shifting the Burden” 11 Figure 4: Causal Loop Diagram of “Accidental Adversaries” 13 Figure 5: Causal Loop Diagram of “Escalation” 15 Figure 6: Causal Loop Diagram of “Drifting Goals” 16 Figure 7: Causal Loop Diagram of “Growth and Underinvestment” 18 Figure 8: Causal Loop Diagram of “Success to the Successful” 20 Figure 9: Causal Loop Diagram of “Limits to Growth” 22 Figure 10: Causal Loop Diagram of “Tragedy of the Commons” 24 Figure 11: Causal Loop Diagram of “Balancing Loop with Delay” 25 Figure 12: Causal Loop Diagram of “The Bow Wave Effect” 33 Figure 13: Causal Loop Diagram of “Firefighting” 37 Figure 14: Causal Loop Diagram of “Everything for Everybody” 39 Figure 15: Causal Loop Diagram of “Feeding the Sacred Cow” 43 Figure 16: Causal Loop Diagram of “PMO vs. Contractor Hostility” 47 Figure 17: Causal Loop Diagram of “Staff Burnout and Turnover” 50 Figure 18: Causal Loop Diagram of “Underbidding the Contract” 53 Figure 19: Causal Loop Diagram of “Longer Begets Bigger” 56 Figure 20: Causal Loop Diagram of “Robbing Peter to Pay Paul” 59 Figure 21: Causal Loop Diagram of “’Happy Path’ Testing” 62 Figure 22: Causal Loop Diagram of “Brooks’ Law” 65 Figure 23: Causal Loop Diagram of “Shooting the Messenger” 69 CMU/SEI-2010-TR-016 | iii CMU/SEI-2010-TR-016 | iv Acknowledgments Many people have contributed to creating this report, both directly and indirectly. It would not have been possible to discuss such a wide range of software acquisition topics without the insights, expertise, and prior work of others. We would also like to thank a key sponsor, the Department of Veterans Affairs, for the original opportunity to perform this work, and our other customers and sponsors who enabled us to work on technical engagements that stimulated and contributed to the ideas expressed here. We extend our thanks to people within the Carnegie Mellon Software Engineering Institute (SEI) and beyond for their comments and reviews of early versions and for their support of this work: Joe Elm, John Foreman, Brian Gallagher, Michael Goodman, Patricia Oberndorf, Robert Rosenstein, and Ray Williams. Finally, the authors are most grateful for the essential help we have received from our editors, Gerald Miller and Barbara White, and our graphic designer, Melissa Neely. Their efforts have resulted in a better document. Carnegie Mellon is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. CMU/SEI-2010-TR-016 | v CMU/SEI-2010-TR-016 | vi Abstract This project on patterns of failure is based on experiences with actual programs and employs concepts from systems thinking to analyze dynamics that have been observed in software development and acquisition practice. The software acquisition and development archetypes, based in part on the general systems archetypes, have been created as part of an ongoing effort to characterize and help manage patterns of counterproductive behavior in software development and acquisition. This report introduces key concepts in systems thinking and the general systems archetypes, and then applies these concepts to the software-reliant acquisition domain. Twelve selected software acquisition and development archetypes are each described and illustrated by a real-life scenario, and guidance is provided on both recovering from and preventing these dynamics. Finally, the authors consider implications of the work and future directions for research. CMU/SEI-2010-TR-016 | vii CMU/SEI-2010-TR-016 | viii 1 Introduction This project on patterns of failure is based on experiences with actual programs and employs concepts from systems thinking to analyze dynamics that have been observed in software development and acquisition practice. The software acquisition and development archetypes have been created as part of an ongoing effort to characterize and help manage patterns of counterproductive behavior in software development and acquisition. These archetypes use the ideas of systems thinking to describe common patterns of failure so that they can be anticipated and prevented. In this report we begin by describing key elements in systems thinking. We proceed with an introduction to the general systems archetypes, and then apply these concepts to the software acquisition domain. Twelve selected software acquisition and development archetypes are described. Each is illustrated in an actual scenario, and guidance is provided on both breaking and preventing these dynamics. Finally, we consider implications and directions for the future—for research and for the use of acquisition archetypes in the field. 1.1 The Problem Perhaps the most puzzling question in the software and systems acquisition world is this: Why do problems persist in software development and systems acquisition, despite the fact that solutions to many of these problems exist and have been known for decades? For 50 years, federally funded research and development centers, think tanks, and advisory bodies—such as the MITRE Corporation, the Aerospace Corporation, the RAND Corporation, and the Defense Science Board—have analyzed barriers and enablers to the acquisition and development of software-intensive systems. For example, the following challenges for policy and topics for research were set down almost 30 years ago [Dews 1979]: Improve the acquisition information data base. Reduce the instability in program funding and scheduling. Strengthen guidance on hardware competition in development. Emphasize production quantity as an element in the requirements process. Continue offering incentives to make program management an attractive service career. Examine the timing of program manager (PM) assignments. This list of topics remains pertinent, although an updated version would highlight additional challenges, including software development and deployment, security, and system-of-systems interoperability. It is disturbing, however, that many long-standing problems associated with the development and acquisition of software-intensive systems remain unresolved—and are growing in magnitude—while proposed solutions remain either untried or have not been sustained.