Technology Readiness Assessment Guide Best Practices for Evaluating the Readiness of Technology for Use in Acquisition Programs and Projects From August 11, 2016 to August 10, GAO-16-410G 2017, GAO is seeking input and feedback August 2016 on this Exposure Draft from all interested parties. See page 9 for more information. Preface .................................................................................. 8 Introduction ........................................................................ 11 The Guide’s Case Studies .................................................................... 13 The Guide’s Readers ........................................................................... 13 Acknowledgments ............................................................................... 14 Chapter 1 ............................................................................ 15 What Is a Technology Readiness Assessment (TRA)? ......................... 15 Definitions and Overview .............................................................. 16 TRAs Inform Technology Development and Identify Potential Concerns ....................................................................................... 18 Overview of Technology Development and Its Relationship to Acquisition Programs .................................................................... 18 Technology Development Models ................................................ 23 TRA’s Relationship to Program Management and Oversight ....... 24 Tailoring TRAs for Different Purposes ........................................... 25 Chapter 2 ............................................................................ 26 Why TRAs Are Important and Understanding Their Limitations ........ 26 Maturity of Technology at Program Start Is an Important Determinant of Success ................................................................ 28 TRA Process Is a Mechanism That Informs Important Acquisition Functions ....................................................................................... 29 Understanding TRAs Can Help Practitioners and Consumers of Information ................................................................................... 31 TRAs Are Snapshots in Time ......................................................... 33 Advancements in Technology Can Pose Challenges in Applying TRAs............................................................................................... 33 Organizational Experience, Culture, and Bias Can Affect TRAs .... 34 Page 1 DRAFT TRAs Depend on the Quality and Availability of Credible Data .... 36 Chapter 3 ............................................................................ 38 Best Practice: A Reliable Process for Conducting Credible TRAs ........ 38 More Frequent Evaluations of Technology Maturity ................... 40 High Quality TRAs .......................................................................... 41 Chapter 4 ............................................................................ 43 Best Practice: Including Technology Maturity Assessments in the Program Strategy, Designing the TRA Plan, and Determining the Team ............................................................................................................. 43 Technical Maturity Assessment Strategy ...................................... 43 The Assessment Team .................................................................. 46 The TRA Plan ................................................................................. 48 Chapter 5 ............................................................................ 52 Best Practice: Selecting Critical Technologies..................................... 52 Critical Technologies Defined ....................................................... 52 Challenges in Selecting Critical Technologies ............................... 54 Steps for Selecting Critical Technologies ...................................... 55 Identifying Other Important Technologies and Programmatic Issues ............................................................................................. 65 Chapter 6 ............................................................................ 68 Best Practice: Evaluating Critical Technologies .................................. 68 Steps for Evaluating Critical Technologies .................................... 68 Relevant Information Must Be Used to Evaluate Critical Technologies ................................................................................. 74 Operational Environment Is Key to Evaluating Critical Technologies ................................................................................. 74 Creating Critical Technology Subsets for Exceptionally Large or Complex Programs ........................................................................ 75 Page 2 DRAFT Chapter 7 ............................................................................ 77 Best Practice: Preparing the TRA Report ............................................ 77 The TRA Report ............................................................................. 77 Steps for Preparing and Coordinating the TRA Report ................. 80 Response to the TRA Report ......................................................... 84 How Dissenting Views Are Documented and Submitted ............. 84 Chapter 8 ............................................................................ 86 Best Practice: Using the TRA Results .................................................. 86 How TRA Reports Are Used .......................................................... 86 TRAs for Governance Decisions .................................................... 87 TRAs as Knowledge-building Exercises ......................................... 88 Identification of Potential Areas of Concern and Risk .................. 89 Early Technology Development .................................................... 91 TRA Process Facilitates Information Sharing Opportunities ......... 92 Basis for Developing a Technology Maturation Plan (TMP) for Immature Technologies ................................................................ 93 Chapter 9 ............................................................................ 95 Best Practice: Preparing a Technology Maturation Plan (TMP) ......... 95 Steps for Preparing a Technology Maturation Plan ...................... 96 Updating a Technology Maturation Plan ...................................... 98 The Technology Maturation Plan Template ............................... 101 Chapter 10 ........................................................................ 106 Practices Are Evolving in Evaluating Software Systems and Systems Integration Using TRAs ...................................................................... 106 Applying TRAs to Software Systems ........................................... 106 Software Embedded Technologies versus Software-only Technologies ............................................................................... 108 Page 3 DRAFT Development of System-level Readiness Metrics ...................... 109 Appendix I: Key Questions to Assess How Well Programs Followed the Six Step Process for Developing Credible TRAs ......................................................................................... 112 Appendix II: Auditing Agencies and Their Websites ........... 117 Appendix III: Case Study Backgrounds ............................... 119 Case Study 1: Immature Technologies Increases Risk, GAO-08-408 119 Case Study 2: Assessments Provide Key Information, GAO-10-675 . 120 Case Study 3: Space Programs Often Underestimate Costs, GAO-07-96 ........................................................................................................... 120 Case Study 4: Program Updates Can Change Critical Technologies, GAO-02-201 ...................................................................................... 121 Case Study 5: Identifying Back-up Critical Technologies, GAO-08- 467SP ................................................................................................. 121 Appendix IV: Experts Who Helped Develop This Guide ...... 123 Appendix V: Contacts and Acknowledgments ................... 130 GAO Contacts .................................................................................... 130 Other Leadership Provided for This Project ..................................... 130 Acknowledgments ............................................................................. 130 Appendix VI: Examples of Various TRL Definitions and Descriptions by Organization ............................................ 131 Appendix VII: Other Types of Readiness Levels ................. 137 Appendix VIII: Agency Websites Where TRA Report Examples Can Be Found .................................................................... 142 References ........................................................................ 143 Image Sources ................................................................... 146 Page 4 DRAFT Tables Table 1: Cost and Schedule Experiences for Products with Mature and Immature Technologies 29 Table 2: Six Steps for Conducting a Technology Readiness Assessment (TRA) 39 Table 3: Characteristics of High Quality Technology Readiness Assessments (TRA) 41 Table 4: Software Implementation Project Work Breakdown Structure 60 Table 5: Technology Readiness Levels (TRLs) Supporting Information for Hardware and Software 72 Table 6: Example Program Management Tools Used with Technology