NASA/SP–2013–604 NASA Software Engineering Benchmarking Study Heather L. Rarick Johnson Space Flight Center, Houston, TX Sara H. Godfrey Goddard Space Flight Center, Greenbelt, MD John C. Kelly NASA Headquarters, Washington, DC Robert T. Crumbley Marshal Space Flight Center, Huntsville, AL Joel M. Wilf Jet Propulsion Laboratory, Pasadena, CA May 2013 NASA STI Program ... in Profile Since its founding, NASA has been dedicated to the • CONFERENCE PUBLICATION. Collected advancement of aeronautics and space science. The papers from scientific and technical conferences, NASA scientific and technical information (STI) pro- symposia, seminars, or other meetings sponsored or gram plays a key part in helping NASA maintain this co-sponsored by NASA. important role. • SPECIAL PUBLICATION. Scientific, technical, The NASA STI program operates under the auspices or historical information from NASA programs, of the Agency Chief Information Officer. It collects, projects, and missions, often concerned with organizes, provides for archiving, and disseminates subjects having substantial public interest. NASA’s STI. The NASA STI program provides access to the NASA Aeronautics and Space Database and its • TECHNICAL TRANSLATION. English-language public interface, the NASA Technical Report Server, translations of foreign scientific and technical thus providing one of the largest collections of aero- material pertinent to NASA’s mission. nautical and space science STI in the world. Results Specialized services also include organizing and are published in both non-NASA channels and by publishing research results, distributing specialized NASA in the NASA STI Report Series, which includes research announcements and feeds, providing help the following report types: desk and personal search support, and enabling data • TECHNICAL PUBLICATION. Reports of exchange services. For more information about the completed research or a major significant phase of NASA STI program, see the following: research that present the results of NASA Programs • Access the NASA STI program home page at and include extensive data or theoretical analysis. http://www.sti.nasa.gov Includes compilations of significant scientific and technical data and information deemed to be of • E-mail your question via the Internet to continuing reference value. NASA counterpart of [email protected] peer-reviewed formal professional papers but has • Fax your question to the NASA STI Help Desk at less stringent limitations on manuscript length and 443-757-5803 extent of graphic presentations. • Phone the NASA STI Help Desk at 443-757-5802 • TECHNICAL MEMORANDUM. Scientific and technical findings that are preliminary or of • Write to: specialized interest, e.g., quick release reports, NASA STI Help Desk working papers, and bibliographies that contain NASA Center for AeroSpace Information minimal annotation. Does not contain extensive 7115 Standard Drive analysis. Hanover, MD 21076-1320 • CONTRACTOR REPORT. Scientific and technical findings by NASA-sponsored contractors and grantees. NASA/SP–2013–604 NASA Software Engineering Benchmarking Study Heather L. Rarick Johnson Space Flight Center, Houston, TX Sara H. Godfrey Goddard Space Flight Center, Greenbelt, MD John C. Kelly NASA Headquarters, Washington, DC Robert T. Crumbley Marshal Space Flight Center, Huntsville, AL Joel M. Wilf Jet Propulsion Laboratory, Pasadena, CA National Aeronautics and Space Administration NASA Headquarters Washington, DC 20546 May 2013 Notice for Copyrighted Information This manuscript is a work of the United States Government authored as part of the official duties of employee(s) of the National Aeronautics and Space Administration. No copyright is claimed in the United States under Title 17, U.S. Code. All other rights are reserved by the United States Government. Any publisher accepting this manuscript for publication acknowledges that the United States Government retains a non-exclusive, irrevocable, worldwide license to prepare derivative works, publish, or reproduce this manuscript, or allow others to do so, for United States Government purposes. Trade names and trademarks are used in this report for identification only. Their usage does not constitute an official endorsement, either expressed or implied, by the National Aeronautics and Space Administration. Level of Review: This material has been technically reviewed by technical management Available from: NASA Center for AeroSpace Information National Technical Information Service 7115 Standard Drive 5285 Port Royal Road Hanover, MD 21076-1320 Springfield, VA 22161 Price Code: A17 NASA/SP-2013-604 NASA Software Engineering Benchmarking Study May 1, 2013 Table of Contents Executive Summary ........................................................................................................vi 1. Introduction .............................................................................................................. 1 1.1. Background .......................................................................................................... 1 2. Purpose .................................................................................................................... 3 3. Scope ....................................................................................................................... 4 3.1. Scope of Topic Areas ........................................................................................... 4 3.2. Scope of Chosen Organizations ........................................................................... 6 4. Methods ................................................................................................................... 8 4.1. Preparation ........................................................................................................... 8 4.2. Interviews ............................................................................................................. 8 4.3. Data Analysis and Report Development ............................................................... 9 5. General Observations ............................................................................................ 11 5.1. Policy .................................................................................................................. 11 5.1.1. Questions .................................................................................................. 11 5.1.2. Discussion ................................................................................................. 12 5.1.2.1. Aerospace Industries .......................................................................... 13 5.1.2.2. Universities and University Research and Development Labs ............ 14 5.1.2.3. Defense Services ................................................................................ 15 5.1.2.4. NASA .................................................................................................. 16 5.1.3. Observations ............................................................................................. 18 5.1.4. Recommendations ..................................................................................... 18 5.2. Acquisition .......................................................................................................... 20 5.2.1. Questions .................................................................................................. 20 5.2.2. Discussion ................................................................................................. 20 5.2.2.1. Aerospace Industries .......................................................................... 22 5.2.2.2. Universities and University Research and Development Labs ............ 22 5.2.2.3. Defense Services ................................................................................ 23 5.2.2.4. NASA .................................................................................................. 23 5.2.3. Observations ............................................................................................. 24 5.2.4. Recommendations ..................................................................................... 24 5.3. Testing ................................................................................................................ 26 5.3.1. Questions .................................................................................................. 26 5.3.2. Discussion ................................................................................................. 26 5.3.2.1. Aerospace Industries .......................................................................... 29 ii NASA/SP-2013-604 NASA Software Engineering Benchmarking Study May 1, 2013 5.3.2.2. Universities and University Research and Development Labs ............ 30 5.3.2.3. Defense Services ................................................................................ 31 5.3.2.4. NASA .................................................................................................. 31 5.3.3. Observations ............................................................................................. 32 5.3.4. Recommendations ..................................................................................... 32 5.4. Assurance........................................................................................................... 33 5.4.1. Questions .................................................................................................. 33 5.4.2. Discussion ................................................................................................