The Systems Test Architect: Enabling the Leap from Testable to Tested
Total Page:16
File Type:pdf, Size:1020Kb
View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Calhoun, Institutional Archive of the Naval Postgraduate School Calhoun: The NPS Institutional Archive Theses and Dissertations Thesis and Dissertation Collection 2016-09 The systems test architect: enabling the leap from testable to tested Rinaldi, Javier A. Monterey, California: Naval Postgraduate School http://hdl.handle.net/10945/50476 NAVAL POSTGRADUATE SCHOOL MONTEREY, CALIFORNIA THESIS THE SYSTEMS TEST ARCHITECT: ENABLING THE LEAP FROM TESTABLE TO TESTED by Javier A. Rinaldi September 2016 Thesis Advisor: Oleg Yakimenko Second Reader: Kristin Giammarco Approved for public release. Distribution is unlimited. THIS PAGE INTENTIONALLY LEFT BLANK REPORT DOCUMENTATION PAGE Form Approved OMB No. 0704-0188 Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188) Washington, DC 20503. 1. AGENCY USE ONLY 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED (Leave blank) September 2016 Master’s thesis 4. TITLE AND SUBTITLE 5. FUNDING NUMBERS THE SYSTEMS TEST ARCHITECT: ENABLING THE LEAP FROM TESTABLE TO TESTED 6. AUTHOR(S) Javier A. Rinaldi 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING Naval Postgraduate School ORGANIZATION REPORT Monterey, CA 93943-5000 NUMBER 9. SPONSORING /MONITORING AGENCY NAME(S) AND 10. SPONSORING / ADDRESS(ES) MONITORING AGENCY N/A REPORT NUMBER 11. SUPPLEMENTARY NOTES The views expressed in this thesis are those of the author and do not reflect the official policy or position of the Department of Defense or the U.S. Government. IRB Protocol number ____N/A____. 12a. DISTRIBUTION / AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE Approved for public release. Distribution is unlimited. 13. ABSTRACT (maximum 200 words) Test and evaluation is one of the cornerstones of the systems engineering process. Not only is it the main vehicle to obtain information about the adequacy of a system design, but it positively influences design decisions and the systems engineering process, if used from the earliest life-cycle stages. Based on its value, the Department of Defense (DOD) and industry both have placed an emphasis on a “shift-left” mentality approach in recent years. Despite this, little guidance or policy is available on how to achieve this mentality within the scope of the systems engineering process. Through the analysis of the documented roles of test and evaluation in systems engineering, this thesis examines the concept that test and evaluation, based on its desired early involvement in the system engineering process, is a stakeholder in that process. In order to participate in that process, test and evaluation as an activity requires a proxy, which this work refers to as the “systems test architect.” The conclusion is that the Systems Test Architect will positively influence the systems engineering process by becoming the proxy stakeholder for test and evaluation. 14. SUBJECT TERMS 15. NUMBER OF test and evaluation, test architect PAGES 87 16. PRICE CODE 17. SECURITY 18. SECURITY 19. SECURITY 20. LIMITATION CLASSIFICATION OF CLASSIFICATION OF THIS CLASSIFICATION OF ABSTRACT REPORT PAGE OF ABSTRACT Unclassified Unclassified Unclassified UU NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89) Prescribed by ANSI Std. 239-18 i THIS PAGE INTENTIONALLY LEFT BLANK ii Approved for public release. Distribution is unlimited. THE SYSTEMS TEST ARCHITECT: ENABLING THE LEAP FROM TESTABLE TO TESTED Javier A. Rinaldi Civilian, Department of the Navy B.S., University of Puerto Rico, 2001 Submitted in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE IN SYSTEMS ENGINEERING from the NAVAL POSTGRADUATE SCHOOL September 2016 Approved by: Oleg Yakimenko, PhD Thesis Advisor Kristin Giammarco, PhD Second Reader Ronald Giachetti, PhD Chair, Department of System Engineering iii THIS PAGE INTENTIONALLY LEFT BLANK iv ABSTRACT Test and evaluation is one of the cornerstones of the systems engineering process. Not only is it the main vehicle to obtain information about the adequacy of a system design, but it positively influences design decisions and the systems engineering process, if used from the earliest life-cycle stages. Based on its value, the Department of Defense (DOD) and industry both have placed an emphasis on a “shift-left” mentality approach in recent years. Despite this, little guidance or policy is available on how to achieve this mentality within the scope of the systems engineering process. Through the analysis of the documented roles of test and evaluation in systems engineering, this thesis examines the concept that test and evaluation, based on its desired early involvement in the system engineering process, is a stakeholder in that process. In order to participate in that process, test and evaluation as an activity requires a proxy, which this work refers to as the “systems test architect.” The conclusion is that the Systems Test Architect will positively influence the systems engineering process by becoming the proxy stakeholder for test and evaluation. v THIS PAGE INTENTIONALLY LEFT BLANK vi TABLE OF CONTENTS I. INTRODUCTION..................................................................................................1 A. BACKGROUND ........................................................................................1 B. PROBLEM STATEMENT .......................................................................2 C. PURPOSE ...................................................................................................2 D. RESEARCH QUESTION .........................................................................2 E. POTENTIAL BENEFIT OF STUDY ......................................................3 F. SCOPE ........................................................................................................3 G. METHODOLOGY ....................................................................................4 II. TEST AND EVALUATION IN SYSTEMS ENGINEERING...........................5 A. DEFINITIONS ...........................................................................................5 B. GENERAL THEORY OF SYSTEMS ENGINEERING .......................6 C. A TEST AND EVALUATION VIEW OF THE SYSTEMS ENGINEERING PROCESS MODEL .....................................................8 1. Definition of the System Requirements........................................9 2. Functional Decomposition and Allocation (Functional Architecting) .................................................................................11 3. Design Synthesis (Physical Architecting) ...................................11 4. Design Implementation ................................................................12 5. Component Verification ..............................................................13 6. System Verification ......................................................................13 7. System Validation ........................................................................14 D. CHAPTER SUMMARY ..........................................................................14 III. TEST AND EVALUATION POLICY AND GUIDANCE...............................15 A. DEPARTMENT OF DEFENSE POLICY ON TEST AND EVALUATION ........................................................................................15 1. United Stated Title, Code 10 – Armed Forces ...........................15 2. Department of Defense Directive 5000.01 ..................................18 3. Department of Defense Directive 5000.02 ..................................18 4. Defense Acquisition Guidebook ...................................................23 5. Test and Evaluation Management Guide ....................................24 B. INDUSTRY GUIDANCE ON TEST AND EVALUATION ................27 1. INCOSE-TP-2003-020-01 – Technical Measurement ................28 2. INCOSE-TP-2010-005-02 – Systems Engineering Measurement Primer ....................................................................30 C. CHAPTER SUMMARY ..........................................................................30 vii IV. THE FIT OF THE SYSTEMS TEST ARCHITECT .......................................33 A. ROLES OF THE SYSTEMS TEST ARCHITECT ..............................33 1. Advocate........................................................................................34 2. Steward .........................................................................................35 3. Advisor ..........................................................................................36 4. Domain Expert .............................................................................37 5. Communicator..............................................................................38 6. Stakeholder ...................................................................................39 B. RESPONSIBILITIES OF THE SYSTEMS TEST ARCHITECT ......39 1. Development of the Test and Evaluation