What T&E’rs Need to Know About

A Webinar Based On A Short Course By: Dr. W. David Bell Dr. C. David Brown SYSTEMS A n d PROGRAM A n d W h y This Webinar Highlights Selected Topics from Our 3 Day Short Course • Overview of Program/ (PM) • Overview of (SE) • Test and Evaluation (T&E) and Interaction with PM and SE • & T&E • Current Trends in T&E – T&E early in programs – Verify vs Validate = DT vs OT – Integrated T&E – T&E of immature technologies – Agile acquisition T&E implications • Case Studies - Army Future Combat Systems & Navy Littoral Surveillance Radar System • DOD Acquisition – Process, Life Cycle, Regulations, Directives, Guidance, & Instructions • Special Topics – Software SE and T&E – Human Systems Integration – Unmanned and Autonomous Sys SE and T&E – Cyber and Info Assurance SE and T&E

31 May 2013, 2100 2 Dave Bell Principal Multi-Discipline Systems Engineer MITRE Corp [email protected] Things done: • Data Analysis • Flight test direction - DT and OT • Scientific experiments • Systems engineering from beginning to end – Requirements definition – Tech development, tech maturing – I&T at all levels - assembly, subsystem & system • Managed S&T programs • VP of operations • Adjunct Professor /Lecturer of systems engineering – SMU, JHU

Education • BS Physics, MBA, D. Engineering

31 May 2013, 2100 3 Dave Brown Consulting Systems Engineer MITRE Corp and Institute for Defense Analyses [email protected]

Things done: • Colonel, US Army Signal Corps • Developing DT instrumentation and test facilities • Development of M&S for test support – Army Virtual Proving Ground • Live Fire T&E • Army Senior Executive – Director of Test and Technology, Army Developmental Test Command • Deputy PM/Executive Director for Test, Army Future Combat Systems Program • Instructor – JHU Masters in Systems Engineering

Education • BS, MS, PhD Electrical Engineering • Masters in National Security Policy, ICAF

31 May 2013, 2100 4 C Bottom Line Up Front What PM elements do T&E’rs need to understand? – Leadership, authority – Stakeholders – – Planning, monitoring, and control – Work breakdown structure (WBS) – Budgeting – Scheduling – (EVM)

Why: – All of the above involve some information for which T&E is the source – Accomplishing T&E involves work (WBS), resources (Budget), and time (Schedule) – Information from T&E is critical to stakeholder communication, risk management, monitoring, and assessing work completion for EVM – T&E planning, resourcing, execution, and completion must be “program managed”

31 May 2013, 2100 5 Bottom Line Up Front C What SE elements do T&E’rs need to understand? – Needs and definition – Requirements analysis – Synthesis – Integration – Maturity/Readiness – Trades – Verification – Validation – Risk management

Why: – All of the above involve some information for which T&E is the source – The test system must be system engineered – Early T&E involvement in programs involves T&E and SE integration – Pushed by GAO and proven by successful programs

31 May 2013, 2100 6 Bottom Line Up Front C

And most important: – T&E’rs must have a thorough knowledge and understanding of the total system and system management process to understand the information and the information needs and perspectives of each stakeholder, So that: – T&E can provide each stakeholder with information that is: •Required •Relevant •Complete •Accurate •Appropriate •Understandable •In the Correct Context

31 May 2013, 2100 7 C Our Challenging World • Terrorism • Global culture clashes • Information overload • Disease – Health Care • Infrastructure • Technical Complexity • Business and Finance Problem Attributes • Technical • Size - Scope • Complexity • Importance • Social, political, fiscal

Systems Engineering and Management is increasingly the field expected to solve the problems. 31 May 2013, 2100 8 C A Systems Approach

Operational Data Collection Lessons Learned Test & Evaluation Capabilities Improvement Product Development & Needs Definition Production

Data Collection Mission Performance Analysis Operations

Technology

Knowledge Prototype Development Performance Demonstration Critical Field Experiments Enabling Science & Technology Hypothesis, Concept Development Trade-offs & Critical Experiments Technology Knowledge Transfer Modeling9 and Simulations This slide adapted from a brief by: 31 May 2013, 2100 Dr. Samuel Seymour, JHU Applied Physics Lab 9 C System Management

Systems Program/Project Engineering Management Configuration Management System Design & Engineering Quality Control Project Administration Interface Contract & Support Management Management

Analysis & Production Data Management Evaluation Management Integration Technical Performance, & Test Cost, Schedule Test & Evaluation 31 May 2013, 2100 10 Senator John McCain, 15 December 2011

To be clear, the military-industrial-congressional complex does not cause programs to fail. But, it does help create poorly-conceived programs — programs that are so fundamentally unsound that they are doomed to be poorly executed. And, it does help keep them alive—long after they should have been ended or restructured.

By “poorly conceived”, I mean major programs that are allowed to begin despite having insufficiently defined requirements; unrealistic cost or schedule estimates; immature technology or too much manufacturing and integration risk; or unrealistic performance expectations.

By “poorly executed”, I am referring to programs that poorly perform because of, among other things, unanticipated design, engineering, manufacturing or technology problems. These sorts of programs should either never have been started to begin with or should have been significantly restructured or terminated at the end of the day.

Specifically, the military-industrial-congressional complex helps ensure that poorly- conceived programs get on rails—and stay there—with “production money” when they are supposed to still be in development. And, for Industry and many of their sponsors in the Pentagon and on the Hill, that’s desirable because it is far more difficult to restructure or terminate a production program—even one that’s performing poorly—than one that’s in development.

Honorable Frank Kendall (USD(AT&L)), ITEA Journal, March 2013 The purpose of developmental testing is simple; to provide data to program leadership so that good decisions can be made as early as possible. I have a sign outside my office that is a quote from G. Edwards Deming: “In God we trust, all others must bring data.” It is our developmental testers who “bring the data” that is needed to make sound decisions during product development. Programs are organized in various ways, but whatever the specific organizational model, testing is the source of the crucial information that provides feedback to , chief engineers, lead system engineers, integrated product teams, and military users on whether their designs meet requirements or not.

Five precepts – DT&E should: Contribute to program efficiency and effective execution, Provide relevant information as early as possible, Integrate DT&E planning across the product life cycle, Focus on support to internal program decisions and verification of compliance with requirements, and Use DT&E to improve the efficiency and validity of OT&E.

We sometimes fail to conduct adequate DT&E prior to the decision to start production. About a year ago, I called a particular decision to enter production on an aircraft program without flight testing “acquisition malpractice.” If a product enters production before the design is stable, the resulting waste in cost increases and schedule slips can be dramatic, and the program is much more likely to be canceled. I stress solid, well defined DT&E results as an important prerequisite for this decision because the pressure to enter production can be overwhelming, and doing so prematurely has major consequences.

Working with program and engineering leadership as key members of the management team, developmental testers provide the information that makes program success possible and much more probable. 12 31 May 2013, 2100 C “The Test and Evaluation (T&E) process is an integral part of the Systems Engineering (SE) process, which identifies levels of performance and assists the developer in correcting deficiencies. It is a significant element in the decision-making process, providing data that support trade-off analysis, risk reduction, and requirements refinement.”*

Both SE and T&E are integral parts of the System/Program Management Process where SE provides the technical direction and T&E provides information to support the technical elements above as well as management processes and information to stakeholders of the System/Program Management Process. These include PMs, decision makers, and Congress.

*DAU Test and Evaluation Management Guide 31 May 2013, 2100 13 Attributes View of a Complex System . Form − Incorporates Many Capabilities − Comprised of Interacting, Diverse Elements − Definable Structure and Interconnections − Bounded; Has Inputs and Outputs

. Function − It Does Something Valuable − It is Dynamic—Not Inert

. Interfaces − Internal − External

This slide adapted from a brief by: 31 May 2013, 2100 Dr. Samuel Seymour, JHU Applied Physics Lab 14 Partitioning View of a Complex System Each system can be broken down into components which in turn can be broken down into smaller components, etc.

Air Defense System System

Search Radar Subsystem Subsystem

T/R Module

Component Component Component Component

Subcomponent Subcomponent Assembly

Screw Part Part

31 May 2013, 2100 This slide adapted from a brief by: 15 Dr. Samuel Seymour, JHU Applied Physics Lab Systems Perspectives View

Systems Thinking Systems Engineering SystemsEngineering Management Systems Focus on Process Focus On Whole Product Focus on Both Process and Product Consideration of Issues Solve Complex Technical Solve Complex Interdisciplinary Problems Technical, Social and Management Issues Evaluation of Multiple Develop and Test Tangible Influence Policy, Processes and use Factors and Influences System Solutions Systems Engineering to Develop Systems Solutions Inclusion of Patterns Need to Meet Requirements, Integrate Human and Technical Relationships, and Common Measure Outcomes and Solve Domain Dynamics and Approaches Understanding Problems

This slide adapted from a brief by: 31 May 2013, 2100 Dr. Samuel Seymour, JHU Applied Physics Lab 16 Key Issues C

• SE and T&E have traditionally been very separate processes accomplished by widely separate communities • SE and PM require information to accomplish knowledge-based development and acquisition • T&E provides information, but it must be the right information, at the right time, fully understood, and correctly used • The Test setup is a system that must be System Engineered. • The T&E program must be Program Managed.

31 May 2013, 2100 17 What is a System?

“A System is a set of interrelated components working together toward some common objective.” - (Kossiakoff, et al)

The organization of hardware, software, materials, facilities, personnel, data and services needed to perform a designated function with specified results...(Defense Acquisition University)

A bounded process involving specific interactions among a number of separately distinguishable functional elements (Amsler, JHU/APL)

31 May 2013, 2100 18 Systems Scope Relationships Examples

Global Family of Systems - Theater Defense Enterprise Systems – BMD Community System of Systems - AEGIS BMD Scale System – STD Missile Subsystem - IR Seeker Component - Sensor Device – Detector

Complexity

This slide adapted from a brief by: 31 May 2013, 2100 Dr. Samuel Seymour, JHU Applied Physics Lab 19 A System Includes…

The System Elements

Benjamin S. Blanchard, System Engineering Management The System Boundry 20 31 May 2013, 2100 System Environment

Benjamin S. Blanchard, System Engineering Management

31 May 2013, 2100 21 Definition of Systems Engineering (SE)

• An iterative and interdisciplinary management and development process that defines and transforms requirements into an operational system.

• Features: Typically, this process involves environmental, economic, political, social, and other non-technological aspects. Activities include conceiving, researching, architecting, utilizing, designing, developing, fabricating, producing, integrating, testing, deploying, operating, sustaining, and retiring system elements.

• Note: The customer for or user of the system usually states the initial version of the requirements. Systems engineering should be used to better define and refine these requirements. Often the requirements change as further decisions are made.

31 May 2013, 2100 22 System Engineering Identifying Qualities • Top down - viewing system as a whole. • Life cycle view. • “Complete” effort to identify system requirements “up-front”. • Interdisciplinary team approach.

31 May 2013, 2100 23 Scientific Method

Problem

Problem Definition

Select Hypothesis

Test Hypothesis

Confirm, Deny or Modify Hypothesis

Conclude

Next Problem 31 May 2013, 2100 24 Systems Engineering Process (Method) (Kossiakoff, et al) Need

1. Requirements Problem 4 Basic SE Activities: executed Analysis Definition repetitively over life cycle leading to successfully Requirements developed and validated system 2. Functional Definition Functions

3. Physical Definition or Design Synthesis Potential Solutions (System Models)

4. Design Validation

Solution(s) 31 May 2013, 2100 25 Systems Engineering Approaches Views

Operation & Maintenance Operational System Functional Production Documentation LinearDeficiencies Specifications Specifications The “V” Systems Engineering Lifecycle

Concept Engineering Post Development Development Development

Technological Defined System Production Installed Operational Opportunities Concept System System

Need 1 Spiral 2 3

4

5 Waterfall Objectives Systems Engineering Requirements Method P&D Analysis Requirements E&MD (Problem Definition)

D&V Functional CE/D Systems Engineering LIFE CYCLE ACTIVITIES

Users - Operators Organizational Structures Definition Project Mgr Attributes/Authority Market Pull Pricing/Estimating Contracting (Functional Analysis & Functions

RFP/| BAA

Discussions Proposal Form Project Office Allocation) Customer Requirements Collaboration WIN! Market Assessment Statement of Work START WORK Product Defn Statement

Brainstorming War rooms Physical Concept New Product Idea Preliminary System/Product Functional/System Work breakdown Risk Assessment Life cycle Technology Push Concept Definition Block Diagram Structure WBS Plan Definition

“Draw a Cartoon”

“PLANNING” (Synthesis, Physical Analysis Budget and Schedules System NEEDS ANALYSIS CONCEPT EXPLORATION PERT and GANTT charts & Allocation) Model CONCEPT/PROGRAM DEFINITION

Production Task Work Orders System Integration Develop Prototype Quantities Specs Work Authorizations Design

and Verification Design

Quality CDR PDR “DIRECTION, MONITOR,CONTROL” Slide adapted from a brief by: Mgt Validation Linear Responsibility Charts System Test and Evaluation Sub System Fabrication Evaluate Prototype Critical Path Analysis (Verification, Evaluation) Configuration “beta tests” Dr. Samuel Seymour, Mgmt PRODUCTION / MANUFACTURING DESIGN / TECHNOLOGY VALIDATION / ENGINEERING DEVELOPMENT Validated Next JHU Applied Physics Lab Technical Performance System Delivery Logistics Phase Project Closeout Field Test and Install/ Operations and Warehousing Acceptance Follow-On? Model Evaluation Maintenance Sales Systems Engineering

T/E AND OPERATIONAL SUPPORT SYSTEM USE Budget Schedule

3126 May 2013, 2100 Communications and Financial Management All the Time 9/20/01 SJSeymour Life Cycle or Linear Approach to Systems Engineering

Phase Needs Concept Concept Technology Engineering Integration Analysis Exploration Definition Validation Design & Evaluation Step

Analyze Analyze Analyze Analyze Requirements Analyze Analyze operational performance functional design Analysis needs reqmts reqmts reqmts reqmts reqmts

Define Define Define Define Define Define Functional component system subsystem subcomponent part functional Definition functions & functions functions functions functions tests interactions

Visualize Visualize Select Specify Specify Specify Physical subsystems, components, components, component subcomponent test Definition technology architectures architectures construction construction equipment T&E is a key part of every phase Simulate, Simulate, Validate Test & Validate Test Design validate validate component evaluate needs, critical Validation performance system design & production feasibility subsystems reqmts effectiveness construction system

31 May 2013, 2100 27 Motivation for Better Systems Management

Benjamin S. Blanchard, System Engineering Management 31 May 2013, 2100 28 The Role of a Systems Engineer • The technical “leader” and “conductor” – Sets the objectives (mission needs, system requirements) – Establishes the plan – Oversees its execution – Monitors and guides progress – Coordinates all technical activities – Enforces requirements – Is the final judge of performance/capabilities demonstrated – Works with Program Manager to orchestrate resources – Makes the difficult technical decisions – Manages resolution of technical problems – Adjusts the plan as necessary – Advises the Program Manager – Ultimately responsible for the technical product

31 May 2013, 2100 29 Systems Enterprise

• System-of-Systems (SOS) - Combination of interrelated systems working with direct interaction to achieve a broader range of activities, with the systems generally fixed and invariant. Ex: Ballistic Missile Defense System

• Family-of-Systems (FOS) – Loosely coupled or non- interacting combination of systems intended to perform a broad function where the systems likely share common components and may or may not directly interact. Ex: Stryker Family of Systems

• System of Systems Architecture - The fundamental organization of a SOS, embodied in its systems, their relationships to each other and to the environment and the principles guiding its design and evolution. Ex: Joint Information Environment (JIE), DoD Information Enterprise Architecture (IEA), Global Information Network Architecture (GINA)

30 31 May 2013, 2100 System of Systems Engineering • Traditionally, SE is focused on the development and implementation of a well-bounded and firmly established set of requirements through the development and integration of subsystems that meet those requirements.

• SoSE leverages independent, interoperable, and integrateable systems that serve a meaningful purpose in a stand-alone mode, to implement operationally flexible capabilities by building a collaborative SoS within an evolving concept of operations to effectively respond to an evolving threat.

• Interoperable = systems that communicate and cooperate

• Integratable = elements that interface in a specified manner to deliver specific functionality or capability

31 May 2013, 2100 31 What makes SoS Engineering Different? • Capability focused (e.g., engineering a capability) • Generally does not have specified requirements • Focus is more on the architecture than the systems • Focus is more on interoperability among systems • Individual systems treated as potentially interchangeable elements • More difficult to test - especially at full-up level in lab • Test challenge – Individual system contribution to SOS • However…similar engineering practices apply – e.g., SE method – Integrate and test

31 May 2013, 2100 32

What is a Project/Program/Task?

• Four Characteristics that Distinguish Projects from Other Managerial Activity: – A Three Dimensional Objective (cost, schedule, performance) – Uniqueness – Use of Resources – Accomplishment by an Organization

• Aspects of Projects That May Affect their Difficulty: – Origin of the Project – Product of the Project – Marketplace or Customer – Size, Location

31 May 2013, 2100 33 Project Planning and Control Work Description And Instructions Network Scheduling

Master / Detailed Goals / Objectives Schedules

Continuous Process Management Decision-Making Budgets

Time / Cost / Performance System Reports Tracking

100 90 80 70 60 50 40 30 20 10 0 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

31 May 2013, 2100 34 Project Planning • Identifies and Communicates: • The Actions to be Taken to Achieve the Project Objectives. (Performance – WBS) • The Sequence and Timing of These Actions Necessary to Achieve the Objectives Efficiently. (Schedule) • The Resources ($$) Required to Support the Actions to Achieve the Objectives. (Budget)

The Triple Constraint Leads to Three Interrelated Aspects of the Project Plan

31 May 2013, 2100 35 The SE Way

• Customer develops requirements • PM leads and manages process • SE guides the development process • T&E informs the process • Customer gets the stuff

36 31 May 2013, 2100 The DOD Way

• DOD Acquisition has two customers – The Warfighter/User – The US Taxpayer/Congress

• Requirements from the User – T&E to Verify/Validate

• Cost and Schedule accountable to the Taxpayer/Congress – T&E to Understand Risks – Programmatic and Technology

37 31 May 2013, 2100 How the Military Works

Deploy

Requirements

Secretary of Defense Service Secretaries

Develop Build Buy

38 31 May 2013, 2100 What Happens to the SE “V”

39 31 May 2013, 2100 What Happens to the SE “V” More Stakeholders The PM

40 31 May 2013, 2100 DOD Acquisition – “The Rules”

Federal Title 10 USC Acquisition Armed Forces Regulation (FAR)

Defense Missile Acquisition Contracting Defense System DOD 5000

Army Air Force Navy Marines

Service Acquisition Regulations, Instructions, Guidance

31 May 2013, 2100 41 National Defense Authorization Acts (NDAA)

• National Defense Authorization Act is the name of a United States federal law that has been enacted for each of the past 48 fiscal years to specify the budget and expenditures of the United States Department of Defense. • Often additional provisions that deal with Defense development and acquisition – 1982 – Nunn-McCurdy Amendment, designed to curtail cost growth in American weapons programs – 1984 – DOT&E – 1996 - Clinger-Cohen Act (CCA), designed to improve the way the federal government acquires, uses and disposes information technology – 2009 – Weapon Systems Acquisition Reform Act (WSARA)

31 May 2013, 2100 42 DOD Acquisition

• The DOD has three principal decision-making support systems associated with military acquisition: – Planning, Programming, Budgeting and Execution (PPBE) Process - Process for strategic planning, program development, and resource determination. – Joint Capabilities Integration and Development System (JCIDS) - The systematic method established by the Joint Chiefs of Staff for assessing gaps in military joint warfighting capabilities and recommending solutions to resolve these gaps. – Defense Acquisition System - The management process used to acquire weapon systems and automated information systems, the operation of which is described in DOD 5000 regulation, instruction, guidance.

31 May 2013, 2100 43 The DOD Way • User Articulates the Needs (Sometimes) • JCIDS Process Develops Requirements (ICD/CDD) • Acquisition Community Develops Contract SOW with Contractor (SE Executer) • PPBE Process Provides the Funding • Acquisition Community Conducts T&E (DT&E) Informed Reviews (Programmatic and Technical) of SE Progression (PDR, CDR, DAES, MS-C) • Service/Agency OTAs and DOT&E Conduct Final Acceptance T&E (IOT&E) & Report to Congress (Representing the US Taxpayer) • User Gets the Stuff

44 31 May 2013, 2100 Operation of the Defense Acquisition System First Acquisition Framework in 1971

FULL- CONCEPTUAL SCALE PRODUCTION/ DEPLOYMENT EFFORT DEVELOPMENT

Program Full-Scale Production Initiation Development Go-ahead Decision Decision

• Decision points: 3 • Phases: 3 • Milestone documents: 1 (Decision Coordinating Paper (DCP))

31 May 2013, 2100 45 The Defense Acquisition Management System DODI 5000.02* • The Materiel Development Decision precedes entry into any phase of the acquisition management system User Needs • Entrance Criteria met before entering phase • Evolutionary Acquisition or Single Step to Full Capability Technology Opportunities & Resources

A B C IOC FOC Materiel Technology Engineering and Production & Operations & Solution Development Manufacturing Development Deployment Support ICDAnalysis

CDD CPD Materiel Post PDR Post CDR FRP Development Assessment Assessment Decision Decision Review AoA PDR PDR CDR Pre-Systems Acquisition or Systems Acquisition Sustainment

Initial Capabilities Capability Development Capability Production Document (ICD) Document (CDD) Document (CPD)

Relationship to JCIDS PDR: Preliminary Design Review IOC: Initial Operational Capability * DOD Instruction 5000.02, Operation of CDR: Critical Design Review FOC: Full Operational Capability FRP: Full Rate Production the Defense Acquisition System, December 8, 2008, USD AT&L 31 May 2013, 2100 46 What Happens to the Development & Acquisition Process

47 31 May 2013, 2100 The Forgotten Part of the Process

The Entire Solution For a Capability

D octrine O rganization Controlled/Developed by Service Chiefs T raining M ateriel Controlled/Developed by L eadership Service Secretaries P ersonnel R esources Big $

Big Industry

31 May 2013, 2100 Big Politics 48 Systems Engineering - Ideal

Validate Full System Requirements Test

Verify Integration Design & Test

Implementation

Time 31 May 2013, 2100 49 Systems Engineering – How it Happens

Full System Requirements Test

Validate Full System Requirements Test

Integration Design & Test

Verify Integration Design & Test

Implementation

Implementation Time

31 May 2013, 2100 Time 50 Systems Engineering – The Result

Full System Requirements Validate Test

Full System Requirements Test

Integration Design & Test Verify Integration Design & Test

Implementation

Implementation Time

31 May 2013, 2100 Time 51 Systems Engineering – How it Should Be

Validate Full System Requirements Test

Verify Design Integration & Test

Implementation

Time

31 May 2013, 2100 52 V&V Implications • DT is more heavily focused around verification – Among other uses, DT is a main player in deciding to accept items from contractors • OT, the focus is both were the requirements met and are they still the right requirements – This is a source of friction with PM’s as they believe they should only be judged by the requirements as stated not as currently needed.

31 May 2013, 2100 53 From: Dave Castellano, Deputy Director, Assessments and Support ODUSD(ATL)

• Over the past 10 years, DoD systems have experienced a 33% cost growth due to “RDT&E mistakes”…

• DoD IOT&E results, FY2001-2006 – 29 systems; mix of ACAT II, 1C, 1D across 3 Services – Approx. 50% were deemed “Not Suitable”, or partially NS – Approx. 33% were deemed “Not Effective”, or partially NE

• Preliminary study conducted by DOT&E (July 2007) determined that lack of Suitability is a significant life cycle sustainment cost driver – Reliability is the main component – Study revealed a strong correlation between reliability growth (mostly Testing and M&S) program investments and reductions in support costs

31 May 2013, 2100 54 …We Don’t Start Them Right • Insufficient requirements analysis and definition at program initiation – Not tangible, measurable, testable, stable – User R&M requirements are not underpinned by sound rationale • Acquisition strategies based on poor technical assumptions, competing budget priorities, and unrealistic expectations • Budget not properly phased • Lack of rigorous systems engineering approach • Schedule realism – success oriented, concurrent, poor estimation and/or planning • Inadequate test planning – breadth, depth, resources • Optimistic/realistic reliability growth – not a priority during development • Inadequate software architectures, design/development discipline, and organizational competencies • Sustainment/life-cycle costs not fully considered (short-sighted)

Red denotes areas in which testers can be especially helpful.

From: Dave Castellano, Deputy Director, Assessments and Support ODUSD(ATL) 31 May 2013, 2100 55 …We Don’t Manage Them Right • Insufficient trade space – Resources, schedule, performance, requirements • Insufficient risk management • Inadequate IMP, IMS, EVMS • Most programs lack quantifiable entrance/exit criteria • Maturing “suitability” (e.g., RAM) is not always a priority • Maturing “effectiveness” is not always a priority • Concurrent test program; inadequate scope due to schedule and resource insufficiencies, etc. • Inadequate OTRR process – no strong DT&E gate prior to IOT&E • Inadequate government staff; Inexperienced and/or limited contractor staffing • Poorly defined IPT roles, responsibilities and authority – Overall poor communications across government and industry staff

Red denotes areas in which testers can be especially helpful.

From: Dave Castellano, Deputy Director, Assessments and Support ODUSD(ATL) 31 May 2013, 2100 56 PMs Avoid Testers As Long As Possible…

• The Situation: – Gov’t program managers typically limit T&E involvement until absolutely forced to do so. • Perceived as adding schedule and fiscal expense in a time-period already under schedule and fiscal stress. – (Cynically) Do not want to ask the question if they might not be able to stand the answer. – We’ll make up lost time and fix the problems later in the last few weeks/months of the program known as T&E. – By forgoing T&E insights the program manager du jour might: • Escape problems on their watch but the DOD does not escape discovering problems and ultimately fixing them • Have their program cancelled or restructured therefore not fielding important warfighter capabilities

31 May 2013, 2100 57 A Word About USC Title X, DOT&E, OT&E, & OTA’s • Do OTAs & DOT&E Represent the User? • Do They Conduct and Evaluate “User Testing” - Is OT&E User T&E? • Involved mostly at end of the process • Report to: – User? – Congress • What happens when the User gets the stuff?

58 31 May 2013, 2100 The DOD Way Updated • Is the stuff ready for OT? • Congress Decides they Shouldn’t Wait Until the End of the Process • WSARA 2009 – Create DASD(SE) – Create DASD(DT&E) – More Emphasis on DT&E (Theoretically throughout the process) – However: Better DT&E does not directly translate to better informed acquisition and development and certainly does not enhance user involvement in the process

59 31 May 2013, 2100 A Word About System-of-Systems (SOS)

• SOS is the way almost all military equipment gets used • But it is the way almost none of it is developed • Recent activity on SOS T&E (mainly Army) – Mandates to conduct SOS T&E – Need to address SOS requirements – Need to manage development by Program of Programs – Can’t just integrate individual programs (Army ASA(ALT) SOS Integration Directorate challenges) – Interoperability testing and integrated testing are not SOS testing

60 31 May 2013, 2100 The Future • Can the Pure SE Way Work for DOD? – Not as Long as the User and Purchaser are Different Customers • Can a Hybrid Work? – Does the Current Hybrid Work? • Congress thinks not • The GAO thinks not • The User thinks not – A Better Hybrid? • COCOM Driven Rapid Acquisition? – Working, but…… – COCOM of current ops gets all the attention and resources – Other COCOMs of possible next conflict get little to nothing • Budget cutting will dictate: – Fewer new systems – More evolution – less revolution – More discipline and rigor – Better informed acquisition – T&E – Better Systems Engineering

61 31 May 2013, 2100

What T&E’rs Need to Know About

A Webinar Based On A Short Course By: Dr. W. David Bell Dr. C. David Brown SYSTEMS ENGINEERING A n d PROGRAM MANAGEMENT A n d W h y