Test& Evaluation

TEST& EVALUATION

Test and evaluation (T&E) ensures that acquired systems meet performance parameters and can be operated and maintained effectively to accomplish the mission. Considerable planning and resources are involved in testing the system, both before and after development. The Test and Evaluation Master Plan (TEMP) documents the test approach, schedule of events, and resource requirements. It supports the development decision at ADE-2B. T&E results support the production, deployment and support decision at ADE-3.

Multiple DHS organizations and players are involved in T&E. Some of the key players are: DHS Undersecretary for Science and Technology establishes T&E policy and processes for DHS acquisitions via the Director, DHS Test and Evaluation Standards Division (TSD).

Director of Operational Test and Evaluation (DOT&E) administers DHS T&E policy and processes and provides independent reports on the T&E of acquisitions to the Acquisition Review Board (ARB).

The Component Acquisition Executive (CAE) approves the TEMP and reviews 0perational T&E results.

The Operational Test Agent (OTA) plans, conducts, and reports independent operational T&E of Level 1 and other selected DHS programs. Operational tests are carried out by typical trained operators and maintainers using production-representative articles in a representative environment. The OTA must be independent of the developer of the system.

Modeling and Simulation

During system development, Modeling and Simulation (M&S) can be used to supplement T&E and provide insight into system performance without physically constructing the system. M&S can help developers better understand the relationships among subsystems and evaluate the expected outcomes of technical decisions. This management tool can reduce program costs, schedule slips, and technical risk by revealing weaknesses during the early stages of design.

M&S can reduce the amount of testing required, but it can never replace it entirely. Moreover, while this tool can be used to reduce technical risk, it also introduces the risk that models do not accurately reflect the system or portray the relevant environmental conditions. M&S requires resources, and it takes effort to verify and validate models and refine them as the system evolves.

Types of Test and Evaluation

There are two basic types of test and evaluation: Developmental T&E and Operational T&E:

Developmental Test & Evaluation (DT&E) supports the systems development process by verifying that the system attains the Critical Technical Parameters (CTPs) in the contract specification and the Operational Requirements Document (ORD). It is normally conducted by the contractor in a highly controlled environment.

Production Qualification Testing (PQT) verifies the effectiveness and efficiency of the manufacturing process.

Production Acceptance Testing (PAT) evaluates the function of sample items as they are produced.

Operational Test & Evaluation (OT&E) verifies that the system works in its intended operational environment. It is conducted by users, under realistic conditions in the field, to address Critical Operational Issues (COIs) that determine the system’s: Operational effectiveness--the ability of a system to accomplish a mission Operational Suitability--the degree to which a system is deployable and sustainable in the field or at sea. This takes into account logistics support considerations such as maintainability of equipment and training of operators. Independent Operational Test and Evaluation is conducted on production representative articles, typically manufactured during Low Rate Initial Production (LRIP). However, it is wise to involve the operational testers earlier in the life cycle. Operational Assessments (OAs) are evaluations of operational effectiveness and suitability by the Operational Test Agent prior to production in order to get an early look at the system in development.

Combined DT&E and OT&E testing, which a single test is used to support both types of testing, can save time and money. However, it requires extensive coordination, and it is difficult to design one test to satisfy both DT&E and OT&E objectives. Note: For more information about testing and evaluation, see MD 026-06, Test and Evaluation.