Dissertation Using the National Training Center Instrumentation System to Aid Simulation- Based Acquisition Andrew Cady This document was submitted as a dissertation in September 2017 in partial fulfillment of the requirements of the doctoral degree in public policy analysis at the Pardee RAND Graduate School. The faculty committee that supervised and approved the dissertation consisted of Bryan Hallmark (Chair), Joe Martz, and Randall Steeb. PARDEE RAND GRADUATE SCHOOL For more information on this publication, visit http://www.rand.org/pubs/rgs_dissertations/RGSD406.html Published 2018 by the RAND Corporation, Santa Monica, Calif. R® is a registered trademark Limited Print and Electronic Distribution Rights This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited. Permission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial use. For information on reprint and linking permissions, please visit www.rand.org/pubs/permissions.html. The RAND Corporation is a research organization that develops solutions to public policy challenges to help make communities throughout the world safer and more secure, healthier and more prosperous. RAND is nonprofit, nonpartisan, and committed to the public interest. RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors. The views expressed in this dissertation are those of the author and do not represent the official policy or position of the United States Air Force, or the U.S. Government. Support RAND Make a tax-deductible charitable contribution at www.rand.org/giving/contribute www.rand.org Abstract Though current data sources for simulation models used in the United States Department of Defense (DoD) acquisition process are many and varied, none adequately represents how weapon systems behave in combat in a robust, quantifiable manner, leading to uncertainty in the acquisition decision making process. The objective of this dissertation is to improve this process by developing empirically derived measures of direct fire behaviors from U.S. Army National Training Center (NTC) data and by demonstrating how these measures can be used to support acquisition decisions based on the output of simulation-based modeling. To accomplish this, I employ a three-part methodology. First, I identify the current data sources for models and simulations used in the defense acquisition process. Of these four data sources—historical combat, operational testing, other simulations, and subject matter expert (SME) judgment—no single source can adequately describe combat behaviors of weapon systems across a wide range of operational environments. Second, I turn to the NTC data as a potential solution to this gap in data sources. I first examine prior NTC-based research and lessons this literature holds for current and future research. I examine the doctrinal underpinnings of maneuver combat behaviors, deriving five important aspects of direct fire—two of which are operationalized in this dissertation: direct fire engagement and movement and maneuver. I examine the NTC instrumentation system data generation process, strengths, and drawbacks to determine if it could measure these. Finally, I describe four measures that I derive to describe maneuver combat behavior—weapon system probability of hit, weapon system rate of fire, unit dispersion, and unit speed. Third, I compare the measures derived in this dissertation against baseline measures from the Joint Conflict and Tactical Simulation (JCATS) simulation model to determine the difference the two sources of measures – actual NTC behavior and JCATS baseline - make in three outcomes: exchange ratio, drawdown of forces rate, and volume of fire. To perform this comparison, I create a scenario based on prior simulation studies along with four excursions to test the influence of changes in mission, enemy, and terrain on the impact of data source. I analyze the results of 300 runs of the JCATS model using a series of linear regressions. For each excursion, regression models indicate a highly significant effect of data source on each model outcome. I conclude this dissertation with a recommendation that the measures described herein form the basis for a larger system of NTC-based behavioral measurement for modeling and simulation (M&S) data. I also recommend several software and hardware improvements to the NTC instrumentation system that could improve its utility as both a data source and a training resource. As future research, I recommend applying advanced analytic techniques to these data, applying these methods to other combat training centers and applying these measures to training and tactics development. iii Table of Contents Abstract .......................................................................................................................................... iii Table of Contents ............................................................................................................................ v Figures........................................................................................................................................... vii Tables ............................................................................................................................................. ix Acknowledgements ........................................................................................................................ xi Abbreviations ............................................................................................................................... xiii 1. Introduction ................................................................................................................................. 1 Background............................................................................................................................................... 1 Objective .................................................................................................................................................. 3 Structure of This Dissertation ................................................................................................................... 4 Limitations ................................................................................................................................................ 7 2. Current Sources of Data for Combat Simulation Modeling ....................................................... 8 Introduction .............................................................................................................................................. 8 Historical Combat ..................................................................................................................................... 9 Operational Testing ................................................................................................................................ 14 Other Simulations ................................................................................................................................... 16 SME Judgment ....................................................................................................................................... 21 Discussion and Summary of Existing Data Sources............................................................................... 25 3. Deriving Behavioral Combat Measures from NTC-IS ............................................................. 27 What Is the NTC and How Can Its Data Be Useful? ............................................................................. 27 Historical Methods for Analyzing the NTC-IS ...................................................................................... 29 Doctrinal Overview of Key Behavioral Aspects of Direct Fire ............................................................. 39 NTC-IS Data Overview .......................................................................................................................... 47 Deriving Measures of Direct Fire Planning and Execution .................................................................... 53 Discussion .............................................................................................................................................. 59 4. Testing the Difference in JCATS Model Outcomes from Using NTC-based Data .................. 60 Model Selection ...................................................................................................................................... 60 Variable Selection .................................................................................................................................. 61 Experimental Design .............................................................................................................................. 65 Analysis Methodology ............................................................................................................................ 72 Model Results ......................................................................................................................................... 73 Discussion .............................................................................................................................................