Upstream Data Fusion: History, Technical Overview, and Applications to Critical Challenges

Andrew J. Newman and Glenn E. Mitzel

pstream data fusion (UDF) refers to the processing, exploitation, and fusion of sensor data as closely to the raw sensor data feed as possible. Upstream processing minimizes information loss that can result from data reduction methods that current legacy systems use to process sensor data; in addition, upstream processing enhances the ability to exploit complementary attributes of different data sources. Since the early 2000s, APL has led a team that pioneered development of UDF techniques. The most mature application is the Air Force Dynamic Time Critical Warfighting Capability program, which fuses a variety of sensor inputs to detect, locate, classify, and report on a specific set of high-value, time-sensitive relocatable ground targets in a tactically actionable time frame. During the late 2000s, APL began expanding the application of UDF techniques to new domains such as space, maritime, and irregular warfare, demonstrating significant improvements in detection versus false-alarm performance, tracking and classification accuracy, reporting latency, and production of actionable intelligence from previously unused or corrupted data. This article introduces the concept, principles, and applicability of UDF, providing a historical account of its development, details on the primary technical elements, and an overview of the challenges to which APL is applying this technology.

INTRODUCTION Sensor and Data Fusion The term “sensor and data fusion” refers to tech- duce new information and inferences and achieve more niques that combine data from multiple sources to pro- complete, clear, precise, accurate, and timely estimates

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) 215 A. J. NEWMAN AND G. E. MITZEL of the unknown quantities than could be achieved by Upstream Data Fusion Concept the use of a single source alone. Fusion of data from “Upstream data fusion” (UDF) refers to the process- multiple sensors provides several advantages over deriv- ing, exploitation, and fusion of sensor data as closely to ing estimates from a single source (see, e.g., Refs. 1 and the raw sensor data feed as possible within the limits 2). First, a statistical advantage is gained by combining imposed by technical feasibility and operational practi- independent observations. Second, the relative posi- cality. Upstream processing minimizes the information tion or motion of sensors can be exploited to provide loss that can result from the data reduction methods a geometric advantage in estimating kinematic states used by current legacy systems that process data from a and other attributes of an observed object. Third, the particular single source; in addition, upstream process- relative strengths and weaknesses of dissimilar data ing enhances the ability of the fusion process to exploit types can be, respectively, magnified and mitigated by the complementary attributes of different data sources. combining them judiciously. Fusing different data types The UDF process taps data at an appropriate point in from multiple sensor sources, in particular from different the processing chain near the sensor source (chosen to sensing phenomenologies (also referred to as modali- acquire the desired information content within engi- ties), broadens the number of physical observables avail- neering feasibility and without disrupting current opera- able to the fusion process, which results in significant tional processes) and bypasses the data reduction and improvements in detection, tracking, and classification detection thresholding steps inherent to a traditional performance as well as resistance to countermeasures single-sensor processing approach. It then exploits the and changing conditions. upstream data by tuning detection sensitivity to respond The U.S. military has acquired and currently oper- to faint signatures and discriminating between the true ates a diverse ensemble of intelligence, surveillance, and targets and the consequent large number of false can- reconnaissance (ISR) assets but tasks and exploits them didates by fusing data across complementary sensor in self-contained enterprises, often referred to as “stove- phenomenologies, diverse view geometries, and differ- pipes,” using a combination of automated and manual ent times. In addition, processing of raw (or nearly raw) processes that are highly specialized to a particular ISR sensor data allows the UDF algorithms to extract and asset, data type, application, or military domain and do exploit measurement data (e.g., target position) and not systematically interact with other such stovepipes. associated uncertainties (e.g., error statistics such as vari- This approach often fails to fully exploit complemen- ance or covariance) with the highest possible precision, tary capabilities and opportunities for collaboration as well as to exploit attribute data that are not normally (see, e.g., Refs. 3–6). The sensors provide an enormous reported in traditional processing chains. Applying UDF volume of data but often do not support precision to operational problems in the ground, maritime, and engagements of targets of interest because of deficien- space domains has demonstrated significant improve- cies in fusion and exploitation of the available data. ments in detection versus false-alarm performance, The process of tasking, collecting, processing, exploit- tracking and classification accuracy, reporting latency, ing, and disseminating ISR data is generally divorced and production of actionable intelligence from previ- from the rapidly evolving tactical picture and the needs ously unused or corrupted data. The benefits of UDF are of the end user (e.g., theater commander or exploita- more fully described in the UDF Benefits section. tion system operator). Improved sensor and data fusion A typical legacy downstream fusion process captures capabilities are needed to satisfy the warfighter’s accu- and fuses post-detection data from the available sources. racy, persistence, and timeliness requirements (see, e.g., Each of the individual sensor systems applies its own Ref. 7). processing, data reduction, and detection threshold- In general, the ability of a multiple-sensor ensemble ing to produce a set of candidate targets (e.g., ground, to provide persistent coverage on all targets, with high maritime, or space targets) in its data stream. In particu- accuracy, high detection probability, and low false- lar, each individual system is tuned to optimize its own alarm rate, will be constrained by limits on the number intrinsic performance, meaning that detection thresh- and diversity of assets, platform speed, maneuverabil- olds are set to relatively high levels to maintain a low ity, view angle, sensor coverage and update rates, sensor false-alarm rate while reducing the data processing load. resolution, and mismatch of target and sensor phe- This reduces the probability of detecting targets whose nomenologies. Coverage gaps, missed detections, false signatures may be only faintly observed in the sensor alarms, errors in the sensor support data, and classifier data (i.e., below the defined threshold). Moreover, data confusion will cause geo-location and classification that are identified as “bad” or “corrupted” are usually errors, incorrect associations, track loss or discontinui- discarded because they may not be reliable enough to ties, and spurious tracks, resulting in an ambiguous and support decision making within that individual process- unreliable tactical ground picture. Fusion and exploi- ing chain, even though these data can often reinforce tation systems must be robust enough to address these (or contradict) the information accrued from other realistic conditions. sensors. The performance of a downstream fusion pro-

216 JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) UPSTREAM DATA FUSION: HISTORY, TECHNICAL OVERVIEW, AND APPLICATIONS cess is therefore inherently limited by the reduced set dissimilar sensor modalities (intercepted signals, images, of thresholded data and the resulting limited number of conventional radar, and ground moving target indica- candidate detections that it receives. tor radar) supplying pieces of complementary upstream Figure 1 illustrates the concept of tapping and exploit- information that are traditionally exploited separately ing upstream sensor data to produce information that is and combined using only products intended for end-user not available through current means of processing single- consumption (if at all). sensor data. The UDF technique captures the raw or partially processed upstream data before decisions are UDF Principles made (i.e., before detection thresholds are applied) and performs an efficient multiple-level screening process UDF is based on the principles of efficient infor- to search the upstream data for candidate detections. mation processing and rigorous model-based evidence Screening thresholds are set very low to ensure that data accrual for multiple, possibly highly dissimilar, data that would otherwise be rejected are considered in the types. Efficiency is a key tenet because of the need to fusion process. This increases the probability of detec- extract all useful information from a potentially overwhelming volume of source data while maintaining tion of actual targets but also increases the number of computational tractability at achievable data transmis- false alarms that are passed to the fusion process. The sion rates. It is achieved through a modular, distributed false alarms will typically be rejected in the fusion pro- architecture in which cess, because, in general, there will not be corroborating evidence from other sensor inputs. In this manner, UDF • software components referred to as “screeners,” spe- discovers targets and activity that would not be found cialized to each sensor, process raw data from each using a legacy fusion process. sensor individually and transmit a dramatically When exploited in their traditional stovepipes, the reduced volume of essential data elements to the different data types, or sensing modalities, that are avail- fusion process; and able for a particular mission can produce very different observables, intermediate information products, and end • fusion and output conditioning software compo- products or releasable report formats. UDF efficiently nents, common to all sensors, exploit upstream extracts, accrues, and reports information by using spe- information from the screeners to quickly but judi- cialized screener, fusion, and output conditioning com- ciously eliminate statistically unlikely possibilities. ponents (described in the UDF Design Overview and Mathematically rigorous model-based evidence Primary Technical Elements section). Figure 2 illustrates accrual is a key tenet (Dynamic Time-Critical Warfight- an example of applying the UDF methodology to a set of ing Capability Feasibility Demonstration Plan, 23 August 2001) because of the need Sensor processing system 1 to properly combine mul- Decisions Sensor 1 tiple, possibly highly dissimilar, data types in situations Tap where each data type con- Fusion of lower- UDF Candidates Decisions tributes essential informa- delity post- screener 1 decision data tion to the fused solution but cannot resolve the targets or Sensor processing system 2 their key attributes individu- Decisions Sensor 2 ally. Each sensing modality produces a distinct set Tap of observables or measured UDF Candidates quantities that may not be screener 2 immediately comparable Fusion of higher- Decisions in form or dimensionality. delity pre- These must be combined decision data Sensor processing system n objectively (e.g., without Decisions Sensor n subjective weights) within a common mathematical Tap framework. Inferences must UDF Candidates be updated as the data are screener n received and must accurately reflect the accumulated Figure 1. UDF concept for tapping upstream sensor data and bypassing single-sensor process- information and aggregate ing stovepipes. uncertainty.

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) 217 A. J. NEWMAN AND G. E. MITZEL

Intermediate processing Product nishing Stovepipe Emitters, Releasable user product Signals Waveform Intercepts parameters reports Tap Tap Traditional UDF screener fusion for signals

Sampled Spectral/spatial Objects, Releasable Images spectra anomalies features reports Tap Tap UDF screener Fusion of for images conditioned Users upstream data Phase Objects, Releasable Radar history Contacts features reports Tap Tap

Sensor system stovepipes UDF screener for radar Output conditioning Detections, Releasable GMTI Waveform HRR pro les Tracks reports Tap Tap UDF screener for GMTI

Figure 2. UDF concept applied to several dissimilar sensing modalities. GMTI, Ground moving target indicator; HRR, high range resolution.

The screeners decide what data to transmit to fusion ciples. For example, it is critical to the evidence accrual- (e.g., candidate detections) and condition those data based fusion process that information supplied by the to prepare them for consumption by a fusion process screener be extracted purely from the source data, unal- adhering to the principle of mathematically rigorous tered by incorporating exogenous sources or assumptions model-based evidence accrual (a process referred to as (referred to as “prior-free” information). (For example, it “data conditioning”). The screeners are physically (e.g., is traditional and appropriate for legacy sensor systems to geographically) distributed, each as close as possible to incorporate terrain elevation data directly into products its corresponding sensor given practical considerations intended for end users.) A serious consequence of vio- (e.g., at a ground station or on board the sensing plat- lating the prior-free prerequisite is the risk of accruing form). They are also logically distributed in the archi- the same exogenous information repeatedly and redun- tecture, meaning that each screener is an independent dantly in fusion as if it represented new and independent functional unit that interfaces to the fusion process data. Also, for example, screeners typically output engi- through a predefined information exchange schema. neering support, measurement, and attribute data with Each screener is specialized to the sensor type and runs fidelity and precision that is not normally passed to an without interfering with the existing operational pro- end user but is necessary to associate observations with cessing flow. The screeners extract candidate detections objects under conditions of high rates of false alarms. with detection thresholds intentionally set low to maxi- The fusion component (or distributed set of fusion mize detection sensitivity so that faint signatures pass components) performs data association, kinematic state reporting thresholds. As a consequence, the screener estimation, and class estimation by applying statisti- passes a much larger number of potentially false detec- cal methods to the candidate detections from multiple tions than would normally be tolerated by a traditional screeners, over time, to identify candidate detections detection process reporting targets to a user or down- with true targets of interest, objects that are not targets stream fusion process. The screeners transmit only the of interest (e.g., persistent clutter), or random noise in essential data elements for candidate detections from the the sensor data. (Current UDF realizations use a central- upstream processing to the fusion process for correlation ized fusion component. Decentralized fusion is a subject and state estimation. of current research that is not covered in this article.) Data conditioning includes some commonly applied The data association process uses a multiple-hypothesis sensor data processing functions, such as extracting formulation, referred to as Multiple Hypothesis Data measurements and features, converting to common Association (MHDA), which applies Bayesian evidence units, and aligning to common reference frames, as well accrual (see, e.g., Ref. 8) to recursively evaluate likeli- as more subtle functions necessary to apply UDF prin- hood ratio statistics on multiple association hypotheses

218 JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) UPSTREAM DATA FUSION: HISTORY, TECHNICAL OVERVIEW, AND APPLICATIONS and enable the correct hypothesis to dominate compet- was followed by a larger conference at APL in March ing incorrect hypotheses over time. The rigorous appli- 2000. At the end of this conference, the three organiza- cation of likelihood-based statistical inference requires tions agreed to form a development team with APL as physics-based and empirically derived models of the sen- the lead. The concept was discussed and refined at the sors (with emphasis on error processes, precisely quan- National Correlation Working Group Workshop, held at tified uncertainties, and sensitivity of measurements to Ft. Monroe, Virginia, on 23–25 May 2000.9 states), targets (with emphasis on statistical distributions By August 2001 the development team was convinced of observable features and signatures), and background that the “Five-Minute War” concept was feasible (docu- environment. This approach properly considers a large mented in Dynamic Time-Critical Warfighting Capability number of uncertain possibilities without making early Feasibility Demonstration Plan, 23 August 2001). Around decisions or applying heuristics with limited applicabil- this time, the term “upstream data fusion” was coined ity. It exploits the complementary attributes of diverse to succinctly refer to the underlying enabling technol- sensor phenomenologies, sensor geometries, and data ogy. The team proposed the initiation of a major new collected over time to maintain a low system-level (post- research program to develop a prototype capability and correlation and report release) false-alarm rate. test it in a series of progressively more challenging dem- Output conditioning is a post-processing component onstrations. The first step, completed in March 2002, that prepares fusion output for use by a human user and was a more detailed investigation of potential benefits determines when the end product is releasable to the user. (as described in the UDF Benefits section) and demonstration of selected benefits by using previously collected sensor data. UDF History Following that success and under the leadership of The development of UDF at APL was initiated by two a visionary APL program manager, Bill Walker (Fig. 3, retired Air Force officers, the late Roy Robinette and the right), a multidisciplinary team was formed by adding 12 late Michael (“Cisco”) Francisco (Fig. 3, left and center, new industry partners to the original three, bringing the respectively). On the basis of their experience as combat total to 15 organizations and more than 200 individuals. pilots in Vietnam, they were convinced that a great deal The new partners were BAE Systems, Defense Consul- of operationally relevant sensor data was not being fully tants Ltd., Dynetics, Francisco & Associates Inc., Keith exploited by the military (later documented in Ref. 7). To S. Peyton, Lockheed Martin Corporation, Orincon, Sci- address the perceived deficiency, they developed a con- ence Applications International Corporation (SAIC), cept they called the “Five-Minute War.” The name was Scitor Corporation, Titan Corporation – Aerospace, chosen to emphasize the potential capability to quickly Veridian Systems Division Inc., and Zel Technologies, find and prosecute lethal threats during the short periods LLC. The team set a goal of conducting a live end-to-end of time when they would expose themselves to surveil- demonstration by May 2004. To handle the complexity lance. Roy and Cisco presented the idea to industry, but and risk associated with the proposed effort, the team it did not generate significant interest because of skepti- made two key decisions. First was the adoption of an agile cism that such a concept could ever be made practical. software development method. Because the concept was Consequently, they convened a nonprofit partnership so new, no firm requirements existed. Traditional software consisting of APL, Georgia Tech Research Institute, and development presumes that the technology and require- Draper Laboratory to assess the feasibility and utility of ments are available at the outset and are stable through- the concept. A preliminary conference was held at the out development. Agile methods accommodate frequent Air Force’s Unmanned Aerial Vehicles Battle Labora- adaptation and were, therefore, well suited to develop- tory, then at Eglin Air Force Base, in January 2000. This ment of the UDF prototype. After careful comparison of the available options, an agile software development method known as “Scrum” (see, e.g., Ref. 10) was selected. Second, the team adopted a philosophy that emphasized cooperation over competition, wherein the prototype was synthesized by choosing, modifying, and integrating pieces of algorithms and code contributed by team members. The team referred to the Figure 3. UDF pioneers Roy Robinette, Michael (“Cisco”) Francisco, and Bill Walker. (Michael cooperative synthesis approach Francisco photo courtesy of Air and Space Power Journal.) as “breed-the-best,” which is a

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) 219 A. J. NEWMAN AND G. E. MITZEL word play contradiction on the commonly used “best-of- fictional revolutionary artificial intelligence system that breed” approach whereby synthesis is achieved through gains self-awareness in the 1984 movie The Terminator). competition by choosing the best finished product or Later, on the day of the live demonstration, raw data component for each individual function. The develop- collected by operational sensors were combined auto- ment philosophy proved surprisingly effective in produc- matically by the prototype. At one instant, with local ing a working prototype within the schedule and funding and remote visitors watching the unscripted events, the constraints, leading the team to affectionately call the combined results showed a dramatic improvement over resulting product “stone soup.” observations seen by one sensor alone. Remote visi- This was truly a team effort, as shown by the many tors were not able to see the results immediately at that contributors across a number of organizations (see moment, but they did note the sudden eruption of clap- Acknowledgments). A few members of that team and its ping and shouting that they heard coming from team successors are mentioned in what follows with the sin- members at the demonstration site. cere hope that neither the authors nor the readers will After the successful May 2004 demonstration, diminish the contributions of others. (All contributors another demonstration was planned for June 2006 are or were affiliated with APL unless otherwise noted.) under more stressful conditions, including some target Dr. David Porter, Dr. James Christ, and Dr. Glenn Mitzel countermeasures. For this demonstration and to sup- developed the original functional architecture design port future development, an entirely new fusion algo- (an early, more detailed version of Fig. 1) in an intense rithm was designed by Dr. David Porter and the late Dr. 2 weeks of isolation. It was subsequently refined by the Larry Levy. This algorithm was designed from scratch larger team and became the framework for all other devel- to employ fully recursive logic and to treat ingested opment. The physical architecture was designed under modalities in a balanced and mathematically rigorous the leadership of Glen Long. The software architecture fashion. The algorithms were implemented in software was developed under the leadership of Steve Wieprecht. by John Florence and Dr. Andrew Newman. Additional The team applied the Scrum software development pro- improvements were made to the data screeners and to cess under the leadership of Will Menner. The screeners the overall software architecture. The demonstration for two of the sensing modalities were developed under in June 2006 was, again, highly successful. Despite the the leadership of Dr. Christopher Boswell, Dr. Patricia countermeasures, targets were reliably detected and Murphy, and Dr. Chung Fu Chang (Lockheed Martin). accurately located with very few false alarms. At one One screener design was based on prior work by Myron point, the Air Force Chief of Staff was invited to watch Brown and extended to include new target features and the live results at the Pentagon. Under the unscripted detection and geo-location techniques. The target fea- conditions, very little was happening during the short tures and the complex code required to extract them period he was able to visit. However, moments after he were contributed by multiple organizations, each having left a very successful event occurred. His subordinates decades of experience with the specific sensing modali- quickly sent him an e-mail message reporting the results. ties. For another sensing modality, a relatively mature Soon afterward UDF became known supportively as automated fusion capability was available and adequate “dumpster diving” and “Trash-INT” to emphasize how it for the first live demonstration. Applying the breed-the- could glean useful information from collected data that best philosophy, that capability was modified to accept would otherwise be discarded. raw data from other screeners under the leadership of The next live demonstration occurred in June 2008, Marshall Alworth and Mike Rector (both with Scitor which successfully showed the capability to process addi- Corporation; Alworth has since joined APL). Output tional sensors and report on additional target types. In conditioning, which turned the impossibly complex March 2009 the prototype was activated for the first output of the automated fusion capability into a final time with continuous operational feeds. This test was end-user product, was developed by Jeff Gilbert. conducted over a 2-week period, primarily to assess The demonstration in May 2004 was highly success- robustness. Operational availability, i.e., the fraction of ful. Real targets were deployed on test ranges where their time the prototype ran unattended, was extremely high. positions were precisely measured for comparison with The prototype was deployed as a limited operational reported locations. During the final preparations for the capability for nearly 3 months in the fall of 2009 and on demonstration, after one long day of work and with ini- several occasions since. During the test and assessment tial installation at the demonstration site complete, the process, the prototype has been undergoing continuous site team turned on the prototype and was ready to leave refinement and maturation based on these operational for the night. Suddenly the first live data appeared and experiences. An initial prototype deployment for use by the prototype processed the data. The site team stayed intelligence analysts at operational sites is planned for for hours convincing themselves that the results seemed the spring of 2013. reasonable. That event became known to the whole team Since 2008, APL has expanded the application as the day “Skynet went operational” (Skynet refers to a of UDF techniques to critical challenges and new

220 JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) UPSTREAM DATA FUSION: HISTORY, TECHNICAL OVERVIEW, AND APPLICATIONS domains including space, maritime, and irregular war- 1.0 fare, described in the UDF Application Areas section. In 0.9 addition, APL has recently initiated internal research 0.8 and development efforts to investigate the feasibility of 0.7 0.6 applying UDF technology to detection and character- 0.5 ization of underground facilities, protection of docked 0.4 submarines carrying nuclear weapons, and area defense 0.3 Single sensor 0.2 against terrorist attacks. Two sensors, one step upstream

Probability of detection 0.1 Two sensors, two steps upstream 0.0 1.00E–06 1.00E–05 1.00E–04 1.00E–03 1.00E–02 1.00E–01 1.00E+00 UDF BENEFITS Probability of false alarm

Detection Figure 5. Comparison of ROC curves for single sensor and UDF The improved detection capabilities of UDF can be one and two steps upstream. SNR = 10 dB. illustrated using a simplified analysis (originally docu- It is common to characterize sensor system per- mented in an APL internal presentation by Mitzel entitled formance by using a receiver operating characteristic “Illustration of the Potential Benefits of Upstream Data (ROC) curve, which plots probability of detection versus Fusion for Target Detection,” dated April 2001) with two false alarm rate over a range of detection thresholds. notional sensor systems with identical processing, deci- Figure 5 shows the performance gain when going from sion logic, and performance characteristics. For the pur- single-sensor detection processing (orange) to one-step poses of this analysis, the UDF concept simply depicted upstream fusion (blue) in which a Boolean AND opera- in Fig. 1 is further simplified to two sensors where the tion is applied to combine the two single sensor decision sensor processing for each sensor is assumed to consist of logics, and further to two-steps upstream fusion (green) computation of a decision variable and the application in which the decision variables are accessed and com- of decision logic to that variable. Figure 4 depicts three bined apart from individual decision logics by using a distinct approaches for determining whether a target is joint probability distribution function. present or absent given the source data from two sen- The single-sensor ROC comes from classical binary sors. First, decisions are produced from each individual detection theory. For normally distributed decision vari- sensor using its intrinsic processing and decision logic ables it can be expressed as (orange). Second, one step upstream (blue), a decision is m produced by combining the decisions made by the two PP= 11–––1 – k , dk,,fk k individual sensor processing chains. This approach may 8 ^ h B require some modification of the individual sensor deci- where k = 1,2 for the individual sensors;  is the cumula- sion logic to adjust the individual decision thresholds for tive distribution function of a zero-mean, unit-variance the purpose of maintaining a fixed probability of detec- normal random variable; m is the decision variable mean tion. Third, two steps upstream (green), a decision is (imposed only by the target);  is the decision variable produced using the internal decision variable of each of standard deviation; Pd denotes probability of correct the two individual sensor processing chains. detection; and Pf denotes probability of false alarm. The signal-to-noise ratio (SNR) is Decision therefore given by m/. The Signal variable Decision Decision orange line in Fig. 5 plots this Sensor 1 Processing logic curve for a common SNR of 10 dB (roughly 3.16). Fused decision Decision For this analysis, the logic two steps rule for fusing decisions one step upstream Fused decision Decision upstream is defined to require one step upstream that the sensors agree on detection decisions. Other logic rules Decision (such as those requiring at least Signal variable Decision Decision Sensor 2 Processing one detection) are possible but logic are not considered here. To Decision variable Decision logic achieve the same probability of X / , assumed to be N(0,1) if a target is absent X / T decide target is absent detection for the fused result as and N(m/ ,1) if a target is present (m/ = SNR) X / > T decide target is present for the individual sensors, the individual sensor probabilities Figure 4. Fusion decision logic applied at varying steps upstream. of detection must be increased

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) 221 A. J. NEWMAN AND G. E. MITZEL by lowering the thresholds. Assuming that the indi- tion of data from multiple sources inevitably requires vidual sensor statistics are independent, the composite assumptions about such matters as the independence probability of detection is the product of the individual of errors, temporal error behavior (e.g., from persistent probabilities of detection. Assuming that the decision biases), and error magnitudes. When the combination variables for both sensors are identically distributed, the is purely downstream, those assumptions are often made ROC one step upstream is given by without understanding or knowledge of the underlying error mechanisms. Pd,1 step upstream = UDF facilitates access to higher-quality engineering 2 –1 m 2 support data. It therefore enables better error accounting PPdf= 11–– ,1 step upstream –  . " , $ 8 ^ h B. and error modeling. Specific examples of the improvement include the elimination of conservative “fudge” fac- The blue line in Fig. 5 plots this curve for a common tors (ad hoc adjustable parameters with weak theoretical SNR of 10 dB. justification), isolation of common error contributions For fused decisions two steps upstream, this analy- (e.g., elevation data), incorporation of correlation dura- sis assumes simultaneous stimuli of the two sensors but tions for slowly varying biases (systematic errors in the with access to the individual decision variables them- sensor measurements), and the ability to estimate and selves. The new decision logic requires the sum of the compensate for bias errors by correlating across different two normalized decision variables to exceed a thresh- sensing modalities without relying on off-line calibration. old. It is easy to see that such decision logic may show Similarly, UDF facilitates access to raw data where subtle improvement over one-step upstream logic. For one- features for target classification can only be exploited. step upstream logic, both sensors must meet individual detection thresholds. For two-step upstream logic, the individual thresholds are irrelevant. A weak decision Timeliness variable measurement in one sensor can be compensated UDF bypasses some of the downstream processing for by a strong measurement in the other variable as long individual sensors (as shown previously in Fig. 2), which as the sum meets the specified sum threshold. The sum typically includes processing that primarily conditions is normally distributed with zero mean and a standard the individual sensor output product for human con- deviation of 2 m/. Therefore, using the same reason- sumption. This reduces system processing latency. Fur- ing as in the single-sensor case but with the summed thermore, the UDF process is implemented recursively, decision variable, so that a very small amount of additional data and processing from one sensor may become the tipping point Pd,2 stepsupstream = for reporting given all the other previous data. –1 m Improvements in timeliness can be dramatic. For 11–– Pf,2 steps upstream – 2  . 8 ^ h B example, for a single sensor, the number of observations to reach a reporting threshold may be large and When comparing this with the ROC for a single may take a significant amount of time to collect. How- sensor, it is apparent that this is equivalent to a gain in ever, in the UDF approach, if a target has been barely SNR of 2 (~3 dB). The green line in Fig. 5 plots this missed in previous sensor observations but the memory curve for a common sensor SNR of 10 dB. of that detection has been recursively retained (consis- For a fixed probability of detection it is obvious that tent with the principle of evidence accrual described in the false alarm rates will be substantially reduced as the UDF Principles section), then the first indications the fusion decision logic is applied further upstream. of the target from another sensor, although themselves For instance, at a detection probability of 0.80 the false not reportable, may result in a combined detection long alarms are reduced by more than a factor of 10 at one step before enough observations from the single sensor would upstream and nearly 100 at two steps upstream. Con- have been collected and exploited. versely, for a fixed false alarm rate, the detection probability will be improved. This implies that the system will achieve detections that would otherwise go unre- Robustness to Countermeasures ported. Although the illustration is simplistic, detection UDF is a logical response to countermeasures that gains of these types have been repeatedly demonstrated are designed to reduce target observables, to confound in the field. a limited set of modalities, or to restrict the exposure of key target signatures. UDF exploits more subtle target Other Benefits features and scales to include more modalities. More- over, as explained in the previous section on timeliness Target Location and Classification benefits, even the most fleeting target signature may Fusing location estimates from multiple independent provide a sufficient amount of corroborating evidence to sources generally improves accuracy. But the combina- detect and report the target.

222 JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) UPSTREAM DATA FUSION: HISTORY, TECHNICAL OVERVIEW, AND APPLICATIONS Required Computational and Communications Capacity Screeners By deploying screeners as far upstream as possible and Each screener ingests data directly from the source, judiciously controlling the release of fused information or with the minimum intervening transmission and pro- to the user according to rigorously computed uncertain- cessing that is feasible, practical, and advantageous to ties and confidence factors, the UDF framework allows system performance. The screener often resides on the a significant reduction in the communications capacity sensor platform or the processing element that it directly required to transmit data among networked components feeds in its traditional processing chain. Screener pro- and to end users. This reduction is particularly dramatic cessing is specialized for the particular data type, using for sensing modalities producing extremely high volumes detailed physics-based and empirical models of the spe- of data such as full-motion video (FMV) and wide-area cific sensor phenomenology. Therefore, screening algo- imaging. End-to-end reductions in required communica- rithms must be developed for each distinct sensor type, tions capacity of many orders of magnitude have been and a screener software component is required to process demonstrated in field tests. In addition, the overall com- the upstream data from each individual sensor. putational load is distributed and the processing capacity The essential functions of each screener are to extract of each individual component can be chosen according candidate targets from upstream sensor data, measure to its specific loading profile. observables, convert all observable data to a common format and units, align data to common reference frames Asset Employment Efficiency where necessary, estimate measurement error with the maximum possible precision, and assure that informa- UDF not only improves awareness of those targets for tion produced is prior-free. High precision prior-free which it is certain, but it also keeps careful track of the observable data are essential for fusion to accurately and ambiguities and uncertainties about targets that are not faithfully compute and represent estimate uncertainties yet reportable. This additional information facilitates and association probabilities. the optimal use of limited surveillance assets to collect In general, each screener may also perform pattern the last piece of accruable evidence as opposed to initi- classification algorithms using the extracted feature ating a new search. Also, accurate assessments of uncer- data to produce conditional likelihoods of the target tainty and ambiguity form a basis for choosing the assets belonging to particular object classes (e.g., target types, and focusing their attention where it will be most useful. persistent clutter, and random clutter). However, the This UDF benefit forms the technical foundation for screener does not perform a traditional target recogni- a more comprehensive concept referred to as “Closed- tion or classification function. It produces only candi- Loop Collaborative ISR” (CLCISR), which is the subject dates and conditional likelihoods rather than making of the article by Newman and DeSena in this issue. final decisions about target class or identity. Feature distributions for objects of interest can be estimated as Gaussian mixture densities by applying maximum UDF DESIGN OVERVIEW AND PRIMARY likelihood expectation maximization and data mining TECHNICAL ELEMENTS techniques to empirical data. In addition, the screener may also provide the functionality to deal with cor- UDF Processing General Framework rupted measurements and dynamically calibrate sensors The general framework for UDF processing consists to remove bias error. of several distributed upstream screeners performing To keep pace with potentially high volumes of sensor detection and data conditioning functions; one or more data, low-complexity design of screener algorithms fusion elements performing data association, kinematic and models for fast computational processing is essen- state estimation, and class estimation functions; and an tial. The techniques and algorithms must be robust to output conditioning element, each of which is described varying conditions (e.g., illumination, collection geom- in general terms in the sections below (some references etries), be adaptive for new targets and backgrounds, and are provided for detailed descriptions, although in many exploit all available information (e.g., spatial, temporal, cases the details are program specific and access may be and spectral). Screener design often applies a multilevel limited). These components, and to a lesser extent the screening algorithm to efficiently search the upstream framework itself, must be specialized for the particular source data in near real time to find candidate targets sensor types and mission while adhering to the general in the data stream. For example, the first-level screening UDF design principles. Current realizations employ a may apply a fast, coarse anomaly-detection technique centralized fusion element, although decentralized fusion to identify a very large number of candidate targets. designs are feasible (see, e.g., Ref. 11) and are the sub- Second and higher levels of screening, as appropriate, ject of current research. The general framework is illus- progressively screen the remaining candidates, applying trated in Fig. 6, which shows one possible realization for more sophisticated algorithms such as pixel-level, object- a ground target surveillance and reconnaissance mission. level, and feature-level algorithms (see, e.g., Ref. 12).

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) 223 A. J. NEWMAN AND G. E. MITZEL

Sensor processing system 1 Target location, error ellipse, UDF screener for sensor 1 target class, and Feature con dence score Releasable Sensor 1 Tap Tap Candidate Detections Feature vectors Object track extraction computation classi cation groups

Output Sensor processing system n conditioning UDF screener for sensor n Groupingng Feature Target class Sensor n Tap Tap Candidate Detections Feature vectors Object likelihoods extraction computation classi cation Releasability

Observables Fusion of conditioned upstream data and error statistics Target class estimation Candidate Discrete Bayes Kinematic state classi er estimation tracks Linear and nonlinear Target location estimates, Data association ltering error covariance (Fisher info), Multiple hypothesis sensor bias estimates and Bayesian Sensor bias sigmas, and lter residuals estimation Target class Validation Kinematic-based likelihoods Location, gating likelihood scoring error stats, p( y | H i ) Target location and target class Tracks L estimation k (associations) and probabilities Cluster Track likelihood Lf Feature-based management scoring likelihood scoring Tracks Altitude LLR L DTED re nement L= k + Hypothesis Hypothesisf Hypothesis Hypothesis expansion scoring pruning cleanup

Candidate tracks

Figure 6. One possible realization of a general framework for UDF processing. DTED, Digital terrain elevation data; LLR, log likelihood ratio.

Most importantly for UDF, the screening thresh- result is a complete partition of the set of screener detec- olds applied by screening are intentionally kept low to tions into subsets corresponding to real objects in the maintain a high probability of detecting actual targets of observed physical space or spurious noise (false alarms). interest. As a consequence, the screener passes a much Each possible such partition is referred to as an associa- higher number of detections, which initially produces a tion hypothesis (see discussion of multiple hypothesis much higher false alarm rate than would be tolerated techniques below). Kinematic state estimation typically by a traditional detection process reporting targets to a refers to the process of inferring the position and motion user or downstream fusion process. The UDF framework variables (e.g., geo-location, velocity, or orbital elements) relies on data association across sensor phenomenolo- of detected objects. Object class estimation refers to the gies, sensor geometries, and data collected over time to process of inferring the types of detected objects chosen maintain a low system-level (post-correlation and report from a set of discrete categories. In certain applications, release) false alarm rate. at a higher fidelity, target identity may also be inferred. These functions within the fusion element are tightly coupled, as described below, and are illustrated by exam- Fusion ple in Fig. 6. The fusion element (or elements) performs data The data association and state estimation func- association, kinematic state estimation, and object tions process candidate detection data from all sensor class estimation. Data association refers to the process screeners asynchronously. This is possible because the of determining the origin of each detection produced screeners are designed to report observable data by using by the screeners as an object already observed (and common data formats and units known to the data asso- tracked), a new object not yet observed, or a false alarm ciation and state estimation functions. However, the induced by random phenomena. This includes correla- techniques applied depend on expected target properties tion across the various sensor types and over time. The and behavior.

224 JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) UPSTREAM DATA FUSION: HISTORY, TECHNICAL OVERVIEW, AND APPLICATIONS

The multiple hypothesis methodology is essential to t1 2 detects t2 2 detects data association in the UDF framework where screener detection sensitivity, and consequent false alarm rates, is set intentionally high. The MHDA approach [often referred to as Multiple Hypothesis Tracking (MHT) in the literature because of its common application to moving target tracking problems13–16] applies Bayesian evidence accrual to recursively evaluate likelihood ratio statistics on multiple association hypotheses and enable the correct hypothesis to dominate competing hypotheses over time. As the term suggests, “evidence accrual” t3 3 detects t4 1 detect refers to a technique for combining information from different sources and over time in which estimates and decisions are updated according to physical and statistical models whenever new data become available. Bayes- ian evidence accrual, or Bayesian inference, applies Bayes’s law relating prior and posterior probability densities of the estimated parameters. It provides a straight- forward and theoretically sound method of recursively updating belief about unknown random variables by t4 truth hypothesis incorporating information from different sources. A true likelihood ratio test is a common statistical test used to false compare the fit of two models to the available data. The log likelihood ratio is the test statistic that quantifies the relative fits. For MHDA, the log likelihood ratio statistic is computed for each association hypothesis under fasle consideration, with the hypothesis that all detections alarms are false alarms serving as the basis of comparison (see Refs. 14 and 15 and references therein for more details). The MHDA methodology generates alternative data association hypotheses whenever the data cause an Figure 7. Example MHDA association of detections into tracks ambiguous or conflicting situation (illustrated by exam- over four time steps. ple in Fig. 7). Each hypothesis is a complete and noncon- flicting set of detection-to-track associations. Decision dynamically estimate observable sensor bias errors where making is delayed by propagating hypotheses in antici- applicable. Recursive linear and nonlinear filtering pation that subsequent data will resolve the uncertainty. techniques are used to update target and bias state esti- Association hypotheses are evaluated, ranked, con- mates with new measurement data and, depending on firmed, and denied via recursive computation of likeli- the application, to propagate estimates forward in time hood statistics derived from the input data and model to correlate with the next set of available measurement parameters. As each new set of measurements is made data. In general, the MHDA algorithms must properly available to the data association process, the algorithm account for cross-track correlations in the bias states. considers the possible ways that the new measurements The class estimation function applies Bayes’s law and the can associate with the existing tracks. Using precise law of total probability to update a discrete probability kinematic information, and based on the consistency distribution on the class of each candidate target object of the measurements with the target (or background) (see, e.g., Refs. 17 and 18). The MHDA computes the log models, the system uses Bayesian likelihood techniques likelihood score for each candidate association of obser- to calculate likelihood measures for the hypotheses. vations (that form a track) recursively with a data update Unlikely hypotheses are pruned, and the probability that combines a kinematic component derived from the mass is re-normalized over the remaining hypotheses. filter residuals and residual covariance (or square root In this manner, true targets are reinforced by evidence information quantities) and a feature-based component from multiple sensor modalities and will pass the report- derived from classifier-generated conditional probabili- ing thresholds; false targets decorrelate and are rejected. ties of detected objects in feature space. The data association function operates in conjunc- The UDF prototypes developed by APL have been tion with the state and class estimation functions. The based on the Reid MHT formulation13 and modern vari- state estimation function estimates the kinematic state ants.14 –16, 19 Traditionally, MHT couples MHDA with of all candidate target objects. It may also be used to independent track filters, commonly Kalman or iterated

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) 225 A. J. NEWMAN AND G. E. MITZEL extended Kalman filter variants that are efficient recur- approach inherently presents situations in which two or sive estimators of the state of a dynamical system from a more competing hypotheses are similar, cannot be fully series of noisy measurements. The APL prototypes instead resolved given the available data, and the remaining use an iterated extended square root information filtering differences are not important for the given application. (SRIF) technique19, 20 for the track filters to recursively Presenting these close, but distinct, hypotheses to a user estimate target and bias states from sensor measurements would be confusing and distracting and would reduce and altitude data.21 The SRIF algorithm has advantages the operator’s confidence in system performance. The over the Kalman filter and its other variants in terms output conditioning function (internal presentations of numerical stability and precision, computational effi- and private communications, J. M. Gilbert, APL, 2003– ciency, and amenability to distributed processing. The 2011) resolves this ambiguity in the MHDA output by MHDA and SRIF implementations estimate and com- statistically aggregating similar association hypotheses pensate for sensor bias errors within the core algorithms, to produce logical entities, called “groups,” that repre- including approximate accounting for cross-track correla- sent possible targets. These groups are the main objects tion (internal presentations and private communications, of interest in determining releasability of target reports D. W. Porter and L. J. Levy, APL, 2003–2006). to the tactical user and computing metrics on fused The APL prototypes have been designed to run in a output. Output conditioning provides a rigorous means multiple-sensor, multiple-target setting where the sensor of aggregating two or more hypotheses or target tracks data may include a mixture of dissimilar data types. The that individually are too ambiguous or have insufficient fusion processing is augmented with support for process- confidence to pass a releasability threshold but can do ing ambiguous, nonlinear, and biased measurement,22 so in aggregate. as well as processing of dissimilar data types with performance that is robust to different orders of the input data.23 The kinematic state estimation function employs UDF APPLICATION AREAS and switches between linear and nonlinear filtering techniques at different steps in the algorithm as needed Relocatable Time-Sensitive Ground Targets to balance the need for estimation accuracy with the The most mature application of UDF is the Air need for computational tractability. The various realiza- Force Dynamic Time Critical Warfighting Capability tions that have been developed at APL are equipped with (DTCWC) program currently managed by the Air Force specialized capabilities to process and fuse specific data Research Laboratory (AFRL). DTCWC applies UDF to types such as angle measurements from screening opti- a variety of sensor inputs to detect, locate, classify, and cal imagery, range and range rate measurements from report on a specific set of high-value, time-sensitive relo- radar sources, and other data types relevant to ground catable ground targets in a tactically actionable time- and maritime surveillance applications such as time difference of arrival from RF signal intercepts and moving target indicator (MTI) contacts from radar and video sources. Some realizations also include a multiple model capability for tracking targets exhibiting different types of motion behavior.

Output Conditioning Output conditioning refers to the process of aggregating and resolving competing hypotheses to provide target reports to the end user. Fusion and output conditioning are tightly coupled and require a coher- ent design to function properly together. The MHDA Figure 8. UDF of diverse sensors against relocatable ground threats.

226 JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) UPSTREAM DATA FUSION: HISTORY, TECHNICAL OVERVIEW, AND APPLICATIONS frame.24–27 Figure 8 illustrates a prototypical mission detection reports were made accessible to trained per- supported by DTCWC. An automated UDF capability sonnel for use in supporting operations, collections, and is the critical enabling technology for DTCWC because analytic decisions. During 2012 and 2013, DTCWC has of the need to prosecute evasive targets employing coun- been deploying an initial prototype for use by intel- termeasures (e.g., frequent relocation, camouflage, and ligence analysts at operational sites as the first step in other mitigation of observables and spoofing of signa- transitioning to an operational capability. tures) during short periods of time when they expose The field demonstrations and LOC assessments themselves to surveillance. DTCWC has successfully have confirmed the feasibility and operational utility of demonstrated an automated end-to-end capability to UDF in support of the DTCWC mission against a set fuse live data feeds in several controlled field demon- of relocatable time-sensitive ground targets. They have strations on test ranges (May 2004, June 2006, and June shown that UDF can achieve high detection probability 2008) as well as in uncontrolled limited operational (e.g., detecting targets that would have been missed by capability (LOC) assessments using operational sensor a single-sensor system) with very low false alarm rates, data against real targets in operational environments and geo-location accuracy capable of supporting target (several during 2009–2012). engagement. Furthermore, they have shown the ability The early history of what was to become DTCWC, to overcome corrupted data, whereby information from including some details about the early field demonstra- one sensor system was used to automatically reject poor tions, is described in the UDF History section. The 2004 measurements from another system, and to automati- field demonstration validated the technical feasibility cally estimate and remove certain types of bias errors. of the UDF concept as applied to the DTCWC mission. The 2006 field demonstration showed the feasibil- Maritime Situational Awareness ity of the MHDA-based fusion approach, the capability to perform successfully in a more challenging natural The Navy needs to track ships globally to coun- 29, 30 environment under more realistic conditions including ter resurgent and emerging surface naval threats. some target countermeasures, and the capability to pro- Worldwide ship tracking typically relies on a combina- vide timely actionable target information to users in a tion of marine radar and vessel self-reporting, such as targeting cell. the Automatic Identification System (AIS), a system In June 2008, DTCWC was exercised over multiple whereby ships broadcast their unique identification, days by using live national and tactical sensor data feeds position, course, and speed. However, these sources of to demonstrate and assess new capabilities including information are not adequate for tracking many impor- integration of a new tactical sensor, reporting on a new tant vessels that are of interest to the Navy and may be target of interest, a reengineered fusion engine, and a attempting to hide in regions of dense maritime traffic, browser-based user interface. DTCWC was operated conceal their identity and position, or otherwise defeat from an Air Force distributed ground system site, which surveillance. controlled the distributed system components resident The maritime surveillance problem presents unique at data collection sites. DTCWC achieved report- challenges in terms of the extremely large area required ing accuracy, a false alarm rate, an estimate error, and for coverage, long track durations, and high density of performance latency that were all within operational vessels in many areas of interest (Fig. 9). Available sensor parameters for tasking the new tactical sensor in its tip- types include surface MTI radars, a variety of imaging ping and cueing role. modalities, and passive systems such as AIS and RF In March 2009, DTCWC was run in an operational signal receivers. However, there is no single sensor that setting for the first time, with the primary objective of detects, tracks, and identifies ships throughout their assessing system availability for users. The prototype ran voyages over wide areas.31 Moreover, individual sensors nearly continuously for more than 2 weeks. With that suffer from weaknesses such as periodic coverage, narrow success to bolster the case, in the fall of 2009 DTCWC coverage, unreliable detection, inaccurate kinematic was assessed at a Technology Readiness Level of 7 by an information, imprecise identification, and high sensitiv- Air Force-led, independent assessment team. The results ity to weather conditions. It is necessary to combine and of these UDF demonstrations and assessments were rec- correlate data from disparate surveillance sensors. This ognized and documented by the Joint Defense Science is further challenged by dense shipping backgrounds and Board Intelligence Science Board Task Force on Inte- unpredictable latencies in data transmission that cause grating Sensor-Collected Intelligence.28 data from different sources to arrive out of sequence rela- From October 2009 through January 2012, the tive to collection time. DTCWC program conducted a series of LOC assess- In a series of efforts using real and simulated sensor ments during which DTCWC processed live operational data, APL showed the potential of UDF to improve data from supported sources and produced detection maritime situational awareness by screening and fusing reports within specified areas of interest. DTCWC upstream data from a variety of operational sensors that

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) 227 A. J. NEWMAN AND G. E. MITZEL

catalog of most, if not all, objects in orbit (Fig. 10). Traditional space surveillance and tracking systems operate with high detection thresholds with an emphasis on rejecting false alarms. The post-detection data are fused to produce orbit tracks. In addition, processing and exploitation of metric data (e.g., satellite position and velocity) and feature data (e.g., pho- tometric and radiometric signatures) are traditionally performed in separate Figure 9. Global maritime ship traffic density. (Reproduced with permission from “Maritime Ship- organizational stovepipes. ping Routes and Strategic Passages,” The Geography of Transport Systems, http://people.hofstra. Traditional methods have edu/geotrans/eng/ch1en/appl1en/maritimeroutes.html, © 1998–2012, Dr. Jean-Paul Rodrigue, Hof- performed well in finding, stra University, New York.) tracking, and identifying are currently exploited separately, and to do so without larger objects in Earth orbit, but smaller objects, includ- interfering with their current operational procedures ing a variety of potential threats, are often lost or missed or capabilities. These efforts emphasized the benefit of entirely. In addition, current methods are severely chal- fusing sensors that provide strong target feature infor- lenged when satellites maneuver from their expected mation with sensors that provide persistent area cover- orbital trajectory or are in close proximity to (and hence age and precise target location information. They also possibly confused with) other satellites. Enhancements showed the adaptability of the UDF framework, with to persistence and timeliness of space situational aware- relatively minor modifications to current realizations, to ness (SSA) are needed to deal with stressing cases such maintaining track custody of maneuvering surface tar- as dim, closely spaced, and maneuvering objects. gets over long periods of time while traveling through In 2009–2010, APL completed a project sponsored dense backgrounds. by the Air Force Space and Missile Systems Center to In this work, track-oriented MHT techniques16 were investigate the operational utility of UDF techniques applied to more efficiently process sensor data for long- to detection and tracking of dim resident space objects duration tracking (by contrast, the DTCWC time- (RSOs) and discrimination of closely spaced RSOs using sensitive targeting mission forced an emphasis on track space surveillance sensor data. The project executed a initiation for targets with short exposure times). Multi- ple motion model filtering techniques (see, e.g., Ref. 32) were applied to enhance tracking of maneuvering ships. A novel nonlinear filtering technique, using an iterated extended SRIF, multiple motion models, and the Dyer–McReynolds smoother,33 was developed to enable tracking of maneuvering ships using out-of-sequence measurement data. The results showed that, even in dense ship traffic environments, the UDF approach can reinforce the existing sensor track picture.34 This improvement could provide valuable intelligence to the warfighter on existing tracks and enable cues to vessels of interest entering the platform’s area of regard.

Space Situational Awareness As our reliance on space systems and space technology grows, so does the potential for collisions and other spacecraft interactions, leading to potentially serious Figure 10. Computer-generated image of objects in Earth orbit. consequences. It is essential to maintain an accurate (Image from NASA Orbital Debris Program Office photo gallery.)

228 JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) UPSTREAM DATA FUSION: HISTORY, TECHNICAL OVERVIEW, AND APPLICATIONS

10-day triple-site collection campaign (October 20–29, macy and influence over the relevant populations.” It 2009) using three geographically separated Raven-class is characterized by the use of “indirect and asymmet- optical sensors (see, e.g., Ref. 35) at Albuquerque, New ric approaches, though it may employ the full range Mexico; Maui, Hawaii; and Kwajalein Atoll, Marshall of military and other capabilities, in order to erode an Islands. The campaign was the first historical instance adversary’s power, influence, and will.”39 The types of of long-baseline simultaneous collections at three sites operations conducted as part of irregular warfare include by Raven-class sensors on common objects. The project counterinsurgency, counterterrorism, and unconven- developed an optical image screener that detects RSO tional warfare. These operations typically require with weak signatures, extracts target features, and gener- time-sensitive prosecution of hidden, fleeting, and ates precise measurement error estimates. The screener maneuvering targets, including individual people. In operates with low detection thresholds to find candidates response to stressing demands on persistence and timeli- with weak signatures in the upstream data. The project ness of ISR, the military has fielded a large number of also adapted the MHDA component used for ground surveillance assets to theaters of operation, producing surveillance applications to the space domain by incor- an overwhelming volume of imagery, FMV, and other porating the appropriate sensor measurement models types of sensor data. UDF technology has the potential corresponding to the Raven-class optical sensors and to mitigate the enormous demand on communications target dynamics models for propagating satellite motion. and computing infrastructure and on human analysts These UDF prototype components processed raw imag- and operators and to turn all of these data into timely, ery, detected a variety of geosynchronous satellites, and actionable intelligence. fused the extracted measurement data to track the satel- APL recently demonstrated a prototype UDF capa- lites with accuracy that was an order of magnitude better bility at the Joint Expeditionary Force Experiment 2010 than that available from the Space Catalog. They also (JEFX 10).4 0, 41 JEFX is a Chief of Staff of the Air Force- discriminated closely spaced geostationary satellites that directed series of experiments that combines live, vir- are commonly cross-tagged by current operations (see tual, and constructive forces to create a near-seamless Refs. 36 and 37). warfighting environment in which to assess the ability During 2010–2011, APL applied its UDF prototype of selected initiatives to provide needed capabilities to as a major component of a project sponsored by AFRL. warfighters. JEFX 10 was focused on irregular warfare The aim was to demonstrate proof of concept for a semi- and assessed emerging technologies and capabilities automated dynamic sensor tasking capability with the against several relevant operational threads, including goal of supporting Joint Space Operations Center’s rapid convoy protection, urban targeting, raid insertion, and decision making in scenarios where the current delib- personnel recovery. The JEFX 10 infrastructure featured erative, manually intensive process for tasking the U.S. an airborne network used to share information among Space Surveillance Network (SSN) is insufficiently airborne platforms and ground nodes, enabling dynamic responsive. The project used UDF software components command and control of airborne platforms and weap- operating in a closed feedback loop with sensor resource ons. The JEFX 10 live fly experiment took place at the management software components to continually retask Nevada Test and Target Range near Las Vegas, Nevada, space-observing sensors to respond quickly to urgent in April 2010. information needs while maintaining minimal aggregate The UDF initiative supported Navy Commander uncertainty (one realization of the CLCISR concept Second Fleet (COMSECFLT) Maritime Operations described in the Asset Employment Efficiency section). Center (MOC) participation in JEFX 10 with the goal of These components were exercised within a dynamic enhancing the MOC ISR tasking, processing, exploita- SSN sensor tasking simulation test bed, which was tion, and dissemination capability and overall maritime used to conduct simulation-based performance assess- domain awareness. The UDF prototype was deployed at ment experiments and to quantify performance benefits the COMSECFLT MOC, Naval Station Norfolk, during in terms of track maintenance (target custody), search the period April 12–23, 2010. The specific objective was efficiency, and responsiveness to emergent information to support the COMSECFLT MOC in executing irregu- needs and changing priorities (see Ref. 38 for details). In lar warfare operational threads by providing intelligence 2012, APL continued this work in support of the AFRL analysts and the collection manager with a common effort on the Ibex program, which was jointly spon- tactical picture of concurrent operations, precise geo- sored by Defense Advanced Research Projects Agency referenced track information from multiple sources, (DARPA) and Air Force Space Command. and the ability to exploit large volumes of data rapidly and efficiently. The UDF prototype was specialized to consume and Irregular Warfare process FMV data collected by a team of unmanned The U.S. DoD defines irregular warfare as “a violent aerial vehicles (UAVs) (illustrated in Fig. 11). The struggle among state and non-state actors for legiti- JEFX 10 operational threads used several Raven and

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) 229 A. J. NEWMAN AND G. E. MITZEL

UAV 1

High bandwidth UAV 1 video Video screener for UAV 1 processing system Moving object detection Low bandwidth Tap Feature Image Object tracking registration localization Video and Detections telemetry Pixel coordinates Metadata Fusion

MHT DTED

UAV n Situational awareness display fused common track picture

High bandwidth UAV n video Video screener for UAV n processing system Moving object detection Tap Feature Image Object tracking registration localization

Figure 11. JEFX 10 concept for UDF of multiple UAV FMV sources. USGS, U.S. Geological Survey; DRG, digital raster graphic.

Buster UAVs as the primary FMV platforms. The pro- More recently, in 2011, APL’s Precision Engagement totype consisted of a video exploitation component Business Area’s Internal Research and Development that automatically detected moving objects (shown in (IR&D) projects extended the UDF-based FMV exploita- Fig. 11 and by example applied to JEFX 10 in Fig. 12), tion capability to include automated target classification a MHT that fused all of the detection data to produce using techniques developed by Murphy and colleagues a common track picture, and a display and user inter- and based on the highly efficient Map Seeking Circuit face component that visualized the common track (MSC) approach (see the article by Murphy et al. in picture along with appropriate geospatial information this issue). Automatic target classification is important such as maps and terrain as well as target coordinates, containment ellipses, and the source video (also shown in Fig. 11). It ran continuously and unattended during operational threads, with rare down-times requiring operator restart and very high operational availability. The system exploited a very high percentage (greater than 90%) of all CHK Raven and Buster video received at Navy COMSECFLT MOC and automatically detected moving targets, including vehicles and people. The detection and false alarm performance varied with video quality and environmental conditions. The desert background was uniform and sparse, which represented a very challenging environment for moving object detection. The UDF detection and track data were also transmitted to another prototype system developed by the Naval Postgraduate School, which used the data to recommend platform orbit adjustments to the Navy Figure 12. Automatic detection of a moving vehicle on a road in COMSECFLT MOC commanders. the desert during JEFX 10.

230 JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) UPSTREAM DATA FUSION: HISTORY, TECHNICAL OVERVIEW, AND APPLICATIONS

UAV 1

UAV 1 video processing system

Tap Track states Video screener for UAV 1 feedback Video and telemetry To maintain Real-time processing sequence Moving object Fusion Moving object Target detections Buffer Image frames detection Pixel-space classi cation MHT (high rate) regions of interest Pixel coordinates Stationary object Object attributes/ detections features Object model DTED correlation Non-real-time processing Image frames (reduced rate) Target classi cation UAV n Track states (feedback)

UAV n video processing system

Tap Detections (feedback) Video screener for UAV n Track states (feedback)

Figure 13. UDF-based FMV exploitation enhanced with automatic target classification. for discriminating among different objects detected in fully automated target recognition. Further research the scene and correctly associating new detections with and development is required. The IR&D projects also existing tracks. The idea is that the likelihood of incor- prototyped and demonstrated the feasibility of using rectly associating a detection with the wrong target is the enhanced tracking and classification capability reduced when the classifying information can be used to transmit alerts to a mobile user equipped with an to strengthen the probability of candidate associations Android-based device. The user is not required to visu- that are similar in terms of classification. The design (depicted in Fig. 13) also uses a feedback processing path that exploits the track information to enhance the target classification. Specifically, the target heading is used to reduce the search space for target orientation by the MSC algorithm. The extended UDF prototype was tested by processing live video continuously from cameras mounted on buildings 13, 17, and 1 on APL’s campus. The system successfully produced detections and tracks on moving vehicles, along with classification scores indicating the degree of confidence in classifying the object as one of the target types of interest (sedan, pickup truck, and APL shuttle bus) (shown by example in Fig. 14). The extended UDF capability supports near-real-time processing and exploitation of multiple FMV sources but is not yet mature enough to classify target types of Figure 14. Example of target detections, tracks, and classifica- interest with sufficient discriminating power to support tion scores in processed FMV.

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) 231 A. J. NEWMAN AND G. E. MITZEL ally monitor the video surveillance but rather receives Michael J. White, Norbert White, Stephen E. Wiepre- automated alerts on user-selected targets of interest cht, Michael P. Williams, Melissa K. Wong, and James C. active in a user-defined area of interest. Wright; Northrop Grumman Corporation: Tyler Elliot, Kyle Gordon, John Noone, Ras Smith, and Chris Tyler; Atlantic Aerospace Corporation: Pete Kittredge, Tamar CONCLUSIONS Peli, and Mon Young; Charles S. Draper Laboratory: APL led a team that pioneered and matured the Frank Geisel, Tony McPherson, and Judy Miller; General development of UDF techniques. UDF combines diverse Dynamics/Veridian: Mark Carlotto, Marty Liggins, and sensor data tapped at points in the sensor processing Mark Nebrich; Lockheed Martin Corporation: Chung-Fu chain that come well before reporting to an end user. Chang, Don Sopiak, Pat Spicer, and Joe Swartz; Orincon UDF has the potential to improve the collective capa- Corporation: James Hawker, Dennis Likes, and Peter bility of a diverse set of sensors to detect, locate, and Shea; Scitor Corporation: Ed Elguthon, Bob Halcomb, classify difficult targets. UDF may relax computational and Mike Rector; Dynetics Corporation: Jason Gallaspy and communications requirements, is inherently distrib- and William Lummus; Georgia Tech Research Institute: uted and scalable, and forms the technical foundation David Millard and Russell Roberts; SAIC: Steve Sayre and for a more futuristic and comprehensive concept called Warren Smith; and Zel Technologies, LLC: Mike Shoop. CLCISR. Potential benefits of UDF have been demon- There have been so many contributors that we sincerely hope we have not missed anyone. strated at different levels of maturity in the laboratory and in the field for applications in time-critical targeting of relocatable ground targets, in maritime situation REFERENCES awareness, in space situational awareness, and in irreg- 1Hall, D. L., and Llinas, J., “Multisensor Data Fusion,” Chap. 1, Hand- ular warfare. Other contributions to critical ISR chal- book of Multisensor Data Fusion: Theory and Practice, 2nd Ed., M. E. lenges are being investigated. Liggins, D. L. Hall, and J. Llinas (eds.), CRC Press, Boca Raton, FL (2009). 2Bath, W., Boswell, C., Sommerer, S., and Wang, I.-J., “Detection Systems Information Fusion,” in Johns Hopkins APL Tech. Dig. 26(4), ACKNOWLEDGMENTS: The authors recognize the vast and 306–314 (2005). varied contributions of the following individuals to the 3Intelligence, Surveillance, and Reconnaissance: Preliminary Observa- development of UDF applied to critical challenges in tions on DOD’s Approach to Managing Requirements for New Systems, Existing Assets, and Systems Development, GAO Report GAO-07-596T the ground, maritime, and space domains (organiza- (Apr 2007). tions are listed in order of number of contributors; con- 4Unmanned Aircraft Systems: Advanced Coordination and Increased tributors within an organization are listed in alphabeti- Visibility Needed to Optimize Capabilities, GAO Report GAO-07-836 (Jul 2007). cal order): APL (and its consultants): Marshall H. Alworth, 5Intelligence, Surveillance, and Reconnaissance: DOD Can Better Assess Randall M. Arvizu, Drew Asher, Bradley O. Bates, Chris W. and Integrate ISR Capabilities and Oversee Development of Future ISR Baumgart, David Bazell, David M. Berstein, Christopher Requirements, GAO Report GAO-08-374 (March 2008). 6Intelligence, Surveillance, and Reconnaissance: Actions Are Needed to M. Boswell, N. Ray Briscoe, Myron Z. Brown, Thomas H. Increase Integration and Efficiencies of DOD’s ISR Enterprise, GAO (Buck) Buchanan, Louis P. Butler, James P. Christ, Cash J. Report GAO-11-465 (June 2011). 7 Costello, Michael A. Cramer, Thomas B. Criss, Colleen A. Deptula, D., and Francisco, M., “Air Force ISR Operations: Hunting Versus Gathering,” Air Space Power J. XXIV(4), 13–17 (Winter 2010). Davis, Jonathan T. DeSena, Vince Doran, Thomas E. Duerr, 8DeGroot, M. H., Optimal Statistical Decisions, Wiley Classics Library, Lien T. Duong, Daniel A. Dutrow, Lance E. Earwood, John Wiley-Interscience (2004). 9 Florence, Michael Francisco, Barry E. Fridling, Chip Frock, Aerospace Command, Control, Intelligence, Surveillance, and Reconnaissance Center, “Study on Optimizing the Information Lee Ann D. Fults, William J. Geckle, Jeffrey M. Gilbert, Process for Warfighting,” in Proc. 2000 National Correlation Working Michael T. Grabbe, Brian E. Gray, Sean D. Healy, Daniel Group Workshop, Ft. Monroe, VA (May 2000). 10Schwaber, K., and Beedle, M., Agile Software Development with Scrum, Hein, Gina R. Hoffman-DeLeone, Tom Jerardi, Bryan A. Prentice Hall (2002). Johnson, Jeffrey D. Jordan, Eliezer G. Kahn, Eric M. Klatt, 11Porter, D. W., “Quantitative Data Fusion: A Distributed/Parallel Susan L. Kirk, Chrisanthe P. Kostalas, Larry J. Levy, Glen Approach to Surveillance, Tracking, and Navigation Using Informa- tion Filtering,” in Proc. Fifth Joint Service Data Fusion Symp., Laurel, A. Long, Mark D. LoPresto, Kevin M. Lyons, Brooke A. MD, pp. 87–127 (Oct 1991). Mays, Jeromy W. McDerment, Nishant L. Mehta, Wil- 12Brown, M. Z., A First Look at SAR ATR for the Five-Minute War, Tech- liam A. Menner, Christopher Michaelis, Robert J. Mitch- nical Memorandum VNT-1-01U-003, JHU/APL, Laurel, MD (4 May 2001). ell, Patricia K. Murphy, Kathleen A. Newtson, Gregory A. 13Reid. D., “An Algorithm for Tracking Multiple Targets,” IEEE Trans. Niswonger, Brian C. Page, Cameron K. Peterson, Keith Automatic Control, 24(6), 843–854 (1979). S. Peyton, David W. Porter, Vignesh R. (Vinny) Ramach- 14Blackman, S., “Multiple Hypothesis Tracking for Multiple Target Tracking,” IEEE Aero. Electron. Sys., 19(1) (Jan 2004). andran, Roy Robinette, Pedro A. Rodriguez, Donald A. 15Bar Shalom, Y., Blackman, S., and Fitzgerald, R., “Dimensionless Ruffle, John Samsundar, Dana M. Schlichting, David Score Function for Multiple Hypothesis Tracking,” IEEE Trans. Aero- P. Silberberg, James C. Spall, Tom Spisz, Jacqueline space Electron. Sys., 43(1) (Jan 2007). 16Kurien, T., “Issues in the Design of Practical Multitarget Tracking A. Stock, Lora S. Vanegas, Meta M. Voelker, Bill Walker, Algorithms,” Multitarget-Multisensor Tracking: Advanced Applications, William C. Walton, Adam S. Watkins, Frank B. Weiskopf, Y. Bar-Shalom (ed.), Artech House, Norwood, MA, pp. 43–83 (1990).

232 JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) UPSTREAM DATA FUSION: HISTORY, TECHNICAL OVERVIEW, AND APPLICATIONS

17Wigren, T., “Target-Type Probability Combining Algorithms for Mul- 29U.S. Maritime Security Policy Coordinating Committee, National tisensor Tracking,” in Signal Processing, Sensor Fusion, and Target Rec- Plan to Achieve Maritime Domain Awareness for The National Strategy ognition X, Proc. SPIE Vol. 4380, I. Kadar (ed.), SPIE, Bellingham, for Maritime Security (Oct 2005). WA, pp. 46–62 (2001). 30Chief of Naval Operations, Navy Maritime Domain Awareness Con- 18Bar-Shalom, Y., Kirubarajan, T., and Gokberk, C., “Tracking with cept (May 2007). Classification-Aided Multiframe Data Association,” IEEE Trans. 31Barton, C., Hobbs, S., Kaina, J., Kessler, O., Lapin, B., Tollefson, E., Aerospace Electron. Sys., 41(3) (Jul 2005). White, F., Maritime Domain Awareness (MDA): Data Fusion Assess- 19Bierman, G. J., Factorization Methods for Discrete Sequential Estima- ment Final Report (Jun 2006). tion, Academic Press (1977). 32Li, X. R., and Jilkov, V. P., “Survey of Maneuvering Target Tracking. 20Bierman, G. J., Belzer, M. R., Vandergraft, J. S., and Porter, D. W., Part V: Multiple-Model Methods,” IEEE Aero. Electron. Sys., 41(4) “Maximum Likelihood Estimation Using Square Root Information (Oct 2005). Filters,” IEEE Trans. Automatic Control, 35(12), 1293–1298 (1990). 33Dyer, P., and McReynolds, S. R., “Extension of Square Root Filtering 21Boswell, C. M., Geolocation Square Root Information Filter Formulation to Include Process Noise,” J. Optimization Theory Applications, 3(6), Using Measurements of Altitude and Measurements of Time Difference 147–158 (1969). of Arrival, Technical Memorandum PSA-05-023, JHU/APL, Laurel, 34McDerment, J. W., Crowning Ships Tracks with Upstream Data Fusion, MD (07 Sep 2005). APL IR&D Completed Project DTIC Report (Sep 2011). 22DeSena, J. T., Bias Treatment under Multiple Hypothesis Data Asso- 35Sabol, C., Kelecy, T., and Murai, M., “Geosynchronous Orbit Deter- ciation using Square Root Information Filters for DTCWC, Technical mination Using the High Accuracy Network Determination System Memorandum GVE-07-0039, JHU/APL, Laurel, MD (6 Jun 2007). (HANDS),” in Proc. 14th AAS/AIAA Space Flight Mechanics Meeting, 23DeSena, J. T., Simulation-Based Study of Processing Order Dependence Maui, HI, paper AAS 04-216 (Feb 2004). for IMINT-ELINT Sensor Ensembles, Technical Memorandum GVE- 36Newman, A. J., Michaelis, C. H., Klatt, E. M., Spisz, T. S., Mehta, 08-0086, JHU/APL, Laurel, MD (9 Oct 2008). N. L., Challa, M. S., Bazell, D., and Casanova, F., “Upstream Data 24Newman, A. J., and Kittredge, P., “The Winchester Experiment— Fusion for Space Situation Awareness,” in Proc. 2009 MSS National Combined Feature-Based SAR Classification and Geo-Observable Symp. on Sensor and Data Fusion, Nellis AFB, NV, paper SA08 (Aug Multiple Hypothesis Tracking for Improved GNCST Data Associa- 2009). tion,” in Proc. 2005 MSS National Symp. on Sensor and Data Fusion, 37Newman, A. J., Michaelis, C. H., Klatt, E. M., Spisz, T. S., Mehta, Monterey, CA, paper SE02 (May 2005). N. L., and Kahn, E. G., Upstream Data Fusion for Space Situation 25Newman, A., Gilbert, J., Boswell, C., Murphy, P., Florence, J., et al., Awareness, Phase 2 Final Technical Report, APL Technical Report “Performance Evaluation of ELINT-IMINT Upstream Sensor Fusion (Jun 2010). for JBMC2,” in Proc. 2006 MSS National Symp. on Sensor and Data 38Newman, A. J., Klatt, E. M., Martin, S., Clarke, J., Rodriguez, B., Fusion, McLean, VA, paper SC02 (Jun 2006). and Mehta, N., “Dynamic SSN Sensor Tasking for Space Situation 26Mitzel, G., Boswell, C., Murphy, P., Walton, W., Newman, A., Flor- Awareness,” in Proc. 2011 MSS National Symp. on Sensor and Data ence, J., Gilbert, J., and Stock, J., “Multimodal Upstream Data Fusion: Fusion, Washington, DC, paper NA04 (Oct 2011). Theoretical Basis, Initial Results, and Future Directions,” in Proc. 39U.S. Department of Defense, Irregular Warfare (IW) Joint Operating 2006 Missile Defense and Intelligence Community Interchange Conf., Concept (JOC), Version 1.0 (Sep 2007). Monterey, CA (Aug 2006). 40Funk, B., Newman, A., and Hiatt, D., “JEFX-10: Autonomy and 27Newman, A. J., Spall, J. C., Mitzel, G. M., Porter, D. W., Walton, Data Fusion,” in Proc. 2010 JHU/APL Small UAS Symp., Laurel, MD W. C., et al., “DTCWC Upstream Data Fusion: Feature-Based Fusion, (Jun 2010). Sensor Bias Compensation, and Northern Edge 2006 Field Dem- 41Funk. B., Castelli, J., Watkins, A., McCubbin, C., Marshall, S., onstration,” in Proc. 2007 MSS National Symp. on Sensor and Data et al., “JEFX 10 Demonstration of Cooperative Hunter Killer Fusion, McLean, VA, paper SE05 (Jun 2007). UAS and Upstream Data Fusion,” Unmanned Systems Technol- 28Defense Science Board, Report of the Joint Defense Science Board Intel- ogy XIII, Proc. SPIE Vol. 8045, D. W. Gage, C. M. Shoemaker, ligence Science Board Task Force on Integrating Sensor-Collected Intel- R. E. Karlsen, and G. R. Gerhart (eds.), SPIE, Bellingham, WA ligence (Nov 2008). (Apr 2011).

Andrew J. Newman is a member of the APL Principal Professional Staff and Section Supervisor of the Data Fusion Anal- Theysis Section of the WeaponAuthors and Targeting Systems Group. His work at APL has focused on sensor and data fusion, target tracking, and dynamic platform and sensor resource management applied to ground, space, and maritime surveillance. He has led tasks on projects mentioned in this article, including DTCWC, UDF for SSA, and JEFX 10 UDF of UAV-Borne Video Sensors. Glenn E. Mitzel is a member of the APL Principal Professional Staff and Chief Scientist for the Precision Engagement Business Area. His interests include optimal estimation and control, remote sensing, and rapid prototyping. Since 2000 he has led large multidisciplinary teams developing, applying, and expanding the UDF concept. For further information on the work reported here, contact Andrew Newman. His e-mail address is [email protected].

The Johns Hopkins APL Technical Digest can be accessed electronically at www.jhuapl.edu/techdigest.

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 3 (2013) 233