We turn now to level-4 processing. The focus of level-4 processing is to seek to optimize the on-going data fusion process (and sub-processes). In particular, we seek to improve the timeliness, relevance, accuracy and comprehensiveness of the fusion process to meet the needs of the system users. This lecture provides an introduction to level 4 processing, describes two broad approaches including optimization and market-based methods, and provide some insight into future research.

In the JDL model, the level-4 process is deliberately shown as being part “inside” the data fusion process and part “outside” the fusion process. This is because the level 4 process is a “meta-process” – it’s a process that monitors the other portions of data fusion processing and seeks to provide a means of optimizing the overall fusion process.

Conceptually we want to monitor the on-going data fusion process (at all levels) and make appropriate changes to improve both the efficiency of the processing and the resulting data fusion “products”. This optimization may include; i) changes in how we task sensors or sources of information, ii) use of “active” (energy emitting) sensors versus “passive” (energy collection) sensors, iii) dynamic selection among alternative processing algorithms, iv) modification of selected algorithms to process the data, v) allocation and re-allocation of information processing among processing resources, vi) information requests to a user or analyst to provide both immediate and long term feedback to the fusion process, etc.

While level-4 processing might be considered as a classic “system optimization” problem, it can be more challenging for several reasons. First, some things might be out of our control – that is, we don’t necessarily control the sensors or sources of information but instead must deal with what’s available. A data fusion system often operates as part of a larger system such as a command and control system, an environmental monitoring system, or other application. Thus, the ‘resources’ are not necessarily ours to control. Second, things that might be recommended to optimize the fusion process may be sub- optimal or even detrimental to the overall process in which our fusion system is embedded. For example, if we’re using multiple passive sensors at night to hunt for, and identify an elusive animal (e.g., the search for Sasquatch!), it might be tempting to turn on search lights (use an “active sensor) to see better. However, turning on the lights or active sensor could very well scare off the thing we’re trying to observe.

There are several factors that motive the evolution of level-4 processing. Some key factors that provide both new opportunities for level 4 processing as well as new complexities for level 4 processing include the following.

1) Intelligent sensor/processing systems - The rapid evolution of intelligent sensor/processing systems provides both opportunities and challenges for more sophisticated tasking. Modern sensors often have the ability to dynamically change their sensing patterns, use advanced waveforms (for active sensors), and perform complex signal and image processing “on the fly” and in reaction to the environment. Simple examples are the advances in mobile phone video and image processing which can adapt to light changes and content of the environment (presence of people). Cell phones are rapidly becoming intelligent multi-sensor devices with sophisticated sensor processing. Such devices provide both the opportunity for dynamic tasking changes but require level-4 algorithms to take advantage of that flexibility.

2) Human observers can augment the use of traditional sensor systems. Humans can provide observations and commentary using mobile devices via text messages. They can also act as “sensor platforms”, carrying smart phones with multi-sensor systems available to collect data. The potential availability of 7 billion human sources and mobile sensor platforms (http://www.digitaltrends.com/mobile/mobile-phone-world-population-2014/#!bkDXk0) is a potential huge source of information for applications such as crisis management, environmental monitoring and citizen science. However, the ability to task, access, and calibrate such sources is quite challenging.

3) Distributed, dynamic, network systems - Information fusion systems are routinely becoming distributed systems. Sensors, sources of information, processing and even users are generally distributed, rather than co-located. Sensors can include sensors embedded in numerous entities (e.g., the Internet of Things), mobile sensors via smart phones, and platforms such as drones and robots. This provides both a rich environment for potential allocation of resources for improved data fusion (and hence situation awareness), as well as more complexity in the tasking and resource allocation.

4) Increased bandwidth of local and global communications systems provides opportunities to share information such as high-definition images, videos, and even three-dimensional image data available from sources such as a Lidar. Thus, level 4 processing must consider not only the availability of increasing bandwidth to collect and distribute more data, but also the potential burdens of such collection and distribution.

5) Service-based architectures are evolving to allow improved connectivity and linkages between previously “unconnected” systems and entities. These evolving tools, data standards, and access to distributed (cloud-based) resources provide opportunities and challenges for level-4 processing.

6) There are ever increasingly complex algorithms that are being developed and are capable of being executed on mobile devices. For example, smart phones routinely allow the application of sophisticated image processing techniques directly on the phone. As meta-data generation methods (e.g., semantic labeling of images become available), this provides new opportunities for information to be available for level-4 processing. In addition, the availability of cloud computing provides opportunities for the use of computationally demanding optimization techniques for level-4 processing.

A partial list of types of functions involved with Level-4 processing is shown in this chart. This is not meant to be an exhaustive list, but rather an indication of the types of functions that can be part of the implementation of a level-4 process. We will summarize these briefly.

• Mission management – Recall that Level-4 processing must be done in the context of the mission or application supported by the data fusion system. The data fusion system must operate within the constraints of the overall system (whether it be a system to monitor the health of a mechanical system or a military situation awareness system, etc.). Thus, it is necessary to represent the mission objectives and constraints of the overall system and develop computational methods to adjudicate between the demands of the system mission and the goal of optimizing the data fusion system performance.

• Target prediction – If we are monitoring or tracking an entity (such as a maneuvering target or fault condition in a machine), it is necessary to represent the current state of an entity (including location, characteristics and identity) and perform state estimation or prediction of future states of the entity.

• Sensor and platform modeling – Part of process refinement involves modeling the location and conditions of our sensor system, the performance of the sensors in the current environment (e.g., to account for effects such as the atmosphere on signal propagation), as well as the interaction between the targets or entities being observed and the signals emitted by the sensors (or signals emitted by the entities).

• System performance modeling – In addition, to the performance of the sensors, we need to model the performance of the data fusion processing (viz., how well the data fusion algorithms are performing), and compute measures of performance and measures of effectiveness.

• System control – Finally, the overall system control must be modeled. For optimization problems this involves i) establishment of optimization criteria (such as data fusion system accuracy, timeliness of the data processing, utilization of resources such as the communications networks, etc.), ii) selection of optimization algorithms, and iii) explicit development of a control philosophy.

Again, we emphasize that optimal control theory is an entire field of study in it’s own right. We provide here only a brief glimpse of issues involved for optimizing the functioning of a data fusion system.

To review, sensor management is a process which seeks to manage and coordinate the use of sensing and information resources in a manner that improves the process of data fusion for improved accuracy and decision-making for human users. Sensor management techniques can be based on variety of classes of techniques including; heuristics, expert systems, utility theory, automated control theory, cognition, decision theoretic approaches, probabalistic based methods, stochastic dynamic programming, linear programming, neural networks, genetic algorithms, and information theory. The purpose or functions of sensor management are to; i) allocate sensor utilization (pointing, moding and emission control for active sensors), ii) prioritize and schedule service requests for use of resources such as sensors and communications infrastructure, iii) assist in sensor data fusion, iv) support the reconfiguration of sensors, communications and computations in challenging or degraded operational modes, v) optimize the use of available sensor assets, and vi) assist in communications modeling and use. In the next series of charts, we will provide an example based on a modern military tactical aircraft. The example provides an opportunity to show the interaction between a data fusion system and a complex mission-focused platform. The concepts introduced for this application and clearly adaptable to other types of systems and domains.

For this example, the aircraft has an integrated avionics system with agile and programmable/steerable sensors which may be both active and passive. The observing environment is challenging due to the very dynamic platform motion and mission, the potential for jamming and deception, and the potentially large number of entities to be observed (which might be closely spaced). In addition, the environment might involve so-called “low observables” in which entities use stealth, camouflage, or other methods to reduce their observability.

While the integrated platform has extensive computing and sensing capabilities, because they are part of a military tactical aircraft, there are certainly limitations and challenges. These include; limited sensor resources, the variability of sensor performance, unplanned sensor failures, and mission constraints such as “emission control” (EMCON) conditions in which the use of active sensors is limited or prohibited based on the overall mission.

This chart shows a conceptual interaction between functions such as; i) mission control/management, ii) evaluation of data fusion and mission performance, and iii) a source and fusion control loop which focuses on the sensor resources and the fusion processing methods.

There is clearly a coupling between three areas; i) the sensor management system, ii) the sensor suite, and iii) the sensor data fusion processing. The sensor management system involves scheduling the sensors, seeking to obtain synergy between different types of sensors (to improve accuracy of locating, characterizing and identifying entities) and understanding the optimal performance of the overall system of sensors. The sensor suite involves a collection of sensing resources, each having different performance specifications, differing ability to contribute to estimating the state of entities, and different performance agility (e.g., ability to change modes, perform internal processing, etc.). Finally, the sensor data processing involves previously discussed functions such as alignment, association and correlation, tracking, etc. The overall mission requirements and priorities drive what design choices are made for these interacting systems.

A useful construct for managing the use of sensors comes from the mathematics of utility theory. The concept of utility was originally introduced by Daniel Bernoulli in 1738 based on the observation that when playing games of chance, people do not make choices based strictly on the potential payoff or money involved, but rather based on an internal assessment of the usefulness or utility of an outcome.

In data fusion, for example, we might seek to optimize the use of sensors to determine the location of an entity. If we are locating a vehicle in a field, we might need to know only an approximate location (e.g., within a few feet of accuracy). Using sensors to determine the location to within an inch, however, may be of little added value. By contrast, if we are seeking the location of a tumor within a patient’s body, it may be necessary to know the location to within a hundredth of an inch. The value of the location is based on the application and our “need” to know the information. Thus, instead of optimizing the absolute accuracy of the location, we optimize a utility function that provides a measure of how useful the information is to us.

For our tactical aircraft example, we may define a utility function to describe aspects of covertness, tracking accuracy, target detection performance, and other factors. The control variables to allow us to optimize the utility may include factors such as time on target (how much time we task a sensor to observe a target), the sampling rate (the number of observations per minute or second), what is measured, and search versus tracking tradeoffs. A search utility might involve a function of the number of detected versus undetected targets. Similarly, a tracking utility function may involve calculations of the track covariance (the accuracy with which we predict that we can observe a target’s location and motion).

This chart provides a summary of some of the types of sensor controls that may be available for the optimization process. For smart sensors with embedded computing capabilities, we may have access to:

• Mode controls include factors such as; turning an sensor on or off, selecting a sensor mode of operation (e.g., power level, waveform or processing mode, scan, track or track-while-scanning), and sensor processing parameters such as decision thresholds and detection, tracking and identification criteria.

• Spatial control functions include; where to point or direct a sensor (either be physically moving an antenna or electronic beam directions, selecting a field of view, scan and search rates, scan and search pattern selection, controlling individual sectors of observation, and parameters for designated targets.

• Temporal control functions include; start and stop times for modes and sector controls, sensor look times, dwell times on target, and maximum allowed emissions (duty cycle)

• Reporting control functions include; filters on reporting based on target attributes (e.g., target class or identity, or filtering by anticipated lethality of an observed target), reporting based on the spatial attributes such as minimum and maximum range limits or altitude limits, and finally prioritized reporting of designated targets.

Factors that may affect how to set the priority of observed targets include; target identity, what information is needed, anticipated threat evaluations, potential opportunities for offensive or defensive actions, and finally fire control needs (if we need to guide a weapon to address a particular target or target type).

Another view of general sensor management functions is provided by Ed Waltz and James Llinas in the book, Multisensor Data Fusion, published by Artech House, Inc. in 1990. The chart shows an expansion of the JDL Level 4 process with connections to Level 1 processing (on the bottom left hand side of the figure) and with the Level 2 and Level 3 processing shown on the top left hand side of the figure. The types of functions described by Waltz and Llinas include; i) prioritization of threats and modeling of what information is “needed” by the other fusion processes, ii) assignment of target or entity priorities (what should be observed first), iii) an assignment algorithm that links targets to sensors that could observe those targets, iv) allocation, scheduling and control of the overall sensor system, v) and an interface to sensors to control or task the sensor observations. These functions are supported by sensor performance and prediction models, and spatial-temporal control computations.

These functions are generic and meant to show the types of data and computations that might be included in the level-4 fusion processing, to optimize the use of sensors to obtain vital information.

The final chart for the optimization discussion is simply a list of potential measures of performance (MOP) and measures of effectiveness (MOE) for the level-4 process. This list is a set of possible metrics that could be used to evaluate how well the sensors are being managed or tasked to obtain information for the remaining part of the data fusion system. Thus, categories of MOP/MOE include; i) detection and leakage – how well are we detecting targets of interest and to what extent are targets “slipping by”, ii) target tracking – how accurately are targets or entities being tracked, iii) identification – how well is our system identifying potential targets or entities, iv) raid assessment – for a military operation such as a raid against opposing targets, how well did we correctly determine the number of targets and their locations, was the information timely enough to support the operation, v) emissions – how well did we manage the overall power used and emitted by the sensors, did we emit more energy than necessary, vi) sensor utilization – were the sensors used effectively both individually and in concert, vii) system response – did we obtain data and information in a sufficiently timely manner to meet the needs of the system users, and viii) computational performance – how much computing, memory and communications bandwidth were used to obtain the results.

We turn now to another concept for level-4 processing, namely models based on commerce. In the electronic-business world, we are very familiar with exchanges such as e-Bay which effectively acts as a global yard sale. Potential consumers use an electronic exchange service to access products being offered by sellers. The consumers don’t know a priori where or who the sellers are, and the sellers don’t know, a priori, where or who the potential consumers/buyers are. An automated auction mechanism is used to match the available services or goods to the requested services or goods by potential buyers. Also, rating systems are available to assist buyers in evaluating the honesty and carry- through of sellers. Other types of exchanges include eLance ( for linking free-lance workers with people needing their services, Amazon for buying books and other goods, and many others.

It is natural to consider the use of e-commerce concepts for level-4 processing. Numerous algorithms, software-based exchange services, and other concepts can be adapted for level-4 processing.

Continuing this concept, we see that potential information consumers, shown on the right hand side of the chart may include analysts, decision-makers, or even other information processing systems. On the left hand side or the chart, information suppliers included sensors, other information processing systems, and communications channels to link the data to the potential users. Thus, conceptually, a sensor manager can be considered as an information/resource broker linking information suppliers (the sensors, external databases, possible human observers) and the communications channels to the information consumers (the analysts, decision-makers and other fusion processes).

Thus, it would appear easy to use the e-commerce, market-based concept for level-4 processing. After all, we now routinely use on-line services to link sellers to consumers. Moreover, dynamic auction techniques allow determining which consumer “wins” the services of which seller or sellers. However, there are some special opportunities and challenges for level-4 processing. Let’s consider two cases in the next two charts.

In our first case, we have a situation in which a single supplier (as sensor) can satisfy multiple consumers with a single product or service. That is, if two different consumers want image data about a particular region, it would certainly be possible for a single data collect (e.g., by an aircraft, drone, or satellite) to provide the requested information. Thus, the “seller” can use the same information to be “sold” to multiple customers. This provides both an opportunity and challenge for level-4 processing using auction methods.

The second case is more challenging. Suppose an information consumer seeks information that cannot be supplied by a single sensor or source. For example, a user wants to know both the location of an entity within a specified level of accuracy AND know the entity’s identity to a specified level. It is common that a single sensor may not be able to satisfy both requirements. Moreover, it may not be possible that any set of sensors can satisfy this information. Thus the market exchange must somehow be able to find a suitable collection of resources that, when fused, best meet the set of information requirements. So, the information broker must be intelligent to do the best job possible in scheduling and tasking the information resources.

An example of a market-based design for sensor allocation is shown in this chart. It is based on work conducted by Tracy Mullen et al. (see V. Avasarala, T.Mullen and D. L. Hall (2009), “A market-based sensor management approach”, Journal of Advances in Information Fusion, April 2009). The design treats sensors and network communications as “sellers” of services and system users as “consumers”. A sensor manager, acts as market auctioneer to seek resources in response to information needs from consumers and a mission manager. In effect the “sellers” (the sensors, sensor processing and transmission channels produce finished products of data related to environmental scans and target tracks. The consumers request or “bid” for information, which is translated into specific types of tasking by the sensor manager. The sensor manager sends information about the status of the sensing and communications system to the mission manager which has responsibility for providing information about resource budgets and availability.

The previously referenced paper by Avasarala, Mullen and Hall describes the actual market-based allocation and auction mechanisms. The auction process involves the concept of tatonnement. This is a type of optimization technique to dynamically seek a price equilibrium which best meets the demands of the “buyers” (viz., what constitutes an acceptable “price” for the requested resources) and the capabilities of the suppliers (viz. what the suppliers can “deliver” at an acceptable “price”). We close the discussion of level-4 processing by making some general remarks about sensor management.

• First, ever increasing capabilities of infrastructure technologies such as improved connectivity and data rates, improved access to cloud-based computing, enhanced sensing and processing capabilities within individual sensors provide opportunities for increasingly sophisticated sensor (and system management).

• Second, sensors and sources of information are required by more than just the fusion process/processor. Sensors and sources of information are not necessarily “owned by” or under the strict control of the data fusion system. As a result, this creates a contention for sensor service as well as a contention for computational and communications resources. Hence, the level-4 process is more complicated than simply a problem of optimal process control.

• Third, in an integrated system (such as our example of the modern tactical aircraft, a coupling exists between sensor management, data fusion, and the sensors. This leads to a challenging global optimization problem which is exacerbated by issues such as the potential inability of any set of resources to satisfy the needs and requests of users.

• Finally, this subject has been “under-researched” in the data fusion community. There are many research opportunities here, especially when considering distributed systems that involve both physical sensors and human observations.

A final tip is that situation awareness for any application such as military situation awareness, control of complex machinery, or monitoring emergency events involves effectively using system resources, such as sensors, computational capabilities and communications resources so that the user gets the information that they need, when the need it. In a sense, it is analogous to how the human brain helps to manage our attention to provide critical information when it is needed, rather than simply trying to react to all of our senses.