Display of Information for Time-Critical Decision Making

Matthew Barry Eric Horvitz

Propulsion Systems Section Decision Theory Group

NASA Johnson Space Center Microsoft Research

Houston, Texas 77058 Redmond, Washington 98025

[email protected] ckwell.com [email protected]

Abstract can also assist engineers with the oine design of user-

interface content and functionality.We will see also

how a decision-theoretic p ersp ective on display high-

We describ e metho ds for managing the com-

lights imp ortant questions ab out human information

plexity of information displayed to p eople

pro cessing. Answers to these questions will allowusto

resp onsible for making high-stakes, time-

further re ne the display-management metho dology.

critical decisions. The techniques provide

to ols for real-time control of the con gura-

Previous related investigation of the use of prob-

tion and quantity of information displayed

ability and utility in display management includes

to a user, and a metho dology for designing

work on controlling the tradeo b etween the com-

exible human-computer interfaces for mon-

pleteness and the complexity of informational displays,

itoring applications. After de ning a proto-

computer-based explanations, and computational b e-

typical set of display decision problems, we

havior [Horvitz et al., 1989], the use of multiattribute

intro duce the exp ected value of revealed in-

utility to control the complexity of presentations and

formation (EVRI) and the related measure

displays [Horvitz, 1987a, Mclaughlin, 1987], and mul-

of exp ected value of displayed information

tiattribute utility for queuing and prioritizing the re-

(EVDI). We describ e how these measures can

sults of diagnostic reasoning [Breese et al., 1991]. In

b e used to enhance computer displays used

other related work, qualitative mo dels, in combination

for monitoring complex systems. We moti-

with several heuristic imp ortance metrics havebeen

vate the presentation by discussing our ef-

employed to select information for monitoring appli-

fortstoemploy decision-theoretic control of

cations [Doyle et al., 1989].

displays for a time-critical monitoring appli-

We will rst review basic results from studies of human

cation at the NASA Mission Control Center

information pro cessing ab out cognitive load, short-

in Houston.

term memory, and decision making. Then, we will de-

scrib e the representation and solution of time-critical

decision problems. We will intro duce the task of mon-

1 INTRODUCTION

itoring and decision making ab out propulsion systems

on the Space Shuttle to motivate the imp ortance of in-

The rapid growth in the use of computers to ac-

formation display in time-critical situations. We will

cess information and to monitor complex systems has

present decision mo dels for the display of information,

brought increased attention to the costs of navigating

and discuss metho ds for evaluating the value of dis-

through large quantities of data in search of critical

played information. We will fo cus rst on the exp ected

information. Problems with accessing and reviewing

value of revealed information (EVRI). After, we will

information are esp ecially salient in high-stakes, time-

intro duce the use of Bayesian mo dels of user b elief and

critical decision-making contexts. We presentwork on

action, and describ e the exp ected value of displayed

enhancing the human{computer interface through ap-

information (EVDI). Finally,we will address practical

plying decision-theoretic inference to control the infor-

approaches to implementing display managers based

mation displayed to p eople resp onsible for monitoring

on EVRI and EVDI. We will summarize by discussing

complex systems. The metho ds can b e employed to

our research directions.

enhance the quality and timeliness of decisions byad-

justing the con guration and quantity of information

displayed to decision makers, dep ending on the current

2 INFORMATION AND

uncertainties and time criticality. Beyond their appli-

COGNITIVE LIMITATIONS

cation in the real-time control displays, the techniques



Why should weworry ab out managing the complexity

In Proceedings of the Eleventh Conference on Uncer-

of displayed information? Decision-theoretic analyses tainty in Arti cial Intel ligence, Montreal, August 1995.

of the value of information show us that gaining cost-

free access to additional information can only enhance Action A,t Duration of Process

the quality of our actions. Unfortunately, information

often comes at a cost. When we compute the net value

e consider the value of information,

of information, w Utility

en uncertainty ab out test results, and the cost of giv System

information. State H, to

Information that is already available to a computer

ypically do es not cost any- ab out a monitored system t E E2 E3

1 En

thing to display.However, in time-critical, high-stakes

situations, the time required by p eople to review infor-

mation, and confusion arising in attempts to pro cess

large amounts of data quickly, can lead to costly delays

Figure 1: An in uence diagram representing the time-

and errors.

dep endent cost asso ciated with allowing an anomalous

condition to p ersist by delaying an e ective resp onse.

Fundamental limitations in the abilities of p eople to

pro cess information explain why decision qualitymay

degrade with increases in the quantity and complex-

ferent times on the temp oral progression of the states

ity of data b eing reviewed, and with diminishment

of a system. In some cases, it may b e appropriate to

in the time available for a resp onse. Human di-

assess an outcome as an equilibrium state, reached at

culties with the pro cessing of information has b een

some time following a set of interventions, and to as-

akey research fo cus within Cognitive Psychology

sess the utility of this equilibrium state [Horvitz, 1990].

[Bruner et al., 1956]. Numerous studies have provided

We can attempt to minimize the detailed mo deling of

evidence that human information pro cessing is pri-

the temp oral progression of system states by assessing

marily sequential in nature [Simon, 1972]. Exp eri-

the time-dep endentchanges in the utility of outcomes

ments haveshown that the sp eed at which sub jects

that are de ned in terms of a system (or world) state

p erform tasks drops as the quantity of information

and interventions made at various times. The utility

b eing considered increases, and that the rate of p er-

asso ciated with allowing one or more states of a system

forming tasks can b e increased by ltering or sup-

to evolve with or without intervention is a function of

pressing irrelevant information [Morrin et al., 1961].

the state, the action, and the time action is taken. It

In a classic study on limitations in human cognition,

may b e p ossible to assess from exp erts time-dep endent

Miller found that humans cannot consider more than

utility functions that capture the changes in utilityat

ve to nine distinct concepts or \chunks" of informa-

progressively later times of intervention for di erent

tion simultaneously [Miller, 1956]. The capacityof

system states [Horvitz and Rutledge, 1991].

decision makers to consider imp ortant in uences on

a decision may b e reduced even further if fast ac-

In one class of time-critical decision problem, repre-

tion is demanded in crisis situations. One cognitive-

senting a large class of high-stakes, time-critical moni-

psychology study demonstrated that p eople cannot

toring tasks, we gain access to information ab out sys-

retain and reason simultaneously ab out more than

tem's b ehavior at time t and attempt to diagnose and

0

two concepts in environments lled with distractions

take action to resp ond to the anomalous state of the

[Waugh and Norman, 1965]. These and other cogni-

system. In many time-critical situations, additional in-

tive psychology ndings provide motivation for auto-

formation ab out the progression of a state over time is

mated metho ds that can balance the value and costs

either not relevantornotavailable b efore action needs

of displayed information.

to b e taken. Outcomes are often particularly sensitive

to the length of time that an anomalous condition p er-

sists. Delays may b e asso ciated with signi cant time-

3 TIME-CRITICAL DECISIONS

dep endentchanges in the utility of outcomes based in

the duration of the condition. A representation of this

The costs versus the b ene ts of sp ending time to re-

typ e of decision problem, assuming a single action or

view additional information are sensitive to the time

xed sequence of actions, is represented by the in u-

criticality of a situation. In time-critical contexts,

ence diagram in Figure 1. We assess time-dep endent

the utilities of outcomes diminish signi cantly with

utilities, as a function of the action A , the state of

i

delays in taking appropriate action. There are sev-

the system H , and the delay b efore action is taken,

j

eral classes of time-dep endent decision problems and

u(A ;H ;t). Given uncertainty ab out the state of sys-

i j

avarietyofways to representvariables and proba-

tem, the exp ected utility (EU) of taking action A at

i

bilistic dep endencies to enco de knowledge ab out time-

time t is

dep endence of outcome and utility [Horvitz, 1987b,

n

Horvitz, 1988]. In one approach, we mo del explicitly

X

the time-dep endent progression of imp ortant states of

EU (A ;t) = p(H jE; )u(A ;H ;t)

i j i j

a system under the in uence of pro cesses that mayper-

j =1

sist over time. We consider the e ects of di erentac-

tions (including not taking any explicit action) at dif- where p(H jE; ) is the probabilityover hyp otheses j 1.0

damaged) 0.5 p(Engine thrown) Probability p(Shrapnel

0.0 0 1 2 3 4 5

Seconds burn continued during propellant failure

Figure 2: Graphs of the probabilities of costly failures

with delayed action, highlighting the p otential time

Figure 3: Primary screen of the traditional display for

criticality of decision making ab out Shuttle propulsion

Shuttle propulsion systems at Mission Control. Left,

systems.

right, and forward RCS data is displayed in panels at

the top of the screen. Data on the right and left OMS

engines are displayed in the lower panels.

ab out di erent systems states, given observations E

and background state of information  .

redirect fuel to alternative engines in a varietyofdif-

4 EXAMPLE: SPACE SHUTTLE

ferentways. The stakes may b e high. For example,

PROPULSION SYSTEMS

continuing a burn during an engine problem can de-

stroy the engines or the entire Space Shuttle. Halting

Our work on display management for time-critical

a burn b efore a critical target velo cityisreached can

decisions has b een a key comp onentofthemulti-

lead to such situations as the forced ditching of the or-

site Vista Pro ject [Horvitz et al., 1992]. The Vista

biter in the o cean, or the missing of a critical key reen-

Pro ject was initiated in 1991 to develop inferential

try opp ortunity. Decisions are not only high-stakes,

to ols to assist ight engineers at the NASA Mission

they may also b e time critical. In many contexts, de-

Control Center in Houston with the interpretation of

layed decisions can b e very costly to the mission and to

telemetry from the Space Shuttle. We designed and

the orbitor itself. The time criticality of propulsion-

implemented decision-theoretic inference and display-

systems decision making is highlighted by the graph

management software to aid engineers monitoring the

in Figure 2, displaying dynamic probabilities, assessed

Shuttle's propulsion systems. To further motivate

from an exp ert, of damaging and destroying an engine

display-management issues, we will review some detail

explosively bycontinuing a burn during a prop ellant

ab out the propulsion-systems monitoring and decision

failure.

problem.

4.1 STATUS QUO FOR DISPLAY

Flight engineers in the Propulsion Section at Johnson

Space Center are resp onsible for monitoring two di er-

Before Vista, the traditional computer displays for

entShuttle thruster systems: the orbital maneuvering

the Propulsion Section resembled other cluttered,

system (OMS) and the reaction control system (RCS).

information-rich displays at the Mission Control Cen-

The large right and left OMS engines are red for such

ter. The primary propulsion-systems display,inuse

critical maneuvers as orbital insertion and orbit circu-

b efore the Vista system was intro duced, is pictured in

larization. The smaller suites of RCS thrusters are

Figure 3. The screen includes information on the sta-

used for translation in space, for such tasks as maneu-

tus of two OMS engines, and the three banks of RCS

vering near ob jects in orbit, as well as for the continual

engines. During missions, if ground controllers b ecome

computer-controlled stabilization of the Shuttle's tra-

concerned ab out the health of one of the propulsion

jectory and p osition.

systems or subsystems, auxiliary screens may b e re-

Flight engineers often face a large quantity of p oten-

quested whichcontain such data as trend information

tially relevant information, esp ecially during crises.

ab out engine consumables. The complex primary and

Propulsion ight engineers must continue to moni-

auxiliary displays of information can b ecome burden-

tor multiple sensors which measure suchvariables as

some in situations that demand quick decision making,

changes in the Shuttle's velo city with burns, pressures

esp ecially for ight engineers in training.

and temp eratures in tanks of consumables (helium,

fuel, nitrogen, and oxidizer), and voltages and currents

4.2 DECISION MODELS FOR

in electrical subsystems.

PROPULSION SYSTEMS

If a problem with the functioning of the propulsion sys-

tems is noted during a critical burn, the op erator must In the rst phase of the Vista pro ject, we develop ed

decide whether to continue the burn, halt the burn, or probabilistic and decision-theoretic mo dels for assist- He P1 Meas He P1 Sens Failure He Tank Leak He Pressure 1 He Pressure He Pressure 2 Helium He P2 Meas He P2 Sens Failure

He P1 Trend He Ox Valve He Fu Valve Hx Fu Temp Prob He P2 Trend Hx Ox Temp Prob

Ox Tank Leak Fu Tank Leak

Ox Temp Prob Fuel Temp Prob

Ox Tank P

Ox Contam Fu Tank P Ox Temp Oxidizer Fuel Fuel Temp Ox Inlet P Fu Contam Ox Tank Pressure Fuel Tank Pressure Fu Inlet P X-over Ox P X-over Fu P Combust P N Tank P2 N P2 Sens Fail

Fuel Inlet Pressure Engine Failure N Tank Pres Ox Inlet Pressure N Tank P1 N P1 Sens Fail N Valve Fail N Accum P Comb Pres Meas N Tank Leak Nitrogen X-over Fu Pressure Comb Sens Failure Comb P Trend Combust Pressure Nitrogen Accum Pres

X-over Ox Pressure

Figure 5: A Bayesian network for the OMS engine.

Figure 4: Schematic of an OMS engine.

of outcomes. We assessed time-dep endent utilityin

terms of the dynamically changing probabilitythata

mission would b e terminated prematurely or, for more

ing ight engineers with key subproblems in the Shut-

catastrophic situations, that the entire orbitor would

tle propulsion domain. First, wedevelop ed Bayesian

b e lost, as a function of the anomaly, the action taken,

networks for the Shuttle's propulsion systems. Figure

and the p ersistence of fault states. We found that it

4 displays a basic schematic of the OMS engine. The

was useful to use multiattribute utility to understand

overall op eration of the propulsion engines is straight-

the tradeo s among dimensions of value in an out-

forward. A tank of helium gas maintains pressure on

come, including suchkey attributes as the p ortion of

tanks of oxidizer and fuel. To re an OMS engine,

the target velo city reached and the probabilityofdam-

valves are op ened which allow the fuel and oxidizer

aging an engine b eing used for the desired impulse.

to mix and combust to provide thrust. In addition to

the basic ows, ground controllers must also consider

The Vista-I I system wentinto service at the Mission

the status of a set of valves b etween various tanks,

Control Center in 1993, following lab oratory validation

and crossover lines that allow prop ellant to b e shared

with a prototyp e named Vista-I. Vista-I I monitors and

by di erent engine systems. Suites of temp erature

interprets live telemetry b eing transmitted from the

and pressure sensors are lo cated at critical lo cations

Shuttle. When the probability of an anomaly (includ-

in the system. The telemetry ab out propulsion sys-

ing sensor faults) exceeds a small threshold, the system

tems transmitted to ground stations consists largely

displays a list of p ossible faults ranked by likeliho o d

of information from these sensors.

with an asso ciated graphical display of the probabili-

ties of the faults. In addition to providing a probability

Figure 5 shows the graphical structure of a Bayesian-

distribution over faults, the system also generates rec-

network for an OMS engine. The Bayesian networks

ommendations ab out ideal action. A list of p ossible

that we constructed for Shuttle propulsion systems are

actions, ranked by exp ected utility is displayed.

notable in that they include rich representations of sen-

sor failures and errors. The mo dels consider failures

of sensors as well as failures of core comp onents of

5 DISPLAY DECISION MAKING

propulsion systems. The sensor-error mo dels included

in the Bayesian networks represent information ab out

Several approaches to the managementofdisplayand

the validity of sensors as wellastheway that di erent

information access were implemented in the initial

sensors fail. Exp erts with long-term exp erience make

Vista systems [Horvitz et al., 1992]. These include

use of evidence ab out sensor failure, suchasnoting

means for exibly controlling the detail of information

that a sp eci c class of sensor is generating a sinusoidal

presented ab out sp eci c subsystems dep ending on the

output or is trending upward when a physical mo del

context and inference ab out anomalies, the use of a

only makes p ossible decreasing quantities of the mea-

list of faults, sorted by probability, as an active index

sured substance.

into related trend information, and the prioritization

of faults by the exp ected cost of delay to review the

4.3 REPRESENTING ACTION AND TIME

faults.

To build a exible approach to display, data Moving from inference ab out anomalous states to the

templates|contiguous sets of related information| realm of actions, we mo deled decisions and outcomes

were designed for each subsystem (the two OMS sys- for di erent anomalies and contexts. Weenumerated

tems and the three RCS systems). A sp ectrum of tem- available actions and considered the time-dep endence

VISTA erator's decisions. Wenow explore metho ds to char-

acterize the costs and b ene ts of displaying di erent

con gurations of information.

Let us rst consider a quantitywe refer to as the

exp ected value of revealed information (EVRI). EVRI

is the exp ected value of considering additional quanti-

ties of information that is available with certainty,yet

is hidden from a decision analysis. The exp ected util-

ity of considering previously hidden information must

be evaluated in the context of a complete decision-

theoretic analysis, taking advantage of al l of the avail-

able information. EVRI di ers from the exp ected

value of information (EVI) in that EVI includes a

consideration of uncertainty ab out the state of obser-

vations. In the case of EVRI, an automated display

Figure 6: A screen from Vista-I displaying how a prob-

manager has access to all available data. EVI is ap-

lem with the left OMS is handled. The left-OMS data

propriate metric for display in cases where a system

template has b een expanded from the core data set

must exp end e ort b efore access is gained to sensor

by the display manager. Relevant trend information is

values.

displayed on the right side of the screen.

Consider the case of monitoring complex systems such

as Shuttle propulsion systems. Human op erators are

charged with reviewing data that is typically accessed

plates were created for each subsystem, spanning a

through a battery of sensors that innervate a mon-

range of completeness from the most detailed to pro-

itored system. We will consider use of a display-

gressively less complete, more abstract presentations

management system to make decisions ab out the na-

of data. We also intro duced levels of detail in the dis-

ture and quantity of evidence E to b e displayed. How-

play of diagnostic information, by allowing the system

ever, we could apply a similar analysis for controlling

to display probabilities of anomalies at more abstract

the display of information ab out other distinctions rep-

levels of the subsystems, such as the probabilityofa

resented in or inferred from a decision mo del.

sensor failure versus a sp eci c sensor failing.

Assume that a monitoring system has access to a

In the main Vista display, templates for the di erent

set of sensed observations, E. The probabilities over

subsystems are con gured in an invariant pattern, in-

hyp otheses of interest H (e.g., failures in a moni-

tro ducing spatial stability in the lo cation of informa-

tored system), inferred with a gold-standard diagnos-

tion ab out di erent subsystems. At run time, the re-

tic mo del that takes into consideration all available

sults of inference are used to mo dify the amountofde-

data, p(H jE;), can b e used to compute the exp ected

tail displayed ab out each subsystem; the data template

G

utility of the gold-standard action, A ,

for a subsystem app ears to telescop e from a compact

X

summary into a larger, more complete presentation.

G

A = arg max u(A ;H )p(H jE;) (1)

i j j

The overall p ositioning of the data templates for di er-

A

j

ent subsystems remains the same during the resizing of

presentations on sp eci c subsystems, minimizing the

Let us now hide some evidence from the analysis and

e ort needed to lo cate information. Preferred levels of

consider, with the same decision mo del, the value

detail for each subsystem were prede ned for di erent

of revealing or displaying a subset of observations,

contexts (e.g., critical OMS burn, orbital coast, etc.).

E  E. We compute a p otentially revised optimal

D 

Beyond automated control, we designed the interface

action, A , based on the revised probabilitydistri-

to allow users easy manual access to any information

bution, p(H jE; ). We compute the b est action by

available to the Vista system.

substituting the revised probability distribution into

Equation 1,

The combination of decision-theoretic inference with

X

exible access to a range of detail on subsystems has

D 

A (E ) = arg max u(A ;H )p(H jE; ) (2)

i j j

worked well in Vista. Nevertheless, wehave continued

A

j

to pursue principles for controlling displays. We will

D 

now fo cus on newer metho ds, several of whicharebe-

Wemust evaluate the exp ected utilityof A with the

ing validated for the forthcoming Vista-I I I system for

gold-standard probability distribution, considering al l

the Mission Control Center.

of the available evidence E,

X

D  D 

eu [A (E )] = u[A (E );H ]p(H jE;) (3)

j j

5.1 EXPECTED VALUE OF REVEALED

j

INFORMATION

We can now de ne the exp ected value of revealed in- The goal of display management in time-critical sit-

formation. The EVRI(e; E ; E) is the exp ected value of uations is to maximize the exp ected utilityofanop- Operator Experience

Displayed Operator Information Decision User model Gold-standard model

Operator

Delay Action,t Duration of Process

Figure 8: Weemploy a user and a gold-standard de-

cision mo del to determine the value of displaying ad- ditional information. The user mo del can b e a gold-

Utility tiations or

System standard mo del acting on a subset of instan

ting a user's causal knowledge and

State a mo del represen preferences.

E E2 E3

1 En

Let us consider the situation where a subset of all in-

formation (e.g., a subset of relevant telemetry from

Figure 7: An in uence diagram representing the prob-

the Shuttle) is revealed by an automated display man-

lem of controlling the display of information in time-

ager to a decision maker charged with resp onsibilityfor

critical situations. We represent user decisions as

making a decision. For now, we assume that the deci-

chance variables.

sion maker is an exp ert who acts in accordance with a

gold-standard diagnostic mo del but requires increasing

amounts of time to review larger quantities of informa-

revealing a set of additional information e in a context

tion. We can use NEVRI to consider the costs versus

de ned by the set of previously revealed information

b ene ts of displaying alternate subsets of available in-

E and the set of all available evidence E. Note that

formation. We can search through all con gurations

the EVRI is zero if the action do es not change with



of evidence to nd a subset of information, e ,that

the revealed information.

maximizes the exp ected utility,

D  D 

EVRI(e; E ; E)=eu[A (E + e; E)] eu [A (E; E)]



e = arg max NEVRI(e; E ; E) (8)

(4)

e

EVRI do es not takeinto account the costs p otentially

For the general case, nding the b est subset of moni-

asso ciated with the review of increasing quantities of

tored information to display requires a searchover all

information. Action maybedelayed if a decision-

combinations of data. We will discuss practical strate-

making agentmust pro cess additional information.

gies that coincide with extensions to current Vista dis-

Such time delays maychange the b est action or in-

play p olicies in Section 7. First, we will generalize

cur signi cant losses in the maximum exp ected utility

EVRI to the exp ected value of displayed information

in time-critical settings. The net exp ected value of re-

(EVDI).

vealing information (NEVRI) includes the costs and

b ene ts of reviewing the additional information. Let

us assume that costs are based solely in deterministic

5.2 GENERALIZATION TO VALUE OF

delays t(e) required to review information e. The b est

DISPLAYED INFORMATION

decision given consideration of only evidence E is,

EVRI considers the costs and b ene ts of revealing sub- X

D 

A (E ) = arg max u[A ;H ;t(E )]p(H jE; )(5)

sets of all available observations in the context of a i j j

A

j

gold-standard mo del. In the general case, we cannot

assume that a user will act in accordance with a gold-

The exp ected value of this decision is,

standard diagnostic or decision mo del. A novice deci-

X

sion maker may make sub optimal decisions relativeto

D  D 

eu[A (E )] = u[A (E );H ;t(E )]p(H jE;)(6)

j j

a gold-standard decision-theoretic mo del yet still b e re-

j

lied up on for action given traditional desires for having

ahuman in the lo op. Alternately,wemay wish to use

conditioning the probability of states of the system on

automated display management to train a user ab out

the complete set of available evidence. The NEVRI

D 

the most imp ortant information to consider. Wenow

can b e computed by considering the b est actions A

generalize the EVRI to the exp ected value of displayed

and exp ected utilities of these actions, given a consid-

information (EVDI) by considering an op erator's ac-

eration of t(E + e)versus t(E ).

tions in resp onse to varying quantities of displayed in-

D  D 

NEVRI(e; E ; E)= eu[A (E + e)] eu [A (E )] (7)

formation and the value of this information to the user

from the p ersp ective of the gold-standard mo del.

Note that we can easily generalize NEVRI to include

We wish to automate decisions ab out display to opti- information ab out the uncertainty asso ciated with the

mize the exp ected value of an op erator's actions. Al- time required to review information.

though we can consider several di erent forms of infor-

H1 H2 H1 H2

mation, including the output of automated inference,

we shall again cast the discussion in terms of decisions

ab out the display of subsets of all monitored observa-

E  E.Tomake a decision ab out the nature tions, E

1 E2 E3 E2 E3

and quantity of evidence to reveal to the system user,

we consider the likeliho o d of alternative user actions

eliho o ds of delay as functions of displayed data.

and lik E4 E4

Within the context of the time-critical decision prob-

lems broadly captured by the in uence diagram in Fig-

Figure 9: Constructing mo dels of the op erator's view

ure 2, the problem of displaying information in time-

of a system. Exp ert trainers of propulsion-systems

critical settings is represented by the in uence diagram

op erators found it useful to initiate the task of de-

displayed in Figure 7. We transform the human op er-

veloping Bayesian mo dels of a trainee's causal knowl-

ator's decision and delaytochance variables that are

edge by pruning away subtle distinctions and dep en-

in uenced by the information displayed. We represent

dencies from the gold-standard Bayesian networks and

the user's exp ertise or background by conditioning the

reassessing probabilities.

action and delaynodesonavariable representing ex-

p ertise. If wehave certain knowledge ab out the user's

background, we condition the decision no de on this

function of displayed information. Assume that a gold-

information, represented by an information arc.

standard mo del indeed represents the preferences and

To simplify our equations, we will assume that the de-

probabilistic relationships of the b est exp ertise avail-

lay in the human op erator's action is a deterministic

able, but that the limited availability of exp erts and

function of the quantity of evidence displayed and is

p olicies of an institution lead to situations that require

indep endent of the ultimate action taken. We will keep

nonexp erts to make decisions. We wish to develop a

the op erator's exp ertise implicit. Given an initial dis-

display manager that continues to reason ab out how

play of system observations E , and a complete set of

alternativequantities of information will change the

evidence E known to a display manager, the net value

nature and timeliness of such decisions.

of displaying additional information (EVDI) e is,

6.1 BAYESIAN MODELS OF USER

BELIEFS

EVDI(e; E ; E)=

X X

To control or design a display based on EVDI, we need

p(A jE; e; ) u[A ;H ;t(E + e)]p(H jE;)

i i j j

to gain access to knowledge ab out p(A jE; )and t(E )

i

i j

(or, more generally, p(t jE ; A;  )). For simpli cation,

k

X X

let us assume that wehave access to deterministic in-

p(A jE; ) u[A ;H ;t(E )]p(H jE;) (9)

i i j j

formation ab out the amount of time required to review

i j

information as a function of the quantity of informa-

tion displayed. We will fo cus on inference ab out user

actions. We explored two approaches to mo deling ac-

We can generalize EVDI to include a consideration

tions of op erators as a function of their training and

of other factors including uncertainty ab out relevant

the data displayed.

information that a user may already knoworhasac-

quired from other sources. The probability distribu-

One approach to mo deling an op erator's actions is to

tion over delaymay dep end on such factors as the op-

work with exp erts with exp erience with training p eo-

erator's uncertainty ab out the b est decision to make

ple in their area of exp ertise, to construct directly

and the p erceived criticality of a situation. We can ex-

mo dels of user action. These mo dels output prob-

tend the EVDI measure by considering uncertaintyin

ability distributions over user actions given assumed

delays, and by conditioning the probability distribu-

sets of observations displayed to an op erator. Build-

tion over length of delayonthequantity of evidence,

ing probabilistic mo dels that make inferences ab out

and on such information as an op erator's predicted

p(AjE; ) action is dicult; the mo deling and assess-

uncertainty ab out the b est action [Horvitz, 1995].

ment of actions typically requires the analysis of a large

numb er of situations. It can b e more ecienttocon-

struct Bayesian user mo dels that represent, from an

6 MODELING USER ACTIONS

exp ert trainer's p ersp ective, the causal probabilistic

relationships assumed by trainees at varying levels of

In time-critical contexts, the goal of a display man-

exp erience. These could b e used to mo del the b eliefs

ager is to assist a user with taking the b est action as

of users ab out anomalies given displayed data.

so on as p ossible. As indicated by Equation 9, a key

task in implementing approaches to display manage- As part of the Vista e ort to build mo dels of user

ment based on EVDI is the development of a proba- action for EVDI-based display, exp erts were asked to

bilistic mo del of an op erator's b eliefs and actions, as a build Bayesian networks that could b e used to make

7 FLEXIBLE DISPLAY OPTIONS inferences ab out the b eliefs of users in resp onse to ob-

servations. We assessed from exp ert trainer's mo dels

of the causal knowledge and uncertain dep endencies

EVRI and EVDI can b e employed in a varietyofways

that represent exp ertise at di erentlevels of training.

to enhance the information displayed for time-critical

The goal of this work was to compute the probabili-

monitoring. EVRI is more appropriate for use in situa-

ties that a user would assign to the presence of system

tions where exp ert engineers will b e making decisions,

faults, after reviewing sets of observations. We found

given the EVRI assumption of gold-standard decision

that exp ert trainers were comfortable building such

making. Wehave found EVRI to b e more appropri-

probabilistic mo dels representing users' b eliefs ab out

ate for real-time monitoring of Shuttle propulsion sys-

a monitored system. The mo dels of user b elief for

tems b ecause real-time decision making is limited to

Shuttle propulsion systems were tested and validating

exp erts. EVDI promises to b e useful for building e ec-

as p erforming like users at some level of training or

tive systems for training and education. Applications

exp erience.

of EVDI require greater amounts of engineering e ort

As highlighted by Figure 9, exp erts found particularly

than EVRI b ecause of the mo deling of user's b eliefs

useful a strategy of b eginning the task of mo deling a

and actions required for EVDI.

nonexp ert's view of diagnosis and decision making by

The most general use of the display metrics involves a

examining the gold-standard decision mo del and prun-

search through all combinations of information avail-

ing away distinctions and dep endencies considered to

able for display. Rather than consider a searchover

b e absent in trainees' mo dels of a monitored system.

D

all subsets of evidence, we can use the metrics at run-

We use p(H jE; ) to refer to the predictions of these

time or design time to make coarser display decisions,

Bayesian user mo dels ab out the b eliefs of a user given

and employapproximate analyses based on single-step

displayed information E .

analysis or on a limited lo okahead.

The metrics can b e used as to ols for evaluating and re-

ning existing information layouts and display strate-

gies. For example, we can examine the value of mo di-

cations to prede ned clusters of related information,

6.2 FROM BELIEFS TO ACTIONS

with an eye to optimizing a static display, or for doing

real-time tailoring of the templates dep ending on con-

text. We can also use the metrics to make decisions

Assume that wehave a mo del that exp erts certify

ab out the display of auxiliary clusters of information.

as providing accurate inference ab out an op erator's

Computer-based monitoring systems often displaya

b eliefs as a function of evidence. Howcanwe use

default, core set of data, yet mayhave access to a

D

p(H jE; ) to compute the probability distribution

large quantity of supp ortive information that is not

over actions by the user, p(A jE; )? If we had ac-

i

typically relevant to decisions. System op erators may

D

cess to an op erator's utilitymodel,u (A ;H ;t), rep-

i j

wish to review a stable pattern of key variables, yet

resenting the user's p erception of outcomes and delays,

have access to supp ortive auxiliary information when

and plausible set of actions and their dep endencies, we

it mightchange their decision. As an example, the

could cho ose the action that maximizes the exp ected

main screen for Shuttle propulsion systems has not

utility from the user's p ersp ective and, thus, generate

traditionally included information ab out sensor trend

D 

a \b est" user action, A .However, we cannot as-

information, yet distinctions ab out trends are repre-

sume that users combine their b eliefs in a coherent,

sented in the diagnostic mo dels and can provide valu-

decision-theoretic manner.

able information the likeliho o d of sensor failures. De-

cisions ab out auxiliary information have b ecome esp e- Several approaches show promise for allowing us to ap-

cially salient when monitoring systems are upgraded proximate p(A jE; ). We assessed from exp erts user-

i

from character-based displays to graphic workstations mo del preferences to explore metho ds for combining

making available more display real-estate. user b eliefs ab out faults into user actions. Exp erts

identi ed di erences b etween the preference mo del of

The EVRI and EVDI metrics can b e used to deter-

a seasoned op erator and a less-exp erienced p erson.

mine when it is valuable to display auxiliary or more

detailed sets of data, by considering the value versus Given these mo dels, wehave explored several metho ds

the costs of displaying auxiliary information. For rea- for mapping the inferred user's b eliefs into the likeli-

soning ab out the value of displaying auxiliary clusters ho o d of actions, p(A jE; ) for display management.

i

of information, wecontinue to monitor telemetry and In one approach, we assume that op erators will at-

p erform decision-theoretic inference. As highlighted in tempt to maximize exp ected utility. In another, we

Figure 5, we compare the exp ected value of decisions assume more conservatively, that the likeliho o d of ac-

cor e

with the core information display, E ,versus with tions is a monotonically increasing function of the in-

cor e aux

. If the additional in- di erent extensions, E + E ferred exp ected utility of the actions given the user's

i

formation leads to decisions with higher exp ected util- preference mo del. Studies of the b ehavior of users with

ity than the decision indicated by the information in di erentlevels of training will b e useful for con rming

the core display, the auxiliary information is displayed. and tuning suchusermodels.

STATUS AND FUTURE WORK Aux Dataset 1 8

Aux Dataset 4

We implemented in a prototyp e of the forthcoming t critical Core Vista-I I I system the use of EVRI to highligh

Dataset information with color to prioritize the review of data.

We are currently exploring the use of EVRI for auto-

mated display of auxiliary information and for control-

Aux Dataset 5 e will

Aux Dataset 2 ling the telescoping of information templates. W

be validating the new functionalities and exp ect the

Aux Dataset 3

Vista-I I I system to b e certi ed for ight at the Mission

Control Center in the coming year. Overall, exp erts

and trainees havebeenenthusiastic ab out the display

Figure 10: Decisions ab out auxiliary data. In one ap-

management and decision-theoretic recommendations

proach to display decision making, a decision-theoretic

provided in the evolving Vista system.

analysis is used to consider the b ene ts of displaying

We are working to further extend the display-

auxiliary information, b eyond a standard core dataset.

management metho ds. In particular, we are ex-

ploring more sophisticated display-managementtech-

niques that takeinto consideration incompleteness or

The display-management metrics can provide a formal

inaccuracy in the mo dels of diagnosis and decision

foundation for such user-interface functionalities as the

making used to compute advice. We wish to endowa

telescoping of templates of information ab out subsys-

reasoning system with explicit metho ds for evaluating

tems that was intro duced in Vista-1. If we are mon-

the con dence in its diagnostic conclusions. Such self-

itoring several subsystems, as in the case of Shuttle

awareness ab out mo del incompleteness, coupled with

propulsion systems, we can generate, for each subsys-

knowledge of when a human decision maker is likely

tem, a set of templates of progressively greater detail

to have deep er insights than the computer-based rea-

and size. EVRI or EVDI can b e used to make decisions

soner, will b e valuable in building genuine decision-

ab out the escalation of templates, from a summary or

making asso ciates.

core set of data to larger, more detailed expansions.

We are concerned with the mo deling e ort that can b e

We do not have to mo del explicitly the costs of re-

required to build systems based on EVDI. We are in-

viewing information to derivevalue from the metrics.

terested in developing more ecient metho ds for build-

For example, we can use EVRI or EVRI to search for

ing and learning mo dels that describ e with delity the

minimal sets of evidence that are consistent with the

b eliefs and actions of users in resp onse to displayed in-

action that has the greatest exp ected utility from the

formation. We are excited ab out recent innovations in

p ersp ective of the gold-standard mo del. The minimal

learning Bayesian networks from data and for combin-

information strategy can b e combined with means that

ing exp ert mo dels with data for building these mo dels

allow user's to request with ease additional variables of

[Heckerman et al., 1994].

interest. As another approach, we can employ EVRI to

We lo ok to future research in cognitive psychology for display auxiliary information whenever the additional

information ab out the costs asso ciated with the re- information can change the b est action taken, regard-

view of information, including studies of how increased less of the criticality, or taking into consideration only

amounts of information and clutter can b e exp ected to gross indications of criticality.

lead to delays, confusion, and, ultimately, to sub opti-

In another application of the metrics, rather than us-

mal decisions. Psychological studies can also giveus

ing EVRI or EVDI to edit the displayed data, we seek

insights ab out the value of such user-interface actions

to make more conservative decisions ab out the high-

as highlighting displayed data with color, and costs

lighting of information as a function of the situation

asso ciated with the instability of dynamically chang-

and user. We again employa myopic or limited lo oka-

ing interface con gurations. We are also interested in

head search through all data b eing displayed, or eval-

gaining deep er insights ab out the p erceptions of users

uate classes of information as clusters. For the my-

ab out automated display decisions versus completely

opic analyses, we compute the value of revealing single

manual access of auxiliary information. Such informa-

pieces of data considered as hidden from the decision

tion ab out preferences and p erformance can allowusto

maker, in the context of the rest of the displayed data.

build more sophisticated, e ectivedisplay managers.

Data can b e highlighted with a range of intensityor

Beyond display management applications, EVRI and color in accordance with the increasing value of data

EVDI can b e harnessed to make automated decisions as indicated by the EVRI or EVDI. Wecancho ose

ab out the most valuable information to transmit to re- to highlight only the top n observations, a number

mote decision makers. In such applications, we balance that may b e determined dynamically by the quantity

the value of information to enhance the timeliness or of data and time criticality. A similar analysis can

quality of decisions with the costs of transmitting the b e used to analyze the overall imp ortance of di erent

data. The metho ds provide a utility-based foundation classes of clustered data to control the highlighting of

for controlling the incremental transmission of time- di erent classes of data.

source constraints. In Proceedings of Third Work- critical information over limited-bandwidth channels.

shop on Uncertainty in Arti cial Intel ligence, pages

We hop e that the metrics and metho ds we describ ed

429{444, Seattle, Washington. American Asso cia-

will b e useful to others tackling issues surrounding

tion for Arti cial Intelligence. Also in L. Kanal, T.

real-time monitoring and control, and oine design

Levitt, and J. Lemmer, ed., Uncertainty in Arti cial

of informational displays. We b elieve that the combi-

Intel ligence3,Elsevier, 1989, pps. 301-324.

nation of automated decision-supp ort and automated

[Horvitz, 1988] Horvitz, E. (1988). Reasoning under display systems will allowpeopletokeep pace with the

varying and uncertain resource constraints. In Pro- growing complexity and stakes asso ciated with under-

standing and controlling the machines and software

ceedings AAAI-88 Seventh National Conferenceon

that we rely on to accomplish critical tasks.

Arti cial Intel ligence, Minneapolis, MN, pages 111{

116. Morgan Kaufmann, San Mateo, CA.

Acknowledgments

[Horvitz, 1990] Horvitz, E. (1990). Computation and

Action Under BoundedResources. PhD thesis, Stan-

ford University.

We appreciate the long-term contributions on the

Vista Pro ject by Jim Martin, Corinne Roukangas,

[Horvitz, 1995] Horvitz, E. (1995). Transmission and

Kevin Scott, Sampath Srinivas, and Cynthia Wells.

display of information: A decision-making p ersp ec-

We thank Jack Breese, Rob ert Fung, David Hecker-

tive. Technical Rep ort Microsoft Technical Rep ort

man, KathyLaskey,Paul Lehner, and Michael Shwe

MSR-TR-95-13, Microsoft Research, Microsoft.

for useful feedback on Vista research. We are grateful

to colleagues at the participating sites, including the

[Horvitz et al., 1989] Horvitz, E., Heckerman, D., Ng,

NASA Johnson Space Center, Ro ckwell Palo Alto Lab-

K., and Nathwani, B. (1989). Heuristic abstrac-

oratory, NASA Ames ResearchCenter, and Stanford

tion in the decision-theoretic Path nder system. In

University. This work was supp orted by the NASA

Proceedings of the Thirteenth Symposium on Com-

Johnson Space Center, Ro ckwell Science Center, the

puter Applications in Medical Care, Washington,

DC, pages 178{182. IEEE Computer So cietyPress, Ro ckwell Space Op erations Company, and National

Los Angeles, CA. Science Foundation Grant IRI-9108385.

[Horvitz et al., 1992] Horvitz, E., Ruokangas, C.,

References

Srinivas, S., and Barry,M.(1992). A decision-

theoretic approach to the display of information for

[Breese et al., 1991] Breese, J., Srinivas, S., and Fiske,

time-critical decisions: The Vista pro ject. In Pro-

R. (1991). A utility-based pilot vehicle interface.

ceedings of the ConferenceonSpaceOperations and

In Proceedings of DARPASymposium on Associate

Automation and Research, January, 1992, NASA

Technology, pages 259{266, George Mason Univer-

Johnson Space Center. National Aeronautics and

sity,Fairfax, VA.

Space Administration.

[Horvitz and Rutledge, 1991] Horvitz, E. and Rut-

[Bruner et al., 1956] Bruner, J., Go o dnow, J., and

ledge, G. (1991). Time-dep endent utility and action

Austin, G. (1956). A study of thinking. Wiley and

under uncertainty.InProceedings of Seventh Con-

Sons.

ference on Uncertainty in Arti cial Intel ligence, Los

[Doyle et al., 1989] Doyle, R., Sellers, S., and Atkin-

Angeles, CA, pages 151{158. Morgan Kaufman, San

son, D. (1989). A fo cused, context-sensitiveap-

Mateo, CA.

proach to monitoring. In Proceedings of the Eleventh

[Mclaughlin, 1987] Mclaughlin, J. (1987). The utility-

International Joint ConferenceonArti cial Intel li-

directed presentation of graphical simulation. Tech-

gence, Detroit, MI, pages 1231{1237. International

nical Rep ort TR-87-59, Stanford University.

Joint Conference on Arti cial Intelligence.

[Miller, 1956] Miller, G. (1956). The magical number

[Heckerman et al., 1994] Heckerman, D., Geiger, D.,

seven plus or minus two; some limits on our capacity

and Chickering, D. (1994). Learning Bayesian net-

for pro cessing information. Psychological Review,

works: The combination of knowledge and statis-

63:81{97.

tical data. In Proceedings of Tenth Conferenceon

[Morrin et al., 1961] Morrin, R., Forin, B., and

Uncertainty in Arti cial Intel ligence, Seattle, WA,

Archer, W. (1961). Information pro cessing b ehav-

pages 293{301. Morgan Kaufmann.

ior, the role of irrelevant stimulus information. Jour-

[Horvitz, 1987a] Horvitz, E. (1987a). A multiattribute

nal of Experimental Psychology, 61:89{96.

utility approach to inference understandability and

[Simon, 1972] Simon, H. (1972). The theory of prob-

explanation. Technical Rep ort KSL-28-87, Medical

lem solving. Information Processing, 71:261{277.

Computer Science Group, Section on Medical Infor-

matics, Stanford University, Stanford, CA.

[Waugh and Norman, 1965] Waugh, N. and Norman,

D. (1965). Primary memory. Psychological Review,

[Horvitz, 1987b] Horvitz, E. (1987b). Reasoning

72:89{104. ab out b eliefs and actions under computational re-