A brief introduction of Detection Theory

Dr. Zhuanghua Shi Changes of Schedule

• Due to NCP Day next Wednesday, Course C schedule changes as follows:

• Date Content Dec. 12 Signal Detection Theory Dec. 19 --- (NCP Day) Jan. 09 Practical – Signal Detection Analysis Jan. 16 Exam: Theoretical part

2 A demo: Detection task

• There will be 4 x-ray photos of travel suitcases presented briefly. Your task is to identify if there is a gun inside. Please write down the number of the photos.

3 +

4 +

5 +

6 +

7 8 Results

Photo No. Gun No Gun Gun No Gun

Yes 11 2 1 2 11

No 15 24 2 11 2

3 0 13

4 0 13

9 Prior knowledge Expectation

Perception & Action Sensitivity Sensation External environment Brain

10 Sensitivity and Criterion

• Performance is a function of TWO properties of the observer: • How well the observer perceives stimuli – or sensitivity • And, how does the observer choose to respond – or criterion • We need methods to assess the two independently • SIGNAL DETECTION THEORY

11 History of signal detection theory

• WWII, researchers concerned with maximization of correct detection responses and minimization of false detection responses • No way to optimize both • First introduced by researchers (Marcum, 1960) • How to discriminate radar signal and noise Signal and noise

Signal Noise S + N = 2 2 2

1.5 1.5 1.5

1 1 1

0.5 0.5 0.5

0 0 0

-0.5 -0.5 -0.5

-1 -1 -1 0 50 100 150 200 250 300 350 400 450 500 0 50 100 150 200 250 300 350 400 450 500 0 50 100 150 200 250 300 350 400 450 500

2 2

1.5 1.5

1 1

0.5 0.5

0 0

-0.5 -0.5

-1 -1 0 50 100 150 200 250 300 350 400 450 500 0 50 100 150 200 250 300 350 400 450 500 Signal Noise Sensory signal and decision making

• are always accompanied with background noise • luminance detection (background luminance) • target detection (against distractors) • For any sensory inputs the brain need to interpret its meaning (e.g., signal or noise) • Decision making (Criterion) • Simple two-alternative force choice • Above that criterion: Yes • Below that criterion: No Confusion matrix – decoding the observer

• Different combinations of stimuli and responses Stimuli Signal Noise Yes Hit False Alarm Response No Miss Correct Rejection What is noise?

External noise Internal noise • interferes the signal • corrupts signal representation Signal and Noise: Statistical view

• Noisy signal representation, Rs:

!" = $ + & where S is signal, N is noise

• How can we decode the signal from !"? Distribution of signal representations

• If noise is white, then the received noisy signal will distribute as a normal distribution • yielding a normal distribution of signal representations • The mean of this distribution would reflect the signal Mean signal, S Probability

Signal representation

18 Distribution of noise

• Gaussian distribution of noise representation near 0 • Usually we assume signal is positive, that is, !" > !$

Mean noise, N Probability

Noise representation

19 Decoding the signal

• Decision can be made based on the information from two distribution (e.g., likelihood)

N S

20 Sometime decision is hard!

N S

21 When you have to make a hard decision, flip a coin. Why? Because when that coin is in the air… You suddenly know what you’re hoping for.

Bias emerges !

22 Decision requires a criterion

• By comparing signal representation with a criterion

RS < C: Noise RS > C: Signal

N C S

23 Easy and hard decisions

• If the signal representation is far from the criterion, the decision is easy • The closer it gets, the harder the decision

Potential S Potential N

N C S

24 Describing the S, N, and C

Misses Hits Correct rejections False alarms p(Rs < C|S) p(Rs > C|S) p(Rs < C|N) p(Rs > C|N)

N C S N C S

S N Y H FA N M CR 100% 100%

25 Describing the S, N and C

• p(H) + p(M) = 1 , so knowing p(H) is sufficient • p(FA) + p(CR) = 1 , so knowing p(FA) is sufficient • We can describe S, N, and C fully by considering only hits and false alarms

S H Y H FA N M CR 100% 100%

26 Describing the S, N and C

• Overall decision difficulty is determined by the signal/noise relationship – sensitivity • Actual performance also depends on decision criterion Sensitivity

N C S

27 Sensitivity and decision criterion

• Sensitivity (Discriminability) • Measure of how close signal and noise are • Supposed to be a property only of the sensory process • Decision Criterion () • Measure of what is considered noise and what signal • Susceptible to motivation, strategy, etc. • Sensitivity and criterion are independent

28 Sensitivity and decision criterion

Unbiased criterion Biased High Low sensitivity Low

29 Computing sensitivity and criterion

• What are the measuring units of S and N? • Independently of the measuring scale (e.g., sound frequency, line length, etc.) we can standardize the distribution by computing deviance scores

Real measure [x] Standard measure [z]

30 Normal distribution

• Assumption – N and S are normally distributed

31 Normal distribution

• For each probability, we can compute a z score

Z score – a horizontal distance measure

• p = 97.7%; z = 2 • p = 2.3%; z = -2

32 Computing sensitivity

• d’ = difference between means of S and N d0 = µ µ S N d’ d0 = z(H)+z(CR) = Z(H) Z(FA)

33 Effects of N and S variability

• If N and s variability differ, individual variability has to be computed • Sensitivity is then d a d’

(µs - µn ) d a = 2 2 s s +s n 2

34 Computing decision criterion

• Criterion influences overall frequency of YES responses: 1. p(H) = 1, p(FA) = .84; d’ = 2 2. p(H) = .84, p(FA) = .16; d’ = 2 3. p(H) = .16, p(FA) = 0; d’ = 2 1 2 3

35 Computing decision criterion

• Likelihood-ratio, β = p(C|S)/p(C|N)

1. p(C1|S) = .004, p(C1|N) = .242, β = .018 – liberal criterion

2. P(C2|S) = .242, p(C2|N) = .242, β = 1 – neutral criterion

3. P(C3|S) = .242, p(C3|N) = .004, β = 55 – conservative

1 2 3 Probability

36 Computing decision criterion

• Response bias, C = -(zH + zFA)/2

1. zH = 3, zFA = 1, C = -2 – liberal criterion

2. zH = 1, zFA = -1, C = 0 – neutral criterion

3. zH = -1, zFA = -3, C = 2 – conservative

1 2 3

Z score

37 Relating criterion and sensitivity – ROC curve

• Receiver operating characteristics – ROC curve • Relation between p(H) and p(FA) for varying response criteria • One curve shows one sensitivity, d’

1,00 0,84; 1 1 2 3 0,16; 0,75 0,84

0,50 p(H) 0,25 0; 0,16 0,00 0,00 0,25 0,50 0,75 1,00 p(FA)

38 ROC and different sensitivities

• Increasing sensitivity increases the slope of ROC curve = switching between ROC curves

39 ROC and different criteria

• Changing criteria shifts the location at a single ROC curve

1 2 3

40 Sensitivity from ROC curve

• The area under ROC reflects sensitivity • Usually, we have only ONE p(H)/p(FA) data point • Problem – how to estimate sensitivity from a single point?

41 Sensitivity from ROC curve

• Solutions: • Assume normal distribution

• Approximate the area

42 Criterion from ROC curve

• The slope of the ROC curve at the observed point

• b = 1 – neutral criterion • b < 1 – liberal criterion • b > 1 – conservative criterion

43 Two-alternative Discrimination task

7.1. THE TWO-ALTERNATIVE DISCRIMINATION TASK 115 • Two signals A and B

• Yes responses are PA and PB • Assume equal variance,

! Figure 7.1: The signal-detection model for two-alternative discrimination, The "′ = % & + % & shaded areas correspond to correct responses. ' ) From Wickens (2002)

more attractive to treat the discrimination stimuli symmetrically. Instead * = −(% & − % & of )speakin/2 g of hits and false alarms, a better representation is to use the ' )proportion of correct responses made to each stimulus. Denote the two signals by A and B and the responses by A and B. The probabilities of correct responses of each type are given by the conditional probabilities PA = P(A|A) and PB = P(B|B). Denote the corresponding proportions by their lowercase equivalents, PA and PB- Figure 7.1 shows the distributional model of the task. The decision axis is represented by a singl44e dimension, and observing a stimulus gives a value along this axis. The distribution of the observations depends on the which stimulus was presented. A decision criterion divides the continuum into response regions associated with the two responses. The shaded areas in the figure correspond to the probabili- ties PA and PB of correctly identifying the two types of stimulus. The decision model of Figure 7.1 also implies the existence of a discrim- ination operating characteristic. By varying the position of the decision criterion, performance can be varied from an emphasis on A responses, with PA 1 and PB 0, to an emphasis on B responses, with PA 0 and PB 1. The relationship of the two probabilities is shown in Figure 7.2. This discrimination operating characteristic is reversed left to right from the detection operating characteristics because the horizontal axis plots correct responses instead of errors. Its interpretation is the same. In par- ticular, the area under the curve has the same meaning as a measure of the distinguishability of the signals. Suppose further that the distributions are Gaussian, with means uA and uB and common variance o2. Because the decision continuum cannot be directly observed, exact values for uA, uB, o2 , and the decision criterion 2-Alternative Force-Choice (2AFC) task

• 2AFC task can be treated as detection of the difference between the two • Suppose Noise (N), and Signal (N+S) both are normal distributed • The variance of the difference of the two has double! (See below)

• !" = ! $% > $' = ! $% − $' > 0 ~ +(-%, 2) 6.3. SECTION B: THEORY 179

dЈ 0.4 S + N 3 0.3 N 2 σ = 1 0.2 !" = 1( ) 2 0.1 0 0.4

2′ = 5 !" 2 density Probability 0.3 0.2 σ = √2 (S + N)−N 0.1 0 −4 −3 −2 −10123456 Sensory magnitude (z units) 45

FIGURE 6.8 Graphical illustration of how d Ј can be calculated for an unbiased 2AFC task. Top: distributions of noise alone ( N ) and signal-plus-noise ( S ϩ N ) separated by d Ј . Bottom: distribution of the difference between the two distributions: ( S ϩ N ) Ϫ N . Note the different σ s for the upper and lower distributions. The z values along the abscissa are normalized to the σ of the two distributions in the top, not bottom panel. See text for further details.

Calculation of d؅ for Biased 2AFC 6.3.4.2 Let us consider the situation in which the two alternatives are presented sequen- tially, i.e., 2IFC. Figure 6.9 plots the distribution of differences in sensory magni- tude between those in the fi rst interval (X1) and those in the second interval (X2), i.e., the distribution of X1 Ϫ X2. Note that there are now two distributions, one for signal present in the fi rst interval and one for signal present in the second interval. The two distributions will be separated by 2d Ј and have σ s of ͙ 2 (see above). If the observer is biased towards responding to one interval more than the other, then their criterion C will be non-zero. The observer’s decision rule is “ 1 ” (fi rst interval) if X1 Ϫ X2 Ͼ C, and “ 2 ” (second interval) if X1 Ϫ X2 Ͻ C. As explained in Section A, the key to calculating d Ј for a biased 2AFC task is to classify the responses in terms of hits and false alarms, where a “ 1 ” response is scored as a hit when the signal is in the fi rst interval and a false alarm when the signal is in the second interval. One can see from Figure 6.9 that:

zpH()(ϭЈϪ d C )/2 (6.17)

and

zpFϭϪЈϪ d C () ( )/2 (6.18) pashler-44093 book December 18, 2001 10:12

Models with Simpler Assumptions (or None at All?) 59

Table 2.8 Evaluation of Sensitivity and Bias in Two-Response Classification

Number of Task Measure Proportions Conclusion

Psychometric function—detection Absolute threshold 1 Bias Psychometric function—discrimination PSE 1 Bias JND 2 Sensitivity Speech classificationStimuli intensity Boundary location and sensitivity 1 Bias Trading relation 2 Change in bias? Sensitivity? •FalseDetection rate is prop. to Difference the sensitivity between 2 Sensitivity “yes” rates for two • Yet, it includes types of lures • Strictly speaking, a null stimulus (noise alone) should be used, so we can distinguish the sensitivity from response bias

1.0 4.0

0.8 Ј

d 3.0 0.6 2.0 (“yes”) 0.4 P

0.2 Cumulative 1.0

0 1 234567 0 1 234567 Stimulus number Stimulus number (a) (b) Handbook of Experimental , vol. 4 Figure 2.10 A psychometric function for detection. NOTE: Stimulus 1 is Noise, and the other stimuli are increasing nonzero levels of intensity. In panel (a), P(“yes”) is plotted, in panel (b), the z score of this value is plotted. The plot in panel (b) can also be 46 interpreted as portraying values of cumulative d′.

the noise-alone stimulus was not immedi- In a discrimination context, a null stimu- ately appreciated, but from an SDT perspec- lus is not used, and it is the difference thresh- tive it is obviously necessary to distinguish old whose value is sought.2 Historical prece- sensitivity from response bias. The observer dence (Fechner is responsible, according to sets a criterion along the decision axis, re- Jones, 1974) and lasting influence belong sponding “yes” to points above and “no” to the method of constant stimuli: A stan- to points below it. Traditionally, the datum dard stimulus (usually drawn from the middle most often abstracted from a psychometric of the stimulus range) is presented on each function ( frequency-of-seeing curve, in vi- trial, and the observer labels each comparison sion) is the (absolute) threshold, the stimu- lus value corresponding to some fixed per- 2Two meanings of threshold must be distinguished. Ear- formance level such as 50% “yes.” This is lier I presented examples of threshold theories that as- sumed discrete representations, but here the term is used just a bias measure, as it depends on a single to refer to the weakest stimulus that can be detected, or proportion. discriminated, and has no theoretical implications. Summary

• Performance in psychophysical paradigm depends both on sensitivity and response/decision criteria • Similar to speed-accuracy trade-off, changes in criteria can mimic changes in sensitivity • It is necessary to develop measures that independently index one and the other • SDT helps us estimate sensitivity and decision criterion independently

47 Additional literature

• Stevens' Handbook of , 3rd Edition, see vol. 4 • : A Practical Introduction, Kingdom & Prins.

48