Familiarity Recognition in Automated Screening Environments Jeffrey G. Proudfoot

Familiarity Recognition in Automated Screening Environments: Utilizing Eye-Tracking Technology as an Intelligence-Gathering Tool

Jeffrey G. Proudfoot The University of Arizona [email protected]

Introduction The September 11th attacks caused a reevaluation of border security policies and procedures in the United States. The systems and manpower allocated to secure infrastructure, citizens, and the border were publicly proven fallible as terrorists involved in the attacks passed through security checkpoints and border crossings unabated. This obvious demonstration of system failure and the perpetual shortage of screening personnel at border crossings, shipping hubs, and centers of public transportation, have collectively served as a catalyst for increased research and development projects to improve screening technologies and overall national security in the United States.

This research proposal addresses the potential application of eye-tracking systems in an automated-screening paradigm to identify individuals possessing knowledge of wanted or dangerous persons. Subjects demonstrating indicators of familiarity during exposure to images of “persons of interest” may be involved in criminal activity or know individuals who are. The implementation of this technology will serve as a valuable instrument for screening agents to identify and investigate suspicious individuals at security checkpoints, as well as glean information from suspects about other persons of interest that can contribute to ongoing investigations.

The Problem Interpersonal communication serves as a primary screening method at border crossings and points of public transportation (airports, bus stations, etc.). Security agents are tasked with the responsibility of identifying, interviewing, and investigating individuals leaking cues of deception or hostile intent during the screening process. Extensive research has been conducted regarding the nature of these deception cues (Ekman, 1985). Unfortunately, the ability of agents to positively identify deception is questionable. Documented success rates are slightly above chance (Miller & Stiff, 1993). Interpersonal Deception Theory posits that the deceiver monitors the target and alters behavior based on cues indicating suspicion or acceptance of the deceit (Buller & Burgoon, 1996). This dynamic approach to deception makes the detection of such tactics extremely difficult. Copious empirical research has shown that humans are incapable of accurately and consistently detecting deception (Derrick, Elkins, Burgoon, Nunamaker, & Zeng, 2010). Familiarity Recognition in Automated Screening Environments Jeffrey G. Proudfoot

An additional challenge encountered by screening agents is that a majority of checkpoints are unable to apply comprehensive screening procedures to each individual due to the high volume of traffic passing through. Whether the cause of this dilemma is a lack of manpower, infrastructure, or automated-screening devices, the outcome is ubiquitous. Individuals are passing through screening environments without being properly evaluated. An obvious consequence of this manpower-volume discrepancy is that people committing crimes, distributing narcotics, trafficking humans, and crossing the border illegally are able to enter the United States without scrutiny.

While a selective-screening strategy is problematic, there is a less-obvious ramification of this approach: an indirect result is the missed opportunity for screening agents to identify individuals who are affiliated with, or have knowledge of, wanted persons or criminal enterprises. A device that could identify familiarity in subjects as they are exposed to images of persons of interest could not only be used to identify candidates for further screening, but it could also allow agents to gather intelligence about wanted persons. Currently, such technology does not exist.

Problem: Screening agents lack the ability, and technology, to identify persons with knowledge of, or familiarity with, wanted persons.

Literature Review Prior research indicates that eye-tracking technology is a feasible solution to detect a subject’s familiarity while viewing images of people’s faces (Ryan Jr, Pavlidis, Rohrbaugh, Marchak, & Kozel, 2003). The following literature review will address the theoretical and empirical foundations upon which this research is based. The purpose of this study is to validate the potential use of this relevant technology to recognize familiarity in automated-screening environments.

The study of eye gaze behavior as an indicator of familiarity is not an undocumented research stream (Ryan Jr, et al., 2003). In a study examining this phenomenon, researchers examined the differences in eye gaze behavior while subjects viewed images of unfamiliar and familiar faces (Althoff & Cohen, 1999). This study revealed that subjects fixated on the internal features of unknown faces more thoroughly than when viewing images of familiar faces (Althoff & Cohen, 1999). In other words, the amount of exposure that individuals have with a person has been proven to alter the eye gaze behavior associated with viewing an image of that person. Interestingly, the findings of this experiment were found to be contradictory to the results of a prior experiment addressing the same issue (Ellis, Shepherd, & Davies, 1979); however, both experiments indicate that familiarity affects eye gaze behavior.

Due to the conflicting outcomes of these methodologically-similar experiments, this study will examine an alternative method of measuring eye gaze behavior as an indicator of familiarity. Guilty Knowledge Tests (GKT) can be used to identify individuals who have specific knowledge Familiarity Recognition in Automated Screening Environments Jeffrey G. Proudfoot of a place, person, object, or other relevant point of interest (Ben-Shakhar & Elaad, 2003). While the traditional format of this method is not directly suitable for rapid screening, the philosophy behind this test can be adapted to a design relevant to this research. In a recent study, researchers merged GKT methods and the evaluation of eye gaze behavior in an attempt to consistently identify subjects possessing guilty knowledge (Derrick, Moffit, & Nunamaker, 2010).

The Derrick et al. (2010) study was comprised of two groups: the bomb-making (guilty) treatment and the control treatment. Participants randomly assigned to the guilty group were tasked with assembling an improvised explosive device (IED). Upon completing this task, each of the participants in the study individually viewed a compilation of images while being monitored with an eye-tracking device. The compilation of photos contained an image of a bomb nearly identical to the one assembled by members of the guilty treatment; however, an obvious component of the bomb was missing in the photo. Later analysis showed that the subjects assigned to assemble the bomb fixated on the location of the missing component, while the control group viewed the image normally (Derrick, Moffit, et al., 2010).

The propensity for bomb-making subjects to focus their eye gaze behavior on the altered portion of the image is similar to the involuntary responses common in concealed information tests (CIT). The purpose of the CIT is to assess the recognition of critical information while a subject is being examined (D. T. Lykken, 1959). The practical application of this theory is frequently exhibited in polygraph examinations. While the lion’s share of CIT research has evaluated this phenomenon in a laboratory setting, a recent study evaluated it in a field setting. Researchers conducting this study found that the field results were similar to those conducted in laboratories, thus strengthening the legitimacy of CITs (Verschuere, Meijer, & Clercq, 2010). Further research examined the cause for subjects to exhibit psychophysiological cues indicative of recognition (Lykken, 1974). Lykken developed Orienting Response (OR) Theory to account for responses to external stimuli that are novel. This theory directly relates to the responses exhibited by subjects in the Derrick et al. (2010) study.

In this study, bomb-making participants demonstrated OR as they viewed an image of an identical bomb in an altered state. The manifestation of this orienting response can be identified as their psychophysiological response to gaze at the altered component due to it being novel. Subjects in the control group had no prior exposure to the altered bomb, and therefor did not exhibit cues indicating an orienting response. The differentiating factor in this study is that the test group was familiar with the bomb while the control group was not. The concept of manufacturing an orienting response indicative of familiarity will be incorporated into a research model.

Research Question Familiarity Recognition in Automated Screening Environments Jeffrey G. Proudfoot

Derrick et al. (2010) solidified the salience of utilizing eye-tracking technology as a medium to administer a GKT; however, the domain of images used in this study was limited to familiar objects, modified objects, and unfamiliar objects. Because the application of eye-tracking technology as a GKT is an emerging research stream, research has yet to be conducted regarding its use in evaluating subjects viewing images of faces.

An additional element of this research will address the effects of applying this technology in a legitimate screening environment. Exposure to an automated screening tool is likely to generate detection apprehension in both guilty and innocent individuals. In other words, even innocent parties are likely to experience slight modifications in behavior during the screening process. This modification in behavior can be attributed to the anxiety resulting from the fear of misclassification (Ekman, 1985). Additionally, guilty parties are likely to consciously alter their eye gaze behavior in an effort to mitigate the eye tracker’s ability to make a correct assessment. Therefore, further research is required to determine if detection apprehension has a negative effect on the proper classification of both guilty and innocent parties.

Figure 1 serves as a visual representation of the variables hypothesized to have an effect on classifying eye gaze behavior as an indicator of familiarity.

Figure 1 – Familiarity Assessment Model

This model will be used as a reference point for the design and implementation of an experiment to address the following research questions:

RQ1: Can eye-tracking technology be used as a GKT to detect familiarity based on eye gaze behavior? RQ2: What impact does detection apprehension have on successful classification?

Methodology and Research Approach A two-treatment between-group experimental design will be the basis for conducting this experiment. It will assess the reliability of eye-tracking technology to evaluate indicators of familiarity, and the variables that positively and negatively impact proper classification. Fifteen volunteers will be enlisted from introductory MIS classes at a large southwestern university to Familiarity Recognition in Automated Screening Environments Jeffrey G. Proudfoot conduct a pilot study ensuring that experimental procedures are properly designed. The first treatment is the control group. Both treatments will be evaluated to determine the effects of orienting responses and familiarity. Participants in the pilot study will be randomly assigned.

Subjects will view a compilation of images while being monitored by an eye-tracking device. A variety of facial images will be displayed to test the various components of the experimental design. Each image will be displayed for twelve seconds. A description of the methodology used to test each component of the experiment is as follows:

 Orienting Response – An image with only one face will be displayed. This image will be altered (an added mole, tattoo, freckles, etc.) yet appear legitimate. Subjects familiar with the individual are likely to focus on the unfamiliar facial characteristics displayed in the photo as an orienting response (Derrick, Moffit, et al., 2010).  Apprehension – To create apprehension, subjects in the guilty group will be given an IED. They will be instructed to pass through a security checkpoint and deliver the IED to a specific individual. They will be shown an image of this individual and instructed to memorize their face as this will be their only way of identifying the recipient. They will be subjected to the same compilation of images during screening at the security checkpoint. One of these images will be of the intended recipient.

The eye tracker will measure the x and y coordinates of each subject’s gaze in relation to the displayed images. I will then average the aggregate x-coordinate and y-coordinate values of each subject’s eye gaze for every image. This will allow me to determine the focal point of interest for each picture and glean information regarding orienting response and detection apprehension.

Expected Results I expect the results of this experiment to be as follows for each of the previously defined conditions:

 Orienting Response – For familiar individuals, I expect that subjects will focus their eye gaze on the altered element of that person’s face.  Apprehension – I expect that subjects delivering the IED will demonstrate erratic eye gaze behavior during exposure to an image of the recipient.

Conclusion Upon approval of this proposal, pilot studies will be engineered to begin researching this phenomenon. The results of this study will be critical in ascertaining the feasibility of using eye- tracking technology to enhance national security efforts in the United States. Familiarity Recognition in Automated Screening Environments Jeffrey G. Proudfoot

References

Althoff, R. R., & Cohen, N. J. (1999). Eye-movement-based memory effect: A reprocessing effect in face perception. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25(4), 997- 1010. doi: 10.1037/0278-7393.25.4.997 Buller, D. B., & Burgoon, J. K. (1996). Interpersonal Deception Theory. Communication Theory, 6(3), 203- 242. doi: 10.1111/j.1468-2885.1996.tb00127.x Derrick, D. C., Elkins, A. C., Burgoon, J. K., Nunamaker, J. F., & Zeng, D. D. (2010). Border Security Credibility Assessments via Heterogeneous Sensor Fusion. Intelligent Systems, IEEE, 25(3), 41-49. Derrick, D. C., Moffit, K., & Nunamaker, J. F. (2010). Eye Gaze Behavior as a Guilty Knowledge Test: Initial Exploration for Use in Automated, Kiosk-based Screening. Paper presented at the HICSS, Hawaii. Ellis, H. D., Shepherd, J. W., & Davies, G. M. (1979). Identification of familiar and unfamiliar faces from internal and external features., 8, 431-439. Lykken. (1974). Psychology and the lie detector industry. American Psychologist, 29(10), 725-739. doi: DOI: 10.1037/h0037441 Lykken, D. T. (1959). The GSR in the detection of guilt. Journal of Applied Psychology, 43, 385-388. Ryan Jr, A. H., Pavlidis, I., Rohrbaugh, J. W., Marchak, F., & Kozel, F. A. (2003). Credibility assessments: operational issues and technology impact for law enforcement applications, Orlando, FL, USA. Verschuere, B., Meijer, E., & Clercq, A. D. (2010). Concealed information under stress: A test of the orienting theory in real-life police interrogations. Legal and Criminological Psychology.