AN “ACTIVE VISION” COMPUTATIONAL MODEL OF VISUAL SEARCH FOR HUMAN-COMPUTER INTERACTION by TIMOTHY E. HALVERSON A DISSERTATION Presented to the Department of Computer and Information Science and the Graduate School of the University of Oregon in partial fulfillment of the requirements for the degree of Doctor of Philosophy December 2008 ii “An ‘Active Vision’ Computational Model of Visual Search for Human-Computer Interaction,” a dissertation prepared by Timothy E. Halverson in partial fulfillment of the requirements for the Doctor of Philosophy degree in the Department of Computer and Information Science. This dissertation has been approved and accepted by: ____________________________________________________________ Anthony J. Hornof, Chair of the Examining Committee ________________________________________ Date Committee in Charge: Dr. Anthony J. Hornof, Chair Dr. Arthur Farley Dr. Stuart Faulk Dr. Edward Vogel Dr. Michal Young Accepted by: ____________________________________________________________ Dean of the Graduate School iii © 2008 Timothy E. Halverson iv An Abstract of the Dissertation of Timothy E. Halverson for the degree of Doctor of Philosophy in the Department of Computer and Information Science to be taken December 2008 Title: AN “ACTIVE VISION” COMPUTATIONAL MODEL OF VISUAL SEARCH FOR HUMAN-COMPUTER INTERACTION Approved: _______________________________________________ Anthony J. Hornof Visual search is an important part of human-computer interaction (HCI). The visual search processes that people use have a substantial effect on the time expended and likelihood of finding the information they seek. This dissertation investigates visual search through experiments and computational cognitive modeling. Computational cognitive modeling is a powerful methodology that uses computer simulation to capture, assert, record, and replay plausible sets of interactions among the many human processes at work during visual search. This dissertation aims to provide a cognitive model of visual search that can be utilized by predictive interface analysis tools and to do so in a manner consistent with a comprehensive theory of human visual processing, namely active vision. The model accounts for the four questions of active vision, the answers to which are important to both practitioners and researchers in HCI: What can be perceived v in a fixation? When do the eyes move? Where do the eyes move? What information is integrated between eye movements? This dissertation presents a principled progression of the development of a computational model of active vision. Three experiments were conducted that investigate the effects of visual layout properties: density, color, and word meaning. The experimental results provide a better understanding of how these factors affect human- computer visual interaction. Three sets of data, two from the experiments reported here, were accurately modeled in the EPIC (Executive Process-Interactive Control) cognitive architecture. This work extends the practice of computational cognitive modeling by (a) informing the process of developing computational models through the use of eye movement data and (b) providing the first detailed instantiation of the theory of active vision in a computational framework. This instantiation allows us to better understand (a) the effects and interactions of visual search processes and (b) how these visual search processes can be used computationally to predict people’s visual search behavior. This research ultimately benefits HCI by giving researchers and practitioners a better understanding of how users visually interact with computers and provides a foundation for tools to predict that interaction. vi CURRICULUM VITAE NAME OF AUTHOR: Tim Halverson PLACE OF BIRTH: Iowa City, IA, USA DATE OF BIRTH: March 31st, 1971 GRADUATE AND UNDERGRADUATE SCHOOLS ATTENDED: University of Oregon, Eugene, OR, USA Lane Community College, Eugene, OR, USA Dickinson State University, Dickinson, ND, USA DEGREES AWARDED: Doctor of Philosophy, Computer and Information Science, 2008, University of Oregon Master of Science, Computer and Information Science, 2003, University of Oregon Bachelor of Science, Computer and Information Science, 2001, University of Oregon AREAS OF SPECIAL INTEREST: Human-Computer Interaction Cognitive Science PROFESSIONAL EXPERIENCE: Cognitive Modeling and Eye Tracking Lab Manager, Dr. Anthony Hornof, September 2006-Current vii Graduate Research Assistant, University of Oregon, 2001-Current Technical Manager and Computer Repair Technician, Galaxy Hardware Publishing 1993-2000 Automatic Tracking Radar Specialist, United States Air Force, 1989-1993 GRANTS, AWARDS AND HONORS: Henry V. Howe Scholarship, University of Oregon, 2007-2008 PUBLICATIONS: Hornof, A. J., Halverson, T., Isaacson, A., & Brown, Erik (2008). Transforming Object Locations on a 2D Visual Display Into Cued Locations in 3D Auditory Space. Proceedings of the Human Factors and Ergonomics Society 52nd Annual Meeting, New York, 1170-1174. Halverson, T. & Hornof, A. J. (2008). The effects of semantic grouping on visual search. Proceedings of ACM CHI 2008: Conference on Human Factors in Computing Systems, Florence, Italy, 3471-3476. Hornof, A. J., Rogers, T., & Halverson, T. (2007). EyeMusic: Performing Live Music and Multimedia Compositions with Eye Movements. NIME 2007: Conference on New Interfaces for Musical Expression, New York, 299-300. Halverson, T., & Hornof, A. J. (2007). A minimal model for predicting visual search in Human-Computer Interaction. Proceedings of ACM CHI 2007: Conference on Human Factors in Computing Systems, San Jose, CA, 431-434. Halverson, T., & Hornof, A. J. (2006). Towards a Flexible, Reusable Model for Predicting Eye Movements During Visual Search of Text. Proceedings of the Annual Meeting of the Cognitive Science Society, Vancouver, BC, Canada, 1428-1433. Brock, D., McClimens, B., Hornof, A. J., & Halverson, T. (2006). Cognitive Models of the Effect of Audio Cueing on Attentional Shifts in a Complex Multimodal Dual-Display Dual-Task. Proceedings of the 28th Annual Meeting of the Cognitive Science Society, Vancouver, BC, Canada, 1044-1049. viii Halverson, T. (2006). Integrating Models of Human-Computer Visual Interaction. ACM CHI 2006: Conference on Human Factors in Computing Systems, Montréal, Québec, Canada, 1747-1750. Halverson, T., & Hornof, A. J. (2004). Explaining eye movements in the visual search of varying density layouts. Proceedings of the Sixth International Conference on Cognitive Modeling, Pittsburgh, PA, 124-129. Halverson, T., & Hornof, A. J. (2004). Local density guides visual search: Sparse groups are first and faster. Proceedings of the Human Factors and Ergonomics Society 48th Annual Meeting, New Orleans, LA, 1860-1864. Halverson, T., & Hornof, A. J. (2004). Strategy shifts in mixed-density search. Proceedings of the 26th Annual Meeting of the Cognitive Science Society, Chicago, IL, 529-534. Halverson, T., & Hornof, A. J. (2004). Link colors guide a search. ACM CHI 2004: Conference on Human Factors in Computing Systems, Vienna, Austria, 1367-1370. Hornof, A. J., & Halverson, T. (2003). Cognitive strategies and eye movements for searching hierarchical computer displays. Proceedings of ACM CHI 2003: Conference on Human Factors in Computing Systems, Ft. Lauderdale, FL, 249-156. Hornof, A. J., & Halverson, T. (2003). Predicting cognitive strategies and eye movements in hierarchical visual search. Proceedings of the Fifth International Conference on Cognitive Modeling, Bamberg, Germany, 261-262. Hornof, A. J., & Halverson, T. (2002). Cleaning up systematic error in eye tracking data by using required fixation locations. Behavior Research Methods, Instruments, and Computers, 34(4), 592-604. ix ACKNOWLEDGEMENTS This research was supported by the National Science Foundation through grant IIS-0308244 and through the Office of Naval Research through grants N00014-06-10054 and N00014-02-10440 to the University of Oregon with Anthony Hornof as the principal investigator. Any opinions, findings, conclusions, or recommendations expressed in this report are those of the participants and do not necessarily represent the official views, opinions, or policy of the National Science Foundation or the Office of Naval Research. I would like to thank my advisor, Anthony Hornof, for the many types of support he provided: intellectual, moral, and so much more. My thanks also go out to my academic “grandfather”, David Kieras. Thank you for your discussions about (and co-creation of) EPIC. I would also like to the thank the research communities that sparked and nurtured my interest: CHI, ICCM, and CogSci. And thanks to my peers that helped in so many ways, specifically: Duncan Brumby, Anna Cavendar, Ishwinder Kaur, Erik Brown, and Andy Isaacson. x To Minerva and all of my family for their support throughout the process: Beth, my parents, my grandparents, Thom, Andy, Melanie, Lucy, Kristen, Jake, and so many others. xi TABLE OF CONTENTS Chapter Page I. INTRODUCTION․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ 1 II. LITERATURE REVIEW․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ 10 2.1 Previous Models of Visual Search in HCI․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ 10 2.2 Active Vision Theory․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ 12 2.2.1 What Can Be Perceived?․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ 13 2.2.2 When Do the Eyes Move?․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․ 15 2.2.3 Where Do the Eyes Move?․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․