Old Dominion University ODU Digital Commons Psychology Theses & Dissertations Psychology Spring 2020 Effects of Transparency and Haze on Trust and Performance During a Full Motion Video Analysis Task Sarah C. Leibner Old Dominion University, [email protected] Follow this and additional works at: https://digitalcommons.odu.edu/psychology_etds Part of the Applied Behavior Analysis Commons, Industrial and Organizational Psychology Commons, and the Industrial Engineering Commons Recommended Citation Leibner, Sarah C.. "Effects of Transparency and Haze on Trust and Performance During a Full Motion Video Analysis Task" (2020). Master of Science (MS), Thesis, Psychology, Old Dominion University, DOI: 10.25777/7sba-p044 https://digitalcommons.odu.edu/psychology_etds/352 This Thesis is brought to you for free and open access by the Psychology at ODU Digital Commons. It has been accepted for inclusion in Psychology Theses & Dissertations by an authorized administrator of ODU Digital Commons. For more information, please contact [email protected]. EFFECTS OF TRANSPARENCY AND HAZE ON TRUST AND PERFORMANCE DURING A FULL MOTION VIDEO ANALYSIS TASK by Sarah C. Leibner B.S. Psychology, May 2017, Old Dominion University A Thesis Submitted to the Faculty of Old Dominion University in Partial Fulfillment of the Requirements for the Degree of MASTER OF SCIENCE PSYCHOLOGY OLD DOMINION UNIVERSITY May 2020 Approved by: James P. Bliss (Director) Mark Scerbo (Member) Konstantin Cigularov (Member) ABSTRACT EFFECTS OF TRANSPARENCY AND HAZE ON TRUST AND PERFORMANCE DURING A FULL MOTION VIDEO ANALYSIS TASK Sarah C. Leibner Old Dominion University, 2020 Director: Dr. James P. Bliss Automation is pervasive across all task domains, but its adoption poses unique challenges within the intelligence, surveillance, and reconnaissance (ISR) domain. When users are unable to establish optimal levels of trust in the automation, task accuracy, speed, and automation usage suffer (Chung & Wark, 2016). Degraded visual environments (DVEs) are a particular problem in ISR; however, their specific effects on trust and task performance are still open to investigation (Narayanaswami, Gandhe, & Mehra, 2010). Research suggests that transparency of automation is necessary for users to accurately calibrate trust levels (Lyons et al., 2017). Chen et al. (2014) proposed three levels of transparency, with varying amounts of information provided to the user at each level. Transparency may reduce the negative effects of DVEs on trust and performance, but the optimal level of transparency has not been established (Nicolau & McKnight, 2006). The current study investigated the effects of varying levels of transparency and image haze on task performance and user trust in automation. A new model predicting trust from attention was also proposed. A secondary aim was to investigate the usefulness of task shedding and accuracy as measures of trust. A group of 48 undergraduates attempted to identify explosive emplacement activity within a series of full motion video (FMV) clips, aided by an automated analyst. The experimental setup was intended to replicate Level 5 automation (Sheridan & Verplank, 1978). Reliability of the automated analyst was primed to participants as 78% historical accuracy. For each clip, participants could shed their decision to an automated analyst. Higher transparency of automation predicted significantly higher accuracy, whereas hazy visual stimuli predicted significantly lower accuracy and 2.24 times greater likelihood of task shedding. Trust significantly predicted accuracy, but not task shedding. Participants were fastest in the medium transparency condition. The proposed model of attention was not supported; however, participants’ scanning behavior differed significantly between hazy and zero haze conditions. The study was limited by task complexity due to efforts to replicate real-world conditions, leading to confusion on the part of some participants. Results suggested that transparency of automation is critical, and should include purpose, process, performance, reason, algorithm, and environment information. Additional research is needed to explain task shedding behavior and to investigate the relationship between degrade visual environments, transparency of automation, and trust in automation. iv Copyright 2019 by Sarah C Leibner, All Rights Reserved. v ACKNOWLEDGEMENTS I would like to acknowledge my advisor and committee chair, Dr. James Bliss, for his patient guidance throughout this process. I would also like to acknowledge Dr. Anne Michalek for her kind assistance and extended loan of eye tracking equipment during data collection. vi TABLE OF CONTENTS Page LIST OF TABLES ....................................................................................................................... viii LIST OF FIGURES ....................................................................................................................... ix Chapter I. INTRODUCTION ...................................................................................................................10 AUTOMATED FMV ANALYSIS .................................................................................…11 LEVELS OF AUTOMATION ............................................................................................17 RELIABILITY ....................................................................................................................18 HUMAN-AUTOMATION TRUST ....................................................................................18 SITUATION AWARENESS AND DECISION MAKING ...............................................20 DEGRADED VISUAL ENVIRONMENTS .......................................................................21 TRANSPARENCY OF AUTOMATION ...........................................................................23 CONVERGENT MEASURES............................................................................................25 CURRENT STUDY ............................................................................................................27 HYPOTHESES ..............................................................................................................28 II. METHOD ................................................................................................................................31 DESIGN ..............................................................................................................................31 INDEPENDENT VARIABLES ....................................................................................31 DEPENDENT VARIABLES ........................................................................................34 PARTICIPANTS .................................................................................................................37 MATERIALS ......................................................................................................................38 MEASURES ........................................................................................................................40 PROCEDURE .....................................................................................................................41 III. RESULTS ...............................................................................................................................44 DATA CLEANING ............................................................................................................44 DATA CODING .................................................................................................................45 ACCURACY .......................................................................................................................46 DECISION SPEED .............................................................................................................47 TASK SHEDDING .............................................................................................................50 ATTENTION ......................................................................................................................52 TRUST ................................................................................................................................55 IV. DISCUSSION .........................................................................................................................60 HAZE AND TRANSPARENCY ........................................................................................60 MEASURING TRUST ........................................................................................................61 ATTENTION ......................................................................................................................64 vii Chapter Page IV. DISCUSSION .........................................................................................................................60 IMPLICATIONS .................................................................................................................65 LIMITATIONS ...................................................................................................................66 CONCLUSION ...................................................................................................................67 REFERENCES ..............................................................................................................................69 APPENDICES ...............................................................................................................................78