MULTIMODAL, ASYNCHRONOUSLY SHARED SLIDE SHOWS AS A DESIGN STRATEGY FOR ENGINEERING DISTRIBUTED WORK IN THE NATIONAL AIRSPACE SYSTEM DISSERTATION Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of the Ohio State University By Roger James Chapman, B.S, M.S. * * * * * The Ohio State University 2002 Dissertation Committee: Approved by: Professor Philip J. Smith Professor Charles E. Billings ___________________________________ Adviser Professor Nadine Sarter Department of Industrial & Systems Engineering Graduate Program Copyright by Roger James Chapman 2002 ABSTRACT This research focused on the design and evaluation of a multimodal asynchronous communications tool that uses an annotated slide show structure for messages, to support collaborative analysis of post-operations in the National Airspace System (NAS). Voice-based asynchronous annotations communicated between writers working collaboratively have been found to support more extensive discussion of problems in draft documents and to be preferred over text based discussion for some types of feedback. People point naturally when discussing the content of a shared image, and voice synchronized with pointing in asynchronous annotation systems has been found to be more efficient in scheduling tasks, than voice-only, or text only communication. Synchronized voice and pointing has also been shown to focus attention and improve retention of information in multimedia presentation systems. This research involved investigating how different communication modes interact with problem solving and the discussion of distributed information, with a view to improving collaborative post operations analysis within and between the Federal Aviation Administration (FAA) and the major airlines in the United States. More specifically how synchronized voice and pointing annotation over asynchronously shared slide shows composed of post operations graphical and tabular data differs in its effect compared to text based annotation, as collections of flights ranked low by standard performance metrics are discussed by FAA and airline representatives. ii A part-task simulation study had 36 AOC dispatchers, chief dispatchers or ATC coordinators from Northwest Airlines communicate with 36 ARTCC (Air Route Traffic Control Center) traffic managers at eight different Centers about performance issues for flights between 9 different city pairs. Each Northwest Airlines participant communicated with one FAA participant, in a one-to-one pairing. A separate slide show consisting of screen captures showing post operations data was created for each city pair. For each slide show, two Northwest Airlines participants were asked to annotate the slides using text, pen marks or arrows, and two were asked to annotate the slides using synchronized voice and pointing, pen marks and arrows. For each slide show, four traffic managers were asked to respond to the annotated message using the same annotation tools the sender used. The results from this research show that: 1. The communications mode failed to show an effect on the number and type of ideas generated for improving performance. 2. The most common strategy proposed was a change of route. 3. Almost all the dispatchers’ ideas were responded to by traffic managers in both modes: 27 out of 27 for text mode participants and 27 out of 29 for voice and pointing mode participants. 4. The dispatchers’ combined problem solving and message creation time was shorter (ANOVA, α < 0.05) when working in the voice and pointing mode than the text based mode, as was the dispatcher-traffic manager pairs’ total combined problem solving and message creation time (ANOVA, α < 0.01). The average task time was 18 minutes 1 second for the voice and pointing mode dispatchers and 28 minutes for the text mode dispatchers (a difference of 9 minutes 59 seconds or 55%). The average dispatcher-traffic manager pair task time was 35 iii minutes and 41 seconds for the voice and pointing mode and 53 minutes and 22 seconds for the text mode. This is a difference of 17 minutes 41 seconds or 50%. 5. In both communication modes the dispatchers normally did not comment on the time of day, time of year, or the number of flights involved on the first slide they made a comment, even though that information was always in the image. The city pair involved was also only mentioned by 5 out of 18 dispatchers working in the text mode and 11 out of 18 dispatchers working in the voice and pointing mode. There was a difference between the two modes in this regard with more dispatchers mentioning each of these categories in the voice and pointing mode (although only the difference between the number mentioning the city pair and time of year were statistically significant at α < 0.05). This is significant because this information contributes towards diagnosing the cause of the flights per performance, and some participants indicated that by stating it was difficult to solve the problem without some of this information. 6. 18/18 (100%) of the dispatchers in the voice and pointing mode made deictic gestures during the creation of their messages. 17/18 (94%) of the traffic managers responding in this mode also made deictic gestures. 7. 15/18 (83%) of the dispatchers and 9/18 (50%) of the traffic managers working in the voice and pointing mode still created static annotations to the objects or locations they referenced with deictic gestures. 8. 9/15 (60%) of the dispatchers and 9/9 (100%) of the traffic managers working in the voice and pointing mode created static annotations while speaking, as opposed to before or after speaking. 9. Dispatchers and traffic managers considered the system they used to be usable and valuable, but dispatchers using the text mode rated the usability and utility iv of the system higher on average than dispatchers using the voice and pointing mode, whereas for traffic managers the reverse was true. 10. The slide image provided a context for comments, but the slide show structure used created a loss of temporal context and comments regarding multiple slides and slides other than the one the comment was placed on meant a slide could not be treated as a disjoint entity. These problems were addressed with a redesign idea to support these contexts. 11. The comments and annotations made provide a vocabulary of syntax and semantics for a more task specific version of the system. Such a system would be tailored for practitioners to work with in this domain, but should be implemented through configuration options to maintain the integrity of the system as a domain free system. v ACKNOWLEDGMENTS I would like to express my thanks to my adviser, Dr. Philip J. Smith, for his guidance throughout my graduate work at Ohio State. I would also like to thank my other committee members, Dr. Charles Billings and Dr. Nadine Sarter, and also Dr. David Woods who has always been willing to share his insight and knowledge of human factors with me. Northwest Airlines and the Federal Aviation Administration made this research possible with their participation. In particular, I would like to thank Charles Bailey, Traffic Management Officer for the New York ARTCC, for his willingness to discuss the National Airspace System with me and for his encouragement for my work. Finally, I would like to acknowledge the support of my family and friends for their faith, and patience. vi VITA March 22, 1962 ................................................... Born, Baldock, England 1986 ..................................................................... B. S., Computer Science summa cum laude Eastern Kentucky University Richmond, Kentucky 1987 ......……......…............................................... M.S., Computer Systems Engineering University of Edinburgh Edinburgh, Scotland 1988 - 1990 ........................................................... Instructor of Computer Science Transylvania University Lexington, Kentucky 1990 - 1995 ........................................................... Associate Professor of Computer Science Rocky Mountain College Billings, Montana 1993 …………………………………………………. Faculty Fellowship Los Alamos National Laboratory Los Alamos, New Mexico 1994, 1995 ………………………………………... Faculty Fellowship Oak Ridge National Laboratory Oak Ridge, Tennessee 1996 - 2000 ........................................................... Graduate Research Associate Industrial & Systems Engineering The Ohio State University 2000 - 2001 ........................................................... Assistant Professor of Computer Science University of Hawaii, Hilo vii PUBLICATIONS 1. Chapman, R. J., Smith, P. J., Billings, C. E., McCoy C. E., and Obradovich J. H. (2001). Collaborative Constraint Propagation as a Planning Strategy in the National Airspace System. Proceedings of the 2001 Annual Meeting of the IEEE Society on Systems, Man and Cybernetics. Tucson, AZ, 3484-3490. 2. Smith, P. J., Billings, C., Chapman, R. J., Obradovich J. H., McCoy, C. E. (2001). Information Management to Support Distributed Decision-Making in the National Airspace System. Proceedings of the Human Factors and Ergonomics Society 45th Annual Meeting, Minneapolis, MN, 54- 57. 3. Chapman, R. J., Smith, P.J., Klopfenstein, M., Jezerinac, J., Obradovich, J., McCoy, C.E., and Orasanu, J. (2000). Supporting collaboration in the National Airspace System with multimodal, asynchronous communications. Proceedings of the 2000 Conference on Human Performance, Situation Awareness and Automation: User-Centered