UNIVERSITY OF CALIFORNIA, SAN DIEGO Studying Episodic Access to Personal Digital Activity: Activity Trails Prototype A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Cognitive Science by Gaston R. Cangiano Committee in charge: Professor James D. Hollan, Chair Professor Barry Brown Professor William Griswold Professor Edwin L. Hutchins Professor David Kirsh 2011 Copyright Gaston R. Cangiano, 2011 All rights reserved. The dissertation of Gaston R. Cangiano is approved, and it is acceptable in quality and form for publication on microfilm and electronically: Chair University of California, San Diego 2011 iii DEDICATION To two, the loneliest number since the number one. iv TABLE OF CONTENTS SignaturePage .................................. iii Dedication..................................... iv TableofContents................................. v ListofFigures .................................. vii Acknowledgements ................................ viii AbstractoftheDissertation . .. .. ix 1 Introduction.................................. 1 1.1 WorkScenario.............................. 3 1.1.1 CurrentPractices........................ 3 1.1.2 Activity Trails . 5 1.2 ThesisStatement ............................ 8 1.3 ResearchApproachandContributions . 8 1.4 ThesisOrganization........................... 10 2 Background .................................. 11 2.1 Analytical Models of Behavior in HCI . 11 2.1.1 Multi-tasking and Interruptions . 12 2.2 RelatedWorkinHCI.......................... 17 2.2.1 TaskSupportSystems . 17 2.3 HumanPerceptionandMemory . 20 2.3.1 Episodic Memory and Context Reinstatement . 20 2.3.2 Episodic Memory and Activity Logging . 22 3 FieldWork .................................. 26 3.1 FieldSite ................................ 26 3.2 DataCollectionMethodology . 28 3.3 FieldData................................ 31 3.3.1 Timeline Representation of Context . 31 3.3.2 HowWorkersBuildContext . 32 3.3.3 ContextReinstatement. 35 4 PrototypeImplementation . 40 4.1 ActivityLogger ............................. 41 4.1.1 DataProcessing......................... 44 TimeSeriesSmoothing. 46 Summarization ......................... 47 v Textual Content and Indexing . 52 4.2 GraphicalUserInterface . 57 5 CaseStudies.................................. 61 SampleCasesonResearch . 63 Sample Cases on On-line Search (Non-Scientific Research) . 66 6 ClosingDiscussion .............................. 69 A UMLDiagramsforActivityTrails. 71 Bibliography ................................... 74 vi LIST OF FIGURES Figure 1.1: Graphical User Interface for Activity Trails. ... 7 Figure3.1: InterviewSetup. .. .. 29 Figure 3.2: Activity Summary Used During Interviews. 30 Figure 3.3: Timeline Representation of Case Histories. 33 Figure 4.1: Logger for Activity Trails. 41 Figure4.2: XMLFrameOutput . 43 Figure4.3: XMLWindowOutput . 43 Figure4.4: XMLInputDevicesOutput . 44 Figure 4.5: Pseudo-code for Computing Window Overlap . 47 Figure 4.6: Plot of Smoothed Time-series. 48 Figure4.7: SAXDataRepresentation.. 50 Figure4.8: SampleOutputfromWordSpy . 52 Figure4.9: VisualSummary DataRepresentation. 53 Figure 4.10: Sample Output Text File for Indexing . 54 Figure 4.11: Sample Dictionary Text File . 56 Figure 4.12: Sample of Filtering . 56 Figure 4.13: Graphical User Interface for Activity Trails. .... 58 Figure4.14:SearchResults. 59 Figure5.1: Userlog. .............................. 62 Figure5.2: Notesandlabnotebook.. 64 Figure A.1: Activity Logger Class Diagram. 71 Figure A.2: Activity Logger Sequence Diagram. 72 Figure A.3: Graphical User Interface Class Diagram. .. 72 Figure A.4: Graphical User Interface Sequence Diagram. .... 73 vii ACKNOWLEDGEMENTS Thanks to whoever deserves credit for Blacks Beach, Porters Pub, and every coffee shop in San Diego. Thanks also to hottubs. viii ABSTRACT OF THE DISSERTATION Studying Episodic Access to Personal Digital Activity: Activity Trails Prototype by Gaston R. Cangiano Doctor of Philosophy in Cognitive Science University of California, San Diego, 2011 Professor James D. Hollan, Chair It was just a generation ago that computers entered the workplace. Back then, they only represented the work we did, nothing else. But today, some sort of computing device is involved in how we play, how we communicate, how we get our news and of course, how we work. What this means is that today almost all aspects of our lives are represented in some digital form. The rapid pace of change in technology and the dramatic shift in the use of computers has a cost associated with it. The legacy design of early computer systems is still prevalent in modern devices and goes unnoticed because of our familiarity with it. The desktop metaphor with its file and folder system, and the application paradigm with its document-centric view of information, both carry the legacy of a design ix that has far surpassed what it was originally designed to do. Digital representations should mediate what we do in the physical world, and since we do much more now than just work through computers, we need new representations that leverage our cognitive abilities in everyday life; in particular, present day computing devices do not facilitate the use of a powerful skill we use in our personal experiences, known as episodic memory. Episodic memory is how we remember our lives through stories. The human perceptual system samples the world continuously in order for the brain to store information, organize it and later recall it efficiently. At least this is the classic view of memory. However, people also leave a physical trace behind each and every one of their actions simply as the byproduct of their interaction with the environment. Because memory is finely tuned to reconstruct the past, our perceptual skills help us make meaning out of these traces. Time, proximity and familiar surroundings provide cues that naturally trigger our recollection of the past. Episodic memory is a human skill that taps into these cues by encoding the context surrounding events therefore allowing us to re-experience the past by recalling specific instances and the context in which they were experienced. Computers, in contrast to humans, only record the consequences of our actions and in doing so, they reduce the type and quantity of the memory cues available. Things have indeed changed dramatically in the last generation. Together with the avalanche of new digital gadgets and the general overwhelming presence of computers in our lives come new opportunities for research. One new and exciting ground for research is the use of activity recordings. The main research question of my work is understanding how to design an interface using these recordings to facilitate the use of episodic memory. There is a growing interest in the field of Human-Computer Interaction in research of life-logging technologies to assist with memory and reflection. But we are only at the very onset of understanding the impact of these technologies, and more importantly, how they might fit into the fabric of our daily lives. The work I present here was motivated by a year-long ethnographic study conducted at a law office. In this study, I used desktop activity recordings as a novel x methodology to learn about the nature and details of work. I learned that what is usually considered multitasking behavior in the literature, is in fact the norm in this setting. Multitasking here is not “crunch mode” type of behavior, but is a self- selected and all together different kind of work style. This style is engendered by both the nature of the legal work and the new digital tools available, in particular communication tools such as instant messaging and email. These tools have had an impact in how paralegals and attorneys interact with clients and with one another. My ethnographic data reveals that with the increasing frequencies and flexibility of the daily interactions comes an increased fragmentation of the context of each work thread. The lack of episodic support in these tools creates a heavy load for workers. Paralegals and attorneys have to put effort to bring together the history of a case from the many separate pieces of the past (email, instant messages, database entries, and so on). In other words, workers have to build context for a case before communicating with clients and this context consists of putting together a timeline representation about the history for a case, containing a chronology of events with the client and a view of any upcoming deadlines or pending issues for the case. My argument is that the tools available presently do not support this context-building process, so in addition to supporting for multitasking and interruptions we need to design support for this process. The second portion of the work in this thesis brings my findings to bear directly on a software design problem for Human-Computer Interaction. It de- scribes the design and implementation of a software prototype tool called Activity Trails with the goal of supporting episodic memory. The thesis ends with a study conducted with researchers at UCSD evaluating the benefits of episodic access for everyday activity through detailed case studies of usage. xi 1 Introduction The interface design for most digital devices — and in particular the preva- lent application/desktop metaphor — supports mainly the use of semantic memory. Every time we access our lives through these interfaces, we are confronted with a design that relies on the use of semantic categories and labels to organize our infor-