Quantifying episodic memories from real-world experience Alyssa Marconi Senior Thesis Department of Biology Emmanuel College & Kreiman Laboratory Boston Children’s Hospital, Harvard Medical School Boston, MA Principal Investigator: Dr. Gabriel Kreiman, Ph.D. Advisor: Dr. Jason Kuehner, Ph.D. April 27th, 2017 Abstract Quantifying episodic memories from real-world experience Alyssa Marconi1 * Gabriel Kreiman2 1Emmanuel College, Boston MA 02115 2Children’s Hospital, Harvard Medical School, Boston MA 02115, USA Emmanuel College, Boston MA Episodic memories form the fundamental fabric of who we are. These memories are full of content and malleable due to the number of inputs impinging upon our senses in our everyday lives. Despite a plethora of laboratory studies about memories for individual words, names, or pictures, little is known about what governs the formation of episodic memories in real life. In this work, we studied real life episodic memories by recording ground truth video and eye movement information during a one hour segment in the life of our subjects and subsequently evaluating their ability to recognize those events. Ten subjects completed an encoding task which took place in two versions: an outdoor and indoor route. Memory recognition tasks were completed at 24 hours up to 3 months post-encoding and performance was consistent across subjects in both versions with an accuracy of up to 62 +/- 7.0%. Previous work used a similar methodology combined with machine learning approaches to predict episodic memory formation in movies, but those results remain inconclusive since movies are artificial stimuli and do not accurately represent real life events. Our study found that real life episodic memories are less memorable than movies likely due to the large amounts of information we filter out in non- laboratory settings. The results allow us to understand episodic memory formation in real life to elucidate the subjective filtering which leads to how we learn and consolidate long-term memories. To evaluate the content properties that influence long-term memory formation, annotations were designed to see what aspects enhance memorability in the real world. These allow us to report what content properties correlate with higher performance in memory recognition. To the best of our knowledge, this work constitutes the first quantitative approach to directly measure memory formation with ground truth data in real life scenarios. Marconi 2 Table of Contents Title Page 1 Abstract 2 Table of Contents 3 List of Figures and tables 4 Introduction 5-18 Episodic Memory Formation and Retrieval from Real-World Experience 5-6 Alzheimer’s Disease and Episodic Memory 7-8 Learning disabilities and Episodic Memory 8 The stages of the human memory process 8-9 Learning and Memory 9-10 Patient H.M. and the evidence for different Memory Systems in humans 10-13 In-lab Episodic Memory Studies from previous literature 14 Memories in Real-Life 14-15 Quantitatively characterizing efficacy of long-term memorability 15-16 Spatiotemporal memory formation in movies vs. real life experience 17-18 Materials and Methods 19-28 Results 29-43 Discussion 44-52 Conclusions 53 Acknowledgements 54 References 55-56 Marconi 3 List of Figures [1] Real-world eye-tracking apparatus for encoding (Part I) and footage for the memory recognition task (Part II) 6 [2] The three stages of the human memory process 9 [3] Overview of the different memory systems in humans 12 [4] Performance on the memory recognition test was consistent at both time points and roughly equal in the categorization of scenes vs. faces 32 [5] Performance on the memory recognition test was retained across subjects on average 3-4 months post-encoding 32 [6] Receiver Operating Characteristic (ROC) Analysis for 30 hours vs. 3-4 months post-encoding showed a decrease in both the False Positive Rate (FPR) and True Positive Rate (TPR). 34 [7] Average Shot Content Properties correlated with performance across the subject pool (A-J). Performance for the memory recognition test that corresponds to if the content was present in the shot or not (mean +/- SEM). 37 [8] Performance in the MFA variant was consistent with version one and had no statistically significant difference in scenes vs. faces 39 [9] Receiver Operating Characteristic (ROC) Analysis for the MFA variant 41 [10] Shot Content Properties correlated with performance (A-H). Performance for the memory recognition test that corresponds to if the content was present in the shot or not (mean +/- SEM). 43 List of Tables [1] Summary of subjects and test conditions in the main experiment and MFA variant 31 Marconi 4 Introduction Quantifying episodic memories from real-world experience Episodic Memory Formation and Retrieval from Real-World Experience Memories are the fabric of who we are and include personal experiences that occurred a specific time or place in the past. Episodic memories, commonly referred to as autobiographical memories, are memories that bring you back to a specific time or place and include the who, what, when, where, and why knowledge (Tulving 2002). These types of memories allow us to travel back to the time of that event and recollect the specific context in which it happened. Episodic memories are types of declarative memory (fig. 3). To understand episodic memories, we must decipher the mechanisms involved in the acquisition, retention, and retrieval of knowledge (Gabrieli 1998). Although episodic memories are detailed, they can also be extremely malleable, and large amounts of the details within them are typically forgotten (Tang et al. 2016). Memory is malleable because it is shared, constructed, elaborated, and exchange with social interactions (Cohen 1996, p.9). The main research question being addressed in this study is quantifying what people remember from episodic memories in real life. In the field of neuroscience, the biological mechanism for the formation and retrieval of the specific contents of episodic memories is still not clear. Previous studies have addressed this question and identified cues but have focused on studying individual words or images, lacking fundamental components of episodic memories including realistic context, which this study aims to address. One of the central challenges to studying memory formation in real life scenarios is the lack of ground truth data to evaluate the accuracy of memory recollections. To study the formation and retrieval of episodic memories in real life, we collected ground truth data from subjects during the first portion of the experiment known as “encoding” (fig. 1). This data from Marconi 5 the subjects’ experience was used in a memory recognition test that took place 24 hours and again at 3 months post-encoding (fig. 1c and 1d). In the context of this study, ground truth data refers to data that we collected about specific sequences of events experienced by the test subjects as described below. In the case of our study, the ground truth data refers to footage obtained with a GoPro camera and Eye-link eye-tracking results collected when the subject performs the encoding portion of the experiment (fig. 1a), either in version 1 (outdoors), or in version 2 which took place indoors (fig. 1e and 1f). The extent to what subjects recall from their episodic experience depends on a multitude of factors including: age, time between encoding and the memory recognition test, context of the episodic memory, and similarity between the clips the subject actually did see (target clips) and control clips the subject did not see (foil clips). By evaluating a subject's memory of their episodic experience using ground truth data in natural conditions we can better understand how we collect and retrieve parts of our everyday lives. In this way, we can quantify what aspects of a memory people can retrieve. Marconi 6 Figure 1: Real-world eye-tracking apparatus for encoding (Part I) and footage for the memory recognition task (Part II). (A) Subjects were fitted with a pair of ASL eye tracking glasses and a supplemental GoPro camera that enhanced the quality and field of vision (FOV) of the recorded video of the subject's visual environment which was mounted to a helmet. (B) Calibration (moving dot) and ASL-to-GoPro synchronization and registration (checkerboard pattern) were automated and standardized across participants using MATLAB psychophysics toolbox. (C + D) An overview of the memory recognition tasks subjects complete post-encoding including"scenes" or "faces." Subjects are shown an equal amount of "target" clips from their experience and "foil" clips those that are not from their experience and asked to choose if they remember the clip in a "yes" or "no" task where clips are pseudo-randomized and appear in one- second segments. (E +F) Route maps for the two versions of this experiment, outdoors (E) and indoors at the MFA (F). These routes were controlled and where subjects were instructed to walk during their encoding experience. Alzheimer’s Disease and Episodic Memory By quantifying what people remember from an episodic memory through a range of characteristics, then we can learn and apply these findings to a number of disciplines within neuroscience as well as gain insight into how people learn. Almost every aspect of our daily lives is influenced by memory. Since memories are the internal mental records we keep, they allow us to access our personal past, which includes all of the skills we have acquired and facts we have learned. When a person is not able to form new memories or recall previous memories, life becomes extremely difficult. This is particularly seen in neurodegenerative disorders where the key cognitive decline includes memory loss such as Alzheimer's and Dementia. Dementia, which is a symptom of Alzheimer's, makes it a truly debilitating disease that is not only becoming widespread in the US but the world. The harsh reality is that as time progresses, the disease is only predicted to become more prevalent because people's life expectancies will increase due to other medical advances.