This Accepted Manuscript has not been copyedited and formatted. The final version may differ from this version. Research Articles: Behavioral/Cognitive Neural integration of stimulus history underlies prediction for naturalistically evolving sequences Brian Maniscalco1,2, Jennifer L. Lee2, Patrice Abry3, Amy Lin1,4, Tom Holroyd5 and Biyu J. He1,2,6 1National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, Maryland 20892 2Neuroscience Institute, New York University Langone Medical Center, New York, NY 3Univ Lyon, Ens de Lyon, Univ Claude Bernard, CNRS, Laboratoire de Physique, F-69342 Lyon, France 4Department of Psychiatry and Biobehavioral Sciences, Semel Institute for Neurocience and Human Behavior, UCLA, Los Angeles, CA 5National Institute of Mental Health, National Institutes of Health, Bethesda, MD 6Departments of Neurology, Neuroscience and Physiology, and Radiology, New York University Langone Medical Center, NY 10016. DOI: 10.1523/JNEUROSCI.1779-17.2017 Received: 26 June 2017 Revised: 19 December 2017 Accepted: 21 December 2017 Published: 8 January 2018 Author contributions: BM and BJH designed the study; PA provided methodological expertise for 1/f stimuli creation and mathematical analysis; BM, JL, and AL collected the data; TH assisted data collection and designed the MEG head localization procedure; BM, JL, and BJH analyzed the data and wrote the manuscript. Conflict of Interest: The authors declare no competing financial interests. This research was supported by the Intramural Research Program of the National Institutes of Health/National Institute of Neurological Disorders and Stroke and New York University Langone Medical Center. BJH further acknowledges support by Leon Levy Foundation and Klingenstein-Simons Neuroscience Fellowship. We thank Ella Podvalny, Adeen Flinker, and Jean-Rémi King for comments on a previous draft of the manuscript. Correspondence should be addressed to To whom correspondence should be addressed. 550 1st Avenue, MSB 460, New York, NY 10016. Email: [email protected]; Tel: +1 (646) 501-2422 Cite as: J. Neurosci ; 10.1523/JNEUROSCI.1779-17.2017 Alerts: Sign up at www.jneurosci.org/cgi/alerts to receive customized email alerts when the fully formatted version of this article is published. Accepted manuscripts are peer-reviewed but have not been through the copyediting, formatting, or proofreading process. Copyright © 2018 the authors 1 Neural integration of stimulus history underlies prediction for naturalistically 2 evolving sequences 3 Abbreviated title: Predictive processing of naturalistic stimuli 4 5 Brian Maniscalco1,2, Jennifer L. Lee2, Patrice Abry3, Amy Lin1,4, Tom Holroyd5, and Biyu J. 6 He1,2,6* 7 1 National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, 8 Maryland 20892 9 2 Neuroscience Institute, New York University Langone Medical Center, New York, NY 10 3 Univ Lyon, Ens de Lyon, Univ Claude Bernard, CNRS, Laboratoire de Physique, F-69342 11 Lyon, France 12 4 Department of Psychiatry and Biobehavioral Sciences, Semel Institute for Neurocience and 13 Human Behavior, UCLA, Los Angeles, CA 14 5 National Institute of Mental Health, National Institutes of Health, Bethesda, MD 15 6 Departments of Neurology, Neuroscience and Physiology, and Radiology, New York University 16 Langone Medical Center, NY 10016. 17 18 * To whom correspondence should be addressed. 550 1st Avenue, MSB 460, New York, NY 19 10016. Email: [email protected]; Tel: +1 (646) 501-2422 20 21 Number of pages: 51; Number of figures: 8; 22 Number of words for Abstract: 158, Introduction: 723, and Discussion: 1,733. 23 Conflict of Interest: The authors declare no competing financial interests. 24 25 Acknowledgements: This research was supported by the Intramural Research Program of the 26 National Institutes of Health / National Institute of Neurological Disorders and Stroke and New 27 York University Langone Medical Center. BJH further acknowledges support by Leon Levy 28 Foundation and Klingenstein-Simons Neuroscience Fellowship. We thank Ella Podvalny, Adeen 29 Flinker, and Jean-Rémi King for comments on a previous draft of the manuscript. 30 31 Author contributions: BM and BJH designed the study; PA provided methodological expertise 32 for 1/f stimuli creation and mathematical analysis; BM, JL, and AL collected the data; TH 33 assisted data collection and designed the MEG head localization procedure; BM, JL, and BJH 34 analyzed the data and wrote the manuscript. 1 35 ABSTRACT 36 Forming valid predictions about the environment is crucial to survival. However, whether 37 humans are able to form valid predictions about natural stimuli based on their temporal 38 statistical regularities remains unknown. Here we presented subjects with tone sequences 39 whose pitch fluctuation over time capture long-range temporal dependence structures prevalent 40 in natural stimuli. We found that subjects were able to exploit such naturalistic statistical 41 regularities to make valid predictions about upcoming items in a sequence. 42 Magnetoencephalography (MEG) recordings revealed that slow, arrhythmic cortical dynamics 43 tracked the evolving pitch sequence over time such that neural activity at a given moment was 44 influenced by the pitch of up to seven previous tones. Importantly, such history integration 45 contained in neural activity predicted the expected pitch of the upcoming tone, providing a 46 concrete computational mechanism for prediction. These results establish humans’ ability to 47 make valid predictions based on temporal regularities inherent in naturalistic stimuli and further 48 reveal the neural mechanisms underlying such predictive computation. 49 50 SIGNIFICANCE 51 A fundamental question in neuroscience is how the brain predicts upcoming events in the 52 environment. To date, this question has primarily been addressed in experiments using 53 relatively simple stimulus sequences. Here, we study predictive processing in the human brain 54 using auditory tone sequences that exhibit temporal statistical regularities similar to those found 55 in natural stimuli. We observed that humans are able to form valid predictions based on such 56 complex temporal statistical regularities. We further show that neural response to a given tone 57 in the sequence reflects integration over the preceding tone sequence, and that this history 58 dependence forms the foundation for prediction. These findings deepen our understanding of 59 how humans form predictions in an ecologically valid environment. 2 60 INTRODUCTION 61 In real-life environments, prior sensory information across multiple time scales continually 62 influences information processing in the present. To date, the relationship between neural 63 integration of information over time and predictive computations remains mysterious (Hasson et 64 al., 2015). The importance of this question is underscored by the fact that forming valid 65 predictions about the features (Summerfield and de Lange, 2014) and timing (Cravo et al., 66 2011; Jepma et al., 2012) of upcoming stimuli in the environment is crucial for survival. 67 Predictions about stimulus features can be generated through a variety of mechanisms, 68 including associative learning (Schlack and Albright, 2007), contextual influences (Bar, 2004), 69 imagination (Schacter et al., 2007), task cues (Wyart et al., 2012; de Lange et al., 2013), and 70 extrapolation from statistical regularities in sensory input (Saffran et al., 1996; Alink et al., 2010; 71 Chalk et al., 2010). Since statistical regularities are ubiquitous in natural stimuli (Dong and Atick, 72 1995; Summerfield and de Lange, 2014), extrapolating from statistical regularities in sensory 73 input should provide a fundamental strategy for forming predictions in natural environments. 74 75 Previous studies investigating predictions based on statistical regularities have largely used 76 relatively simple stimuli, such as the oddball paradigm where a novel stimulus is embedded 77 within a sequence of repeated stimuli (Yaron et al., 2012), local-global paradigm where a 78 sequence including two values contains both local and global regularities (Bekinschtein et al., 79 2009), and repeated presentations of a fixed sequence of stimuli (Erickson and Desimone, 80 1999; Meyer and Olson, 2011; Gavornik and Bear, 2014). While such stimuli allow for tight 81 experimental control, they induce the formation of predictions by repeated presentations of 82 items or sequences of items. By contrast, temporally varying stimuli encountered in the natural 83 environment have rich statistical structures that allow for more complex forms of predictions. At 84 present, whether humans can form valid predictions based on temporal statistical regularities 85 inherent in natural stimuli remains unknown. 3 86 87 In particular, one pervasive statistical feature of natural stimuli is that they exhibit power spectra 88 following a P v 1 / f β pattern in spatial (Tolhurst et al., 2007) and temporal frequency domains, 89 where β is an exponent commonly ranging between 0 and 2. In the temporal domain, a 1 / f β 90 pattern is observed in the loudness and pitch fluctuations of music, speech, and ambient noise 91 in urban and rural settings (Voss and Clarke, 1975; De Coensel et al., 2003), and in the rhythms 92 of music (Levitin et al., 2012). Previous work has shown that human perception and neural 93 activity are sensitive to the scaling parameter β of dynamic auditory stimuli such as tone 94 sequences