Inhibitory Learning As Prediction Error Feedback Loop: 10 a Neurocognitive Framework & Model 11 12 Matthew S
Total Page：16
File Type：pdf, Size：1020Kb
1 text count = 4128 2 text pages = 15 3 tables = 1 4 figures = 1 5 references = 65 6 7Running Head: Inhibitory Learning Prediction Error Feedback Loop 8 9 Inhibitory Learning as Prediction Error Feedback Loop: 10 A Neurocognitive Framework & Model 11 12 Matthew S. Price1 13Affiliation: 141University of Texas MD Anderson Cancer Center UTHealth Graduate School of Biomedical 15Sciences 16Corresponding author: 17Matthew S. Price, 6767 Bertner Ave, Houston, TX 77030. [email protected] 18Prior presentations: None. 19Conflicts of interest: None. 20Declarations of interest: None. 21Funding sources: No public, commercial, or not-for-profit-sector grant funding was received. 22Keywords: prediction error, inhibitory learning, emotion regulation, intolerance of uncertainty 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 Inhibitory Learning Prediction Error Feedback Loop 1Abstract 2Inhibitory learning promotes emotion regulation via systematic exposure to fear-inducing stimuli. Given 3that inconsistencies between expectations, states, and outcomes may be experienced as elements of 4inhibitory learning, to what extent are prediction errors – mismatches between expectations and 5outcomes – a core neural element of inhibitory learning? This paper takes a complex systems approach 6to prediction errors and postulates that a prediction error feedback loop – a series of self-perpetuating 7disparities between expected and perceived outcomes – could be a correlate of or responsible for 8improved emotion regulation from inhibitory learning. The inhibitory learning prediction error feedback 9loop may additionally elucidate how human and animal studies demonstrate improved emotion 10regulation in the form of reduced fear responses without exposure to specific fear-inducing stimuli. 11Introduction 12Inhibitory learning is a behavioral learning approach employed by many modern psychotherapies that 13promote emotion regulation via systematic interaction with fear-inducing stimuli (Abramowitz & Arch 142014; Blakey & Abramowitz 2016; Gross 2015; Knowles & Olatunji 2019). Given that inhibitory learning 15involves discrepancies between: a) expectations; b) subjective states experienced due to exposure to 16fear-inducing stimuli; and c) the eventual outcome (emotion regulation), could a neuroeconomic 17process that relies on prediction errors – neural signaling that occurs as a result of mismatches 18between expectations and perceived outcomes – account for improvements in emotion regulation from 19the behavioral strategy? 20This paper aims to conceptualize inhibitory learning as a prediction error feedback loop, i.e., a neural 21algorithm of self-promoting mismatches between perceived and expected outcomes, that correlates 22with or promotes emotion regulation. Features of the inhibitory learning prediction error feedback loop 23may elucidate how emotion regulation in the form of reduced fear responses can be demonstrated 24without exposure to fear-inducing stimuli in animal and human studies. 1 Inhibitory Learning Prediction Error Feedback Loop 1Temporal Distance Learning & Prediction Errors 2Temporal difference learning is a type of unsupervised learning derived from the highly-influential 3Rescorla-Wagner model that suggests that stimulus–outcome associations may be formed during 4instrumental (or operant) learning — learning about rewards and punishments to guide appetitive and 5avoidant behavior — via comparisons of expectations or predictions to present experiences (Maia & 6Frank 2011; Miller, Barnet, & Grahame 1995; Robinson, Overstreet, Charney, Vytal, & Grillon 2013). 7During temporal difference learning, mental associations are mediated by differences or mismatch 8between predicted and perceived outcomes called prediction errors, which are hypothesized to encode 9the intensity of the arousal (surprise), as well as the valence (positivity/negativity), of a discrepancy 10between an expected reward or punishment and an outcome (Fouragnan, Retzler, & Philiastides 2018). 11The following types of prediction errors have been postulated to exist: 12 1. Reward prediction errors (RPE), which encode mismatch between expected and perceived 13 rewards. Behaviorally, RPE appear to update learned action valuations — differences between 14 a stimulus value (the ‘innate’ cost-benefit equation for the stimulus targeted for action) and 15 action costs (the cost of performing the behavior) (Suri, Sheppes, & Gross 2015). RPE may thus 16 provide information relevant to action selection — which action should be selected — not action 17 specification — how a selected action should be performed (Frömer, Nassar, Stürmer, Sommer, 18 & Yeung 2018). Two types of RPE are posited to exist: 19 a. positive RPE, which occur when outcomes are better than expected; 20 b. negative RPE, which occur due to failure to attain an expected reward; 21 2. Aversive prediction errors (APE), which encode mismatch between expected and perceived 22 punishments. Two types of APE are posited to exist: 23 a. positive APE, which occur when outcomes are worse than expected; 24 b. negative APE, which occur due to absence of an expected punishment (Roy et al. 2014). 2 Inhibitory Learning Prediction Error Feedback Loop 1Corticothalamic Circuits & the Development & Treatment of 2Mental Illness 3Significant research has focused on the roles of dopamine (DA) and cortico-basal ganglia- 4thalamocortical loops (CBGTC circuits) in operant learning. Mesolimbic DA encoding plays a central 5role in CBGTC circuits and prediction error signaling (Maia & Frank 2011) in a process known as 6predictive processing (Kaaronen 2018). CBGTC circuits compose feedback loops used to process 7affective, cognitive, and motor information (Marchand 2012) extensively involved in inhibitory control 8(Wei & Wang 2016). In the salience network (SN), CBGTC circuits are postulated to formulate a 9mechanism for the development and treatment of many psychiatric conditions (Peters, Dunlop, & 10Downar 2016). These circuits play a role in disorders such as addiction, schizophrenia, obsessive- 11compulsive disorder (OCD), posttraumatic stress disorder (PTSD), chronic pain, depression, 12anhedonia, and suicide (Cisler et al. 2018; Gradin et al. 2011; Maia & Frank 2011; Ploner, Sorg, & 13Gross 2017; Schmaal et al. 2019; Shin & Liberzon 2009; Ubl et al. 2015), which share emotion 14dysregulation as an etiological and/or maintenance factor (Akram et al. 2020; Der-Avakian & Markou 152012; Dvir, Ford, Hill, & Frazier 2014; Eskelund, Karstoft, & Andersen 2018; Garfield, Lubman, & Yücel 162014; Riquino, Priddy, Howard, & Garland 2018; Winer et al. 2017). Thus, abnormalities in CBGTC 17circuits and prediction errors are not only associated with mental health problems characterized by a 18lack of cognitive control over maladaptive thoughts, impulsive behaviors, and inattention to relevant 19internal and external stimuli (Peters et al. 2016), but these circuits may also be relevant to the treatment 20of disorders which share emotion dysregulation more generally as an etiological and/or maintenance 21factor. 22Expected and Actual Reward Values & Emotions 23Subjective states such as emotions may rely in whole or in part on prediction error encoding. Prediction 24errors and emotions commonly signify intensity of arousal (surprise) and are positively or negatively 3 Inhibitory Learning Prediction Error Feedback Loop 1valenced; however, emotions also depend on other cognitive factors such as uncertainty and 2perceptions/interpretations of interoceptive responses to external and internal stimuli (Anderson, 3Carleton, Diefenbach, & Han 2019; Fouragnan et al. 2018; Seth & Critchley 2013). In this regard, 4subjective states such as emotions may rely in whole or in part on mismatches between the actual 5value of a reward or punishment — which is thought to consist of three components: 1) the magnitude 6or size; 2) probability; and 3) timing (immediate or delayed) of a predicted reward or punishment — and 7the expected value of a reward or punishment, defined as the product of the probability and magnitude 8of a reward or punishment (Abler, Walter, Erk, Kammerer, & Spitzer 2006; Fouragnan et al. 2018). 9For example, in rodents and monkeys, optogenetic DA inhibition – negative RPE – has been found to 10induce avoidance behaviors (Schultz 2017). In humans, the emotion ‘frustration’ is associated with 11negative RPE (Toates 1988), which may in turn be linked to the cognitive state of uncertainty. Though 12uncertainty may arise when we doubt whether a situation or outcome will or will not occur (Wever, 13Smeets, & Sternheim 2015), uncertainty may rather be said to exist “when the likelihood of future 14events [rewards and/or punishments] is indefinite or incalculable” (Knight 1921). In other words, 15uncertainty may be said to exist when a null value (ø) is present in the probability component of an 16expected value of a reward or punishment. However, “probability or risk is calculable or estimable” 17(Knight 1921), i.e., is a result of environmental feedback updating expected values of rewards and 18punishments with actual values. 19The ability to calculate the probability of attaining a reward is important for a number of reasons, one of 20which is that action cost (theoretically a function of expected value) enhances RPE signaling in 21midbrain DA neurons (Tanaka, O’Doherty, & Sakagami 2019). Thus, the duration of time that a null 22value (ø) is not updated in the probability component of an actual value of a reward can become 23associated with a likelihood