Opinion TRENDS in Neurosciences Vol.26 No.8 August 2003 423 A computational substrate for incentive salience Samuel M. McClure1, Nathaniel D. Daw2 and P. Read Montague1 1Center for Theoretical Neuroscience, Human Neuroimaging Laboratory, Baylor College of Medicine, 1 Baylor Plaza, Houston, TX 77030, USA 2Computer Science Department, Center for the Neural Basis of Cognition, Carnegie Mellon University, Pittsburgh, PA 15213, USA Theories of dopamine function are at a crossroads. On the contrary, we find that the degree of overlap of the Computational models derived from single-unit record- experimental findings represents a tremendous improve- ings capture changes in dopaminergic neuron firing ment in our integrated understanding of dopaminergic rate as a prediction error signal. These models employ neuron function. the prediction error signal in two roles: learning to We begin with a description of an experiment that predict future rewarding events and biasing action captures the core of the incentive salience hypothesis of choice. Conversely, pharmacological inhibition or lesion dopamine function. This description is then seated within of dopaminergic neuron function diminishes the ability the prediction error theory of dopamine function, which of an animal to motivate behaviors directed at acquiring suggests that phasic changes in dopaminergic cell firing rewards. These lesion experiments have raised the encode an error in the prediction of future reward (Eqn 3) possibility that dopamine release encodes a measure of [3,4]. This approach offers an extendable, mathematical the incentive value of a contemplated behavioral act. model that captures the concept of incentive salience. We The most complete psychological idea that captures also show how this model explains a group of earlier this notion frames the dopamine signal as carrying behavioral experiments that reveal clearly the dual ‘incentive salience’. On the surface, these two compet- function of dopamine release as both a learning signal ing accounts of dopamine function seem incommensu- and an action bias. rate. To the contrary, we demonstrate that both of these functions can be captured in a single compu- The incentive salience hypothesis tational model of the involvement of dopamine in The incentive salience hypothesis is grounded in a growing reward prediction for the purpose of reward seeking. history of behavioral effects that accrue in response to manipulation of dopaminergic neuron function. One Many aspects of behavior are dedicated to identifying, singular finding is that dopamine receptor antagonism judging and pursuing objects and goals. It has long been does not change the appetitive value of rewards, but recognized that the mesencephalic dopamine system is appears to inhibit the ability to initiate actions necessary centrally involved in these processes for a range of to acquire the rewards [2,5–7]. These findings have led to behaviors, over and above its role in purely motor the idea that dopamine receptor antagonism selectively functions. One classic attempt to account for dopamine- inhibits the capacity to initiate reward seeking actions, but mediated events in reward and motivation was the has no effect on the value of the reward to the animal anhedonia hypothesis of dopamine dysfunction [1], [2,5–7]. That is, the internal valuation of the reward by which proposed that dopamine release was the hedonic the animal is not changed, but its capacity to act on that signal of the brain that represented the pleasure associ- valuation is inhibited. This view of dopamine function is ated with a primary reward. According to this hypothesis, supported by an enormous body of literature, which we do deficits caused by interference with dopamine followed not seek to summarize. In our opinion, the most fruitful from the lack of this hedonic signal. A variety of summary of these effects is captured in the idea of incentive subsequent experimental results have challenged this salience as proposed by Berridge and Robinson [2] (but see initial formulation. In particular, manipulations leading to Refs [5,6] for modifications of their basic proposal). The idea reduced dopamine-mediated function do not affect hedonic of incentive salience is that dopamine release assigns responses to primary rewards [2], and increased activity of incentive value to objects or behavioral acts. These assigned dopaminergic neurons is often tied to events that precede values are then available to be used by some action selection reward consumption [3]. In light of these data, several system that makes more valuable actions more likely. theories have been posited to replace the anhedonia Accordingly, antagonism of dopamine receptor function hypothesis, all of which implicate dopamine in reward does not influence the assignment of value, but does anticipation or reward seeking [2,4–6]. These theories are inhibit the use of these values in choosing actions. generally depicted as being inconsistent with one another. An experiment by Ikemoto and Panksepp [8] (Fig. 1) highlights the essential features of the incentive salience Corresponding author: P. Read Montague ([email protected]). hypothesis. Rats were trained to traverse a one-arm maze http://tins.trends.com 0166-2236/03/$ - see front matter q 2003 Elsevier Science Ltd. All rights reserved. doi:10.1016/S0166-2236(03)00177-2 424 Opinion TRENDS in Neurosciences Vol.26 No.8 August 2003 into the ventral tegmental area and injection of the (a) dopamine receptor agonist cis-flupentixol into the nucleus accumbens, caused dramatic behavioral deficiencies (Fig. 1b). The baseline levels of movement X Start 1 of the rats outside the context of the task were dramatically decreased. Furthermore, their running X2 Drinking bottle speeds within the maze were significantly less than X3 Goal those of saline controls. However, the treated rats maintained the same desire for the sucrose solution: when the investigators moved them to the bottle, they (b) drank normal volumes of liquid during a 30 s reward NAc VTA period (Fig. 1b, gray bars). This latter observation Intake 100 100 demonstrates that dopaminergic neuron antagonism 80 80 60 60 Running did not affect the motivation for the reward of the rats. 40 40 These results have been interpreted to mean that Activity 20 20 dopamine does not encode the pleasure associated with % Control level 0 0 cis-flupentixol GABA rewards, as the anhedonia hypothesis suggests, but instead enables reward-seeking behaviors. Changes in TRENDS in Neurosciences dopamine-mediated activity are appropriately timed for Fig. 1. The incentive salience hypothesis. (a) Rats were trained to traverse a one- such a role: increased activation precedes reward-motiv- armed maze to obtain sucrose solution at the end. Photosensors (arrows) were ated actions, instead of being linked to the time of reward used to determine running speed [blue bars in (b)], in addition to the baseline level consumption [4]. Moreover, behavioral measures of affec- of movement in the start box [green bars in (b)], while access to the runway was tive reactions to primary rewards are unaltered by blocked by a door. X1,X2 and X3 represent intermediate states in the model, as shown in Fig. 3. (b) After training, dopaminergic neuron activity was reduced dopaminergic neuron antagonism [2]. Based on these either by application of the dopamine receptor antagonist cis-flupentixol in the behavioral data, Berridge and Robinson suggest that nucleus accumbens (NAc) or by injection of GABA into the ventral tegmental area (VTA). Both manipulations reduced the ability of the rats to initiate the running dopamine is responsible for assigning ‘incentive salience’ needed to acquire the sucrose solution (P , 0:01; loss of ability to ascribe incentive to objects and behavioral acts. Incentive salience maps salience), while leaving the volume of sucrose they consumed unaffected (gray ‘liked’ objects or acts to ‘wanted’ objects or acts (i.e. objects bars). Using data from Ref. [8]. or acts that an animal will ‘work to acquire’) [2,6]. In their to obtain access to sucrose solution at the end (Fig. 1a). words, dopamine ‘transforms the neural representation of After training, investigators administered either saline or a stimulus into an object of attraction that animals will one of three pharmacological agents into specific areas in work to acquire’, and this attraction is transferred to the brain. Two of the manipulations degraded dopamin- conditioned stimuli associated with reward [2]. This ergic neuron function. Both of these, injection of GABA mapping is illustrated qualitatively in Fig. 2a. (a) (b) Expected reward Actual reward γ Like V(st+1) – V(st) r(t) Dopamine Dopamine + neurons neurons Attribution of δ(t) incentive salience Want ∆ α δ –m(δ(t) – b) –1 wi = (t) P = (e ) Learning Action selection TRENDS in Neurosciences Fig. 2. Theories of dopamine function. (a) The incentive salience hypothesis. Dopamine is proposed to be responsible for converting an object or action that is ‘liked’ into one that is ‘wanted’. In other words, dopamine ascribes incentive salience to objects and actions. This process is necessary to motivate actions aimed at acquiring rewards. (b) The prediction error hypothesis. Changes in dopaminergic neuron firing rate are hypothesized to encode a prediction error ðdÞ signal as part of a computational system dedicated to seeking and predicting rewards (temporal-difference model). Inputs from the world (states, st) are used to generate an internal estimate of a value function ðV