Journal of Neuroscience, Psychology, and Economics © 2010 American Psychological Association 2010, Vol. 3, No. 1, 27–45 1937-321X/10/$12.00 DOI: 10.1037/a0018046 Functional Neuroimaging of Intertemporal Choice Models: A Review

R. McKell Carter, Justin R. Meyer, and Scott A. Huettel Duke University

People often forsake a larger reward later for a smaller reward sooner. The process of devaluing the larger, later prize is called temporal discounting or delay discounting, which lies at the core of intertemporal choice. Here, we describe the methodology and findings of research on the mechanisms of intertemporal choice, with a focus on those that utilize functional MRI (fMRI). We consider the neural bases for the most common economic models of intertemporal choice and examine whether these models require neural processes that are common or distinct across types of decision making. Con- sidered as a whole, current research points to potentially distinct contributions from brain systems associated with valuation and with prospective thought, which may be reflected in separable foci in posterior cingulate cortex. Based on open questions in the field, we suggest two core goals for future research: identifying aspects of valuation that are unique to intertemporal choice and evaluating direct or indirect interactions be- tween delay and prize magnitude.

Keywords: delay discounting, intertemporal choice, neuroimaging, fMRI, decision making

Adaptive decision making facilitates the ef- Because most people judge rewards to be less fective consumption and allocation of resources valuable when obtained later in time, this pro- in the pursuit of survival. Some decisions in- cess is known as temporal discounting. Dys- volve choices between two immediately avail- functions in intertemporal choice mechanisms able options that differ in aspects of their value are thought to contribute to a wide-range of (e.g., relative preferences between attributes) or pathologies of decision making, from failures of their probability of occurrence (e.g., risk and planning and investment, and inequities in re- uncertainty). However, an important class of source allocation (Angeletos, Laibson, Repetto, decisions involves comparison of options that Tobacman, & Weinberg, 2001; Laibson, 1997), differ in the time at which they would be avail- to pathologies of decision making in addiction able, as when choosing between a smaller re- (Bickel & Marsch, 2001; Perry & Carroll, ward available immediately and a larger reward 2008), and attention deficit disorder (Critchfield only obtainable after some delay. In such inter- & Kollins, 2001; Plichta et al., 2009). temporal choice situations, decision makers Individual differences in intertemporal choice must adjust the subjective value of the later can have striking consequences. In a seminal reward to account for the delay until its arrival. study, Mischel and colleagues examined cognitive and social competencies in children who varied in their ability to delay gratification (Mischel, Shoda, &Peake,1988).Preschoolchildrenwereshowna R. McKell Carter, Center for Cognitive Neuroscience, Department of Neurobiology, Duke University; Justin R. small food reward and then offered the oppor- Meyer, Center for Cognitive Neuroscience, Duke Univer- tunity to obtain a more desirable food reward, if sity; and Scott A. Huettel, Center for Cognitive Neuro- they waited until the experimenter returned to science, Psychology and Neuroscience, Duke University. the room to eat the food. Those children who We thank John Clithero for assistance in preparation of the manuscript and Vinod Venkatraman and O’Dhaniel were able to wait longer in this simple paradigm Mullette-Gillman for their comments on the manuscript. scored better on cognitive, coping, and social SAH was supported by an incubator award from the Duke competency measures in a follow-up Study 10 Institute for Brain Sciences. years later. Variation in tolerance for reward Correspondence concerning this article should be addressed to Scott A. Huettel, Box 90999, Duke University, Durham, NC delay can be attributed to the use of specific 27708. E-mail: [email protected] cognitive strategies, which points toward im-

27 28 CARTER, MEYER, AND HUETTEL portant directions for improving decision mak- ternative models that have these characteristics ing through a better understanding of intertem- include a hyperbolic function (Kirby & Marak- poral choice. ovic, 1995; Laibson, 1997, 1998; Strotz, 1956) There exists a substantial literature on inter- as well as a quasi-hyperbolic function utilizing temporal choice behavior. As first described by two exponential decay functions (Loewenstein, Samuelson (1937) and now replicated and ex- 1996; Phelps, 1968; Shefrin & Thaler, 1988). tended in many studies, most individuals will- Given the importance of intertemporal ingly sacrifice value to obtain a reward within a choice – and the ongoing debate about its con- shorter time interval (for reviews, see Frederick, stituent processes – there has been substantial Loewenstein, & O’Donoghue, 2002; Kalen- recent interest in using neuroscience methods to scher & Pennartz, 2008; Loewenstein & Prelec, elucidate its underlying mechanisms. Reflecting 1992). Such discounting effects are ubiquitous this interest, more than a dozen recent func- for both primary rewards (e.g., food, juice; Ain- tional neuroimaging studies (see Table 1) have slie, 1974; Kobayashi & Schultz, 2008; explored a range of intertemporal choice phe- McClure, Ericson, Laibson, Loewenstein, & nomena. We begin this review with an overview Cohen, 2007; Richards, Mitchell, Wit, & Sei- of these studies’ methods, with particular focus den, 1997) and secondary rewards (e.g., money; on how designs used in neuroimaging research Thaler, 1981; for review see Loewenstein & differ from those used in behavioral research Prelec, 1992; Strotz, 1956; Tesch & Sanfey, (both in humans and animals). We then examine 2008), although an opposite phenomenon is the evidence that addresses two core questions sometimes observed for aversive stimuli, which about intertemporal choice. How is the value of can become more negative if farther away in a delayed reward computed? And, are there time (i.e., increasingly dreaded; Berns et al., aspects of the valuation and decision process 2006). In Samuelson’s formulation, temporal that are unique to intertemporal choice? Together, these questions bear on a topic discounting was modeled as an exponential de- central to neuroeconomic research: the exis- cay function where the subjective value (U), tence of a common neural currency for reward after a delay D, is given by U ϭ AeϪ␤D (Equa- (Deaner, Khera, & Platt, 2005; Montague & tion 1), where A is the amount of the prize and Berns, 2002; Montague & King-Casas, 2007). ␤ is the discount rate. Using an exponential At one extreme, evidence that intertemporal function can be readily shown to be normative, choice shares valuation and decision mecha- in that it treats all future delay intervals simi- nisms with other forms of choice (Benzion, larly and thus cannot lead to inconsistent or Rapoport, & Yagil, 1989; Green & Myerson, time-varying preferences. 1996) would support the conclusion that subjec- More recent formulations have adopted alter- tive value is represented in a single neural cur- native mathematical frameworks to account for rency. Under this perspective, temporal delay is anomalies in intertemporal choice behavior. one of many factors that contribute to a repre- Most notably, individuals tend to exhibit incon- sentation of value for a choice option, but that sistent preferences depending on the time until the value representation itself has no special rewards are available (Prelec & Loewenstein, qualities because of temporal delay. At another 1991; Thaler, 1981). For example, when choos- extreme, evidence of fundamental differences ing between $100 now and $110 in 2 weeks, between intertemporal choice and other types of most individuals prefer the smaller and more choice could lead to the conclusion that inter- immediate reward. However, when choosing temporal choice reflects, at least in part, the between $100 in 36 weeks and $110 in 38 workings of a separate system for valuation weeks, nearly all people choose the larger re- (Chapman & Weber, 2006; Prelec & Loewen- ward—even though this latter scenario will be- stein, 1991). Thus considered, the subjective come identical to the former in 36 weeks. This value of a delayed outcome would only be com- inconsistency, often called the immediacy ef- pared with other value signals at a final step fect, implies that the mechanism used to calcu- before output (e.g., response selection). As a late intertemporal value shows a strong prefer- pre´cis of our conclusions, the current evidence ence for immediate rewards but a shallower from neuroimaging studies supports an interme- discounting curve at longer times. Proposed al- diate perspective: that initial aspects of inter- Table 1 CHOICE INTERTEMPORAL OF NEUROIMAGING ISSUE: SPECIAL Characteristics of Studies Included in This Review Immediate prize Delay prize Delay length Reference Model Prize Min Max Min Max Min Max McClure 2004 ␤␦ Gift certificate $5 $39 $6 $40 0 wk 6 wk Tanaka 2004 TD Cumulative direct pay $0.09 $0.26 Ϫ$0.26 $0.95 0 steps 3 steps Hariri 2006 k Hypothetical $0.1 $105 $100 $100 0 d 5 yr Boettiger 2007 ICR Hypothetical $0.7 $95 $1 $100 1 wk 6 mo Kable 2007 k Debit cards $20 $20 $20.25 $110 6 hr 180 d McClure 2007 ␤␦ Juice/Water 1 ml 2 ml 2 ml 3 ml 0 min 20 min Wittmann 2007 ␤␦ Hypothetical $0 $524 $476 $524 5 d 10 yr Ballard 2009 k, ␤␦,&␤ Direct pay $10 $10 $10 $25 0 d 180 d Bjork 2008 (VBM) k Direct pay $0.25 $10 $10 $10 0 d 1 yr Ersner-Hershfield 2008 k Direct pay $0.01 $10.5 $10 $10 0 d 1 yr Hoffman 2008 k N/A $1 $99 $100 $100 0 d 1 yr Luhmann 2008 ␤ Cumulative direct pay $0.1 $0.2 $0.1 $0.2 1 s 9 s Shamosh 2008 AUC Hypothetical $10 $40,000 $200 $40,000 1 mo 8 yr Weber 2008 k Gift certificate $5 $7 $6 $12.6 0 mo 5 mo Clithero 2009 k Gift certificate NA NA $15 $50 1 wk 16 wk Plichta 2009 ICR Direct pay $6.86 $54.84 $7.2 $82.27 0 wk 4 wk s 13.5 s 4.0 20$ 20$ ء20$ 0$ ءGregorios-Pippas 2009 k & ␤ Cumulative direct pay Note. Prizes paid in foreign currencies were converted to US dollars using the average exchange rate from the year before the publication of the study. TD ϭ temporal difference; ICR ϭ impulsive choice ratio; AUC ϭ area under the curve; Hypo. ϭ hypothetical; N/A ϭ not applicable. .Paid only a fraction of the total cumulative ء 29 30 CARTER, MEYER, AND HUETTEL temporal valuation reflect neural mechanisms the more-delayed option (3 studies). The param- that differ from that of other forms of choice, eters for these functions are normally calculated but that the associated value signals are later on an individual-by-individual basis by offering represented within a common reward system. each participant a series of choices and then determining pairs of immediate and delayed rewards with equivalent value. Alternatively, Methodological Differences in Delay some studies have estimated the parameters that Discounting Studies best fit the collection of choices made by the participant. In determining discounting functions, The effects of temporal delay upon subjec- the majority of the studies (nine) vary the magni- tive value are, considered generally, predict- tude of both the delay and immediate prize, while able and intuitive: as the delay until a reward the remainders vary the magnitude of only one or increases, its subjective value decreases. The neither of the two prizes. The practice of varying generality of this phenomenon – which oper- only the delay prize (fixed immediate reward) ates across a wide range of rewards and time elicits greater discounting from participants than scales – has allowed researchers to use a only varying the immediate prize (fixed delay broad range of methods, reward modalities, reward; Tesch & Sanfey, 2008). choice options, and analysis parameters in their experiments. Here, we describe the ap- proaches used in published studies that have Prize Modalities used functional neuroimaging methods (in all There has been remarkable variability in the cases, functional MRI, or fMRI) to investi- reward parameters used within neuroimaging gate the neural basis for delay discounting. studies of intertemporal choice. Some of this Even within this restricted sample, there has diversity is driven by differences in delay pa- been a great deal of diversity in the methods rameters. To elicit variability in intertemporal used to isolate the effects of delay on rewards, choice, it is necessary to structure delays and despite the shared interest in a nominally sim- prize magnitudes so that participants sometimes ilar cognitive process. Listed in Table 1 are choose the delayed option and sometimes all (to our knowledge) published neuroimag- choose the immediate option. Thus, longer de- ing studies of intertemporal choice, along lays are often correlated with larger prizes. with their key methodological parameters. However, prize choice is also driven by the payment method. For reference, nearly all such Discounting Functions studies within behavioral economics have ex- amined choices for monetary rewards, the ma- As noted above, while the effects of delay jority of which have involved hypothetical re- upon value can be modeled normatively using a wards (see Frederick et al., 2002, for review). single exponential decay function, more com- Within our study sample, the prizes offered plex functions are needed to describe actual included direct monetary payments (cumulative choice behavior (Fudenberg & Levine, 2006; or random, 7 studies), gift certificates or debit Kalenscher & Pennartz, 2008; Loewenstein, cards (4 studies), hypothetical monetary pay- O’Donoghue, & Rabin, 2003; O’Donoghue & ments (4 studies), or juice/water (1 study). The Rabin, 2000; Ok & Masatlioglu, 2007). So far, range of symbolic prize magnitude, for a single neuroimaging studies of intertemporal choice trial, across studies is very large, from a mini- have modeled the effects of delay using five mum of $0.01 (in Ersner-Hershfield, Wimmer, different approaches: a single exponential decay & Knutson, 2008) to a maximum of $40,000 (in function (3 studies), a dual exponential decay (Shamosh et al., 2008)). Studies using a hypo- function (i.e., the beta-delta model; 4 studies), a thetical payment method have a median maxi- temporal difference model (i.e., performing a mum prize of $312 while those paying directly multiple regression analysis using a temporal (cash or gift certificate, usually on a few ran- difference model with a decay parameter rang- domly chosen trials) have a median maximum ing from 0 to 0.99; 1 study), a hyperbolic func- prize of $32.50. Those studies using a cumula- tion (i.e., calculating a k value; 8 studies), and tive direct pay method have the smallest (per some measure of the proportion of choices of trial) median maximum prize ($0.95). SPECIAL ISSUE: NEUROIMAGING OF INTERTEMPORAL CHOICE 31

We note that there are examples of prior year with delays of greater than one year, find- behavioral research that provide indications that ing that regions involved with reward evalua- hypothetical prizes can be treated differently tion, among others, differentiated these two cat- than real prizes, especially at short delays egories. Moreover, using real but nonimmediate (Camerer & Hogarth, 1999; Smith & Walker, rewards (i.e., those delivered after the partici- 1993), though there are also examples where no pant leaves the laboratory session) risks incur- difference was found (Baker, Johnson, & ring transaction costs that can influence subjec- Bickel, 2003; Johnson & Bickel, 2002; Lagorio tive value and that are not present in animal & Madden, 2005; Madden, Begotka, Raiff, & research. Note that, in contrast, McClure and Kastern, 2003; Madden et al., 2004). Random colleagues have used both monetary rewards single-trial payments have also been shown to with delays of weeks (McClure, Laibson, Loe- evoke different discount rates than cumulative wenstein, & Cohen, 2004), and juice rewards payment methods (Kagel & Roth, 1997). Most with delays of minutes (McClure et al., 2007), neuroeconomic studies of risky choice, for com- and report relatively similar patterns of activa- parison, now use a standard payment scheme: tion across the two studies. choices involve real monetary rewards in the The variability in models, payment methods, range of a few tens of dollars (or equivalent in and delay intervals used in intertemporal choice other currency) and pay participants for a subset studies has surely contributed to differences in of their choices, selected randomly; see (Clith- their results. Yet, it is important to note that the ero, Carter, & Huettel, 2009; Hsu, Bhatt, Ado- use of a standardized set of conditions poses sig- lphs, Tranel, & Camerer, 2005; Huettel, Stowe, nificant limitations. For example, when studying Gordon, Warner, & Platt, 2006) for examples. impulsivity in choice, the promise of an abstract No such convergence exists for research on intertem- prize later may not elicit the desired behavior. poral choice. We return to the motivation behind this Utilizing substantive prizes that are large enough diversity in the summary paragraph below. to elicit a tradeoff at longer delays may also raise ethical concerns. Nevertheless, consistency in Delay Intervals methods may often improve connections to other research without compromising the purpose of the Similarly, there has been little consistency in study. temporal interval across studies. The majority of neuroimaging studies included in this review Summary of Neuroimaging Studies: (12 studies) use delays in the range of weeks to ALE Analysis years, with the remainder using much shorter delays of seconds to minutes for rewards ob- In spite of the array of methods catalogued tained during the experiment itself (McClure et above, studies of delay discounting show sub- al., 2007) and for symbolic rewards that are stantial overlap. In order to provide a coarse accumulated (Gregorios-Pippas, Tobler, & representation of the regions of the brain con- Schultz, 2009; Luhmann, Chun, Yi, Lee, & sistently involved in studies of delay discount- Wang, 2008; Tanaka et al., 2004). While the ing, we performed activation likelihood estima- former range is generally compatible with that tion (ALE) using GingerALE (Laird et al., used in behavioral economics research, it differs 2005). We included activation peaks from each dramatically from that of the large animal liter- of the studies listed in bold in Table 1. Three ature on delay discounting, which normally uses studies were excluded because they used ROI- delays on the order of several seconds (Hayden based analyses (Bjork, Momenan, & Hommer, & Platt, 2007; Kalenscher et al., 2005; Kalen- in press; Clithero et al., 2009; Gregorios-Pippas scher et al., 2004; Rosati, Stevens, & Hauser, et al., 2009). One study was excluded because 2006; Stevens, Hallinan, & Hauser, 2005; its coordinates were reported in an unconven- Stevens, Rosati, Ross, & Hauser, 2005). These tional space (Hoffman et al., 2008). Analysis of very different time scales may involve different 378 foci in the remaining 13 studies us- neural substrates (Lane, Cherek, Pietras, & Tch- ing 10,000 permutations (10 mm FWHM) re- eremissine, 2003). As an example, a study by turned 25 significant clusters (see Table 2), rep- Wittmann and colleagues (2007) contrasted de- resenting regions that were more likely to be lays (for hypothetical rewards) of less than one activated during tasks that involve delay dis- 32 CARTER, MEYER, AND HUETTEL

Table 2 Results of ALE Analysis Cluster peak (TAL) Cluster Volume, mm3 xyzNetwork Medial prefrontal cortex 9,888 Ϫ2 40 18 Both Ventral striatum 9,256 14 10 0 VAL Posterior cingulate cortex 1,976 Ϫ8 Ϫ36 36 VAL Right insula (R) 1,824 36 18 Ϫ2 VAL Temporal parietal cortex (L) 1,776 Ϫ48 Ϫ66 14 CN Splenial posterior cingulate 1,736 Ϫ2 Ϫ50 20 CN Left insula (L) 1,328 Ϫ30 18 Ϫ6 VAL Left inferior PFC (L) 1,016 Ϫ50 0 12 Neither Left inferior PFC (L) 1,000 Ϫ42 4 30 Neither Left temporal pole (L) 800 Ϫ46 4 Ϫ24 Neither Posterior parietal cortex (L) 688 Ϫ28 Ϫ60 44 Neither Posterior parietal cortex 656 26 Ϫ60 46 Neither Occipital cortex 576 Ϫ12 Ϫ88 10 Neither Inferior parietal lobule 520 Ϫ50 Ϫ52 24 CN Occipital cortex 384 12 Ϫ82 14 Neither Left superior frontal gyrus 368 Ϫ18 24 42 Neither Somatosensory cortex 344 30 Ϫ42 50 Neither Lateral orbitofrontal cortex 336 40 46 Ϫ12 VAL Inferior parietal lobule 248 56 Ϫ42 26 Neither Right anterior inferior PFC 232 48 34 10 CN Right anterior inferior PFC 216 46 48 6 CN Middle temporal gyrus 208 Ϫ64 Ϫ46 4 CN Left posterior insula 184 Ϫ30 0 6 Neither Somatosensory cortex 168 Ϫ44 Ϫ32 46 Neither Midbrain 160 6 Ϫ22 Ϫ8 VAL Note. Significant clusters from an anatomical likelihood analysis of 13 delay discounting fMRI studies. VAL indicates this region is normally part of the value network. CN indicates this region is normally part of the core network. PFC ϭ prefrontal cortex; L ϭ left; R ϭ right. counting (see Figure 1). Consistent with delay the methodologies discussed above, these two discounting functioning as a complex decision, networks could be reliably extracted. Table 2 shows areas associated with a wide In summary, the most obvious feature of the range of processes. Figure 1 shows regions in- neuroimaging literature on intertemporal choice cluded in a network of areas known to be sen- is its methodological variability. Nevertheless, sitive to value (ventral striatum, medial prefron- two sorts of research questions have recurred tal cortex, orbitofrontal cortex, anterior insula; throughout this sample. First, there has been Gottfried, O’Doherty, & Dolan, 2003; Knutson, consistent interest in the neural mechanisms Adams, Fong, & Hommer, 2001; Montague & underlying the construction of value for tem- Berns, 2002; O’Doherty, Deichmann, Critchley, porally delayed prizes, both in how it relates &Dolan,2002)orsubjectivevalue(PCC; to specific functional forms and whether dis- Kable & Glimcher, 2007), shown colored in tinct brain systems are engaged by temporal red. Regions shaded in blue are often associated delay. Second, intertemporal choice has been with a “core network” (Spreng, Mar, & Kim, set forth as a prototypic example of how 2009) that supports prospective processes like cognitive control can override impulsive, autobiographical memory, theory of mind, and reward-seeking choice (cf., Kahneman & planning for the future. These included inferior Tversky, 1979), with concomitant interest in prefrontal cortex, medial prefrontal cortex, tem- the contributions of neural control systems for poral-parietal cortex, and peri-splenial posterior selection of delayed outcomes. We discuss cingulate. Although the list of contrasts in- how neuroimaging data can bear on these cluded in the ALE analysis were as diverse as questions in the following sections. Finally, SPECIAL ISSUE: NEUROIMAGING OF INTERTEMPORAL CHOICE 33

Figure 1. ALE Analysis of Delay Discounting fMRI Studies. Shown in (A) are clusters identified as associated with intertemporal choice, based on an activation likelihood estimate (ALE) meta-analysis of 13 delay discounting studies (see text for details). Regions often associated with a value network are shaded in red, while regions associated with a core network for self-directed thought are shaded in blue. Within the posterior cingulate cortex (B), there is evidence for a division between a more anterior and dorsal region of posterior cingulate cortex (PCC) that is associated with value and a retrosplenial region associated with self-directed and prospective thought (C). Shown in boxes are centroids of PCC activation from seven relevant studies: 1 Ballard & Knutson, 2009; 2 Harrington et al., 2004; 3 Kable & Glimcher, 2007; 4 Luhmann et al., 2008; 5 Spreng et al., 2009; 6 Weber, & Huettel, 2008; 7 Wittmann et al., 2007. Vstr—ventral striatum, MPFC – medial prefrontal cortex, PCC – posterior cingulate cortex. we consider the implications of the current underlying neural mechanisms. Samuelson’s literature for methodology and theoretical initial model described delay discounting using grounding of future studies. Considered an exponential decay function (see Equation 1), broadly, nearly all existing studies use para- an elegant and simple solution that utilized the digms drawn from the behavioral economics math common in interest calculation at the time. literature, such that decisions are regarding Yet, numerous violations of this model are ev- midsized to large monetary rewards obtain- ident (Prelec & Loewenstein, 1991; Thaler, able over a period of weeks to years. Yet, 1981), some noted by Samuelson at the time of within this broad category there exist numer- the original work. Collectively, these violations ous differences that make comparison across imply a function with a much steeper discount- studies challenging: few studies have adopted ing over short time scales. Functional neuroim- such similar procedures that methodological aging studies have sought brain regions whose differences can be ruled out as explanations activation tracks specific parameters of these func- of their different results. We will return to tions and, therefore, potentially contribute to val- the implications of this methodological vari- uation. Because different studies have adopted ability for future research at the end of this rather different functions for explaining their ob- review. served behavior, there have been areas of dis- agreement within the literature that point to How Is Value Constructed for Temporally clear avenues for future experimentation. Here, Delayed Prizes? we present two proposed models of intertempo- ral choice, quasi-hyperbolic and hyperbolic dis- Anomalies in human intertemporal choice counting, using studies from McClure and col- behavior suggest potential constraints for the leagues (2004) and Kable and Glimcher (2007). 34 CARTER, MEYER, AND HUETTEL

These studies differ in many aspects of their interpreted to provide evidence for two conclu- methods; for example, they differ in the amount sions: generally, that the value of a delayed of training participants receive. Even so, they option depends on the interaction between two provide a unique opportunity to contrast the distinct sets of neural systems and, specifically, neural substrates postulated to support distinct that impulsive choices reflect an overactivation models of intertemporal choice. of the reward system (compared to control mechanisms). Moreover, the basic results were Quasi-Hyperbolic Discounting replicated by the same group in a 2007 paper (McClure et al., 2007) that used liquid rewards In an early fMRI study of intertemporal (juice and water). In this latter study, partici- choice, published in 2004 by McClure and col- pants were offered a choice between a small leagues (McClure et al., 2004), participants amount of a liquid reward now, and an amount chose between pairs of real monetary rewards as much as three times larger after a maximum that differed in their value (over a range of a few delay of 20 minutes. Utilizing a beta/delta tens of dollars) and delay before delivery (over model, they find reward circuitry activation for a range of a few weeks). To guide their analy- choices involving an immediate reward and lat- ses, the authors adopted a two-process frame- eral prefrontal and parietal activation for trials work that models discounting using a quasi- comparing delay prizes. They also find no dif- hyperbolic function for subjective value (U), for ference between striatal BOLD responses to a prize of amount A, at a given delay D, as prizes delayed 10 minutes and those delayed an U ϭ A␤␦D (Equation 2; see Laibson, 1997; amount comparable to the first experiment. Loewenstein, 1996; Shefrin & Thaler, 1988). These results have been interpreted as strong Note that this perspective matches well to the neural evidence in favor of a two-component pervasive dual-systems account of behavioral model of intertemporal choice. control (Bechara, 2005; Ernst & Paulus, 2005; Kahneman, 2003; Loewenstein, 1996). Their Hyperbolic Discounting analyses sought brain regions whose activation was consistent with each of two model param- An alternative perspective was advanced by eters. The first parameter, ␤, is also referred to Kable and Glimcher (2007) who used experi- as the “hot” or visceral system. It reflects a mental procedures largely similar to those of strong preference for immediate rewards when McClure and colleagues (McClure et al., 2004) compared to any delay. Through a contrast be- but began with a different framework: they hy- tween decisions involving an immediately pothesized that neural systems for valuation available reward and those involving only de- reflected a single hyperbolic discounting func- layed rewards, they found that regions normally tion that operated regardless of delay interval associated with value representation, such as the A U ϭ (Equation 3, U ϭ subjective value, ventral striatum, medial prefrontal cortex, me- 1 ϩ kD dial orbitofrontal cortex, and posterior cingu- A ϭ prize amount, D ϭ delay magnitude, k ϭ late, increased with the presence of an immedi- discounting parameter; Laibson, 1997; Mazur, ate reward. The second parameter, ␦, is also 1987; Strotz, 1956). For each participant, they referred to as the “cold” or rational system. It measured discounting preferences based on in- produces a shallow discounting rate (and thus dependent behavioral sessions (and subse- reduced discriminability between reward val- quently matched to data from the fMRI ses- ues) at longer delays; within their analyses, sion). Their key analyses compared distinct delta-related regions were defined based on functional forms to determine which best having similar activation levels regardless of matched activation in brain regions associated intertemporal delay. This approach revealed with valuation. They found that subjective activation within dorsolateral prefrontal cor- value, as estimated for each participant using tex, parietal cortex, lateral orbitofrontal cor- asingle-parameterhyperbolicdecayfunction, tex, as well as regions associated with motor was very well matched to activation in ventral and visual processing. striatum, ventromedial prefrontal cortex, and The results of McClure and colleagues have posterior cingulate cortex. They also demon- been highly influential, in that they have been strated that the hyperbolic function fit their SPECIAL ISSUE: NEUROIMAGING OF INTERTEMPORAL CHOICE 35 fMRI data better than both a single-compo- relatively short delays (Wittmann et al., 2007), nent decay function as well as a more shallow would be consistent with either model. A po- decay function similar to the delta component tential strategy would be to focus model fit tests of the beta-delta model. Control analyses ex- to only the specific range of values where strong cluded potential confounding factors ranging from discrepancies are predicted. absolute amount of reward to delay until reward Second, the observed activation in value- delivery. relevant brain areas describes a point in the Favoring the simplest interpretation of these construction of an intertemporal choice, not the results, Kable and Glimcher concluded that the entire mechanism. Other processes, such as an value of a delayed outcome was represented in impulsive choice system, could have upstream canonical reward-related brain regions, whose or downstream influences on a subjective-value activation was largely consistent with a stan- representation. As summarized by Tesch and dard hyperbolic discounting model. (See Glim- Sanfey (2008), a range of behavioral effects can cher, Kable, & Louie, 2007, for additional dis- modulate subjective value in intertemporal cussion by the same authors about the potential choice: a prize magnitude effect, greater dis- importance of these results for economic mod- counting for smaller rewards (Raineri & Rach- eling.) Hyperbolic discounting functions have lin, 1993; Thaler, 1981); an effect of the number been shown to have a number of advantages of choices, more choices increases discounting over other models of discounting (Kirby & (Read, 2001); an order effect, increasing se- Marakovic, 1995). The conclusion that compu- quences are valued more than decreasing se- tation of the value of a delayed outcome relies quences (Loewenstein & Sicherman, 1991); a on similar mechanisms as other rewards is con- preference for speeding up rather than slowing sistent with work by Hariri and colleagues payment schedules (Loewenstein, 1988); a pref- (Hariri et al., 2006), which shows that reward erence for distributed rewards (Loewenstein & sensitivity in the ventral striatum scales with a Prelec, 1993); an effect of reward type, food is participant’s hyperbolic discounting coefficient. discounted more strongly than money (Odum & The results of Kable and Glimcher provide Rainaud, 2003); and an effect of reward va- compelling evidence that delay valuation relies lence, gains are more steeply discounted than on the same neural mechanisms as do other types of valuation. Nevertheless, two main ca- losses (Thaler, 1981). In addition, discounting veats are in order. First, analyses of this type preferences can be modified by distraction need extensive and well-distributed data to (Mischel, 1972). There are also cases where the make fine distinctions between functional forms mental state of the participant preceding the (Berns, Laibson, & Loewenstein, 2007). Acqui- choice modulates discounting (summarized in sition of such data requires a sampling process Berns et al., 2007; Montague & King-Casas, that grows very rapidly with small increases in 2007). Discounting rates are increased by depri- coverage and resolution of the prize parameter vation of self control (Baumeister & Heather- space. Even comprehensive analyses like that of ton, 1996), when emotionally or sexually Kable and Glimcher (Kable & Glimcher, 2007) aroused (described in Loewenstein, 1996), or only imperfectly sample the full parameter when smokers are deprived of nicotine (Field, space: they have relatively few participants and, Santarcangelo, Sumnall, Goudie, & Cole, 2006; thus, limited examples of discounting parame- Mitchell, 2004). Schweighofer and colleagues ters, and they present only a limited set of (2006) propose a more general conclusion that stimuli to define distribution biases. For exam- the participant’s choices reflect the optimal dis- ple, changes in the initial delay to prize have not counting function for the task. Collectively, been described in the literature (but see Glim- these phenomena make it difficult to argue for a cher et al., 2007). Moreover, different functions single hyperbolic discounting function utilized do not lead to drastically different predictions in the brain. Very recent data also pose difficul- for the majority of intertemporal choice scenar- ties for the simple conclusion that hyperbolic ios, reflecting the general similarity between the discounting behavior reflects hyperbolic activa- hyperbolic and quasi-hyperbolic functions. Be- tion of the reward system. Gregorios-Pippas, cause of this imprecision, results from other Tobler, and Schultz (2009) found that an expo- studies, such as increased striatal activation to nential decay function was a slightly better 36 CARTER, MEYER, AND HUETTEL model of activity in the ventral striatum during leading to the common currency map can be delay discounting than a hyperbolic function. consistent across them. Given these and similar concerns and the results described in the previ- Are There Neural Mechanisms Specific to ous section, the cardinal outstanding question Intertemporal Choice? for neuroimaging studies of intertemporal choice is not “How is subjective value repre- In biological systems, selection pressures can sented?” but “Are there aspects of subjective lead to coopting of existing mechanisms for value computation and comparison that are spe- new purposes. The use of a single mechanism cific to intertemporal choice?” for valuation would provide significant advan- We note that prior behavioral studies provide tages (Montague & Berns, 2002; Sugrue, Cor- equivocal answers to this latter question. Some rado, & Newsome, 2004), most notably by theorists have noted that all forms of reward providing a common currency for comparisons discounting, whether intertemporal or probabi- between different forms of reward (Klein, listic, may share aspects of calculation and com- Deaner, & Platt, 2008). As described above, parison (Cardinal, 2006; Green & Myerson, there exists substantial evidence that brain re- 2004; Prelec & Loewenstein, 1991). As exam- gions associated with the evaluation of imme- ples, rewards become less valuable when asso- diately obtained primary and secondary rewards ciated with risk or ambiguity, as can be modeled (Delgado, Nystrom, Fissell, Noll, & Fiez, 2000; using simple decay functions (Von Neumann & Knutson et al., 2001; Schultz, Dayan, & Morgenstern, 1944). One perspective is that de- Montague, 1997), both sure and probabilistic lay discounting involves estimation of risk: (Kuhnen & Knutson, 2005), are modulated by each time point leading up to delivery of a prize aspects of rewards obtainable after some delay, can be modeled with a constant hazard; there- including both the relative immediacy of those fore, more temporally distant prizes are less rewards (McClure et al., 2007; McClure et al., likely to be obtained. Probabilistic prizes, con- 2004; Wittmann et al., 2007) and the subjective versely, could be modeled according to their value of those rewards (Kable & Glimcher, estimated delay until delivery: lower probability 2007). One natural conclusion from these find- prizes require more attempts and, therefore, a ings is that intertemporal choice merely reflects longer delay before they are obtained. More- one application of a general-purpose decision- over, some anomalies are common across prob- making mechanism. In this section, we evaluate ability and delay discounting models (Prelec & an alternative (and more nuanced) perspective: Loewenstein, 1991); probabilistically certain that distinct brain regions play important roles outcomes are overvalued (Allais, 1953), just in intertemporal choice, compared to other like immediately available outcomes. There are forms of decision making. notable exceptions, however. Increasing magni- Explaining intertemporal choice using only tude of reward usually has opposite effects on mechanisms for reward evaluation would pose probability and delay: large rewards reduce the several challenges. First, a strict common- immediacy effect but increase the certainty ef- currency explanation does not readily account fect (Chapman & Weber, 2006; Prelec & Loe- for modality-specific manipulations of value, wenstein, 1991; Rachlin, Brown, & Cross, despite examples of delay-specific framing ef- 2000; Weber & Chapman, 2005). In addition, fects as introduced above (Berns et al., 2007; relationships between an individual’s risk pref- Tesch & Sanfey, 2008). Second, that a common erences and delay preferences have been found currency exists at relatively late stages of pro- in some cases (Crean, de Wit, & Richards, cessing – which seems tautological given individ- 2000; Myerson, Green, Hanson, Holt, & Estle, uals’ ability to compare distinct rewards – does 2003; Reynolds, Karraker, Horn, & Richards, not preclude initially separate computations of 2003) but not in others (Ohmura, Takahashi, & value that are then combined at a later stage for Kitamura, 2005; Reynolds, Richards, Horn, & comparison purposes. Rewards that are delayed Karraker, 2004). in time are likely to be represented and evalu- Three recent fMRI studies have combined ated, at least initially, through different mecha- intertemporal and risky choice within the same nisms. Prize types differ in enough aspects of paradigm. Weber and Huettel examined differ- their representation that not all neural substrates ences in the neural substrates of risk and delay SPECIAL ISSUE: NEUROIMAGING OF INTERTEMPORAL CHOICE 37 valuation (Weber & Huettel, 2008) by asking results of Weber and Huettel (2008). Also participants to choose between pairs of mone- consistent with a role for the posterior cingu- tary outcomes differing in their delay or their late in delay prize valuation, Wittmann and risk. (Note that all choices were within- colleagues (2007) note a correlation between modality; participants never compared a de- posterior cingulate activity and a participant’s layed but certain outcome to a probabilistic but choice of the delayed option. These studies immediate outcome.) Their results indicated provide strong evidence that activity in the that there was some differentiation in the re- posterior cingulate is related to delayed, but gions of the brain supporting each type of val- not risky, prize valuation. uation. Decision making involving risk pro- If activity in the posterior cingulate is specif- duced greater activation, compared to delay and ically relevant to the valuation of temporally control tasks, in a number of regions, including delayed prizes, what components of delay prize the posterior parietal cortex, the dorsolateral valuation could be carried out there? One pos- prefrontal cortex, the anterior insula, the ante- sibility is that activity in the posterior cingulate rior cingulate cortex, and the orbital frontal cor- correlates with the subjective value of a delayed tex. These regions largely overlapped with reward but not the subjective value of a risk those reported in previous studies of risky reward. In fact, Kable and Glimcher found ev- choice (Dickhaut et al., 2003; Hsu et al., 2005; idence of posterior cingulate activity that corre- Huettel et al., 2006; Kuhnen & Knutson, 2005; lates with subjective prize value (Kable & Paulus, Rogalsky, Simmons, Feinstein, & Stein, Glimcher, 2007). A second and even more in- 2003; Rogers et al., 1999). Decision making triguing possibility is that the posterior cingu- involving delay evoked greater activation, com- late lies earlier in the delay prize valuation pared to risk and control tasks, in the posterior pathway, computing the primitives of delayed cingulate cortex. A similar regional differentia- prize valuation. If this is true, we would then tion was observed in a second study from the expect activity in the posterior cingulate to cor- same laboratory that focused specifically on val- relate with the temporal component of the delay uation of delayed or probabilistic rewards. Us- prize or the objective delay prize magnitude, but ing a pattern-classification analysis derived not subjective value. Though Kable and Glim- from machine learning, Clithero, Carter, and cher demonstrated correlations with subjective Huettel (2009) found that patterns of activation value in the posterior cingulate, activation in an within voxels in posterior parietal cortex (and, adjacent and possibly distinct region may cor- secondarily, within posterior cingulate cortex) relate with its more primitive components (see uniquely predicted whether a participant was Figure 1B and C). valuing a delay or risk prize, even when con- Evidence of objective prize magnitude and trolling for the overall activation level within a temporal delay sensitivity was presented in a brain region. very recent paper by Ballard and colleagues In a third study examining potential differ- (Ballard & Knutson, 2009). Dissociating tem- ences between risk and delay discounting, Lu- poral delay and objective prize magnitude, they hman and colleagues (2008) asked participants show that the peri-splenial aspect of the PCC to choose between a small prize available im- responds to the displayed value of the prize (the mediately, with certainty, and a larger prize objective value; Ballard & Knutson, 2009). available either with some risk or with both risk They also found evidence of temporal sensitiv- and a temporal delay. The probability that a ity in the peri-splenial aspect of the posterior prize would be received was represented using cingulate. Specifically, increases in this region’s a set of one to nine squares, each of which had responses to temporal delay were negatively a constant hazard of 0.1. In the delay condition, correlated with the participants’ individual dis- the same squares indicated both the risk previ- count parameters. More evidence of temporal ously described and a delay of 1 second per sensitivity in the peri-splenial posterior cingu- square. Luhman and colleagues found that ac- late was presented in a study by Ersner- tivity in posterior cingulate correlated with the Hershfield and colleagues (2008). Their partic- addition of the delay condition, an aspect of the ipants viewed a series of adjectives, evaluating risk and delay valuation process that was not how relevant some were to their current self and common to both. This finding agrees with the how relevant others were to a future (envi- 38 CARTER, MEYER, AND HUETTEL sioned) self. The authors found that a similar that a normative exponential model cannot ac- region in the posterior cingulate is more active count for real-world discounting behavior (see when considering a person in the present than Ballard & Knutson, 2009; Gregorios-Pippas et when considering a person in the future. A al., 2009; Schweighofer et al., 2006, for possi- similar portion of the posterior cingulate is also ble exceptions), there exists no consensus about implicated in the encoding of time intervals its replacement. Hyperbolic models of discount- (Harrington et al., 2004), see Figure 1C. Given ing are both simple and well-fit to fMRI data in these commonalities, future work will need to at least some experiments (Glimcher et al., better incorporate cognitive neuroscience re- 2007; Kable & Glimcher, 2007). However, hy- search into time representation (Buhusi & perbolic models leave significant portions of Meck, 2005). behavior unexplained (such as the framing and Adjacent representation of subjective value context effects described above). And, other and delay prize valuation primitives provides a researchers have found little or no difference convenient location for the construction of between how well distinct functions fit fMRI delayed prize value primitives and their inter- data (Ballard & Knutson, 2009; Gregorios- action with a broader system for common cur- Pippas et al., 2009) or have found that the best rency. This intriguing possibility provides a tar- model depends on the experimental conditions get for future studies seeking to identify the (Schweighofer et al., 2006). Dual-system mod- mechanism of interaction between delay prize els match intuitive descriptions of intertemporal representation and subjective value. choice: Everyone would prefer to have the larger prize sooner but most exercise the nec- essary restraint to make what they believe to be Unanswered Questions and Future a smarter choice, often choosing the delayed Directions prize. This self restraint has limits; it is not an unlimited resource, and there are many exam- Building on the structure provided by a rich ples (see the previous sections) where discount- behavioral literature, neuroimaging data could, ing can be altered by depleting self restraint in principle, not only provide concrete descrip- before a choice. These counterpoints do not tions of abstract model features but also even make the models mutually exclusive. It is dif- provide a means for adjudicating between sim- ficult to imagine a neural mechanism that would ilar models. Indeed, the series of innovative predict framing effects specific to delay dis- studies described in this review have collec- counting that does not also provide a delay- tively made significant progress toward this specific representation of value. A likely possi- goal. Regions of the brain that show significant bility would be that there are secondary influ- correlations with aspects of intertemporal ences on early and unique aspects of the delayed choice, such as the striatum, posterior cingu- prize value calculation. This value could then be late cortex, insula, and medial prefrontal cor- passed on to a common currency system where tex, provide a solid basis for further elabora- different value types are compared. Given that tion. Even so, the core questions identified at anomalies of delayed prize valuation remain the outset of our review – and targeted by the unexplained, more complex models seem nec- studies we describe – remain only imperfectly essary. A number of promising candidates answered. We conclude by identifying three await testing (e.g., Fudenberg & Levine, outstanding challenges for future research on 2006; Kalenscher & Pennartz, 2008; Loewen- intertemporal choice, along with some sugges- stein et al., 2003; O’Donoghue & Rabin, tions for methodological improvements to meet 2000; Ok & Masatlioglu, 2007). those challenges. The diverse set of methods used in fMRI studies of intertemporal choice has not only Distinguishing Models and Their provided a wealth of potential conclusions but Constituent Processes also strengthened (through converging data) those findings that have been consistent across What steps should researchers take to better studies. Yet, specific replications of those find- test the distinguishing features of competing ings are necessary. An approach that makes models? While there is now general acceptance great strides toward reconciling studies whose SPECIAL ISSUE: NEUROIMAGING OF INTERTEMPORAL CHOICE 39 results differ is a direct comparison of the fit of that some animals possess cognitive capacities the alternate economic models to neural data, that are potentially homologous to the mecha- see Kable and Glimcher (2007) as well as Gre- nisms of delay discounting. As examples, apes gorios-Pippas, Tobler and Schultz (2009) and show long-term planning by retaining tools for Ballard and Knutson (2009) for examples. In use 14 hours later (Mulcahy & Call, 2006) but order to facilitate comparisons across studies, have not yet demonstrated that they can reject we recommend that, for most research ques- an immediate prize for a larger one that will tions, future experiments should use the most occur more than a few minutes later (Chimps common parameters and design methods: delay 123s; Rosati, Stevens, Hare, & Hauser, 2007). periods in the range of weeks; real payments, Corvids and many other animals cache food for through a subset of the trials completed; vari- much later use (Raby, Alexis, Dickinson, & ance of both the immediate and future prize Clayton, 2007), albeit in a largely stereotyped amount; and variance in the order and place- format. In addition, intertemporal choice is ment of prizes presented. We do, however, strongly affected by the loss of portions of the again note that there are constraints that may prefrontal cortex (Bechara, Tranel, & Damasio, require deviations from these guidelines (e.g., 2000), which may have very different func- for juice rewards, a delay of weeks would be tional properties and connectivity in human and unreasonable). nonhuman animals. Directly relevant to this Finally, we note that functional neuroimaging point, the relative volume of human prefrontal data are not, in themselves, sufficient for distin- cortex is correlated with smaller discounting guishing models of intertemporal choice. At rates (Bjork et al., 2009). There is a disconnect root, the empirical data needed to establish between this result and those described in the whether discounting functions follow a hyper- animal literature (Cardinal, Pennicott, Sugatha- bolic, quasi-hyperbolic, or some other function pala, Robbins, & Everitt, 2001; Izawa, Zachar, will come from rigorous behavioral testing. Yanagihara, & Matsushima, 2003; Winstanley, What functional neuroimaging data can pro- Theobald, Cardinal, & Robbins, 2004; but see vide, however, is the ability to evaluate what Kheramin et al., 2003). Given the difficulty in classes of models are biologically plausible and, demonstrating long-term delay discounting in thus, are good candidates for future behavioral animal models, there are likely to be significant testing (see Clithero, Tankersley, & Huettel, differences between the neural substrates of hu- 2008, for additional discussion). As an example, man intertemporal choice behavior and that Delgado and colleagues have demonstrated that studied in model organisms. An important a neurobiologically based model can motivate bridging approach will be assessing choices the design of auctions in a study of overbidding over delays similar to that present in animal behavior (Delgado, Schotter, Ozbay, & Phelps, models (Luhmann et al., 2008; McClure et al., 2008). Neuroimaging (or other neuroscience) 2007). As an intriguing example, Gregorios- data could also provide needed constraints for Pippas, Tobler, and Schultz (2009) used a any model: identifying conditions under which Pavlovian paradigm to assess the value of their parameters might change, predicting the rewards received after delays of, at maxi- effects of additional manipulations (e.g., time mum, 13.5 seconds. They find activity in the pressure, depression), and providing insight into ventral striatum is modified by delay and is individual differences (e.g., effects of aging). slightly better fit by an exponential decay than ahyperbolicfunction. Improving the Connection to Another study that provides a point of com- Animal Research parison for animal models comes from Tanaka and colleagues (Tanaka et al., 2004), who used Many of the most successful areas of neuro- a state-dependent Markov-chain task to look for science (e.g., vision science, memory) have in- the neural correlates of delayed prize choice. volved close interaction between human and Unlike the majority of studies included in this animal research. While no studies have hereto- review, these authors looked for neural differ- fore demonstrated intertemporal choice in ani- ences evoked by a participant’s choice of an mals at delays similar to typical human experi- immediate reward compared to a choice that ments (i.e., days or longer), research suggests required several small losses prior to winning a 40 CARTER, MEYER, AND HUETTEL larger prize (i.e., effort/action-related delay, events. These component processes, in many rather than forced waits). This method incorpo- cases, are better described within the psycho- rates temporal delay according to the number of logical and cognitive neuroscience literatures losses before the prize rather than requiring the than within economic theory. participant to focus on the delay period, which Consider the evaluation of delay, itself. To would increase discounting (Mischel, 1972). explain both temporal discounting and the va- Tanaka and colleagues found ventral-dorsal riety of framing effects described earlier, there gradients in the striatum and insula according to must be both representations of time as well as the duration of temporal delay in the prize being mechanisms for those representations to alter chosen. They hypothesized that this result re- subjective value. Describing these effects will flects multiple cortico-basal ganglia networks require the incorporation of a larger body of that each represent value at a different delay. At behavioral and neuroscience research on time first consideration, it seems difficult to compare perception and its underlying mechanisms this study with the other studies listed in Ta- (Buhusi & Meck, 2005). While a full consider- ble 1. However, because they use a temporal ation of time processing is well beyond the difference (Sutton & Barto, 1998) formula- scope of this review, there are recent and ten- tion of delay discounting, there are potential tative first steps toward integrating delay dis- connections to a large body of literature in- counting with the cognitive neuroscience of volving learned associations and studies on timing and prospection. As noted in a previous decision making. Specifically, delay discount- section, Ersner-Hershfield and colleagues ing is related to recent neuroscience work (2008) asked participants to rate the relevance positing an actor/critic model of decision of adjectives to themselves as well as a future making (O’Doherty et al., 2004; Paulus, Fein- version of themselves. They found that delay stein, Tapert, & Liu, 2004; Preuschoff, Bos- discounting rates correlated with the amount of saerts, & Quartz, 2006; Seymour et al., 2004; activity in the rostral anterior cingulate when Sutton & Barto, 1998). There is also a signifi- evaluating adjectives for themselves versus cant literature on temporal difference learning evaluating the relevance of adjectives for their in animal models (Barto, 1995; Bayer & future selves. They also find an effect of time in Glimcher, 2005; Lau & Glimcher, 2008; Mor- the posterior cingulate, noted previously. Incor- ris, Nevet, Arkadir, Vaadia, & Bergman, 2006; porating methods of studying time from the Samejima, Ueda, Doya, & Kimura, 2005; broader cognitive neuroscience literature holds Schultz et al., 1997; Seo, Barraclough, & Lee, promise for differentiating components of the 2007; Seo & Lee, 2007), providing an addi- broad network of activations associated with tional method for better connecting their work delay discounting studies. to animal research. Work by Schultz (Schultz et Moreover, a growing body of literature im- al., 1997) has shown that the dopamine trans- plicates specific clinical disorders (e.g., addic- mitter system is an excellent candidate for im- tion) that shape how people evaluate rewards plementing these proposed learning models, available immediately and after delays. While leading to well motivated hypotheses this review focused on nonclinical research as a (Sagvolden, Johansen, Aase, & Russell, 2005; way of constraining the scope of our survey, Tripp & Wickens, 2008) that could explain future research should incorporate clinical con- pathological failures of intertemporal choice siderations when trying to understand the mech- (e.g., in attention deficit/hyperactivity disorder, anisms of intertemporal choice. Specifically, ef- ADHD). fects like symmetric past and future discounting (Bickel, Yi, Kowal, & Gatchalian, 2008), the Building Links to the Larger Cognitive hidden zero effect (Magen, Dweck, & Gross, Neuroscience Literature 2008), and changes in discounting with drug state history (e.g., Giordano et al., 2002), all The behavioral models of delay discounting suggest novel pieces for the intertemporal described here require a complex set of compo- choice puzzle. nent processes: representations of value and Neuroeconomic research typically has the time, prospective thought and planning for the virtue of precision in modeling and rigor in future, and methods for comparing disparate methods; research typically involves a large SPECIAL ISSUE: NEUROIMAGING OF INTERTEMPORAL CHOICE 41 number of participants and involves thoroughly Bechara, A. (2005). Decision making, impulse con- discussed hypotheses. As we mentioned earlier, trol and loss of willpower to resist drugs: A neu- this makes economic phenomena excellent tar- rocognitive perspective. Nature Neuroscience, 8, gets for neuroimaging studies that can identify 1458–1463. components of models and provide neurobio- Bechara, A., Tranel, D., & Damasio, H. (2000). Char- logical constraints for further testing. The one acterization of the decision-making deficit of pa- tients with ventromedial prefrontal cortex lesions. potential pitfall of this sort of exchange lies in Brain, 123, 2189–2202. the use of reverse inference (Poldrack, 2006). Benzion, U., Rapoport, A., & Yagil, J. (1989). Dis- Although intertemporal choice involves regions count rates inferred from decisions: An experimen- associated both with valuation and with core tal study. Management Science, 35, 270–284. prospective and self-directed thought, the spe- Berns, G. S., Chappelow, J., Cekic, M., Zink, C. F., cific computations supported by these regions Pagnoni, G., & Martin-Skurski, M. E. (2006). Neu- remain unknown. Thus, experimental designs robiological substrates of dread. Science, 312, should also extend beyond simple economic 754–758. paradigms to consider additional forms of inter- Berns, G. S., Laibson, D., & Loewenstein, G. (2007). temporal choice to improve both the specificity Intertemporal choice: Toward an integrative and generalizability of these results. Despite framework. Trends in Cognitive Sciences, 11, substantial progress over the past few years, 482–488. there remain many unexplored avenues for fu- Bickel, W. K., & Marsch, L. A. (2001). Toward a ture research. behavioral economic understanding of drug depen- dence: Delay discounting processes. Addiction (Abingdon, England), 96, 73–86. Bickel, W. K., Yi, R., Kowal, B. P., & Gatchalian, References K. M. (2008). Cigarette smokers discount past and future rewards symmetrically and more than con- Ainslie, G. W. (1974). Impulse control in pigeons. trols: Is discounting a measure of impulsivity? Journal of the Experimental Analysis of Behavior, Drug and Alcohol Dependence, 96, 256–262. 21(3), 485–489. Bjork, J. M., Momenan, R., & Hommer, D. W. (in Allais, M. (1953). Le comportement de l’homme press). Delay discounting correlates with propor- rationnel devant le risque: Critique des postulats et tional lateral frontal cortex volumes. Biological axiomes de l’Ecole Americaine. Econometrica, 21, Psychiatry. 503–546. Buhusi, C. V., & Meck, W. H. (2005). What makes Angeletos, G. M., Laibson, D., Repetto, A., Tobac- us tick? Functional and neural mechanisms of in- man, J., & Weinberg, S. (2001). The hyperbolic terval timing. Nature Reviews Neuroscience, 6, consumption model: Calibration, simulation, and 755–765. empirical evaluation. Journal of Economic Per- Camerer, C. F., & Hogarth, R. M. (1999). The effects spectives, 15, 47–68. Baker, F., Johnson, M. W., & Bickel, W. K. (2003). of financial incentives in experiments: A review Delay discounting in current and never-before cig- and capital-labor-production framework. Journal arette smokers: Similarities and differences across of Risk and Uncertainty, 19, 7–42. commodity, sign, and magnitude. Journal of Ab- Cardinal, R. N. (2006). Neural systems implicated in normal Psychology, 112, 382–392. delayed and probabilistic reinforcement. Neural Ballard, K., & Knutson, B. (2009). Dissociable neu- Networks, 19, 1277–1301. ral representations of future reward magnitude and Cardinal, R. N., Pennicott, D. R., Sugathapala, C. L., delay during temporal discounting. NeuroIm- Robbins, T. W., & Everitt, B. J. (2001). Impulsive age, 45, 143–150. choice induced in rats by lesions of the nucleus Barto, A. G. (1995). Adaptive critics and the basal accumbens core. Science, 292, 2499–2501. ganglia. In J. C. Houk, J. Davis, & D. Beiser Chapman, G. B., & Weber, B. J. (2006). Decision (Eds.), Models of information processing in the biases in intertemporal choice and choice under basal ganglia (pp. 215–232). Cambridge, MA: uncertainty: Testing a common account. Memory MIT Press. & Cognition, 34, 589–602. Baumeister, R. F., & Heatherton, T. F. (1996). Self- Clithero, J. A., Carter, R. M., & Huettel, S. A. (2009). regulation failure: An overview. Psychological In- Local pattern classification differentiates processes quiry, 7, 1–15. of economic valuation. NeuroImage, 45, 1329– Bayer, H. M., & Glimcher, P. W. (2005). Midbrain 1338. dopamine neurons encode a quantitative reward Clithero, J. A., Tankersley, D., & Huettel, S. A. prediction error signal. Neuron, 47, 129–141. (2008). Foundations of neuroeconomics: From 42 CARTER, MEYER, AND HUETTEL

philosophy to practice. Plos Biology, 6, 2348– just as soon as possible? American Economic Re- 2353. view, 97, 142–147. Crean, J. P., de Wit, H., & Richards, J. B. (2000). Gottfried, J. A., O’Doherty, J., & Dolan, R. J. (2003). Reward discounting as a measure of impulsive Encoding predictive reward value in human amyg- behavior in a psychiatric outpatient population. dala and orbitofrontal cortex. Science, 301, 1104– Experimental and Clinical Psychopharmacol- 1107. ogy, 8, 155–162. Green, L., & Myerson, J. (1996). Exponential versus Critchfield, T. S., & Kollins, S. H. (2001). Temporal hyperbolic discounting of delayed outcomes: Risk discounting: Basic research and the analysis of and waiting time. American Zoologist, 36, 496– socially important behavior. Journal of Applied 505. Behavior Analysis, 34, 101–122. Green, L., & Myerson, J. (2004). A discounting Deaner, R. O., Khera, A. V., & Platt, M. L. (2005). framework for choice with delayed and probabi- Monkeys pay per view: Adaptive valuation of so- listic rewards. Psychological Bulletin, 130, 769– cial images by rhesus macaques. Current Biology, 792. 15(6), 543–548. Gregorios-Pippas, L., Tobler, P. N., & Schultz, W. Delgado, M. R., Nystrom, L. E., Fissell, C., Noll, (2009). Short-term temporal discounting of reward D. C., & Fiez, J. A. (2000). Tracking the hemody- value in human ventral striatum. Journal of Neu- namic responses to reward and punishment in the rophysiology, 101, 1507–1523. striatum. Journal of Neurophysiology, 84, 3072– Hariri, A. R., Brown, S. M., Williamson, D. E., Flory, 3077. J. D., de Wit, H., & Manuck, S. B. (2006). Pref- Delgado, M. R., Schotter, A., Ozbay, E. Y., & erence for immediate over delayed rewards is as- Phelps, E. A. (2008). Understanding overbidding: sociated with magnitude of ventral striatal activity. Using the neural circuitry of reward to design Journal of Neuroscience, 26, 13213–13217. economic auctions. Science, 321, 1849–1852. Harrington, D. L., Boyd, L. A., Mayer, A. R., Shel- Dickhaut, J., McCabe, K., Nagode, J. C., Rustichini, traw, D. M., Lee, R. R., Huang, M. X., & Rao, A., Smith, K., & Pardo, J. V. (2003). The impact of S. M. (2004). Neural representation of interval the certainty context on the process of choice. encoding and decision making. Cognitive Brain Proceedings of the National Academy of Sciences, Research, 21, 193–205. USA, 100, 3536–3541. Hayden, B. Y., & Platt, M. L. (2007). Temporal Ernst, M., & Paulus, M. P. (2005). Neurobiology of discounting predicts risk sensitivity in rhesus ma- decision making: A selective review from a neu- caques. Current Biology, 17, 49–53. rocognitive and clinical perspective. Biological Psychiatry, 58, 597–604. Hoffman, W., Schwartz, D., Huckans, M., McFarland, Ersner-Hershfield, H., Wimmer, G. E., & Knutson, B., Meiri, G., Stevens, A., & Mitchell, S. H. B. (2008). Saving for the future self: Neural mea- (2008). Cortical activation during delay discount- sures of future self-continuity predict temporal dis- ing in abstinent methamphetamine dependent in- counting. Social Cognitive and Affective Neuro- dividuals. Psychopharmacology, 201, 183–193. science, 4, 85–92. Hsu, M., Bhatt, M., Adolphs, R., Tranel, D., & Cam- Field, M., Santarcangelo, M., Sumnall, H., Goudie, erer, C. F. (2005). Neural systems responding to A., & Cole, J. (2006). Delay discounting and the degrees of uncertainty in human decision-making. behavioural economics of cigarette purchases in Science, 310, 1680–1683. smokers: The effects of nicotine deprivation. Psy- Huettel, S. A., Stowe, C. J., Gordon, E. M., Warner, chopharmacology, 186, 255–263. B. T., & Platt, M. L. (2006). Neural signatures of Frederick, S., Loewenstein, G., & O’Donoghue, T. economic preferences for risk and ambiguity. Neu- (2002). Time discounting and time preference: A ron, 49, 765–775. critical review. Journal of Economic Litera- Izawa, E. I., Zachar, G., Yanagihara, S., & Matsus- ture, 40, 351–401. hima, T. (2003). Localized lesion of caudal part of Fudenberg, D., & Levine, D. K. (2006). A dual-self lobus parolfactorius caused impulsive choice in the model of impulse control. American Economic Re- domestic chick: Evolutionarily conserved function view, 96, 1449–1476. of ventral striatum. Journal of Neuroscience, 23, Giordano, L. A., Bickel, W. K., Loewenstein, G., 1894–1902. Jacobs, E. A., Marsch, L., & Badger, G. J. (2002). Johnson, M. W., & Bickel, W. K. (2002). Within- Mild opioid deprivation increases the degree that subject comparison of real and hypothetical money opioid-dependent outpatients discount delayed rewards in delay discounting. Journal of the Ex- heroin and money. Psychopharmacology (Berl), perimental Analysis of Behavior, 77, 129–146. 163, 174–182. Kable, J. W., & Glimcher, P. W. (2007). The neural Glimcher, P. W., Kable, J., & Louie, K. (2007). correlates of subjective value during intertemporal Neuroeconomic studies of impulsivity: Now or choice. Nature Neuroscience, 10, 1625–1633. SPECIAL ISSUE: NEUROIMAGING OF INTERTEMPORAL CHOICE 43

Kagel, J. H., & Roth, A. E. (1997). The handbook of Laird, A. R., Fox, P. M., Price, C. J., Glahn, D. C., experimental economics. Princeton, NJ: Princeton Uecker, A. M., Lancaster, J. L., . . . Fox, P. T. University Press. (2005). ALE meta-analysis: Controlling the false Kahneman, D. (2003). Maps of bounded rationality: discovery rate and performing statistical contrasts. Psychology for behavioral economics. American Human Brain Mapping, 25, 155–164. Economic Review, 93, 1449–1475. Lane, S. D., Cherek, D. R., Pietras, C. J., & Tcher- Kahneman, D., & Tversky, A. (1979). Prospect the- emissine, O. V. (2003). Measurement of delay ory: An analysis of decision under risk. Economet- discounting using trial-by-trial consequences. Be- rica, 47, 263–291. havioural Processes, 64, 287–303. Kalenscher, T., & Pennartz, C. M. A. (2008). Is a bird Lau, B., & Glimcher, P. W. (2008). Value represen- in the hand worth two in the future? The neuro- tations in the primate striatum during matching economics of intertemporal decision-making. behavior. Neuron, 58, 451–463. Progress in Neurobiology, 84, 284–315. Loewenstein, G. (1996). Out of control: Visceral Kalenscher, T., Windmann, S., Diekamp, B., Rose, influences on behavior. Organizational Behavior J., Gunturkun, O., & Colombo, M. (2005). Single and Human Decision Processes, 65, 272–292. units in the pigeon brain integrate reward amount Loewenstein, G., O’Donoghue, T., & Rabin, M. and time-to-reward inan impulsive choice task. (2003). Projection bias in predicting future utility. Current Biology, 15, 594–602. Quarterly Journal of Economics, 118, 1209–1248. Kalenscher, T., Windmann, S., Gunturkun, O., Rose, Loewenstein, G., & Prelec, D. (1992). Anomalies in J., Diekamp, B., & Colombo, M. (2004). Single intertemporal choice: Evidence and an interpreta- neurons code delay- and amount-dependent sub- tion. Quarterly Journal of Economics, 107, 573– jective reward value and relative reward prefer- 597. ence during impulsive decision making. Journal of Loewenstein, G., & Sicherman, N. (1991). Do work- Psychophysiology, 18, 202–202. ers prefer increasing wage profiles. Journal of La- Kheramin, S., Body, S., Ho, M. Y., Velazquez- bor Economics, 9, 67–84. Martinez, D. N., Bradshaw, C. M., Szabadi, E., . . . Loewenstein, G. F. (1988). Frames of mind in inter- Anderson, I. M. (2003). Role of the orbital pre- temporal choice. Management Science, 34, 200– frontal cortex in choice between delayed and un- 214. certain reinforcers: A quantitative analysis. Behav- Loewenstein, G. F., & Prelec, D. (1993). Preferences ioural Processes, 64, 239–250. for sequences of outcomes. Psychological Review, Kirby, K. N., & Marakovic, N. N. (1995). Modeling 100, 91–108. myopic decisions: Evidence for hyperbolic delay- Luhmann, C. C., Chun, M. M., Yi, D.-J., Lee, D., & discounting within-subjects and amounts. Organi- Wang, X.-J. (2008). Neural Dissociation of Delay zational Behavior and Human Decision Pro- and Uncertainty in Intertemporal Choice. Journal cesses, 64, 22–30. of Neuroscience, 28, 14459–14466. Klein, J. T., Deaner, R. O., & Platt, M. L. (2008). Madden, G. J., Begotka, A. M., Raiff, B. R., & Neural correlates of social target value in macaque Kastern, L. L. (2003). Delay discounting of real parietal cortex. Current Biology, 18, 419–424. and hypothetical rewards. Experimental and Clin- Knutson, B., Adams, C. M., Fong, G. W., & Hom- ical Psychopharmacology, 11, 139–145. mer, D. (2001). Anticipation of increasing mone- Madden, G. J., Raiff, B. R., Lagorio, C. H., Begotka, tary reward selectively recruits nucleus accum- A. M., Mueller, A. M., Hehli, D. J., & Begotka, bens. Journal of Neuroscience, 21, 1–5. A. M. (2004). Delay discounting of potentially real Kobayashi, S., & Schultz, W. (2008). Influence of and hypothetical rewards: II. Between- and within- reward delays on responses of dopamine neurons. subject comparisons. Experimental and Clinical Journal of Neuroscience, 28, 7837–7846. Psychopharmacology, 12, 251–261. Kuhnen, C. M., & Knutson, B. (2005). The neural Magen, E., Dweck, C. S., & Gross, J. J. (2008). The basis of financial risk taking. Neuron, 47, 763–770. hidden-zero effect: Representing a single choice as Lagorio, C. H., & Madden, G. J. (2005). Delay dis- an extended sequence reduces impulsive choice. counting of real and hypothetical rewards III: Psychological Science, 19, 648–649. Steady-state assessments, forced-choice trials, and Mazur, J. E. (1987). An adjusting procedure for all real rewards. Behavioural Processes, 69, 173– studying delayed reinforcement. In M. L. Com- 187. mons, J. E. Mazur, J. A. Nevin, & H. Rachlin Laibson, D. (1997). Golden eggs and hyperbolic dis- (Eds.), Quantitative Analysis of Behavior (Vol. 5, counting. Quarterly Journal of Economics, 112, pp. 55–73). Hillsdale, NJ: Erlbaum. 443–477. McClure, S. M., Ericson, K. M., Laibson, D. I., Laibson, D. (1998). Life-cycle consumption and hy- Loewenstein, G., & Cohen, J. D. (2007). Time perbolic discount functions. European Economic discounting for primary rewards. Journal of Neu- Review, 42, 861–871. roscience, 27, 5796–5804. 44 CARTER, MEYER, AND HUETTEL

McClure, S. M., Laibson, D. I., Loewenstein, G., & Paulus, M. P., Rogalsky, C., Simmons, A., Feinstein, Cohen, J. D. (2004). Separate neural systems value J. S., & Stein, M. B. (2003). Increased activation in immediate and delayed monetary rewards. Sci- the right insula during risk-taking decision making ence, 306, 503–507. is related to harm avoidance and neuroticism. Neu- Mischel, T. (1972). Explanation in Behavioural Sci- roImage, 19, 1439–1448. ences–Borger, R. and Cioffi, F. Contemporary Perry, J. L., & Carroll, M. E. (2008). The role of Psychology, 17, 51–52. impulsive behavior in drug abuse. Psychopharma- Mischel, W., Shoda, Y., & Peake, P. K. (1988). The cology, 200, 1–26. nature of adolescent competencies predicted by Phelps, E. S. (1968). Myopia and inconsistency in preschool delay of gratification. Journal of Per- dynamic utility maximization. The Review of Eco- sonality and Social Psychology, 54, 687–696. nomic Studies, 35(185). Mitchell, S. H. (2004). Effects of short-term nicotine Plichta, M. M., Vasic, N., Wolf, R. C., Lesch, K.-P., deprivation on decision-making: Delay, uncer- Brummer, D., Jacob, C., Fallgatter, A. J., & Gron, tainty and effort discounting. Nicotine & Tobacco G. (2009). Neural hyporesponsiveness and hyper- Research, 6, 819–828. responsiveness during immediate and delayed re- Montague, P. R., & Berns, G. S. (2002). Neural ward processing in adult attention-deficit/hyperac- economics and the Biological substrates of valua- tivity disorder. Biological Psychiatry, 65, 7–14. tion. Neuron, 36, 265–284. Poldrack, R. A. (2006). Can cognitive processes be Montague, P. R., & King-Casas, B. (2007). Efficient inferred from neuroimaging data? Trends in Cog- statistics, common currencies and the problem of nitive Sciences, 10, 59–63. reward-harvesting. Trends in Cognitive Sci- Prelec, D., & Loewenstein, G. (1991). Decision- ences, 11, 514–519. making over time and under uncertainty: A com- Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & mon approach. Management Science, 37, 770– Bergman, H. (2006). Midbrain dopamine neurons 786. encode decisions for future action. Nature Neuro- Preuschoff, K., Bossaerts, P., & Quartz, S. R. (2006). science, 9, 1057–1063. Neural differentiation of expected reward and risk Mulcahy, N. J., & Call, J. (2006). Apes save tools for in human subcortical structures. Neuron, 51, 381– future use. Science, 312, 1038–1040. 390. Myerson, J., Green, L., Hanson, J. S., Holt, D. D., & Raby, C. R., Alexis, D. M., Dickinson, A., & Clay- Estle, S. J. (2003). Discounting delayed and prob- ton, N. S. (2007). Planning for the future by west- abilistic rewards: Processes and traits. Journal of ern scrub-jays. Nature, 445, 919–921. Economic Psychology, 24, 619–635. Rachlin, H., Brown, J., & Cross, D. (2000). Discount- O’Doherty, J., Dayan, P., Schultz, J., Deichmann, R., ing in judgments of delay and probability. Journal Friston, K., & Dolan, R. J. (2004). Dissociable of Behavioral Decision Making, 13, 145–159. roles of ventral and dorsal striatum in instrumental Raineri, A., & Rachlin, H. (1993). The effect of conditioning. Science, 304, 452–454. temporal constraints on the value of money and O’Doherty, J. P., Deichmann, R., Critchley, H. D., & other commodities. Journal of Behavioral Deci- Dolan, R. J. (2002). Neural responses during an- sion Making, 6, 77–94. ticipation of a primary taste reward. Neuron, 33, Read, D. (2001). Is time-discounting hyperbolic or 815–826. subadditive? Journal of Risk and Uncertainty, 23, O’Donoghue, T., & Rabin, M. (2000). The econom- 5–32. ics of immediate gratification. Journal of Behav- Reynolds, B., Karraker, K., Horn, K., & Richards, ioral Decision Making, 13, 233–250. J. B. (2003). Delay and probability discounting as Odum, A. L., & Rainaud, C. P. (2003). Discounting related to different stages of adolescent smoking of delayed hypothetical money, alcohol, and food. and non-smoking. Behavioural Processes, 64, Behavioural Processes, 64, 305–313. 333–344. Ohmura, Y., Takahashi, T., & Kitamura, N. (2005). Reynolds, B., Richards, J. B., Horn, K., & Karraker, Discounting delayed and probabilistic monetary K. (2004). Delay discounting and probability dis- gains and losses by smokers of cigarettes. Psycho- counting as related to cigarette smoking status in pharmacology, 182, 508–515. adults. Behavioural Processes, 65, 35–42. Ok, E. A., & Masatlioglu, Y. (2007). A theory of Richards, J. B., Mitchell, S. H., Wit, H. D., & Seiden, (relative) discounting. Journal of Economic The- L. S. (1997). Determination of discount functions ory, 137, 214–245. in rats with an adjusting-amount procedure. Jour- Paulus, M. P., Feinstein, J. S., Tapert, S. F., & Liu, nal of the Experimental Analysis of Behavior, 67, T. T. (2004). Trend detection via temporal differ- 353–366. ence model predicts inferior prefrontal cortex ac- Rogers, R. D., Owen, A. M., Middleton, H. C., Wil- tivation during acquisition of advantageous action liams, E. J., Pickard, J. D., Sahakian, B. J., & selection. NeuroImage, 21, 733–743. Robbins, T. W. (1999). Choosing between small, SPECIAL ISSUE: NEUROIMAGING OF INTERTEMPORAL CHOICE 45

likely rewards and large, unlikely rewards acti- Spreng, R. N., Mar, R. A., & Kim, A. S. (2009). The vates inferior and orbital prefrontal cortex. Journal common neural basis of autobiographical memory, of Neuroscience, 19, 9029–9038. prospection, navigation, theory of mind, and the Rosati, A. G., Stevens, J. R., Hare, B., & Hauser, default mode: A quantitative meta-analysis. Jour- M. D. (2007). The evolutionary origins of human nal of Cognitive Neuroscience, 21, 489–510. patience: Temporal preferences in chimpanzees, Stevens, J. R., Hallinan, E. V., & Hauser, M. D. (2005). bonobos, and human adults. Current Biology, 17, The ecology and evolution of patience in two New 1663–1668. World monkeys. Biology Letters, 1, 223–226. Rosati, A. G., Stevens, J. R., & Hauser, M. D. (2006). Stevens, J. R., Rosati, A. G., Ross, K. R., & Hauser, The effect of handling time on temporal discount- M. D. (2005). Will travel for food: Spatial dis- ing in two New World primates. Animal Behav- counting in two new world monkeys. Current Bi- iour, 71, 1379–1387. ology, 15, 1855–1860. Sagvolden, T., Johansen, E. B., Aase, H., & Russell, Strotz, R. H. (1956). Myopia and inconsistency in V. A. (2005). A dynamic developmental theory of dynamic utility maximization. Review of Eco- attention-deficit/hyperactivity disorder (ADHD) nomic Studies, 23, 165–180. predominantly hyperactive/impulsive and com- Sugrue, L. P., Corrado, G. S., & Newsome, W. T. bined subtypes. Behavioral and Brain Sci- (2004). Matching behavior and the representation ences, 28, 397–419. of value in the parietal cortex. Science, 304, 1782– Samejima, K., Ueda, Y., Doya, K., & Kimura, M. 1787. (2005). Representation of action-specific reward Sutton, R. S., & Barto, A. G. (1998). Reinforcement values in the striatum. Science, 310, 1337–1340. learning. Cambridge, MA: MIT Press. Samuelson, P. A. (1937). A note on measurement of Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Oka- utility. The Review of Economic Studies, 4, 155– moto, Y., & Yamawaki, S. (2004). Prediction of 161. immediate and future rewards differentially re- Schultz, W., Dayan, P., & Montague, P. R. (1997). A cruits cortico-basal ganglia loops. Nature Neuro- science, 7, 887–893. neural substrate of prediction and reward. Science, Tesch, A. D., & Sanfey, A. G. (2008). Models and 275, 1593–1599. methods in delay discounting. Annals of the New Schweighofer, N., Shishida, K., Han, C. E., Oka- York Academy of Sciences, 1128, 90–94. moto, Y., Tanaka, S. C., Yamawaki, S., . . . Doya, Thaler, R. (1981). Some empirical evidence on dy- K. (2006). Humans can adopt optimal discounting namic inconsistency. Economics Letters, 8, 201– strategy under real-time constraints. Plos Compu- 207. tational Biology, 2, 1349–1356. Tripp, G., & Wickens, J. R. (2008). Research review: Seo, H., Barraclough, D. J., & Lee, D. (2007). Dy- Dopamine transfer deficit: A neurobiological the- namic signals related to choices and outcomes in ory of altered reinforcement mechanisms in the dorslateral prefrontal cortex. Cerebral Cor- ADHD. Journal of Child Psychology and Psychi- tex, 17, I110–I117. atry and Allied Disciplines, 49, 691–704. Seo, H., & Lee, D. (2007). Temporal filtering of Von Neumann, J., & Morgenstern, O. (1944). Theory reward signals in the dorsal anterior cingulate cor- of games and economic behavior. Princeton, NJ: tex during a mixed-strategy game. Journal of Neu- Princeton University Press. roscience, 27, 8366–8377. Weber, B. J., & Chapman, G. B. (2005). Playing for Seymour, B., O’Doherty, J. P., Dayan, P., Koltzen- peanuts: Why is risk seeking more common for burg, M., Jones, A. K., Dolan, R. J., . . . Frackow- low-stakes gambles? Organizational Behavior and iak, R. S. (2004). Temporal difference models de- Human Decision Processes, 97, 31–46. scribe higher-order learning in humans. Nature, Weber, B. J., & Huettel, S. A. (2008). The neural 429, 664–667. substrates of probabilistic and intertemporal deci- Shamosh, N. A., DeYoung, C. G., Green, A. E., Reis, sion making. Brain Research, 1234, 104–115. D. L., Johnson, M. R., Conway, A. R. A., . . . Gray, Winstanley, C. A., Theobald, D. E. H., Cardinal, J. R. (2008). Individual differences in delay dis- R. N., & Robbins, T. W. (2004). Contrasting roles counting: Relation to intelligence, working mem- of basolateral amygdala and orbitofrontal cortex in ory, and anterior prefrontal cortex. Psychological impulsive choice. Journal of Neuroscience, 24, Science, 19, 904–911. 4718–4722. Shefrin, H. M., & Thaler, R. H. (1988). The behav- Wittmann, M., Leland, D. S., & Paulus, M. P. (2007). ioral life-cycle hypothesis. Economic Inquiry, 26, Time and decision making: Differential contribu- 609–643. tion of the posterior insular cortex and the striatum Smith, V. L., & Walker, J. M. (1993). Monetary during a delay discounting task. Experimentelle rewards and decision cost in experimental eco- Hirnforschung Expe´rimentation Ce´re´brale, 179, nomics. Economic Inquiry, 31, 245–261. 643–653.