Kalenscher Pennartz 2008.Pdf
Total Page:16
File Type:pdf, Size:1020Kb
Progress in Neurobiology 84 (2008) 284–315 www.elsevier.com/locate/pneurobio Is a bird in the hand worth two in the future? The neuroeconomics of intertemporal decision-making Tobias Kalenscher *, Cyriel M.A. Pennartz Animal Physiology and Cognitive Neuroscience, Swammerdam Institute for Life Sciences, Faculty of Science, University of Amsterdam, Kruislaan 320, 1098 SM Amsterdam, The Netherlands Received 3 May 2007; received in revised form 28 November 2007; accepted 29 November 2007 Abstract When making intertemporal decisions, i.e., decisions between outcomes occurring at different instants in time, humans and animals prefer rewards with short-term availability over rewards that become available in the long run. Discounted utility theory (DUT) is an influential normative model for intertemporal decisions that attempts to capture preference over time. It prescribes which is the best decision to takewith respect to consistent, coherent and optimal choice. Over the last few decades, DUT’s descriptive validity has been critically challenged. Empirical studies found systematic violations of several of DUT’s assumptions, including time-consistency of preferences, stationarity,constant discounting and utility maximisation. To account for theseanomalies,alternativemodelshavebeendevisedinvariousacademicdisciplines,includingeconomics,psychology,biology,philosophy,andmost lately, cognitive neuroscience. This article reviews the most recent literature on the behavioural and neural processes underlying intertemporal choices, and elucidates to which extent these findings can be used to explainviolations of DUT’s assumptions. In the first three sections, DUT is introduced, and behavioural anomalies are discussed. The fourth part focuses on the neuroscience of intertemporal choice, including its functional neuroanatomy, attempts to find a discounted value signal in the brain, and recent efforts to identify neural mechanisms producing time-inconsistencies. In the last section, the computational literature on neural learning mechanisms is reviewed. Then, a new, biologically plausible computational model of intertemporal choice is proposed that is able to explain many of the behavioural anomalies. The implications of these results help to understand why humansandanimalsfrequentlydecideirrationallyandtotheirlong-termeconomicdisadvantage,andwhichneuralmechanismsunderlysuchdecisions. # 2007 Elsevier Ltd. All rights reserved. Keywords: Neuroeconomics; Reward; Decision-making; Intertemporal decisions; Delay discounting; Risk; Impulsivity; Self-control; Dopamine; Glutamate; Temporal difference learning; Neural replay Contents 1. Time and probability . .......................................................................... 285 1.1. Decisions under risk and intertemporal decisions . ................................................. 285 1.2. Probabilistic and dated outcomes . ............................................................. 286 2. Rational intertemporal decisions . ................................................................. 286 2.1. (Ir)rational decisions . ..................................................................... 286 2.2. Expected utility theory as a normative framework for decision-making under risk ............................ 286 2.3. Discounted utility theory as a normative framework for intertemporal decisions . ............................ 287 3. Irrational intertemporal decisions: anomalies in intertemporal choice . ........................................ 288 3.1. Violation of the stationarity axiom ............................................................. 288 3.2. Violation of the assumption of constant discounting ................................................. 289 Abbreviations: ACC, anterior cingulate cortex; BLA, basolateral amygdala; BOLD, blood-oxygen-level dependent; cAMP, cyclic adenosine monophosphate; CS, conditioned stimulus; DLPFC, dorsolateral prefrontal cortex; EUT, expected utility theory; HSAT, Hebbian synapses with adaptive thresholds model; LTD, long- term depression; LTP, long-term potentiation; mPFC, medial prefrontal cortex; NAc, nucleus accumbens; NCL, nidopallium caudolaterale; NMDA, n-methyl D- aspartate; OFC, orbitofrontal cortex; PFC, prefrontal cortex; PSTH, peri-stimulus time histogram; RPM, reinforcement processing module; STN, subthalamic nucleus; TD, temporal difference; VMPFC, ventromedial prefrontal cortex. * Corresponding author. Tel.: +31 20 525 7637; fax: +31 20 525 7709. E-mail address: [email protected] (T. Kalenscher). 0301-0082/$ – see front matter # 2007 Elsevier Ltd. All rights reserved. doi:10.1016/j.pneurobio.2007.11.004 T. Kalenscher, C.M.A. Pennartz / Progress in Neurobiology 84 (2008) 284–315 285 3.3. Violation of the assumption of utility maximisation . ............................................... 290 3.4. Gains and losses in intertemporal decision-making . ............................................... 291 3.5. Further anomalies. ........................................................................ 291 3.6. Alternative theories........................................................................ 291 3.7. Summary. ............................................................................ 292 4. The neuroscience of intertemporal choices . ........................................................... 293 4.1. Intertemporal decision-making—a challenge for the cognitive neurosciences . .............................. 293 4.2. Neural representation of the decision variables ‘reward delay’ and ‘reward amount’. .......................... 293 4.3. Neuroanatomy of intertemporal decisions . ....................................................... 294 4.4. Neural integration of ‘reward delay’ and ‘reward amount’. .......................................... 295 4.5. Is the discounted reward value represented on a single-cell or population level? . .......................... 296 4.6. Utility from anticipation—the value of passage of time in the brain ...................................... 297 4.7. A single valuation mechanism or multiple choice systems? . .......................................... 298 4.8. Evidence for multiple decision networks in the brain . ............................................... 298 4.9. Challenging the multiple systems hypothesis . ................................................... 299 5. Towards a neurocomputational model of intertemporal choice behaviour . ...................................... 302 5.1. Delay discounting and neurocomputational modelling. ............................................... 302 5.2. Temporal difference learning . ............................................................... 302 5.3. Neural implementations of a discounting function: TD model .......................................... 304 5.4. Neural implementations: climbing-firing profile . ................................................... 305 5.5. Neural implementations: eligibility trace . ....................................................... 305 5.6. Neural implementations: reverse replay. ....................................................... 307 5.7. A model of intertemporal choices based on reverse replay and Hebbian learning . .......................... 308 6. Summary and outlook . ........................................................................ 309 Acknowledgements ............................................................................ 310 References . ................................................................................ 310 1. Time and probability decision-making, intertemporal choices, is illustrated by the pension fund example: when thinking about investing into 1.1. Decisions under risk and intertemporal decisions the retirement provision, you will face little risk or uncertainty, as the insurance company is reliable and will pay out the pension Your door bell rings, and when you open it, an insurance when you reach the age. However, your decision would involve salesman smiles into your face. He wants to sell you two calculating and trading-off choice outcomes that would be different insurance certificates. The first certificate is a fire realised at different points in time, i.e., you would have to invest insurance. It would cost you only $10 per month, and would money now to obtain benefits that are yet to come. On the other cover all the damage to your house in the unlikely, but hand, if you decided against the retirement provision, you could shattering case it was destroyed by a terrible fire. The second save the monthly $100 premium. This would put you in a better policy is a pension fund. The premium is $100 each month, but financial position to treat yourself to things that you fancy now, you would receive a guaranteed life-long retirement pay of at for example an expensive dinner each month. Hence, your choice least $700 per month once you reach the age of 65. depends on whether you are willing to forgo short-term benefits Those two examples nicely illustrate two problems in choice in order to invest into an option that is probably more reasonable theory that keep decision researchers from many different in the long run. Accordingly, intertemporal decision making can scientific disciplines busy, including psychology, biology, be defined as choices between outcomes that occur at different neuroscience, economics, law, politics and philosophy: deci- points in time. sion-making under risk and intertemporal decision-making. This review will focus on the neurobiology underlying the When you deliberate whether to