REINFORCERS AND CONTROL TOWARDS A COMPUTATIONAL ÆTIOLOGY OF DEPRESSION QUENTIN JM HUYS DISSERTATION SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY OF THE UNIVERSITY OF LONDON GATSBY COMPUTATIONAL NEUROSCIENCE UNIT UNIVERSITY COLLEGE LONDON DECLARATION I, Quentin Jan Marie Huys, confirm that the work pre- sented in this thesis is my own. Where information has been derived from other sources, I confirm that this has been indicated in the thesis. June 30, 2007 2 ABSTRACT Depression, like many psychiatric disorders, is a disorder of affect. Over the past decades, a large number of affective issues in depression have been characterised, both in human experiments and animal models of the dis- order. Over the same period, experimental neuroscience, helped by com- putational theories such as reinforcement learning, has provided detailed descriptions of the psychology and neurobiology of affective decision mak- ing. Here, we attempt to harvest the advances in the understanding of the brain’s normal dealings with rewards and punishments to dissect out and define more clearly the components that make up depression. We start by exploring changes to primary reinforcer sensitivity in the learned helpless- ness animal models of depression. Then, a detailed formalisation of control in a goal-directed decision making framework is presented and related to animal and human data. Finally, we show how serotonin’s joint involve- ment in reporting negative values and inhibiting actions may explain some aspects of its involvement in depression. Throughout, aspects of depres- sion are seen as emerging from normal affective function and reinforcement learning, and we thus conclude that computational descriptions of normal affective function provide one possible avenue by which to define an ætiol- ogy of depression. 3 CONTENTS Abstract .............................................. 3 Contents ............................................. 4 List of figures .......................................... 7 List of tables ........................................... 9 Preface .............................................. 10 1 Introduction 13 1.1 Depression ......................................... 13 1.2 Neurobiology of affective decision making ...................... 15 1.2.1 Dopamine and serotonin ............................ 17 1.3 Modelling ......................................... 18 1.4 Organisation of the thesis ................................ 19 2 Literature review: Affective decisions in depression 21 2.1 Background ........................................ 22 2.1.1 Epidemiology ................................... 22 2.1.2 Diagnosis ..................................... 23 2.1.3 Treatment ..................................... 25 2.1.4 Neuromodulators ................................. 25 2.2 Primary reinforcer sensitivity .............................. 25 2.2.1 Reward ....................................... 26 2.2.2 Punishment .................................... 27 2.2.3 Stress: cortisol levels ............................... 28 2.3 Pavlovian actions ..................................... 29 2.3.1 Serotonin in depression ............................. 29 2.4 Goal-directed decisions and control ........................... 31 2.4.1 Learned helplessness ............................... 32 2.4.2 Contingency judgements ............................ 34 2.4.3 Planning ...................................... 34 2.5 Habitual learning ..................................... 35 2.5.1 Appetitive habits ................................. 35 2.5.2 Aversive habits .................................. 37 2.6 Motivation ......................................... 39 2.6.1 Psychomotor retardation ............................ 39 2.6.2 Behavioural evidence .............................. 40 2.6.3 Emotional induction studies .......................... 40 2.6.4 Dopamine ..................................... 41 2.6.5 Motivation in depression ............................ 42 2.7 The depressive state: recapitulation of human evidence ............... 42 2.8 Human data on the ætiology of depression ...................... 43 2.8.1 Stress and genetics ................................ 43 2.8.2 Controllability ................................... 45 2.9 Animal models of depression .............................. 45 2.9.1 Validity ....................................... 45 4 2.9.2 Learned helplessness ............................... 46 2.9.3 Chronic mild stress ................................ 48 2.9.4 Neuromodulators ................................. 49 2.10 Induction: recapitulation ................................. 53 3 Blunting 54 3.1 Introduction ........................................ 54 3.2 Shock size in learned helplessness ........................... 56 3.2.1 Model definition ................................. 56 3.2.2 Learning ...................................... 58 3.2.3 Results ....................................... 59 3.3 Conditioned suppression ................................. 61 3.3.1 Model ....................................... 62 3.3.2 Results ....................................... 64 3.4 Generalisation ....................................... 66 3.4.1 Methods ...................................... 67 3.4.2 Results ....................................... 68 3.5 Discussion ......................................... 70 3.5.1 Modulation of pain sensitivity in animal experiments ............ 70 3.5.2 Serotonin ..................................... 71 3.5.3 Generalisation ................................... 71 3.5.4 Valence generalisation .............................. 73 3.5.5 Predictability ................................... 73 4 Control 74 4.1 Introduction ........................................ 74 4.2 Notions of control ..................................... 76 4.3 Results ........................................... 78 4.3.1 Outcome entropy ................................. 78 4.3.2 Fraction of controllable outcomes ........................ 80 4.3.3 Generalisation ................................... 81 4.3.4 Reinforcement-sensitive control ........................ 84 4.3.5 Animal models of depression .......................... 84 4.4 Discussion ......................................... 89 4.4.1 Normative model of control ........................... 89 4.4.2 Dopamine ..................................... 90 4.4.3 Symmetry between rewards and punishments ................ 91 4.4.4 Human data on control ............................. 91 4.4.5 Animal data on control ............................. 93 5 Pavlovian inhibition 95 5.1 Introduction ........................................ 95 5.2 Methods .......................................... 97 5.2.1 The model ..................................... 97 5.2.2 Serotonin ..................................... 98 5.2.3 Learning ...................................... 99 5.3 Results ........................................... 99 5.3.1 Behavioural inhibition .............................. 99 5.3.2 Serotonin depletion ................................ 99 5.3.3 Recall bias .....................................101 5.3.4 Reward seeking ..................................102 5.3.5 Impulsivity ....................................102 5.4 Discussion .........................................104 5.4.1 Behavioural Inhibition System .........................104 5.4.2 Tryptophan depletion ..............................105 5.4.3 Serotonin and dopamine .............................105 5 5.4.4 Depression .....................................106 6 General Conclusions 108 6.1 Contributions .......................................108 6.2 Limitations .........................................109 6.3 Future work ........................................110 6.4 Synthesis ..........................................110 Appendix 113 A Reinforcement learning and affective decisions 114 A.1 Reinforcement learning ..................................114 A.2 Value from tree search ..................................115 A.3 Model-free estimates of value ..............................117 A.4 Policies from values ....................................118 A.5 Decision tree ........................................118 B Statistical descriptions of control 120 B.1 Control as conditional entropy / outcome set size ..................120 B.2 Multiple actions with independent outcomes .....................123 B.2.1 Control as fraction of controllably attainable outcomes ........... 124 B.3 Control over desirable outcomes ............................127 B.4 Control variability across environments ........................129 C Notation and Abbreviations 131 Bibliography 133 6 LIST OF FIGURES 1.1 Basic decision making setup ............................... 15 2.1 Pavlovian inhibition mediated by serotonin ...................... 30 2.2 The learned helplessness paradigm ........................... 32 2.3 Asymmetrically reinforced discrimination task. .................... 36 2.4 Interaction of life stress and 5HTTLPR ......................... 44 3.1 Pain-induced analgesia .................................. 56 3.2 Model of LH paradigm .................................. 57 3.3 Model reversal latencies ................................. 59 3.4 Model Q values over time. ................................ 60 3.5 Inescapable shock effect on contingency ........................ 62 3.6 Model of the Jackson et al. (1978) experiment four .................. 63 3.7 Contingency results I ................................... 65 3.8 Contingency results II .................................. 65 3.9 Contingency results III