Encoding and Decoding of Pain Relief in the Human Brain Suyi Zhang Department of Engineering University of Cambridge This dissertation is submitted for the degree of Doctor of Philosophy Peterhouse December 2018 Abstract Encoding and Decoding of Pain Relief in the Human Brain. Suyi Zhang The studies in this thesis explored how pain and its relief are represented in the human brain. Pain and relief are important survival signals that motivate escape from danger and search for safety, however, they are often evaluated by subjective descriptions only. Studying how humans learn and adapt to pain and relief allows objective investigation of the information processing and neural circuitry underlying these internal experiences. My research set out to use computational learning models to provide mechanistic expla- nations for the behavioural and functional neuroimaging data collected in pain/relief learning experiments with independent groups of healthy human participants. With a Pavlovian acute pain conditioning task in Experiment 1, I found that ‘associability’ (a form of uncertainty signal) had a crucial role in controlling the learning rates of different conditioned responses, and can be used to anatomically dissociate underlying neural systems. Experiment 2 focused on relief learning of terminating a tonic pain stimulus, in which the priority for relief-seeking is in conflict with the general suppression of cognition and attention. I showed that associability during active learning not only controls the relief learning rate, but also correlates with endogenously modulated (reduced) ongoing pain. This finding was confirmed in Experiment 3 using an independent active relief learning paradigm in a complex dynamic environment. Critically, both experiments showed that associa- bility was correlated with responses in the pregenual anterior cingulate cortex (pgACC), a brain region previously implicated in aspects of endogenous pain control related to attention and controllability. This provided a potential computational account of an information-sensitive endogenous analgesic mechanism. In Experiment 4, I explored the implications of endogenous controllability for technology- based pain therapeutics. I designed an adaptive closed-loop system that learned to control pain iv stimulation using decoded real-time pain representations from the brain. Subjects were shown to actively enhance the discriminability of pain only in the pgACC, and uncertainty during learning again correlated with endogenously modulated pain and were associated with pgACC responses. Together, these studies (i) show the importance of uncertainty in controlling learning during both acute and tonic pain, (ii) describe how uncertainty also flexibly modulates pain to maximise the impact of learning, (iii) illustrate a central role for the pgACC in this process, and (iv) reveal the implications for future technology-based therapeutic systems. Declaration I hereby declare that except where specific reference is made to the work of others, the contents of this dissertation are original and have not been submitted in whole or in part for consideration for any other degree or qualification in this, or any other university. This dissertation ismyown work and contains nothing which is the outcome of work done in collaboration with others, except as specified in the text and Acknowledgements. This dissertation contains fewer than 65,000 words including appendices, bibliography, footnotes, tables and equations and has fewer than 150 figures. All neuroimaging experiments were performed by myself, with contributions and assistance from collaborators and colleagues, at the Center for Information and Neural Networks, Japan, and the Advanced Telecommunications Research Institute, Japan. Suyi Zhang December 2018 Acknowledgements I would like to thank my supervisor Dr Ben Seymour. The work presented here would not exist without his invaluable support and guidance. I would also like to express my gratitude to all my colleagues and collaborators in Cambridge and Japan, especially the imaging teams at the Center for Information and Neural Networks and the Advanced Telecommunications Research Institute for their assistance in performing the studies. Thank you. Financial support was generously provided by the WD Armstrong Fund and the Cambridge Trust. Table of contents List of publications xi List of figures xiii List of tables xv Nomenclature xvii 1 Introduction1 1.1 Pain, relief, and motivation . .2 1.2 Learning through reinforcement . .8 1.3 The neurobiological basis of pain/relief motivation . 14 1.4 Clinical implications and therapeutic potentials . 22 1.5 Thesis structure . 23 2 Methods 25 2.1 Functional magnetic resonance imaging (fMRI) . 26 2.2 Physiological and behavioural measurements . 31 2.3 Pain stimulation . 33 2.4 Computational modelling . 34 3 Experiment 1: Dissociable learning processes underlie pain conditioning 39 3.1 Introduction . 40 3.2 Methods . 40 3.3 Results . 45 3.4 Discussion . 51 3.5 Tables . 54 3.6 Supplementary figures . 55 x Table of contents 4 Experiment 2: Comparing active and passive relief learning 59 4.1 Introduction . 60 4.2 Methods . 61 4.3 Results . 72 4.4 Discussion . 78 4.5 Tables . 81 4.6 Supplementary figures . 84 5 Experiment 3: Active relief learning in a dynamic environment 87 5.1 Introduction . 88 5.2 Methods . 89 5.3 Results . 95 5.4 Discussion . 99 5.5 Tables . 103 5.6 Supplementary figures . 106 6 Experiment 4: Endogenous controllability of brain-machine interfaces for pain 109 6.1 Introduction . 110 6.2 Methods . 111 6.3 Results . 120 6.4 Discussion . 130 6.5 Tables . 133 7 Discussion 135 7.1 Contributions . 135 7.2 Future work . 144 References 149 List of publications Research articles • Zhang S, Mano H, Ganesh G, Robbins T, and Seymour B. Dissociable Learning Processes Underlie Human Pain Conditioning. Current Biology 26.1 (2016), 52-58. (Experiment 1) • Zhang S, Mano H, Lee M, Yoshida W, Robbins T, Kawato M, and Seymour B. The Control of Tonic Pain by Active Relief Learning. eLife 7 (2018), e31949. (Experiment 2 and 3) • Zhang S, Mano H, Yoshida W, Yanagisawa T, Shibata K, Kawato M, and Seymour B. Endogenous Controllability of Brain-Machine Interfaces for Pain. bioRxiv (2018), 369736. (Experiment 4) Review article • Zhang S and Seymour B. Technology for Chronic Pain. Current Biology 24.18 (2014), R930-R935. Collaboration • Mano H, Yoshida W, Shibata K, Zhang S, Koltzenburg M, Kawato M, and Seymour B. Thermosensory Perceptual Learning Is Associated with Structural Brain Changes in Parietal-Opercular (SII) Cortex. Journal of Neuroscience 37.39 (2017), 9380-9388. List of figures 1.1 Introduction: Motivation and affective valence. .4 1.2 Introduction: The neural pathways of pain and pain relief. 20 2.1 Methods summary. 37 3.1 Experiment 1: Task paradigm and example trial. 41 3.2 Experiment 1: Behavioural results. 48 3.3 Experiment 1: Neuroimaging results. 50 3.4 Experiment 1 Supplementary: Recording and stimulating apparatus placement. 55 3.5 Experiment 1 Supplementary: Heart rate, early/late trial SCRs, ROI betas. 57 4.1 Experiment 2: Task paradigm, contingency, and tonic pain/relief stimulation. 62 4.2 Experiment 2: Behavioural results. 74 4.3 Experiment 2: Neuroimaging results. 77 4.4 Experiment 2 Supplementary: Skin conductance raw data. 84 4.5 Experiment 2 Supplementary: Model protected exceedance probability. 85 5.1 Experiment 3: Task paradigm, unstable relief probability traces. 90 5.2 Experiment 3: Behavioural results. 98 5.3 Experiment 3: Neuroimaging results. 100 5.4 Experiment 3 Supplementary: Skin conductance raw data. 106 5.5 Experiment 3 Supplementary: Model protected exceedance probability. 107 5.6 Experiment 3 Supplementary: Overlaying relief learning clusters. 108 6.1 Experiment 4: Task paradigm, rated trial example, and insula ROI. 112 6.2 Experiment 4: Behavioural results. 123 6.3 Experiment 4: Whole brain comparison between days results. 124 6.4 Experiment 4: Decoder comparison and searchlight analysis results. 125 6.5 Experiment 4: Switch trials differences. 127 6.6 Experiment 4: Frequency learning evidence. 129 List of tables 3.1 Experiment 1: Model comparison. 54 3.2 Experiment 1: Neuroimaging ROI analysis. 54 4.1 Experiment 2: Rating details. 81 4.2 Experiment 2: Model fitting evidence and winning models. 81 4.3 Experiment 2: Model fitting results. 82 4.4 Experiment 2: Neuroimaging ROI analysis. 83 5.1 Experiment 3: Rating details. 103 5.2 Experiment 3: Model fitting evidence and winning models. 103 5.3 Experiment 3: Model fitting results. 104 5.4 Experiment 3: Neuroimaging ROI analysis. 105 6.1 Experiment 4: Decoder testing performance. 121 6.2 Experiment 4: Neuroimaging ROI analysis. 134 Nomenclature Roman Symbols A Action H Entropy Q Value (action / state-action) R Reinforcement / Outcome S State T Time (trial) or State transition function V Value (state) Greek Symbols a Associability / Learning rate d Prediction error e Explore probability (e-greedy rule) g Discount factor p Policy t Temperature (softmax rule) Acronyms / Abbreviations A-O Action-Outcome AAL Automated anatomical labelling (fMRI atlas) ACC Anterior cingulate cortex BIC Bayesian Information Criterion BLA Basolateral component of amygdala BMI Brain-machine interface BMS Bayesian model selection BOLD Blood oxygenation level dependent CeA Central nucleus of amygdala CHEPS Contact heat-evoked potential stimulator CR Conditional response CS Conditional stimulus xviii Nomenclature DBS Deep brain stimulation DLPFC Dorsolateral prefrontal cortex EMG