Journal of Mathematical Psychology 99 (2020) 102447 Contents lists available at ScienceDirect Journal of Mathematical Psychology journal homepage: www.elsevier.com/locate/jmp Review Active inference on discrete state-spaces: A synthesis ∗ Lancelot Da Costa a,b, , Thomas Parr b, Noor Sajid b, Sebastijan Veselic b, Victorita Neacsu b, Karl Friston b a Department of Mathematics, Imperial College London, London, SW7 2RH, United Kingdom b Wellcome Centre for Human Neuroimaging, University College London, London, WC1N 3AR, United Kingdom article info a b s t r a c t Article history: Active inference is a normative principle underwriting perception, action, planning, decision-making Received 17 April 2020 and learning in biological or artificial agents. From its inception, its associated process theory has Received in revised form 23 July 2020 grown to incorporate complex generative models, enabling simulation of a wide range of complex Accepted 3 September 2020 behaviours. Due to successive developments in active inference, it is often difficult to see how its Available online xxxx underlying principle relates to process theories and practical implementation. In this paper, we try to Keywords: bridge this gap by providing a complete mathematical synthesis of active inference on discrete state- Active inference space models. This technical summary provides an overview of the theory, derives neuronal dynamics Free energy principle from first principles and relates this dynamics to biological processes. Furthermore, this paper provides Process theory a fundamental building block needed to understand active inference for mixed generative models; Variational Bayesian inference allowing continuous sensations to inform discrete representations. This paper may be used as follows: Markov decision process to guide research towards outstanding challenges, a practical guide on how to implement active Mathematical review inference to simulate experimental behaviour, or a pointer towards various in-silico neurophysiological responses that may be used to make empirical predictions. ' 2020 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Contents 1. Introduction.........................................................................................................................................................................................................................3 2. Active inference..................................................................................................................................................................................................................3 3. Discrete state-space generative models...........................................................................................................................................................................6 4. Variational Bayesian inference..........................................................................................................................................................................................6 4.1. Free energy and model evidence.........................................................................................................................................................................6 4.2. On the family of approximate posteriors ...........................................................................................................................................................7 4.3. Computing the variational free energy...............................................................................................................................................................8 5. Perception............................................................................................................................................................................................................................9 5.1. Plausibility of neuronal dynamics........................................................................................................................................................................ 10 6. Planning, decision-making and action selection............................................................................................................................................................. 10 6.1. Planning and decision-making ............................................................................................................................................................................. 10 6.2. Action selection, policy-independent state-estimation ..................................................................................................................................... 11 6.3. Biological plausibility ............................................................................................................................................................................................ 11 6.4. Pruning of policy trees.......................................................................................................................................................................................... 11 6.5. Discussion of the action–perception cycle ......................................................................................................................................................... 11 7. Properties of the expected free energy ........................................................................................................................................................................... 12 8. Learning............................................................................................................................................................................................................................... 12 9. Structure learning............................................................................................................................................................................................................... 14 9.1. Bayesian model reduction .................................................................................................................................................................................... 14 9.2. Bayesian model expansion ................................................................................................................................................................................... 15 10. Discussion............................................................................................................................................................................................................................ 16 11. Conclusion ........................................................................................................................................................................................................................... 17 Declaration of competing interest.................................................................................................................................................................................... 17 ∗ Corresponding author at: Department of Mathematics, Imperial College London, London, SW7 2RH, United Kingdom. E-mail address: [email protected] (L. Da Costa). https://doi.org/10.1016/j.jmp.2020.102447 0022-2496/' 2020 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by- nc-nd/4.0/). L. Da Costa, T. Parr, N. Sajid et al. Journal of Mathematical Psychology 99 (2020) 102447 .............................................................................................................................................................................................................................................. 17 Appendix A. More complex generative models.............................................................................................................................................................. 17 A.1. Learning B and D ................................................................................................................................................................................................... 17 A.2. Complexifying the prior over policies................................................................................................................................................................. 17 A.3. Multiple state and outcome modalities .............................................................................................................................................................. 18 A.4. Deep temporal models.......................................................................................................................................................................................... 18 Appendix B. Computation of dynamics underlying perception.................................................................................................................................... 18 B.1. Free energy conditioned upon a policy .............................................................................................................................................................. 18 B.2. Free energy gradients...........................................................................................................................................................................................