PLOS ONE RESEARCH ARTICLE ToyArchitecture: Unsupervised learning of interpretable models of the environment ☯ ☯ Jaroslav VõÂtkůID *, Petr DluhosÏ , Joseph Davidson, Matěj Nikl, Simon Andersson, Přemysl PasÏka, Jan SÏ inkora, Petr Hlubuček, Martin StraÂnskyÂ, Martin Hyben, Martin Poliak, Jan Feyereisl, Marek Rosa GoodAI Research s.r.o., KarolinskaÂ, Prague, Czech Republic ☯ These authors contributed equally to this work. * [email protected] a1111111111 a1111111111 a1111111111 a1111111111 Abstract a1111111111 Research in Artificial Intelligence (AI) has focused mostly on two extremes: either on small improvements in narrow AI domains, or on universal theoretical frameworks which are often uncomputable, or lack practical implementations. In this paper we attempt to follow a big pic- ture view while also providing a particular theory and its implementation to present a novel, OPEN ACCESS purposely simple, and interpretable hierarchical architecture. This architecture incorporates Citation: VõÂtků J, DluhosÏ P, Davidson J, Nikl M, the unsupervised learning of a model of the environment, learning the influence of one's Andersson S, PasÏka P, et al. (2020) own actions, model-based reinforcement learning, hierarchical planning, and symbolic/sub- ToyArchitecture: Unsupervised learning of interpretable models of the environment. PLoS symbolic integration in general. The learned model is stored in the form of hierarchical repre- ONE 15(5): e0230432. https://doi.org/10.1371/ sentations which are increasingly more abstract, but can retain details when needed. We journal.pone.0230432 demonstrate the universality of the architecture by testing it on a series of diverse environ- Editor: Qichun Zhang, University of Bradford, ments ranging from audio/visual compression to discrete and continuous action spaces, to UNITED KINGDOM learning disentangled representations. Received: April 15, 2019 Accepted: February 29, 2020 Published: May 18, 2020 Copyright: © 2020 VõÂtků et al. This is an open Motivation access article distributed under the terms of the Creative Commons Attribution License, which Despite the fact that strong AI capable of handling a diverse set of human-level tasks was envi- permits unrestricted use, distribution, and sioned decades ago, and there has been significant progress in developing AI for narrow tasks, reproduction in any medium, provided the original we are still far away from having a single system which would be able to learn with efficiency author and source are credited. and generality comparable to human beings or animals. While practical research has focused Data Availability Statement: Data and mostly on small improvements in narrow AI domains, research in the area of Artificial Gen- implementation are available open source at eral Intelligence (AGI) has tended to focus on frameworks of truly general theories, like AIXI https://github.com/GoodAI/torchsim. [1], Causal Entropic Forces [2], or PowerPlay [3]. These are usually uncomputable, incompati- Funding: At the time of working on this study and ble with theories of biological intelligence, and/or lack practical implementations. the manuscript, all of the authors were employed Another class of algorithm that can be mentioned encompasses systems that are usually by a commercial company: GoodAI Research s.r. somewhere on the edge of cognitive architectures and adaptive general problem-solving sys- o., which was the sole funder of this research. The tems. Examples of such systems are: the Non-Axiomatic Reasoning System [4], Growing funder provided support in the form of salaries for all authors, provided the hardware for performing Recursive Self-Improvers [5], recursive data compression architecture [6], OpenCog [7], experiments and approved the final decision to Never-Ending Language Learning [8], Ikon Flux [9], MicroPsi [10], Lida [11] and many others publish. The funder did not have any additional role [12]. These systems usually have a fixed structure with adaptive parts and are in some cases PLOS ONE | https://doi.org/10.1371/journal.pone.0230432 May 18, 2020 1 / 50 PLOS ONE ToyArchitecture: Unsupervised learning of interpretable models of the environment in the study design, data collection and analysis, or able to learn from real-world data. There is often a trade-off between scalability and domain preparation of the manuscript. The specific roles of specificity, therefore they are usually outperformed by deep learning systems, which are gen- the authors are articulated in the `Author eral and highly scalable given enough data, and therefore increasingly more applicable to real- Contributions' section. world problems. Competing interests: At the time of working on Finally, at the end of this spectrum there are theoretical roadmaps that are envisioning this study and the manuscript, all the authors were promising future directions of research. These usually suggest combining deep learning with employed by the commercial company GoodAI Research s.r.o. This does not alter our adherence additional structures enabling, for example, more sample-efficient learning, more human-like to PLOS ONE policies on sharing data and reasoning, and other attributes [13, 14]. materials. The company does not hold any patents Our approach could be framed as something between the ones described above. It is an pertaining to the work described in the paper, and attempt to propose a reasonably unified AI architecture which takes into account the big pic- there are no other restrictions on the sharing of ture, and states the required properties right from the beginning as design constraints (as in data and/or materials published in this manuscript. [15]), is interpretable, and yet there is a simple mapping to deep learning systems if necessary. In this paper, we present an initial version of the theory (and its proof-of-concept imple- mentation) defining a unified architecture which should fill the aforementioned gap. Namely, the goals are to: · Provide a hierarchical and decentralized architecture capable of robust learning and infer- ence across a variety of tasks with noisy and partially-observable data. · Produce one simple architecture which either solves, or has the potential to solve as many of the requirements for general intelligence as possible, according to the holistic design princi- ples of [13, 16]. · Emphasize simplicity and interpretability and avoid premature optimization, so that prob- lems and their solutions become easier to identify. Thus the name “ToyArchitecture”. This paper is structured as follows: first, we state the basic premises for a situated intelligent agent and review the important areas in which current Deep Learning (DL) methods do not perform well (see Required Properties of the Agent). Next, in section Environment Descrip- tion and Implications for the Learned Model, we describe the properties of the class of envi- ronments in which the agent should be able to act. We try to place restrictions on those environments such that we make the problem practically solvable but do not rule out the real- istic environments we are interested in. Section Design Requirements on the Architecture then transforms the expected properties of the environments into design requirements on the architecture. In section Description of the Prototype Architecture the functionality of the pro- totype architecture is explained with reference to the required properties and the formal defi- nition in the Appendix. Section Experiments presents some basic experiments on which the theoretical properties of the architecture are illustrated. Finally, Discussion and Conclusions compares the ToyArchitecture to existing models of AI, discusses its current limitations, and proposes avenues for future research. Required properties of the agent This section describes the basic requirements of an autonomous agent situated in a realistic environment, and discusses how they are addressed by current Deep Learning frameworks. 1. Learning: Most of the information received by an agent during its lifetime comes without any supervision or reward signal. Therefore, the architecture should learn in a primarily unsupervised way, but should support other learning types for the occasions when feedback is supplied. PLOS ONE | https://doi.org/10.1371/journal.pone.0230432 May 18, 2020 2 / 50 PLOS ONE ToyArchitecture: Unsupervised learning of interpretable models of the environment 2. Situated cognition: The architecture should be usable as a learning and decision making system by an agent which is situated in a realistic environment, so it should have abilities such as learning from non-i.i.d. and partially observable data, active learning [17], etc. 3. Reasoning: It should also be capable of higher-level cognitive reasoning (such as goal- directed, decentralized planning, zero shot learning, etc.). However, instead of needing to decide when to switch between symbolic/sub-symbolic reasoning, the entire system should hierarchically learn to compress high-dimensional inputs to lower-dimensional (a similar concept to the semantic pointer [18]), slower changing [19], and more structured [20] rep- resentations. At each level of the hierarchy, the same inference mechanisms should be com- patible with both (simple) symbolic and sub-symbolic terms. This refers to one of the most fundamental problems in AIÐchunking: how to efficiently convert raw sensory data into a structured and separate format [21, 22]. The system should be able to learn and store repre- sentations of both simple and complex concepts