TRANSFORMER NETWORKS OF CONCEPTUAL KNOWLEDGE 1 Transformer Networks of Human Conceptual Knowledge Sudeep Bhatia and Russell Richie University of Pennsylvania March 1, 2021 Send correspondence to Sudeep Bhatia, Department of Psychology, University of Pennsylvania, Philadelphia, PA. Email:
[email protected]. Funding was received from the National Science Foundation grant SES-1847794. TRANSFORMER NETWORKS OF CONCEPTUAL KNOWLEDGE 2 Abstract We present a computational model capable of simulating aspects of human knowledge for thousands of real-world concepts. Our approach involves fine-tuning a transformer network for natural language processing on participant-generated feature norms. We show that such a model can successfully extrapolate from its training dataset, and predict human knowledge for novel concepts and features. We also apply our model to stimuli from twenty-three previous experiments in semantic cognition research, and show that it reproduces fifteen classic findings involving semantic verification, concept typicality, feature distribution, and semantic similarity. We interpret these results using established properties of classic connectionist networks. The success of our approach shows how the combination of natural language data and psychological data can be used to build cognitive models with rich world knowledge. Such models can be used in the service of new psychological applications, such as the cognitive process modeling of naturalistic semantic verification and knowledge retrieval, as well as the modeling of real-world categorization, decision making, and reasoning. Keywords: Conceptual knowledge; Semantic cognition; Distributional semantics; Connectionist modeling; Transformer networks TRANSFORMER NETWORKS OF CONCEPTUAL KNOWLEDGE 3 Introduction Knowledge of concepts and their features is one of the fundamental topics of inquiry in cognitive science (Murphy, 2004; Rips et al., 2012).