Towards Unsupervised Knowledge Extraction Dorothea Tsatsoua,b, Konstantinos Karageorgosa, Anastasios Dimoua, Javier Carbob, Jose M. Molinab and Petros Darasa aInformation Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH), 6th km Charilaou-Thermi Road, 57001, Thermi, Thessaloniki, Greece bComputer Science Department, University Carlos III of Madrid, Av. Universidad 30, Leganes, Madrid 28911, Spain Abstract Integration of symbolic and sub-symbolic approaches is rapidly emerging as an Artificial Intelligence (AI) paradigm. This paper presents a proof-of-concept approach towards an unsupervised learning method, based on Restricted Boltzmann Machines (RBMs), for extracting semantic associations among prominent entities within data. Validation of the approach is performed in two datasets that connect lan- guage and vision, namely Visual Genome and GQA. A methodology to formally structure the extracted knowledge for subsequent use through reasoning engines is also offered. Keywords knowledge extraction, unsupervised learning, spectral analysis, formal knowledge representation, symbolic AI, sub-symbolic AI, neuro-symbolic integration 1. Introduction Nowadays, artificial intelligence (AI) is linked mostly to machine learning (ML) solutions, enabling machines to learn from data and subsequently make predictions based on unidentified patterns in data, taking advantage of neural network (NN)-based methods. However, AI is still far from encompassing human-like cognitive capacities, which include not only learning but also understanding1, abstracting, planning, representing knowledge and logically reasoning over it. On the other hand, Knowledge Representation and Reasoning (KRR) techniques allow machines to reason about structured knowledge, in order to perform human-like complex problem solving and decision-making. AI foundations propose that all the aforementioned cognitive processes (learning, abstracting, representing, reasoning) need to be integrated under a unified strategy, in order to advance to In A. Martin, K. Hinkelmann, H.-G. Fill, A. Gerber, D. Lenat, R. Stolle, F. van Harmelen (Eds.), Proceedings of the AAAI 2021 Spring Symposium on Combining Machine Learning and Knowledge Engineering (AAAI-MAKE 2021) - Stanford University, Palo Alto, California, USA, March 22-24, 2021. " [email protected] (D. Tsatsou); [email protected] (K. Karageorgos); [email protected] (A. Dimou); [email protected] (J. Carbo); [email protected] (J.M. Molina); [email protected] (P. Daras) 0000-0003-0554-9679 (D. Tsatsou); 0000-0002-5426-447X (K. Karageorgos); 0000-0003-2763-4217 (A. Dimou); 0000-0001-7794-3398 (J. Carbo); 0000-0002-7484-7357 (J.M. Molina); 0000-0003-3814-6710 (P. Daras) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR http://ceur-ws.org Workshop ISSN 1613-!!" CEUR Workshop Proceedings (CEUR-WS.org) Proceedings 1"Understanding" is a term that touches complex human cognitive capacities and is equivocal when applied to machine intelligence. In the premises of this paper, we use the term to denote ’digestion’ of disparate information to capture the major premises (semantics) of an entire domain, or of particular content, tasks or any other contextual premise. Figure 1: Analogies of cognitive development theory (Piagetean-inspired) to modern AI. stronger AI. This integration involves the progress from intuitive, sub-symbolic intelligence to symbolic, logic-based cognitive processes. Bridging sub-symbolic (connectionist) AI with symbolic AI has long been considered crucial towards effective AI solutions [1]. Recognizing the need of symbolic and sub-symbolic integration, the work of this paper is inspired by the amalgamation of different learning and epistemology theories on human cognitive development. Besides their functional differences, most theories converge to a devel- opmental process by which human cognition evolves. In this process, sub-symbolic learning is fundamental (both in the sense of rudimentary, as well as in the sense of necessary) to obtain symbolic function, subsequently used for concrete or abstract logic and reasoning [2]. The current success of NN-based sub-symbolic learning and the plethora of symbolic KRR solutions long available, signify that AI is now equipped to complete its development cycle, but with one missing link between two ends: non-manual acquisition of symbolic knowledge based on the distillation of information hidden in sub-symbolic models. Figure1 portrays the analogies with the Piagetean [3]2 cognitive development stages to AI’s processes. The work of this paper corresponds to analogies understanding, i.e. making connections be- tween concepts. Consequently, we propose an unsupervised approach for extracting knowledge based on the patterns formed in trained neural networks, namely from Restricted Boltzmann Machines (RBMs) [4]. The extracted knowledge can be represented formally and thus shared, used and re-used for logical inference over any domain. The idea is in its early fruition stage and the paper performs a sanity check over the proposed approach, with an interest in its applicability to different domains and/or data, and presents a concrete plan for the evolution of the approach. To this end, Section2 offers an overview of related work; Section3 presents the implemented method, along with first experimental results and observations; Section4 provides a conclusion over the initial approach and a concrete overview of future work. 2. Related Work Integration of sub-symbolic and symbolic methods pertains to three major research directions: a) symbolic representation learning through neural approaches; b) induction of structured 2Piaget’s theory has received criticism in terms of terminology, stage transition and contents, causation of gained attributes, etc. This is the reason why the specific stage names are not used in Figure1. However, it does provide a concrete flow of the development of cognitive capacities in humans, useful to depict the analogies in human vs machine intelligence developmental processes. knowledge and/or some logical inference capacities in neural approaches, e.g. [5], [6]; c) hybrid methods that combine symbolic and sub-symbolic solutions to solve different parts of specific problems, e.g. [7], [8]. The scope of this paper lies in (a) - symbolic representation learning. This can be further divided to: i) recognition of pertinent symbolic representations of salient concepts within a domain and/or their hierarchy; ii) recognition of prominent relations that associate particular concepts; iii) pattern mining for extracting particular associative rules within a domain of discourse. Representation learning focuses on identifying/automatically constructing the input features needed to perform a specific ML task. In computer vision, representations learned often donot have any symbolic manifestation, rather remain unstructured, black box vehicles in ML-based classification and feature detection [9], with few approaches aligning representations learned to specific symbolic labels [10]. In NLP, representation learning revolves mostly around the construction of word embeddings, associating words of high lexical proximity. Symbolic representation of natural language components is inherent, often accompanied with underlying semantics [11]. Several supervised methods have tried to move beyond mere term proximity identification to deeper semantics recognition, usually delving into encoding taxonomic relations between words within term embeddings [12]. Symbolic representation of non-taxonomic relations, however, is one of the most pertinent tasks towards structured knowledge acquisition. To this end, [13] employ modular neural networks, combining different NN models (e.g. image and text) with common features fused in a cumulative model, thus allowing to learn how textual relations associate objects in images. The Neuro-Symbolic Concept Learner [14], not only learns object representations in scenes through natural supervision – i.e. no labeled data required – but also learns non-specified binary relations between recognized objects, combining visual and textual cues. Ultimately, knowledge learned is formalized and reasoned upon, in a question-answering task. The method is however yet confined to very few specific objects and predefined relations between them. Still, itpaves the way towards pragmatic neural extraction of knowledge and subsequently towards symbolic inferencing over new knowledge. The Neural Symbolic Cognitive Agent (NSCA) [15] uses a Recurrent Temporal Restricted Boltzmann Machine (RTRBM) [16] to learn complex temporal relations between data and subsequently formalize these relations into propositional rules, to be used for subsequent inferencing. The method has been applied to a restricted domain task and only to enrich existing knowledge with non-persistent, non-verified (to be exact, not needing verification) knowledge, however it unveils the capacity of RBMs as a powerful neural method for obtaining associations and rules among entities. [17] also employ Restricted Boltzmann Machines (RBMs) in order to extract fuzzy rules, en- capsulating the uncertainty/vagueness of probabilistic machine learning within logic-compliant rules, something missing from the crisp NSCA method. However, this method lacks compre- hensive description of the formalization method for the extracted rules. The hybrid Differential Inductive Logic Programming (DILP) method [18] combines