Natural Language Generation Using Link Grammar for General Conversational Intelligence Vignav Ramesh1,2 and Anton Kolonin2,3,4 1 Saratoga High School, Saratoga CA 95070, USA 2 SingularityNET Foundation, Amsterdam, Netherlands 3 Aigents, Novosibirsk, Russian Federation 4 Novosibirsk State University, Russian Federation {rvignav, akolonin}@gmail.com Abstract. Many current artificial general intelligence (AGI) and natural language processing (NLP) architectures do not possess general conversational intelli- gence—that is, they either do not deal with language or are unable to convey knowledge in a form similar to the human language without manual, labor-inten- sive methods such as template-based customization. In this paper, we propose a new technique to automatically generate grammatically valid sentences using the Link Grammar database. This natural language generation method far outper- forms current state-of-the-art baselines and may serve as the final component in a proto-AGI question answering pipeline that understandably handles natural lan- guage material. Keywords: Interpretable Artificial Intelligence, Formal Grammar, Natural Lan- guage Generation, Natural Language Processing. 1 Introduction 1.1 Artificial General Intelligence and Natural Language Processing Currently, the fields of AGI and NLP have little overlap, with few existing AGI archi- tectures capable of comprehending natural language and nearly all NLP systems founded upon specialized, hardcoded rules and language-specific frameworks that are not generalizable to the various complex domains of human language [1]. Furthermore, while some NLP and AGI frameworks do incorporate some form of natural language understanding, they are still unable to convey knowledge in a form similar to the human language without manual, labor-intensive methods such as template-based customiza- tion. 1.2 Motivations Interpretable Language Processing. Unlike explainable artificial intelligence (XAI), which refers to methods and techniques in the application of artificial intelligence (AI) such that the results generated by the AI model can be understood by humans, inter- 2 pretable AI (IAI) requires both the AI model itself as well as its results to be under- standable. It contrasts with the concept of “black box” algorithms in machine learning where even developers cannot explain how the AI arrived at a specific decision. Inter- pretable language processing (ILP) is an extension of the IAI concept to NLP; ILP is expected to allow for acquisition of natural language, comprehension of textual com- munications, and production of textual messages in a reasonable and transparent way [2]. Our natural language generation (NLG) architecture intends to serve as an ILP method that allows for the generation of grammatically valid sentences in an interpret- able manner. Our work not only provides explainable results but also provides an inter- pretable model for sentence generation, since we rely on Link Grammar which is com- prehensible in itself. Unsupervised Language Learning. Current methods of grammar learning, such as deep neural networks (DNNs), require extensive supervised training on large corpora. However, humans are capable of acquiring explainable and reasonable rules of building sentences from words based on grammatical rules and conversational patterns and clearly understand the grammatical and semantic categories of words. This raises the idea of unsupervised language learning (ULL), which enables acquisition of language grammar from unlabeled text corpora programmatically in an unsupervised way. In a ULL system, the learned knowledge is stored in a human-readable and reasonable rep- resentation [2]. Examples of ULL systems include the OpenCog AI platform’s cogni- tive pipeline that enables unsupervised grammar learning [3] and Glushchenko et al.’s grammar induction technology that unsupervisedly learns grammar from only a single input corpus [4]. The ULL pipeline consists of five main components: a text pre-cleaner (which pre- processes corpus files with configurable cleanup and normalization options), a sense pre-disambiguator (which performs word disambiguation and builds senses from to- kens), a text parser (which parses sentences into word tokens or senses), a grammar learner (which learns word categories and rules from parses), and a tester that evaluates the quality of the inferred grammar. The ultimate output of the ULL pipeline is a model of the human language; one such model is the Link Grammar dictionary itself, which serves as the knowledge base behind our NLG architecture [5]. Question Answering. Question answering is a computer science discipline within the fields of information retrieval and natural language processing concerned with building systems that automatically answer questions posed by humans in a natural language. An explainable question answering pipeline would consist of two main components: natural language comprehension, and natural language generation. Natural language comprehension involves first parsing the question (otherwise known as the input query) based on a formal grammar such as Link Grammar, and then performing semantic in- terpretation (extracting the concept represented by the query and determining relation- ships between individual parts of the parsed query obtained in the previous step). The natural language generation component then involves semantic query execution (deter- mining the answer to the input question based on the semantic relationships extracted in the previous step) and the use of a formal grammar to construct grammatically valid 3 sentences from the words associated with the semantic relationships derived during query execution [2]. A diagram of the question answering pipeline is shown below. Fig. 1. Question answering workflow Our current work is concerned with the final part of the question answering pipeline; with Link Grammar serving as the requisite formal grammar, the proposed NLG archi- tecture automates sentence construction from the words derived in the penultimate part of the pipeline and thereby enables question answering with minimal to no human in- terference. 1.3 Link Grammar Link Grammar is founded upon the idea that each word possesses a feature structure consisting of a set of typed connectors and disjuncts associating those connectors. 4 Rules, which correspond to lexical entries or grammatical categories, describe a list of words and a set of their defining disjuncts. Generation involves matching up connectors from one word with connectors from another, given the known set of disjuncts for both of them. A connector represents either the left or right half of a grammatical link of a given type, and different types of connectors are denoted by letters or pairs of letters like S or SX. For instance, if a word A has the connector S+, this means that A can have an S link to its right. If a word b has the connector S-, this means that b can have an S link to its left. In this case, if A occurs to the left of b in a sentence, then the two words can be joined together with an S link [1]. The “Link Grammar dictionary,” then, is a data- base that maps all common words of a given language to the connectors that define them. Disjuncts are sets of connectors that constitute the legal use of a given word. For instance, if word A has the expression {C-} & (X+ or Y+), where {} denotes optional- ity, then A has the following four disjuncts: C- X+, X+, C- Y+, Y+. Macros are single symbols that define large connector expressions. For instance, let the macro <macro> define the expression {C-} & (X+ or Y+) as above. Then, the ex- pression <macro> or {W+ & Z-} is simply shorthand for ({C-} & (X+ or Y+)) or {W+ & Z-}. Macros are used throughout the Link Grammar dictionary to reduce redundancy of connector expressions and increase readability. In Link Grammar notation, a sentence is a set of tokens (words, punctuation, and other syntactic structures) that may be linked by matching connectors between pairs of tokens. For instance, consider the sentence, “The cat caught a mouse.” The Link Gram- mar parse structure for this sentence is: Fig. 2. Link Grammar parse of “The cat caught a mouse.” The rules of Link Grammar impose additional constraints beyond the matching of connectors, namely the planarity and connectivity metarules. Planarity means that links do not cross, while connectivity means that the links and words of a sentence must form a connected graph—all the words must be linked to the other words in the sentence via at least one path. Overall, the structure of grammatical knowledge as stored in Link Grammar appears interpretable, human-readable, and maintainable, including the ability to add new rules and amend existing ones based on the intent of a human linguist maintaining the knowledge base or a programmatic language learning framework such as the ULL pipe- line discussed in Section 1.2 [3, 4]. Our proposed architecture utilizes the English Link Grammar dictionaries to perform natural language generation. Link Grammar Database. The Link Grammar database is comprised of dictionaries for each of more than 10 languages. In this paper, we focus on the English database, 5 which includes approximately 1,430 distinct rules that each correspond to word clus- ters—groups of words with the same sets of grammar properties—and 86,863 word forms. Dictionaries may be represented as hierarchical trees of files, where a master “.dict” file maps all common words to their defining connector expressions and refer- ences supporting