Exploring Neural Models for Parsing Natural Language into First-Order Logic Hrituraj Singh ∗ Milan Aggrawal Balaji Krishnamurthy Adobe Research Adobe Systems Adobe Systems Abstract a knowledge base (such as FreeBase (Bollacker et al., 2008)) for retrieving concise answers (Fur- Semantic parsing is the task of obtaining bach et al., 2010). Such representations can be used machine-interpretable representations from to specify instructions to robots (Artzi and Zettle- natural language text. We consider one such formal representation - First-Order Logic moyer, 2013) or conversational agents (Artzi and (FOL) and explore the capability of neural Zettlemoyer, 2011) for executing desired action(s) models in parsing English sentences to FOL. in an environment. Similarly, natural language We model FOL parsing as a sequence to se- queries are transformed into executable database quence mapping task where given a natural programming language instructions (such as SQL) language sentence, it is encoded into an in- to retrieve or generate correct results in a database termediate representation using an LSTM fol- (Sun et al., 2018; Zhong et al., 2017). lowed by a decoder which sequentially gener- ates the predicates in the corresponding FOL A variety of logical forms and meaning represen- formula. We improve the standard encoder- tations have been proposed for text. These include decoder model by introducing a variable align- graph-based formalisms (Banarescu et al., 2013; ment mechanism that enables it to align vari- Abend and Rappoport, 2013; Oepen et al., 2014; ables across predicates in the predicted FOL. Kollar et al., 2018) where text is represented as We further show the effectiveness of predict- a typed graph. The entities and action events are ing the category of FOL entity - Unary, Binary, represented as nodes with labeled edges depicting Variables and Scoped Entities, at each decoder relations between them. Semantic dependency tree step as an auxiliary task on improving the con- sistency of generated FOL. We perform rigor- (Oepen et al., 2014) is a directed graph depicting ous evaluations and extensive ablations. We the syntactic structure of a sentence in the form of also aim to release our code as well as large modifier relations between its words. AMR (Ab- scale FOL dataset along with models to aid fur- stract Meaning Representation) graphs (Banarescu ther research in logic-based parsing and infer- et al., 2013) use variables to annotate nodes fol- ence in NLP. lowing neo-Davidsonian style (Davidson, 1969). Lambda Dependency-based Compositional Seman- 1 Introduction tics (λ-DCS) (Liang, 2013) was proposed as a for- Semantic parsing aims at mapping natural language mal language adapting Dependency-Based Compo- arXiv:2002.06544v1 [cs.CL] 16 Feb 2020 to structured meaning representations. This en- sitional Semantics (Liang et al., 2013) borrowing ables a machine to understand unstructured text the expressiveness of lambda calculus (Barendregt better which is central to many tasks requiring nat- et al., 1984) but aiming to remove explicit use of ural language understanding such as question an- variables. swering (Berant et al., 2013; Pasupat and Liang, In this work, we focus on first-order logic (FOL) 2015), robot navigation (MacMahon et al., 2006; (Smullyan, 2012) as the language formalism for Artzi and Zettlemoyer, 2013), database querying text. FOL represents entities and actions in natu- (Zelle and Mooney, 1996) etc. For question an- ral language through quantified variables and con- swering, natural language question is converted to sists of functions (called predicates) which take formal semantics which facilitates interaction with variables as arguments. The predicates attach se- This∗ work was done when the author was working as a mantics to variables and express relations between Software Developer at Adobe Systems objects (Blackburn, 2005). For instance, a simple sentence - “a man is eating” can be represented neural approaches owing to complexities in its rep- through FOL as resentation. Since it is one of the first such explo- ration for FOL, we treat the popular sequence to 9A(9B(man(A) ^ eat(B) ^ agent(B; A))) sequence model coupled with attention mechanism Advanced natural language concepts as in sen- (Bahdanau et al., 2014) as our baseline. We pro- tence “the man and woman are seated facing each pose to disentangle the prediction of different types other” can be expressed as of FOL syntactic entities (unary and binary pred- icates, variables etc) while parsing sentences and 9A(9B(9D(9C(man(A) ^ woman(B) ^ show improvements through performing category seat(D) ^ subset of(A; C) ^ type prediction as an auxiliary task. We further subset of(B; C) ^ theme(D; C) ^ show major improvements by explicitly constrain- not(9E(other(E) ^ not(9F (face(F ) ^ ing the decoder to align variables across unary and theme(F; E) ^ agent(F; C))))))))) binary predicates. This restricts the model to main- where “man” and “woman” are represented to- tain consistency while expressing standalone entity gether through shared variableC and “facing each attributes and relations between them. other” is represented by negating the existence of a Our contributions can be enumerated as: 1) We thing E for which C is not facing E holds true. explore and develop an open domain neural se- The success of learning based neural approaches mantic parser to parse natural language sentences in NLP tasks like machine translation (Cho et al., to FOL using Seq2Seq framework; 2) We pro- 2014; Sutskever et al., 2014; Vaswani et al., 2017), pose disentangled FOL entity type prediction along paraphrase generation (Prakash et al., 2016; Gupta with FOL parsing under multi-task learning and et al., 2018), dialog modeling (Vinyals and Le, FOL variable alignment through decoder alignment 2015; Kottur et al., 2017), machine comprehension mechanism. We perform extensive ablation studies (Wang et al., 2017), logical inference (Kim et al., to establish the improvements registered; 3) We 2019) has motivated their use for semantic parsing also aim to release our code, models and large (Kociskˇ y` et al., 2016; Buys and Blunsom, 2017; scale dataset used comprising of sentence-FOL Cheng et al., 2017; Liu et al., 2018; Li et al., 2018) mappings to aid further research in FOL based as well. Many such works use the encoder-decoder NLP. framework to model it as a sequence transduction task. Since they were designed for solving specific 2 Background tasks like question answering, such methods (Jia and Liang, 2016; Dong and Lapata, 2016) have Text to FOL Conversion : In this section, we mainly focussed on confined logical formalism for give a brief overview of syntactic-semantic analy- specific domains such as flight reservation, restau- sis pipeline used for obtaining the mappings data rant booking, etc (Wang et al., 2015) capturing through Boxer (Bos, 2008) based on Combina- limited vocabulary and semantic concepts. tory Categorial Grammar (CCG) (Steedman and In this paper, we aim at developing a general- Baldridge, 2011) and Discourse Representation purpose open-domain neural first-order logic parser Theory (DRT) (Kamp et al., 2011). CCG is phrase- for natural language sentences to examine the ca- level grammar which defines rules for generating pabilities of such models. We train our model constituency-based structures. CCG comprises of by obtaining a large corpus of text-FOL pairs for syntactically typed lexical items such that each sentences in SNLI Dataset (Bowman et al., 2015) item is a lambda-expression and uses combinatorial through C&C parser (Clark and Curran, 2007) logic (lambda calculus) to combine them through and Boxer (Bos, 2008) (discussed later in detail).1 the application of combinators. CCG derivation Apart from meaning depiction, parsing sentences guides semantic composition to obtain Discourse to FOL would enable neural models to capture Representation Structures (DRS) from CCG parses. complex relationships between entities resulting in DRS comprises of discourse referents and condi- richer embeddings which might be useful in several tions defined on them which can be recursive. DRS other NLP tasks. Such an examination would help is capable of representing varied linguistic phe- understand challenges in generating FOL through nomena such as anaphora, presupposition, tense and aspect. These DRSs are compatible and can be 1https://github.com/valeriobasile/candcapi converted to FOL through a set of syntactic trans- formations (Bunt et al., 2001). Formally, predi- on the input sentence and the previously generated cates in FOL are atomic formulas that are combined tokens. Our input X consists of a sequence of through logical connectives - logical and (^), logi- m tokens fx0; x1; :::; xmg which get encoded into cal or (_) ; and quantifiers. In general, a predicate hidden contextual representations by an Encoder. P (v1; v2; :::; vn) is an n-ary function of variables. The Decoder, then, generates an output sequence There are two types of quantifiers, universal (8)- of n tokens fy0; y1; :::; yng. which specifies that sub-formula within its scope n is true for all instances of the variable and exis- Y P (Y jX) = P (y jy ::y ;X) (1) tential (9) - which asserts existence of at least one i 0 i−1 i=0 instance represented by a variable under which the sub-formula holds true. For example, “All humans 3.1 Encoder eat” can be represented as Our Encoder E is a bidirectional LSTM (biLSTM) 8A(9B(human(A) ^ eat(B) ^ agent(B; A))) which encodes a sequence of input tokens X : Following generalized De Morgan’s law (John- fx0; x1:::; xmg into a sequence of hidden states H : fh ; h ; :::; h g h 2 Rdh stone, 1979), universal quantifiers can be repre- e e0 e1 em , ei to capture con- sented through existential quantification and nega- textual information from the input that is eventu- tion (not) preserving the semantics as ally used by the decoder to produce the output FOL sequence. The biLSTM block takes word embed- not(9A(not(9B(human(A) ^ eat(B) ^ dings for the input tokens Ee : fee0 ; ee1 ; :::; eem g, agent(B; A))))) D eei 2 R as input and processes them to calculate Output and Mapping Format : Given a text sen- the contextual representations tence, we obtain the following FOL output.