A Survey on Semantic Parsing from the Perspective of Compositionality

Pawan Kumar Srikanta Bedathur Indian Institute of Technology Delhi, Indian Institute of Technology Delhi, New Delhi, Hauzkhas-110016 New Delhi, Hauzkhas-110016 [email protected] [email protected]

Abstract compositionality: evident by the choice of the for- mal language and the construction mechanism and Different from previous surveys in seman- lexical variation: grounding the words/phrases to tic parsing (Kamath and Das, 2018) and appropriate KB entity/relation. knowledge base question answering(KBQA) (Chakraborty et al., 2019; Zhu et al., 2019; Language Compositionality: According to Hoffner¨ et al., 2017) we try to takes a dif- Frege’s principle of compositionality: “the ferent perspective on the study of semantic meaning of a whole is a function of the meaning parsing. Specifically, we will focus on (a) meaning composition from syntactical struc- of the parts. The study of formal semantics ture (Partee, 1975), and (b) the ability of for natural language to appropriately represent semantic parsers to handle lexical variation sentence meaning has a long history (Szab, 2017). given the context of a knowledge base (KB). Most semantic formalism follow the tradition of In the following section after an introduction Montague’s grammar (Partee, 1975) i.e. there is of the field of semantic parsing and its uses a one-to-one correspondence between syntax and in KBQA, we will describe meaning represen- semantics e.g. CCG (Zettlemoyer and Collins, tation using grammar formalism CCG (Steed- 2005). We will not be delving into the concept of man, 1996). We will discuss semantic compo- sition using formal languages in2. In section3 semantic representation in formal language in this we will consider systems that uses formal lan- survey. guages e.g. λ-calculus (Steedman, 1996), λ- Lexical Variance: Lexical variation in human DCS (Liang, 2013). Section4 and5 consider semantic parser using structured-language for language is huge. Differences in the surface form logical form. Section6 is on different bench- of words in the natural language and the label of the mark dataset ComplexQuestions (Bao et al., corresponding entity/relation in the KB is mainly 2016) and GraphQuestions (Su et al., 2016) due to the polysemy. For example attend may be that can be used to evaluate semantic parser on referred to by label Education (Berant et al., 2013). their ability to answer complex questions that Similarly paraphrases of a sentence may have dif- are highly compositional in nature. ferent phrases to mean the same thing, e.g. ‘What 1 Introduction is your profession’, ‘What do you do for a living’. ‘What is your source of earning’ all these variation arXiv:2009.14116v1 [cs.CL] 29 Sep 2020 One of the main challenge in Knowledge Base may points to label profession (Berant and Liang, Question Answering (KBQA) is semantic parsing - 2014). the construction of a complete, formal, symbolic, The two challenges are elegantly summed up meaning representation (MR) of a sentence (Wong in a function p = f(a, b, R, K) by Mitchell and and Mooney, 2006). Most commonly used formal Lapata(2010) i.e. the combined meaning of sym- frameworks use a combination of λ-calculus and bol a and b is function of lexicon a and lexicon b first order logic (FOL) e.g. CCG (Zettlemoyer and under the syntactic relation R and the context K. Collins, 2005), λ-DCS (Liang, 2013). The logical- We propose that the context K be the knowledge expression further needs to be grounded in a knowl- base (KB). KBQA provides an appropriate ground edge base (KB), in the case of KBQA, the challenge for testing different semantic parsing approaches here is lexical variation. The two main challenges empirically. There are some KBQA systems which that any KBQA system has to tackle are language use semantic parser as a module in their pipeline (Reddy et al., 2014; Cheng et al., 2017) where the parse the natural language sentence into functional purview of semantic parsing is to get to the logical- composition of meaningful parts greatly affects its expression, and a downstream process takes up capacity (expressiveness). Such a system is known lexicon grounding or disambiguation. There are as Semantic Parser. some systems which don’t have such seperations e.g. SEMPRE(Berant et al., 2013). However, both 2.1 Formal Language type of systems do resort to some formal language First order predicate logic (FOPL) can be used to or intermediate logical form. We exclude KBQA represent meaning of natural language sentence, systems which use non-symbolic representation however it fails to represent some concepts in the (Cohen et al., 2020). natural language, e.g. “How many primes are less We describe here some terminology commonly than 10?” (Liang, 2016) - FOPL doesn’t have a used in the study of semantic parsing (Diefenbach function to count the number of elements. The for- et al., 2018; Kamath and Das, 2018). 1. Intermedi- mal semantics can use a higher-order language e.g. ate logical form: represents the complete meaning λ-calculus, say a higher order function count ex- of the natural language using formal language e.g. ists, that can count the number of elements in a set. λ-calculus (Zettlemoyer and Collins, 2005; Haki- Thus we can represent the previous questions as mov et al., 2015), λ-DCS (Liang, 2013) or struc- count(λx.prime(x) ∧ less(x, 10)). Without go- tured language Yih et al.(2015); Hu et al.(2018). ing into further details of the formal language, let’s This is the main output of semantic parsing. 2. consider an example here showing compositional phrase-mapping: mapping phrases in the question use of λ-calculus to represent the meaning of a to their corresponding resource in the KB is re- complex sentence. E.g. the sentence “Those who quired to provide a real-word context to the inter- had children born in Seattle.” Liang(2013). mediate logical form. This process is also called grounding of the logical form, thus obtaining a λx.∃y.children(x, y) ∧ P laceOfBirth(y, Seattle) grounded logical form. 3. Disambiguation: of Take another example showing coordination in many resources obtained in the phrase-mapping λ-calculus ”Sqaure blue or round yellow pillow” process only a few will be right according to the (Artzi et al., 2013) which is represented as semantic of the natural language. 4. Query con- struction: Querying the KB-endpoint requires trans- λx.pillow(x) ∧ (square(x) ∧ blue(x))∨ lation of the grounded logical form into a query lan- (round(x) ∧ yellow(x))
guage e.g. SPARQL. Translation from grounded logical form to the query language is a determinis- Many semantic parsing systems use only part of the tic process. operators available in λ-calculus thus they are lim- ited in expressiveness by their choice of operators, 2 Compositionality in Formal Semantics not by the choice of formal language.

Considering the compositionality in the natural 2.2 Structured language language in the sense that meaning of the whole A graph-structured logical form or a tree-structured sentence is constructed from meaning of its parts. logical form can also be used to represent the mean- According to Pelletier(2011) this is a composi- ing of a natural language sentence. tionality in the “functional sense”: something is compositional if it is a complex thing with some Graph-structured language E.g. Semantic property that can be defined in terms of a func- Query Graph(SQG) (Hu et al., 2018), has nodes tion of the same property of its parts (with due representing constant/values and edges represent- consideration to the way the parts are combined). ing relation. The edges could be seen as analo- In formal semantics the complex things is a syn- gous to the binary relation of the logical formalism. tactically complex sentence and the property of This definition of SQG directly corresponds to the interest is meaning, while combining the parts due many knowledge graphs K like dbpedia (Auer et al., consideration has to be given to how those parts 2007), freebase (Bollacker et al., 2008). With four are syntactically present in the complex sentence. primitive operations to manipulate a graph struc- The choice of a formal language to represent mean- ture: connect and merge that operate on pair of ing of a complex sentence in a system trying to nodes, expand and fold that operate on single node (Hu et al., 2018), and with higher-order functions 3 Logic Based Formalism attached to nodes (Yih et al., 2015), graph-structure makes for good candidate for logical form of a nat- Many semantic parsing systems use higher-order ural language sentence e.g. Figure1. formal logic to represent meaning of natural lan- guage sentence e.g. λ−calculus (Zettlemoyer and λ− less Collins, 2005), DCS (Berant et al., 2013; Be- 10 Prime Number Count rant and Liang, 2014). A first order predicate logic can only express simple natural language sentence Figure 1: logical form for “How many primes are less of type yes/no or one that seek set of elements ful- than 10?” as graph-structure filling a logical expression. To operate on set of elements the formal languages are augmented with Tree-structured language Logical languages higer-order function, eg. count(A) that would re- with tree-hierarchy can represent such hierarchy turn cardinality of set A. λ−Calculus Carpenter of natural language as well. FunQL Cheng et al. (1997) is a higher order functional language, it is (2017); Zelle and Mooney(1996a); Kate et al. more expressive, it can represent natural language (2005) is a variable free functional language en- constructs like count, superlative etc. λ−DCS coding tree-hierarchy. It has a predicate-arguments (Liang, 2013) which doesn’t use existential vari- form and a recursive tree structure, where the non- able, borrows hugely from λ−Calculus. Logi- terminal are predicate and the terminal nodes in the cal formalism that was introduced by Liang et al. tree makes for an argument, e.g. sentence “which (2013) as DCS-tree, will be discussed in Section5 states do not border texas?” is: 3.1 Statistical Models answer(exclude(states(all), border(texas))) Zettlemoyer and Collins(2005) have used λ−Calculus as intermediate logical form to rep- Dependency based Compositional Semantics(DCS) resent meaning of the natural language sentence. (Liang et al., 2013) is another tree-structured logi- The construction mechanism requires an initial cal forms, where the logical-form, a tree, is called set of CCG lexicon Λ0, where the lexicons DCS Trees. In its basic version DCS proposes only have also been assigned semantic meaning using two operations join and aggregate and the full ver- λ−abstractions, e.g. borders := (S\NP )/NP : sion comes with higher order function like argmax λx.λy.borders(y, x) for sentence “Utah borders etc. readers are referred to Liang et al.(2013). An Idaho”. Given a sentence and its logical-expression, example of DCS tree as logical form is shown in a rule based function GENLEX(Si,Li) creates figure2. DCS tree proposes to reduce the com- lexicon specific to the sentence. Together with Λ0, plexity in compositinally creating the logical form the new set λ = Λ0 ∪ GENLEX(Si,Li) form of a sentence, however, its being a tree-structured the search space for the probabilistic CCG parser. logical form brings some limitations, such as it The parser uses beam-search to come up with a can’t be used to represent bound anaphora as in high probability parse of the sentence. There are sentence “those who had a child who influenced sequence of derivation stages Ti used by a parser them” (Liang, 2013). to reach to the logical form Li, however they are taken as hidden variable while learning the param- eter of the parser. The set of lexicon Λ will swell when trained over larger datasets, even though the algorithm adds only lexicon that are part of the final logical expression. This work was state of the art at the time, giving precision of 96.25% and recall 79.29% on Geo880 dataset. Berant et al.(2013) was perhaps the first work using a simplified λ−DCS as the intermediate log- ical form, there system is called SEMPRE. The logical form is built recursively in a bottom up Figure 2: DCS tree for sentence “Major city in Califor- fashion starting with a lexicon leading upto the fi- nia” nal logical form. The lexicon are logical predicates in KG which are used for a natural language phrase. meaning of a sentence. There are many works em- The process of determining a lexicon involves cre- ploying ecoder-decoder architecture in the context ating typed-NL phrases R1[t1, t2] by aligning a of semantic parsing e.g. (Dong and Lapata, 2016; text-corpus with KG (Lin et al., 2012), then the set Jia and Liang, 2016; Dong and Lapata, 2018). Un- of supporting entity-pairs F(R1) is determined, like statistical models, described above which re- the logical predicate R2 makes for a lexicon if quire feature engineering and high quality lexicon F(R1) ∩ F(R2) 6= φ. Using typed-phrases also (Zettlemoyer and Collins, 2005), the Neural Net- helps in tackling polysemy e.g. ”born in” could work based model can be trained to learn the fea- mean PlaceOfBirth or DateOfBirth which tures required for semantic parsing by themselves. can be tackled using typed ”born in”[Person, Dong and Lapata(2016) have considered the Location] and ”born in”[Person, Date] problem of semantic parsing as sequence trans- respectively. For composition of lexicon into logi- duction task, converting a sequence of words in cal form there are small set of rules: say z1, z2 are natural language into a sequence of λ-terms in two lexicons, they can undergo following opera- the logical form. The encoder-decoder architec- tions join z1.z2, aggregations z1(z2), intersections ture is made of an LSTM-encoder and an LSTM- z1 ∩ z2 or bridging z1 ∩ p.z2. The last operation decoder. The index of the words in NL sentence bridging is used when the relation between two q = x1 . . . x|q| is first converted to a vector using entity is weakly implied or it is implicit. If the an embedding matrix, then passed to encoder’s in- two entities(also called unary z) have types, say put layer (l = 0), one-by-one in sequence until t1, t2 bridging introduces all binary predicates p time step t|q|, when encoder has seen all the words that have type structure [t1, t2], thus the logical in the input sentence. Next time step and onward form z1 ∩ p.z2. During the process of generating belongs to the decoder. The input to the decoder the logical form, a feature vector take shape as well, in its first time step is hidden state of encoder hL t|q| which is later used to score the candidate logical and the index of the start of sequence symbol hsi. form with a log-linear model. During evaluation At each time step afterward the decoder takes at the logical-form which is most probable pθ(d|x) input layer (l0) word vector corresponding to the gets used. SEMPRE scores F1 32.9% on WebQues- previous predicted word yt−1 and gives the prob- tions which Berant et al.(2013) introduced. ability p(yt|y

3.2 Neural Encoder-Decoder Models Jia and Liang(2016) as well have employed A neural encoder-decoder architecture can learn encoder-decoder architecture and train their model to generate formal logical expression representing on augmented dataset. The augmented dataset is obtained using production rules of context free Semantic Query Graph: A semantic query grammar, e.g. replacing an entity by its type, or graph SQG, by Hu et al.(2018), is a graph, in replacing a word of, say type t1 by a whole-phrase which each vertex vi is associated with an entity when the phrase type checks out t1. phrase, class phrase or wild-cards in the natural Dong and Lapata(2018) improves upon language (NL) sentence ; and each edge eij is as- its previous work slightly by introducing sociated with a relation phrase in the NL sentence, another intermediate decoder called sketch- where 1 ≤ i, j ≤ |V |. decoder. Thus it has for modules in the The semantic parsing pipeline used by Reddy network: input-encoder → sketch-decoder → et al.(2014) takes a natural language (NL) sentence sketch-encoder → output-decoder. The idea and uses a syntactic parser for CCG by Clark and being to gloss over the low-level information Curran(2004). A CCG grammar provides with one- like variable names, their values to create a to-one correspondence between syntax and seman- coarse-representation of the input sentence, which tic, syntax being the word-category and semantic can they guide the output-decoder into generating represented using a λ-expression, together they are better formed logical expressions. referred as CCG-lexicon. Further in the pipeline there is a graph-parser parsing the λ-expression Discussion: LSTMs are easy engineering tools as a label graph. The nodes in the graph may be for sequence-transduction, but they don’t learn the attached to math functions like unique, count etc. rules of the grammar (Sennhauser and Berwick, as required by the semantics of the NL sentence. 2018). Sennhauser and Berwick(2018) used 4 The label-graph is not yet contextualized against production rules and generated a 2Dyck language, a world/KG, therefore also called an ungrounded which is a string having only brackets ‘[’ ‘]’ and graph. A contextualization process of nodes and ‘{’‘}’, e.g. string “{[{}[]]} ”. They showed that edges of the ungrounded graphs against a KG is LSTM don’t generalise well, the error rate for out- carried out using beam search. So, given an un- of-sample test data is 8-14 times high compared to grounded graph (u), labels on its nodes and edges in-sample error-rate of 0.3% for LSTM using 50 are run through KG to find possible candidate en- hidden units. They observe that LSTM learn the tities and relations respectively. Thus obtaining a sequential-statistical correlation and that they don’t set of possible grounded graphs {g}. A structured- store irrelevant information, but LSTM fail to learn perceptron is then used to wade through candidate the 4-rules of the CFG used to generate the lan- graphs by scoring their feature vectors Φ as dot guage. They observe that novel architecture may product with model parameters θ. be required to learn the grammar-rules, and there are works in that directions e.g. dynamic network Yih et al.(2015) have tried to bring in a part architecture (Looks et al., 2017) of hardwired struc- of λ-calculus into a graphical representation by tural constrain (Kiperwasser and Goldberg, 2016; introducing variable nodes (corresponding to ex- Joulin and Mikolov, 2015) istential variables) and answer nodes (correspond- ing to the bounded variable in lambda-expression), 4 Graph based formalism as well as aggregation function such as argmin, argmax, count into semantic query graph. The The semantic parser using graph based formal- generation of a query graph is formulated as tran- ism resort to a labeled-graph also called Semantic sition of states, each state being a subgraph in the Query Graph(SQG) as semantic representation of KG thus always grounded. Starting with an empty the natural language (NL). Graph as an abstraction query graph φ to state Se with only entity nodes, is widely used and Label graph are a good way the next state add path to answer node known as to represent the meaning of NL sentence, where state Sp finally adding the constraints to obtain the the nodes represent a concept/person and the edges full query graph as state Sc. The transition from between represent their inter-dependency. Besides, one state to another is defined in terms of a well graph is also the language of many Knowledge defined set of actions A = ∪{Ae, Ap, Ac, Aa} i.e. Graphs(KG). There are many works using SQG add entity nodes, path nodes, constraints nodes as logical formalism liek Reddy et al.(2014); Yih and aggregation respectively. Candidate Se states, et al.(2015); Bao et al.(2016); Hu et al.(2018); which only contains single entity nodes, are de- Hu et al.(2018). cided using an entity linking system (Yang and Chang, 2016). Transitioning to state Sp requires tween nodes: wherever two entity-mentions are making 1-hops/2-hops and scoring all the candi- adjacent in the dependency-tree the relation edges date paths for a possible predicate by comparing in the SQG is kept unlabelled (implicit relation) its similarity with questions-pattern (obtained from and where they are far apart, the words in the the question after replacing the entity in Se with dependency-tree makes for the label of the rela- place holders ¡e¿) using two CNN models. The tions edge in the SQG. The query evaluation in construction of the final SQG which is state Sc both the framework is similar i.e. finding top-k requires adding constraints (entity/class) or math matching subgraph in KG corresponding to a SQG. functions, which is guided by heuristics, e.g. of Each node and edge in the SQG comes with a all the resource-nodes in KG attached to variable candidate-list and each candidate gets a tf-idf score R E nodes in Sp select one if it is an entity occurring when retrieved from D /D . The only difference in the question. The candidate SQG thus obtained being SQG in the NFF may have few edges left are ranked by the F1 score based on answers they unmatched. Hu et al.(2018) also propose to use receive when executed on KG. bottom-up approach of forming the SQG and find- ing a correct match. Starting with a single node Hu et al. introduced two frameworks, RFF (Re- of the SQG and finding a match in the KG, then lation (edge)-First Framework) and NFF (Node expanding the node to a partial SQG and scoring First Framework), for semantic parsing capitaliz- its corresponding match. ing upon semantic query graph (SQG). The con- structions mechanism in both the framework uses Sun et al.(2020) allude to the fact that current dependency parse of a sentence, say Y (N). In RFF dependency parser err in longer and complex sen- the relation mentions is a subtree y in Y (N). The tences and they propose to first do a coarse (skele- relation mention should have a matching relation ton) parsing of the complex sentence into auxiliary in a predefined set T = {rel1, rel2, . . . , reln}.A clauses and pass it over to NFF (Hu et al., 2018) matching relation reli must have all its words in the to take up the fine-grained semantic parsing. The subtree y. The associated nodes/arguments to rela- skeleton parsing is modeled as set of four steps tion edge hreli, arg1, arg2i are determined based 1. identifying whether a sentence could be parsed on pos-tags coming out from the subtree y. Further, into main-clause and auxiliary-clause, 2. identify- the relation mentions and the node/arg-phrases, ing the text-span in the sentence that makes for an which are in their surface form, are mapped to auxiliary clause after the first step is true, 3. iden- predicates/predicate paths and entities/classes re- tifying the headword in the sentence that governs spectively in the KG. The mapping from reli to the texts-span and 4. identifying the dependency predicates/predicate paths uses a relation mention relation between headword and the text-span. The dictionary DR, which utilizes the set of relations four above steps are modelled using BERT (Devlin mentions and their Supporting-Entities as in Patty et al., 2019). Four different BERTBASE models (Nakashole et al., 2012). Similarly, the mapping are fine-tuned as a task in single sentence classifi- from nodes/args-phrase uses an entity mentions cation (SSC), question answering(QA), questions dictionary (DE) CrossWikis by Spitkovsky and answering and sentence pair classification (SPC) Chang(2012). The second framework NFF is pit- respectively for step 1, 2, 3 and 4. Different from ted to be robust to errors in dependency parsing NFF which uses tf-idf score for ranking candidates as compared to RFF. RFF uses dependency-tree to and the matched query graph, Sun et al. have used a determine relation mentions, dependency structure sentence-level scorer and a word-level scorer. The and pos-tags, however NFF uses dependency-tree sentence-level scorer scores the similarity of a test- only to decide if there should be an edge between sentence against a training-sentence, favoring train- to entity nodes. Another advantage of using NFF is ing sentences which can provide their underlying that it allows unlabelled edges between entities to query graph to the test-sentence which when exe- represent implicit relations eg. an unlabelled edge cuted on the KG should retrieve non-empty result. between nodes Chinese and actor. The implicit re- The sentence-level scorer is a fine-tuned BERT lations are resolved during query evaluations. NFF for the task of SPC. The word-level scorers scores requires first to extract all entity mention using bag-of-words (BOW) in sentence (after removing a dictionary-based approach Deng et al.(2015), entities and stop words) and BOW in query graph then using dependency tree introduce edges be- which mainly consists of predicates, thus scoring appropriateness of the predicates used in the SQG. where et is concatenation of bt and st and Wa,Ws are weight matrix. The mapping of lexicon to Discussion: We summarized four semantic pars- database entity and predicates is done using a bi- ing systems above, which use graph as intermediate linear neural network logical form. They differ mainly in the way they generate the SQG, while Reddy et al.(2014) and T p(gt|ut) ∝ exp ut · Wug · gt Hu et al.(2018) use a syntactic parser (CCG and dependency-tree) Yih et al.(2015) doesn’t use any The training objective is to maximize the likelihood suntactic parser. It uses a state-transition based ap- of the grounded logical form, therefore it considers proach to generate the SQG keeping partial states the ungrounded logical form as latent variable and always grounded. We see that on WebQuestions marginalise over it. (Yih et al., 2015) STAGG achieves 52.5%F 1. The state-transition based approach with a different set Discussion The transition based approach with of actions (connect, merge, expand and fold) is just tree-operation can generate all the tree- used in a different work by Sen Hu (Hu et al., 2018) structured candidate logical-form. It makes a good which got an F1 of 53.6% on WebQuestions. case for its adoptions where the inherent structure of the language could be limited to tree-like. 5 Tree structured logical form 6 Benchmark Datasets Cheng et al.(2017) uses nested tree-structured log- ical form known as funQL. FunQL has a predicate- The benchmark dataset used to evaluate the few argument structure, where the predicate is a non- systems discussed in this survey are described in terminal (NT) and the arguments may be another table1. Benchmark datasets have grown in com- sub-tree or a terminal node. The s-expression in plexity as well as in size overtime. Partly driven funQL besides telling how the semantics are com- by demand of larger and larger machine learning posed also tells how it could be derived by nesting model which require large data to train and partly words of natural language taken either as termi- to find a better semantic parser which would be nal or non-terminal node of the tree. The seman- able to parse complex and varied set of questions. tic parsing system by Cheng et al.(2017) sepa- Geo880: Introduced by (Zelle and Mooney, rate the generation of logical form and mapping 1996b) is a closed domain dataset of questions re- of lexicon to knowledge base entity and relation lated to US geography. Semantic parsing systems into two different stages. The logical form gener- based on encoder-decoder architecture seems to ation is done in a task-independent fashion. The favour close domain dataset. Thus purely evaluat- logical form obtained is called ungrounded logi- ing the ability of the semantic parser to generate cal form. The model generates the ungrounded correct logical form. logical form by applying sequence of actions in actions-set A = {NT,TER,RED} on the stack WebQuestions (Berant et al., 2013) introduced of s-expression, and choosing a word from an input- this dataset, collected from google suggest api. The buffer (used to store words in the sentence) ran- dataset has question and denotation paris, collected domly. The model is trained to learn a distribution from Freebase(Bollacker et al., 2008). (Yih et al., of possible action set and the word to be chosen 2016) later annotated the data with logical forms, from the input buffer conditioned on the sentence to show that annotation help in learning. and the state of the stack of s-expression. The stack is represented using a stack-LSTM (Dyer et al., GraphQuestions (Su et al., 2016) introduced this dataset which was obtained by presenting Aa- 2015), say (st). The state of input-buffer is en- coded using a Bi-LSTM (Hochreiter and Schmid- mazon Mechanical Turk workers with 500 Free- huber, 1997) and is adaptively weighted at each base graph queries and asking them to verbalise it into natural language. time step using the stack-state st. The two proba- bilities are given below: ComplexWebQuestions ComplexWebQues- tions v1.1 (Talmor and Berant, 2018) is a dataset p(at|a

Table 1: Benchmark Dataset

Li Dong and Mirella Lapata. 2018. Coarse-to-fine de- Rohit J Kate, Yuk Wah Wong, and Raymond J Mooney. coding for neural semantic parsing. In Proceedings 2005. Learning to transform natural to formal lan- of the 56th Annual Meeting of the Association for guages. In AAAI, pages 1062–1068. Computational Linguistics (Volume 1: Long Papers), pages 731–742. Eliyahu Kiperwasser and Yoav Goldberg. 2016. Sim- ple and accurate dependency parsing using bidirec- Chris Dyer, Miguel Ballesteros, Wang Ling, Austin tional lstm feature representations. Transactions Matthews, and Noah A Smith. 2015. Transition- of the Association for Computational Linguistics, based dependency parsing with stack long short- 4:313327. term memory. arXiv preprint arXiv:1505.08075. Percy Liang. 2013. Lambda dependency-based compo- Sherzod Hakimov, Christina Unger, Sebastian Walter, sitional semantics. arXiv preprint arXiv:1309.4408. and Philipp Cimiano. 2015. Applying semantic pars- ing to question answering over linked data: Address- Percy Liang. 2016. Learning executable semantic ing the lexical gap. In Natural Language Processing parsers for natural language understanding. Com- and Information Systems: 20th International Con- mun. ACM, 59(9):6876. ference on Applications of Natural Language to In- formation Systems, NLDB 2015, Passau, Germany, Percy Liang, Michael I Jordan, and Dan Klein. 2013. June 17-19, 2015, Proceedings, volume 9103. Learning dependency-based compositional seman- tics. Computational Linguistics, 39(2):389–446. Sepp Hochreiter and Jurgen¨ Schmidhuber. 1997. Long short-term memory. Neural computation, Thomas Lin, Oren Etzioni, et al. 2012. Entity linking 9(8):1735–1780. at web scale. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Konrad Hoffner,¨ Sebastian Walter, Edgard Marx, Ri- Web-scale Knowledge Extraction (AKBC-WEKEX), cardo Usbeck, Jens Lehmann, and Axel-Cyrille pages 84–88. Ngonga Ngomo. 2017. Survey on challenges of question answering in the semantic web. Semantic Moshe Looks, Marcello Herreshoff, DeLesley Web, 8(6):895–920. Hutchins, and Peter Norvig. 2017. Deep learning S. Hu, L. Zou, J. X. Yu, H. Wang, and D. Zhao. with dynamic computation graphs. 2018. Answering natural language questions by Jeff Mitchell and Mirella Lapata. 2010. Composition subgraph matching over knowledge graphs. IEEE in distributional models of semantics. Cognitive sci- Transactions on Knowledge and Data Engineering, ence, 34(8):1388–1429. 30(5):824–837. Sen Hu, Lei Zou, and Xinbo Zhang. 2018. A state- Ndapandula Nakashole, Gerhard Weikum, and Fabian transition framework to answer complex questions Suchanek. 2012. Patty: a taxonomy of relational over knowledge base. In Proceedings of the 2018 patterns with semantic types. In Proceedings of the Conference on Empirical Methods in Natural Lan- 2012 Joint Conference on Empirical Methods in Nat- guage Processing, pages 2098–2108. ural Language Processing and Computational Natu- ral Language Learning, pages 1135–1145. Robin Jia and Percy Liang. 2016. Data recombina- tion for neural semantic parsing. arXiv preprint Barbara Partee. 1975. Montague grammar and trans- arXiv:1606.03622. formational grammar. Linguistic Inquiry, 6(2):203– 300. Armand Joulin and Tomas Mikolov. 2015. Inferring algorithmic patterns with stack-augmented recurrent Jeff Pelletier. 2011. Compositionality. nets. Siva Reddy, Mirella Lapata, and Mark Steedman. 2014. Aishwarya Kamath and Rajarshi Das. 2018. A Large-scale semantic parsing without question- survey on semantic parsing. arXiv preprint answer pairs. Transactions of the Association for arXiv:1812.00978. Computational Linguistics, 2:377–392. Luzi Sennhauser and Robert C Berwick. 2018. Evalu- Luke S. Zettlemoyer and Michael Collins. 2005. Learn- ating the ability of lstms to learn context-free gram- ing to map sentences to logical form: Structured mars. arXiv preprint arXiv:1811.02611. classification with probabilistic categorial grammars. In Proceedings of the Twenty-First Conference on Valentin I Spitkovsky and Angel X Chang. 2012. A Uncertainty in Artificial Intelligence, UAI05, page cross-lingual dictionary for english wikipedia con- 658666, Arlington, Virginia, USA. AUAI Press. cepts. Qile Zhu, Xiyao Ma, and Xiaolin Li. 2019. Statistical Mark Steedman. 1996. Surface structure and interpre- learning for semantic parsing: A survey. Big Data tation. MIT press. Mining and Analytics, 2(4):217–239. Yu Su, Huan Sun, Brian Sadler, Mudhakar Srivatsa, Izzeddin Gur,¨ Zenghui Yan, and Xifeng Yan. 2016. On generating characteristic-rich question sets for qa evaluation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing, pages 562–572.

Yawei Sun, Lingling Zhang, Gong Cheng, and Yuzhong Qu. 2020. Sparqa: Skeleton-based seman- tic parsing for complex questions over knowledge bases. In AAAI, pages 8952–8959.

Zoltn Gendler Szab. 2017. Compositionality. In Ed- ward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, summer 2017 edition. Metaphysics Re- search Lab, Stanford University.

Alon Talmor and Jonathan Berant. 2018. Reparti- tioning of the complexwebquestions dataset. arXiv preprint arXiv:1807.09623.

Yuk Wah Wong and Raymond J Mooney. 2006. Learn- ing for semantic parsing with statistical machine translation. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Com- putational Linguistics, pages 439–446. Association for Computational Linguistics.

Yi Yang and Ming-Wei Chang. 2016. S-mart: Novel tree-based structured learning algorithms ap- plied to tweet entity linking. arXiv preprint arXiv:1609.08075.

Scott Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base.

Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201–206.

John M Zelle and Raymond J Mooney. 1996a. Learn- ing to parse database queries using inductive logic programming. In Proceedings of the national con- ference on artificial intelligence, pages 1050–1055.

John M Zelle and Raymond J Mooney. 1996b. Learn- ing to parse database queries using inductive logic programming. In Proceedings of the national con- ference on artificial intelligence, pages 1050–1055.