Arxiv:2006.10022V3 [Cs.AI] 2 Feb 2021 Cus 2015; Sakaguchi Et Al
Total Page:16
File Type:pdf, Size:1020Kb
Conversational Neuro-Symbolic Commonsense Reasoning Forough Arabshahi1*, Jennifer Lee1, Mikayla Gawarecki2 Kathryn Mazaitis2, Amos Azaria3, Tom Mitchell2 1Facebook, 2Carnegie Mellon University, 3Ariel University fforough, [email protected], fmgawarec, [email protected], [email protected], [email protected] Abstract commands from human subjects, revealed that humans often under-specify conditions in their statements; perhaps because In order for conversational AI systems to hold more natural and broad-ranging conversations, they will require much more they are used to speaking with other humans who possess the commonsense, including the ability to identify unstated pre- common sense needed to infer their more specific intent by sumptions of their conversational partners. For example, in making presumptions about their statement. the command “If it snows at night then wake me up early The inability to make these presumptions makes it chal- because I don’t want to be late for work” the speaker relies on lenging for computers to engage in natural sounding conver- commonsense reasoning of the listener to infer the implicit pre- sations with humans. While conversational AI systems such sumption that they wish to be woken only if it snows enough as Siri, Alexa, and others are entering our daily lives, their to cause traffic slowdowns. We consider here the problem of conversations with us humans remains limited to a set of pre- understanding such imprecisely stated natural language com- programmed tasks. We propose that handling unseen tasks mands given in the form of if-(state), then-(action), because- (goal) statements. More precisely, we consider the problem requires conversational agents to develop common sense. of identifying the unstated presumptions of the speaker that Therefore, we propose a new commonsense reasoning allow the requested action to achieve the desired goal from benchmark for conversational agents where the task is to the given state (perhaps elaborated by making the implicit pre- infer commonsense presumptions in commands of the form sumptions explicit). We release a benchmark data set for this “If state holds Then perform action Because I want to task, collected from humans and annotated with commonsense achieve goal .” The if-(state), then-(action) clause arises presumptions. We present a neuro-symbolic theorem prover when humans instruct new conditional tasks to conversational that extracts multi-hop reasoning chains, and apply it to this agents (Azaria, Krishnamurthy, and Mitchell 2016; Labutov, problem. Furthermore, to accommodate the reality that current AI commonsense systems lack full coverage, we also present Srivastava, and Mitchell 2018). The reason for including the an interactive conversational framework built on our neuro- because-(goal) clause in the commands is that some pre- symbolic system, that conversationally evokes commonsense sumptions are ambiguous without knowing the user’s pur- knowledge from humans to complete its reasoning chains. pose, or goal. For instance, if Alice’s goal in the previous example was to see snow for the first time, Bob would have presumed that even a snow flurry would be excuse enough to 1 Introduction wake her up. Since humans frequently omit details when stat- Despite the remarkable success of artificial intelligence (AI) ing such commands, a computer possessing common sense and machine learning in the last few decades, commonsense should be able to infer the hidden presumptions; that is, the reasoning remains an unsolved problem at the heart of AI additional unstated conditions on the If and/or Then portion (Levesque, Davis, and Morgenstern 2012; Davis and Mar- of the command. Please refer to Tab. 1 for some examples. arXiv:2006.10022v3 [cs.AI] 2 Feb 2021 cus 2015; Sakaguchi et al. 2019). Common sense allows us In this paper, in addition to the proposal of this novel task humans to engage in conversations with one another and to and the release of a new dataset to study it, we propose a novel convey our thoughts efficiently, without the need to spec- initial approach that infers such missing presumptions, by ex- ify much detail (Grice 1975). For example, if Alice asks tracting a chain of reasoning that shows how the commanded Bob to “wake her up early whenever it snows at night” so action will achieve the desired goal when the state holds. that she can get to work on time, Alice assumes that Bob Whenever any additional reasoning steps appear in this rea- will wake her up only if it snows enough to cause traffic soning chain, they are output by our system as assumed slowdowns, and only if it is a working day. Alice does not implicit presumptions associated with the command. For our explicitly state these conditions since Bob makes such pre- reasoning method we propose a neuro-symbolic interactive, sumptions without much effort thanks to his common sense. conversational approach, in which the computer combines its A study, in which we collected such if(state)-then(action) own commonsense knowledge with conversationally evoked *work done when FA and JL were at Carnegie Mellon University. knowledge provided by a human user. The reasoning chain Copyright © 2021, Association for the Advancement of Artificial is extracted using our neuro-symbolic theorem prover that Intelligence (www.aaai.org). All rights reserved. learns sub-symbolic representations (embeddings) for logical Table 1: Statistics of if-(state), then-(action), because-(goal) commands collected from a pool of human subjects. The table shows four distinct types of because-clauses we found, the count of commands of each type, examples of each and their corresponding commonsense presumption annotations. Restricted domain includes commands whose state is limited to checking email, calendar, maps, alarms, and weather. Everyday domain includes commands concerning more general day-to-day activities. Annotations are tuples of (index, presumption) where index shows the starting word index of where the missing presumption should be in the command, highlighted with a red arrow. Index starts at 0 and is calculated for the original command. if then because Annotation: Domain clause clause clause Example Commonsense Presumptions Count # If it’s going to rain in the afternoon ( · ) Restricted domain state action goal # (8, and I am outside) 76 then remind me to bring an umbrella ( · ) (15, before I leave the house) because I want to remain dry # If I have an upcoming bill payment ( · ) Restricted domain state action anti-goal # (7, in the next few days) 3 then remind me to pay it ( · ) (13, before the bill payment deadline) because I don’t want to pay a late fee # If my flight ( · ) is from 2am to 4am Restricted domain state action modifier # (3, take off time) 2 then book me a supershuttle ( · ) (13, for 2 hours before my flight take off time) because it will be difficult to find ubers. # If I receive emails about sales on basketball shoes ( · ) Restricted domain state action conjunction # (9, my size) 2 then let me know ( · ) (13, there is a sale) because I need them and I want to save money. SUM 83 # If there is an upcoming election ( · ) (6, in the next few months) # # Everyday domain state action goal then remind me to register ( · ) and vote ( · ) (6, and I am eligible to vote) 55 because I want my voice to be heard. (11, to vote), (13, in the election) If it’s been two weeks since my last call with my mentee # and I don’t have an upcoming appointment with her ( · ) (21, in the next few days) Everyday domain state action anti-goal # 4 then remind me to send her an email ( · ) (29, to schedule our next appointment) because we forgot to schedule our next chat # If I have difficulty sleeping ( · ) Everyday domain state action modifier then play a lullaby (5, at night) 12 because it soothes me. # If the power goes out ( · ) Everyday domain state action conjunction then when it comes back on remind me to restart the house furnace (5, in the Winter) 6 because it doesn’t come back on by itself and I want to stay warm SUM 77 statements, making it robust to variations of natural language 1990), ConceptNet (Liu and Singh 2004; Havasi, Speer, and encountered in a conversational interaction setting. Alonso 2007; Speer, Chin, and Havasi 2017) and ATOMIC (Sap et al. 2019; Rashkin et al. 2018) are examples of such Contributions This paper presents three main contribu- KBs (see (Davis and Marcus 2015) for a comprehensive list). tions. 1) We propose a benchmark task for commonsense Recently, Bosselut et al. (2019) proposed COMET, a neural reasoning in conversational agents and release a data set con- knowledge graph that generates knowledge tuples by learning taining if-(state), then-(action), because-(goal) commands, on examples of structured knowledge. These KBs provide annotated with commonsense presumptions. 2) We present background knowledge for tasks that require common sense. CORGI (COmmonsense ReasoninG by Instruction), a system However, it is known that knowledge bases are incomplete, that performs soft logical inference. CORGI uses our pro- and most have ambiguities and inconsistencies (Davis and posed neuro-symbolic theorem prover and applies it to extract Marcus 2015) that must be clarified for particular reasoning a multi-hop reasoning chain that reveals commonsense pre- tasks. Therefore, we argue that reasoning engines can benefit sumptions. 3) We equip CORGI with a conversational inter- greatly from a conversational interaction strategy to ask hu- action mechanism that enables it to collect just-in-time com- mans about their missing or inconsistent knowledge. Closest monsense knowledge from humans. Our user-study shows (a) in nature to this proposal is the work by Hixon, Clark, and Ha- the plausibility of relying on humans to evoke commonsense jishirzi (2015) on relation extraction through conversation for knowledge and (b) the effectiveness of our theorem prover, question answering and Wu et al.