Articles WatsonPaths: Scenario-Based Question Answering and Inference over Unstructured Information Adam Lally, Sugato Bagchi, Michael A. Barborak, David W. Buchanan, Jennifer Chu-Carroll, David A. Ferrucci, Michael R. Glass, Aditya Kalyanpur, Erik T. Mueller, J. William Murdock, Siddharth Patwardhan, John M. Prager n We present WatsonPaths, a novel BM Watson is a question-answering system that takes nat - system that can answer scenario-based ural language questions as input and produces precise questions. These include medical ques - Ianswers along with accurate confidences as output (Fer - tions that present a patient summary rucci et al. 2010). In 2011, in a modified version of the quiz and ask for the most likely diagnosis or show Jeopardy!, Watson defeated two of the best human play - most appropriate treatment. Watson - Paths builds on the IBM Watson ques - ers. tion-answering system. WatsonPaths Jeopardy! questions are usually factoid questions — the breaks down the input scenario into answer and supporting evidence are usually stated explicitly individual pieces of information, asks in some document in the corpus. While in practice we may relevant subquestions of Watson to con - retrieve multiple redundant documents, in principle the clude new information, and represents answer could be expressed succinctly in one. The main chal - these results in a graphic model. Proba - lenges for a factoid question-answering system are retrieving bilistic inference is performed over the the correct document, and then extracting the correct answer graph to conclude the answer. On a set of medical test preparation questions, from the document. At the core of Watson’s question answer - WatsonPaths shows a significant ing is a suite of algorithms that match passages containing improvement in accuracy over multiple candidate answers to the original question. These algorithms baselines. have been described in a series of articles (Chu-Carroll et al. 2012; Ferrucci 2012; Gondek et al. 2012; Lally et al. 2012; McCord, Murdock, and Boguraev 2012; Murdock et al. 2012a; 2012b). In some important applications, however, questions do not have this “factoid” character. Consider the Copyright © 2017, Association for the Advancement of Artificial Intelligence. All rights reserved. ISSN 0738-4602 Summer 2017 59 Articles “A 32-year-old woman with type 1 diabetes mellitus has had progressive renal failure... Her hemoglobin concentration is 9 g/dL... A blood smear shows normochromic, normocytic cells. What is the problem? Patient’s hemoglobin conc. Patient’s blood smear Patient has renal failure is 9 g/dL [low] shows normocytic cells Evidence: “Low hemoglobin conc. indicates anemia.” Evidence: “Erythropoietin is produced in the kidneys.” Patient has anemia Evidence: “Normocytic anemia is a type of anemia Patient is at risk for with normal red blood Erythropoietin de!ciency cells.” Patient has normocytic anemia Evidence: “Erythropoietin de!ciency is a cause of normocytic anemia.” Most likely cause of low hemoglobin conc. is Erythropoietin de!ciency Figure 1. A Simple Diagnosis Graph for a Patient with Erythropoietin Deficiency. following questions, one from medicine and one knowledge to a specific case, as in a medical scenario from taxation: about a patient. A 32-year-old woman with type 1 diabetes mellitus has Before beginning work on automated scenario- had progressive renal failure. Her hemoglobin con - based question answering, we investigated how centration is 9 g/dL. A blood smear shows nor - humans solve such questions. We asked domain mochromic, normocytic cells. What is the problem? experts to describe their approach to solving a set of I inherited real-estate from a relative who died 5 years scenario-based questions in the medical domain. An ago via a trust that was created before his death. The example is shown in figure 1. Many drew a graph of property was sold this year after dissolution of the initial signs and symptoms leading to their most like - trust, and the money was put in a Roth-IRA. Which ly possible causes and connecting them to a final tax form(s) do I need to file? conclusion. This motivated us to look into graph- We will call these types of questions scenario-based based methods as a way of answering scenario-based questions. In these types of questions, it is not gener - questions automatically. ally the case that the answer and supporting evidence In this article, we describe WatsonPaths, a system can be contained in one document. Rather, for many that builds on Watson to answer scenario-based ques - scenario-based questions, information from multiple tions. The core idea is to break the question down documents and other sources must generally be into parts, over which we can ask and answer factoid retrieved and then integrated to answer the questions subquestions using Watson, then integrate these properly. Furthermore, we must often apply general answers into a graphic model that can be used to 60 AI MAGAZINE Articles answer the larger scenario-based question. We show Scenario Analysis that WatsonPaths not only outperforms a baseline The first step in the pipeline is scenario analysis, system that uses simple information retrieval, but where we identify factors in the input scenario that also outperforms its own subcomponent, Watson, in may be of importance. In the medical domain, the answering a set of scenario-based questions from the factors may include demographics (“32-year old medical domain. woman”), preexisting conditions (“type 1 diabetes mellitus”), signs and symptoms (“progressive renal WatsonPaths Medical Use Case failure”), and test results (“hemoglobin concentra - tion is 9 g/dL,” “normochromic cells,” “normocytic Although WatsonPaths is intended as a domain-gen - cells”). The extracted factors become nodes in a eral technology for scenario-based question answer - graph structure called the assertion graph, on which ing, we decided to start by focusing our attention on the remaining steps of the process will operate. the medical domain. We focused on the problem of patient scenario analysis, where the goal is typically Node Prioritization a diagnosis or a treatment recommendation. The next step is node prioritization, where we decide To explore this kind of problem solving, we which nodes in the graph are most important for obtained a set of medical test preparation questions. solving the problem. In a small scenario like this These are multiple-choice medical questions based example, we may be able to explore everything, but on an unstructured or semistructured natural lan - in general this will not be the case. Factors that affect guage description of a patient. Although Watson - the priority of a node may include the system’s con - Paths is not restricted to multiple-choice questions, fidence in the node assertion or the system’s estima - we saw multiple-choice questions as a good starting tion of how fruitful it would be to expand a node. For point for development. Many of these questions example, normal test results and demographic infor - involve diagnosis, either as the entire question, as in mation are generally less useful for starting a diagno - the previous medical example, or as an intermediate sis than symptoms and abnormal test results. step, as in the following example: A 63-year old patient is sent to the neurologist with a Relation Generation clinical picture of resting tremor that began 2 years The relation-generation step builds the assertion ago. At first it was only on the left hand, but now it graph. We do this primarily by asking Watson ques - compromises the whole arm. At physical exam, the tions about the factors. In medicine we want to know patient has an unexpressive face and difficulty in the causes of the findings and abnormal test results walking, and a continuous movement of the tip of the that are consistent with the patient’s demographic first digit over the tip of the second digit of the left information and normal test results. Given the sce - hand is seen at rest. What part of his nervous system nario in the Introduction, we could ask, “What does is most likely affected? type 1 diabetes mellitus cause?” We use a medical For this question, it is useful to diagnose that the ontology to guide the process of formulating sub - patient has Parkinson’s disease before determining questions to ask Watson. Relevant factors may also be which part of his nervous system is most likely affect - combined to form a single, more targeted question. ed. These multistep inferences are a natural fit for the Because in this step we want to emphasize recall, we graphs that WatsonPaths constructs. In this example, take several of Watson’s highly ranked answers. The the diagnosis is the missing link on the way to the exact number of answers taken, or the confidence final answer. threshold, are parameters that must be tuned. Given a set of answers, we add them to the graph as nodes, Scenario-Based Question Answering with edges from nodes that were used in questions to nodes that were answers. The edge is labeled with the In scenario-based question answering, the system predicate used to formulate the question (like causes receives a scenario description that ends with a or indicates ), and the strength of the edge is initially punch line question. For instance, the punch line set to Watson’s confidence in the answer. question in the Parkinson’s example is “What part of Although Watson is the primary way we add edges his nervous system is most likely affected?” Instead of to the graph, WatsonPaths allows for any number of treating the entire scenario as one monolithic ques - relation generator components to post edges to the tion as would Watson, WatsonPaths explores multi - graph. For instance, we apply term matchers to pairs ple facts in the scenario in parallel and reasons with of nodes, and post a relation between nodes that the results of its exploration as a whole to arrive at match.