Schema Curation via Causal Association Rule Mining Noah Weber∗1, Anton Belyy∗1, Nils Holzenberger 1, Rachel Rudinger2, Benjamin Van Durme1 1Johns Hopkins University 2University of Maryland, College Park Abstract Event schemas are structured knowledge sources defining typical real-world scenarios (e.g., going to an airport). We present a frame- work for efficient human-in-the-loop construc- tion of a schema library, based on a novel mechanism for schema induction and a well- crafted interface that allows non-experts to ”program” complex event structures. Associ- ated with this work we release a machine read- able resource (schema library) of 232 detailed event schemas, each of which describe a dis- tinct typical scenario in terms of its relevant sub-event structure (what happens in the sce- nario), participants (who plays a role in the scenario), fine-grained typing of each partic- ipant, and the implied relational constraints between them. Our custom annotation inter- Figure 1: An example schema from our schema library, face, SchemaBlocks, and the event schemas induced from a skeleton mined by Causal ARM (Sec- are available online.12 tion 3) and fully fleshed out by an annotator using our SchemaBlocks annotation interface (Section 4). 1 Introduction What is implied by the invocation of a real-world scenario such as, say, a criminal trial? From one’s Mooney and DeJong, 1985; Mueller, 1999). While knowledge of the world, one makes a myriad of in- this facilitates quite impressive inferences, it re- ferences: that the scenario typically starts with the quires an onerous annotation effort carried out by defendant being accused and brought to court, that experts, and hence tends to be difficult to scale. On it likely contains events such as the presentation of the other end, one may employ data-driven methods evidence by a prosecutor, and that it ends with the to automatically induce this knowledge (Chambers judge announcing the final verdict. and Jurafsky, 2009, 2010; Balasubramanian et al., arXiv:2104.08811v1 [cs.CL] 18 Apr 2021 Though this type of scenario-level knowledge 2013; Rudinger et al., 2015), at the price of noise, is recognized as being vital for text understanding and a severe loss of detail in the type of knowledge (Schank and Abelson, 1977; Minsky, 1974; Bower extracted. et al., 1979; Abbott et al., 1985), explicitly anno- One may also try to take a semi-automatic ap- tating this knowledge in a way useful to language proach, taking advantage of both automatic and processing systems has proven to be a difficult task, annotator driven components. This is the approach subject to a seemingly fundamental trade-off. At taken in Wanzare et al.(2016) (see also Regneri one end, one may try to hand engineer this knowl- et al.(2010)) who use an initial human annotation edge in a richly detailed format (DeJong, 1983; to obtain various high quality event sequence de- scriptions for a target scenario, before using semi- *Equal∗ contribution. Order decided via wheel 1https://nlp.jhu.edu/demos/sb supervised clustering to aggregate these annota- 2https://bit.ly/3gwpx2X tions (Wanzare et al., 2017). In this paper we also adopt a semi-automatic ap- that make up this chain are drawn from a custom proach in order to facilitate the creation of a new structured ontology which defines a taxonomy of annotated resource of structured, machine readable events types (eg. an Acquit event, a Transportation event schemas. Each event schema characterizes a event), in addition to event-specific slot types to be real-world scenario, describing the events the sce- filled by arguments (eg. the Defendant or Trans- nario typically involves, the participants of these porter slot). While we use the term “chain” to de- events (as well as their role and typing informa- scribe the sequence of events defined in a schema, tion), and the implied relations between these par- the schemas presented here need not always be ticipants. An example visualization of an event ordered as a linear chain. In our schemas, subse- schema is given in Figure 1. Our workflow follows quences of events in the schema may be marked two main steps: either as occurring in a linear temporal order, in an arbitrary temporal order, or as forming mutually 1. Automatically induce what we term as skele- exclusive “branches.” ton schemas: argumentless event sequences that form the outline of an event schema. This Entities Entities fill the slots specified by each is done via a new automatic script induction event defined in the schema. The same entity can method which leverages and combines two (and usually will) be used to fill different slots recent advances in automatic script induction: across different events in order to indicate a co- an Association Rule Mining-based approach referring relationship. All entities may also take proposed in Belyy and Van Durme(2020), on types, either coarse grained types defined in the and a causal inference based scoring metric KAIROS ontology (including types such as person, proposed in Weber et al.(2020). commercial item, etc), or fine grained types defined by the Wikidata qnodes. Our annotated schemas 2. Using our SchemaBlocks interface, have hu- utilize both typing ontologies. man annotators “flesh out” the manually se- lected skeleton schemas by adding argument, Relations Relations between participating enti- role, typing, and relational information, in ad- ties are the last ingredient of the schemas defined dition to a name and description of the sce- here. These relations are also drawn from the nario the schema describes. KAIROS ontology. As of now, all relations are defined between two entities, each of which partic- Via this process, we create a resource of 232 ipate in at least one event defined in the schema. schemas, 150 of which are semi-automatically in- duced, with the rest being annotated from scratch 3 Automatic Induction of Skeleton from textual scenarios directly with SchemaBlocks. Schemas The content and format of this dataset will form the rest of the paper below. Our system first automatically induces what we term as skeleton schemas: argumentless event se- 2 The Anatomy of a Schema quences which form an outline of a potential event schema. It is a selected group of these skeleton Conceptualizations of what constitutes a “schema” schemas which are then passed forward to annota- differ across the literature. A schema in our re- tors to manually extend into full event schemas. source is constructed from three basic elements: By starting the schema creation with an auto- events, the entities that participate in these events, matic, data-driven step, we allow the data to “speak and the relations between these entities. The atomic for itself” with regards to what kinds of topics and types of events, entities, and relations in this work scenarios we might want to target given some spec- are defined by the DARPA KAIROS Phase 1 (v3.0) ified domain. The fact that the base of the schemas ontology. It consists of 67 event types, 24 coarse- has some connection to our targeted domain gives grained entity types, and 46 relation types.3 at least some assurance that the final schemas will Events The backbone for the meaning of a be applicable towards making common sense in- schema in this work is the temporally ordered chain ferences in our domain when used in real-world of events that it describes. The individual events applications. 3The full ontology definition can be accessed at this link: The automatic system for skeleton schema in- https://bit.ly/3mIWJoN duction combines two recent advances in schema induction: (1) an Association Rule Mining (ARM) inference based scoring approach presented in We- based algorithm presented in Belyy and Van Durme ber et al.(2020). (2020) which efficiently finds all event subse- The approach developed in Weber et al.(2020) quences which have enough support in the data, defines a scoring function, cscore(·; ·) which, tak- and (2) a script compatibility scoring model pre- ing in two events e1 and e2, outputs a score propor- sented in Weber et al.(2020) which finds high tional to the aptness of e2 following e1 in a script. quality subsequences output by the ARM method The approach builds upon reasonable assumptions and combines them together to form full skeleton on the data generating process to overcome con- schemas. We give a brief overview of each of these ceptual weaknesses in prior approaches and was approaches and how they are used in our system shown to output scores more in line with human below. judgments of script knowledge. We refer readers to the paper for details. 3.1 Association Rule Mining for Script Induction In order to create our skeleton schemas, we first use the trained scoring module from Weber et al. Belyy and Van Durme(2020) show how prior clas- (2020) to score all subsequences obtained via the sic work in automatic script induction (primarily process described in Section 3.1. Since the causal the line of work following Chambers and Jurafsky scoring module is only defined pairwise, we take (2008)) can be better recast as a problem of Asso- the following average as the assigned score for a ciation Rule Mining. ARM works with a dataset subsequence, S = (e1; :::; eN ), of length N: where each datapoint is a set of items. In the script induction setting, an item is an event and a data- N−1 N 2 X X point is the set of events appearing in a document score(S) = cscore(e ; e ) N(N − 1) i j and sharing some co-referring argument. The ARM i=1 j=i+1 approach consist of two distinct stages: 1.