About Inferences in a Crowdsourced Lexical-

Manel Zarrouk Mathieu Lafourcade Alain Joubert UM2-LIRMM UM2-LIRMM UM2-LIRMM 161 rue Ada 161 rue Ada 161 rue Ada 34095 Montpellier, FRANCE 34095 Montpellier, FRANCE 34095 Montpellier, FRANCE [email protected] [email protected] [email protected]

Abstract French version of WordNet, were built by auto- mated crossing of WordNet and other lexical re- Automatically inferring new relations sources along with some manual checking. Nav- from already existing ones is a way to igli (2010) constructed automatically BabelNet a improve the quality of a lexical network large multilingual lexical network from term co- by relation densification and error de- occurrences in Wikipedia. tection. In this paper, we devise such an approach for the JeuxDeMots lexi- A lexical-semantic network can contain lem- cal network, which is a freely avalaible mas, forms and multi-word expressions as lexical network for French. We first entry points (nodes) along with word meanings present deduction (generic to specific) and concepts. The idea itself of word senses in and induction (specific to generic) which the lexicographic tradition may be debatable in are two inference schemes ontologically the context of resources for semantic analysis, founded. We then propose abduction and we generally prefer to consider word us- as a third form of inference scheme, ages. A given polysemous word, as identified which exploits examples similar to a tar- by locutors, has several usages that might dif- get term. fer substantially from word senses as classically defined. A given usage can also in turn have 1 Introduction several deeper refinements and the whole set Building resources for Computational Linguis- of usages can take the form of a decision tree. tics (CL) is of crucial interest. Most of exist- For example, frigate can be a bird or a ship. A ing lexical-semantic networks have been built frigate boat can be distinguished as a modern > by hand (like for instance WordNet (Miller et ship with missiles and radar or an ancient vessel al., 1990)) and, despite that tools are generally with sails. In the context of a collaborative con- designed for consistency checking, the task re- struction, such a should be con- mains time consuming and costly. Fully auto- sidered as being constantly evolving and a gen- mated approaches are generally limited to term eral rule of thumb is to have no definite certi- co-occurrences as extracting precise semantic tude about the state of an entry. For a polysemic relations between terms from corpora remains term, some refinements might be just missing at really difficult. Meanwhile, crowdsourcing ap- a given time notwithstanding evolution of lan- proaches are flowering in CL especially with guage which might be very fast, especially in the advent of Amazon Mechanical Turk or in a technical domains. There is no way (unless by broader scope Wikipedia and Wiktionary, to cite inspection) to know if a given entry refinements the most well-known examples. WordNet is such are fully completed, and even if this question is a lexical network, constructed by hand at great really relevant. cost, based on synsets which can be roughly The building of a collaborative lexical network considered as concepts (Fellbaum, 1988). Eu- (or, in all generality, any similar resource) can roWordnet (Vossen., 1998) a multilingual ver- be devised according to two broad strategies. sion of WordNet and WOLF (Sagot., 2008) a First, it can be designed as a contributive system

174 Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 174–182, Gothenburg, Sweden, April 26-30 2014. c 2014 Association for Computational Linguistics like Wikipedia where people willingly add and ontology-based inference. OntoLearn (Velardi, complete entries (like for Wiktionary). Second, 2006) is a system that automatically build on- contributions can be made indirectly thanks to tologies of specific domains from texts and also games (better known as GWAP (vonAhn, 2008)) makes use of inferences. There have been and in this case players do not need to be aware also researchs on taxonomy induction based on that while playing they are helping building a WordNet (Snow, 2006). Although extensive work lexical resource. In any case, the built lexical on inference from texts or handcrafted resources network is not free of errors which are corrected has been done, almost none endogenously on along their discovery. Thus, a large number of lexical network built by the crowds. Most prob- obvious relations are not contained in the lexi- ably the main reason of that situation is the lack cal network but are indeed necessary for a high of such specific resources. quality resources usable in various NLP applica- In this article, we first present the principles tions and notably semantic analysis. For exam- behind the lexical network construction with ple, contributors seldom indicate that a particu- crowdsourcing and games with a purpose (also lar bird type can fly, as it is considered as an obvi- know as human-based computation games) and ous generality. Only notable facts which are not illustrated them with the JeuxDeMots (JDM) easily deductible are naturally contributed. Well project. Then, we present the outline of an elici- known exceptions are also generally contributed tation engine based on an inference engine using and take the form of a negative weight and anno- deduction, induction and especially abduction agent: 100 tated as such (for example, fly − ostrich schemes. An experimentation is then presented. −−−−−−−→ [exception: bird]). In order to consolidate the lexical network, 2 Crowdsourced Lexical Networks we adopt a strategy based on a simple in- For validating our approach, we used the JDM ference mechanism to propose new relations lexical network, which is constructed thanks to from those already existing. The approach is a set of associatory games (Lafourcade, 2007) strictly endogenous (i.e. self-contained) as it and has been made freely available by its au- doesn’t rely on any other external resources. In- thors. There is an increasing trend of using on- ferred relations are submitted either to contrib- line GWAPs (game with a purpose (Thaler et utors for voting or to experts for direct valida- al., 2011)) method for feeding such resources. tion/invalidation. A large percentage of the in- Beside manual or automated strategies, con- ferred relations has been found to be correct tributive approaches are flowering and becom- however, a non-negligible part of them are found ing more and more popular as they are both to be wrong and understanding why is both in- cheap to set up and efficient in quality. teresting and useful. The explanation process The network is composed of terms (as ver- can be viewed as a reconciliation between the in- tices) and typed relations (as links between ference engine and contributors who are guided vertices) with weight. It contains terms and through a dialog to explain why they found possible refinements. There are more than 50 the considered relation incorrect. The possible types of relations, that range from ontological causes for a wrong inferred relation may come (hypernym, hyponym), to lexical-semantic from three possible origins: false premises that (, antonym) and to semantic role were used by the inference engine, exception or (agent, patient, instrument). The weight of a confusion due to some polysemy. relation is interpreted as a strength, but not In (Sajous et al., 2013) an endogenous enrich- directly as a probability of being valid. The JDM ment of Wiktionary is done thanks to a crowd- network is not an ontology with some clean sourcing tool. A quite similar approach of us- hierarchy of concepts or terms. A given term ing crowdsourcing has been considered by (Ze- can have a substantial set of hypernyms that ichner, 2012) for evaluating inference rules that covers a large part of the ontological chain to are discovered from texts. In (Krachina, 2006), upper concepts. For example, hypernym(cat) = some specific inference methods are conducted {feline,mammal,living being,pet,vertebrate,...}. on text with the help of an ontology. Simi- Heavier weights associated to relations are those larly, (Besnard, 2008) capture explanation with felt by users as being the most relevant. The

175 1st January 2014, there are more than 6 700 000 and ways to avoid probably wrong inferences relations and roughly 310 000 lexical items in the can be done through a logical blocking: if JDM lexical network (according to the figures there are two distinct meanings for B that hold given by the game site: http://jeuxdemots.org). respectively the first and the second relation, To our knowledge, there is no other existing then most probably the inferred relation R(3) freely available crowdsourced lexical-network, is wrong (see figure 1) and hence should be especially with weighted relations, thus enabling blocked. Moreover, if one of the premises is strongly heuristic methods. tagged by contributors as true but irrelevant, then the inference is blocked.

3 Inferring with Deduction & Induction 1 (2) R : w B

Adding new relations to the JDM lexical network w 2 may rely on two components: (a) an inference (1) is-a : Bi Bj engine and (b) a reconciliator. The inference en- (5) R gine proposes relations as a contributor to be A (4) is-a C validated by other human contributors or ex- perts. In case of invalidation of an inferred re- (3) R? : w3 lation, the reconciliator is invoked to try to as- Figure 1: Triangular inference scheme where the sess why the inferred relation was found wrong. logical blocking based on the polysemy of the Elicitation here should be understood as the pro- central term B which has two distinct meanings cess to transform some implicit knowledge of the Bi and B j is applied. The two arrows without la- user into explicit relations in the lexical network. bel are those of word meanings. The core ideas about inferences in our engine are It is possible to evaluate a confidence level (on the following: an open scale) for each produced inference, in a way that dubious inferences can be eliminated • inferring is to derive new premises (as out through statistical filtering. The weight w relations between terms) from previously of an inferred relation is the geometric mean of known premises, which are existing rela- the weight of the premises (relations (1) and (2) tions; in Figure 1). If the second premise has a nega- • candidate inferences may be logically tive value, the weight is not a number and the blocked on the basis of the presence or the proposal is discarded. As the geometric mean is absence of some other relations; less tolerant to small values than the arithmetic • candidate inferences can be filtered out on mean, inferences which are not based on two the basis of a strength evaluation. rather strong relations (premises) are unlikely to pass. 3.1 Deduction Scheme R i s a R w(A C) = ( w(A − B) w(B C))1/2 −→ −−−→ × −→ Inferring by deduction is a top-down scheme w3 = (w1 w2)1/2 based on the transitivity of the relation is-a (hy- Inducing a⇒ transitive closure× over a knowledge pernym). If a term A is a kind of B and B holds base is not new, but doing so considering word some relation R with C, then we can expect that A meanings over a crowdsourced lexical network is holds the same relation type with C. The scheme an original approach. i s a can be formally written as follows: A − B 3.2 Induction Scheme R R ∃ −−−→ B C A C. As for the deductive inference, induction ex- ∧ ∃ −→ ⇒ −→ i s a For example, shark − fish and fish ploits the transitivity of the relation is-a. If a term has par t −−−→ − fin, thus we can expect that shark A is a kind of B and A holds a relation R with C, −−−−−−−→has par t then we might expect that B could hold the same − fin. The inference engine is applied −−−−−−−→ type of relation with C. More formally we can on terms having at least one hypernym (the i s a R R write: A − B A C B C. scheme could not be applied otherwise). Of ∃ −−−→ ∧i s a ∃ −→ ⇒ has−→par t course, this scheme is far too naive, especially For example, shark − fish and shark − −−−→ has par−−−−−→ t considering the resource we are dealing with jaw, thus we might expect that fish − jaw. −−−−−→ and may produce wrong relations (noise). In This scheme is a generalization inference. The effect, the central term B is possibly polysemous principle is similar to the one applied to the de-

176 duction scheme and similarly some logical and the lexical network with a negative weight and statistical filtering may be undertaken. annotated as exception. Relations that are ex- ceptions do not participate further as premises 1 w B for deducing. For the induction, in case we have two trusted relations, the reconciliator asks the (1) is-a :

Ai 2 R (5) is-a

w validators if the relation (A C) (which served −→ A as premise) is an exception relatively to the term B. If it is the case, in addition to storing the false (4) R R (2) R : Aj (3) R ? : inferred relation (B C) in the lexical network −→ R with a negative weight, the relation (A C) is w −→ 3 C annotated as exception. In the induction case, the exception is a true premise which leads to a Figure 2: (1) and (2) are the premises, and (3) false induced relation. In both cases of induc- is the induction proposed for validation. Term tion and deduction, the exception tag concerns A may be polysemous with meanings holding R always the relation (A C). Once this relation premises, thus inducing a probably wrong rela- −→ is annotated as an exception, it will not partic- tion. ipate as a premise in inferring generalized rela- The central term here A, is possibly polyse- tions (bottom-up model) but can still be used in mous (as shown in Figure 2). In that case, we inducing specified relations (top-down model). have the same polysemy issues than with the de- Errors due to Polysemy. If the central term duction, and the inference may be blocked. The (B for deduction and A for induction) present- estimated weight for the induced relation is: ing a polysemy is mentioned as polysemous R R 2 i s a w(B C) = (w(A C)) / w(A − B) in the network, the refinement terms ter m1, −→ −→ 2 −−−→ w2 = (w3) /w1 ter m ,... ter m are presented to the validator ⇒ 2 n 3.3 Performing Reconciliation so she/he can choose the appropriate one. The Inferred relations are presented to the validator validator can propose new terms as refinements to decide of their status. In case of invalida- if she/he is not satisfied with the listed ones (in- tion, a reconciliation procedure is launched in ducing the creation of new appropriate refine- order to diagnose the reasons: error in one of the ments). If there is no meta information indicat- premises (previously existing relations are false), ing that the term is polysemous, we ask first the exception or confusion due to polysemy (the in- validator if it is indeed the case. After this proce- ference has been made on a polysemous central dure, new relations will be included in the net- term). A dialog is initiated with the user (Cohen’s work with positive values and the inference en- kappa of 0.79). To know in which order to pro- gine will use them later on as premises. ceed, the reconciliator checks if the weights of 4 Abductive Inference the premises are rather strong or weak. The last inferring scheme is built upon abduc- Errors in the premises. We suppose that rela- tion and can be viewed as an example based tion (1) (in Figure 1 and 2) has a relatively low strategy. Hence abduction relies on similarity weight. The reconciliation process asks the val- between terms, which may be formalized in our idator if the relation (1) is true. It sets a negative context as sharing some outgoing relations be- weight to this relation if not so that the engine tween terms. The abductive inferring layout blocks further inferences. Else, if relation (1) is supposes that relations held by a term can be true, we ask about relation (2) and proceed as proposed to similar terms. Here, abduction first above if the answer is negative. Otherwise, we selects a set of similar terms to the target term A check the other cases (exception, polysemy). which are considered as proper examples. The Errors due to Exceptions. For the deduction, in outgoing relations from the examples which are case we have two trusted relations, the reconcil- not common with those of A are proposed as iation process asks the validators if the inferred potential relations for A and then presented for relation is a kind of exception relatively to the validation/invalidation to users. Unlike induc- term B. If it is the case, the relation is stored in tion and deduction, abduction can be applied on

177 terms with missing or irrelevant ontological rela- tions, and can generate ontological relations to ω max(25,mwogr({x }) 0.5) (4) 2 = i × be used afterward by the inference loop. where mwogr({xi }) is the mean of weights of out- 4.1 Abduction Scheme going relations from the set of examples xi . We note an outgoing relation as a 3-uple of a type If a term A is sharing at least δ1 relations, hav- t, a weight w and a target node n:R = t , w , n ing a weight over ω1, of the total of the rela- i 〈 i i i . For example, consider the term A having n tions R1, R2,... Rp toward terms T1, T2,... Tp 〉 outgoing relations. Amongst these relations, we with a group of examples x1, x2,... xn, we admit have for example: that this term has a degree of similarity strong has par t location enough with these examples. After building up beak − A & nest A. • ←−−− • ←−−− a set of examples on which we can apply our ab- We found 3 examples sharing those two rela- duction engine we proceed with the second part tions with the term A: has par t of the strategy. If we have at least δ2 examples xi beak − {ex1,ex2,ex3} • location←−−− holding a specific relation Rk0 weighting over ω2 nest {ex1,ex2,ex3} with a term B , more formally R = t , w , • ←−−− k k0 ω2 We consider these terms as a set of exam- 〈 ≥ Bk , we can suppose that the term A may hold ples to follow and similar to A. These examples 〉 this same relation Rk0 with the same target term have also other outgoing relations which are pro- Bk (figure 3). posed as potential relations for A. For example : R10 ? agent 1 car ac A {ex1,ex2} − fly {ex2} colorful • −−−→has par t • −−−→ − {ex1,ex2,ex3} feather R1 R10 • agent 1 −−−→ T1 x1 B1 {ex3} − sing • −−−→ R We infer that A can hold these relations and we R2 x2 20 T Rq0 ? B propose them for validation. 2 2 agent 1 has par t x3 A − fly ? A − feather ? • car−−−→ ac • −−−→agent 1 Rp Rq0 A colorful ? A − sing ? T B • −−−→ • −−−→ p q 4.2 Abduction Filtering xn Applying the abduction procedure crudely on the terms generates a lot of waste as a consid- Figure 3: Abduction scheme with examples xi erable amount of erroneous inferred relations. sharing relations with A and proposing new ab- Hence, we elaborated a filtering strategy to avoid ducted relations. having a lot of dubious proposed candidates. For On figure 3, we simplified thresholds to 2 this purpose, we define two different threshold for illustrative purpose. So, to be selected, the pairs. The first threshold pair (δ1, ω1) is used to examples x1 ,x2, x3,... xn must have at least 2 select proper examples x1,x2...xn and is defined common relations with A. A relation R0 must as follows: 1 q be hold by at least 2 examples to be proposed→ as δ max(3,nbogr(A) 0.1) (1) a potential relation for A. More clearly: 1 = × where nbogr(A) is the number of outgoing rela- R10 R10 x B and x B R0 : 2 tions from the term A. Ï 1 −−−→ 1 2 −−−→ 1 ⇒ 1 R10 ? ω max(25,mwogr(A) 0.5) (2) propose A B1 1 =⇒ −−−→ = × R20 x B R0 : 1 where mwogr(A) is the mean of weights of outgo- Ï n −−−→ 2 ⇒ 2 do not propose this relation. ing relations from A. The second threshold pair =⇒ Rq0 Rq0 Rq0 (δ2, ω2) is used to select proper candidate re- x1 Bq , x3 Bq and xn Bq lations from outgoing relations of the examples Ï −−−→ −−−→ −−−→ Rq0 : 3 R ,R ...R . ⇒ 10 20 q0 Rq0 ? propose A Bq δ max(3,{x } 0.1) (3) =⇒ −−−→ 2 = i × where {xi } is the cardinal of the set {xi }. For statistical filtering, we can act on the

178 Relation type Proposed % threshold (δ2, ω2) as the minimum number of is-a (x is a type of y) 6.1 examples x being R related with a target term i 0 has-parts (x is composed of y) 25.1 Bk . It is also possible to evaluate the weight of holonym (y specific of x) 7.2 the abducted relation as following: typical place (of x) 7.2 charac (x as characteristic y) 13.7 n,p,q R0 agent-1 (x can do y) 13.3 k 1 X 3 w(A Bk ) pw1w2w3 (5) instr-1 (x instrument of y) 1.7 −→ = nbR0 i 1,j 1,k 1 cd = = = patient-1 (x can be y) 1 place-1 (x located in the place y) 9.8 where nb is the number of the relations R0 Rcd0 place > action (y can be done in place x) 3.4 R j candidate to be proposed and w =A T & object > mater (x is made of y) 0.3 1 −−−→ j R R j k0 Table 1: Global percentages of relations pro- w2=xi T j & w3=xi Bk . This−−−→ filtering parameters−−−→ are adjustable ac- posed per type for deduction and induction. cording to the user’s requirements, so it can fulfil various expectations. Constant values in thresh- Deduction % valid % error Relation type rlvt rlvnt prem excep pol old formulas have been determined empirically. ¬ is-a 76% 13% 2% 0% 9% 5 Experimentation has-parts 65% 8% 4% 13% 10% holonym 57% 16% 2% 20% 5% We made an experiment with a unique run of typical place 78% 12% 1% 4% 5% charac 82% 4% 2% 8% 4% the deduction, induction and abduction engines agent-1 81% 11% 1% 4% 3% over the lexical network. Contributors have ei- instr-1 62% 21% 1% 10% 6% ther accepted or rejected a subset of those can- patient-1 47% 32% 3% 7% 11% place-1 72% 12% 2% 10% 6% didates during the normal course of their activ- place > action 67% 25% 1% 4% 3% ity. This experiment is for an evaluation pur- object > mater 60% 3% 7% 18% 12% pose only, as actually the system is running iter- Table 2: Number of propositions produced by atively along with contributors and games. The deduction and ratio of relations found as true or experiment has been done with the parameters false. given previously, which are determined empri- cally as those maximizing recall and precision In tables 2 and 3 are presented some evalu- (over a very small subset of the JDM lexical net- ations of the status of the inferences proposed work, around 1‰). by the inference engine through deduction and induction respectively. Inferences are valid for 5.1 Appliying Deductions and Inductions an overall of 80-90% with around 10% valid but has par ts We applied the inference engine on around not relevant (like for instance dog − pro- −−−−−−−→ 25 000 randomly selected terms having at least ton). We observe that error number in premises one hypernym or one hyponym and thus pro- is quite low, and nevertheless errors can be eas- duced by deduction more than 1 500 000 infer- ily corrected. Of course, not all possible errors ences and produced by induction over 360 000 are detected through this process. More inter- relation candidates. The threshold for filtering estingly, the reconciliation allows in 5% of the was set to a weight of 25. This value is relevant cases to identify polysemous terms and refine- as when a human contributor proposed relation ments. Globally false negatives (inferences voted is validated by experts, it is introduced with a de- false while being true) and false positives (infer- fault weight of 25. ences voted true while being false) are evaluated The transitive is-a (Table1) is not very produc- to less than 0.5%. tive which might seems surprising at first glance. For the induction process, the relation is-a is In fact, the is-a relation is already quite popu- not obvious (a lexical network is not reductible lated in the network, and as such, fewer new re- to an ontology and multiple inheritance is possi- lations can be inferred. The figures are inverted ble). Result seems about 5% better than for the for some other relations that are not so well pop- deduction process: inferences are valid for an ulated in the lexical network but still are poten- overall of 80-95%. The error number is very low. tially valid. The has-parts relation and the agent The main difference with the deduction process semantic role (the agent-1 relation) are by far the is on errors due to polysemy which is lower with most productive types. the induction process.

179 To try to assess a baseline for those results, wrong abducted relations. we compute the full closure of the lexical net- Figure 4 presents two types of data: (1) the work, i.e. we produce iteratively all possible can- percentage of correct abducted relations accord- didate relations until no more could be found, ing to the number of examples required to pro- each candidate being considered as correct and duce the inference, and (2) the proportion be- participating to the process. We got more than tween the produced relations and the total of 6 000 000 relations out of which 45% were wrong 107 416 relations according to the minimal num- (evaluation on around 1 000 candidates ran- ber of examples allowed. What can clearly be domly chosen). seen is that when the number of required ex- amples is increased, the ratio of correct abduc- 5.2 Unleashing the Abductive Engine tions increases accordingly, but the number of We applied systematically the abduction engine proposed relations dramaticaly falls. The num- on the lexical items contained in the network, ber of abductions is an inverse power law of the and produce 629 987 abducted relations out of number of examples required. which 137 416 were not already existing in the network. Those 137 416 are candidate relations concerning 10 889 distinct lexical entries, hence producing a mean of around 12 new relations per entry. The distribution of the proposed re- lations follows a power law, which is not totally surprising as the relation distribution in the lex- ical network is by itself governed by such a dis- tribution. Those figures indicate that abduction seems to be still quite productive in terms of raw candidates, even not relying on ontological ex- Figure 4: Production of abducted relations and isting relations. percentage of correctness according to examples The table 4 presents the number of relations number. proposed by the inference engine through ab- duction. The different relation types are var- At 3 examples, only 40% of the proposed re- iously productive, and this is mainly due to lations are correct, and with a minimum of 6 the number of existing relations and the dis- examples, more than 3/4 of the proposals are tribution of their type. The most productive deemed correct. The balanced F-score is opti- relation is has-part and the least one is holo mal at the intersection of both curves, that is to (holonym/whole). Correct relations represent say for at least 4 examples. around 80% of the relations that have been eval- In figure 5, is showed the mean number of uated (around 5.6% of the total number of pro- new relations during an iteration of the infer- duced relations). ence engine on abduction. Between two runs, One suprising fact, is that the 80% seem to users and validators are invited to accept or re- be quite constant notwithstanding the relation ject abducted relations. This process is done type, the lowest value being 77% (for instr-1 at their discretion and users may leave some which is the relation specifying what can be done propostions unvoted. Experiments showed that with x as an instrument) and the highest being users are willing to validate strongly true rela- 85% (for action-place which is the relation asso- tions and invalidate clearly false relations. Rela- ciating for an action the typical locations where tions whose status may be difficult are more of- it can occur). The abduction process is not onto- ten left aside than other easiest proposals. The logically based, and hence does not rely on the third run is the most productive with a mean of generic (is-a) or specific (hyponym) relations, almost 20 new abducted relations. After 3 runs, but on the contrary on any set of examples that the abductive process begins to be less produc- seems to be alike the target term. The apparent tive by attrition of new possible candidates. No- stability of 80% correct abducted relations may tice that the abduction process may, on subse- be a positive consequence of relying on a set of quent runs, remove some previsouly done pro- examples, with a potentially irreductible of 20% posals and as such is not monotonous.

180 ments showed that abduction is quite produc- tive (compared to deduction and induction), and is stable in correctness. User evaluation showed that wrong abducted relations (around 20% of all abducted relations) are still logically sound and could not have been dismissed a priori. Ab- Figure 5: Mean number of new relations rela- duction can conclusively be considered as a use- tively to runs in iterated abduction. full and efficient tool for relation inference. The main difficulty relies in setting the various pa- 5.3 Figures on Reconciliation rameter in order to achieve a fragile tradeoff be- Reconciliation in abduction is simpler than in tween an overrestrictive filter (many false nega- deduction or induction, as the potential adverse tives, resulting in information losses) and the op- effect of polysemy is counterbalanced by the posite (many false postive, more human effort). statistical approach implemented by the large The elicitation engine we presented through number of examples (when available). The rec- schemas based on deduction, induction and ab- onciliation in the case of abduction is to deter- duction is an efficient error detector, a polysemy mine if the wrong proposal has been produced identifier but also a classifier by abduction. The logically considering the support examples. In actions taken during the reconciliation forbid 97% of the cases, the wrong abducted relation an inference proven wrong or exceptional to be has been qualified as wrong but logical by vot- inferred again. Each inference scheme is sup- ers or validators. For examples: Boeing ported by the two others, and if a given inference has par t pl ace• 747 − propeller* whale lake * has been produced by more than one of these −−−−−→agent 1 • −−−→ pelican − sing *. All those wrong ab- three schemas, it is almost surely correct. • −−−−−→ ducted relations given as examples above might Induction % valid % error have been correct. Considering the examples ex- Relation types rlvt rlvnt prem excep pol ¬ ploited to produce the candidates, in those cases is-a - - - - - there is no possible way to guess those relations has-parts 78% 10% 3% 2% 7% are wrong. This is even reinforced by the fact that holonyme 68% 17% 2% 8% 5% typical place 81% 13% 1% 2% 3% abduction does not rely on ontological relations, charac 87% 6% 2% 2% 3% which in some cases could have avoided wrong agent-1 84% 12% 1% 2% 1% abduction. However, abduction compared to in- instr-1 68% 24% 1% 4% 3% patient-1 57% 36% 3% 2% 2% duction and deduction, can be used on terms place-1 75% 16% 2% 5% 2% that do not hold ontological relations, either they place > action 67% 28% 1% 3% 1% are missing or they are not relevant (for verbs, in- object > mater 75% 10% 7% 5% 3% stances...). Table 3: Number of propositions produced by in- duction and ratio of relations found as true or 6 Conclusion false. We presented some issues in inferring new rela- Abduction #prop #eval (%) True (%) False (%) tions from existing ones to consolidate a lexical- is-a 7141 421 (5.9) 343 (81.5) 78 (18.5) semantic network built with games and user has-parts 26517 720 (2.7) 578 (80.3) 142 (19.7) contributions. New inferred relations are stored holo 1592 153 (9.6) 124 (81) 29 (18.9) agent 7739 298 (3.9) 236 (79.2) 62 (20.8) to avoid having to infer them again and again dy- place 17148 304 (1.8) 253 (83.2) 51 (16.8) namically. To be able to enhance the network instr 10790 431 (4) 356 (82.6) 75 (17.4) quality and coverage, we proposed an elicitation charac 7443 319 (4.3) 251 (78.7) 68 (21.3) agent-1 18147 955 (5.3) 780 (81.7) 175 (18.3) engine based on inferences (induction, deduc- instr-1 11867 886 (7.5) 682 (77) 204 (23) tion and abduction) and reconciliation. If an in- place-1 14787 1106 (7.5) 896 (81) 210 (19) ferred relation is proven wrong, a reconciliation place>act 8268 270 (3.3) 214 (79.3) 56 (20.7) process is conducted in order to identify the un- act>place 5976 170 (2.8) 145 (85.3) 25 (14.7) Total 137416 6033 (4.3) 4858 (81) 1175 (19) derlying cause and solve the problem. The ab- Table 4: Number of propositions produced by duction scheme does not rely on the ontologi- abduction and ratio of relations found as true or cal relation (is-a) but merely on examples that false. are similarly close to the target term. Experi-

181 References Siorpaes, K. and Hepp, M. 2008. Games with a Pur- pose for the Semantic Web. in IEEE Intelligent Sys- von Ahn, L. and Dabbish, L. 2008. Designing games tems, number 3, volume 23.p 50-60. with a purpose. in Communications of the ACM, number 8, volume 51. p 58-67. Snow, R. Jurafsky, D., Y. Ng., A. 2006. Semantic tax- onomy induction from heterogenous evidence. in Besnard, P. Cordier, M.-O., and Moinard, Y. 2008. Proceedings of COLING/ACL 2006, 8 p. Ontology-based inference for causal explanation. Integrated Computer-Aided Engineering , IOS Thaler, S and Siorpaes, K and Simperl, E. and Hofer, Press, Amsterdam, Vol. 15 , No. 4, 351-367, 2008. C. 2011. A Survey on Games for Knowledge Acqui- sition. STI Technical Report, May 2011.19 p. Fellbaum, C. and Miller, G. 1988. (eds) WordNet. The MIT Press. Velardi, P. Navigli, R. Cucchiarelli, A. Neri, F. 2006. Evaluation of OntoLearn, a methodology for Auto- Krachina, O., Raskin, V. 2006. Ontology-Based Infer- matic Learning of Ontologies. in Ontology Learn- ence Methods. CERIAS TR 2006-76, 6 p. ing and Population, Paul Buitelaar Philipp Cim- miano and Bernardo Magnini Editors, IOS press Lafourcade, M. 2007. Making people play for Lex- 2006). ical Acquisition. In Proc. SNLP 2007, 7th Sym- posium on Natural Processing. Pattaya, Vossen, P. 2011. EuroWordNet: a multilingual Thailande, 13-15 December. 8 p. with lexical semantic networks. Kluwer Academic Publishers.Norwell, MA, USA.200 p. Lafourcade, M., Joubert, A. 2012. Long Tail in Weighted Lexical Networks. In proc of Cogni- Zeichner, N., Berant J., and Dagan I. 2012. Crowd- tive Aspects of the (CogAlex-III), COLING, sourcing Inference-Rule Evaluation. in proc of ACL Mumbai, India, December 2012. 2012 (short papers).

Lieberman, H, Smith, D. A and Teeters, A 2007. Common consensus: a web-based game for col- lecting commonsense goals. In Proc. of IUI, Hawaii,2007.12 p .

Marchetti, A and Tesconi, M and Ronzano, F and Mosella, M and Minutoli, S. 2007. SemKey: A Se- mantic Collaborative Tagging System. in Procs of WWW2007, Banff, Canada. 9 p.

Mihalcea, R and Chklovski, T. 2003. Open MindWord Expert: Creating large annotated data collections with web users help.. In Proceedings of the EACL 2003, Workshop on Linguistically Annotated Cor- pora (LINC). 10 p.

Miller, G.A. and Beckwith, R. and Fellbaum, C. and Gross, D. and Miller, K.J. 1990. Introduction to WordNet: an on-line lexical database. Interna- tional Journal of . Volume 3, p 235- 244.

Navigli, R and Ponzetto, S. 2010. BabelNet: Build- ing a very large multilingual semantic network. in Proceedings of the 48th Annual Meeting of the As- sociation for Computational Linguistics, Uppsala, Sweden, 11-16 July 2010.p 216-225.

Sagot, B. and Fier, D. 2010. Construction d’un word- net libre du français à partir de ressources multi- lingues. in Proceedings of TALN 2008, Avignon, France, 2008.12 p.

Sajous, F., Navarro, E., Gaume, B,. Prévot, L. and Chudy, Y. 2013. Semi-Automatic Enrichment of Crowdsourced Synonymy Networks: The WISIG- OTH system applied to Wiktionary. Language Re- sources & Evaluation, 47(1), pp. 63-96.

182