TALEN: Tool for of Low-resource ENtities

Stephen Mayhew Dan Roth University of Pennsylvania University of Pennsylvania Philadelphia, PA Philadelphia, PA [email protected] [email protected]

Abstract work on non-expert annotators for natural lan- guage tasks (Snow et al., 2008), where the anno- We present a new web-based interface, tators lack specific skills related to the task, there TALEN, designed for named entity an- has been little to no work on situations where an- notation in low-resource settings where notators, expert or not, do not speak the language. the annotators do not speak the language. To this end, we present a web-based interface de- To address this non-traditional scenario, signed for users to annotate text quickly and easily TALEN includes such features as in-place in a language they do not speak. lexicon integration, TF-IDF token statis- TALEN aids non-speaker annotators1 with sev- tics, Internet search, and entity propaga- eral different helps and nudges that would be un- tion, all implemented so as to make this necessary in cases of a native speaker. The main difficult task efficient and frictionless. We features, described in detail in Section2, are a conduct a small user study to compare Named Entity (NE) specific interface, entity prop- against a popular annotation tool, show- agation, lexicon integration, token statistics infor- ing that TALEN achieves higher precision mation, and Internet search. and recall against ground-truth annota- The tool operates in two separate modes, each tions, and that users strongly prefer it over with all the helps described above. The first mode the alternative. displays atomic documents in a manner analogous to nearly all prior annotation . The second TALEN is available at: github.com/CogComp/talen. mode operates on the sentence level, patterned on bootstrapping with a human in the loop, and de- 1 Introduction signed for efficient discovery and annotation. In addition to being useful for non-speaker an- Named entity recognition (NER), the task of find- notations, TALEN can be used as a lightweight in- ing and classifying named entities in text, has been spection and annotation tool for within-language well-studied in English, and a select few other lan- named entity annotation. TALEN is agnostic to la- guages, resulting in a wealth of resources, par- belset, which means that it can also be used for a ticularly annotated training data. But for most wide variety of sequence tagging tasks. languages, no training data exists, and annotators who speak the language can be hard or impossible 2 Main Features to find. This low-resource scenario calls for new In this section, we describe the main features indi- methods for gathering training data. Several works vidually in detail. address this with automatic techniques (Tsai et al., 2016; Zhang et al., 2016; Mayhew et al., 2017), Named Entity-Specific Interface but often a good starting point is to elicit manual The interface is designed specifically for Named from annotators who do not speak the Entity (NE) annotation, where entities are rela- target language. Language annotation strategies and software 1We use ‘non-speaker’ to denote a person who does not speak or understand the language, in contrast with ‘non- have historically assumed that annotators speak native speaker’, which implies at least a shallow understand- the language in question. Although there has been ing of the language.

80 Proceedings of the 56th Annual Meeting of the Association for Computational -System Demonstrations, pages 80–86 Melbourne, Australia, July 15 - 20, 2018. c 2018 Association for Computational Linguistics Figure 1: Document-based annotation screen. A romanized document from an Amharic corpus is shown. The user has selected “nagaso gidadane” (Negasso Gidada) for tagging, indicated by the thin gray border, and by the popover component. The lexicon (dictionary) is active, and is displaying the definition (in italics) for “doctor” immediately prior to “nagaso”. (URL and document title are obscured). tively rare in the document. The text is intention- annotate all of them. ally displayed organically, in a way that is familiar In the future, we plan to make this entity prop- and compact, so that the annotator can see as much agation smarter by allowing propagation to near as possible on any given screen. This makes it surface forms (in cases where a suffix or prefix easy for an annotator to make document-level de- differs from the target phrase) or to cancel prop- cisions, for example, if an unusual-looking phrase agation on incorrect phrases, such as stopwords. appears several times. To add an annotation, as shown in Figure1, the annotator clicks (and drags) Lexicon Integration on a word (or phrase), and a popover appears The main difficulty for non-speakers is that they with buttons corresponding to label choices as de- do not understand the text they are annotating. An fined in the configuration file. Most words are not important feature of TALEN is in-place lexicon in- names, so a default label of non-name (O) is as- tegration, which replaces words in the annotation signed to all untouched tokens, keeping the num- screen with their translation from the lexicon. For ber of clicks to a minimum. In contrast, a part-of- example, in Figure1, the English word doctor is speech (POS) annotation system, SAWT (Samih displayed in line (translations are marked by ital- et al., 2016), for example, is designed so that ev- ics). As before, this is built on the notion that an- ery token requires a decision and a click. notation is easiest when text looks organic, with translations seamlessly integrated. Users can click Entity Propagation on a token and add a definition, which is immedi- In a low-resource scenario, it can be difficult to ately saved to disk. The next time the annotator discover and notice names because all words look encounters this word, the definition will be dis- unfamiliar. To ease this burden, and to save on played. If available, a bilingual lexicon (perhaps clicks, the interface propagates all annotation de- from PanLex (Kamholz et al., 2014)), can kickstart cisions to every matching surface in the document. the process. A side effect of the definition addition For example, if ’iteyop’eya (Ethiopia) shows up feature is a new or updated lexicon, which can be many times in a document, then a single click will shared with other users, or used for other tasks.

81 Figure 2: Sentence-based annotation screen, with sentences corresponding to seed term ba’iteyop’eya (annotated in the first sentence, not in the second). Two sentences are shown, and the remaining three sentences associated with this seed term are lower on the page. Notice that hayelamareyame dasalane˜ (Hailemariam Desalegn) has also been annotated in the first sentence. This will become a new seed term for future iterations. (URL and sentence IDs are obscured).

Token Statistics 3 Annotation Modes

When one has no knowledge of a language, it may There are two annotation modes: a document- be useful to know various statistics about tokens, based mode, and a sentence-based mode. Each has such as document count, corpus count, or TF-IDF. all the helps described above, but they display doc- For example, if a token appears many times in ev- uments and sentences to users in different ways. ery document, it is likely not a name. Our an- notation screen shows a table with statistics over 3.1 Document-based Annotation tokens, including document count, percentage of documents containing it, and TF-IDF. At first, it The document-based annotation is identical to the shows the top 10 tokens by TF-IDF, but the user common paradigm of document annotation: the can click on any token to get individual statistics. administrator provides a group of documents, and For example, in Figure1, nagaso appears 4 times creates a configuration file for that set. The anno- in this document, appears in 10% of the total doc- tator views one document at a time, and moves on uments, and has a TF-IDF score of 9.21. In prac- to the next when they are satisfied with the anno- tice, we have found that this helps to give an idea tations. Annotation proceeds in a linear fashion, of the topic of the document, and often names will although annotators may revisit old documents to have high TF-IDF in a document. fix earlier mistakes. Figure1 shows an example of usage in the document-based annotation mode. Internet Search 3.2 Sentence-based Annotation Upon selection of any token or phrase, the popover includes an external link to search the Internet for The sentence-based annotation mode is modeled that phrase. This can be helpful in deciding if a after bootstrapping methods. In this mode, the phrase is a name or not, as search results may re- configuration file is given a path to a corpus of turn images, or even autocorrect the phrase to an documents (usually a very large corpus) and a English standard spelling. small number (less than 10) seed entities, which

82 Figure 3: Sentence-based annotation screen showing 4 seed terms available for annotation. Notice the Unannotated and Annotated tabs. These terms are in the active Unannotated tab because each term has some sentences that have not yet been labeled with that seed term. For example, of the 5 sentences found for ba’iteyop’eya, only 1 has this seed term labeled (see Figure2). can be easily acquired from Wikipedia. This cor- 4 Related Work pus is indexed at the sentence level, and for each seed entity, k sentences are retrieved. These are There are several tools designed for similar pur- presented to the annotator, as in Figure2, who will poses, although to our knowledge none are de- mark all names in the sentences, starting with the signed specifically for non-speaker annotations. entity used to gather the sentence, and hopefully Many of the following tools are powerful and flex- discover other names in the process. As names are ible, but would require significant refactoring to discovered, they are added to the list of seed enti- accommodate non-speakers. ties, as shown in Figure3. New sentences are then The most prominent and popular is brat: rapid retrieved, and the process continues until a pre- annotation tool (brat) (Stenetorp et al., 2012), a defined number of sentences has been retrieved. web-based general purpose annotation tool capa- At this point, the data set is frozen, no new sen- ble of a handling a wide variety of annotation tences are added, and the annotator is expected to tasks, including span annotations, and relations thoroughly examine each sentence to discover all between spans. brat is open source, reliable, and named entities. available to download and run. There are a number of tools with similar func- tionality. Sapient (Liakata et al., 2009) is a web- In practice, we found that the size of k, which is based system for annotating sentences in scientific the number of sentences retrieved per seed term, documents. WebAnno (de Castilho et al., 2016) affects the overall corpus diversity. If k is large uses the frontend visualization from brat with a relative to the desired number of sentences, then new backend designed to make large-scale project the annotation is fast (because entity propagation management easier. EasyTree (Little and Tratz, can annotate all sentences with one click), but the 2016) is a simple web-based tool for annotating method produces a smaller number of unique en- dependency trees. Callisto2 is a desktop tool for tities. However, if k is small, annotation may be rich linguistic annotation. slower, but return more diverse entities. In prac- tice, we use a default value of k = 5. 2http://mitre.github.io/callisto/

83 SAWT (Samih et al., 2016) is a sequence an- DATASET PRECISION RECALL F1 notation tool with a focus on being simple and brat 51.4 8.7 14.2 lightweight, which is also a focus of ours. One TALEN 53.6 12.6 20.0 key difference is that this expects that annotators will want to provide a tag for every word. This DATASET TOTAL NAMES UNIQUENAMES is inefficient for NER, where many tokens should Gold 2260 1154 take the default non-name label. brat 405 189 5 Experiment: Compare to brat TALEN 457 174 The brat rapid annotation tool (brat) (Stenetorp Figure 4: Performance results. The precision, re- et al., 2012) is a popular and well-featured anno- call, and F1 are measured against the gold standard tation tool, which makes for a natural comparison Amharic training data. When counting Unique to TALEN. In this experiment, we compare tools Names, each unique surface form is counted once. qualitatively and quantitatively by hiring a group We emphasize that these results are calculated of annotators. We can compare performance be- over a very small amount of data annotated over a tween TALEN and brat by measuring the results half-hour period by annotators with no experience after having annotators use both tools. with TALEN or Amharic. These only show a quick We chose to annotate Amharic, a language from and dirty comparison to brat, and are not intended Ethiopia. We have gold training and test data for to demonstrate high-quality performance. this language from the Linguistic Data Consor- tium (LDC2016E87). The corpus is composed of strategies. The tags used were Person, Organiza- several different genres, including newswire, dis- tion, Location, and Geo-Political Entity. As for cussion forums, web blogs, and social network strategy, we instructed them to move quickly, an- (Twitter). In the interest of controlling for the notating names only if they are confident (e.g. if domain, we chose only 125 newswire documents they know the English version of that name), and (NW) from the gold data, and removed all annota- to prioritize diversity of discovered surface forms tions before distribution. Since Amharic is written over exhaustiveness of annotation. When using in Ge’ez script, we romanized it, so it can be read TALEN, we encouraged them to make heavy use by English speakers. We partitioned the newswire of the lexicon. We provided a short list (less than documents into 12 (roughly even) groups of docu- 20 names) of English names that are likely to be ments, and assigned each annotator 2 groups: one found in documents from Ethiopia: local politi- to be annotated in brat, the other with TALEN. This cians, cities in the region, etc. way, every annotator will use both interfaces, and The annotation period lasted 1 hour, and con- every document will be annotated by both inter- sisted of two half hour sessions. For the first faces. We chose one fully annotated gold docu- session, we randomly assigned half the annota- ment and copied it into each group, so that the an- tors to use brat, and the other half to use TALEN. notators have an annotation example. When this 30 minute period was over, all annota- We employed 12 annotators chosen from our tors switched tools. Those who had used brat use NLP research group. Before the annotation pe- ours, and vice versa. We did this because users are riod, all participants were given a survey about likely to get better at annotating over time, so the tool usage and language fluency. No users had fa- second tool presented should give better results. miliarity with TALEN, and only one had any fa- Our switching procedure mitigates this effect. miliarity with brat. Of the annotators, none spoke At the end of the second session, each document Amharic or any related language, although one an- group had been annotated twice: once by some notator had some familiarity with Hebrew, which annotator using brat, and once by some annotator shares a common ancestry with Amharic, and one using TALEN. These annotations were separate, so annotator was from West Africa.3 each tool started with a fresh copy of the data. Immediately prior to the annotation period, we We report results in two ways: first, annotation gave a 15 minute presentation with instructions on quality as measured against a gold standard, and tool usage, annotation guidelines, and annotation second, annotator feedback. Figure4 shows ba- 3Ethiopia is in East Africa. sic statistics on the datasets. Since the documents

84 we gave to the annotators came from a gold an- TextAnnotation data structure from the CogComp notated set, we calculated precision, recall, and NLP library (Khashabi et al., 2018) to represent F1 with respect to the gold labels. First, we see and process text. that TALEN gives a 5.8 point F1 improvement over In cases where it is not practical to run a brat. This comes mostly from the recall, which im- backend server (for example, Amazon Mechani- proves by 3.9 points. This may be due to the auto- cal Turk5), we include a version written entirely in matic propagation, or it may be that having a lexi- javascript, called annotate-local.js. con helped users discover more names by proxim- We allow a number of standard input file for- ity to known translations like president. In a less mats, and attempt to automatically discover the time-constrained environment, users of brat might format. The formats we allow are: column, in be more likely select and annotate all surfaces of a which each (word, tag) pair is on a single line, name, but the reality is that all annotation projects serialized TextAnnotation format (from CogComp are time-constrained, and any help is valuable. NLP (Khashabi et al., 2018)) in both Java serial- The bottom part of the table shows the anno- ization and JSON, and CoNLL column format, in tation statistics from TALEN compared with brat. which the tag and word are in columns 0 and 5, TALEN yielded more name spans than brat, but respectively. fewer unique names, meaning that many of the When a user logs in and chooses a dataset to an- names from TALEN are copies. This is also likely notate, a new folder is created automatically with a product of the name propagation feature. the username in the path. When that user logs in We gathered qualitative results from a feedback again, the user folder is reloaded on top of the orig- form filled out by each annotator after the eval- inal data folder. This means that multiple users can uation. All but one of the annotators preferred annotate the same corpus, and their annotations TALEN for this task. In another question, they are saved in separate folders. This is in contrast were asked to select an option for 3 qualities of to brat, where each dataset is modified in place. each tool: efficiency, ease of use, and presentation. Each quality could take the options Bad, Neutral, 7 Conclusions and Future Work or Good. On each of these qualities, brat had a We have presented TALEN, a powerful interface majority of Neutral, and TALEN had a majority for annotation of named entities in low-resource of Good. For TALEN, Efficiency was the highest languages. We have explained the usage with rated quality, with 10 respondents choosing Good. screenshots and descriptions, and outlined a short We also presented respondents with the 4 ma- user study showing that TALEN outperforms brat jor features of TALEN (TF-IDF box, lexicon, en- in a low-resource task in both qualitative and quan- tity propagation, Internet search), and asked them titative measures. to rate them as Useful or Not useful in their ex- perience. Only 4 people found the TF-IDF box Acknowledgements useful; 10 people found the lexicon useful; all 12 We would like to thank our annotators for their people found the entity propagation useful; 7 peo- time and help in the small annotation project. ple found the Internet search useful. These results This material is based on research spon- are also reflected in the free text feedback. Most sored by the US Defense Advanced Research respondents were favorable towards the lexicon, Projects Agency (DARPA) under agreement num- and some respondents wrote that the TF-IDF box bers FA8750-13-2-0008 and HR0011-15-2-0025. would be useful with more exposure, or with bet- The U.S. Government is authorized to reproduce ter integration (e.g. highlight on hover). and distribute reprints for Governmental purposes 6 Technical Details notwithstanding any copyright notation thereon. The views and conclusions contained herein are The interface is web-based, with a Java backend those of the authors and should not be interpreted server. The frontend is built using Twitter boot- as necessarily representing the official policies 4 strap framework, and a custom javascript library or endorsements, either expressed or implied, of called annotate.js. The backend is built with DARPA or the U.S. Government. the Spring Framework, written in Java, using the

4https://getbootstrap.com/ 5mturk.com

85 References putational Natural Language (CoNLL). http://cogcomp.org/papers/TsaiMaRo16.pdf. Richard Eckart de Castilho, Eva Mujdricza-Maydt, Muhie Seid Yimam, Silvana Hartmann, Iryna Boliang Zhang, Xiaoman Pan, Tianlu Wang, Ashish Gurevych, Anette Frank, and Chris Biemann. 2016. Vaswani, Heng Ji, Kevin Knight, and Daniel Marcu. A web-based tool for the integrated annotation of se- 2016. Name tagging for low-resource incident lan- mantic and syntactic structures. In Proceedings of guages based on expectation-driven learning. In the Workshop on Language Technology Resources NAACL. and Tools for (LT4DH). pages 76–84. David Kamholz, Jonathan Pool, and Susan M Colow- ick. 2014. Panlex: Building a resource for panlin- gual lexical translation. In LREC. pages 3145–3150. Daniel Khashabi, Mark Sammons, Ben Zhou, Tom Redman, Christos Christodoulopoulos, Vivek Sriku- mar, Nicholas Rizzolo, Lev Ratinov, Guanheng Luo, Quang Do, Chen-Tse Tsai, Subhro Roy, Stephen Mayhew, Zhilli Feng, John Wieting, Xiaodong Yu, Yangqiu Song, Shashank Gupta, Shyam Upadhyay, Naveen Arivazhagan, Qiang Ning, Shaoshi Ling, and Dan Roth. 2018. Cogcompnlp: Your swiss army knife for nlp. In 11th Language Resources and Eval- uation Conference. Maria Liakata, Claire Q, and Larisa N. Soldatova. 2009. Semantic annotation of papers: Interface & enrichment tool (sapient). In BioNLP@HLT- NAACL. Alexa Little and Stephen Tratz. 2016. Easytree: A graphical tool for dependency tree annotation. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Gro- belnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Tenth Interna- tional Conference on Language Resources and Eval- uation (LREC 2016). European Language Resources Association (ELRA), Paris, France. Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017. Cheap translation for cross- lingual named entity recognition. In EMNLP. http://cogcomp.org/papers/MayhewTsRo17.pdf. Younes Samih, Wolfgang Maier, Laura Kallmeyer, and Heinrich Heine. 2016. Sawt: Sequence annotation web tool. Rion Snow, Brendan T. O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In EMNLP. Pontus Stenetorp, Sampo Pyysalo, Goran Topic,´ Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsu- jii. 2012. brat: a web-based tool for NLP-assisted text annotation. In Proceedings of the Demonstra- tions Session at EACL 2012. Association for Com- putational Linguistics, Avignon, France. Chen-Tse Tsai, Stephen Mayhew, and Dan Roth. 2016. Cross-lingual named entity recognition via wiki- fication. In Proc. of the Conference on Com-

86