LaTeCH 2013

Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities (LaTeCH 2013)

August 8, 2013 Sofia, Bulgaria Production and Manufacturing by Omnipress, Inc. 2600 Anderson Street Madison, WI 53704 USA

c 2013 The Association for Computational Linguistics

Order copies of this and other ACL proceedings from:

Association for Computational Linguistics (ACL) 209 N. Eighth Street Stroudsburg, PA 18360 USA Tel: +1-570-476-8006 Fax: +1-570-476-0860 [email protected]

ISBN 978-1-937284-62-6

ii Preface

We are delighted to present you with this volume containing the papers accepted for presentation at the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, endorsed by the ACL SIGHUM interest group.

Language technology has by now pervaded core processing procedures targeting the cleaning, searching, linking, enriching, and mining of digitized data from these fields. After six previous LaTeCH workshops, we are happy to carry on with aggregating and disseminating the most interesting studies – selected by a thorough peer-review process – concerning current hot topics in the area of natural language processing. Acceptance rate for LaTeCH-2013 was 62%. We would especially like to thank the members of the programme committee for willing to share their expertise by providing detailed reviews and insightful input to all the submitting authors.

On top of the regular paper presentations, the organisers are proud to integrate the SIGHUM annual business meeting into the programme.

We wish you a well-spent workshop day!

Piroska Lendvai and Kalliopi Zervanou Chairs of LaTeCH-2013

iii

Organizers

Workshop Chairs: Piroska Lendvai, Research Institute for Linguistics (Hungary) Kalliopi Zervanou, Radboud University Nijmegen (The Netherlands)

Organizing Committe: Piroska Lendvai, Research Institute for Linguistics (Hungary) Caroline Sporleder, Saarland University / Trier University, Germany Antal van den Bosch, Radboud University Nijmegen (The Netherlands) Kalliopi Zervanou, Radboud University Nijmegen (The Netherlands)

Program Committee Members: Ion Androutsopoulos, Athens University of Economics and Business, Greece David Bamman, Carnegie Mellon University, USA Rens Bod, Universiteit van Amsterdam, The Netherlands Toine Bogers, Royal School of Library and Information Science, Copenhagen, Denmark Paul Buitelaar, DERI Galway, Ireland Mick O‘Donnell, Universidad Autonoma de Madrid, Spain Julio Gonzalo, Universidad Nacional de Educacion a Distancia, Spain Ben Hachey, Macquarie University, Australia Iris Hendrickx, Radboud University Nijmegen, The Netherlands Elias Iosif, Technical University of Crete, Greece Jaap Kamps, Universiteit van Amsterdam, The Netherlands Vangelis Karkaletsis, NCSR Demokritos, Greece Mike Kestermont, University of Antwerp / Research Foundation Flanders, Belgium Dimitrios Kokkinakis, University of Gothenburg, Sweden Stasinos Konstantopoulos, NCSR Demokritos, Greece Barbara McGillivray, Oxford University Press, UK Joakim Nivre, Uppsala University, Sweden Csaba Oravecz, Research Institute for Linguistics, Hungary Petya Osenova, Bulgarian Academy of Sciences, Bulgaria Katerina Pastra, Cognitive Systems Research Institute, Greece Michael Piotrowski, University of Zurich, Switzerland Georg Rehm, DFKI Berlin, Germany Martin Reynaert, Tilburg University, The Netherlands Eszter Simon, Research Institute for Linguistics, Hungary Herman Stehouwer, Max Planck Institute for Psycholinguistics, The Netherlands Mark Stevenson, University of Sheffield, UK Mariët Theune, University of Twente, The Netherlands Suzan Verberne, Radboud University Nijmegen, The Netherlands Cristina Vertan, University of Hamburg, Germany Menno van Zaanen, Tilburg University, The Netherlands Svitlana Zinger, TU Eindhoven, The Netherlands

v

Table of Contents

Generating Paths through Cultural Heritage Collections Samuel Fernando, Paula Goodale, Paul Clough, Mark Stevenson, Mark Hall and Eneko Agirre . . 1

Using character overlap to improve language transformation Sander Wubben, Emiel Krahmer and Antal van den Bosch ...... 11

Comparison between historical population archives and decentralized databases Marijn Schraagen and Dionysius Huijsmans ...... 20

Semi-automatic Construction of Cross-period Thesaurus Chaya Liebeskind, Ido Dagan and Jonathan Schler ...... 29

Language Technology for Agile Social Media Science Simon Wibberley, David Weir and Jeremy Reffin ...... 36

Morphological annotation of Old and Middle Hungarian corpora Attila Novak, György Orosz and Nóra Wenszky ...... 43

Argument extraction for supporting public policy formulation Eirini Florou, Stasinos Konstantopoulos, Antonis Koukourikos and Pythagoras Karampiperis . . 49

Towards a Tool for Interactive Concept Building for Large Scale Analysis in the Humanities Andre Blessing, Jonathan Sonntag, Fritz Kliche, Ulrich Heid, Jonas Kuhn and Manfred Stede . . 55

Learning to Extract Folktale Keywords Dolf Trieschnigg, Dong Nguyen and Mariët Theune ...... 65

Towards Creating Precision Grammars from Interlinear Glossed Text: Inferring Large-Scale Typological Properties Emily M. Bender, Michael Wayne Goodman, Joshua Crowgey and Fei Xia ...... 74

Using Comparable Collections of Historical Texts for Building a Diachronic Dictionary for Spelling Normalization Marilisa Amoia and José Manuel Martínez ...... 84

Integration of the Thesaurus for the Social Sciences (TheSoz) in an Information Extraction System Thierry Declerck ...... 90

The (Un)faithful Machine Translator Ruth Jones and Ann Irvine...... 96

Temporal classification for historical Romanian texts Alina Maria Ciobanu, Anca Dinu, Liviu Dinu, Vlad Niculae and Octavia-Maria ¸Sulea...... 102

Multilingual access to cultural heritage content on the Semantic Web Dana Dannells, Aarne Ranta, Ramona Enache, Mariana Damova and Maria Mateva ...... 107

vii

Conference Program

Thursday August 8, 2013

9:00-9:15 Opening

9:15–9.35 Generating Paths through Cultural Heritage Collections Samuel Fernando, Paula Goodale, Paul Clough, Mark Stevenson, Mark Hall and Eneko Agirre

9:35–9.55 Using character overlap to improve language transformation Sander Wubben, Emiel Krahmer and Antal van den Bosch

9:55–10:15 Comparison between historical population archives and decentralized databases Marijn Schraagen and Dionysius Huijsmans

10:15–10:30 Semi-automatic Construction of Cross-period Thesaurus Chaya Liebeskind, Ido Dagan and Jonathan Schler

10:30-11:00 Coffee Break

11:00–11:15 Language Technology for Agile Social Media Science Simon Wibberley, David Weir and Jeremy Reffin

11:15–11:30 Morphological annotation of Old and Middle Hungarian corpora Attila Novák, György Orosz and Nóra Wenszky

11:30–11:45 Argument extraction for supporting public policy formulation Eirini Florou, Stasinos Konstantopoulos, Antonis Koukourikos and Pythagoras Karampiperis

11:45-12:30 SIGHUM annual business meeting

12:30-14:00 Lunch Break

14:00–14:20 Towards a Tool for Interactive Concept Building for Large Scale Analysis in the Humanities Andre Blessing, Jonathan Sonntag, Fritz Kliche, Ulrich Heid, Jonas Kuhn and Man- fred Stede

14:20–14:40 Learning to Extract Folktale Keywords Dolf Trieschnigg, Dong Nguyen and Mariët Theune

ix Thursday August 8, 2013 (continued)

14:40–15:00 Towards Creating Precision Grammars from Interlinear Glossed Text: Inferring Large- Scale Typological Properties Emily M. Bender, Michael Wayne Goodman, Joshua Crowgey and Fei Xia

15:00–15:15 Using Comparable Collections of Historical Texts for Building a Diachronic Dictionary for Spelling Normalization Marilisa Amoia and José Manuel Martínez

15:15–15:30 Integration of the Thesaurus for the Social Sciences (TheSoz) in an Information Extraction System Thierry Declerck

15:30-16:00 Coffee Break

16:00–16:15 The (Un)faithful Machine Translator Ruth Jones and Ann Irvine

16:15–16:30 Temporal classification for historical Romanian texts Alina Maria Ciobanu, Anca Dinu, Liviu Dinu, Vlad Niculae and Octavia-Maria ¸Sulea

16:30–16:50 Multilingual access to cultural heritage content on the Semantic Web Dana Dannells, Aarne Ranta, Ramona Enache, Mariana Damova and Maria Mateva

16:50-17:00 Closing

x Generating Paths through Cultural Heritage Collections Samuel Fernando1, Paula Goodale2, Paul Clough2, Mark Stevenson1, Mark Hall2, Eneko Agirre3 1Department of Computer Science, University of Sheffield 2Information School, University of Sheffield 3Computer Science Department, University of the Basque Country {s.fernando, p.goodale, p.d.clough, r.m.stevenson, m.mhall}@sheffield.ac.uk [email protected]

Abstract educational perspective paths can provide tangible learning objects, created by teachers and followed Cultural heritage collections usually or- by students. Alternatively from a cultural her- ganise sets of items into exhibitions or itage perspective paths can be used to create activ- guided tours. These items are often ity trails and guided tours support exploration by accompanied by text that describes the visitors through collections of cultural artefacts. theme and topic of the exhibition and pro- This echoes the organised galleries and guided vides background context and details of tours found in physical museums. The existance connections with other items. The PATHS of tools, such as Walden’s paths1, Trailmeme2 and project brings the idea of guided tours Storify3, provide functionalities for users to record to digital library collections where a tool and share paths through web resources and digital to create virtual paths are used to assist libraries. From this perspective everyone can take with navigation and provide guides on par- on role of curator and provide access to their own ticular subjects and topics. In this pa- personal collections. per we characterise and analyse paths of We have developed an online system called items created by users of our online sys- PATHS that allows curators and end-users to cre- tem. The analysis highlights that most ate and view paths to navigate through the Eu- users spend time selecting items relevant ropeana4 cultural heriage collection. As part of to their chosen topic, but few users took evaluations of the prototype PATHS system par- time to add background information to the ticipants have created paths on various topics. In paths. In order to address this, we con- this paper we describe a number of these paths and ducted preliminary investigations to test their characteristics. Analysing paths that are cre- whether Wikipedia can be used to au- ated manually and characterising them can be seen tomatically add background text for se- as a first step towards developing methods to sup- quences of items. In the future we would port the creation of paths automatically and semi- like to explore the automatic creation of automatically. Within the context of the PATHS full paths. project this is being considered to deal with the following limitations of manual creation of paths. 1 Introduction Firstly, the effort required in generating them often Paths (or trails) have been studied as a means of means that a sufficient number of paths on a vari- assisting users with the navigation of digital col- ety of topics are not available. Secondly, the man- lections as an alternative to standard keyword- ual creation of paths is a very time-consuming pro- based search (Furuta et al., 1997; Reich et al., cess that would benefit from computational sup- 1999; Shipman et al., 2000; White and Huang, port in whatever form this might take. This pa- 2010). Paths can be particularly useful to users per presents initial work in automatically creat- who are unfamilar with the content of digital col- ing paths and provides the following novel con- lections (e.g. historical documents) and may find it difficult to formulate appropriate queries (Wil- 1http://www.csdl.tamu.edu/walden/ 2http://open.xerox.com/Services/ son et al., 2010). Paths can be used to assist users xerox-trails with the navigation of collections through the pro- 3http://storify.com/ vision of narratives and subject guides. From an 4http://www.europeana.eu/

1 Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 1–10, Sofia, Bulgaria, August 8 2013. c 2013 Association for Computational Linguistics tributions: (1) we present results of user stud- However, since “narrative is constructed by link- ies describing what people want from paths and ing elements of this database in a particular or- how they use them to navigate digital collections; der” (Manovich, 1999), it is possible to offer users (2) we analyse a set of manually-created paths any number of explicit ‘trajectories’ (narratives) to identify their properties and be able to charac- through a digital information space, and by merg- terise them; and (3) we present work on automati- ing database and narrative in this way, creating cally generating background text for sequences of a more dynamic, discovery-led experience. This items, thus providing an efficient way to enrich view might be interpreted at its simplest level as paths with additional information with little man- a virtual representation of the guided tours rou- ual input required. tinely offered in physical CH spaces, and indeed The paper is structured as follows: Section 2 de- there is a small strand of research into the creation scribes related work on the use of narratives in cul- of systems for generating and exploring online ex- tural heritage and previous approaches to automat- hibitions and tours from items held within digital ically generate paths; Section 3 defines the prob- collections. A scenario of users creating and edit- lem of generating paths and describes the datasets ing trails in a CH context is described by Walker used in the experiments; Section 4 presents analy- (2006), including functionality for collecting, or- sis of manually-created paths; Section 5 shows re- dering and annotating museum objects. sults of using automatic methods to generate back- ground text; and finally Section 6 concludes the 2.2 Automatically Creating Paths paper and provides avenues for further work. Generation of implicit trails through physical and 2 Related Work virtual museum spaces has been related to the learning process (Peterson and Levene, 2003). In 2.1 Narratives and Cultural Heritage this example, trails are automatically created by The potential of narrative in digital CH to sup- users as they navigate their way through an infor- port learning, creativity and exploration is clear, mation space, and may be used for individual or providing opportunities for supporting a more ac- collaborative purposes. Research on the applica- tive user interaction, including deeper engagement tion of curated pathways in web environments has with context, representation of the collecting pro- often focused on providing trails pre-prepared by cess, and facilitation of a more entertaining expe- experts (e.g. curators, educationalists) as a means rience of learning (Mulholland and Collins, 2002). of assisting novice users to navigate information Walker et al. (2013) also propose narrative as a online (Shipman et al., 2000). Indeed, it has been major element of interaction and informal learn- found that domain knowledge or expertise can ing, suggesting that meaning is made when the considerably enhance the quality of trails created links between people and artefacts, and interpreta- (Yuan and White, 2012). Automatic extraction tion and ideas are surfaced, especially within so- and generation of trails in information spaces has cial groups. Their experiments involve the use been explored as a means of harnessing the wis- of mobile and handheld technologies in a physi- dom of crowds, using the mass actions of earlier cal museum environment, capturing audio annota- user behaviour to establish relevance, and recom- tions, but have much in common with experimen- mend content or navigation routes to later users. tal systems designed for path creation online. In a Such trails can be readily mined from search en- similar vein the StoryBank project utilises collec- gine transaction logs and have been shown to pro- tions of photographs and audio narratives to create vide added value (White and Huang, 2010; Has- and share stories as information in the developing san and White, 2012; Liao et al., 2012). West and world (Frohlich and Rachovides, 2008). Leskovec (2012) take this notion a stage further Whilst technologies have aided the creation and and attempt to identify wayfinding strategies em- sharing of narratives in physical cultural encoun- ployed by browsers in Wikipedia, with the goal of ters, Manovich (1999) critiques the lack of narra- assisting future users in their navigation by surfac- tive in digital cultural environments, offering that ing potentially useful hyperlinks. online collections and many CH web sites are Guided tours or pathways are essentially more databases with constantly changing content that structured, purposeful forms of trails, taking the inevitably lack a cohesive and persistent story. user through a specific sequence of information

2 nodes and may also be automatically generated, 3.1 Utilisation of Paths rather than manually curated as in the examples Initial user requirements interviews with 22 ex- above. Wheeldon and Levene (2003) offer an al- pert users in the heritage, education and profes- gorithm for generating trails from site-search, en- sional domains found a strong affinity with the abling elements of structure and context to be in- path metaphor, revealing a range of different in- corporated into the trails created in this way, but terpretations of what it means in the CH context noting potential scalability issues for web scale and how they could be employed in an online en- search tasks. In the CH domain, a small num- vironment to engage with key audiences. Eight ber of projects have attempted to automatically interpretations of the path metaphor emerged: generate digital content in the form of exhibi- 1. Path as search history tions, tours and trails. Makel¨ a¨ et al. (2007) de- 2. Path as information seeking journey scribe a system which utilises semantically an- 3. Path as linked metadata notated content to generate personalised ‘exhi- 4. Path as a starting point or way in bitions’ from a structured narrative-based search 5. Path as a route through query. Similarly, Zdrahal et al. (2008) demonstrate 6. Path as augmented reality how pathways can be generated through a collec- 7. Path as information literacy journey / learn- tion of semantically related documents to provide ing process a means of exploration, using non-NLP cluster- 8. Path as transaction process ing and path creation techniques. Sophisticated approaches such as linear programming and evo- The first three of these are closest to the idea lutionary algorithms have also been proposed for of hypertext trails, with trails defined by user in- generating summaries and stories (McIntyre and teraction in 1 and 2, and trails defined automati- Lapata, 2010; Woodsend and Lapata, 2010). In cally, by the system in 3. Variations 4-6 are more contrast, Wang et al. (2007) use a recommender creative interpretations, all suggesting opportuni- system approach to generate museum tours on ties for guiding the user into and through collec- the basis of ratings stored within a dynamic user tions, encouraging exploration and/or offering an model, and Pechenizkiy and Calders (2007) pro- immersive experience, conducive with our initial pose the additional use of data mining techniques vision for the PATHS system. on log data to improve this type of tour personali- In addition to expert-defined routes, 5 also in- sation. corporates the idea of users being able to see and follow “well-trodden path” defined by the cumula- In summary, online tours and trails are made tive interactions of other users, thus extending the possible either through manually curated content opportunities for utilizing search histories. Con- generated through the efforts of experts or other versely, 7 and 8 are both process oriented, al- end users, or have been automatically generated though 7 is experiential, user-defined, learning- from the mining of large scale search logs, or from oriented, typified by trial and error and unique to collections benefitting from semantically-linked the individual, whilst 8 is a rigid process designed content and/or detailed user models. to escort all users consistently through a standard process of pre-defined steps. 3 Methodology A strong emphasis was placed on path content being carefully selected or ‘curated’ by the path- This study brings together work from several ar- creator, with the addition of context and interpre- eas of the PATHS project. An analysis of what tation so that the objects within the path convey paths might be used for and what form they are ex- a narrative or meaning. Content may be derived pected to take, has had implications for the system from one collection, but there were seen to be sig- design and functionality and evaluation measures. nificant benefits from including objects from di- A user study focused upon evaluation of the first verse collections, along with other materials from prototype has provided manually-created paths as external web sites. a basis for analysing path content and attributes, Paths facilitate topic-based information re- which in turn informs the desired characteristics trieval typified by the berry-picking mode of in- of automated paths and the algorithm designed for teraction (Bates, 1989), rather than known item generating paths automatically. searching. Furthermore, paths may be a useful tool

3 for personal information management in both for- • 1x long simulated work task - path creation mal and informal research scenarios, enabling the (30 minutes) user to record, reuse and share their research activ- • Task feedback questionnaire ity, or helping them to organize their ideas. Cre- • Session/system feedback questionnaire ativity is also encouraged, as user-generated paths • Think-after interview based upon the com- provide the means to repurpose CH objects into plex task users’ own narratives for private or public con- Of most interest here is the simulated work task, sumption. with associated observations, feedback and reflec- A summary of specific user scenarios high- tions. This task focused on the creation of a path, lighted by participants is given below: using a scenario adapted to the type of user. Free- • Teachers/lecturers presentations and class- dom was given in choosing a subject for the path, room activities and limited instructions were provided in what • Museum personnel curating collections, giv- might be needed to complete the task, for exam- ing an overview, or covering a topic in depth ple: • Leisure users browsing, collecting interest- “Imagine you are a student who has been asked ing and/or visually appealing content to create a path as part of a university assignment. • Researchers to aid image-based research, You have been asked to use primary source ma- sharing and discussing findings with fellow terials to create a mini online exhibition suitable researchers and supervisors for a target group within the general public and/or • Non-academic specialists (e.g. local histori- school visitor categories. Your goal is to introduce ans) collecting and sharing items of interest a historical or art-focussed topic in a popular, ac- with other enthusiasts cessible way, and to encourage further use and ex- ploration of cultural heritage resources.” 3.2 Defining the Problem Data on the tasks was captured via log files, as To create a path or narrative that guides a user well as screen recording and observations using through a set of items from a collection, whether the Morae usability software. Detailed analysis as a manual process or automatically, there are was undertaken of user behaviour in the process of three main activities: (1) the selection of items to completing the task, and of the paths created, from include in the path; (2) the arrangement of items both quantitative and qualitative perspectives. to form a path or narrative and (3) the annota- 4 Analysing Manually-created Paths tion of the path to with descriptive text and back- ground information. We envision techniques to In this section we describe the results of analysing automate the entire process; however, a first step is the 22 paths created manually in the PATHS pro- to analyse existing manually-created paths to iden- totype system. tify their characteristics and inform the automatic creation of similar structures. 4.1 User behaviour On average users spend 25.3 mins on creating a 3.3 User Study path (min=11.7; max=33.6) with an average of The manually generated paths used for this study 201 mouse clicks (min=53; max=380). From the were created as part of a more detailed user study observations, it was noted that some participants to evaluate the first prototype, conducted using spent quite a lot of time thinking about the task a protocol informed by the Interactive IR eval- and pondering their next move, whilst others en- uation framework (Borlund, 2003). Twenty-two gaged in more rapid fire activity in the face of users, including subject experts, students and gen- uncertainty. Analysis of the screen recordings eral users (subject novices), each completed a 2- showed a variety of primary interaction styles for hour session, during which they participated in the this task, with a fairly even split between serial following activities: searching (33%) and serial browsing (39%), as the • Profile questionnaire and cognitive style test two most popular strategies. Serial searching in- • Familiarisation with the system volves repetitive search and reformulation, with • 4x short information seeking tasks (5 minutes only a page or two of search results viewed before each) searching again, and serial browsing involves very

4 few searches, with large numbers of search re- as a chronological or narrative sequence. These sults pages viewed (over 50 pages in some cases). more advanced features were used by significantly These are then in effect, polar opposites of interac- fewer users, which could indicate a learning issue, tion. Only 6% engaged primarily in exploring be- a lack of need, or a time constraint. On reviewing haviour (using the explore and similar items con- the paths created by our evaluation participants it tent), and 22% of participants occupied the middle is found that in 41% of cases, contextual informa- ground, utilising a mix of search, browse and ex- tion was not added to any items in the path. There plore, with no strong preference for any one style. are however 32% in which annotations were added to all items (generally these were shorter paths 4.2 Properties of paths with fewer items), and a further 27% where anno- The mean number of items in a path was 10.7 (std tations were added to some or most of the items. dev=6.7 items) with a minimum of 5 items and In 72% of cases the items in the paths created maximum of 29 items. Most popular bin is 6- were re-ordered to some degree, with 17% spend- 10 items in a path (59%). We found 85% of the ing a considerable amount of time on this activity. items included in the paths included an image with This finding is encouraging, as the default is for the metadata. The paths created were manually items to be included in the path in the order they categorised by theme to ascertain whether there were saved to the workspace, and re-ordering in- are any distinct preferences for the subject mat- dicates that users are thinking about their path as a ter of content included. The most popular cate- whole and trying to make sense of the information gories were paths about places (23%), art subjects it is intended to convey. Typical types of ordering (23%) and history subjects (32%). These themes included chronology (32%), narrative (23%), ge- are likely to have been influenced at least partly ography (for example, a walking tour - 9%), theme by what content is currently available in our col- (9%) and ‘interestingness’ (5%). lection, although the amount of art-related content is much less than for history, and also appear to 5 Enriching paths with background have been influenced by the topics covered in ex- information isting paths in the system (e.g. places, topics re- This section describes preliminary work on the lated to the world wars). There were, however a task of semi-automated path creation. In par- significant number of expert users who attempted ticular we describe efforts to enrich paths with to build paths related to their own research inter- background contextual information using relevant ests, with varying degrees of success. Wikipedia articles. The related work described 4.3 Descriptions and ordering in Section 2.2 shows that there have been previ- ous efforts to automatically select cultural heritage Once items have been selected and they have been items to form paths, trails and exhibitions. How- transferred in the path creation workspace, users ever to our knowledge no significant effort has have the opportunity to modify and enhance their been made to automatically annotate such paths path with a number of tools for adding content and with descriptive or contextual information. The metadata, and for re-ordering the content. On cre- interviews described in Section 3.1 highlighted ating the path, most users immediately went to the the importance CH experts placed on having ad- metadata fields and added information for the path ditional information to give context for the items description and duration fields, as well as a num- in the path. It was also noted during the manual ber of tags (or keywords). A short 1-2 line de- path-creation exercise (Section 4.3) that a signif- scription of the path appears to be the norm and icant number of the users did not add any such was added in 91% of cases. Tags were added by information to the path. The reasons for this are 82% of users and a duration by only 46% of users. unclear, but nevertheless there seems to be suffi- It is clear from further investigation that the tags cient motivation to devise automatic methods for were added incorrectly (without commas between this task. Although the methods have previously them) by a significant number of users and a tip been well established in other tasks5 , we believe for successful use is required. The items within a path can be annotated with 5INEX Tweet Contextualization Track (https: //inex.mmci.uni-saarland.de/tracks/qa/) the user’s own contextual information, and can be and Link-the-wiki Track (http://www.inex.otago. re-ordered into a more meaningful sequence, such ac.nz/tracks/wiki-link/wiki-link.asp)

5 this is the first time they have been applied for the of article titles W comprising the titles of all the task of annotating sequences of items in this way. linked articles which were found in the text fields of S. For each title in W we also have the associ- 5.1 Method ated confidence value for the link as calculated by Wikipedia Miner. The next step is to select from Manually generated paths contain sequences of K the most relevant sentences to output as the gen- items selected from Europeana on some topic or erated text. For each sentence in K a score is as- theme. Creators provide their own title, subject signed if any of the words in the sentence match keywords and description for the path. To aid one of the titles in W . The score is then simply the creation of paths we explore whether background sum of the confidence values associated with these information could be generated automatically for titles. The top scoring sentences are then output as such paths. An approach is presented here which the background text. This method can be consid- shows promise as a potential way to achieve this ered to be a kind of query based summarisation task. The input for this approach is a sequence of (Jurafsky and Martin, 2008). items and a key Wikipedia article which describes the overall topic of the path. The output comprises 5.2 Results sentences taken from a relevant Wikipedia article. The aim is for this output to provide useful and The automatic approaches for generating back- interesting additional background information re- ground text were run over the items in the 22 man- lated to the items and theme of the path. In this ually created paths described in the previous sec- paper experiments are focussed on how to select tion. To ensure a fair test the user-added text was good quality text to present as additional informa- was discarded and only the original Europeana tion for the path. For this reason the key Wikipedia metadata for the items was used as source for the article is manually chosen, and the task is to find a methods. good approach for selecting the most relevant sen- For each path a single key Wikipedia article was tences from this key article for the text. manually found which best captured the overall Two methods are tested in this paper. The first theme or topic of the path. For all paths at least method simply takes the first n sentences of the one reasonably good article was found, although article and outputs this. Since Wikipedia articles for some paths the articles were better matches for are always structured to have a summary of the the path topic than for others. Some examples are article in the first paragraph we can expect this text shown in Table 1. to perform well as a summary of the path topic. With the input ready, the approaches described The second method is more advanced and at- were run over the items and key articles to gener- tempts to find text in the article that is relevant to ate the additional text. This was limited to 3 sen- the actual items that have been chosen for the path. tences to provide a short but useful piece of text This approach uses the Wikipedia Miner software for viewers of the paths. For example one path (Milne and Witten, 2008) to add inline links to was entitled “Bronze Age metallurgy”. The user the text in the items for this approach. This soft- provided description was as follows: ware disambiguates terms in the text and then de- Bronze Age metalsmiths produce a variety of tects links using various features such as the com- objects, however, we learn as much from the tools, monness of the term, the overall relatedness of the leftover debris and the way in which objects were terms in the text and so on. The result is text en- deposited as we do from the finished objects them- riched with inline links to relevant Wikipedia arti- selves. cles. Each link also has an associated confidence The key article chosen for the path was “Bronze value which indicates how sure the software is that Age”. The items in the path included various ar- the link is correctly disambiguated and relevant to tifacts and working debris from the Bronze Age. the text. For the first approach the output is the first 3 sen- The approach works as follows for a sequence tences from this article: of items S and a key article K. First Wikipedia The Bronze Age is a period characterized by Miner is run over the items in S. The text input to the use of copper and its alloy bronze as the chief Wikipedia Miner comprises the title, subject and hard materials in the manufacture of some imple- description fields of each item. The output is a set ments and weapons. Chronologically, it stands be-

6 Path title Key Wikipedia article Canals in England Canals of the United Kingdom A brief history of Sheffield History of Sheffield Art and Criticism of John Ruskin John Ruskin Early medieval features in English Architecture of the medieval cathedrals Parish Churches of England Brooches through the ages Brooch

Table 1: Key articles for paths tween the Stone Age and Iron Age. The term Stone Annotators were asked to grade from A (very Age implies the inability to smelt any ore, the term good) to E (very poor) on each dimension. The Bronze Age implies the inability to smelt iron ore results are shown in Figure 1. The results for and the term Iron Age implies the ability to man- the first 3 sentences are shown as First3 and for ufacture artifacts in any of the three types of hard the weighted approach as Weighted. For each di- material. mension, the distribution of judgements across the For the second approach the top 3 highest scor- paths is shown. The First3 approach was found ing sentences were output: to be superior in every dimension. For relevance A region could be in the Bronze Age either by scores 90% of the scores were either A or B com- smelting its own copper and alloying with tin or pared to 63% for the Weighted approach. Sim- by trading for bronze from production areas else- ilarly for the coherence judgements 97% were A where. The Bronze Age in Ireland commenced or B compared to 62% for the weighted approach. around 2000 BC, when copper was alloyed with The reason for this superior performance seems to tin and used to manufacture Ballybeg type flat axes be that the first few sentences of Wikipedia arti- and associated metalwork. The Bronze Age is a cles are deliberately created to give a short sum- period characterized by the use of copper and its mary introduction of the topic of the article. This alloy bronze as the chief hard materials in the explains the high scores for relevance and coher- manufacture of some implements and weapons. ence. These sentences scored most highly since they Both approaches scored lower on the contex- contained the most high-confidence terms from tualisation dimension, with First3 getting 67% the items, for example terms such as ‘copper’, ‘al- A or B grades and the Weighted approach get- loy’ and ‘Bronze Age’. ting 43%. There may be several reasons for this. 5.3 Evaluation Firstly one problem is that the auto-generated text sometimes repeats information that is already in To evaluate the two approaches, 5 annotators were the path and item descriptions; thus the text fails presented with the paths and the text and asked to to meet the requirement of ‘useful additional in- rate each path on 3 dimensions: formation’. Secondly the text is sometimes quite • The relevance of the text to the theme and general and vague, rather than focussing on spe- items of the path. Text which relates strongly cific details which might be most relevant to the to the path is scored highly while off-topic or items chosen for the path. irrelevant text is given a low score. • The coherence or quality of the text itself. To measure the agreement among the annotators Text which appears well-written and well- the following approach was used. First the scores structured is scored highly, while poorly writ- were converted to numeric values; A to 1, B to 2 ten or incoherent text is given a low score. and so on. Then the scores for each annotator were • The contextualisation of the text in relation compared to the average of the scores of all the to the path. To achieve a high score the other annotators. The correlation was computing text should offer useful or interesting addi- using Spearman’s correlation coefficient. These tional information which is not found else- scores were then averaged amongst all annotators where within the content, i.e. the text helps to give a final agreement value. The results are to provide a context for items in the path. shown in Table 2.

7 Figure 1: Comparing the results of the two methods.

First3 Weighted tion to understand the meaning of the items in the Relevance 0.57 0.57 path. The results from the user studies showed that Coherence 0.28 0.56 this need was not being met in a significant num- Contextualisation 0.56 0.78 ber of cases; users were putting items together on a topic but adding little or no descriptive text about Table 2: Agreement amongst annotators. the topic and the items in the path. Therefore we identified this as a key task which might benefit For both approaches there was good agreement from automatic methods. The simpler approach on the Relevance dimension. For the Coherence which output the first n sentences from the key dimension the First3 approach got quite a low Wikipedia article was found to generate the best score. This may be because one annotator gave results. The resulting generated text was found to lower scores for all paths, while the others all gave be relevant and coherent. In most cases the text consistently high scores, which seems to have was also found to add useful context about the skewed the correlation co-efficient. For the con- topic. textualisation dimension the correlation scores for Future work will further refine the text genera- high for both approaches, and the Weighted ap- tion approach. The approach depends on success- proach in particular achieved a very high agree- fully identifying a good key article for each path. ment value. In these experiments the key article was manually chosen, however we are devising methods to se- 6 Conclusions lect this article automatically. To correct the prob- lem with repeated information a filtering approach This paper presented results of interviews about could eliminate information that is already con- creating paths through cultural heritage collec- tained within the paths. tions. These results inform us on how people want to navigate through cultural heritage collec- Acknowledgments tions using the path metaphor, how they wish to make use of paths for their work and education, The research leading to these results was car- and what information and qualities they consider ried out as part of the PATHS project (http: it important for a path to contain. The paper also //paths-project.eu) funded by the Eu- presents results from studies using the PATHS pro- ropean Community’s Seventh Framework Pro- totype software where users were able to search gramme (FP7/2007-2013) under grant agreement and explore a large digital library collection and no. 270082. create their own paths of items from the collection on topics of their interest. From the interviews it was clear that the experts References considered it important that the paths contain ad- Marcia J Bates. 1989. The design of browsing and ditional information to convey contextual informa- berrypicking techniques for the online search inter-

8 face. Online Information Review. Don Peterson and Mark Levene. 2003. Trail records and navigational learning. London review of Educa- Pia Borlund. 2003. The IIR evaluation model: a tion, 1(3):207–216. framework for evaluation of interactive information retrieval systems. Information research, 8(3). S. Reich, L. Carr, D. De Roure, and W. Hall. 1999. Where have you been from here? Trails in hypertext David M Frohlich and Dorothy Rachovides. 2008. Us- systems. ACM Computing Surveys, 31. ing digital stories for local and global information sharing. In Community and International Develop- Frank M Shipman, Richard Furuta, Donald Brenner, ment, CHI 2008 Workshop. Chung-Chi Chung, and Hao-wei Hsieh. 2000. Guided paths through web-based collections: De- R. Furuta, F.. Shipman, C. Marshall, D. Brenner, and sign, experiences, and adaptations. Journal of H. Hsieh. 1997. Hypertext paths and the World- the American Society for Information Science, Wide Web: experiences with Walden’s Paths. In 51(3):260–272. Proceedings of the eighth ACM conference on Hy- pertext, pages 167–176, New York, NY. K. Walker, A. Main, and Fass. J. 2013. User- Generated Trails in Third Places. In HCI-3P Work- Ahmed Hassan and Ryen W White. 2012. Task tours: shop on Human Computer Interaction for Third helping users tackle complex search tasks. In Pro- Places at Computer Human Interaction 2013. ceedings of the 21st ACM international conference on Information and knowledge management, pages Kevin Walker. 2006. Story structures. building nar- 1885–1889. ACM. rative trails in museums. In Technology-Mediated Narrative Environments for Learning, pages 103– Daniel Jurafsky and James H. Martin. 2008. Speech 114. Sense Publishers. and Language Processing (2nd Edition) (Prentice Hall Series in Artificial Intelligence). Prentice Hall. Yiwen Wang, Lora M Aroyo, Natalia Stash, and Lloyd Rutledge. 2007. Interactive user modeling for per- Zhen Liao, Yang Song, Li-wei He, and Yalou Huang. sonalized access to museum collections: The ri- 2012. Evaluating the effectiveness of search task jksmuseum case study. In User Modeling 2007, trails. In Proceedings of the 21st international con- pages 385–389. Springer. ference on World Wide Web, pages 489–498. ACM.

Eetu Makel¨ a,¨ Osma Suominen, and Eero Hyvonen.¨ Robert West and Jure Leskovec. 2012. Human 2007. Automatic exhibition generation based on wayfinding in information networks. In Proceed- semantic cultural content. In Proc. of the Cultural ings of the 21st international conference on World Heritage on the Semantic Web Workshop at ISWC+ Wide Web, pages 619–628. ACM. ASWC, volume 2007. Richard Wheeldon and Mark Levene. 2003. The best Lev Manovich. 1999. Database as symbolic form. trail algorithm for assisted navigation of web sites. Convergence: The International Journal of Re- In Web Congress, 2003. Proceedings. First Latin search into New Media Technologies, 5(2):80–99. American, pages 166–178. IEEE.

Neil McIntyre and Mirella Lapata. 2010. Plot induc- Ryen W White and Jeff Huang. 2010. Assessing the tion and evolutionary search for story generation. In scenic route: measuring the value of search trails in Proceedings of the 48th Annual Meeting of the Asso- web logs. In Proceedings of the 33rd international ciation for Computational Linguistics, pages 1562– ACM SIGIR conference on Research and develop- 1572. Association for Computational Linguistics. ment in information retrieval, pages 587–594. ACM.

D. Milne and I.H. Witten. 2008. Learning to link with M. Wilson, Kulesm B., M. Schraefel, and B. Schnei- wikipedia. In Proceeding of the 17th ACM confer- derman. 2010. From keyword search to explo- ence on Information and knowledge management, ration: Designing future search interfaces for the pages 509–518. ACM. web. Foundations and Trends in Web Science, 2(1):1–97. Paul Mulholland and Trevor Collins. 2002. Using dig- ital narratives to support the collaborative learning Kristian Woodsend and Mirella Lapata. 2010. Auto- and exploration of cultural heritage. In Database matic generation of story highlights. In Proceedings and Expert Systems Applications, 2002. Proceed- of the 48th Annual Meeting of the Association for ings. 13th International Workshop on, pages 527– Computational Linguistics, pages 565–574. Associ- 531. IEEE. ation for Computational Linguistics.

Mykola Pechenizkiy and Toon Calders. 2007. A Xiaojun Yuan and Ryen White. 2012. Building the framework for guiding the museum tours person- trail best traveled: effects of domain knowledge on alization. In Proceedings of the Workshop on Per- web search trailblazing. In Proceedings of the 2012 sonalised Access to Cultural Heritage (PATCH07), ACM annual conference on Human Factors in Com- pages 11–28. puting Systems, pages 1795–1804. ACM.

9 Zdenek Zdrahal, Paul Mulholland, and Trevor Collins. 2008. Exploring pathways across stories. In Proc. of International Conference on Distributed Human- Machine Systems.

10 Using character overlap to improve language transformation

Sander Wubben Emiel Krahmer Antal van den Bosch Tilburg University Tilburg University Radboud University Nijmegen P.O. Box 90135 P.O. Box 90135 P.O. Box 9103 5000 LE Tilburg 5000 LE Tilburg 6500 HD Nijmegen The Netherlands The Netherlands The Netherlands [email protected] [email protected] [email protected]

Abstract area that is now defined as the Netherlands and parts of Belgium. One of the factors that make Language transformation can be defined Middle Dutch difficult to read is the fact that at as translating between diachronically dis- the time no overarching standard language existed. tinct language variants. We investigate the Modern Dutch is defined as Dutch as spoken from transformation of Middle Dutch into Mod- 1500. The variant we investigate is contempo- ern Dutch by means of machine transla- rary Dutch. An important difference with regular tion. We demonstrate that by using char- paraphrasing is the amount of parallel data avail- acter overlap the performance of the ma- able. The amount of parallel data for the vari- chine translation process can be improved ant pair Middle Dutch - Modern Dutch is sev- for this task. eral orders of magnitude smaller than bilingual parallel corpora typically used in machine trans- 1 Introduction lation (Koehn, 2005) or monolingual parallel cor- In this paper we aim to develop a system to para- pora used for paraphrase generation by machine phrase between diachronically distinct language translation (Wubben et al., 2010). variants. For research into history, historical lin- We do expect many etymologically related guistics and diachronic language change, histori- words to show a certain amount of character cal texts are of great value. Specifically from ear- overlap between the Middle and Modern variants. lier periods, texts are often the only forms of in- An example of the data is given below, from the formation that have been preserved. One prob- work ’Van den vos Reynaerde’ (‘About Reynard lem that arises when studying these texts is the the Fox’), part of the Comburg manuscript that difference between the language the text is writ- was written between 1380-1425. Here, the first ten in and the modern variant that the researchers text is the original text, the second text is a who want to study the texts know and speak them- modern translation in Dutch by Walter Verniers selves. It takes a great deal of deciphering and in- and a translation in English is added below that terpretation to be able to grasp these texts. Our aim for clarity. is to facilitate scholars such as historians who do not possess extensive knowledge of Middle Dutch “Doe al dat hof versamet was, who are studying medieval texts. We do this by at- Was daer niemen, sonder die das, tempting to generate literal translations of the sen- Hine hadde te claghene over Reynaerde, tences in the text into modern language. In par- Den fellen metten grijsen baerde.” ticular we focus on the task of translating Middle Dutch to modern Dutch. The transformation be- “Toen iedereen verzameld was, tween language variants, either synchronically or was er niemand -behalve de das- diachronically, can be seen as a paraphrase and a die niet kwam klagen over Reynaert, translation task, as it is often impossible to catego- die deugniet met zijn grijze baard.” rize two languages as either variants or different languages. “When everyone was gathered, We define Middle Dutch as a collection of there was noone -except the badger- closely related West Germanic dialects that were who did not complain about Reynaert, spoken and written between 1150 and 1500 in the that rascal with his grey beard.”

11 Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 11–19, Sofia, Bulgaria, August 8 2013. c 2013 Association for Computational Linguistics dictionary and a list of spelling rules. By plug- We can observe that although the two Dutch ging in other dictionaries and spelling rules, the texts are quite different, there is a large amount tool can be adapted for other tasks. Kestemont et of character overlap in the words that are used. al. (2010) describe a machine learning approach Our aim is to use this character overlap to compen- to normalize the spelling in Middle Dutch Text sate for the lower amount of data that is available. from the 12th century. They do this by converting We compare three different approaches to trans- the historical spelling variants to single canonical late Middle Dutch into Modern Dutch: a standard (present-day) lemmas. Memory-based learning is Phrase-Based machine translation (PBMT) ap- used to learn intra-lemma spelling variation. Al- proach, a PBMT approach with additional prepro- though these approaches normalize the text, they cessing based on Needleman-Wunsch sequence do not provide a translation. alignment, and a character bigram based PBMT More work has been done in the area of trans- approach. The PBMT approach with preprocess- lating between closely related languages and deal- ing identifies likely translations based on character ing with data sparsity that occurs within these lan- overlap and adds them as a dictionary to improve guage pairs (Hajicˇ et al., 2000; Van Huyssteen the statistical alignment process. The PBMT ap- and Pilon, 2009). Koehn et al. (2003) have shown proach based on character bigrams rather than that there is a direct negative correlation between translating words, transliterates character bigrams the size of the vocabulary of a language and the and in this way improves the transformation pro- accuracy of the translation. Alignment models cess. We demonstrate that these two approaches are directly affected by data sparsity. Uncommon outperform standard PBMT in this task, and that words are more likely to be aligned incorrectly to the PBMT transliteration approach based on char- other words or, even worse, to large segments of acter bigrams performs best. words (Och and Ney, 2003). Out of vocabulary (OOV) words also pose a problem in the trans- 2 Related work lation process, as systems are unable to provide translations for these words. A standard heuristic Language transformation by machine translation is to project them into the translated sentence un- within a language is a task that has not been stud- translated. ied extensively before. Related work is the study by Xu et al. (2012). They evaluate paraphrase sys- Various solutions to data sparsity have been tems that attempt to paraphrase a specific style of studied, among them the use of part-of-speech writing into another style. The plays of William tags, suffixes and word stems to normalize Shakespeare and the modern translations of these words (Popovic and Ney, 2004; De Gispert and works are used in this study. They show that their Marino, 2006), the treatment of compound words models outperform baselines based on dictionar- in translation (Koehn and Knight, 2003), translit- ies and out-of-domain parallel text. Their work eration of names and named entities, and advanced differs from our work in that they target writing models that combine transliteration and transla- in a specific literary style and we are interested in tion (Kondrak et al., 2003; Finch et al., 2012) translating between diachronic variants of a lan- or learn unknown words by analogical reason- guage. ing (Langlais and Patry, 2007). Work that is slightly comparable is the work by Vilar et al. (2007) investigate a way to han- Zeldes (2007), who extrapolates correspondences dle data sparsity in machine translation between in a small parallel corpus taken from the Modern closely related languages by translating between and Middle Polish Bible. The correspondences are characters as opposed to words. The words in the extracted using machine translation with the aim parallel sentences are segmented into characters. of deriving historical grammar and lexical items. Spaces between words are marked with a special A larger amount of work has been published about character. The sequences of characters are then spelling normalization of historical texts. Baron used to train a standard machine translation model and Rayson (2008) developed tools for research in and a language model with n-grams up to n = 16. Early Modern English. Their tool, VARD 2, finds They apply their system to the translation between candidate modern form replacements for spelling the related languages Spanish and Catalan, and variants in historical texts. It makes use of a find that a word based system outperforms their

12 letter-based system. However, a combined sys- tion 7. We close with a discussion of our results in tem performs marginally better in terms of BLEU Section 8. scores. Tiedemann (2009) shows that combining Middle Dutch Modern Dutch character-based translation with phrase-based versamet verzameld translation improves machine translation quality was was in terms of BLEU and NIST scores when trans- niemen niemand lating between Swedish and Norwegian if the die de OOV-words are translated beforehand with the das das character-based model. claghene klagen Nakov and Tiedemann (2012) investigate the over over use of character-level models in the translation be- Reynaerde Reynaert tween Macedonian and Bulgarian movie subtitles. metten met zijn Their aim is to translate between the resource poor grijsen grijze language Macedonian to the related language Bul- baerde baard garian, in order to use Bulgarian as a pivot in or- Table 1: Examples of character overlap in words der to translate to other languages such as English. from a fragment of ’Van den vos Reynaerde’ Their research shows that using character bigrams shows improvement over a word-based baseline. It seems clear that character overlap can be used to improve translation quality in related languages. 4 Language transformation Models We therefore use character overlap in language 4.1 PBMT baseline transformation between two diachronic variants of a language. For our baseline we use the Moses software to train a phrase based machine translation (PBMT) 3 This study model (Koehn et al., 2007). In general, a statistical machine translation model normally finds a best In this study we investigate the task of translating translation E˜ of a text in language F for a text from Middle Dutch to Modern Dutch. Similarly in language E by combining a translation model to resource poor languages, one of the problems P (F |E) with a language model P (E): that are apparent is the small amount of parallel Middle Dutch - Modern Dutch data that is avail- E˜ = arg max P (F |E)P (E) able. To combat the data sparseness we aim to use E∈E∗ the character overlap that exists between the Mid- In phrase-based machine translation the sen- dle Dutch words and their Modern Dutch counter- tence F is segmented into a sequence of I phrases parts. Examples of overlap in some of the words during decoding. Each source phrase is then trans- given in the example can be viewed in Table1. We lated into a target phrase to form sentence E. are interested in the question how we can use this Phrases may be reordered. overlap to improve the performance of the transla- The GIZA++ statistical alignment package tion model. We consider three approaches: (A) (Och and Ney, 2003) is used to perform the word Perform normal PBMT without any preprocess- alignments, which are later combined into phrase ing, (B) Apply a preprocessing step in which we alignments in the Moses pipeline to build the pinpoint words and phrases that can be aligned language transformation model. GIZA++ imple- based on character overlap and (C) perform ma- ments IBM Models 1 to 5 and an HMM word chine translation not to words but to character bi- alignment model to find statistically motivated grams in order to make use of the character over- alignments between words. We first tokenize our lap. data. We then lowercase all data and use all sen- We will first discuss the PBMT baseline, fol- tences from the Modern Dutch part of the cor- lowed by the PBMT + overlap system and the pus to train an n-gram language model with the character bigram PBMT transliteration system in SRILM toolkit (Stolcke, 2002). Then we run the Section 4. We then describe the experiment with GIZA++ aligner using the training pairs of sen- human judges in Section 6, and its results in Sec- tences in Middle Dutch and Modern Dutch. We

13 execute GIZA++ with standard settings and we in the matrix C(i, j) where i = 1, 2, ..., N and optimize using minimum error rate training with j = 1, 2, ..., M, the score of any cell C(i, j) is BLEU scores. The Moses decoder is used to gen- then the maximum of: erate the translations.

4.2 PBMT with overlap-based alignment Before using the Moses pipeline we perform a qdiag = C(i − 1, j − 1) + s(i, j) preprocessing alignment step based on character q = C(i − 1, j) + g overlap. Word and phrase pairs that exhibit a large down amount of character overlap are added to the par- qright = C(i, j − 1) + g allel corpus that GIZA++ is trained on. Every time we find a phrase or word pair with large overlap it is added to the corpus. This helps bias the align- Here, s(i, j) is the substitution score for let- ment procedure towards aligning similar words ters i and j, and g is the gap penalty score. If i and reduces alignment errors. To perform the pre- and j match, the substitution score is in fact the processing step we use the Needleman-Wunsch matching score. The table is filled this way recur- algorithm (Needleman and Wunsch, 1970). The sively, filling each cell with the maximum score of Needleman-Wunsch algorithm is a dynamic pro- the three possible options (diagonally, down and gramming algorithm that performs a global align- right). After this is done, an optimal path can ment on two sequences. Sequence alignment is be found by performing a traceback, starting in a method to find commonalities in two (or more) the lower right corner of the table and ending in sequences of some items or characters. One of- the upper left corner, by visiting cells horizontally, ten used example is the comparison of sequences vertically or diagonally, but only those cells with of DNA to find evolutionary differences and sim- the highest score. After this process we end up ilarities. Sequence alignment is also used in lin- with an alignment. guistics, where it is applied to finding the longest We use the Needleman-Wunsch algorithm to common substring or the differences or similari- find an optimal alignment of the Middle Dutch ties between strings. - Modern Dutch sentence pairs. We regard each The Needleman-Wunsch algorithm is a se- line as a sentence. In case of rhyming text, a quence alignment algorithm that optimizes a score frequent phenomenon in Middle Dutch text, lines function to find an optimal alignment of a pair of are usually parts of sentences. We then consider sequences. Each possible alignment is scored ac- each line a string, and we try to align as many cording to the score function, where the alignment characters and whitespaces to their equivalents in giving the highest similarity score is the optimal the parallel line. We split the aligned sentences alignment of a pair of sequences. If more than in each position where two whitespaces align and one alignment yields the highest score, there are we consider the resulting aligned words or phrases multiple optimal solutions. The algorithm uses as alignments. For each aligned word or phrase an iterative matrix to calculate the optimal solu- pair we calculate the Jaccard coefficient and if that tion. All possible pairs of characters containing is equal or higher than a threshold we add the one character from each sequence are plotted in aligned words or phrases to the training material. a 2-dimensional matrix. Then, all possible align- We heuristically set this threshold to 0.5. By using ments between those characters can be represented this method we already can find many-to-one and by pathways through the matrix. Insertions and one-to-many alignments. In this way we help the deletions are allowed, but can be penalized by GIZA++ alignment process by biasing it towards means of a gap penalty in the alignment. aligning words and phrases that show overlap. Ta- The first step is to initialize the matrix and fill in ble 2 illustrates this process for two lines. the gap scores in the top row and leftmost column. In our case we heuristically set the values of the 4.3 Character bigram transliteration scores to +1 for matches, -2 for mismatches and Another somewhat novel approach we propose -1 for gaps after evaluating on our development for Language Transformation is Character-based set. After initialization, we can label the cells transliteration. To circumvent the problem of

14 Middle Dutch: hine hadde+ ++te claghene over Reynaerde , Modern Dutch: di+e ++niet kwam klag+en+ over Reynaer+t , Jaccard 0.4 0.14 0 0.63 1 0.70 1

Middle Dutch: +den fe++llen met++ten grijsen baerde . Modern Dutch: die+ deugniet met zijn grijze+ baard+ . Jaccard 0.50 0.09 0.50 0.71 0.8 1

Table 2: Alignment of lines with Jaccard scores for the aligned phrases. A + indicates a gap introduced by the Needleman Wunsch alignment.

OOV-words and use the benefits of character over- #d da at t# en n# #d da aa ar lap more directly in the MT system, we build #d da at t# et t# #s st a translation model based on character bigrams, #d da at t# et t# #s similar to (Nakov and Tiedemann, 2012). Where #d da at t# ie et t# #s st they use this approach to translate between closely #d da at t# ie et t# #s related languages, we use it to translate between #d da at t# la an n# diachronic variants of a language. The sentences #d da at t# le et t# in the parallel corpus are broken into charac- #d da at t# n# #d da aa ar ro ter bigrams, with a special character representing #d da at t# n# #d da aa ar whitespaces. These bigrams are used to train the #d da at t# n# translation model and the language model. An ex- #d da at t# rd da at t# ample of the segmentation process is displayed in #d da at ts s# #d da at t# Table 3. We train an SRILM language model on #d da at ts si i# #h he eb bb be en the character bigrams and model sequences of up #d da at ts #d da at t# to 10 bigrams. We then run the standard Moses #d da at ts #w wa at t# pipeline, using GIZA++ with standard settings to #d da at tt tu #w wa at t# #j generate the phrase-table and we use the Moses decoder to decode the bigram sequences. A num- Table 4: Example entries from the character bi- ber of sample entries are shown in Table 4. As a gram Phrase-table, without scores. final step, we recombine the bigrams into words. The different sizes of the Phrase-table for the dif- system lines ferent approaches can be observed in Table 5. PBMT 20,092 PBMT + overlap 27,885 original segmented character bigram transliteration 93,594 Hine #H Hi in ne e# hadde #h ha ad dd de e# Table 5: Phrase-table sizes of the different models te #t te e# claghene #c cl la ag gh he en ne e# over #o ov ve er r# Table 6. All works are from the Digital Library of 1 Reynaerde #R Re ey yn na ae er rd de e# Dutch Literature . “Middle Dutch” is a very broad , #, ,# definition. It encompasses all Dutch language spo- ken and written between 1150 and 1500 in the Table 3: Input and output of the character bigram Netherlands and parts of Belgium. Works stem- segmenter ming from different centuries, regions and writers can differ greatly in their orthography and spelling conventions. No variant of Dutch was considered 5 Data Set standard or the norm; Middle Dutch can be con- sidered a collection of related lects (regiolects, di- Our training data consists of various Middle Dutch alects). This only adds to the problem of data literary works with their modern Dutch transla- tion. A breakdown of the different works is in 1http://www.dbnl.org

15 sparsity. Our test set consists of a selection of 6.2 Participants sentences from the Middle Dutch work Beatrijs, The participants in this evaluation study were 22 a Maria legend written around 1374 by an anony- volunteers. All participants were native speakers mous author. of Dutch, and participated through an online in- source text lines date of origin terface. All participants were adults, and 12 were Van den vos Reynaerde 3428 around 1260 male and 10 female. In addition to the 22 partici- Sint Brandaan 2312 12th century pants, one expert in the field of Middle Dutch also Gruuthuuse gedichten 224 around 1400 ’t Prieel van Trojen 104 13th century performed the experiment, in order to be able to Various poems 42 12th-14th cent. compare the judgements of the laymen and the ex- pert. Table 6: Middle Dutch works in the training set 6.3 Procedure The participants were asked to rank three different Middle Si seide: ’Ic vergheeft u dan. Dutch Ghi sijt mijn troest voer alle man automatic literal translations of Middle Dutch text. Modern Ze zei: ’ik vergeef het je dan. For reference, they were also shown a modern (of- Dutch Je bent voor mij de enige man ten not literal) translation of the text by Dutch au- PBMT Ze zei : ’ Ik vergheeft u dan . Gij ze alles in mijn enige voor al man thor Willem Wilmink. The order of items to judge PBMT + Ze zei : ’ Ik vergheeft u dan . was randomized for each participant, as well as the Overlap dat ze mijn troest voor al man order of the output of the systems for each sen- Char. Bigram Ze zeide : ’ Ik vergeeft u dan . PBMT Gij zijt mijn troost voor alle man tence. The criterium for ranking was the extent to Middle Dat si daer snachts mochte bliven. which the sentences could be deciphered and un- Dutch ’Ic mocht u qualijc verdriven,’ derstood. The participants were asked to always Modern omdat ze nu niet verder kon reizen. provide a ranking and were not allowed to assign Dutch ’Ik kan u echt de deur niet wijzen,’ PBMT dat ze daar snachts kon bliven . the same rank to multiple sentences. In this way, ’ Ik kon u qualijc verdriven , ’ each participant provided 25 rankings where each PBMT + dat ze daar s nachts kon blijven . pair of sentences received a distinct rank. The pair Overlap ’ Ik kon u qualijc verdriven , ’ Char. Bigram dat zij daar snachts mocht blijven . with rank 1 is considered best and the pair with 3 PBMT ’ Ik mocht u kwalijk verdrijven , is considered worst.

Table 7: Example output system mean rank 95 % c. i. PBMT 2.44 (0.03) 2.38 - 2.51 PBMT + Overlap 2.00 (0.03) 1.94 - 2.06 6 Experiment char. bigram PBMT 1.56 (0.03) 1.50 - 1.62

In order to evaluate the systems, we ran an exper- Table 8: Mean scores assigned by human subjects, iment to collect human rankings of the output of with the standard error between brackets and the the systems. We also performed automatic evalu- lower and upper bound of the 95 % confidence in- ation. terval 6.1 Materials Because of the nature of our data, in which sen- 7 Results tences often span multiple lines, it is hard to eval- uate the output on the level of separate lines. We 7.1 Human judgements therefore choose to evaluate pairs of lines. We ran- In this section we report on results of the exper- domly choose a line, and check if it is part of a iment with judges ranking the output of the sys- sensible sentence that can be understood without tems. To test for significance of the difference more context. If that is the case, we select it to in- in the ranking of the different systems we ran re- clude in our test set. In this way we select 25 pairs peated measures analyses of variance with system of lines. We evaluate the translations produced by (PBMT, PBMT + Overlap, character bigram MT) the three different systems for these sentences. Ex- as the independent variable, and the ranking of the amples of the selected sentences and the generated output as the dependent variable. Mauchly‘s test corresponding output are displayed in Table 7. for sphericity was used to test for homogeneity of

16 PBMT PBMT + char. X2 PBMT + Overlap system and the character bigram overlap bigram system scored highest for each of these cases at PBMT p < .005. These results are detailed in Table 9. 2.05 2.59 1.36 16.636** When comparing the judgements of the partici- 2.77 1.82 1.41 21.545** pants with the judgements of an expert, we find 2.50 1.27 2.23 18.273** a significant medium Pearson correlation of .65 1.95 1.45 2.59 14.273** (p < .001) between the judgements of the expert 2.18 2.36 1.45 10.182** and the mean of the judgements of the participants. 2.45 2.00 1.55 9.091* This indicates that the judgements of the laymen 2.91 1.77 1.32 29.545** are indeed useful. 2.18 2.27 1.55 6.903* 2.14 2.00 1.86 0.818 7.2 Automatic judgements 2.27 1.73 2.00 3.273 In order to attempt to measure the quality of the 2.68 1.68 1.64 15.364** transformations made by the different systems au- 2.82 1.95 1.23 27.909** tomatically, we measured NIST scores by compar- 2.68 2.09 1.23 23.545** ing the output of the systems to the reference trans- 1.95 2.55 1.50 12.091** lation. We do realize that the reference translation 2.77 1.86 1.36 22.455** is in fact a literary interpretation and not a literal 2.32 2.23 1.45 9.909** translation, making automatic assessment harder. 2.86 1.91 1.23 29.727** Having said that, we still hope to find some effect 2.18 1.09 2.73 30.545** by using these automatic measures. We only re- 2.05 2.09 1.86 0.636 port NIST scores here, because BLEU turned out 2.73 2.18 1.09 30.545** to be very uninformative. In many cases sentences 2.41 2.27 1.32 15.545** would receive a BLEU score of 0. Mauchly‘s test 2.68 2.18 1.14 27.364** for sphericity was used to test for homogeneity of 1.82 2.95 1.23 33.909** variance for the NIST scores, and was not signifi- 2.73 1.95 1.32 21.909** cant. We ran a repeated measures test with planned 2.91 1.77 1.32 29.545** pairwise comparisons made with the Bonferroni method. We found that the NIST scores for the dif- Table 9: Results of the Friedman test on each of ferent systems differed significantly (F (2, 48) = the 25 sentences. Results marked * are significant 6.404, p < .005, η2 = .211). The average NIST at p < 0.05 and results marked ** are significant p scores with standard error and the lower and up- at p < 0.01 per bound of the 95 % confidence interval can be seen in Table 10. The character bigram translit- eration model scores highest with 2.43, followed variance, but was not significant, so no corrections by the PBMT + Overlap model with a score of had to be applied. Planned pairwise comparisons 2.30 and finally the MT model scores lowest with were made with the Bonferroni method. The mean a NIST score of 1.95. We find that the scores ranking can be found in Table 8 together with the for the PBMT model differ significantly from the standard deviation and 95 % confidence interval. PBMT + Overlap model (p < .01) and the charac- We find that participants ranked the three systems 2 ter bigram PBMT model (p < .05), but the scores differently, F (2, 42) = 135, 604, p < .001, ηp = for the PBMT + Overlap and the character bigram .866. All pairwise comparisons are significant at PBMT model do not differ significantly. When p < .001. The character bigram model receives we compare the automatic scores to the human as- the best mean rank (1.56), then the PBMT + Over- signed ranks we find no significant Pearson corre- lap system (2.00) and the standard PBMT system lation. is ranked lowest (2.44). We used a Friedman test to detect differences across multiple rankings. We 8 Conclusion ran the test on each of the 25 K-related samples, and found that for 13 sentences the ranking pro- In this paper we have described two modifications vided by the test subjects was equal to the mean of the standard PBMT framework to improve the ranking: the PBMT system ranked lowest, then the transformation of Middle Dutch to Modern Dutch

17 system mean NIST 95 % c. i. ple is the disappearing silent ’h’ after a ’g’. This PBMT 1.96 (0.18) 1.58 - 2.33 often leads to transliterated words of which the PBMT + overlap 2.30 (0.21) 1.87 - 2.72 spelling is only partially correct. Apparently the char. bigram PBMT 2.43 (0.20) 2.01 - 2.84 human judges rate these ’half words’ higher than completely wrong words, but automatic measures Table 10: Mean NIST scores, with the standard such as NIST are insensitive to this. error between brackets and the lower and upper We have also reported the NIST scores for the bound of the 95 % confidence interval output of the standard PBMT approach and the two proposed variants. We see that the NIST scores show a similar patterns as the human judge- by using character overlap in the two variants. We ments: the PBMT + Overlap and character bigram described one approach that helps the alignment PBMT systems both achieve significantly higher process by adding words that exhibit a certain NIST scores than the normal PBMT system. How- amount of character overlap to the parallel data. ever, the PBMT + Overlap and character bigram We also described another approach that translates PBMT models do not differ significantly in scores. sequences of character bigrams instead of words. It is interesting that we find no significant corre- Reviewing the results we conclude that the use of lation between human and automatic judgements, character overlap between diachronic variants of a leading us to believe that automatic judgements language is beneficial in the translation process. are not viable in this particular scenario. This is More specifically, the model that uses character perhaps due to the fact that the reference transla- bigrams in translation instead of words is ranked tions the automatic measures rely on are in this best. Also ranked significantly better than a stan- case not literal translations but rather more loosely dard Phrase Based machine translation approach translated literary interpretations of the source text is the approach using the Needleman-Wunsch al- in modern Dutch. The fact that both versions are gorithm to align sentences and identify words or written on rhyme only worsens this problem, as phrases that exhibit a significant amount of char- the author of the Modern Dutch version is often acter overlap to help the GIZA++ statistical align- very creative. ment process towards aligning the correct words We think techniques such as the ones described and phrases. We have seen that one issue that here can be of great benefit to laymen wishing to is encountered when considering the task of lan- investigate works that are not written in contem- guage transformation from Middle Dutch to Mod- porary language, resulting in improved access to ern Dutch is data sparseness. The number of lines these older works. Our character bigram translit- used to train on amounts to a few thousand, and eration model may also play some role as a com- not millions as is more common in SMT. It is putational model in the study of the evolution of therefore crucial to use the inherent character over- orthography in language variants, as it often will lap in this task to compensate for the lack of data generate words that are strictly speaking not cor- and to make more informed decisions. The char- rect, but do resemble Modern Dutch in some way. acter bigram approach is able to generate a trans- Automatic evaluation is another topic for future lation for out of vocabulary words, which is also work. It would be interesting to see if an automatic a solution to the data sparseness problem. One measure operating on the character level correlates area where the character bigram model often fails, better with human judgements. is translating Middle Dutch words into Modern Dutch words that are significantly different. One example can be seen in Table 7, where ’mocht’ References is translated by the PBMT and PBMT + Overlap Alistair Baron and Paul Rayson. 2008. VARD 2: A systems to ’kon’ and left the same by the charac- tool for dealing with spelling variation in historical ter bigram transliteration model. This is probably corpora. In Proceedings of the Postgraduate Confer- ence in Corpus Linguistics, Birmingham, UK. Aston due to the fact that ’mocht’ still exists in Dutch, University. but is not as common as ’kon’ (meaning ’could’). A. De Gispert and J. B. Marino. 2006. Linguistic another issue to consider is the fact that the char- knowledge in statistical phrase-based word align- acter bigram model learns character mappings that ment. Natural Language Engineering, 12(1):91– are occurring trough out the language. One exam- 108.

18 Andrew Finch, Paul Dixon, and Eiichiro Sumita. 2012. Franz Josef Och and Hermann Ney. 2003. A sys- Rescoring a phrase-based machine transliteration tematic comparison of various statistical alignment system with recurrent neural network language mod- models. Computational Linguistics, 29(1):19–51. els. In Proceedings of the 4th Named Entity Work- shop (NEWS) 2012, pages 47–51, Jeju, Korea, July. Maja Popovic and Hermann Ney. 2004. Towards the Association for Computational Linguistics. use of word stems and suffixes for statistical ma- chine translation. In LREC. European Language Re- Jan Hajic,ˇ Jan Hric, and Vladislav Kubon.ˇ 2000. Ma- sources Association. chine translation of very close languages. In Pro- ceedings of the sixth conference on Applied natural Andreas Stolcke. 2002. SRILM - an extensible lan- language processing, pages 7–12. Association for guage modeling toolkit. In In Proc. Int. Conf. on Computational Linguistics. Spoken Language Processing, pages 901–904, Den- ver, Colorado. Mike Kestemont, Walter Daelemans, and Guy De Pauw. 2010. Weigh your words—memory- Jorg¨ Tiedemann. 2009. Character-based PSMT for based lemmatization for Middle Dutch. Literary closely related languages. In Llu´ıs Marques´ and and Linguistic Computing, 25(3):287–301, Septem- Harold Somers, editors, Proceedings of 13th An- ber. nual Conference of the European Association for Machine Translation (EAMT’09), pages 12 – 19, Philipp Koehn and Kevin Knight. 2003. Feature-rich Barcelona, Spain, May. statistical translation of noun phrases. In ACL ’03: Proceedings of the 41st Annual Meeting on Asso- Gerhard B Van Huyssteen and Sulene´ Pilon. 2009. ciation for Computational Linguistics, pages 311– Rule-based conversion of closely-related languages: 318, Morristown, NJ, USA. Association for Compu- a dutch-to-afrikaans convertor. 20th Annual Sympo- tational Linguistics. sium of the Pattern Recognition Association of South Africa, December. Philip Koehn, Franz Josef Och, and Daniel Marcu. David Vilar, Jan-Thorsten Peter, and Hermann Ney. 2003. Statistical phrase-based translation. In Second Work- Proceedings of the 2003 Conference of the North 2007. Can we translate letters? In shop on Statistical Machine Translation American Chapter of the Association for Computa- , pages 33– tional Linguistics on Human Language Technology- 39, Prague, Czech Republic, jun. Association for Computational Linguistics. Volume 1, pages 48–54. Association for Computa- tional Linguistics. Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2010. Paraphrase generation as mono- Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris C. lingual translation: Data and evaluation. In Burch, Marcello Federico, Nicola Bertoldi, Brooke B. Mac Namee J. Kelleher and I. van der Sluis, ed- Cowan, Wade Shen, Christine Moran, Richard Zens, itors, Proceedings of the 10th International Work- Chris Dyer, Ondrej Bojar, Alexandra Constantin, shop on Natural Language Generation (INLG and Evan Herbst. 2007. Moses: Open source toolkit 2010), pages 203–207, Dublin. for statistical machine translation. In ACL. The As- sociation for Computer Linguistics. Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for style. In P. Koehn. 2005. Europarl: A parallel corpus for statis- COLING, pages 2899–2914. tical machine translation. In MT summit, volume 5. Amir Zeldes. 2007. Machine translation between lan- Grzegorz Kondrak, Daniel Marcu, and Kevin Knight. guage stages: Extracting historical grammar from a 2003. Cognates can improve statistical translation parallel diachronic corpus of Polish. In Matthew models. In HLT-NAACL. Davies, Paul Rayson, Susan Hunston, and Pernilla Danielsson, editors, Proceedings of the Corpus Lin- Philippe Langlais and Alexandre Patry. 2007. Trans- guistics Conference CL2007. University of Birming- lating unknown words by analogical learning. In ham. EMNLP-CoNLL, pages 877–886. ACL.

Preslav Nakov and Jorg¨ Tiedemann. 2012. Combin- ing word-level and character-level models for ma- chine translation between closely-related languages. In Proceedings of the 50th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 301–305, Jeju Island, Korea, July. Association for Computational Linguistics.

S. B. Needleman and C. D. Wunsch. 1970. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of molecular biology, 48(3):443–453, March.

19 Comparison between historical population archives and decentralized databases

Marijn Schraagen and Dionysius Huijsmans Leiden Institute of Advanced Computer Science (LIACS) Leiden University, The Netherlands schraage,huijsman @liacs.nl { }

Abstract and results for a test database. An application of the matching in a family tree visualization tool is Differences between large-scale historical provided in Section 7. A conclusion and directions population archives and small decentral- for future research are provided in Section 8. ized databases can be used to improve data The most important concepts used throughout quality and record connectedness in both this paper are defined as follows: types of databases. A parser is developed Record. Unit of matching and linkage. A record to account for differences in syntax and refers to a Genlias certificate (Section 3) or a data representation models. A matching Gedcom certificate reconstruction (Sections 3 and procedure is described to discover records 4), unless stated otherwise. from different databases referring to the Record match. A pair of records that refer to the same historical event. The problem of ver- same event (birth, marriage, or death). ification without reliable benchmark data Record link. A pair of records that refer to related is addressed by matching on a subset of events (e.g., birth and death of the same person). record attributes and measuring support Field similarity measure. Similarity between for the match using a different subset of field values, e.g., number of days between dates. attributes. An application of the matching Record similarity measure. Similarity require- procedure for comparison of family trees ments for selected fields and relations between the is discussed. A visualization tool is de- requirements. scribed to present an interactive overview Edit distance. Minimum number of character of comparison results. insertions, deletions and/or substitutions needed to transform one string into another (Levenshtein 1 Introduction distance). In the historical demographics and genealogy do- Name sequence. Concatenation of person names main, research data can be collected from cen- from a record. tralized databases such as a historical census or Person name. Single name, i.e., given name or civil registry. Alternatively, decentralized data can family name. be collected from, e.g., personal archives or lo- cal organizations. Complementary and conflicting 2 Related work information between these sources can be valu- able for research if the overlap, i.e., matching Automatic matching and linkage of historical records, is correctly identified. This paper de- records has been researched for several decades. scribes a method to discover matching records in An early example can be found in (Winchester, centralized and decentralized data for the prob- 1970), using Soundex as field similarity measure lem of family reconstruction. An overview of re- to compare census data. An approach from the lated work is presented in Section 2. Section 3 French-speaking Canadian province of Quebec, describes the two different data formats. Section 4 using a custom phonetic code, is described in provides a mapping procedure between the differ- (Bouchard and Pouyez, 1980). In this approach ent data formats. In Section 5 the matching proce- different types of data are merged together. The dure is explained at the conceptual and technical researchers state that “the most fundamental rule level. Section 6 provides a verification procedure is that we never try to link individuals, but rather

20 Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 20–28, Sofia, Bulgaria, August 8 2013. c 2013 Association for Computational Linguistics pairs of individuals; that is: couples [. . . ] It can have been proposed (Wilson, 2001) that take into be demonstrated easily that individual linkage is account matches between individuals and match- liable to result in uncertain, false, or missed links, ing links between individuals. More elaborate whereas the result of the linkage of couples is very linkage of pedigree data is described in (Quass and reliable”. The approach is implemented as fol- Starkey, 2003), using feature weights and thresh- lows: “we accept as candidates for linkage those olds to increase linkage performance. pairs which have at least two exactly identical el- Using various definitions of record, such as a ements”. Experience with the dataset used in the single individual, multiple individuals, families current research has resulted in a similar approach (i.e., multiple individuals in a family relation), or (see Section 5). Linkage on the Quebec dataset has events (i.e., multiple individuals in a certain rela- been developed by the same authors (Bouchard, tion at a specific point in time), most research in 1992). The 1992 paper discusses the use of field record linkage is either directed towards matching values: “the various fields can serve as identi- of records, i.e., asserting equal reference, or link- fiers (linkage), controls (validation), or variables age of related (but not equal) records using match- (analysis).” The notion of internal validation is ing of record elements (e.g., a birth record linked discussed further in Section 6. Later approaches to a marriage record based on a match between to record linkage have focussed on scalability of the child and the bridegroom). In social networks linkage methods for large amounts of data, see, research a different type of linkage is common, e.g., (Christen and Gayler, 2008). where records are linked but not matched (e.g., two people sharing common interests). Occasion- A detailed overview of elements from genealo- ally this type of link is used in historical record gical records and the application of each element linkage as well (Smith and Giraud-Carrier, 2006). for linkage is provided in (Wilson, 2011). Besides Test corpora have been developed (Schone et historical and/or genealogical data, various other al., 2012), (Bouchard, 1992), however these are in- types of data have been used for development and trinsically domain- and language-specific. More- testing of algorithms, such as hospital records, over, these corpora are generally not readily avail- phone book records, customer records, etc. How- able for research. ever, algorithms generally assume a certain level of uniformity in data representation, both at a tech- 3 Data formats nical and at a conceptual level. This means that generally pedigrees are linked to other pedigrees The centralized data used in the experiments is ex- 1 but not to civil certificates, and vice versa. Events tracted from the Dutch Genlias database. Genlias and individuals (actors) have been modelled to- contains civil certificates (around 15 million in to- gether using NLP techniques (Segers et al., 2011), tal) from the Netherlands, for the events of birth, however these approaches are mostly not applica- marriage and death. Most documents originate ble for genealogical data both because of the lack from the 19th and early 20th century. A record of natural language resources to identify and link (see Figure 1 for an example) consists of the type instances of actors and events, as well as the differ- of event, a serial number, date and place, and par- ence in scope of the model (participants of histor- ticipant details. The parents are also listed for the ically significant events vs. every person that ex- main subject(s) of the document, i.e., the newborn isted during a certain time period). Some attempts child, bride and groom, and deceased person for have been made to facilitate data exchange and birth, marriage and death certificates, respectively. accessability in the genealogical domain, either The documents do not contain identifiers for indi- by presenting a standardized format (the Gedcom viduals. No links are provided between documents standard (GEDCOM Team, 1996) being the most or individuals. succesful example), by conversion into a standard- The decentralized data is extracted from a fam- ized format (Kay, 2006; Kay, 2004), by enforcing ily tree database in the Gedcom (Genealogical a Semantic Web ontology (Zandhuis, 2005), or Data Communication) format. In this format ge- by defining a framework that accepts custom data nealogical data is stored based on individuals and models as metadata to be provided upon exchange nuclear (immediate) families, instead of events as of the genealogical data itself (Woodfield, 2012). 1The Genlias data is currently maintained by WieWasWie, Algorithmic solutions for merging of pedigrees see http://www.wiewaswie.nl (in Dutch)

21 Type: birth certificate 0 @F294@ FAM Serial number: 176 1 HUSB @I840@ Date: 16 - 05 - 1883 1 WIFE @I787@ Place: Wonseradeel 1 MARR Child: Sierk Rolsma 2 DATE 30 MAY 1874 Father: Sjoerd Rolsma 2 PLAC Wonseradeel Mother: Agnes Weldring 1 CHIL @I847@ 1 CHIL @I848@ 1 CHIL @I849@ Figure 1: Genlias birth certificate. 0 @I840@ INDI 1 NAME Sjoerd/Rolsma/ 1 BIRT 2 DATE 13 FEB 1849 in Genlias. Every individual or family in Gedcom 1 DEAT is assigned a unique identifier. Records for indi- 2 DATE 17 JAN 1936 viduals usually contain personal information like 1 FAMS @F294@ 0 @I787@ INDI names, birth and death date, etc. The families in 1 NAME Agnes/Welderink/ which the individual participates, either as child 1 SEX F 1 BIRT or as parent, are also indicated. A family record 2 DATE ca 1850 lists the individuals and their roles. Marriage in- 1 FAMS @F294@ formation (date, place) can also be present in a 0 @I849@ INDI 1 NAME Sierk/Rolsma/ family record. Using the record identifiers, a link 1 BIRT network between individuals and families can be 2 DATE 16 MAY 1883 constructed. 2 PLAC Wonseradeel 2 SOUR Gedcom is a text-based free entry format. The 3 REFN 176 standard (GEDCOM Team, 1996) states that “A 1 FAMC @F294@ record is represented as a sequence of tagged, variable-length lines, arranged in a hierarchy. A Figure 2: Gedcom database fragment, showing a line always contains a hierarchical level number, a selection of fields from a FAM record (family) tag, and an optional value. A line may also con- and three INDI records (individual). tain a cross-reference identifier or a pointer.” (see Figure 2 for an example). The Gedcom standard is used by a wide variety of genealogical appli- com is the preferred option. cations, ranging from full-featured commercially There are various tools available to perform the available software to small scripts. The imple- required data transformation. Many genealogy mentation of the standard can differ between ap- programs can export Gedcom data to, e.g., XML plications, as well as the content format entered by or SQL databases which can be queried to con- users. The next section describes a parsing proce- struct events. Alternatively, dedicated Gedcom dure designed to process this kind of data. parsers exist for a number of programming lan- guages (such as Perl (Johnson, 2013), C (Verthez, 4 Parsing 2004), Python (Ball, 2012), XSLT (Kay, 2004)) Prior to the actual record matching, a mapping be- that provide data structures to manipulate the Ged- tween the data formats must be performed. This com data from within code. However, the data requires either a reconstruction of events from structures are still centered around individuals and the Gedcom file, or vice versa a reconstruction families and the performance of the tools is to a of individuals and nuclear families from Genlias. greater or lesser degree sensitive to violations of The first option requires links between Gedcom (some version of) the Gedcom standard. The rest records, for example to construct a birth record of this section describes a more general parsing from the three individual records of the child algorithm that can be applied to any kind of level- and parents using the intermediate family record. numbered textual data. The second option requires links between Gen- The parser (see Figure 3) uses a Prolog DCG- lias certificates, for example to construct a family style grammar to specify the elements of target record from the birth certificates of several chil- records (see Figure 4 for an example). Tags found dren. Record links are available in Gedcom only, in lines from the database file are pushed on a and therefore reconstruction of events from Ged- stack one by one. Before a tag is pushed, all cur-

22 S while←L ∅ readline(database) do Protorecord ← if(L.level = 0) then [@], [fam,chil(2)]:[indi,birt,date], id L.value [fam,chil(2)]:[indi,birt,plac], ← while(S.top.level L.level) do [fam,chil(2)]:[indi,birt,sour,refn], [fam,chil(2)]:[@], ≥ S.pop() [fam,chil(2)]:[indi,name], [fam,husb]:[@], S.push(L.tag) [fam,husb]:[indi,name], [fam,wife]:[@], foreach terminalList grammar do [fam,wife]:[indi,name] if(S = terminalList)∈ then index(id,terminalList) L.value Certificate foreach id index do ← foreach∈target grammar do @F294@, 16 MAY 1883, Wonseradeel, 176, if(pointerList∈ target) then @I849@, Sierk/Rolsma/, @I840@, Sjoerd/Rolsma/, duplicate(target,id,pointerList)∈ @I787@, Agnes/Welderink/ foreach protoRecord ( target duplicates) do foreach terminalList∈ { protoRecord} ∪ do output index(id,terminalList)∈ Figure 5: Parsing example for index value output record← separator ← @F294@ using the pointer [@F294@,CHIL(2)], which is @I849@. Figure 3: Parser algorithm. similar data formats. Additionally, only informa- birthcertificate > [@],[fam,chil(+)]:birthbasic, tion that corresponds to an element of a target −− [fam,husb]:personname, [fam,wife]:personname. record is indexed, resulting in a light-weight pro- birthbasic > birthdate, birthplace, birthref, personname. birthdate −−> [indi,birt,date]. cedure. The output of the parser can be directly birthplace−− > [indi,birt,plac]. used for record matching, which is described in birthref −−> [indi,birt,sour,refn]. personname−− > [@],[indi,name]. the next section. target > −−birthcertificate. −− 5 Matching

Figure 4: Grammar fragment. Special characters: After parsing, both databases are represented in ’@’ level 0-value (record id), ’+’ pointer list, the same data format. This enables a definition ’:’ pointer dereference. of similarity between records based on the val- ues of corresponding fields. In the current exper- iments a partial similarity measure is used, mean- rent elements with an equal or higher level num- ing that any sufficiently large subset of the corre- ber are popped, which makes the stack correspond sponding fields must be similar whereas the com- to the current branch in the database hierarchy. If plement set remains unchecked. This approach as- the stack corresponds to a list of terminal symbols sumes sparseness of high-dimensional data, which in the grammar, then the current line is indexed implies that the set of field values of each record is for later use by the value at level 0. All grammar unique and moreover any large subset of field val- rules are expanded to terminal symbols and subse- ues is also unique. This property can easily be ver- quently dereferenced for each of the index values ified on a given database and if it holds, the simi- in the previous step. If an expanded rule contains a larity measure can be simplified accordingly. For pointer list (indicated by a + symbol) then the rule the current experiments this allows for name vari- is duplicated for each element of the pointer list ation in civil certificates which is hard to detect associated to the current index value before dere- automatically by similarity measures. A certifi- ferencing. As an example the algorithm in Fig- cate, as discussed in Section 3, generally contains ure 3 applied to the database in Figure 2 using the at least three individuals, which amounts to six grammar in Figure 4 on the index value @F294@ names in total (given names and family names). will produce three duplicate protorecords which If one of the names is subject to large variation in can be dereferenced to certificates. Figure 5 pro- two matching records (for example Elizabeth vs. vides an example that matches the Genlias certifi- Lisa), this match might be undetected when us- cate in Figure 1. Note that the family name of the ing all names in the record comparison. However, mother differs between the databases. by ignoring this field in a partial comparison the The use of a domain-independent grammar pro- match will be discovered. vides a flexible parser for Gedcom or structurally A partial record similarity measure can be de-

23 fined by stating similarity requirements for each Edit distance threshold 5 of the fields used in the measure and relations be- Large year margin 10 tween the requirements. As an example, consider Small year margin the matching between marriage certificates based marriage 2 on the year of marriage and the names of the bride birth, death 0 and bridegroom (four names in total) which is used Marriage match 153 in the current experiments, as stated in Figure 6. no match 23 Note that the first clause in this definition requires Birth match 335 an exact match on person names. This has the con- no match 276 ceptual advantage that exact matching is more re- Death match 100 liable than similarity matching based on, e.g., edit no match 239 distance. Additionally, exact matching allows for straightforward string indexing and efficient look- Table 1: Matching parameters and results. up. Memory consumption is less efficient, the ex- ample index of two names out of four requires 4 knowledge of the domain. A subset of the Gedcom 2 = 6 entries per record. Therefore it might be necessary to adjust the similarity measure to meet records is used to match the timeframe of the Gen- ( ) computational resources. lias database (1796–1920). Settings and match- ing results are displayed in Table 1. The match- ing is performed for the three main types of civil At least two out of four names are exactly certificates: birth, marriage and death. For birth equal, and and death certificates the marriage record similar- the year of marriage is equal or different ity measure (Figure 6) is used replacing the roles by a small margin, or of bride and bridegroom by mother and father of the child or deceased person for birth and death the year of marriage is different by a certificates respectively (i.e., the name of the child larger margin and the edit distance be- or deceased person itself is not used). To avoid tween name sequences is below a small confusion with other siblings, the small year mar- threshold, or gin for birth and death certificates is set to zero. If multiple matching candidates are found using the year of marriage in a record is spec- the record similarity measure, the match with the ified as a range and the year of marriage smallest edit distance between name sequences is in another record is within this range, used. The large amount of missed matches for and the edit distance between name se- birth and death certificates is expected, because quences is below a small threshold. the Genlias database is still under active develop- ment and a significant number of birth and death Figure 6: Record similarity measure certificates are not yet digitized. Moreover, Ged- for marriage certificates. com databases generally contain many peripheral individuals for which no parents are listed (usu- ally inlaws of the family of interest), prohibiting 6 Results and verification the reconstruction of birth and death certificates. The record similarity measure in Figure 6 is ap- Verification of record matches should ideally be plied to the Genlias database and a sample Ged- performed using a test set of known matches (a com database containing 1327 individuals and 423 gold standard). However, for this particular com- families. As preprocessing, given names are re- bination of databases such a test set is not avail- duced to the first token (for example: Quentin able. The lack of test sets extends to the majority Jerome Tarantino Quentin Tarantino). Sepa- of decentralized historical data, as well as Genlias rate family name prefixes,→ which are common in itself (which does not have any kind of internal Dutch, are stripped using a stop list (for example: links or verification sets). This is a quite unde- Vincent van Gogh Vincent Gogh). The edit dis- sirable situation given the large variation in data tance threshold and→ year margins required by the quality and coverage between databases in the his- similarity measure are set according to empirical torical domain. Because the characteristics of any

24 two databases regarding the contents can differ to names). The year is used in the similarity measure, a large degree, the performance of a matching al- but the day and month can be used for support. For gorithm obtained on one database is not indicative the current experiments three levels of support are for other databases. Put differently: every appli- defined: exact date match, a difference of 1 to 7 cation of a matching algorithm has to perform its days, or a difference of more than 7 days. own verification, which is difficult in the absence In case of limited support from the verification of test sets. fields, edit distance (or any other string similarity measure) can be used as an indication of the cor- 6.1 Internal verification rectness of a match. A possible solution for the verification problem 6.2 Toponym mapping is to re-use the sparseness assumption to obtain Toponyms cannot always be compared directly, a measure of support for a match. The matches because of the difference in use between Genlias returned by the similarity measure are based on a and most Gedcom databases. In Genlias the to- subset of fields. If other field values are equal or ponym that denotes the location of the event is al- similar as well, they provide additional support for ways the municipality that has issued the certifi- the match independent of the similarity measure. cate. In a Gedcom database often the actual lo- Note that this solution is only applicable if there cation of the event is used, which can be a town are fields available which are not used in the record that is part of a larger municipality. A comparison similarity measure. Moreover these fields should between toponyms is therefore more informative have a certain discriminative power, which rules after mapping each toponym to the corresponding out categorical variables like gender or religion. municipality. In the current experiments a refer- For many linkage tasks extra fields are not avail- ence database of Dutch toponyms is used to per- able, for example linking a marriage certificate of form the mapping. Because the municipal organi- a person to the marriage certificate of this person’s zation in the Netherlands changes over time, the parents, in which case the only available informa- year of the event is required for a correct map- tion about the parents are the person names. How- ping. Ambiguity for toponyms (i.e., multiple lo- ever, in the current experiments a certificate from cations with the same name) can generally be re- one database is being matched to the same certifi- solved using the province of the event. In case that cate in another database, therefore the amount of the province is not recorded the toponym can be available information is much larger. disambiguated by choosing the location with the A candidate field for verification is the serial most inhabitants by default. number, which has been recorded since the start of the civil registry in the Netherlands. The num- 6.3 Interpretation of support figures bers are assigned per year by the municipality Table 2 shows the results of verification using se- issuing the certificate, meaning that the combi- rial numbers, dates and mapped toponyms as sup- nation of municipality, year and serial number port fields. The support figures should be inter- uniquely references a certificate (also known as preted with the distribution of data values in mind. a persistent identifier or PID). A shared PID be- The first two rows of Table 2 represent matches tween two records in a match therefore provides with equal serial numbers. Most of these matches strong support for this match. However, in a Ged- have equal PIDs (toponym and year equal as well). com database serial numbers are not necessarily Given that each PID is unique these matches are included. The source of the data can be something correct. Differences in toponym are usually small different than the civil registry, such as church for matches with equal serial numbers, therefore records, or the database author might just have a PID match can be assumed (although support omitted the serial number. Moreover, if the source is higher for true PID matches). The third row of the Gedcom record is the civil registry, then the represents matches with the same toponym and match is not very indicative of the performance date, and also two names equal (by definition of of the similarity measure in combining different the similarity measure). Note again that the match data sources. Therefore, the serial number is of was selected using the names and the year only, limited use only for verification purposes. Other and verified using the toponym and the full date. candidate fields are dates and toponyms (location These matches could be incorrect, because it is

25 possible that different couples with (partially) the field marriage birth death same name got married on the same day in the s t d e same place, for example. In the Genlias database + + + 69 170 9 this is the case for around 0.3% of all marriage + - + 2 30 0 certificates. Therefore, the sparseness assump- – + + 41 20 1 tion largely holds for this set of fields and these – + 0 33 6 ∼ matches can also be considered correct. Similarly, – + – 2 1 0 other verification field values can be interpreted in – – + 10 2 7 terms of confidence in a match (based on the va- –– 2 5 10 ∼ lidity of the sparseness assumption) or counterev- ––– 3 11 2 3 ≤ idence against a match (in case of large differ- ––– > 3 16 72 64 ences in field values). For the current experiments, total 153 335 100 the last row of matches should be considered in- correct. The relatively large number of incorrect Table 2: Verification results. matches for birth and death certificates can be at- Columns: (s)erial number, (t)oponym, (d)ate, tributed to the lack of coverage in Genlias. The (e)dit distance. Support level: + equal, 1–7 days difference,– not equal (s,t) or best match is returned, however this assumes true ∼ matches to be present in the data set. The record > 7 days difference (d). Edit distance is only similarity match can be adjusted using the verifi- used for the matches without support from the cation fields, however it is preferred to keep simi- verification fields (final two rows). larity computation and verification separated.

7 Application To demonstrate the application of the method, a comparison is performed on links between mar- The previous sections have discussed matching riage certificates in Genlias and corresponding records from different databases that refer to the links in the sample Gedcom database used in the same event. However, most research in histori- matching experiments. A marriage certificate con- cal record linkage is focussed on links between tains the marriage couple and the parents of both events, such as a birth and the marriage of the bride and bridegroom. A link can be defined be- parents listed in the birth certificate. These links tween a marriage and the marriage of one of the can be added to a database by manual annota- parent couples (see Figure 7). For the Genlias tion or using automatic linkage methods. Differ- database links have been constructed by select- ent databases in the same population domain are ing all record pairs with a maximum Levenshtein likely to contain complementary and conflicting edit distance of 3 between name sequences. Ad- links, which can be used to increase the quality ditional record links are computed by converting and quantity of links in both databases. To com- each name to a base form and selecting record pare links between databases the records need to pairs with matching base name sequences. The be matched first, which can be achieved using the details of the link computation are beyond the record matching method from the current research. scope of this paper, for the current experiments

Certificate 09915109 Certificate 09914931 Municipality: Mijnsheerenland Municipality: Mijnsheerenland Date: September 18, 1868 Date: August 29, 1895 Serial number: 9 Serial number: 13

Parents groom Parents bride Parents groom Parents bride Jacob Kaptein Arij Dekker Ewoud Kaptein Souverein Smits Dirksje van den Broek Kniertje Verschoor Geertrui Dekker Lijntje Delgens

Marriage couple Marriage couple Ewout Kaptein Jacob Kaptein Geertrui Dekker Lijntje Smits

Figure 7: Example of a link between two Genlias marriage certificates, containing a small spelling variation: Ewout vs. Ewoud.

26 @F100@ 01-05-1824 Sikke Sasses van der Zee Aafke Klazes de Boer Afke de Boer

@F171@ @F15@ 13-05-1848 09-05-1857 Sjoerd Riemerts Riemersma Jan Johannes Altena Johanna Sikkes van der Zee Klaaske Sikkes van der Zee

@F16@ @F17@ @F18@ @F13@ @F19@ 9797998 02-07-1892 16-11-1889 09-01-1896 13-06-1896 ~1900 08-05-1895 Johannes Altena Eke Foekema Sikke Altena Ruurd Altena H Wesseling Hendrikus Wesseling Elisabeth Vonk Aaltje Altena Cornelia Verkooyen Anna Jans Rolsma Agatha Altena Agatha Altena Cornelia Verkooijen

@F122@ @F123@ @F124@ ~1920 ~1925 18-05-1923 Sikkes ? Bartolomeus Mathias van Oerle Sikke Altena IJbeltje Altena Klaaske Altena Trijntje Homminga

Figure 8: Visualization of link comparison. only the resulting set of links between Genlias 8 Conclusion and future work marriage certificates is of interest. In the Ged- com database links between marriages are already In this paper a method is described to compare present. The link comparison procedure is as fol- a dataset based on events (Genlias) to a dataset lows: first, marriage certificates are matched us- based on individuals (the Gedcom model). This ing the method described in Section 5. For every method is complementary to most current ap- matched certificate the marriages of the children proaches in record linkage, in which only datasets are identified using the links from Genlias and the with the same conceptual structure are compared. Gedcom database (cf. Figure 7). These two sets of The parser (Section 4) facilitates the transforma- marriages are aligned using a slightly more strict tion of data formats. A combination of multi- version of the record similarity measure in Fig- ple string indexing and field similarity measures ure 6, to accomodate for the inherent similarity in provides a computationally efficient and flexible names and timeframe of sibling marriages. Us- record matching method, as described in Sec- ing the alignment, the links can be divided into tion 5. The problem of verification without gold three categories: present in both databases, present standard test data is addressed in Section 6. An in the Gedcom database only, or present in Gen- application of the method in a visualization tool is lias only. A visualization tool is developed that presented in Section 7. shows the results of the comparison in a link tree In future research, other Gedcom databases (see Figure 8), which can be browsed by expand- can be presented to the matching procedure. A ing or collapsing record links. Colours indicate crowdsourcing set-up can be envisioned to per- differences between databases (red and blue for form large-scale data collection and evaluation of the Gedcom database and Genlias, respectively). the approach. The matching procedure itself can Records @F19@ and 9797998 are an example of be refined by improving the record similarity mea- a false negative. The lower row is found in the sure or by incorporating a network approach in Gedcom database only because these records are which record links can contribute to matching. Fi- outside of the Genlias timeframe. The tool enables nally, functionality can be added to the visualiza- users to provide their own Gedcom database and tion tool, preferably resulting in a public release. identify differences with the nation-wide Genlias Acknowledgment database. Due to data licensing issues the tool has not yet been released, however it could be inte- This work is part of the research programme grated in the Genlias website in the future. LINKS, which is financed by the Netherlands Or- ganisation for Scientific Research (NWO), grant 640.004.804. The authors would like to thank Tom Altena for the use of his Gedcom database.

27 References Peter Verthez. 2004. The Gedcom parser library. http://gedcom-parse.sourceforge. Madeleine Ball. 2012. python-gedcom: Python net/. module for parsing, analyzing, and manipulat- ing GEDCOM files. https://github.com/ D. Randall Wilson. 2001. Graph-based remerging of madprime/python-gedcom/. genealogical databases. In Proceedings of the 1st Annual Family History Technology Workshop. Fam- Gerard´ Bouchard and Christian Pouyez. 1980. Name ilySearch. variations and computerized record linkage. Histor- ical Methods: A Journal of Quantitative and Inter- D. Randall Wilson. 2011. Genealogical record link- disciplinary History, 13(2):119–125. age: Features for automated person matching. In Proceedings of RootsTech 2011, pages 331–340. Gerard´ Bouchard. 1992. Current issues and new FamilySearch. prospects for computerized record linkage in the province of Quebec.´ Historical Methods: A Jour- Ian Winchester. 1970. The linkage of historical nal of Quantitative and Interdisciplinary History, records by man and computer: Techniques and prob- 25(2):67–73. lems. The Journal of Interdisciplinary History, 1(1):107–124. Peter Christen and Ross Gayler. 2008. Towards scal- Scott Woodfield. 2012. Effective sharing of family able real-time entity resolution using a similarity- history information. In Proceedings of the 12th An- aware inverted index approach. In Seventh Aus- nual Family History Technology Workshop. Family- tralasian Data Mining Conference (AusDM 2008), Search. volume 87, pages 51–60. ACS. Ivo Zandhuis. 2005. Towards a genealogical ontol- GEDCOM Team. 1996. The GEDCOM standard ogy for the semantic web. In Humanities, Comput- release 5.5. Technical report, Family and Church ers and Cultural Heritage: Proceedings of the XVI History Department, The Church of Jesus Christ of international conference of the Association for His- Latter-day Saints, Salt Lake City. tory and Computing, pages 296–300. Paul Johnson. 2013. Gedcom — a module to manip- ulate Gedcom genealogy files. http://search. cpan.org/˜pjcj/Gedcom-1.18/.

Michael Kay. 2004. Up-conversion using XSLT 2.0. In Proceedings of XML: From Syntax to Solutions. IDEAlliance.

Michael H. Kay. 2006. Positional grouping in XQuery. In Proceedings of the 3rd International Workshop on XQuery Implementation, Experience and Perspec- tives (XIME-P).

Dallan Quass and Paul Starkey. 2003. Record linkage for genealogical databases. In KDD-2003 Workshop on Data Cleaning, Record Linkage, and Object Con- solidation, pages 40–42.

Patrick Schone, Chris Cummings, Stuart Davey, Michael Jones, Barry Nay, and Mark Ward. 2012. Comprehensive evaluation of name matching across historic and linguistic boundaries. In Proceedings of the 12th Annual Family History Technology Work- shop. FamilySearch.

Roxane Segers, Marieke van Erp, and Lourens van der Meij. 2011. Hacking history via event extraction. In Proceedings of the sixth international conference on Knowledge capture, K-CAP ’11, pages 161–162. ACM.

Matthew Smith and Christophe Giraud-Carrier. 2006. Genealogical implicit affinity network. In Proceed- ings of the 6th Annual Family History Technology Workshop. FamilySearch.

28 Semi-automatic Construction of Cross-period Thesaurus

Chaya Liebeskind, Ido Dagan, Jonathan Schler Computer Science Department Bar-Ilan University Ramat-Gan, Israel [email protected], [email protected], [email protected]

Abstract earlier historical periods, different aspects of that term were mentioned. For example, in our Jewish Cross-period (diachronic) thesaurus con- historical corpora, the modern term birth control, struction aims to enable potential users to has no equivalent ancient term, However, different search for modern terms and obtain se- contraceptive methods were described in our his- mantically related terms from earlier pe- torical texts that are semantically similar to birth riods in history. This is a complex task not control. Thus, a related term is considered sim- previously addressed computationally. In ilar to the target term when it refers to the same this paper we introduce a semi-automatic concept. iterative Query Expansion (QE) scheme The goal of our research is to support con- for supporting cross-period thesaurus con- structing a high-quality publishable thesaurus, as struction. We demonstrate the empirical a cultural resource on its own, alongside being benefit of our scheme for a Jewish cross- a useful tool for supporting searches in the do- period thesaurus and evaluate its impact on main. Since the precision of fully automatically- recall and on the effectiveness of lexicog- constructed thesauri is typically low (e.g. (Mi- rapher manual effort. halcea et al., 2006)), we present a semi-automatic setting for supporting thesaurus construction by a 1 Introduction and Background domain expert lexicographer. Our recall-oriented In the last decade, there is a growing interest in ap- setting assumes that manual effort is worthwhile plying Natural Language Processing (NLP) meth- for increasing recall as long as it is being utilized ods to historical texts due to the increased avail- effectively. ability of these texts in digital form (Sporleder, Corpus-based thesaurus construction is an ac- 2010; Sanchez-Marco´ et al., 2011; Piotrowski, tive research area (Curran and Moens, 2002; Kil- 2012). The specific linguistic properties of histor- garriff, 2003; Rychly´ and Kilgarriff, 2007; Liebe- ical texts, such as nonstandard orthography, gram- skind et al., 2012; Zohar et al., 2013). Typi- mar and abbreviations, pose special challenges for cally, two statistical approaches for identifying se- NLP. One of this challenges, which has not been mantic relatedness between words were investi- addressed so far, is the problem of bridging the gated: first-order (co-occurrence-based) similarity lexical gap between modern and ancient language. and second-order (distributional) similarity (Lin, In this paper, we address the interesting task 1998; Gasperin et al., 2001; Weeds and Weir, of cross-period thesaurus (a.k.a. diachronic the- 2003; Kotlerman et al., 2010). In this research, saurus) construction. A thesaurus usually contains we focus on statistical measures of first-order sim- thousands of entries, denoted here as target terms. ilarity (see Section 2). These methods were found Each entry includes a list of related terms, cover- to be effective for thesaurus construction as stand- ing various semantic relations. A cross-period the- alone methods and as complementary to second- saurus aims to enable the potential user to search order methods (Peirsman et al., 2008). First-order for a modern term and get related terms from ear- measures assume that words that frequently occur lier periods. Thus, in a cross-period thesaurus together are topically related (Schutze¨ and Peder- the target terms are modern while their related sen, 1997). Thus, co-occurrence provides an ap- terms are ancient. In many cases, while the actual propriate approach to identify highly related terms modern term (or its synynom) does not appear in for the thesaurus entries.

29 Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 29–35, Sofia, Bulgaria, August 8 2013. c 2013 Association for Computational Linguistics In general, there are two types of historically- mary in Sections 5 and 6. relevant corpora: ancient corpora of ancient lan- guage, and modern corpora with references and 2 Automatic Thesaurus Construction mixed mentions to ancient language (termed here Automatic thesaurus construction focuses on the corpora ). Since in our setting the thesaurus’ target process of extracting a ranked list of candidate terms are modern terms, which do not appear in related terms (termed candidate terms) for each ancient corpora, co-occurrence methods would be given target term. We assume that the top ranked directly applicable only over a mixed corpus. In candidates will be further examined (manually) by a preliminary experiment, we applied the Liebe- a lexicographer, who will select the eventual re- skind et al. (2012) algorithmic scheme, which ap- lated terms for the thesaurus entry. plies first-order similarity and morphological as- Statistical measures of first-order similarity pects of corpus-based thesaurus construction, on (word co-occurrence), such as Dice coefficient a mixed corpus of our historical domain. We ob- (Smadja et al., 1996) and Pointwise Mutual In- served that the target terms had low frequency in formation (PMI) (Church and Hanks, 1990), were this corpus. Since statistical co-occurrence mea- commonly used to extract ranked lists of candi- sures have poor performance over low statistics, date related terms. These measures consider the the experiment’s results were not satisfactory. We number of times in which each candidate term co- therefore looked for ways to increase the number occurs with the target term, in the same document, of documents in the statistical extraction process, relative to their total frequencies in the corpus. and decided that applying query expansion (QE) In our setting, we construct a thesaurus for a techniques might be a viable solution. morphologically rich language (Hebrew). There- We recognized two potential types of sources fore, we followed the Liebeskind et al. (2012) al- of lexical expansions for the target terms. The gorithmic scheme designed for these cases, sum- first is lexical resources available over the inter- marized below. First, our target term is repre- net for extracting different types of semantic rela- sented in its lemma form. For each target term we tions (Shnarch et al., 2009; Bollegala et al., 2011; retrieve all the corpus documents containing this Hashimoto et al., 2011). The second is lists of given target term. Then, we define a set of candi- related terms extracted from a mixed corpus by date terms, which are represented in their surface a first-order co-occurrence measure. These lists form, that consists of all the terms in all these doc- contain both ancient and modern terms. Although uments. Next, the Dice co-occurrence score be- only ancient terms will be included in the final tween the target term and each of the candidates thesaurus, modern terms can be utilized for QE is calculated, based on their document-level statis- to increase thesaurus coverage. Furthermore, ex- tics in the corpus. After sorting the terms based on panding the target term with ancient related terms their scores, the highest rated candidate terms are enables the use of ancient-only corpora for co- clustered into lemma-based clusters. Finally, we occurrence extraction. rank the clusters by summing the co-occurrence Following these observations, we present an it- scores of their members and the highest rated clus- erative interactive QE scheme for bootstrapping ters constitute the candidate terms for the given thesaurus construction. This approach is used to target term, to be presented to a domain expert. bridge the lexical gap between modern and ancient terminology by means of statistical co-occurrence 3 Iterative Semi-automatic Scheme for approaches. We demonstrate the empirical advan- Cross-period Thesaurus Construction tage of our scheme over a cross-period Jewish do- As explained in Section 1, our research focuses main and evaluate its impact on recall and on the on a semi-automatic setting for supporting cross- effectiveness of the lexicographer manual effort. period thesaurus construction by a lexicographer. The remainder of this paper is organized as fol- In this work, we assume that a list of modern tar- lows: we start with a description of the statistical get terms is given as input. Then, we automatically thesaurus construction method that we utilize in extract a ranked list of candidate related terms for our scheme. Our main contribution of the itera- each target term using statistical measures, as de- tive scheme is described in Section 3, followed by tailed in Section 2. Notice that at this first step re- a case-study in Section 4 and evaluation and sum- lated terms can be extracted only from the mixed

30 corpora, in which the given (modern) target term may occur. Next, a lexicographer manually se- lects, from the top ranked candidates, ancient re- lated terms for the thesaurus entry as well as terms for QE. The QE terms may be either ancient or modern terms from the candidate list, or terms from a lexical resource. Our iterative QE scheme iterates over the QE terms. In each iteration, a QE term replaces the target term’s role in the statistics extraction process. Candidate related terms are extracted for the QE term and the lexicographer judges their relevancy with respect to the original target term. Notice that if the QE term is modern, only the mixed corpora can be utilized. However, Figure 1: Semi-automatic Algorithmic Scheme if the QE term is ancient, the ancient corpora are also utilized and may contribute additional related For efficiency, only new candidates that were not terms. judged in pervious iterations are given for judge- The algorithmic scheme we developed for the- ment. The stopping criterion is when there are no saurus construction is illustrated in Figure 1. Our additional expansions in the expansions list. input is a modern target term. First, we au- Since the scheme is recall-oriented, the aim of tomatically extract candidates by statistical co- the annotation process is to maximize the the- occurrence measures, as described in Section 2. saurus coverage. In each iteration, the domain Then, a domain-expert annotates the candidates. expert annotates the extracted ranked list of can- didate terms until k sequential candidates were The manual selection process includes two de- judged as irrelevant. This stopping criterion for cisions on each candidate (either modern or an- each iteration controls the efforts to increase recall cient): (i) whether the candidate is related to the while maintaining a low, but reasonable precision. target term and should be included in its thesaurus In our setting, we extract ancient related terms entry, and (ii) whether this candidate can be used for modern terms. Therefore, in order to utilize as a QE term for the original target term. The co-occurrence statistics extraction, our scheme re- second decision provides input to the QE process, quires both ancient and mixed corpora, where which triggers the subsequent iterations. Follow- the first iteration utilizes only the mixed corpora. ing the first decision we filter the modern terms Then, our iterative scheme enables subsequent it- and include only ancient ones in the actual the- erations to utilize the ancient corpora as well. saurus. The classification of a candidate term as ancient 4 Case Study: Cross-period Jewish or modern is done automatically by a simple clas- Thesaurus sification rule: If a term appears in an ancient cor- pus, then it is necessarily an ancient term; other- Our research targets the construction of a cross- 1 wise, it is a modern term (notice that the converse period thesaurus for the Responsa project . The is not true, since an ancient term might appear in corpus includes questions on various daily issues modern documents). posed to rabbis and their detailed rabbinic an- swers, collected over fourteen centuries, and was In parallel to extracting candidate related terms used for previous IR and NLP research (Choueka from the corpus, we extract candidate terms also et al., 1971; Choueka et al., 1987; HaCohen- from our lexical resources, and the domain expert Kerner et al., 2008; Liebeskind et al., 2012; Zohar judges their fitness as well. Our iterative process et al., 2013). is applied over the expansions list. In each itera- The Responsa corpus’ documents are divided to tion, we take out an expansion term and automat- four periods: the 11th century until the end of the ically extract related candidates for it. Then, the 15th century, the 16th century, the 17th through annotator selects both ancient related terms for the the 19th centuries, and the 20th century until to- thesaurus and suitable terms, either modern or an- cient, for the expansion list for further iterations. 1Corpus kindly provided: http://biu.ac.il/jh/Responsa/

31 day. We considered the first three periods as our Method RT R Pro J ancient corpora along with the RaMBaM (Hebrew First-iteration 50 0.31 0.038 1307 acronym for Rabbi Mosheh Ben Maimon) writ- Baseline 63 0.39 0.024 2640 ings from the 12th century. For the mixed corpus Iterative 151 0.94 0.057 2640 we used the corpus’ documents from the last pe- riod, but due to relatively low volume of modern Table 1: Results Comparison documents we enriched it with additional modern collections (Tchumin collection 2, ASSIA (a Jour- selected in our case) sequential irrelevant candi- nal of Jewish Ethics and Halacha), the Medical- dates (First-iteration). Halachic Encyclopedia3, a collection of questions To evaluate our scheme’s performance, we used and answers written by Rabbi Shaul Israeli4, and several measures: total number of ancient related the Talmudic Encyclopedia (a Hebrew language terms extracted (RT), relative recall (R) and pro- encyclopedia that summarizes halachic topics of ductivity (Pro). Since we do not have any pre- the Talmud in alphabetical order). Hebrew Wik- defined thesaurus, our micro-averaged relative- tionary was used as a lexical resource for syn- recall considered the number of ancient related onyms. terms from the output of both methods (baseline For statistics extraction, we applied (Liebeskind and iterative) as the full set of related terms. Pro- et al., 2012) algorithmic scheme using Dice coef- ductivity was measured by dividing the total num- ficient as our co-occurrence measure (see Section ber of ancient related terms extracted (RT) by the 2). Statistics were calculated over bigrams from total number of the judgments performed for the corpora consisting of 81993 documents. method (J). 5.2 Results 5 Evaluation Table 1 compares the performance of our semi- 5.1 Evaluation Setting automatic iterative scheme with that of the base- We assessed our iterative algorithmic scheme by line over a test set of 30 modern target terms. Our evaluating its ability to increase the thesaurus cov- iterative scheme increases the average number of erage, compared to a similar non-iterative co- extracted related terms from 2.1 to 5, i.e., increas- occurrence-based thesaurus construction method. ing recall by 240%. The relative recall of the first- In our experiments, we assumed that it is worth iteration (0.31) is included in the relative recall of spending the lexicographer’s time as long as it is both the baseline and our iterative method. Iterat- productive, thus, all the manual annotations were ing over the first iteration increases recall by 300% based on the lexicographer efforts to increase re- (from 50 to 151 terms), while adding more judge- call until reaching the stopping criterion. ments to the non-iterative method increases recall We used Liebeskind et al. (2012) algorithmic only by 26% (to 63 terms). The productivity of the scheme as our non-iterative baseline (Baseline). iterative process is higher even than the productiv- For comparison, we ran our iterative scheme, cal- ity of the first iteration, showing that the iterative culated the average number of judgments per tar- process optimizes the lexicographer’s manual ef- get term (88) and set the baseline stopping crite- fort. rion to be the same number of judgements per tar- Table 2 shows examples of thesaurus target get. Thus, we ensured that the number of judge- terms and their ancient related terms, which were 5 ments for our iterative algorithm and for the base- added by our iterative scheme . Since the related line is equal, and thus coverage increase is due to a terms are ancient Halachic terms, we explain them better use of lexicographer’s effort. For complete- rather than translate them to English. ness, we present the results of the non-iterative al- We further analyze our scheme by comparing gorithm with the stopping criterion of the iterative the use of ancient versus modern terms in the itera- algorithm, when reaching k (k=10 was empirically tive process. Although modern related terms were not included in our cross-period thesaurus, in the 2http://www.zomet.org.il/?CategoryID=170 judgement process the lexicographer judged their 3http://medethics.org.il/website/index.php/en/research- 2/encyclopedia-of-jewish-medical-ethics 5To facilitate readability we use a transliteration of He- 4http://www.eretzhemdah.org/data/uploadedfiles/ebooks/14- brew using Roman characters; the letters used, in Hebrew sfile.pdf lexico-graphic order, are abgdhwzxTiklmns`pcqrst.ˇ

32 Figure 2: The extraction of ancient terms versus modern terms in the iterative process

Target term Related term zkwiwt iwcrim (copyright) hsgt gbwl (trespassing) iwrd lamnwt xbrw ([competitively] enter his friend’s profession) `ni hmhpk bxrrh (a poor man is deciding whether to buy a cake and another person comes and takes it) hmtt xsd (euthanasia) rwb gwssin lmith (most dying people die) xii s`hˇ (living for the moment) hpsqt hriwn (abortion) xwtkin h`wbr (killing the fetus) hwrg npsˇ (killing a person) rwdp (pursuer, a fetus endangering its mother’s life) tiknwn hmspxhˇ (birth control) slwˇ sˇ nsimˇ msmˇ swtˇ bmwk (three types of women allowed to use cotton di- aphragm) dsˇ mbpnim wzwrh mbxwc (withdrawal method) hprt xwzh (breach of contract) biTwl mqx (cancelling a purchase) dina dgrmi (indirect damage) mqx t`wt (erroneous bargain) srwb pqwdh (insubordination) mwrd bmlkwt (rebel against the sovereign [government]) imrh at pik (to disobey) Avner and khni Nob (a biblical story: king Saul ordered to slay Ahimilech to- gether with 85 priests. Avner, the captain of Saul’s guard, disobeyed the order.)

Table 2: Examples for the iterative scheme’s contribution relevancy too. In Figure 2, we report the number the effectiveness of QE for information retrieval of modern related terms in comparison to the num- (Voorhees, 1994; Xu and Croft, 1996), our results ber of ancient related terms for each iteration. In show that QE for thesaurus construction in an iter- parallel, we illustrate the number of ancient expan- ative interactive setting is beneficial for increasing sions in proportion to the number of modern ex- thesaurus’ coverage substantially. pansions. The x-axis’ values denote the iterations, while the y-axis’ values denote the number of ex- 6 Conclusions and Future Work pansions and related terms respectively. For each We introduced an iterative interactive scheme for iteration, the expansions chart presents the expan- cross-period thesaurus construction, utilizing QE sions that were extracted while the related terms techniques. Our semi-automatic algorithm signif- chart presents the extracted related terms, of which icantly increased thesaurus coverage, while op- the ancient ones were included in the thesaurus. timizing the lexicographer manual effort. The Since the input for our scheme is a modern target scheme was investigated for Hebrew, but can be terms, the first iteration extracted more modern re- generically applied for other languages. lated terms than ancient terms and utilized more We plan to further explore the suggested scheme modern expansions than ancient. However, this by utilizing additional lexical resources and QE proportion changed in the second iteration, prob- algorithms. We also plan to adopt second-order ably thanks to the ancient expansions retrieved in distributional similarity methods for cross-period the first iteration. thesaurus construction. Although there are often mixed results on

33 References Dekang Lin. 1998. Automatic retrieval and clustering of similar words. In Proceedings of the 36th Annual D. Bollegala, Y. Matsuo, and M. Ishizuka. 2011. Meeting of the Association for Computational Lin- A web search engine-based approach to mea- guistics and 17th International Conference on Com- sure semantic similarity between words. Knowl- putational Linguistics - Volume 2, ACL ’98, pages edge and Data Engineering, IEEE Transactions on, 768–774, Stroudsburg, PA, USA. Association for 23(7):977–990. Computational Linguistics.

Yaacov Choueka, M. Cohen, J. Dueck, Aviezri S. Rada Mihalcea, Courtney Corley, and Carlo Strappa- Fraenkel, and M. Slae. 1971. Full text document rava. 2006. Corpus-based and knowledge-based retrieval: Hebrew legal texts. In SIGIR, pages 61– measures of text semantic similarity. In Proceedings 79. of the 21st national conference on Artificial intelli- gence - Volume 1, AAAI’06, pages 775–780. AAAI Yaacov Choueka, Aviezri S. Fraenkel, Shmuel T. Klein, Press. and E. Segal. 1987. Improved techniques for pro- cessing queries in full-text systems. In SIGIR, pages Yves Peirsman, Kris Heylen, and Dirk Speelman. 306–315. 2008. Putting things in order. First and second order context models for the calculation of semantic sim- Kenneth Ward Church and Patrick Hanks. 1990. Word ilarity. In 9es Journees´ internationales d’Analyse association norms, mutual information, and lexicog- statistique des Donnees´ Textuelles (JADT 2008). raphy. Comput. Linguist., 16(1):22–29, March. Lyon, France. James R. Curran and Marc Moens. 2002. Improve- ments in automatic thesaurus extraction. In Pro- Michael Piotrowski. 2012. Natural language process- ceedings of the ACL-02 workshop on Unsupervised ing for historical texts. Synthesis Lectures on Hu- lexical acquisition - Volume 9, ULA ’02, pages 59– man Language Technologies, 5(2):1–157. 66, Stroudsburg, PA, USA. Association for Compu- tational Linguistics. Pavel Rychly´ and Adam Kilgarriff. 2007. An ef- ficient algorithm for building a distributional the- Caroline Gasperin, Pablo Gamallo, Alexandre Agus- saurus (and other sketch engine developments). In tini, Gabriel Lopes, Vera De Lima, et al. 2001. Us- Proceedings of the 45th Annual Meeting of the ACL ing syntactic contexts for measuring word similarity. on Interactive Poster and Demonstration Sessions, In Workshop on Knowledge Acquisition and Catego- ACL ’07, pages 41–44, Stroudsburg, PA, USA. As- rization, ESSLLI, Helsinki, Finland. sociation for Computational Linguistics.

Yaakov HaCohen-Kerner, Ariel Kass, and Ariel Peretz. Cristina Sanchez-Marco,´ Gemma Boleda, and Llu´ıs 2008. Combined one sense disambiguation of ab- Padro.´ 2011. Extending the tool, or how to anno- breviations. Proceedings of ACL08: HLT, Short Pa- tate historical language varieties. In Proceedings of pers, pages 61–64. the 5th ACL-HLT Workshop on Language Technol- ogy for Cultural Heritage, Social Sciences, and Hu- Chikara Hashimoto, Kentaro Torisawa, Stijn manities, LaTeCH ’11, pages 1–9, Stroudsburg, PA, De Saeger, Junichi Kazama, and Sadao Kuro- USA. Association for Computational Linguistics. hashi. 2011. Extracting paraphrases from definition sentences on the web. In Proceedings of the 49th Hinrich Schutze¨ and Jan O. Pedersen. 1997. A Annual Meeting of the Association for Computa- cooccurrence-based thesaurus and two applications tional Linguistics (ACL), pages 1087–1097. to information retrieval. Inf. Process. Manage., 33(3):307–318, May. Adam Kilgarriff. 2003. Thesauruses for natu- ral language processing. In Proceedings of the Eyal Shnarch, Libby Barak, and Ido Dagan. 2009. Ex- Joint Conference on Natural Language Processing tracting lexical reference rules from wikipedia. In and Knowledge Engineering, pages 5–13, Beijing, Proceedings of the Joint Conference of the 47th An- China. nual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan of the AFNLP: Volume 1 - Volume 1, ACL ’09, pages Zhitomirsky-geffet. 2010. Directional distributional 450–458, Stroudsburg, PA, USA. Association for similarity for lexical inference. Nat. Lang. Eng., Computational Linguistics. 16(4):359–389, October. Frank Smadja, Kathleen R. McKeown, and Vasileios Chaya Liebeskind, Ido Dagan, and Jonathan Schler. Hatzivassiloglou. 1996. Translating collocations for 2012. Statistical thesaurus construction for a mor- bilingual lexicons: a statistical approach. Comput. phologically rich language. In *SEM 2012: The Linguist., 22(1):1–38, March. First Joint Conference on Lexical and Computa- tional Semantics, pages 59–64, Montreal,´ Canada, Caroline Sporleder. 2010. Natural language process- 7-8 June. Association for Computational Linguis- ing for cultural heritage domains. Language and tics. Linguistics Compass, 4(9):750–768.

34 Ellen M. Voorhees. 1994. Query expansion using lexical-semantic relations. In Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information re- trieval, SIGIR ’94, pages 61–69, New York, NY, USA. Springer-Verlag New York, Inc. Julie Weeds and David Weir. 2003. A general frame- work for distributional similarity. In Proceedings of the 2003 conference on Empirical methods in nat- ural language processing, EMNLP ’03, pages 81– 88, Stroudsburg, PA, USA. Association for Compu- tational Linguistics. Jinxi Xu and W. Bruce Croft. 1996. Query expan- sion using local and global document analysis. In Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, SIGIR ’96, pages 4–11, New York, NY, USA. ACM.

Hadas Zohar, Chaya Liebeskind, Jonathan Schler, and Ido Dagan. 2013. Automatic thesaurus construction for cross generation corpus. Journal on Comput- ing and Cultural Heritage (JOCCH), 6(1):4:1–4:19, April.

35 Language Technology for Agile Social Media Science

Simon Wibberley Jeremy Reffin David Weir Department of Informatics Department of Informatics Department of Informatics University of Sussex University of Sussex University of Sussex [email protected] [email protected] [email protected]

Abstract is being said about their brands and products, with language processing technology being used We present an extension of the DUALIST to capture relevant comments or conversations and tool that enables social scientists to engage apply some form of sentiment analysis (SA), in or- directly with large Twitter datasets. Our der to derive insights into what is being said. This approach supports collaborative construc- paper forms part of a growing body of work that tion of classifiers and associated gold stan- is attempting to broaden the scope of SMA be- dard data sets. The tool can be used to yond the realm of product marketing, and into ar- build classifier cascades that decomposes eas of concern to social scientists (Carvalho et al., tweet streams, and provide analysis of tar- 2011; Diakopoulos and Shamma, 2010; Gonzalez- geted conversations. A central concern is Bailon et al., 2010; Marchetti-Bowick and Cham- to provide an environment in which social bers, 2012; O’Connor et al., 2010; Tumasjan et al., science researchers can rapidly develop an 2011; Tumasjan et al., 2010). informed sense of what the datasets look Social media presents an enormous opportunity like. The intent is that they develop, not for the social science research community, consti- only an informed view as to how the data tuting a window into what large numbers of people could be fruitfully analysed, but also how are talking. There are, however, significant obsta- feasible it is to analyse it in that way. cles facing social scientists interested in making 1 Introduction use of big social media datasets, and it is important for the NLP research community to gain a better In recent years, automatic social media analysis understanding as to how language technology can (SMA) has emerged, not only as a major focus support such explorations. of attention within the academic NLP community, A key requirement, and the focus of this paper, but as an area that is of increasing interest to a va- is agility: the social scientist needs to be able to riety of business and public sectors organisations. engage with the data in a way that supports an it- Among the many social media platforms in use to- erative process, homing in on a way of analysing day, the one that has received the most attention is the data that is likely to produce valuable insight. Twitter, the second most popular social media net- Given what is typically a rather broad topic as a work in the world with over 400 million tweets starting point, there is a need to see what issues re- sent each day. The popularity of Twitter as a tar- lated to that topic are being discussed and to what get of SMA derives from both the public nature extent. It can be important to get a feeling for the of tweets, and the availability of the Twitter API kind of language being used in these discussions, which provides a variety of flexible methods for and there is a need to rapidly assess the accuracy scraping tweets from the live Twitter stream. of the automated decision making. There is little A plethora of social media monitoring plat- value in developing an analysis of the data on an forms now exist, that are mostly concerned with approach that relies on the technology making de- providing product marketing oriented services1. cisions that are so nuanced that the method being For example, brand monitoring services seek to used is highly unreliable. As the answers to these provide companies with an understanding of what questions are being exposed, insights emerge from 1http://wiki.kenburbary.com/social-media-monitoring- the data, and it becomes possible for the social sci- wiki lists 230 Social Media Monitoring Solutions entist to progressively refine the topics that are be-

36 Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 36–42, Sofia, Bulgaria, August 8 2013. c 2013 Association for Computational Linguistics ing targetted, and ultimately create a way of au- to predict election outcomes by counting mentions tomatically analysing the data that is likely to be of political parties and/or politicians as ‘votes’ in insightful. various ways. Further content analysis is then Supporting this agile methodology presents se- overlaid, such as sentiment or mood anlysis, in vere challenges from an NLP perspective, where an attempt to improve performance. However the the predominant approaches use classifiers that generic language-analysis techniques that are ap- involve supervised machine learning. The need plied lead to little or no gain, often causing ad- for substantial quantities of training data, and justments to target question to something with less the detrimental impact on performance that re- strict assessment criteria, such as poll trend instead sults when applying them to “out-of-domain” data of election outcome (Tumasjan et al., 2010; Tu- mean that exisiting approaches cannot support the masjan et al., 2011; O’Connor et al., 2010; Sang agility that is so important when social scientists and Bos, 2012). This research has been criticised engage with big social media datasets. for applying out-of-domain techniques in a ‘black We describe a tool being developed in collab- box’ fashion, and questions have been raised as oration with a team of social scientists to support to how sensitive the results are to parameters cho- this agile methodology. We have built a frame- sen (Gayo-Avello, 2012; Jungherr et al., 2012). work based on DUALIST, an active learning tool • Approaches that employ manual analysis of for building classifiers (Settles, 2011; Settles and the data by researchers with a tailored analyti- Zhu, 2012). This framework provides a way for cal approach (Bermingham and Smeaton, 2011; a group of social scientists to collaboratively en- Castillo et al., 2011).This approach reflects tra- gage with a stream of tweets, with a goal of con- ditional research methods in the social sciences. structing a chain (or cascade) of automatic docu- Through manual annotation effort, researchers en- ment classification layers that isolate and analyse gage closely with the data in a manual but in- targeted conversions on Twitter. Section 4 dis- teractive fashion, and this effort enables them to cusses ways in which the design of our frame- uncover patterns in the data and make inferences work is intended to support the agile methodol- as to how SM was being used in the context of ogy mentioned above, with particular emphasis on the sociocultural phenomena under investigation. the value of DUALIST’s active learning approach, This research suffers form either being restricted and the crucial role of the collaborative gold stan- to fairly small datasets. dard and model building activities. Section 4.3 discusses additional data processing step that have • Approaches that employ tailored automatic been introduced to increase the frameworks use- data analysis, using a supervised machine-learning fulness, and section 5 introduces some projects to approach(Carvalho et al., 2011; Papacharissi and which the framework is being applied. de Fatima Oliveira, 2012; Meraz and Papacharissi, 2013; Hopkins and King, 2010). This research in- 2 Related Work fers properties of the SM data using statistics from Work that focuses on addressing sociological their bespoke machine learning analysis. Mannual questions with SMA broadly fall into one of three annotation effort is required to train the classifiers categories. and is typically applied in a batch process at the commencement of the investigation. • Approaches that employ automatic data analy- sis without tailoring the analysis to the specifics of Our work aims to expand this last category, im- the situation e.g. (Tumasjan et al., 2010; Tumas- proving the quality of research by capturing more jan et al., 2011; O’Connor et al., 2010; Gonzalez- of the insight-provoking engagement with the data Bailon et al., 2010; Sang and Bos, 2012; Bollen seen in more traditional research. et al., 2011). This body of research involves lit- 3 DUALIST tle or no manual inspection of the data. An an- alytical technique is selected a-priori, applied to Our approach is built around DUALIST (Settles, the SM stream, and the results from that analy- 2011; Settles and Zhu, 2012), an open-source sis are then aligned with a real-world phenomenon project designed to enable non-technical analysts in order to draw predictive or correlative conclu- to build machine-learning classifiers by annotat- sions about social media. A typical approach is ing documents with just a few minutes of effort.

37 In Section 4, we discuss various ways in which the introduction, our vision is that a team of social we have extended DUALIST, including function- scientists will be able to use this tool to collabora- ality allowing multiple annotators to work in par- tively work towards the construction of a cascade allel; incorporating functionality to create ‘gold- of automatic document classification layers that standard’ test sets and measure inter-annotator carve up an incoming Twitter data stream in order agreement; and supporting on-going performance to pick out one or more targeted ‘conversations’, evaluation against the gold standard during the and provide an analysis of what is being discussed process of building a classifier. DUALIST pro- in each of these ‘conversations’. In what follows, vides a graphical interface with which an annota- we refer to the social scientists as the researchers tor is able to build a Na¨ıve Bayes’ classifier given and the activity during which the researchers are a collection of unlabelled documents. During the working towards delivering a useful classifier cas- process of building a classifier, the annotator is cade as data engagement. presented with a selection of documents (in our 4.1 Facilitating data engagement case tweets) that he/she has an opportunity to la- bel (with one of the class labels), and, for each When embarking on the process of building one class, a selection of features (tokens) that the an- of the classifiers in the cascade, researchers bring notator has an opportunity to mark as being strong preconceptions as to the basis for the classifica- features for that class. tion. It is only when engaging with the data that Active learning is used to select both the docu- it becomes possible to develop an adequate clas- ments and the features being presented for annota- sification policy. For example, when looking for tion. Documents are selected on the basis of those tweets that express some attitude about a targeted that the current model is most uncertain about issue, one needs a policy as to how a tweet that (as measured by posterior class entropy), and fea- shares a link to an opinion piece on that topic tures are selected for a given class on the basis without any further comment should be classified. of those with highest information gain occurring There are a number of ways in which we support frequently with that class. After a batch of docu- the classification policy development process. ments and features have been annotated, a revised • One of the impacts of the active learning ap- model is built using both the labelled data and the proach adopted in DUALIST is that by presenting current model’s predictions for the remaining un- tweets that the current model is most unsure of, labelled data, through the use of the Expectation- DUALIST will very rapidly expose issues around Maximization algorithm. This new model is then how to make decisions on boundary cases. used as the basis for selecting the set of documents and features that will be presented to the annotator • We have extended DUALIST to allow multi- for the next iteration of the model building pro- ple researchers to build a classifier concurrently. cess. Full details can be found in Settles (2011). In addition to reducing the time it takes to build classifiers, this fosters a collaborative approach to The upshot of this is two-fold: not only can a classification policy development. reasonable model be rapidly created, but the re- searcher is exposed to an interesting non-uniform • We have added functionality that allows for the sample of the training data. Examples that are rel- collaborative construction of gold standard data atively easy for the model to classify, i.e. those sets. Not only does this provide feedback dur- with low entropy, are ranked lower in the list of ing the model building process as to when perfor- unlabelled data awaiting annotation. The effect of mance begins to plateau, but, as a gold standard this is that the training process facilitates a form of is being built, researchers are shown the current data exploration that exposes the user to the hard- inter-annotator agreement score, and are shown est border cases. examples of tweets where there is disagreement among annotators. This constitutes yet another 4 Extending DUALIST for Social Media way in which researchers are confronted with the Science Research most problematic examples.

This section describes ways in which we have ex- 4.2 Building classifier cascades tended DUALIST to provide an integrated data ex- Having considered issues that relate to the con- ploration tool for social scientists. As outlined in struction of an individual classifier, we end this

38 section by briefly considering issues relating to lowing researchers to review changes over time the classifier cascade. The Twitter API provides in any classifier’s input or output tweet flows basic boolean search functionality that is used to (classes). This extends the common approach of scrape the Twitter stream, producing the input to sentiment tracking over time to tracking over time the cascade. A typical strategy is to select query any attitudinal (or other) response whose essen- terms for the boolean search with a view to achiev- tial features can be captured by a classifier of this ing a reasonably high recall of relevant tweets2. kind. These class-volume-by-time-interval plots An effective choice of query terms that actually can provide insight into how and when the stream achieves this is one of the things that is not well changes in response to external events. understood in advance, but which we expect to emerge during the data engagement phase. Cap- Link analysis. It is becoming apparent that link turing an input stream that contains a sufficiently sharing (attaching a URL to a tweet, typically large proportion of interesting (relevant) tweets is pointing to a media story) is an important aspect of usually achieved at the expense of precision (the how information propagates through social media, proportion of tweets in the stream being scraped particularly on Twitter. For example, the mean- that are relevant). As a result, the first task that is ing of a tweet can sometimes only be discerned by typically undertaken during the data engagement inspecting the link to which it points. We are in- phase involves building a relevancy classifier, to troducing to the framework automatic expansion be deployed at the top of the classifier cascade, of shortened URLs and the ability to inspect link that is designed to filter out irrelevant tweets from URL contents, allowing researchers to interpret the stream of tweets being scraped. tweets more rapidly and accurately. A combina- When building the relevancy classifier, the re- tion of link analysis with time series analysis is searchers begin to see how well their preconcep- also providing researchers with insights into how tions match the reality of the data stream. It is only mainstream media stories propagate through soci- through the process of building this classifier that ety and shape opinion in the social media age. the researchers begin to get a feel for the compo- sition of the relevant data stream. This drives the Language use analysis. Once a classifier has researcher’s conception as to how best to divide been initially established, the framework analyses up the stream into useful sub-streams, and, as a the language employed in the input tweets using result, provides the first insights into an appropri- an information gain (IG) measure. High IG fea- ate cascade architecture. Our experience is that in tures are those that have occurrence distributions many cases, classifiers at upper levels of the cas- that closely align the document classification dis- cade are involved in decomposing data streams in tributions; essentially they are highly indicative of useful ways, and classifiers that are lower down the class. This information is proving useful to so- in the cascade are designed to measure some facet cial science researchers for three purposes. First, (e.g. sentiment polarity) of the material on some it helps identify the words and phrases people em- particular sub-stream. ploy to convey a particular attitude or opinion in the domain of interest. Second, it can provide in- 4.3 Tools for Data Analysis formation on how the language employed shifts As social scientists are starting to engage with over time, for example as new topics are intro- real-world data using this framework, it has duced or external events occur. Third, it can be emerged that certain patterns of downstream data used to select candidate keywords with which to analysis are of particular use. augment the stream’s boolean scraper query. In this last case, however, we need to augment the Time series analysis. For many social phenom- analysis; many high IG terms make poor scraper ena, the timing and sequence of social media mes- terms because they are poorly selective in the more sages are of critical importance, particularly for a general case (i.e. outside of the context of the ex- platform such as Twitter. Our framework supports isting query-selected sample). We take a sample tweet volume analysis across any time frame, al- using the candidate term alone with the search API and estimate the relevancy precision of the scraped 2In many cases it is very hard to estimate recall since there is no way to estimate accurately the volume of relevant tweets tweet sample by passing the tweets through the in the full Twitter stream. first-level relevancy classifier. The precision of the

39 new candidate term can be compared to the preci- repeated high recall, high precision searches of sion of existing terms and a decision made. large data sets for a specific entity. As an example, this approach is being employed in the EU attitu- 5 Applications and Extensions dinal survey study. The framework’s flexibility enables it to be applied Repeated complex search. In situations where to any task that can be broken down into a series of a fixed but complex search needs to be performed classification decisions, or indeed where this ap- repeatedly over a relatively long period of time, proach materially assists the social scientist in ad- then a supervised-learning classifier can be ex- dressing the issue at hand. In order to explore its pected both to produce the best results and to be application, our framework is being applied to a cost-effective in terms of the effort required to variety of tasks: train it. The authors have employed this approach Identifying patterns of usage. People use the in a commercial environment (Lyra et al., 2012), same language for different purposes; the frame- and the ability to train classifiers more quickly work is proving to be a valuable tool for eluci- with this framework reduces the cost still further dating these usage patterns and for isolating data and makes this a practical approach in a wider sets that illustrate these patterns. As an example, range of circumstances. the authors (in collaboration with a team of so- With regard to extension of the framework, we cial scientists) are studying the differing ways in have identified a number of avenues for expansion which people employ ethnically and racially sensi- and improvement that will significantly increase tive language in conversations on-line. The frame- its usefulness and applicability to real-world sce- work has helped to reveal and isolate a number of narios, and we have recently commenced an 18- distinct patterns of usage. month research programme to formalise and ex- tend the framework and its associated methodol- Tracking changes in opinion over time. Sen- ogy for use in social science research3. timent classifiers trained in one domain perform poorly when applied to another domain, even Conclusions and Future Work when the domains are apparently closely related We describe an agile analysis framework built (Pang and Lee, 2008). Traditionally, this has around the DUALIST tool designed to support ef- forced a choice between building bespoke clas- fective exploration of large twitter data sets by sifiers (at significant cost), or using generic sen- social scientists. The functionality of DUAL- timent classifiers (which sacrifice performance). IST has been extended to allow the scraping of The ability to rapidly construct sentiment classi- tweets through access to the Twitter API, collab- fiers that are specifically tuned to the precise do- orative construction of both gold standard data main can significantly increase classifier perfor- sets and Na¨ıve Bayes’ classifiers, an Information mance without imposing major additional costs. Gain-based method for automatic discovery of Moving beyond sentiment, with these bespoke new search terms, and support for the construction classifiers it is in principle possible to track over of classifier cascades. Further extensions currently time any form of opinion that is reflected in lan- under development include grouping tweets into guage. In a second study, the authors are (in col- threads conversations, and automatic clustering of laboration with a team of social scientists) build- relevant tweets in order to discover subtopics un- ing cascades of bespoke classifiers to investigate der discussion. shifts in citizens’ attitudes over time (as expressed in social media) to a range of political and social Acknowledgments issues arising across the European Union. We are grateful to our collaborators at the Cen- Entity disambiguation. References to individ- tre for the Analysis of social media, Jamie Bartlett uals are often ambiguous. In the general case, and Carl Miller for valuable contributions to this word sense disambiguation is most success- work. We thank the anonymous reviewers for their fully performed by supervised-learning classifiers helpful comments. This work was partially sup- (Marquez` et al., 2006), and the low cost of pro- ported by the Open Society Foundation. ducing classifiers using this framework makes this 3Towards a Social Media Science, funded by the UK approach practical for situations where we require ESRC National Centre for Research Methods.

40 References International Conference on Artificial Intelligence (AI-2012). [Bermingham and Smeaton2011] Adam Bermingham and Alan F Smeaton. 2011. On using Twitter to [Marchetti-Bowick and Chambers2012] Micol monitor political sentiment and predict election Marchetti-Bowick and Nathanael Chambers. results. In Proceedings of the Workshop on Senti- 2012. Learning for microblogs with distant su- ment Analysis where AI meets Psychology (SAAIP), pervision: political forecasting with Twitter. In IJCNLP 2011, pages 2–10. Proceedings of the 13th Conference of the European Chapter of the Association for Computational [Bollen et al.2011] Johan Bollen, Alberto Pepe, and Linguistics, pages 603–612. Huina Mao. 2011. Modeling public mood and emo- tion: Twitter sentiment and socio-economic phe- [Marquez` et al.2006] Llu´ıs Marquez,` Gerard Escudero, nomena. In Proceedings of the Fifth International David Mart´ınez, and German Rigau. 2006. Su- AAAI Conference on Weblogs and Social Media, pervised corpus-based methods for wsd. In Eneko pages 450–453. Agirre and Philip Edmonds, editors, Word Sense Disambiguation, volume 33 of Text, Speech and [Carvalho et al.2011] Paula Carvalho, Lu´ıs Sarmento, Language Technology, pages 167–216. Springer Jorge Teixeira, and Mario´ J. Silva. 2011. Liars and Netherlands. saviors in a sentiment annotated corpus of comments to political debates. In Proceedings of the 49th An- [Meraz and Papacharissi2013] Sharon Meraz and Zizi nual Meeting of the Association for Computational Papacharissi. 2013. Networked gatekeeping and Linguistics: Human Language Technologies: short networked framing on #egypt. The International papers - Volume 2, pages 564–568, Stroudsburg, PA, Journal of Press/Politics, 18(2):138–166. USA. [O’Connor et al.2010] Brendan O’Connor, Ramnath [Castillo et al.2011] Carlos Castillo, Marcelo Mendoza, Balasubramanyan, Bryan R Routledge, and Noah A and Barbara Poblete. 2011. Information credibility Smith. 2010. From tweets to polls: Linking text on Twitter. In Proceedings of the 20th International sentiment to public opinion time series. In Proceed- Conference on World wide web, pages 675–684. ings of the International AAAI Conference on We- blogs and Social Media, pages 122–129. [Diakopoulos and Shamma2010] Nicholas A Di- akopoulos and David A Shamma. 2010. Charac- [Pang and Lee2008] Bo Pang and Lillian Lee. 2008. terizing debate performance via aggregated Twitter Opinion mining and sentiment analysis. Founda- sentiment. In Proceedings of the 28th international tions and trends in Information Retrieval, 2(1-2):1– conference on Human factors in computing systems, 135. pages 1195–1198. [Papacharissi and de Fatima Oliveira2012] Zizi Pa- pacharissi and Maria de Fatima Oliveira. 2012. [Gayo-Avello2012] Daniel Gayo-Avello. 2012. I Affective news and networked publics: the rhythms wanted to predict elections with twitter and all i of news storytelling on #egypt. Journal of Commu- got was this lousy paper a balanced survey on elec- nication, 62(2):266–282. tion prediction using Twitter data. arXiv preprint arXiv:1204.6441. [Sang and Bos2012] Erik Tjong Kim Sang and Johan Bos. 2012. Predicting the 2011 dutch senate elec- [Gonzalez-Bailon et al.2010] Sandra Gonzalez-Bailon, tion results with Twitter. Proceedings of the Euro- Rafael E Banchs, and Andreas Kaltenbrunner. 2010. pean Chapter of the Association for Computational Emotional reactions and the pulse of public opin- Linguistics 2012, page 53. ion: Measuring the impact of political events on the sentiment of online discussions. arXiv preprint [Settles and Zhu2012] Burr Settles and Xiaojin Zhu. arXiv:1009.4019. 2012. Behavioral factors in interactive training of text classifiers. In Proceedings of the 2012 Confer- [Hopkins and King2010] Daniel J. Hopkins and Gary ence of the North American Chapter of the Associ- King. 2010. A method of automated nonparametric ation for Computational Linguistics: Human Lan- content analysis for social science. American Jour- guage Technologies, pages 563–567. nal of Political Science, 54(1):229–247. [Settles2011] Burr Settles. 2011. Closing the loop: [Jungherr et al.2012] Andreas Jungherr, Pascal Jurgens,¨ Fast, interactive semi-supervised annotation with and Harald Schoen. 2012. Why the Pirate Party queries on features and instances. In Proceedings won the German election of 2009 or the trouble of the Conference on Empirical Methods in Natural with predictions: A response to Tumasjan, Sprenger, Language Processing, pages 1467–1478. Sander, & Welpe. Social Science Computer Review, 30(2):229–234. [Tumasjan et al.2010] Andranik Tumasjan, Timm O Sprenger, Philipp G Sandner, and Isabell M Welpe. [Lyra et al.2012] Matti Lyra, Daoud Clarke, Hamish 2010. Predicting elections with Twitter: What 140 Morgan, Jeremy Reffin, and David Weir. 2012. characters reveal about political sentiment. In Pro- Challenges in applying machine learning to media ceedings of the fourth international AAAI confer- monitoring. In Proceedings of Thirty-second SGAI ence on weblogs and social media, pages 178–185.

41 [Tumasjan et al.2011] Andranik Tumasjan, Timm O Sprenger, Philipp G Sandner, and Isabell M Welpe. 2011. Election forecasts with Twitter how 140 char- acters reflect the political landscape. Social Science Computer Review, 29(4):402–418.

42 Morphological annotation of Old and Middle Hungarian corpora Attila Novak´ 1,2 Gyorgy¨ Orosz2 Nora´ Wenszky2 1Research Institute for Linguistics, Hungarian Academy of Sciences Benczur´ u. 33., Budapest, Hungary 2MTA-PPKE Natural Language Research Group Faculty of Information Technology, Pazm´ any´ Peter´ Catholic University Prater´ u. 50/a, Budapest, Hungary {novak.attila,oroszgy}@itk.ppke.hu, [email protected]

Abstract 2 Preprocessing The overwhelming majority of extant texts from In our paper, we present a computational the Old Hungarian period are codices, mainly con- morphology for Old and Middle Hungar- taining texts translated from Latin. The texts se- ian used in two research projects that aim lected for the Corpus of Informal Language Use, at creating morphologically annotated cor- however, are much closer to spoken language: pora of Old and Middle Hungarian. In ad- minutes taken at court trials, such as witch tri- dition, we present the web-based disam- als, and letters sent by noblemen and serfs. In biguation tool used in the semi-automatic the case of the latter corpus, metadata belonging to disambiguation of the annotations and the the texts are also of primary importance, as these structured corpus query tool that has a make the corpus fit for historical-sociolinguistic unique but very useful feature of making research. corrections to the annotation in the query results possible. 2.1 Digitization All the texts selected for our corpora were orig- 1 Introduction inally hand-written. However, the basis for the One of the aims of two parallel OTKA projects of digitized version was always a printed edition of the Research Institute for Linguistics of the Hun- the texts published earlier. The printed texts were garian Academy of Sciences1 is to create mor- scanned and converted to a character stream us- phologically analyzed and searchable corpora of ing OCR. This was not a trivial task, especially texts from the Old Hungarian and Middle Hungar- in the case of Old Hungarian texts, owing to the ian period. In the course of the projects, the Hu- extensive use of unusual characters and diacrit- mor Hungarian morphological analyzer (Novak,´ ics. In the lack of an orthographic norm, each 2003; Prosz´ eky´ and Novak,´ 2005) was extended text applied a different set of characters; moreover, to be capable of analyzing words containing mor- the printed publications used different fonts. Thus phological constructions, suffix allomorphs, suf- the only way to get acceptable results was to re- fix morphemes, paradigms or stems that were used train the OCR program2 for each text from scratch in Old and Middle Hungarian but no longer exist since the out-of-the-box Hungarian language and in present-day Hungarian. In the sections below, glyph models of the software did not fit any of our we describe how the morphological analyzer was texts. Subsequently, all the automatically recog- adapted to the task, the problems we encountered nized documents had to be manually checked and and how they were solved. We also present the corrected, but even so, this workflow proved to be automatic and the manual disambiguation system much faster than attempting to type in the texts. used for the morphosyntactic annotation of texts 2.2 Normalization and the corpus manager with the help of which the annotated corpora can be searched and main- The next step of preprocessing was normalization, tained. i.e. making the texts uniform regarding their or- thography and phonology. Normalization, which 1Hungarian historical generative syntax [OTKA NK78074], and Morphologically analysed corpus of Old and 2We used FineReader, which makes full customization of Middle Hungarian texts representative of informal language glyph models possible, including the total exclusion of out- use [OTKA 81189] of-the-box models.

43 Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 43–48, Sofia, Bulgaria, August 8 2013. c 2013 Association for Computational Linguistics was done manually, in our case meant modern- databases, the locus of word token occurrences ization to present-day orthography. Note that this only identified codex page, column and line num- also implies differences in tokenization into indi- ber, but there was no information concerning the vidual words between the original and the normal- order of words within a line. The databases also ized version. During this process, which also in- contain morphological analyses, but they were en- cluded segmentation of the texts into clauses, cer- coded in a hard-to-read numerical format, which tain phonological dialectal variations were neu- occasionally was incorrect and often incomplete. tralized. Furthermore, the categorization was in many re- Morphological variation, however, was left un- spects incompatible with our system. However, fi- touched: no extinct morphemes were replaced by nally we managed to re-create the original texts. their present day counterparts. We also retained First the order of words was manually restored extinct allomorphs unless the variation was purely and incomplete and erroneous analyses were fixed. phonological. In the case of potential irresolvable Missing lemmas were added to the lexicon of the ambiguity, the ambiguity was preserved as well, adapted computational morphology, and the nor- even if it was due to the vagueness of the orthog- malized version of the texts was generated using raphy of the era. the morphology as a word form generator. Finally, An example of this is the non-consistent mark- the normalized texts were reanalyzed to get analy- ing of vowel length. The definite and indefinite ses compatible with the annotation scheme applied 3rd person singular imperfect of the frequently to the other texts in the corpora. used word mond ‘say’ was monda´ ∼ monda re- spectively, but accents are often missing from the 3 The morphological analyzer texts. Furthermore, in many texts in the corpus, The digitized and normalized texts have been an- these two forms were used with a clearly differ- alyzed with an extended version of the Humor ent distribution from their present day counterparts analyzer for Hungarian. The lexicon of lemmas mondta ∼ mondott. Therefore, in many cases, nei- and the affix inventory of the program have been ther the orthography, nor the usage was consistent augmented with items that have disappeared from enough to decide unambiguously how a certain ap- the language but are present in the historical cor- pearance of monda should be annotated concern- pora. Just the affix inventory had to be supple- ing definiteness. mented with 50 new affixes (not counting their al- Another example of inherent ambiguity is a di- lomorphs). alectal variant of possessive marking, which is Certain affixes have not disappeared, but their very frequent in these corpora and often neutral- productivity has diminished compared to the Old izes singular and plural possessed forms. For ex- Hungarian era. Although words with these mor- ample, cselekedetinek could both mean ‘of his/her phemes are still present in the language, they are deed’ or ‘of his/her deeds’, which in many cases generally lexicalized items, often with a changed cannot be disambiguated based on the context meaning. An example of such a suffix is –At, even for human annotators. Such ambiguous cases which used to be a fully productive nomen actio- were annotated as inherently ambiguous regarding nis suffix. Today, this function belongs to the suf- number/definiteness etc. fix –As´ . The (now lexicalized) words, however, that end in –At mark the (tangible) result of an ac- 2.3 Jakab’s databases tion (i.e. nomen acti) in present-day standard Hun- Some of the Old Hungarian codices (Jokai´ (Jakab, garian, as in falazat ‘wall’ vs. falazas´ ‘building a 2002), Guary (Jakab and Kiss, 1994), Apor (Jakab wall’. and Kiss, 1997), and Festetics (Jakab and Kiss, One factor that made adaptation of the morpho- 2001)) were not digitized using the OCR tech- logical model difficult was that there are no reli- nique described above, as these were available in able accounts on the changes of paradigms. Data the form of historical linguistic databases, created concerning which affix allomorphs could be at- by Jakab Laszl´ o´ and his colleagues between 1978 tached to which stem allomorphs had to be ex- and 2002. However, the re-creation of the origi- tracted from the texts themselves. Certain mor- nal texts out of these lexical databases was a dif- phological constructions that had already disap- ficult task. The first problem was that, in the peared by the end of the Old Hungarian era were

44 rather rare (such as some participle forms) and of- titive,4 (iii) a lot of inherent ambiguities described ten some items in these rare subparadigms have al- above. ternative analyses. This made the formal descrip- The workflow for disambiguation of mor- tion of these paradigms rather difficult. phosyntactic annotation was a semi-automatic However, the most time consuming task was the process: an automatically pre-disambiguated ver- enlargement of the stem inventory. Beside the ad- sion of each text was checked and corrected manu- dition of a number of new lemmas, the entries of ally. For a very short time, we considered using the several items already listed in the lexicon of the Jakab databases as a training corpus, but recover- present-day analyzer had to be modified for our ing them required so much development and man- purposes. The causes were various: some roots ual labor and the analyses in them lacked so much now belong to another part of speech, or in some distinction we wanted to make that we opted for constructions they had to be analyzed differently creating the training data completely from scratch from their present analysis. instead. Furthermore, the number of pronouns was con- siderably higher in the examined period than 4.1 The manual disambiguation interface today. The description of their extensive and To support the process of manual checking and rather irregular paradigms was really challenging the initial manual disambiguation of the training as some forms were underrepresented in the cor- corpus a web-based interface was created using pora. JavaScript and Ajax where disambiguation and Some enhancements of the morphological an- normalization errors can be corrected very effec- alyzer made during the corpus annotation projects tively. The system presents the document to the were also applicable to the morphological descrip- user using an interlinear annotation format that is tion of standard modern Hungarian. One such easy and natural to read. An alternative analysis modification was a new annotation scheme ap- can be chosen from a pop-up menu containing a plied to time adverbials that are lexicalized suf- list of analyses applicable to the word that appears fixed (or unsuffixed) forms of nouns, like reggel when the mouse cursor is placed over the problem- ‘morning/in the morning’ or nappal ‘daytime/in atic word. Note that the list only contains gram- daytime’, quite a few of which can be modified by matically relevant tags and lemmas for the word adjectives when used adverbially, such as fenyes´ returned by the morphological analyzer. This is nappal ‘in broad daylight’. This latter fact sheds very important, since, due to the agglutinating na- light on a double nature of these words that could ture of Hungarian, there are thousands of possible be captured in an annotation of these forms as spe- tags (see Figure 1). cially suffixed forms of nouns instead of atomic adverbs, an analysis that is compatible with X-bar theory (Jackendoff, 1977).

4 Disambiguation With the exception of already analyzed sources (i.e. the ones recovered from the Jakab databases), Figure 1: The web-based disambiguation interface the morphological annotation had to be disam- biguated. The ambiguity rate of the output of The original and the normalized word forms as the extended morphological analyzer on historical well as the analyses can also be edited by clicking texts is higher than that for the standard Humor them, and an immediate reanalysis by the morpho- 3 analyzer for present-day corpora (2.21 vs. 1.92 logical analyzer running on the web server can be analyses/word with an identical (high) granularity initiated by double clicking the word. We use Ajax of analyses). This is due to several factors: (i) the technology to update only the part of the page be- historical analyzer is less strict, (ii) there are sev- longing to the given token, so the update is imme- eral formally identical members of the enlarged diate. Afterwards, a new analysis can be selected verbal paradigms including massively ambiguous from the updated pop-up menu. subparadigms like that of the passive and the fac- 4This ambiguity is absent from modern standard Hungar- 3measured on newswire text ian because the passive is not used any more.

45 As there is an inherent difference between the (fel)dolgoz ‘process’). Always selecting the –ik- original and normalized tokenization, and be- less variant is not a good bet in the case of many cause, even after thorough proofreading of the nor- frequent words in this ambiguity class. malized version, there may remain tokenization Recently, we replaced HunPos with another errors in the texts, it is important that tokens and HMM-based trigram tagger, PurePos (Orosz and clauses can also be split and joined using the dis- Novak,´ 2012), that has many nice extra features. It ambiguation interface. can process morphologically analyzed ambiguous The automatic annotation system was created in input and/or use an integrated analyzer constrain- a way that makes it possible that details of the ing possible analyses to those proposed by the an- annotation scheme be modified in the course of alyzer or read from the input. This boosts the pre- work. One such modification was e.g. the change cision of the tagger dramatically in the case of lan- to the annotation of time adverbs mentioned in guages like Hungarian and small training corpora. Section 3 above. The modified annotation can The fact that PurePos can be fed analyzed input be applied to texts analyzed and disambiguated makes it easy to combine with constraint-based prior to the modification relatively easily. This tools that can further improve the accuracy of the is achieved by the fact that, in the course of re- tagging by handling long distance agreement phe- analysis, the program chooses the analysis most nomena not covered by the trigram model or sim- similar to the previously selected analysis (based ply removing impossible tag sequences from the on a letter trigram similarity measure). Neverthe- search space of the tool. less, the system highlights all tokens the reanaly- PurePos can perform lemmatization, even for sis of which resulted in a change of annotation, so words unknown to the morphological analyzer that these spots can be easily checked manually. (and not annotated on the input) learning a suffix- For changes in the annotation scheme where the based lemmatization model from the training cor- simple similarity-based heuristic could not be ex- pus along with a similar suffix-based tag guessing pected to yield an appropriate result (e.g. when model, thus it assigns a full morphological anal- we decided to use a more detailed analysis of de- ysis to each token. It is also capable of generat- rived verb forms as before), a more sophisticated ing an n-best list of annotations for the input sen- method was devised to update the annotations: old tence when using beam search instead of the de- analyses were replaced using automatically gener- fault Viterbi decoding algorithm. ated regular expressions. 4.3 Disambiguation performance 4.2 Automatic disambiguation We performed an evaluation of the accuracy of While the first few documents were disambiguated PurePos on an 84000-word manually checked completely manually using the web-based tool, part of the historical corpus using five-fold cross- we soon started to train and use a tagger for pre- validation with a training corpus of about 67000 disambiguation applying the tagger incrementally, words and a test corpus of about 17000 words in trained on an increasing number of disambiguated each round. The ratio of words unknown to the and checked text. First the HMM-based trigram MA in this corpus is rather low: 0.32%. tagger HunPos (Halacsy´ et al., 2007) was used. The average accuracy of tagging, lemmatiza- HunPos is not capable of lemmatization, but we tion and full annotation for different versions of used a straightforward method to get a full anal- the tagger are shown in Table 1. In addition to ysis: we applied reanalysis to the text annotated token accuracy, we also present sentence accu- only by the tags assigned by HunPos using the racy values in the table. Note that, in contrast to automatic similarity-based ranking of the analy- the usual way of evaluating taggers, these values ses. This approach yielded quite good results, but were calculated excluding the always unambigu- one problem with it was that the similarity-based ous punctuation tokens from the evaluation. The ranking always prefers shorter lemmas, which was baseline tagger uses no morphological information not appropriate for handling the case of a fre- at all. Its current lemmatization implementation quent lemma ambiguity for verbs with one of the uses suffix guessing in all cases (even for words lemma candidates ending in an –ik suffix and the seen in the training corpus) and selects the most other lacking a suffix (such as dolgozik ‘work’ vs. frequent lemma, which is obviously not an ideal

46 solution. 2004). In addition to queries formulated using The disambiguator using morphology performs MQL, the query language of Emdros, either typed significantly better. Its clause-level accuracy is in at the query box or assembled using controls 81.50%, which means that only every fifth clause of the query interface, advanced users can use contains a tagging error. The tag set we used in a custom-made corpus-specific query language the corpus differentiates constructions which are (MEQL), which makes a much more compact for- not generally differentiated at the tag level in Hun- mulation of queries possible than MQL. It is e.g. garian corpora, e.g. deictic pronouns (ebben ‘in extremely simple to locate a specific locus in the this’) vs. deictic pre-determiners (ebben a hazban´ corpus: one simply needs to type in the sequence ‘in this house’). Many of these can only be dis- of words one is looking for. Queries formulated ambiguated using long-distance dependencies, i.e. in MEQL are automatically converted to MQL information often not available to the trigram tag- queries by the query processor. ger. Combination of the tagger with a constraint- The search engine makes it possible to search based tool (see e.g. Hulden and Francom (2012)) inside sentences, clauses, or texts containing would presumably improve accuracy significantly. grammatical constructions and/or tagged with In the rightmost column, we listed a theoreti- metadata matching the criteria specified in the cal upper limit of the performance of the current query. Units longer than a sentence can also be trigram tagger implementation using 5-best output searched for. The context displayed by default for and an ideal oracle that can select the best annota- each hit is the enclosing sentence with focus words tion. highlighted. Clauses may be non-continuous: this is often the case for embedded subordinate baseline morph 5-best+o clauses, but the corpus also contains many injected token Tag 90.17% 96.44% 98.97% parenthetical coordinate clauses and many exam- Lem. 91.52% 98.19% 99.11% ples where the topic of a subordinate clause pre- Full 87.29% 95.90% 98.53% cedes its main clause with the net effect of the clause Tag 62.48% 83.81% 93.99% subordinate clause being interrupted by the main Full 54.68% 81.50% 91.47% clause. The query example in Figure 2 shows a Table 1: Disambiguation performance of the tag- sentence containing several clauses with gaps: the ger clauses enclosed in angle brackets are wedged be- tween the topic and comment part of the clauses which they interrupt. Emdros is capable of repre- 5 Searching the corpus senting these interrupted clauses as single linguis- tic objects with the interrupting clause not being The web-based tool we created as a corpus query considered part of the interrupted one. interface does not only make it possible to search for different grammatical constructions in the 6 Conclusion texts, but it is also an effective correction tool. Er- rors discovered in the annotation or the text ap- In our paper, we described the most important pearing in the “results” box can immediately be steps of the creation of a morphological annota- corrected and the corrected text and annotation tion framework for the analysis of Old and Mid- is recorded in the database. Naturally, this latter dle Hungarian extant texts consisting of a mor- functionality of the corpus manager is only avail- phological analyzer, an automatic disambiguation able to expert users having the necessary privi- tool and an intuitive web-based manual disam- leges. biguation tool. Certain problems arising during A fast and effective way of correcting errors in this process were discussed together with their so- the annotation is to search for presumably incor- lution. We also presented our corpus manager, rect structures and to correct the truly problematic which serves both as a structured corpus query tool ones at once. The corrected corpus can be ex- and as a correction tool. ported after this procedure and the tagger can be The morphological analyzer is used for the an- retrained on it. notation of the constantly growing Old and Mid- The database used for the corpus manager is dle Hungarian corpora. Part of these corpora are based on the Emdros corpus manager (Petersen, already searchable by the public. The Old Hun-

47 Ray Jackendoff. 1977. X-bar-Syntax: A Study of Phrase Structure. Linguistic Inquiry Monograph 2. MIT Press, Cambridge, MA. Laszl´ o´ Jakab and Antal Kiss. 1994. A Guary-k¨ odex´ ab´ ec´ erendes´ adattara´ . Szam´ ´ıtog´ epes´ nyelvtort¨ eneti´ adattar.´ Debreceni Egyetem, Debrecen. Laszl´ o´ Jakab and Antal Kiss. 1997. Az Apor-kodex´ ab´ ec´ erendes´ adattara´ . Szam´ ´ıtog´ epes´ nyelvtort¨ eneti´ adattar.´ Debreceni Egyetem, Debrecen. Laszl´ o´ Jakab and Antal Kiss. 2001. A Festetics-kodex´ ab´ ec´ erendes´ adattara´ . Szam´ ´ıtog´ epes´ nyelvtort¨ eneti´ adattar.´ Debreceni Egyetem, Debrecen. Laszl´ o´ Jakab. 2002. AJokai-k´ odex´ mint nyelvi emlek:´ szot´ arszer´ u¨ feldolgozasban´ . Szam´ ´ıtog´ epes´ Nyelvtort¨ eneti´ Adattar.´ Debreceni Egyetem, Debre- cen. Attila Novak.´ 2003. Milyen a jo´ Humor? [What is good Humor like?]. In I. Magyar Szam´ ´ıtog´ epes´ Figure 2: The query interface Nyelveszeti´ Konferencia, pages 138–144, Szeged. SZTE. garian Corpus is available at http://rmk.nytud.hu, Gyorgy¨ Orosz and Attila Novak.´ 2012. PurePos – an while the analyzed part of the Historical Corpus open source morphological disambiguator. In Pro- ceedings of the 9th International Workshop on Nat- of Informal Language Use can be searched at ural Language Processing and Cognitive Science., http://tmk.nytud.hu. Wroclaw, Poland.

Acknowledgments Ulrik Petersen. 2004. Emdros — a text database en- gine for analyzed or annotated text. In In: Proceed- Research reported in this paper was supported by ings of COLING 2004. (2004) 1190–1193. the research project grants OTKA NK78074 and Gabor´ Prosz´ eky´ and Balazs´ Kis. 1999. A unification- OTKA 81189. In addition, we gratefully acknowl- based approach to morpho-syntactic parsing of ag- edge support by the grants TAMOP-4.2.1./B-´ glutinative and other (highly) inflectional languages. ´ In Proceedings of the 37th annual meeting of the 11/2/KMR-2011-002 and TAMOP-4.2.2./B-10/1- Association for Computational Linguistics on Com- 2010-0014. We would also like to thank anony- putational Linguistics, ACL ’99, pages 261–268, mous reviewers of the paper for their helpful com- Stroudsburg, PA, USA. Association for Computa- ments and suggestions. tional Linguistics. Gabor´ Prosz´ eky´ and Attila Novak.´ 2005. Compu- tational Morphologies for Small Uralic Languages. References In Inquiries into Words, Constraints and Contexts., pages 150–157, Stanford, California. Peter´ Halacsy,´ Andras´ Kornai, and Csaba Oravecz. 2007. HunPos: an open source trigram tagger. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, ACL ’07, pages 209–212, Stroudsburg, PA, USA. Association for Computational Linguistics.

Mans Hulden and Jerid Francom. 2012. Boost- ing statistical tagger accuracy with simple rule- based grammars. In Nicoletta Calzolari (Confer- ence Chair), Khalid Choukri, Thierry Declerck, Mehmet Ugur˘ Dogan,˘ Bente Maegaard, Joseph Mar- iani, Jan Odijk, and Stelios Piperidis, editors, Pro- ceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), Is- tanbul, Turkey. European Language Resources As- sociation (ELRA).

48 Argument extraction for supporting public policy formulation Eirini Florou Stasinos Konstantopoulos Dept of Linguistics, Faculty of Philosophy Institute of Informatics and University of Athens, Greece Telecommunications, NCSR ‘Demokritos’ [email protected] [email protected]

Antonis Kukurikos Pythagoras Karampiperis Institute of Informatics and Telecommunications, NCSR ‘Demokritos’, Athens, Greece {kukurik,pythk}@iit.demokritos.gr

Abstract pressed in support or opposition of opinions with In this paper we describe an application little semantic similarity to our topic of interest. of language technology to policy formula- As a rough example, consider how drafting envi- tion, where it can support policy makers ronmental policy can benefit from access to statis- assess the acceptance of a yet-unpublished tics about how people felt about industrial growth policy before the policy enters public con- at the expense of environmental concerns when sultation. One of the key concepts is that other policy in completely different domains was instead of relying on thematic similarity, on public consultation: many of the arguments we extract arguments expressed in support about the relative merits of industrial growth and or opposition of positions that are general environmental concerns can retain their structure statements that are, themselves, consistent and be thematically transferred to the new domain, with the policy or not. The focus of this helping draft a policy that best addresses people’s paper in this overall pipeline, is identify- concerns. ing arguments in text: we present and em- Of paramount importance for implementing pirically evaluate the hypothesis that ver- such an approach is the linguistic tools for identi- bal tense and mood are good indicators of fying arguments. In this paper, we first motivate arguments that have not been explored in the inclusion of argument extraction inside the the relevant literature. larger policy formulation and validation cycle and present the position of an argument extraction tool 1 Introduction inside a computational system for supporting this The large-scale acquisition, thematic classifica- cycle (Section2). We then proceed to a present the tion, and sentiment analysis of Web content has argument extraction literature (Section3) and our been extensively applied to brand monitoring, dig- hypothesis that verbal morpho-syntactic features ital reputation management, product development, are good discriminators of arguments (Section4). and a variety of similar applications. More re- We close the paper by presenting and discussing cently, it has also seen application in public policy empirical results (Section5) and concluding (Sec- validation, where the ‘brand’ to be monitored is tion6). a publicized and widely commented government 2 Policy formulation and validation policy. All these methods typically rely on the seman- Our work is carried out in the context of a project tic similarity between a given text or set of terms that develops computational tools for the early and Web content; often using domain-specific on- phases of policy making, before policy drafts have tological and terminological resources in order to been made available for public consultation.1 measure this similarity. This approach, however At that stage, the policy’s impact on public requires that all parties involved discourse on the opinion cannot be estimated by similarity-based same topic; that is to say, that we are seeking the searching for relevant Web content, since the pol- collective opinion of the Web on a topic that has icy text has not been announced yet — or even been publicized enough to attract the attention of fully authored for that matter. One of the core the Web. 1Full details about the project have been suppressed to In this paper we present a slightly different ap- preserve anonymity, but will be included in the camera-ready. proach, where we are looking for arguments ex-

49 Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 49–54, Sofia, Bulgaria, August 8 2013. c 2013 Association for Computational Linguistics ideas of the project is that in order to assist the the expense of precision. As a second step, clean policy formulation process, a tool needs to esti- text is extracted from the raw Web content using mate the acceptance of a yet unpublished docu- the Boilerpipe library (Kohlschutter¨ et al., 2010) ment based on Web content that is not themati- in order to remove HTML tags, active compo- cally similar, but is rather supporting or opposing a nents (e.g. JavaScript snippets), and content that more general position or maxim that also supports is irrelevant to the main content (menus, ad sec- or opposes the policy under formulation. tions, links to other web pages), and also to replace To make this more concrete, consider a new pol- HTML entities with their textual equivalent, e.g., icy for increasing the penetration of wind power replacing ‘&’ with the character ‘&’. production, setting specific conditions and prior- The resulting text is tokenized and sentence- ities. The project is developing an authoring en- splitted and each sentence classified as relevant or vironment where specific policy statements are not using standard information retrieval methods linked to more general statements, such as: to assess the semantic similarity of each sentence to the general policy statements. This is based on (1) Greenhouse gas emissions should not be a both general-purpose resources2 and the domain concern at all. ontology for the particular policy. Consecutive (2) It is desired to reduce greenhouse gas sentences that are classified as positive are joined emissions, but this should be balanced into a segment. against other concerns. The main objective of the work described here is the classification of these segments as being (3) It is desired to reduce greenhouse gas representative of a stance that would also support emissions at all costs. or oppose the policy being formulated, given the premise of the general statements (1)–(3). Our ap- We have, thus, created a formulation of ‘relevant proach is to apply the following criteria: content’ that includes Examples 1 and 2 below. These are taken from different domains, are com- • That they are semantically similar to the gen- menting policies and laws that are already formu- eral statements associated with the policy. lated and made public, and can be used to infer • the level of support for the new wind power policy That they are arguments, rather than state- although no textual similarity exists. ments of fact or other types of prose. • That their polarity towards the general state- (4) In case hard packaging is made compulsory ments is expressed. by law, producers will be forced to consume more energy, leading to more greenhouse In order to be able to assess segments, we thus gas emissions. need a linguistic pipeline that can calculate seman- tic similarity, identify arguments, and extract their (5) Tidal power production does not emit structure (premises/consequences) and polarity (in greenhouse gases, but other environmental support or opposition). problems are associated with its widespread The focus of the work described here is iden- deployment. tifying arguments, although we also outline how Leaving aside the ontological conceptualization the features we are proposing can also be used in that achieves this matching, which is reported else- order to classify chunks of text as premises or con- where, we will now discuss the language process- sequences. ing pipeline that retrieves and classifies relevant 3 Related Work Web content. Content is acquired via focused crawling, us- The first approaches of argument extraction were ing search engine APIs to retrieve public Web concentrated on building wide-coverage argument pages and social network APIs to retrieve con- 2WordNets are publicly available for both English and tent from social content sharing platforms. Con- Greek, that is the language of the experiments reported here. tent is searched and filtered (in case of feed-like Simpler semantic taxonomies can also be used; the accuracy of the semantic similarity measured here does not have a ma- APIs) based on fairly permissive semantic similar- jor bearing on the argument extraction experiments that are ity measures, emphasising a high retrieval rate at the main contribution of this paper.

50 structure lexicons, originally manually Fitzpatrick 4 Approach and Experimental Setup and Sager(1980, 1981) and later from elec- As seen above, shallow techniques are typically tronic versions of conventional dictionaries, since based on connectives and other discourse mark- such dictionaries contain morpho-syntactic fea- ers in order to define shallow argument patterns. tures Briscoe et al.(1987). More recently, the fo- What has not been investigated is whether shallow cus shifted to automatically extracting these lex- morpho-syntactic features, such as the tense and ical resources from corpora Brent(1993) and to mood of the verbal constructs in a passage, can hybrid approaches using dictionaries and corpora. also indicate argumentative discourse. Works using syntactic features to extract top- Our hypothesis is that future and conditional ics and holders of opinions are numerous (Bethard tenses and moods often indicate conjectures and et al., 2005). Semantic role analysis has also hypotheses which are commonly used in argumen- proven useful: Kim and Hovy(2006) used a tation techniques such as illustration, justification, FrameNet-based semantic role labeler to deter- rebuttal where the effects of a position counter to mine holder and topic of opinions. Similarly, Choi the speaker’s argument are analysed. Naturally, and Cardie(2006) successfully used a PropBank- such features cannot be the sole basis of argument based semantic role labeler for opinion holder ex- identification, so we need to experiment regarding traction. their interaction with discourse markers. Somasundaran et al. (2008; 2010) argued that To make this more concrete, consider the exam- semantic role techniques are useful but not com- ples in Section2: although both are perfectly valid pletely sufficient for holder and topic identifica- arguments that can help us infer the acceptance or tion, and that other linguistic phenomena must be rejection of a policy, in the first one future tense studied as well. In particular, they studied dis- is used to speculate about the effects of a policy; course structure and found specific cue phrases in the second example there is no explicit marker that are strong features for use in argument extrac- that the effects of large-scale tidal power produc- tion. Discourse markers that are strongly associ- tion are also a conjecture. ated with pragmatic functions can be used to pre- Another difficulty is that conditional and fu- dict the class of content, therefore useful features ture verbal groups are constructed using auxil- include the presence of a known marker such as iary verbs and (in some languages) other auxil- ‘actually’, ‘because’, ‘but’. iary pointers. Consider, for example, the follow- Tseronis(2011) describes three main ap- ing PoS-tagged and chunked Greek translation of proaches to describing argument markers: Geneva Example4: School, Argument within Language Theory and the Pragma-dialectical Approach. According the (6) [oi paragogoi]np Geneva School, there are three main types of [the-NomPl producers-NounNomPl] markers/connective, organisation markers, illocu- [tha ipochreothoun]vp tionary function markers (the relations between [Pointer-Fut force-Perf-3PP] acts) and interactive function markers. Argument [na katanalosoun]vp within Language Theory is a study of individual [Pointer-Subj consume-Inf] words and phrases. The words identified are argu- ‘producers will be forced to consume’ ment connectors: these describe an argumentative function of a text span and change the potential In order to be able to correctly assign simple fu- of it either realising or de-realising the span. The ture, information from the future pointer ‘tha’ Pragma-dialectical Approach looks at the context needs to be combined with the perfective feature beyond words and expressions that directly refer to of finite verb form. Conditionals, future perfect, the argument. It attempts to identify words and ex- past perfect, and similar tenses or moods like sub- pressions that refer to any moves in the argument juncive also involve the tense of the auxiliary verb, process. Similarly to Marcu and Echihabi(2002), besides the future pointer and the main verb. the approach is to create a model of an ideal argu- We have carried out our experiments in Greek ment and annotate relevant units. language texts, for which we have developed a JAPE grammar3 that extract the tense and mood of

3JAPE is finite state transducer over GATE annota-

51 Table 1: Categories of morpho-syntactic features extracted from text segments. Label Description Features DM Absolute number of occurrences of discourse markers 5 numerical features from a given category Rel Relative frequency of each of the 6 tenses and each of the 6 moods 12 numerical features RCm Relative frequency of each tense/mood 9 numerical features combination (only for those that actually appear). Bin Appearance of each of the 6 tenses and each of the 6 moods 12 binary features Dom Most frequent tense, mood, and tense/mood combination 3 string features TOTAL 41 features each verb chunk. The grammar uses patterns that • explanation, matching patterns such as ‘in combine the features of pointers and auxiliary and other words’, ‘for instance’, quotesfor this main verbs, without enforcing any restrictions on reason(s), etc. what words (e.g., adverbs) might be interjected in the chunk. That is to say, the chunker is responsi- • deduction, ‘as a consequence’, ‘in accor- ble for identifying verb groups and our grammar is dance with the above’, ‘proving that’, etc. restricted to propagating and combining the right • rebuttal, ‘despite’, ‘however’, etc. features from each of the chunk’s constituents to the chunk’s own feature structure. • conditionals, ‘supposing that’, ‘in case that’, PoS-tagging and chunking annotations have etc. been previously assigned by the ILSP suite of Greek NLP tools (Papageorgiou et al., 2000; All features extracted by this process are given Prokopidis et al., 2011), as provided by the rele- on Table1. 4 vant ILSP Web Services to get PoS tagged and 5 Results and Discussion chunked texts in the GATE XML format. At a second layer of processing, we create one We have used the method described in Section2 data instance for each segment (as defined in Sec- in order to obtain 677 text segments, with a size tion2 above) and for each such segment we ex- ranging between 10 and 100 words, with an av- tract features relating to verbal tense/mood and to erage of 60 words. Of these, 332 were manually the appearance of discourse markers. The former annotated to not be arguments; the remaining 345 are different ways to aggregate the various tenses positive examples were obtained by oversampling and moods found in the whole segment, by mea- the 69 segments in our corpus that we have manu- suring relative frequencies, recording the appear- ally annotated to be arguments.5 ance of a tense or mood even once, and naming We have then applied the feature extraction de- the predominant (most frequent) tense and mood; scribed in Section4 in order to set up a classifi- tense and mood are seen both individually and as cation task for J48, the Weka6 implementation of tense/mood combinations. the C4.5 decision tree learning algorithm (Quin- Furthermore, we have defined five absolute fre- lan, 1992). We have applied a moderate confi- quency features which record the matching against dence factor of 0.25, noting that experimenting the several patterns and keywords provided for the with the confidence factor did not yield any sub- following five categories of arguments: stantially different results. In order to better understand the feature space, • justification, matching patterns such as ‘be- we have run a series of experiments, with quan- cause’, ‘the reason being’, ‘due to’, etc. titative results summarized in Table2. The first tions. Please see http://gate.ac.uk/sale/tao/ 5The data and relevant scripts for carrying out these splitch8.html The JAPE grammar we have developed experiments are available at http://users.iit. will be made available on-line; location cannot be yet dis- demokritos.gr/˜konstant/dload/arguments. closed in order to preserve anonymity. tgz 4 Currently at http://ilp.ilsp.gr 6Please see http://www.cs.waikato.ac.nz/ ml/weka

52 Table 2: Precision and recall for retrieving arguments using different feature mixtures. Please cf. Table1 for an explanation of the feature labels. The results shown are the 10-fold cross-validation mean. Morpho-syntactic With Discourse Markers Without Discourse Markers features used Prec. Rec. Fβ=1 Prec. Rec. Fβ=1 All 75.8% 71.9% 73.8% 75.5% 70.4% 72.9% no Dom 79.8% 73.3% 76.4% 74.0% 71.9% 72.9% no Rel 74.5% 72.8% 73.8% 73.1% 69.3% 71.1% no RCm 76.3% 71.0% 73.6% 76.8% 70.1% 73.3% no Bin 70.0% 70.4% 70.2% 66.7% 69.6% 68.1% Rel 73.4% 75.9% 74.6% 70.3% 72.2% 71.2% Dom 57.1% 98.8% 72.4% 54.9% 94.2% 69.4% RCm 69.3% 66.7% 67.9% 71.9% 62.9% 67.1% Bin 71.7% 49.9% 58.8% 70.1% 44.9% 54.8% None 67.9% 20.9% 31.9% — observation is that both morpho-syntactic features each verb tense/mood and the binary appearance and discourse markers are needed, because if ei- of each verb tense/mood inside a text segment are ther category is omitted results deteriorate. How- as discriminative of argumentative text as the (typ- ever, not all morpho-syntactic features are needed: ically used) discourse markers; and that classifica- note how omitting the Dom, Rel, or RCm cate- tion is improved by combining discourse marker gories yields identical or improved results. On the features with our verbal tense/mood features. For other hand, the binary presence feature category doing this, we developed a regular grammar that Bin is significant (cf. 5th row). We cannot, how- combines the PoS tags of the members of a verb ever, claim that only the Bin category is sufficient, chunk in order to assign tense and mood to the and, in fact, if one category has to be chosen that chunk. In this manner, our approach depends on would have to be that of relative frequency fea- PoS taggin and chunking only. tures (cf. rows 6-9). In subsequent steps of our investigation, we are planning to refine our approach to extracting ar- 6 Conclusion gument structure: it would be interesting to test We describe here an application of language tech- if argument premises tend to correlate with cer- nology to policy formulation, and, in particular, to tain tenses or moods, distinguishing them from using Web content to assess the acceptance of a from conclusions. Further experiments can also yet-unpublished policy before public consultation. examine if the simultaneous appearance of con- The core of the idea is that classifying Web con- crete tenses at the same sentence is an indicator tent as similar to the policy or not does apply, be- of an argument. Finally, we plan to examine the cause the policy document has not been made pub- predicates of an argument, and especially if the lic yet; but that we should rather extract arguments head word of each sentence (be it verb or deverbal from Web content and assess whether these argue noun) and its seat at the boundaries of the sentence in favour or against general concepts that are (or may contribute to extract an argument or not, espe- are not) consistent with the policy being formu- cially for impersonal, modal, and auxiliary verbs. lated. Acknowledgements As a first step to this end, our paper focuses on the identification of arguments in Greek lan- The research leading to these results has re- guage content using shallow features. Based on ceived funding from the European Union’s Sev- our observation that verb tense appears to be a sig- enth Framework Programme (FP7/2007-2013) un- nificant feature that is not exploited by the rele- der grant agreement no 288513. For more de- vant literature, we have carried out an empirical tails, please see the NOMAD project’s website, evaluation of this hypothesis. We have, in partic- http://www.nomad-project.eu ular, demonstrated that the relative frequency of

53 References Prokopis Prokopidis, Byron Georgantopoulos, and Haris Papageorgiou. 2011. A suite of NLP tools Steven Bethard, Hong Yu, Ashley Thornton, for Greek. In Proceedings of the 10th Interna- Vasileios Hatzivassiloglou, and Dan Jurafsky. tional Conference of Greek Linguistics (ICGL 2005. Extracting opinion propositions and opin- 2011), Komotini, Greece. ion holders using syntactic and lexical cues. In James G. Shanahan, Yan Qu, and Janyce Wiebe, J. Ross Quinlan. 1992. C4.5: Programs for Ma- editors, Computing Attitude and Affect in Text: chine Learning. Morgan Kaufmann, San Mateo, Theory and Applications. Springer. CA, USA. Michael Brent. 1993. From grammar to lexi- Swapna Somasundaran and Janyce Wiebe. 2010. con: unsupervised learning of lexical syntax. Recognizing stances in ideological on-line de- In Computational Linguistics - Special issue on bates. In Proc. NAACL HLT Workshop on Com- using large corpora: II, Volume 19 Issue 2, putational Approaches to Analysis and Genera- pages 243–262. tion of Emotion in Text (CAAGET 2010) Ted Briscoe, Claire Grover, Bran Boguraev, and Swapna Somasundaran, Janyce Wiebe, and Joseph John. Carroll. 1987. The derivation of a Ruppenhofer. 2008. Discourse level opinion in- grammatically-indexed lexicon from the long- terpretation. In Proceedings of the 22nd Inter- man dictionary of contemporary english. In national Conference on Computational Linguis- Proceeding of ACL ’87. tics - Volume 1, COLING ’08, Stroudsburg, PA, USA. Association for Computational Linguis- Yejin Choi and Claire Cardie. 2006. Joint extrac- tics. tion of entities and relations for opinion recog- nition. In Proceedings of EMNLP 2006. Assimakis Tseronis. 2011. From connectives to argumentative markers: A quest for markers Eileen Fitzpatrick and Naomi Sager. 1980. The of argumentative moves and of related aspects Lexical subclasses of the LSP English Gram- of argumentative discourse. Argumentation, mar. Linguistic String Project, New York Uni- 25(4):427–444. versity. Eileen Fitzpatrick and Naomi Sager. 1981. The lexical subclasses of the lsp english grammar. In N. Sager (ed), Natural Language Information Processing, Addison- Wesley, Reading, Ma. Soo-Min Kim and Eduard Hovy. 2006. Extract- ing opinions, opinion holders, and topics ex- pressedin online news media text. In Proceed- ings of ACL/COLING Workshop on Sentiment and Subjectivity in Text. Christian Kohlschutter,¨ Peter Fankhauser, and Wolfgang Nejdl. 2010. Boilerplate detection us- ing shallow text features. In Proc. 3rd ACM International Conference on Web Search and Data Mining (WSDM 2010) New York, USA. Daniel Marcu and Abdessamad Echihabi. 2002. An unsupervised approach to recognizing dis- course relations. In Proc. ACL ’02. Haris Papageorgiou, Prokopis Prokopidis, Voula Giouli, and Stelios Piperidis. 2000. A unified POS tagging architecture and its application to Greek. In Proceedings of the 2nd Language Resources and Evaluation Conference (LREC 2000), Athens, Greece, pages 1455–1462.

54 Towards a Tool for Interactive Concept Building for Large Scale Analysis in the Humanities Andre Blessing1 Jonathan Sonntag2 Fritz Kliche3 Ulrich Heid3 Jonas Kuhn1 Manfred Stede2 1Institute for Natural Language Processing Universitaet Stuttgart, Germany 2Institute for Applied Computational Linguistics University of Potsdam, Germany 3Institute for Information Science and Natural Language Processing University of Hildesheim, Germany

Abstract precise results due to a rather unspecific search as well as semantically invalid or ambigious search We develop a pipeline consisting of var- words. On the other hand, large amounts of e.g. ious text processing tools which is de- news texts are available, also over longer periods signed to assist political scientists in find- of time, such that e.g. tendencies over time can ing specific, complex concepts within be derived. The corpora we are currently working large amounts of text. Our main focus is on contain ca. 700,000 articles from British, Irish, the interaction between the political scien- German and Austrian newspapers, as well as (yet tists and the natural language processing unexplored) material in French. groups to ensure a beneficial assistance for Figure 1 depicts a simple example of a quantita- the political scientists and new application tive analysis.1 The example shows how often two challenges for NLP. It is of particular im- terms, Friedensmission(‘peace operation’), and portance to find a “common language” be- Auslandseinsatz(‘foreign intervention’) are used tween the different disciplines. Therefore, in the last two decades in newspaper texts about we use an interactive web-interface which interventions and wars. The long-term goal of the is easily usable by non-experts. It inter- project is to provide similar analysis for complex faces an active learning algorithm which concepts. An example of a complex concept is is complemented by the NLP pipeline to the evocation of collective identities in political provide a rich feature selection. Political contexts, as indirect in the news. Examples for scientists are thus enabled to use their own such collective identities are: the Europeans, the intuitions to find custom concepts. French, the Catholics. 1 Introduction The objective of the work we are going to dis- cuss in this paper is to provide NLP methods and In this paper, we give examples of how NLP meth- tools for assisting political scientists in the ex- ods and tools can be used to provide support for ploration of large data sets, with a view to both, complex tasks in political sciences. Many con- a detailed qualitative analysis of text instances, cepts of political science are complex and faceted; and a quantitative overview of trends over time, they tend to come in different linguistic realiza- at the level of corpora. The examples discussed tions, often in complex ones; many concepts are here have to do with (possibly multiple) collective not directly identifiable by means of (a small set identities. Typical context of such identities tend of) individual lexical items, but require some in- to report communication, as direct or as indirect terpretation. speech. Examples of such contexts are given in 1. Many researchers in political sciences either work qualitatively on small amounts of data which they interpret instance-wise, or, if they are in- (1) Die Europaer¨ wurden¨ die Lucke¨ fullen,¨ terested in quantitative trends, they use compara- The Europeans would the gap fill, tively simple tools, such as keyword-based search in corpora or text classification on the basis of 1The figure shows a screenshot of our web-based terms only; this latter approach may lead to im- prototype.

55 Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 55–64, Sofia, Bulgaria, August 8 2013. c 2013 Association for Computational Linguistics Figure 1: The screenshot of our web-based system shows a simple quantitative analysis of the frequency of two terms in news articles over time. While in the 90s the term Friedensmission (peace operation) was predominant a reverse tendency can be observed since 2001 with Auslandseinsatz (foreign intervention) being now frequently used.

sagte Ruhe.¨ lated to the infrastructural standards in use in the said Ruhe.¨ CLARIN community. In section 4, we exemplify ,,The Europeans would fill the gap, Ruhe¨ said.” the intended use of the methods with case studies about steps necessary for identifying evocation: The tool support is meant to be semi-automatic, being able to separate reports from comments, and as the automatic tools propose candidates that strategies for identifying indirect speech. Section need to be validated or refused by the political sci- 6 is devoted to a conclusion and to the discussion entists. of future work. We combine a chain of corpus processing tools with classifier-based tools, e.g. for topic clas- 2 Project Goals sifiers, commentary/report classifiers, etc., make the tools interoperable to ensure flexible data ex- A collaboration between political scientists and change and multiple usage scenarios, and we em- computational linguists necessarily involves find- bed the tool collection under a web (service) - ing a common language in order to agree on based user interface. the precise objectives of a project. For exam- The remainder of this paper is structured as fol- ple, social scientists use the term codebook for lows. In section 2, we present an outline of the ar- manual annotations of text, similar to annotation chitecture of our tool collection, and we motivate schemes or guidelines in NLP. Both disciplines the architecture. Section 3 presents examples of share methodologies of interactive text analysis implemented modules, both from corpus process- which combine term based search, manual an- ing and search and retrieval of instances of com- notation and learning-based annotation of large plex concepts. We also show how our tools are re- amounts of data. In this section, we give a brief

56 summary of the goals from the perspective of each automatic analysis components have to be com- of the two disciplines, and then describe the text bined with manual annotation or interactive inter- corpus that is used in the project. Section 3 will vention of the human analyst. describe our approach to devising a system archi- In addition to this principled challenge, there tecture that serves to realize the goals. may be more mundane issues resulting from pro- cessing corpora whose origin stretches over many 2.1 Social Science Research Issue years. In our case, the data collection phase coin- Given the complexity of the underlying research cided with a spelling reform in German-speaking issues (cf. Section 1) and the methodological tra- countries. Many aspects of spelling changed twice dition of manual text coding by very well-trained (in 1996 and in 2006), and thus it is the responsi- annotators in the social science and particular in bility of the NLP branch of the project to provide political science, our project does not aim at any an abstraction over such changes and to enable to- fully-automatic solution for empirical issues in po- day’s users to run a homogeneous search over the litical science. Instead, the goal is to provide as texts using only the current spelling. While this much assistance to the human text analyst as possi- might be less important for generic web search ap- ble, by means of a workbench that integrates many plications, it is of great importance for our project, tasks that otherwise would have to be carried out where the overall objective is a combination of with different software tools (e.g., corpus prepro- quantitative and qualitative text analysis. cessing, KWIC searches, statistics). In our project, In our processing chain, we first need to harmo- the human analyst is concerned specifically with nize the data formats so that the processing tools manifestations of collective identities in newspa- operate on a common format. Rather than defin- per texts on issues of war and military interven- ing these from scratch, we aim at compatibility tions: who are the actors in political crisis man- with the standardization efforts of CLARIN3 and agement or conflict? How is this perspective of DARIAH4, two large language technology infras- responsible actors characterized in different news- tructure projects in Europe that in particular target papers (with different political orientation; in dif- eHumanities applications. One of the objectives ferent countries)? The analyst wants to find doc- is to provide advanced tools to discover, explore, uments that contain facets of such constellations, exploit, annotate, analyse or combine textual re- which requires search techniques involving con- sources. In the next section we give more details cepts on different levels of abstraction, ranging about how we interact which the CLARIN-D in- from specific words or named entities (which may frastructure (Boehlke et al., 2013). appear with different names in different texts) to event types (which may be realized with different 3 Architecture verb-argument configurations). Thus the text cor- The main goal is to provide a web-based user- pus should be enriched with information relevant interface to the social scientist to avoid any soft- to such queries, and the workbench shall provide ware installation. Figure 2 presents the workflow a comfortable interface for building such queries. of the different processing steps in this project. Moreover, various types and (possibly concurrent) The first part considers format issues that occur layers of human annotations have to complement if documents from different sources are used. The the automatic analysis, and the manual annota- main challenge is to recognize metadata correctly. tion would benefit from automatic control of code- 2 Date and source name are two types of metadata book compliance and the convergence of coding which are required for analyses in the social sci- decisions. ences. But also the separation of document con- 2.2 Natural Language Processing Research tent (text) and metadata is important to ensure that Issue only real content is processed with the NLP meth- ods. The results are stored in a repository which Large collections of text provide an excellent op- uses a relational database as a back-end. All fur- portunity for computational linguists to scale their ther modules are used to add more annotations to methods. In the scenario of a project like ours, this the textual data. First a complex linguistic pro- becomes especially challenging, because standard 3http://www.clarin.eu/ 2or, in NLP terms: annotation scheme. 4http://www.dariah.eu/

57 cessing chain is used to provide state-of-the-art sentation. The conversion tools account for differ- corpus linguistic annotations (see Section 3.2 for ent file encodings and convert input files to Uni- details). Then, to ensure that statistics over oc- code (UTF-8). We currently work on newspa- currence counts of words, word combinations and per articles wrapped with metadata. Text mining constructions are valid and not blurred by the mul- components read out those metadata and identify tiple presence of texts or text passages in the cor- text content in the documents. Metadata appear pus, we filter duplicates. Duplicates can occur at varying positions and in diverse notations, e.g. if our document set contains the same document for dates, indications of authors or newspaper sec- twice or if two documents are very similar, e.g. tions. The components account for these varia- they differ in only one sentence. tions and convert them to a consistent machine readable format. The extracted metadata are ap- pended to the XML representation. The result- ing XML is the starting point for further compu- Raw documents tational linguistic processing of the source docu- Import ments. Exploration Workbench The workbench contains a tool to identify text Duplicate filter duplicates and semi-duplicates via similarity mea- sures of pairs of articles (Kantner et al., 2011). Repository: Linguistic analysis Sentence splitter Tokenizer The method is based on a comparison of 5-grams, Tagger Parser Metadata Coref NER Structural data weighted by significance (tf-idf measure (Salton Textual data Complex Concept Builder and Buckley, 1988)). For a pair of documents it Topic filter yields a value on a “similarity scale” ranging from Concept detection 0 to 1. Values at medium range (0.4 to 0.8) are considered semi-duplicates. Web-based Data cleaning is important for the data-driven Userinterface studies. Not only duplicate articles have a nega- tive impact, also articles which are not of interest for the given topic have to be filtered out. There Figure 2: Overview of the complete processing are different approaches to classify articles into a chain. range of predefined topics. In the last years LDA (Blei et al., 2003; Niekler and Jahnichen,¨ 2012) We split the workflow for the user into two is one of the most successful methods to find top- parts: The first part is only used if the user im- ics in articles. But for social scientists the cate- ports new data into the repository. For that he gories typically used in LDA are not sufficient. We can use the exploration workbench (Section 3.1). follow the idea of Dualist (Settles, 2011; Settles Secondly, all steps for analyzing the data are done and Zhu, 2012) which is an interactive method for with the Complex Concept Builder (Section 3.2). classification. The architecture of Dualist is based 3.1 Exploration Workbench on MALLET (McCallum, 2002) which is easily Formal corpus inhomogeneity (e.g. various data integrable into our architecture. Our goal is to formats and inconsistent data structures) are a ma- design the correct feature to find relevant articles jor issue for researchers working on text corpora. for a given topic. Word features are not sufficient The web-based “Exploration Workbench” allows since we have to model more complex features (cf. for the creation of a consistent corpus from vari- Section 2.1). ous types of data and prepares data for further pro- The workbench is not exclusively geared to the cessing with computational linguistic tools. The data of the current project. We chose a modular workbench can interact with to existing computa- set-up of the tools of the workbench and provide tional linguistic infrastructure (e.g. CLARIN) and user-modifiable templates for the extraction of var- provides input for the repository also used by the ious kinds of metadata, in order to keep the work- Complex Concept Builder. bench adaptable to new data and to develop tools The workbench converts several input formats suitable for data beyond the scope of the current (TXT, RTF, HTML) to a consistent XML repre- corpus.

58 3.2 Complex Concept Builder can then be improved continuously. This aspect of the Complex Concept Builder is especially im- A central problem for political scientists who in- portant because new or adapted concepts that may tend to work on large corpora is the linguistic va- be looked for can be found without further help of riety in the expression of technical terms and com- natural language processing experts. plex concepts. An editorial or a politician cited in a news item can mobilize a collective identity 3.3 Implementation which can be construed from e.g. regional or so- cial affiliation, nationality or religion. A reason- We decided to use a web-based platform for our able goal in the context of the search for collec- system since the social scientist needs no software tive identity evocation contexts is therefore to find installation and we are independent of the used all texts which (possibly) contain collective iden- operating system. Only a state-of-the-art web- tities. Moreover, while we are training our inter- browser is needed. On the server side, we use a active tools on a corpus on wars and military in- tomcat installation that interacts with our UIMA terventions the same collective identities might be pipeline (Ferrucci and Lally, 2004). A HTML- expressed in different ways in a corpus i.e. on the rendering component designed in the project (and Eurocrisis. parametrizable) allows for a flexible presentation From a computational point of view, many dif- of the data. A major issue of our work is interac- ferent tools need to be joined to detect interest- tion. To solve this, we use JQuery and AJAX to ing texts. An example application could be a case dynamically interact between client- and server- where a political scientist intends to extract news- side. paper articles that cite a politician who tries to rally support for his political party. In order to 4 Case Study detect such text, we need a system to identify di- In this section we explore the interaction between rect and indirect speech and a sentiment system to various sub-systems and how they collaborate to determine the orientation of the statement. These find complex political concepts. The following systems in turn need various kinds of preprocess- Section 4.1 describes the detection of direct and ing starting from tokenization over syntactic pars- indirect speech and its evaluation follows in Sec- ing up to coreference resolution. The Complex tion 4.2. Section 4.3 is a general exploration of a Concept Builder is the collection of all these sys- few selected sub-systems which require, or benefit tems with the goal to assist the political scientists. from direct and indirect speech. Finally, Section So far, the Complex Concept Builder imple- 4.4 discusses a specific usage scenario for indirect ments tokenization (Schmid, 2009), lemmatisation speech. (Schmid, 1995), part-of-speech tagging (Schmid and Laws, 2008), named entity detection (Faruqui 4.1 Identifying Indirect Speech and Pado,´ 2010), syntactical parsing (Bohnet, 2010), coreference analysis for German (Lappin The Complex Concept Builder provides analy- and Leass, 1994; Stuckardt, 2001), relation extrac- ses on different linguistic levels (currently mor- tion (Blessing et al., 2012) and sentiment analysis phosyntax, dependency syntax, named entities) of for English (Taboada et al., 2011). annotation. We exploit this knowledge to identify It is important for a researcher of the humanities indirect speech along with a mentioned speaker. to be able to adapt existing classification systems Our indirect speech recognizer is based on three according to his own needs. A common procedure conditions: i) Consider all sentences that contain in both, NLP and political sciences, is to annotate at least one word which is tagged as subjunctive data. Therefore, one major goal of the project and (i.e. “*.SUBJ”) by the RFTagger. ii) This verb the Complex Concept Builder is to provide ma- has to be a direct successor of another verb in the chine learning systems with a wide range of pos- dependency tree. iii) This verb needs to have a subject. sible features — including high level information Figure 3 depicts the dependency parse tree of like sentiment, text type, relations to other texts, sentence 2. etc. — that can be used by non-experts for semi- automatic annotation and text selection. Active (2) Der Einsatz werde wegen der Risiken fur¨ die learning is used to provide immediate results that unbewaffneten Beobachter ausgesetzt, teilte

59 Additionally, for each sentence we extracted the speaker and the used verb of speech. We man-

VFIN.Full.3.Sg.Past.Ind ually evaluated 200 extracted triples (sentence, RFTags teilte informed speaker, verb of speech): The precision of our

OC SB system is: 92.5%

VFIN.Aux.3.Sg.Pres.Subj Examples 2, 3 and 4 present good candidates werde Missionschef am mit will be head of mission on which are helpful for further investigations on col- lective identities. In example 3 Cardinal Lehmann is a representative speaker of the Catholic commu- ausgesetzt Mood Samstag . nity which is a collective identity. Our extracted stopped Mood Saturday . sentences accelerate the search for such candidates which amounts to looking manually for needles in

Einsatz wegen , Robert a haystack. mission because of , Robert

example speaker verb of speech

Der Risiko the risks (2) Robert Mood teilte (told) (3) Kardinal Karl Lehmann sagte (said) (4) Sergej Ordzhonikidse sagte (said) (5) Bild (picture) truben¨ (tarnish) (6) sein (be) sein (be) Figure 3: Dependency parse of a sentence that contains indirect speech (see Sentence 2). Examples 5 and 6 show problems of our first approach. In this case, the speaker is not a person Missionschef Robert Mood am Samstag mit. or an organisation, and the verb is not a verb of speech. The mission will be stopped because of the risks to the unarmed observers, informed Head of Mission Robert (3) Ein Angriffskrieg jeder Art sei ” sit- Mood on Saturday. tlich verwerflich ”, sagte der Vorsitzende The speaker of the indirect speech in Sentence der Bischoffskonferenz, Kardinal Karl 2 is correctly identified as Missionschef (Head of Lehmann. Mission) and the corresponding verb is teilte mit Any kind of war of aggression is ”morally reprehen- sible,” said the chairman of the Bishops’ Conference, (from mitteilen) (to inform). Cardinal Karl Lehmann. The parsing-based analysis helps to identify the speaker of the citation which is a necessary in- (4) Derartige Erklarungen¨ eines Staatschefs formation for the later interpretation of the cita- seien im Rahmen der internationalen tion. As a further advantage, such an approach Beziehungen inakzeptabel, sagte der UN- helps to minimize the need of lexical knowledge Generaldirektor Sergej Ordzhonikidse for the identification of indirect speech. Our er- gestern in Genf. ror analysis below will show that in some cases Such statements of heads of states are unacceptable in a lexicon can help to avoid false positives. A lexi- the context of international relations, said UN General Director Sergei Ordzhonikidse in Geneva yesterday. con of verbs of communication can easily be boot- strapped by using our approach to identify candi- (5) Wurden¨ die Wahlen verschoben, trubte¨ sich dates for the list of verbs which then restrict the das geschonte¨ Bild. classifier in order to achieve a higher precision. Would the elections be postponed, the embellished im- age would tarnish. 4.2 Indirect Speech Evaluation (6) Dies sei alles andere als einfach, ist aus Of- For a first impression, we present a list of sen- fizierskreisen zu horen.¨ tences which were automatically annotated as pos- This is anything but simple, is to hear from military itive instances by our indirect speech detector. circles. The sentences were rated by political scientists.

60 identities are mobilised by entities of political de- most used speech verbs bate (i.e. persons, organisations, etc.); the detec-

erzählen (tell) tion of indirect speech is mandatory for any such betonen (emphasize) werden (will) analysis. haben (have) schreiben (write) 4.4 Commentary/Report Classification mitteilen (share) berichten (report) A useful distinction for political scientists dealing meinen (state) erklären (explain) with newspaper articles is the distinction between sagen (say) articles that report objectively on events or back- 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00% 40.00% grounds and editorials or press commentaries. We first extracted opinionated and objective texts from DeReKo corpus (Stede, 2004; Kupietz Figure 4: 10 most used verbs (lemma) in indirect speech. et al., 2010). Some texts were removed in order to balance the corpus. The balanced corpus contains 2848 documents and has been split into a develop- 4.3 Using Indirect Speech ment and a training and test set. 570 documents Other modules benefit from the identification of were used for the manual creation of features. The indirect speech, as can be seen from Sentence 7. remaining 2278 documents were used to train and The sentiment system assigns a negative polarity evaluate classifiers using 10-fold cross-validation of −2.15 to the sentence. The nested sentiment with the WEKA machine learning toolkit (Hall et sources, as described by (Wiebe et al., 2005), of al., 2009) and various classifiers (cf. Table 1). this sentence require a) a direct speech with the The challenge is that the newspaper articles speaker “Mazower” and b) an indirect speech with from the training and evaluation corpus come from 5 the speaker “no one” to be found. different newspapers and, of course, from differ- ent authors. Commentaries in the yellow press (7) ”There were serious arguments about what tend to have a very different style and vocabulary should happen to the Slavs and Poles in east- than commentaries from broadsheet press. There- ern Europe,” says Mazower, ”and how many fore, special attention needs to be paid to the in- of them should be sent to the camps and what dependence of the classifier from different authors proportion could be Germanised ... No one and different newspapers. For this reason, we use ever came out and directly said Hitler had got hand-crafted features tailored to this problem. In it wrong, but there was plenty of implied crit- return, this means omitting surface-form features icism through comparisons with the Roman (i.e. words themselves). empire. [...]”6 The support vector machine used the SMO al- A collective identity evoked in Sentence 7 is gorithm (Platt and others, 1998) with a polynomial “the Germans”— although the term is not explic- kernel K(x, y) =< x, y > e with e = 2. All other itly mentioned. This collective identity is de- algorithms were used with default settings. scribed as non-homogeneous in the citation and can be further explored manually by the political precision recall f-score scientists. SVM 0.819 0.814 0.813 The following are further applications of the Naive Bayes 0.79 0.768 0.764 identified indirect speeches a) using the frequency Multilayer Percep- 0.796 0.795 0.794 of speeches per text as a feature for classifica- tron tion; e.g. a classification system for news re- ports/commentaries as described in Section 4.4 b) Table 1: Results of a 10-fold cross-validation for a project-goal is to find texts in which collective various machine learning algorithms. 5The reported sentiment value for the whole sentence is applicable only to the direct speech. The indirect speech (i.e. A qualitative evaluation shows that direct and “Hitler had got it wrong”) needs a more fine-grained polarity indirect speech is a problem for the classifier. score. Since our Complex Concept Builder is very flexible, it Opinions voiced via indirect speech should not is trivial to score each component separately. 6http://www.guardian.co.uk/education/2008/jul lead to a classification as ‘Commentary’, but /01/academicexperts.highereducationprofile should be ignored. Additionally, the number of

61 uses of direct and indirect speech by the author can textual data and the encoding of the data is hard provide insight into the intention of the author. A for non-experts. Furthermore, WebLicht does not common way to voice one’s own opinion, without yet support the combination of manual and au- having to do so explicitly, is to use indirect speech tomatic annotation needed for text exploration in that the author agrees with. Therefore, the number the social science. Our approach is based on the of direct and indirect speech uses will be added webservices used by WebLicht. But in contrast to to the classifier. First experiments indicate that the WebLicht, we provide two additional components inclusion of direct and indirect speech increase the that simplify the integration (exploration work- performance of the classifier. bench) and the interpretation (complex concept builder) of the research data. The former is in- 5 Related Work tended, in the medium term, to be made available in the CLARIN framework. Many approaches exist to assist social scientists in dealing with large scale data. We discuss some 6 Conclusion and Outlook well-known ones and highlight differences to the approach described above. We developed and implemented a pipeline of var- The Europe Media Monitor (EMM) (Stein- ious text processing tools which is designed to as- berger et al., 2009) analyses large amounts of sist political scientists in finding specific, complex newspaper articles and assists anyone interested in concepts within large amounts of text. Our case news. It allows its users to search for specific top- studies showed that our approach can provide ben- ics and automatically clusters articles from differ- eficial assistance for the research of political sci- ent sources. This is a key concept of the EMM, entists as well as researcher from other social sci- because it collects about 100, 000 articles in ap- ences and the humanities. A future aspect will be proximately 50 languages per day and it is impos- to find metrics to evaluate our pipeline. In recently sible to scan through these by hand. EMM users started annotation experiments on topic classifica- are EU institutions, national institutions of the EU tion Cohen’s kappa coefficient (Carletta, 1996) is member states, international organisations and the mediocre. It may very well be possible that the public (Steinberger et al., 2009). complex concepts, like multiple collective identi- The topic clusters provide insight into “hot” ties, are intrinsically hard to detect, and the anno- topics by simply counting the amount of articles tations cannot be improved substantially. per cluster or by measuring the amount of news on The extension of the NLP pipeline will be an- a specific topic with regards to its normal amount other major working area in the future. Examples of news. Articles are also data-mined for geo- are sentiment analysis for German, adding world graphical information, e.g. to update in which knowledge about named entities (e.g. persons and geographical region the article was written and events), identification of relations between enti- where the topic is located. Social network infor- ties. mation is gathered and visualised as well. Finally, all these systems need to be evaluated Major differences between the EMM and our not only in terms of f-score, precision and recall, approach are the user group and the domain of but also in terms of usability for the political scien- the corpus. The complex concepts political sci- tists. This also includes a detailed investigation of entists are interested in are much more nuanced various political science concepts and if they can than the concepts relevant for topic detection and be detected automatically or if natural language the construction of social networks. Additionally, processing can help the political scientists to de- the EMM does not allow its users to look for their tect their concepts semi-automatically. The defini- own concepts and issues, while this interactivity tion of such evaluation is an open research topic in is a central contribution of our approach (cf. Sec- itself. tions 1, 2.1 and 3.2). Acknowledgements The CLARIN-D project also provides a web- based platform to create NLP-chains. It is called The research leading to these results has been WebLicht (Hinrichs et al., 2010), but in its cur- done in the project eIdentity which is funded from rent form, the tool is not immediately usable for the Federal Ministry of Education and Research social scientists as the separation of metadata and (BMBF) under grant agreement 01UG1234.

62 References Andrew Kachites McCallum. 2002. Mal- let: A machine learning for language toolkit. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. http://www.cs.umass.edu/ mccallum/mallet. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993–1022, March. Andreas Niekler and Patrick Jahnichen.¨ 2012. Match- ing results of latent dirichlet allocation for text. Andre Blessing, Jens Stegmann, and Jonas Kuhn. In Proceedings of ICCM 2012, 11th International 2012. SOA meets relation extraction: Less may be Conference on Cognitive Modeling, pages 317–322. more in interaction. In Proceedings of the Work- Universitatsverlag¨ der TU Berlin. shop on Service-oriented Architectures (SOAs) for the Humanities: Solutions and Impacts, Digital Hu- John Platt et al. 1998. Sequential minimal optimiza- manities, pages 6–11. tion: A fast algorithm for training support vector machines. technical report msr-tr-98-14, Microsoft Volker Boehlke, Gerhard Heyer, and Peter Wittenburg. Research. 2013. IT-based research infrastructures for the hu- manities and social sciences - developments, exam- Gerard Salton and Christopher Buckley. 1988. Term- ples, standards, and technology. it - Information weighting approaches in automatic text retrieval. In- Technology, 55(1):26–33, February. formation processing & management, 24(5):513– 523. Bernd Bohnet. 2010. Top accuracy and fast depen- dency parsing is not a contradiction. In Proceedings Helmut Schmid and Florian Laws. 2008. Estima- of the 23rd International Conference on Computa- tion of conditional probabilities with decision trees tional, pages 89–97. and an application to fine-grained POS tagging. In Proceedings of the 22nd International Conference Jean Carletta. 1996. Assessing agreement on classi- on Computational Linguistics (Coling 2008) fication tasks: The kappa statistic. Computational , pages Linguistics, 22(2):249–254. 777–784, Manchester, UK, August.

Manaal Faruqui and Sebastian Pado.´ 2010. Train- Helmut Schmid. 1995. Improvements in part-of- ing and evaluating a german named entity recog- speech tagging with an application to german. In nizer with semantic generalization. In Proceedings Proceedings of the ACL SIGDAT-Workshop, pages of KONVENS 2010, Saarbrucken,¨ Germany. 47–50.

D. Ferrucci and A. Lally. 2004. UIMA: an architec- Helmut Schmid, 2009. Corpus Linguistics: An In- tural approach to unstructured information process- ternational Handbook, chapter Tokenizing and Part- ing in the corporate research environment. Natural of-Speech Tagging. Handbooks of Linguistics and Language Engineering, 10(3-4):327–348. Communication Science. Walter de Gruyter, Berlin.

Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Burr Settles and Xiaojin Zhu. 2012. Behavioral fac- Pfahringer, Peter Reutemann, and Ian H Witten. tors in interactive training of text classifiers. In Pro- 2009. The weka data mining software: an update. ceedings of the 2012 Conference of the North Amer- ACM SIGKDD Explorations Newsletter, 11(1):10– ican Chapter of the Association for Computational 18. Linguistics: Human Language Technologies, pages 563–567. Association for Computational Linguis- Erhard W. Hinrichs, Marie Hinrichs, and Thomas Za- tics. strow. 2010. WebLicht: Web-Based LRT Services for German. In Proceedings of the ACL 2010 System Burr Settles. 2011. Closing the loop: Fast, inter- Demonstrations, pages 25–29. active semi-supervised annotation with queries on features and instances. In Proceedings of the Con- Cathleen Kantner, Amelie Kutter, Andreas Hilde- ference on Empirical Methods in Natural Language brandt, and Mark Puettcher. 2011. How to get rid Processing, pages 1467–1478. Association for Com- of the noise in the corpus: Cleaning large samples putational Linguistics. of digital newspaper texts. International Relations Online Working Paper, 2, July. Manfred Stede. 2004. The potsdam commentary corpus. In Proceedings of the 2004 ACL Work- Marc Kupietz, Cyril Belica, Holger Keibel, and An- shop on Discourse Annotation, DiscAnnotation ’04, dreas Witt. 2010. The german reference corpus pages 96–102, Stroudsburg, PA, USA. Association dereko: a primordial sample for linguistic research. for Computational Linguistics. In Proceedings of the 7th conference on interna- tional language resources and evaluation (LREC Ralf Steinberger, Bruno Pouliquen, and Erik Van 2010), pages 1848–1854. Der Goot. 2009. An introduction to the europe me- dia monitor family of applications. In Proceedings Shalom Lappin and Herbert J Leass. 1994. An algo- of the Information Access in a Multilingual World- rithm for pronominal anaphora resolution. Compu- Proceedings of the SIGIR 2009 Workshop, pages 1– tational linguistics, 20(4):535–561. 8.

63 Roland Stuckardt. 2001. Design and enhanced evalua- tion of a robust anaphor resolution algorithm. Com- putational Linguistics, 27(4):479–506. Maite Taboada, Julian Brooke, Milan Tofiloski, Kim- berly Voll, and Manfred Stede. 2011. Lexicon- based methods for sentiment analysis. Computa- tional linguistics, 37(2):267–307. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emo- tions in language. Language Resources and Evalu- ation, 39(2-3):165–210.

64 Learning to Extract Folktale Keywords

Dolf Trieschnigg, Dong Nguyen and Mariet¨ Theune University of Twente Enschede, The Netherlands {d.trieschnigg,d.nguyen,m.theune}@utwente.nl

Abstract keywords are a selection of the words or phrases appearing in the document itself (Turney, 2000; Manually assigned keywords provide a Frank et al., 1999). valuable means for accessing large docu- In this paper we look into keyword extraction ment collections. They can serve as a shal- in the domain of cultural heritage, in particular low document summary and enable more for extracting keywords from folktale narratives efficient retrieval and aggregation of infor- found in the Dutch Folktale Database (more on mation. In this paper we investigate key- this collection in section 3). These narratives words in the context of the Dutch Folk- might require a different approach for extraction tale Database, a large collection of stories than in other domains, such as news stories and including fairy tales, jokes and urban leg- scholarly articles (Jiang et al., 2009). Stories in the ends. We carry out a quantitative and qual- Dutch Folktale Database are annotated with un- itative analysis of the keywords in the col- controlled, free-text, keywords. Because suggest- lection. Up to 80% of the assigned key- ing keywords which do not appear in the text is a words (or a minor variation) appear in the considerably harder task to automate and to eval- text itself. Human annotators show moder- uate, we restrict ourselves to keywords extracted ate to substantial agreement in their judg- from the text itself. ment of keywords. Finally, we evaluate a In the first part of this paper we study the cur- learning to rank approach to extract and rent practice of keyword assignment for this col- rank keyword candidates. We conclude lection. We analyze the assigned keywords in the that this is a promising approach to auto- collection as a whole and present a more fine- mate this time intensive task. grained analysis of a sample of documents. More- over, we investigate to what extent human anno- 1 Introduction tators agree on suitable keywords extracted from Keywords are frequently used as a simple way the text. Manually assigning keywords is an ex- to provide descriptive metadata about collections pensive and time-consuming process. Automatic of documents. A set of keywords can concisely assignment would bring down the cost and time to present the most important aspects of a document archive material. In the second part of this paper and enable quick summaries of multiple docu- we evaluate a number of automatic keyword ex- ments. The word cloud in Figure 1, for instance, traction methods. We show that a learning to rank gives a quick impression of the most important approach gives promising results. topics in a collection of over 40,000 documents (a The overview of this paper is as follows. We collection of Dutch folktales). first describe related work in automatic keyword Keyword assignment or generation is the task assignment. In section 3 we introduce the Dutch of finding the most important, topical keywords or Folktale Database. In section 4 we present an anal- keyphrases to describe a document (Turney, 2000; ysis of the keywords currently used in the folktale Frank et al., 1999). Based on keywords, small database. In section 5 we investigate the agree- groups of documents (Hammouda et al., 2005) or ment of human annotators on keyword extraction. large collections of documents (Park et al., 2002) In section 6 we present and evaluate an automatic can be summarized. Keyword extraction is a re- method for extracting and ranking keywords. We stricted case of keyword assignment: the assigned end with a discussion and conclusion in section 7.

65 Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 65–73, Sofia, Bulgaria, August 8 2013. c 2013 Association for Computational Linguistics Keyword (translation) Frequency

dood (death) 5,861 man (man) 4,547 vrouw (woman) 4,154 sterven (to die) 3,168 huis (house) 2,894 spokerij (haunting) 2,491 duivel (devil) 2,487 nacht (night) 2,449 voorteken (omen) 2,380 voorloop (forerunnings) 2,372 geld (money) 2,370 toverij (sorcery) 2,322 zien (to see) 2,270 heks (witch) 2,233 boer (farmer) 2,189 water (water) 2,177 angst (fear) 2,091 hekserij (witchcraft) 1,911 kind (child) 1,853 spoken (ghosts) 1,747 spook (ghost) 1,742 seks (sex) 1,659

Figure 1: Frequent keywords in the Dutch Folktale Database

2 Related Work the frequency of a potential keyword in the doc- ument and the inverse document frequency in the Because of space limitations, we limit our dis- collection (Turney, 2000; Hulth, 2003; Frank et cussion of related work to keyword extraction in al., 1999). Examples of more complex features the context of free-text indexing. Automated con- are: features based on characteristics of lexical trolled vocabulary indexing is a fundamentally chains, requiring a lexical database with word different task (see for instance Medelyan and Wit- meanings (Ercan and Cicekli, 2007); features re- ten (2006) and Plaunt and Norgard (1998)). lated to frequencies in external document collec- Typically, keyword extraction consists of two tions and query logs (Bendersky and Croft, 2008; steps. In the first step candidate keywords are de- Yih et al., 2006; Liu et al., 2009b; Xu et al., 2010); termined and features, such as the frequency or and a feature to determine the cohesiveness of re- position in the document, are calculated to char- trieved documents with that keyword (Bendersky acterize these keywords. In the second step the and Croft, 2008). candidates are filtered and ranked based on these features. Both unsupervised and supervised algo- 2.3 Unsupervised Methods for Keyword rithms have been used to do this. Extraction 2.1 Candidate Extraction Unsupervised methods for keyword extraction Candidate keywords can be extracted in a number typically rely on heuristics to filter and rank the of ways. The simplest approach is to treat each keywords in order of importance. For instance, single word as a candidate keyword, optionally by ranking the candidates by their importance in filtering out stop words or only selecting words the collection – estimated by the inverse docu- with a particular Part-of-Speech (Liu et al., 2009a; ment frequency. Another approach is to apply the Jiang et al., 2009). More sophisticated approaches PageRank algorithm to determine the most impor- allow for multi-word keywords, by extracting con- tant keywords based on their co-occurrence link- secutive words from the text, optionally limited to structure (Mihalcea and Tarau, 2004). Liu et al. keywords adhering to specific lexical patterns (Os- (2009b) employed clustering to extract keywords inski and Weiss, 2005; Hulth, 2003; Rose et al., that cover all important topics from the original 2010; Frank et al., 1999; Turney, 2000). text. From each topic cluster an exemplar is deter- mined and for each exemplar the best correspond- 2.2 Features to Characterize Keywords ing keyword is determined. Many features for characterizing candidate key- 2.4 Supervised Methods for Keyword words have been investigated previously, with Extraction varying computational complexities and resource requirements. The simplest features are based Early supervised methods used training data to set on document and collection statistics, for instance the optimal parameters for (unsupervised) systems

66 based on heuristics (Turney, 2000). Other methods 2000 approached keyword extraction as a binary classi- fication problem: given a candidate keyword it has 1500 to be classified as either a keyword or not. Meth- ods include decision trees (Bendersky and Croft, 1000 2008), Naive Bayes (Frank et al., 1999) and Sup- port Vector Machines (Zhang et al., 2006). Zhang Document frequency 500 et al. (2008) approached keyword extraction as a labeling problem for which they employed condi- 0 0 10 20 30 40 50 tional random fields. Recently, keyword extrac- Number of assigned keywords tion has been cast as a ranking problem and learn- ing to rank techniques have been applied to solve Figure 2: Number of assigned keywords per doc- it (Jiang et al., 2009). Jiang et al. (2009) concluded ument that learning to rank approaches performed better than binary classifiers in the context of extracting keywords from scholarly texts and websites. Dif- assigned to the collection as a whole. After that we ferent variations of learning to rank exist, see (Li, make a more fine-grained analysis of the keywords 2011) for an overview. assigned to a selection of the documents. 3 The Dutch Folktale Database 4.1 Quantitative Analysis The Dutch Folktale Database is a repository of over 40,000 folktales in Dutch, old Dutch, Frisian We analyzed a snapshot from the Dutch Folktale and a large number of Dutch dialects. The mate- Database (from early 2012) that consists of 41,336 rial has been collected in the 19th, 20th and 21th folktales. On average, 15 keywords have been as- centuries, and consists of stories from various pe- signed to each of these documents (see Figure 2). riods, including the Middle Ages and the Renais- The median number of assigned keywords is 10, sance. The collection has both an archival and a however. The keywords vocabulary has 43,195 research function. It preserves an important part unique keywords, most of which consist of a sin- of the oral cultural heritage of the Netherlands and gle word (90%). Figure 1 shows a word cloud can be used for comparative folk narrative studies. of keywords used in the collection; more frequent Since 2004 the database is available online1. keyword types appear larger. On the right, it lists The real value of the database does not only lie the most frequent keyword types (and their trans- the stories themselves, but also in their manually lations). The assignment of keywords to docu- added set of descriptive metadata fields. These ments has a Zipfian distribution: a few keyword fields include, for example, a summary in Dutch, types are assigned to many documents, whereas a list of proper names present in the folktales, and many keyword types are assigned to few docu- a list of keywords. Adding these metadata is a ments. time-consuming and demanding task. In fact, the When we limit our collection to stories in Dutch amount of work involved hampers the growth of (15,147 documents), we can determine how many the folktale database. A large backlog of digitized of the manually assigned keywords can be found folktales is awaiting metadata assignment before literally in the story text2. We define the keyword they can be archived in the collection. Being able coverage of a document as the fraction of its as- to automatically assign keywords to these docu- signed keywords which is found in the full text ments would be a first step to speed up the archiv- or its summary. The average keyword coverage ing process. of the Dutch stories is 65%. Figure 3 shows a 4 Analysis of Assigned Keywords histogram of the coverage. It shows that most of the documents have a keyword coverage of 0.5 or In this section we analyze the keywords that have more. been manually assigned to the stories in the Dutch Folktale Database. First we look at the keywords 2Stories in other languages or dialects have been assigned 1http://www.verhalenbank.nl, in Dutch only Dutch keywords.

67 3000 Classification Count Perc. 2500 Literal 669 67.6% Almost literal 120 12.1% 2000 Synonym 49 5.0% Hypernym 23 2.3% 1500 Typing error 2 0.2% Other 126 12.7% 1000 Total 989 100.0% Document frequency

500 Table 1: Keyword types in a set of 1000 folktales 0 0.0 0.2 0.4 0.6 0.8 1.0 Keyword coverage by full-text and summary of more abstract keywords, which we will study in Figure 3: Keyword coverage of folktales in Dutch future research.

5 Evaluating Agreement in Keyword 4.2 Qualitative Analysis Assignment The quantitative analysis does not provide insight into what kind of keywords have been assigned. The previous analyses raise the question whether Therefore, we analyzed a selection of documents the keywords have been consistently assigned: do more thoroughly. For each of the five largest gen- annotators choose the same keywords when pre- res in the collection (fairy tale, traditional legend, sented with the same text? Moreover, knowing joke, urban legend and riddle) we sampled 10 tales the difficulty of the task for human annotators will and manually classified the keywords assigned to give us an indication of the level of performance these folktales. A total of almost 1000 keywords we may expect from automatic keyword assign- was analyzed. Table 1 summarizes the statistics of ment. To determine the agreement between an- this analysis. Almost 80% of the keywords appear notators we asked ten annotators to classify the vocabulary of five folktales from different genres. literally or almost literally in the text. The almost 3 literal appearances include keywords which differ Frog (van den Bosch et al., 2007) was used to in quantity (plural versus singular form) and verb extract the vocabulary of lemmas. After carefully forms. Verb forms vary in tense (present rather reading a folktale, the annotator classified the al- than past tense) and infinitive keywords of sepa- phabetically sorted list of lemmas extracted from rable verbs. An example of the latter is the as- the text. Each lemma was classified as either: 1) signment of the keyword “terugkeren”, to return, not a relevant keyword – should not be assigned non where “keren” (∼ turn) and “terug” (∼ back) are to this document ( ); 2) a relevant keyword – rel highly used in a sentence. Of the analyzed keywords should be assigned ( ); 3) a relevant key- hrel 5% are synonyms of words appearing the text and word – should definitely be assigned ( ). The 2.3% are hypernyms of words appearing the text three levels of relevance were used to see whether (e.g. “wapen”, weapon, is used as a keyword with annotators have a preference for certain keywords. “mes”, knife, mentioned in the text). The remain- The pairwise agreement between annotators was ing 13% of the keywords represent abstract topic, measured using Cohen’s kappa. Each document event and activity descriptions. For example, the was judged twice, totaling a set of 25 documents. keyword “wegsturen”, to send away, when one of Most of the annotators were familiar with the folk- the characters explicitly asks someone to leave. tale database and its keywords; two were active Other examples are the keywords “baan”, job, and contributors to the database and thus had previous “arbeid”, labor, when the story is about an unem- experience in assigning keywords to folktales. ployed person. On average, the annotators judged 79% of the vocabulary as non-relevant as keywords. 9% and Based on these numbers we can conclude that 12% of the vocabulary was judged as relevant and based on extraction techniques alone we should highly relevant respectively, but there was a large be able to reproduce a large portion of the manual variation in these percentages: some annotators keyword assignment. When thesauri are employed assigned more highly relevant keywords, others to find synonyms and hypernyms, up to 87% of the assigned more relevant keywords. manually assigned keywords could be found. A much harder task is to obtain the remaining 13% 3http://ilk.uvt.nl/frog/

68 Cohen’s Kappa what criteria should be met by the keywords. Such Classes Average σ Min Max instructions are not available to current annotators non, rel, hrel 0.48 0.14 0.16 0.77 of the collection either. non, rel + hrel 0.62 0.16 0.25 0.92 non + rel, hrel 0.47 0.20 0.0 0.84 We conclude that annotators typically agree on the keywords from a text, but have a varying Table 2: Classification agreement between annota- notion of highly relevant keywords. The aver- tors. Non: non-relevant, rel: relevant, hrel: highly age keywords-based representation strongly con- relevant. denses the documents vocabulary: a document can be represented by a fifth (21%) of its vocabulary5. This value can be used as a cut-off point for meth- The two experienced annotators showed a con- ods ranking extracted keywords, discussed here- sistently higher average agreement in comparison after. to the other annotators (0.56 and 0.50 for non, rel, hrel; 0.7 and 0.64 for non, rel + hrel; 0.56 and 0.50 6 Automatically Extracting Keywords for non + rel, hrel). Moreover, they assigned more (relevant and highly relevant) keywords to the doc- In the last part of this paper we look into automati- uments on average. cally extracting keywords. We compare a learning Table 2 summarizes the agreement measured to rank classifier to baselines based on frequency between annotators. The first row indicates the and reuse in their ability to reproduce keywords agreement when considering agreement over all found in manually classified folktales. three classes; the second row indicates the agree- In all cases we use the same method for extract- ment when treating relevant and highly relevant ing keyword candidates. Since most of the manual keywords as the same class; the last row shows keywords are single words (90% of the used key- the agreement in indicating the same highly rel- word types in the collection), we simply extract evant keywords. The numbers indicate moder- single words as keyword candidates. We use Frog ate agreement between annotators over all three for tokenization and part of speech tagging. Stop classes and when considering the choice of highly words are not removed. relevant keywords. Annotators show substantial agreement on deciding between non-relevant and 6.1 Baseline Systems relevant keywords. Table 3 shows the agreement We use a basic unsupervised baseline for keyword between annotators on keywords with different extraction: the words are ranked according to de- parts of speech (CGN4 tagset). Most disagree- scending TF-IDF. We refer to this system as TF- ments are on nouns, adjectives and verbs. Verbs IDF. TF, term frequency, and IDF, inverse docu- and adjectives show few agreements on relevant ment frequency, are indicators of the term’s local and highly relevant keywords. In contrast, on and global importance and are frequently used in 20% of the nouns annotators agree on their rele- information retrieval to indicate the relative impor- vance. It appears that the annotators do not agree tance of a word (Baeza-Yates and Ribeiro-Neto, whether adjectives and verbs should be used as 2011). keywords at all. We can give three other reasons Note that a word appearing once in the collec- why annotators did not agree. First, for longer tion has the highest IDF score. This would imply stories annotators were presented with long lists that the most uncommon words are also the most of candidate keywords. Sometimes relevant key- important resulting in a bias towards spelling er- words might have been simply overlooked. Sec- rors, proper names, and other uncommon words. ond, it turned out that some annotators selected Hence, our second baseline takes into account some keywords in favor to other keywords (for in- whether a keyword has been used before in a train- stance a hyponym rather than a hypernym), where ing set. Again, the candidates are ranked by de- others simply annotated both as relevant. Third, scending TF-IDF, but now keywords appearing in the disagreement can be explained by lack of de- the training collection are ranked above the key- tailed instructions. The annotators were not told words not appearing in the collection. We refer to how many (highly) relevant keywords to select or this baseline as TF-IDF-T.

4Corpus Gesproken Nederlands (Spoken Dutch Corpus), 5Based on the figures that on average 9% of the vocabu- http://lands.let.kun.nl/cgn/ehome.htm lary is judged as relevant and 12% as highly relevant

69 Part of speech Adjective Adverb Noun Special Numeral Prep. Verb Number of words 272 257 646 131 53 268 664 Agreement non 70% 96% 40% 95% 81% 99% 73% rel 4% 0% 6% 0% 0% 0% 3% hrel 1% 0% 14% 2% 2% 0% 4% Disagreement non ↔ rel 15% 2% 17% 2% 11% 0% 12% non ↔ hrel 5% 1% 8% 2% 4% 1% 5% rel ↔ hrel 5% 0% 15% 0% 2% 0% 4%

Table 3: Agreement and disagreement of annotators on keywords with different parts of speech. Values are column-wise percentages. Tags with full agreement are not shown.

6.2 Learning to Rank Keywords 6.2.2 Document Context Following Jiang et al. (2009) we apply a learn- We use the following document context features: ing to rank technique to rank the list of extracted tf: the term frequency indicates the number of ap- keywords. We train an SVM to classify the rel- pearances of the word divided by the total number ative ordering of pairs of keywords. Words cor- of tokens in the document. responding to manual keywords should be ranked first offset: indicates the offset of the word’s higher than other words appearing in the docu- first appearance in the document, normalized by ment. We use SVM-rank to train a linear ranking the number of tokens in the document (follow- SVM (Joachims, 2006). We use the following fea- ing Zhang et al. (2008)). Important (key)words are tures. expected to be mentioned early. first sentence offset: indicates the offset of the 6.2.1 Word Context first sentence in which the token appears, normal- We use the following word context features: ized by the number of sentences in the document. starts uppercase: indicates whether the token sentence importance: indicates the maxi- starts with an uppercase letter (1) or not (0). Since mum importance of a sentence in which the proper names are not used as keywords in the folk- word appears, as measured by the SumBasic tale database, this feature is expected to be a neg- score (Nenkova and Vanderwende, 2005). Sum- ative indicator of a word being a keyword. Basic determines the relative importance of sen- contains space: indicates whether the token con- tences solely on word probability distributions in tains a space (Frog extracts some Dutch multi- the text. word phrases as a single token). Tokens with dispersion: indicates the dispersion or scattering spaces are not very common. of the word in the document. Words which are is number: indicates whether the token consists highly dispersed are expected to be more impor- of only digits. Numbers are expected not to be a tant. The DPnorm is used as a dispersion measure, keyword. proposed in Gries (2008). contains letters: indicates whether the token con- tains at least a single letter. Keywords are expected 6.2.3 Collection Context to contain letters. We use the following features from the collec- all capital letters: indicates whether the token tion/training context: consists of only capital letters. Words with only idf: the inverse document frequency indicates the capital letters are not expected to be keywords. collection importance of the word based on fre- single letter: indicates whether the token consists quency: frequent terms in the collection are less of only one letter. One letter keywords are very important than rare terms in the collection. uncommon. tf.idf: combines the tf and idf features by multi- contains punctuation: indicates whether the to- plying them. It indicates a trade-off between local ken contains punctuation such as apostrophes. and global word importance. Keywords are expected not to contain punctuation. is training keyword: indicates whether the word part of speech: indicates the part of speech of is used in the training collection as a keyword. the token (each tag is a binary feature). Nouns assignment ratio: indicates the percentage of are expected to be a positive indicator of key- documents in which the term is present in the text words (Jiang et al., 2009). and in which it is also assigned as a keyword.

70 6.3 Evaluation Method System MAP P@5 P@R TF-IDF 0.260 0.394 0.317 We evaluate the ranking methods on their ability TF-IDF-T 0.336 0.541 0.384 to reproduce the manual assignment of keywords. rank-SVM 0.399 0.631 0.453 Ideally the ranking methods rank these manual Table 4: Keyword extraction effectiveness. The keywords highest. We measure the effectiveness differences between systems are statistically sig- of ranking in terms of (mean) average precision nificant (paired t-test, p< 0.001) (MAP), precision at rank 5 (P@5) and precision at rank R (P@R), similar to Jiang et al. (2009). Note Change in that we use all the manually assigned keywords Feature MAP P@5 P@R as a ground truth, including words which do not assignment ratio -0.036 -0.056 -0.038 occur in the text itself. This lowers the highest is training keyword 0.006 0.002 0.005 achievable performance, but it will give a better tf.idf -0.004 -0.010 -0.002 part of speech -0.003 -0.007 0.000 idea of the performance for the real task. dispersion -0.001 -0.001 0.000 We perform a 10-fold stratified cross-validation idf 0.001 0.002 0.000 starts uppercase 0.000 0.000 -0.001 with a set of 10,900 documents from the Dutch first offset 0.000 0.000 0.000 Folktale Database, all written in modern Dutch. tf 0.000 0.000 0.000 contains space 0.000 0.000 0.000 is number 0.000 0.000 0.000 all capital letters 0.000 0.000 0.000 6.4 Results contains punctuation 0.000 0.000 0.000 contains letters 0.000 0.000 0.000 Table 4 lists the performance of the three tested sentence importance 0.000 0.000 0.000 TF-IDF first sentence offset 0.000 0.000 0.000 systems. The system performs worst, single letter 0.000 0.000 0.000 and is significantly outperformed by the TF-IDF- T system, which in turn is significantly outper- Table 5: Differences in performance when leaving formed by the rank-SVM system. On average, out features. The features are ordered by descend- rank-SVM returns 3 relevant keywords in its top ing difference in MAP. 5. The reported mean average precision values are affected by manual keywords which are not present in the text itself. To put these numbers 6.5 Feature Ablation in perspective: if we would put the manual key- To determine the added value of the individual fea- words which are in the text in an optimal ranking, tures we carried out an ablation study. Table 5 i.e. return these keywords first, we would achieve lists the changes in performance when leaving out an upper bound mean average precision of 0.5675. a particular feature (or group of features in case Taking into account the likelihood that some of of part of speech). It turns out that many features the highly ranked false positives are relevant af- can be left out without hurting the performance. ter all (the annotator might have missed a relevant All the features testing simple word characteristics keyword) and considering the difficulty of the task (such as single letter) do not, or only marginally (given the variation in agreement between manual influence the results. Also taking into account the annotators), we argue that the rank-SVM performs importance of sentences (sentence importance), or quite well. the first appearance of a word (first offset and first Jiang et al. (2009) reported MAPs of 0.288 and sentence offset) does not contribute to the results. 0.503 on the ranking of extracted keyphrases from scholarly articles and tags from websites respec- System MAP P@5 P@R tively. Based on these numbers, we could argue rank-SVM 0.399 0.631 0.453 that assigning keywords to folktales is harder than minimum set 0.405 0.631 0.459 reproducing the tags of websites, and slightly eas- ier than reproducing keyphrases from scientific ar- Table 6: Results using the full set of features and ticles. Because of differences in the experimental the minimum set of features (assignment ratio, setup (e.g. size of the training set, features and tf.idf, part of speech and dispersion). Differences system used), it is difficult to make strong claims between systems are statistically significant (t-test, on the difficulty of the task. p < 0.001).

71 Genre (# stories) MAP P@5 P@R Keyword annotators show moderate to substantial Trad. legend (3783) 0.439 0.662 0.494 agreement in extracting the same keywords for a Joke (2793) 0.353 0.599 0.405 story. We showed that a learning to rank method Urban legend (1729) 0.398 0.653 0.459 Riddle (1067) 0.391 0.573 0.415 using features based on assignment ratio, tf.idf, Fairy tale (558) 0.404 0.670 0.477 part of speech and dispersion can be effectively Pers. narrative (514) 0.376 0.593 0.437 used to extract and rank keyword candidates. We Legend (221) 0.409 0.622 0.478 None (122) 0.366 0.602 0.421 believe that this system can be used to suggest Other (113) 0.405 0.648 0.472 highly relevant keyword candidates to human an- All (10900) 0.399 0.631 0.453 notators to speed up the archiving process. Table 7: SVM performance split according to In our evaluation we aimed to reproduce the story genre. Values in bold are significantly dif- manual annotations, but it is unclear whether bet- ferent from the results on the other genres (inde- ter performing systems are actually more helpful pendent t-test, p-value < 0.01) to the user. In an ad hoc retrieval scenario, in which the user issues a single query and reviews a list of retrieved documents, extracted keywords These observations suggest that almost identi- might be used to boost the early precision of the cal results can be obtained using only the features results. However, a user might not even notice assignment ratio, tf.idf, part of speech and disper- a difference when a different keyword extraction sion. The results reported in Table 6 confirm this system is used. Moreover, the more abstract key- (we do note that these results were obtained by op- words which do not appear in the text might be timizing on the test set). more important for the user experience. In fu- ture work we want to get insight in how keywords 6.6 Performance on Folktale Genres contribute to the end user experience. Ideally, the The folktale database contains stories from differ- evaluation should directly measure how useful the ent folktale genres, varying from legends to fairy various keywords are for accessing the collection. tales and jokes. Table 7 lists the performance mea- In this work we considered only extracting key- sures per story genre. Values in bold indicate sig- words from the text we want to annotate. Given nificant differences with the stories from the other the multilingual content of the database this is a genres combined. The performance on traditional limited approach: if the goal of assigning key- legends turns out to be significantly better than words is to obtain a normalized representation of other genres: this could be explained by the fact the stories, this approach will require translation that on average these stories are longer and there- of either the source text (before extraction) or the fore contain more keywords. Similarly, the de- extracted keywords. Even in the monolingual sce- crease can be explained for jokes, which are much nario, the extraction of keywords is limited in deal- shorter on average. Another explanation could be ing with differences in style and word use. Writers that more abstract keywords are used to indicate may use different words or use words in a differ- the type of joke. Interestingly, the riddles, which ent way; ideally the representation based on key- are even shorter than jokes, do not perform sig- words is a normalized representation which closes nificantly worse than the other genres. Personal this semantic gap. In future work we will look into narratives also underperformed in comparison to annotation with keywords from multi-lingual the- the other genres. We cannot readily explain this, sauri combined with free-text keywords extracted but we suspect it may have something to do with from the text itself. Finally, we want to look into the fact that personal narratives are more varied in classification of abstract themes and topics. content and contain more proper names. Acknowledgments 7 Discussion and Conclusion

In this work we analyzed keywords in the context This research was supported by the Folktales as of the Dutch Folktale Database. In this database, Classifiable Texts (FACT) project, part of the on average 15 keywords have been assigned to a CATCH programme funded by the Netherlands story, many of which are single keywords which Organisation for Scientific Research (NWO). appear literally or almost literally in the text itself.

72 References R. Mihalcea and P. Tarau. 2004. Textrank: Bringing order into texts. In Proceedings of EMNLP, vol- R Baeza-Yates and B. Ribeiro-Neto. 2011. Modern In- ume 4, pages 404–411. Barcelona, Spain. formation Retrieval. The Concepts and Technology Behind Search. Addison-Wesley. A. Nenkova and L. Vanderwende. 2005. The impact of frequency on summarization. Microsoft Research, M. Bendersky and W.B. Croft. 2008. Discovering key Redmond, Washington, Tech. Rep. MSR-TR-2005- concepts in verbose queries. In Proceedings of SI- 101. GIR 2008, pages 491–498. S. Osinski and D. Weiss. 2005. A concept-driven al- G. Ercan and I. Cicekli. 2007. Using lexical chains gorithm for clustering search results. Intelligent Sys- for keyword extraction. Information Processing & tems, IEEE, 20(3):48–54. Management, 43(6):1705–1714. Y. Park, R.J. Byrd, and B.K. Boguraev. 2002. Auto- E. Frank, G.W. Paynter, I.H. Witten, C. Gutwin, and matic glossary extraction: beyond terminology iden- C.G. Nevill-Manning. 1999. Domain-specific tification. In Proceedings of COLING 2002, pages keyphrase extraction. In Proceedings of IJCAI-99, 1–7. Association for Computational Linguistics. pages 668—673. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. Christian Plaunt and Barbara A Norgard. 1998. An Association Based Method for Automatic Indexing Stefan Th. Gries. 2008. Dispersions and adjusted fre- with a Controlled Vocabulary. Journal of the Ameri- quencies in corpora. International Journal of Cor- can Society for Information Science and Technology, pus Linguistics, 13(4):403–437. 49(10):888–902.

K. Hammouda, D. Matute, and M. Kamel. 2005. S. Rose, D. Engel, N. Cramer, and W. Cowley. 2010. Corephrase: Keyphrase extraction for document Automatic keyword extraction from individual doc- clustering. Machine Learning and Data Mining in uments. In Michael W. Berry and Jacob Kogan, ed- Pattern Recognition, pages 265–274. itors, Text Mining: Applications and Theory, pages 3–20. John Wiley & Sons. A. Hulth. 2003. Improved automatic keyword extrac- tion given more linguistic knowledge. In Proceed- P.D. Turney. 2000. Learning algorithms for keyphrase ings of EMNLP, volume 10, pages 216–223, Morris- extraction. Information Retrieval, 2(4):303–336. town, NJ, USA. Association for Computational Lin- A. van den Bosch, G.J. Busser, W. Daelemans, and guistics. S Canisius. 2007. An efficient memory-based mor- phosyntactic tagger and parser for Dutch. In F. van X. Jiang, Y. Hu, and H. Li. 2009. A ranking ap- Eynde, P. Dirix, I. Schuurman, and V. Vandeghinste, proach to keyphrase extraction. In Proceedings of editors, Selected Papers of the 17th Computational the 32nd international ACM SIGIR conference on Linguistics in the Netherlands Meeting, pages 99– Research and development in information retrieval, 114, Leuven, Belgium. pages 756–757. ACM. S. Xu, S. Yang, and F.C.M. Lau. 2010. Keyword ex- T. Joachims. 2006. Training Linear SVMs in Lin- traction and headline generation using novel word ear Time. In the 12th ACM SIGKDD international features. Proceedings of the Twenty-Fourth AAAI conference, pages 217–226, New York, NY, USA. Conference on Artificial Intelligence (AAAI-10). ACM. W. Yih, J. Goodman, and V.R. Carvalho. 2006. Find- H. Li. 2011. Learning to Rank for Information Re- ing advertising keywords on web pages. In Proceed- trieval and Natural Language Processing. Synthesis ings of the 15th international conference on World Lectures on Human Language Technology. Morgan Wide Web, pages 213–222. ACM. & Claypool Publishers. K. Zhang, H. Xu, J. Tang, and J. Li. 2006. Keyword F. Liu, D. Pennell, F. Liu, and Y. Liu. 2009a. Unsu- extraction using support vector machine. Advances pervised approaches for automatic keyword extrac- in Web-Age Information Management, pages 85–96. tion using meeting transcripts. In Proceedings of NAACL 2009, pages 620–628. Association for Com- C. Zhang, H. Wang, Y. Liu, D. Wu, Y. Liao, and putational Linguistics. B. Wang. 2008. Automatic keyword extrac- tion from documents using conditional random Z. Liu, P. Li, Y. Zheng, and M. Sun. 2009b. Cluster- fields. Journal of Computational Information Sys- ing to find exemplar terms for keyphrase extraction. tems, 4(3):1169–1180. In Proceedings of EMNLP, pages 257–266. Associ- ation for Computational Linguistics.

O Medelyan and Ian H Witten. 2006. Thesaurus based automatic keyphrase indexing. In JCDL 2006, pages 296–297. ACM.

73 Towards Creating Precision Grammars from Interlinear Glossed Text: Inferring Large-Scale Typological Properties

Emily M. Bender Michael Wayne Goodman Joshua Crowgey Fei Xia Department of Linguistics University of Washington Seattle WA 98195-4340 {ebender,goodmami,jcrowgey,fxia}@uw.edu

Abstract the creation of such precision grammars is time We propose to bring together two kinds of consuming, and the cost of developing them must linguistic resources—interlinear glossed be brought down if they are to be effectively inte- text (IGT) and a language-independent grated into language documentation projects. precision grammar resource—to automat- In this work, we are interested in leveraging ically create precision grammars in the existing linguistic resources of two distinct types context of language documentation. This in order to facilitate the development of precision paper takes the first steps in that direction grammars for language documentation. The first by extracting major-constituent word or- type of linguistic resource is collections of inter- der and case system properties from IGT linear glossed text (IGT), a typical format for dis- for a diverse sample of languages. playing linguistic examples. A sample of IGT from Shona is shown in (1). 1 Introduction (1) Ndakanga ndakatenga muchero Hale et al. (1992) predicted that more than 90% ndi-aka-nga ndi-aka-teng-a mu-chero SBJ.1SG-RP-AUX SBJ.1SG-RP-buy-FV CL3-fruit of the world’s approximately 7,000 languages will ‘I had bought fruit.’ [sna] (Toews, 2009:34) become extinct by the year 2100. This is a crisis not only for the field of linguistics—on track to The annotations in IGT result from deep linguistic lose the majority of its primary data—but also a analysis and represent much effort on the part of crisis for the social sciences more broadly as lan- field linguists. These rich annotations include the guages are a key piece of cultural heritage. The segmentation of the source line into morphemes, field of linguistics has responded with increased the glossing of those individual morphemes, and efforts to document endangered languages. Lan- the translation into a language of broader commu- guage documentation not only captures key lin- nication. The IGT format was developed to com- guistic data (both primary data and analytical pactly display this information to other linguists. facts) but also supports language revitalization ef- Here, we propose to repurpose such data in the au- forts. It must include both primary data collec- tomatic development of further resources. tion (as in Abney and Bird’s (2010) universal cor- The second resource we will be working with pus) and analytical work elucidating the linguistic is the LinGO Grammar Matrix (Bender et al., structures of each language. As such, the outputs 2002; 2010), an open source repository of imple- of documentary linguistics are dictionaries, de- mented linguistic analyses. The Grammar Matrix scriptive (prose) grammars as well as transcribed pairs a core grammar, shared across all grammars and translated texts (Woodbury, 2003). it creates, with a series of libraries of analyses Traditionally, these outputs were printed ar- of cross-linguistically variable phenomena. Users tifacts, but the field of documentary linguistics access the system through a web-based question- has increasingly realized the benefits of producing naire which elicits linguistic descriptions of lan- digital artifacts as well (Nordhoff and Poggeman, guages and then outputs working HPSG (Pol- 2012). Bender et al. (2012a) argue that the docu- lard and Sag, 1994) grammar fragments compat- mentary value of electronic descriptive grammars ible with DELPH-IN (www.delph-in.net) tools can be significantly enhanced by pairing them with based on those descriptions. For present purposes, implemented (machine-readable) precision gram- this system can be viewed as a function which mars and grammar-derived treebanks. However, maps simple descriptions of languages to preci-

74 Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 74–83, Sofia, Bulgaria, August 8 2013. c 2013 Association for Computational Linguistics sion grammar fragments. These fragments are rel- pend on earlier choices) and as the input to the cus- atively modest, yet they relate linguistic strings to tomization script that actually produces the gram- semantic representations (and vice versa) and are mar fragments to spec. The customization sys- ready to be built out to broad coverage. tem distinguishes between choices files which are Thus we ask whether the information encoded complete and consistent (and can be used to cre- by documentary linguists in IGT can be lever- ate working grammar fragments) and those which aged to answer the Grammar Matrix’s question- do not yet have answers to required questions or naire and create a precision grammar fragment give answers which are inconsistent according to automatically. The information required by the the underlying grammatical theory. The ultimate Grammar Matrix questionnaire concerns five dif- goal of the present project is to be able to automat- ferent aspects of linguistic systems: (i) constituent ically create complete and consistent choices files ordering (including the presence/absence of con- on the basis of IGT, and in fact to create complete stituent types), (ii) morphosyntactic systems, (iii) and consistent choices files which take maximal morphosyntactic features, (iv) lexical types and advantage of the analyses stored in the Grammar their instances and (v) morphological rules. In this Matrix customization system, answering not only initial work, we target examples of types (i) and the minimal set of questions required but in fact all (ii): the major constituent word order and the gen- which are relevant and possible to answer based on eral type of case system in a language. The Gram- the information in the IGT. mar Matrix and other related work are described Creating such complete and consistent choices in further in §2. In §3 we present our test data and files is a long-term project, with different ap- experimental set-up. §§4–5 describe our method- proches required for the different types of ques- ology and results for the two tasks, respectively, tions outlined in §1. Bender et al. (2012b) take with further discussion and outlook in §§6–7. some initial steps towards answering the questions which define lexical rules. We envision answering 2 Background and Related Work the questions regarding morphosyntactic features 2.1 The Grammar Matrix through an analysis of the grams that appear on the gloss line, with reference to the GOLD ontology The Grammar Matrix produces precision gram- (Farrar and Langendoen, 2003). The implementa- mars on the basis of description of languages tion of such systems in such a way that they are that include both high-level typological informa- robust to potentially noisy data will undoubtedly tion and more specific detail. Among the for- be non-trivial. The contribution of this paper is mer are aspects (i)–(iii) listed in §1. The third the development of systems to handle one example of these (morphosyntactic features) concerns the each of the questions of types (i) and (ii), namely type and range of grammaticized information that detecting major constituent word order and the un- a language marks in its morphology and/or syn- derlying case system. For the first, we build di- tax. This includes person/number systems (e.g., rectly on the work of Lewis and Xia (2008) (see is there an inclusive/exclusive distinction in non- §2.2). Our experiment can be viewed as an at- singular first person forms?), the range of aspec- tempt to reproduce their results in the context of tual distinctions a language marks, and the range the specific view of word order possibilities devel- of cases (if any) in a language, inter alia. The an- oped in the Grammar Matrix. The second question swers to these questions in turn cause the system (that of case systems) is in some ways more sub- to provide relevant features that the user can ref- tle, requiring not only analysis of IGT instances in erence in providing the more specific information isolation and aggregation of the results, but also elicited by the questionnaire ((iv) and (v) above), identification of particular kinds of IGT instances viz., the definition of both lexical types (e.g., first and comparison across them. person dual exclusive pronouns) and morphologi- cal rules (e.g., nominative case marking on nouns). 2.2 RiPLes The information input by the user to the Gram- mar Matrix questionnaire is stored in a file called The RiPLes project has two intertwined goals. a ‘choices file’. The choices file is used both in The first goal is to create a framework that allows the dynamic definition of the html pages (so that the rapid development of resources for resource- the features available for lexical definitions de- poor languages (RPLs), which is accomplished by

75 ties of languages, including Daume´ III and Camp- bell’s (2007) Bayesian approach to discovering ty- pological implications and Georgi et al.’s (2010) work on predicting (unknown) typological proper- ties by clustering languages based on known prop- erties. Both projects use the typological database WALS (Haspelmath et al., 2008), which has in- formation about 192 different typological proper- ties and about 2,678 different languages (though the matrix is very sparse). This approach is com- plementary to ours, and it remains an interesting Figure 1: Welsh IGT with alignment and projected question whether our results could be improved syntactic structure by bringing in information about other typological properties of the language (either extracted from the IGT or looked up in a typological database). bootstrapping NLP tools with initial seeds created Another strand of related work concerns the col- by projecting syntactic information from resource- lection and curation of IGT, including the ODIN rich languages to RPLs through IGT. Projecting project (Lewis, 2006; Xia and Lewis, 2008), syntactic structures has two steps. First, the words which harvests IGT from linguistics publications in the language line and the translation line are available over the web and TypeCraft (Beermann aligned via the gloss line. Second, the transla- and Mihaylov, 2009), which facilitates the collab- tion line is parsed by a parser for the resource-rich orative development of IGT annotations. TerraL- language and the parse tree is then projected to ing/SSWL2 (Syntactic Structures of the World’s the language line using word alignment and some Languages) has begun a database which combines heuristics as illustrated in Figure 1 (adapted from both typological properties and IGT illustrating Xia and Lewis (2009)).1 Previous work has ap- those properties, contributed by linguists. plied these projected trees to enhance the perfor- Finally, Beerman and Hellan (2011) represents mance of statistical parsers (Georgi et al., 2012). another approach to inducing grammars from IGT, Though the projected trees are noisy, they contain by bringing the hand-built linguistic knowledge enough information for those tasks. sources closer together: On the one hand, their The second goal of RiPLes is to use the au- cross-linguistic grammar resource (TypeGram) in- tomatically created resources to perform cross- cludes a mechanism for mapping from strings lingual study on a large number of languages specifying verb valence and valence-altering lex- to discover linguistic knowledge. For instance, ical rules to sets of grammar constraints. On Lewis and Xia (2008) showed that IGT data en- the other hand, their IGT authoring environment riched with the projected syntactic structure could (TypeCraft) provides support for annotating exam- be used to determine the word order property of a ples with those strings. The approach advocated language with a high accuracy (see §4). Naseem here attempts to bridge the gap between IGT and et al. (2012) use this type of information (in their grammar specification algorithmically, instead. case, drawn from the WALS database (Haspel- math et al., 2008)) to improve multilingual depen- 3 Development and Test Data dency parsing. Here, we build on this aspect of RiPLes and begin to extend it towards the wider Our long-term goal is to produce working gram- range of linguistic phenomena and more detailed mar fragments from IGT produced in documen- classification within phenomena required by the tary linguistics projects. However, in order to Grammar Matrix questionnaire. evaluate the performance of approaches to answer- ing the high-level questions in the Grammar Ma- 2.3 Other Related Work trix questionnaire, we need both IGT and gold- Our work is also situated with respect to attempts standard answers for a reasonably-sized sample of to automatically characterize typological proper- languages. We have constructed development and test data for this purpose on the basis of work done 1The details of the algorithm and experimental results were reported in (Xia and Lewis, 2007). 2http://sswl.railsplayground.net/, accessed 4/25/13

76 Sets of languages DEV1 (n=10) DEV2 (n=10) TEST (n=11) Range of testsuite sizes 16–359 11–229 48–216 Median testsuite size 91 87 76 Language families Indo-European (4), Niger- Indo-European (3), Indo-European (2), Afro-Asiatic, Congo (2), Afro-Asiatic, Dravidian (2), Algic, Austro-Asiatic, Austronesian, Japanese, Nadahup, Creole, Niger-Congo, Arauan, Carib, Karvelian, Sino-Tibetan Quechuan, Salishan N. Caucasian, Tai-Kadai, Isolate

Table 1: Language families and testsuites sizes (in number of grammatical examples) by students in a class that uses the Grammar Ma- formation. The parameter most closely relevant to trix (Bender, 2007). In this class, students work the present work is Order of Words in a Sentence with descriptive resources for languages they are (Dryer, 2011). For this parameter, Lewis and Xia typically not familiar with to create testsuites (cu- tested their method in 97 languages and found that rated collections of grammatical and ungrammat- their system had 99% accuracy provided the IGT ical examples) and Grammar Matrix choices files. collections had at least 40 instances per language. Later on in the class, the students extend the gram- The Grammar Matrix’s word order questions mar fragments output by the customization system differ somewhat from the typological classifi- to handle a broader fragment of the language. Ac- cation that Lewis and Xia (2008) were using. cordingly, the testsuites cover phenomena which Answering the Grammar Matrix questionnaire go beyond the customization system. amounts to more than making a descriptive state- Testsuites for grammars, especially in their ment about a language. The Grammar Matrix cus- early stages of development, require examples that tomization system translates collections of such are simple (isolating the phenomena illustrated by descriptive statements into working grammar frag- the examples to the extent possible), built out of ments. In the case of word order, this most di- a small vocabulary, and include both grammati- rectly effects the number and nature of phrase cal and ungrammatical examples (Lehmann et al., structure rules included in the output grammar, but 1996). The examples included in descriptive re- can also interact with other aspects of the gram- sources often don’t fit these requirements exactly. mar (e.g., the treatment of argument optionality). As a result, the data we are working with include More broadly, specifying the word order system examples invented by the students on the basis of of a grammar determines both grammaticality (ac- the descriptive statements in their resources.3 cepting some strings, ruling out others) and, for In total, we have testsuites and associated the fixed word orders at least, aspects of the map- choices files for 31 languages, spanning 17 lan- ping of syntactic to semantic arguments. guage families (plus one creole and one language Lewis and Xia (2008), like Dryer (2011), gave isolate). The most well-represented family is the six fixed orders of S, O and V plus “no dom- Indo-European, with nine languages. We used 20 inant order”. In contrast, the Grammar Matrix languages, in two dev sets, for algorithm develop- distinguishes Free (pragmatically constrained), V- ment (including manual error analysis), and saved final, V-initial, and V2 orders, in addition to the 11 languages as a held-out test set to verify the six fixed orders. It is important to note that the generalizability of our approach. Table 1 lists the relationship between the word order type of a lan- language families and the range of testsuite sizes guage and the actual orders attested in sentences for each of these sets of languages. can be somewhat indirect. For a fixed word order language, we would expect the order declared as 4 Inferring Word Order its type to be the most common in running text, but not the only type available. English, for exam- Lewis and Xia (2008) show how IGT from ODIN ple, is an SVO language, but several constructions (Lewis, 2006) can be used to determine, with high allow for other orders, including subject-auxiliary accuracy, the word order properties of a language. inversion, so-called topicalization, and others: They identify 14 typological parameters related to (2) Did Kim leave? word order for which WALS (Haspelmath et al., (3) The book, Kim forgot. 2008) or other typological resources provide in- In a language with more word order flexibility in 3Such examples are flagged in the testsuites’ meta-data. general, there may still be a preferred word order

77 which is the most common due to pragmatic or us three axes: one for the tendency to exhibit VS other constraints. Users of the Grammar Matrix or SV order, another for VO or OV order, and an- are advised to choose one of the fixed word orders other for OS or SO order. By counting the ob- if the deviations from that order can generally be served word orders in the IGT examples, we can accounted for by specific syntactic constructions, place the language in this three-dimensional space. and a freer word order otherwise. Figure 4.1 depicts this space with the positions of The relationship between the correct word or- canonical word orders.4 The canonical word order der choice for the Grammar Matrix customization positions are those found under homogeneous ob- system and the distribution of actual token word servations. For example, the canonical position for orders in our development and test data is affected SOV order is when 100% of the sentences exhibit by another factor, related to Lewis and Xia’s ‘IGT SO, OV, and SV orders; and the canonical position bias’ which we dub ‘testsuite bias’. The collec- for Free word order is when each observed order tions of IGT we are using were constructed as test- occurs with equal frequency to its opposite order suites for grammar engineering projects and thus (on the same axis; e.g. VO and OV). We select the comprise examples selected or constructed to il- underlying word order by finding which canoni- lustrate specific grammatical properties in a test- cal word order position has the shortest Euclidean ing regime where one example is enough to repre- distance to the observed word order position. sent each sentence type of interest. Therefore, they When a language is selected as Free word or- do not represent a natural distribution of word or- der, we employ a secondary heuristic to decide if der types. For example, the testsuite authors may it is actually V2 word order. The V2 order cannot show the full range of possible word orders in the be easily recognized only with the binary word or- word order section of the testsuite and then default ders, so it is not given a unique point in the three- to one particular choice for other portions (those dimensional space. Rather, we try to recognize it illustrating e.g., case systems or negation). by comparing the ternary orders. A Free-order lan- guage is reclassified as V2 if SVO and OVS occur 4.1 Methodology more frequently than SOV and OSV.5

Our first stpes mirror the RiPLes approach, pars- OVS ing parse the English translation of each sentence OSV OS and projecting the parsed structure onto the source VOS language line. Functional tags, such as SBJ and OV OBJ, are added to the NP nodes on the English V-final VS side based on our knowledge of English word or- Free/V2 SV der and then carried over to the source language V-initial VO side during the projection of parse trees. The trees SOV are then searched for any of ten patterns: SOV, SO SVO, OSV, OVS, VSO, VOS, SV, VS, OV, and VSO VO. The six ternary patterns match when both ver- SVO bal arguments are present in the same clause. The four binary patterns are for intransitive sentences Figure 2: Three axes of basic word order and the or those with dropped arguments. These ten pat- positions of canonical word orders. terns make up the observed word orders. Given our relatively limited data set (each lan- guage is one data point), we present an initial 4.2 Results approach to determining underlying word order Table 2 shows the results we obtained for our dev based on heuristics informed by general linguis- and test sets. For comparison, we use a most- tic knowledge. We compare the distribution of ob- 4Of the eight vertices of this cube, six represent canoni- served word orders to distributions we expect to cal word orders the other two impossible combinations: The see for canonical examples of underlying word or- vertex for (SV, VO, OS) (e.g.) has S both before and after O. ders. We accomplish this by first deconstructing 5The VOS and VSO patterns are excluded from this com- parison, since they can go either way—there may be un- the ternary observed-word-orders into binary pat- aligned constituents (i.e. not a S, O, or V) before the verb terns (the four above plus SO and OS). This gives which are ignored by our system.

78 frequent-type baseline, selecting SOV for all lan- Case Case grams present system NOM ∨ ACCERG ∨ ABS guages, based on Dryer’s (2011) survey. We get none high accuracy for DEV1, low accuracy for DEV2, nom-acc X erg-abs and moderate accuracy for TEST, but all are sig- X split-erg XX nificantly higher than the baseline. (conditioned on V)

Dataset Inferred WO Baseline DEV1 0.900 0.200 Table 3: GRAM case system assignment rules DEV2 0.500 0.100 TEST 0.727 0.091 (O). Among languages which make use of case, Table 2: Accuracy of word-order inference the most common alignment type is a nominative- accusative system (Comrie 2011a,b). In this Hand analysis of the errors in the dev sets type, S takes the same kind of marking as A.6 show that some languages fall victim to the test- The Grammar Matrix case library provides nine suite bias, such as Russian, Quechua, and Tamil. options, including none, nominative-accusative, All of these languages have Free word order, but ergative-absolutive (S marked like O), tripartite (S, our system infers SVO for Russian and SOV for A and O all distinct) and several more intricate Quechua and Tamil, because the authors of the types. For example, in a language with one type testsuites used one order significantly more than of split case system the alignment is nominative- the others. Similarly, the Free word order lan- accusative in non-past tense clauses, but ergative- guage Nishnaabemwin is inferred as V2 because absolutive in past tense ones. there are more SVO and OVS patterns given than As with major constituent word order, the con- others. We also see errors due to misalignment straints implementing a case system in a grammar from RiPLes’ syntactic projection. The VSO lan- serve to model both grammaticality and the map- guage Welsh is inferred as SVO because the near- ping between syntactic and semantic arguments. ubiquitous sentence-initial auxiliary doesn’t align Here too, the distribution of tokens may be some- to the main verb of the English translation. thing other than a pure expression of the case alignment type. Sources of noise in the distri- 5 Inferring Case Systems bution include: argument optionality (e.g., tran- Case refers to linguistic phenomena in which the sitives with one or more covert arguments), ar- form of a noun phrase (NP) varies depending on gument frames other than simple intransitives or the function of the NP in a sentence (Blake, 2001). transitives, and quirky case (verbs that use a non- The Grammar Matrix’s case library (Drellishak, standard case frame for their arguments, such as 2009) focuses on case marking of core arguments the German verb helfen which selects a dative ar- of verbs. Specifying a grammar for case involves gument, though the language’s general system is both choosing the high-level case system to be nominative-accusative (Drellishak, 2009)). modeled as well as associating verb types with 5.1 Methodology case frames and defining the lexical items or lex- We explore two possible methodologies for infer- ical rules which mark the case on the NPs. Here, ring case systems, one relatively na¨ıve and one we focus on the high-level case system question more elaborate, and compare them to a most- as it is logically prior, and in some ways more in- frequent-type baseline. Method 1, called GRAM, teresting than the lexical details: Answering this considers only the gloss line of the IGT and as- question requires identifying case frames of verbs sumes that it complies with the Leipzig Glossing in particular examples and then comparing across Rules (Bickel et al., 2008). These rules not only those examples, as described below. prescribe formatting aspects of IGT but also pro- The high-level case system of a language con- vide a set of licensed ‘grams’, or tags for grammat- cerns the alignment of case marking between tran- ical properties that appear in the gloss line. GRAM sitive and intransitive clauses. The three ele- scans for the grams associated with case, and as- ments in question are the subjects of intransi- signs case systems according to Table 3. tives (dubbed S), the subjects (or agent-like ar- This methodology is simple to implement and guments) of transitives (dubbed A) and the ob- jects (or patient-like arguments) of intransitives 6English’s residual case system is of this type.

79 expected to work well given Leipzig-compliant the verb. These could be distinguished by looking IGT. However, since it does not model the func- for intransitive verbs that appear more than once in tion of case, it is dependent on the IGT authors’ the data and checking whether their S arguments choice of gram symbols, and may be confused by all have consistently A or O marking. either alternative case names (e.g., SBJ and OBJ for This system is agnostic as to the spelling of the nominative and accusative or LOC for ergative in case grams. By relying on more analysis of the languages where it is homophonous with the loca- IGT than GRAM, it also introduces new kinds of tive case) or by other grams which collide with the brittleness. Recognizing the difference between case name-space (such as NOM for nominalizer). grams being present and (virtually) absent makes It also only handles four of the nine case systems the system susceptible to noise. (albeit the most frequent ones). 5.2 Results Method 2, called SAO, is more theoretically motivated, builds on the RiPLes approach used Table 4 shows the results for the inference of case- in inferring word order, and is designed to be marking systems. Currently GRAM performs best, robust to idiosyncratic glossing conventions. In but both methods generally perform better than this methodology, we first identify the S, A and the baseline. The better performance of GRAM O arguments by projecting the information from is expected, given the small size and generally the parse of the English translation (including Leipzig-compliant glossing of our data sets. In the function tags) to the source sentence (and its future work, we plan to incorporate data from glosses). We discard all items which do not appear ODIN, which is likely less consistently annotated to be simple transitive or intransitive clauses with but more voluminous, and we expect SAO to be all arguments overt, and then collect all grams for more robust than GRAM to this kind of data. each argument type (from all words within in the Dataset GRAMSAO Baseline NP, including head nouns as well as determiners DEV1 0.900 0.700 0.400 DEV2 0.900 0.500 0.500 and adjectives). While there are many grammati- TEST 0.545 0.545 0.455 cal features that can be marked on NPs (such as number, definiteness, honorifics, etc.), the only Table 4: Accuracy of case-marking inference ones that should correlate strongly with grammat- ical function are case-marking grams. Further- We find that GRAM is sometimes able to do well more, in any given NP, while case may be multi- when RiPLes gives alignment errors. For exam- ply marked, we only expect one type of case gram ple, Old Japanese is a NOM-ACC language, but the to appear. We thus assume that the most frequent case-marking grams (associated to postpositions) gram for each argument type is a case marker (if are not aligned to the NP arguments, so SAO is not there are any) and assign the case system accord- able to judge their distribution. On the other hand, ing to the following rules, where Sg,Og and Ag de- SAO prevails when non-standard grams are used, note the most frequent grams associated with these such as the NOM-ACC language Hupdeh, which is argument positions, respectively: annotated with SUBJ and OBJ grams. This comple- mentarity suggests scope for system combination, • Nominative-accusative: Sg=Ag,Sg6=Og which we leave to future work. • Ergative-absolutive: Sg=Og,Sg6=Ag • No case: Sg=Ag=Og, or Sg6=Ag6=Og and Sg, 6 Discussion and Future Work Ag,Og also present on each of the other ar- gument types Our initial results are promising, but also show • Tripartite: Sg6=Ag6=Og, and Sg,Ag,Og (vir- remaining room for improvement. Error analysis tually) absent from the other argument types suggests two main directions to pursue: • 6= 6= Split-S: Sg Ag Og, and Ag and Og are both Overcoming testsuite bias In both the word or- present in the list for the S argument type der and case system tasks, we see the effect of Here, we’re using Split-S to stand in for both testsuite bias on our system results. The testsuites Split-S and Fluid-S. These are both systems where for freer word order languages can be artificially some S arguments are marked like A, and some dominated by a particular word order that the test- like O. In Split-S, which is taken depends on the suite author found convenient. Further, the re- verb. In Fluid-S, it depends on the interpretation of stricted vocabulary used in testsuites, combined

80 with a general preference for animates as subjects, been able to beat the most-frequent-type baseline. leads to stems and certain grams potentially being Better handling of these unaligned words is misidentified as case markers. a non-trivial task, and will require bringing in We believe that these aspects of testsuite bias sources of knowledge other than the structure of are not typical of our true target input data, viz., the English translation. The information we have the larger collections of IGT created by field to leverage in this regard comes mainly from the projects. On the other hand, there may be other as- gloss line and from general linguistic/typological pects of testsuites which are simplifying the prob- knowledge which can be added to the algorithm. lem and to which our current methods are over- That is, there are types of grams which are canon- fitted. To address these issues, we intend to look ically associated with verbal projections and types to larger datasets in future work, both IGT collec- of grams canonically associated with nominal pro- tions from field projects and IGT from ODIN. For jections. When these grams occur on unaligned the field projects, we will need to construct choices elements, we can hypothesize that the elements files. For ODIN, we can search for data from the are auxiliaries and case-marking adpositions re- languages we already have choices files for. spectively. Further typological considerations will As we move from testsuites to test corpora motivate heuristics for modifying tree structures (e.g., narratives collected in documentary linguis- based on these classifications. tics projects), we expect to find different distribu- Other directions for future work include extend- tions of word order types. Our current methodol- ing this methodology to other aspects of grammat- ogy for extracting word order is based on idealized ical description, including additional high-level locations in our word order space for each strict systems (e.g., argument optionality), discovering word order type. Working with naturally occurring the range of morphosyntactic features active in a corpora it should be possible to gain a more em- language, and describing and populating lexical pirically based understanding of the relationship types (e.g., common nouns with a particular gen- between underlying word order and sentence type der). Once we are able to answer enough of the distributions. It will be particularly interesting to questionnaire that the customization system is able see how stable these relationships are across lan- to output a grammar, interesting options for de- guages with the same underlying word order type tailed evaluation will become available. In par- but from different language families and/or with ticular, we will be able to parse the IGT (includ- differences in other typological characteristics. ing held-out examples) with the resulting gram- mar, and then compare the resulting semantic rep- Better handling of unaligned words The other resentations to those produced by parsing the En- main source of error is words that remain un- glish translations with tools that produce compara- aligned in the projected syntactic structure and ble semantic representations for English (using the thus only loosely incorporated into the syntax English Resource Grammar (Flickinger, 2000)). trees. This includes items like case marking adpo- sitions in Japanese, which are unaligned because 7 Conclusions and Future Work there is no corresponding word in English, and auxiliaries in Welsh, which are unaligned when In this paper we have presented an approach to the English translation doesn’t happen to use an combining two types of linguistic resources—IGT, auxiliary. In the former case, our SAO method as produced by documentary linguists and a cross- for case system extraction doesn’t include the case linguistic grammar resource supporting preci- grams in the set of grams for each NP. In the latter, sion parsing and generation—to create language- the word order inference system is unable to pick specific resources which can help enrich language up on the VSO order represented as Aux+S+[VP]. documentation and support language revitaliza- Simply fixing the attachment of the auxiliaries will tion efforts. In addition to presenting the broad vi- not be enough in this case, as the word order infer- sion of the project, we have reported initial results ence algorithm will need to be extended to han- in two case studies as a proof-of-concept. Though dle auxiliaries, but fixing the alignment is the first there is still a ways to go, we find these initial re- step. Alignment problems are also the main reason sults a promising indication of the approach’s abil- our initial attempts to extract information about ity to assist in the preservation of the key type of the order of determiners and nouns haven’t yet cultural heritage that is linguistic systems.

81 Acknowledgments French verbal morphology. In LSA Annual Meeting Extended Abstracts 2012. We are grateful to the students in Ling 567 at the University of Washington who created the test- Emily M. Bender. 2007. Combining research and suites and choices files used as development and pedagogy in the development of a crosslinguistic test data in this work and to the three anonymous grammar resource. In Tracy Holloway King and Emily M. Bender, editors, Proceedings of the GEAF reviewers for helpful comments and discussion. 2007 Workshop, Stanford, CA. CSLI Publications. This material is based upon work supported by the National Science Foundation under Grant Balthasar Bickel, Bernard Comrie, and Martin Haspel- No. BCS-1160274. Any opinions, findings, and math. 2008. The Leipzig glossing rules: Con- ventions for interlinear morpheme-by-morpheme conclusions or recommendations expressed in this glosses. Max Planck Institute for Evolutionary An- material are those of the author(s) and do not nec- thropology and Department of Linguistics, Univer- essarily reflect the views of the National Science sity of Leipzig. Foundation. Barry J. Blake. 2001. Case. Cambridge University Press, Cambridge, second edition. References Bernard Comrie. 2011a. Alignment of case marking of Steven Abney and Steven Bird. 2010. The human full noun phrases. In Matthew S. Dryer and Martin language project: building a universal corpus of the Haspelmath, editors, The World Atlas of Language world’s languages. In Proceedings of the 48th An- Structures Online. Max Planck Digital Library, Mu- nual Meeting of the Association for Computational nich. Linguistics, ACL ’10, pages 88–97, Stroudsburg, PA, USA. Association for Computational Linguis- Bernard Comrie. 2011b. Alignment of case marking of tics. pronouns. In Matthew S. Dryer and Martin Haspel- math, editors, The World Atlas of Language Struc- Dorothee Beerman and Lars Hellan. 2011. Induc- tures Online. Max Planck Digital Library, Munich. ing grammar from IGT. In Proceedings of the 5th Language and Technology Conference: Human Lan- Hal Daume´ III and Lyle Campbell. 2007. A Bayesian guage Technologies as a Challenge for Computer model for discovering typological implications. In Science and Linguistics (LTC 2011). Proceedings of the 45th Annual Meeting of the As- Dorothee Beermann and Pavel Mihaylov. 2009. Type- sociation of Computational Linguistics, pages 65– Craft: Linguistic data and knowledge sharing, open 72, Prague, Czech Republic, June. Association for access and linguistic methodology. Paper presented Computational Linguistics. at the Workshop on Small Tools in Cross-linguistic Research, University of Utrecht. The Netherlands. Scott Drellishak. 2009. Widespread But Not Uni- versal: Improving the Typological Coverage of the Emily M. Bender, Dan Flickinger, and Stephan Oepen. Grammar Matrix. Ph.D. thesis, University of Wash- 2002. The grammar matrix: An open-source starter- ington. kit for the rapid development of cross-linguistically consistent broad-coverage precision grammars. In Matthew S. Dryer. 2011. Order of subject, object and John Carroll, Nelleke Oostdijk, and Richard Sut- verb. In Matthew S. Dryer and Martin Haspelmath, cliffe, editors, Proceedings of the Workshop on editors, The World Atlas of Language Structures On- Grammar Engineering and Evaluation at the 19th line. Max Planck Digital Library, Munich. International Conference on Computational Lin- guistics, pages 8–14, Taipei, Taiwan. Scott Farrar and Terry Langendoen. 2003. A linguistic ontology for the semantic web. Glot International, Emily M. Bender, Scott Drellishak, Antske Fokkens, 7:97–100. Laurie Poulson, and Safiyyah Saleem. 2010. Gram- mar customization. Research on Language & Com- Dan Flickinger. 2000. On building a more efficient putation, pages 1–50. 10.1007/s11168-010-9070-1. grammar by exploiting types. Natural Language Emily M. Bender, Sumukh Ghodke, Timothy Baldwin, Engineering, 6 (1) (Special Issue on Efficient Pro- and Rebecca Dridan. 2012a. From database to tree- cessing with HPSG):15 – 28. bank: Enhancing hypertext grammars with grammar engineering and treebank search. In Sebastian Nord- Ryan Georgi, Fei Xia, and William Lewis. 2010. hoff and Karl-Ludwig G. Poggeman, editors, Elec- Comparing language similarity across genetic and tronic Grammaticography, pages 179–206. Univer- typologically-based groupings. In Proceedings of sity of Hawaii Press, Honolulu. the 23rd International Conference on Computa- tional Linguistics (Coling 2010), pages 385–393, Emily M. Bender, David Wax, and Michael Wayne Beijing, China, August. Coling 2010 Organizing Goodman. 2012b. From IGT to precision grammar: Committee.

82 Ryan Georgi, Fei Xia, and William Lewis. 2012. Im- Fei Xia and William D. Lewis. 2007. Multilin- proving dependency parsing with interlinear glossed gual structural projection across interlinear text. text and syntactic projection. In Proceedings of In Proc. of the Conference on Human Language COLING 2012: Posters, pages 371–380, Mumbai, Technologies (HLT/NAACL 2007), pages 452–459, India, December. Rochester, New York.

Ken Hale, Michael Krauss, Lucille J. Wata- Fei Xia and William D. Lewis. 2008. Repurposing homigie, Akira Y. Yamamoto, Colette Craig, theoretical linguistic data for tool development and LaVerne Masayesva Jeanne, and Nora C. England. search. In Proceedings of the Third International 1992. Endangered languages. Language, 68(1):pp. Joint Conference on Natural Language Processing, 1–42. pages 529–536, Hyderabad, India.

Martin Haspelmath, Matthew S. Dryer, David Gil, and Fei Xia and William Lewis. 2009. Applying NLP Bernard Comrie, editors. 2008. The World Atlas technologies to the collection and enrichment of lan- of Language Structures Online. Max Planck Digital guage data on the web to aid linguistic research. In Library, Munich. http://wals.info. Proceedings of the EACL 2009 Workshop on Lan- guage Technology and Resources for Cultural Her- Sabine Lehmann, Stephan Oepen, Sylvie Regnier- itage, Social Sciences, Humanities, and Education Prost, Klaus Netter, Veronika Lux, Judith Klein, (LaTeCH – SHELT&R 2009), pages 51–59, Athens, Kirsten Falkedal, Frederik Fouvry, Dominique Esti- Greece, March. Association for Computational Lin- val, Eva Dauphin, Herve` Compagnion, Judith Baur, guistics. Lorna Balkan, and Doug Arnold. 1996. TSNLP: Test suites for natural language processing. In Pro- ceedings of the 16th conference on Computational linguistics - Volume 2, COLING ’96, pages 711– 716, Stroudsburg, PA, USA. Association for Com- putational Linguistics.

William D. Lewis and Fei Xia. 2008. Automati- cally identifying computationally relevant typolog- ical features. In Proceedings of the Third Interna- tional Joint Conference on Natural Language Pro- cessing, pages 685–690, Hyderabad, India.

William D. Lewis. 2006. ODIN: A model for adapting and enriching legacy infrastructure. In Proceedings of the e-Humanities Workshop, Held in cooperation with e-Science, Amsterdam.

Tahira Naseem, Regina Barzilay, and Amir Globerson. 2012. Selective sharing for multilingual dependency parsing. In Proceedings of the 50th Annual Meet- ing of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 629–637, Jeju Island, Korea, July. Association for Computational Linguistics.

Sebastian Nordhoff and Karl-Ludwig G. Poggeman, editors. 2012. Electronic Grammaticography. Uni- versity of Hawaii Press, Honolulu.

Carl Pollard and Ivan A. Sag. 1994. Head-Driven Phrase Structure Grammar. Studies in Contempo- rary Linguistics. The University of Chicago Press and CSLI Publications, Chicago, IL and Stanford, CA.

Carmela Toews. 2009. The expression of tense and as- pect in Shona. Selected Proceedings of the 39th An- nual Converence on African Linguistics, pages 32– 41.

Tony Woodbury. 2003. Defining documentary lin- guistics. Language documentation and description, 1(1):35.

83 Using Comparable Collections of Historical Texts for Building a Diachronic Dictionary for Spelling Normalization

Marilisa Amoia Jose Manuel Martinez Saarland University Saarland University [email protected] [email protected]

Abstract et al., 2011b) for German and many others). Such In this paper, we argue that compara- resources are generally annotated with shallow ble collections of historical written re- metadata (e.g. year of publication, author, ge- sources can help overcoming typical chal- ographical location) for allowing fast retrieval. lenges posed by heritage texts enhanc- However, the annotation of richer linguistic and ing spelling normalization, POS-tagging semantic information still poses a series of chal- and subsequent diachronic linguistic anal- lenges that have to be overcome, such as (i) yses. Thus, we present a comparable cor- the noise introduced by deviant linguistic data pus of historical German recipes and show (spelling/orthography variation, lack of sentence how such a comparable text collection to- boundaries, etc.) typical of this kind of material, gether with the application of innovative due to the lack of standardized writing conventions MT inspired strategies allow us (i) to ad- in terms of words and punctuation and hence (ii) dress the word form normalization prob- the higher error rates obtained when applying stan- lem and (ii) to automatically generate a dard NLP methods. diachronic dictionary of spelling variants. Further, standardization of spelling variation in Such a diachronic dictionary can be used historical texts can be broken down at least into both for spelling normalization and for ex- two subproblems: tracting new ”translation” (word forma- 1. the old word forms often differ from the mod- tion/change) rules for diachronic spelling ern orthography of the same items. Consider, variants. Moreover, our approach can be for instance, the diachronic variants of the applied virtually to any diachronic collec- third person singular of present tense of the tion of texts regardless of the time span verb werden in German (which means ‘be- they represent. A first evaluation shows come’ as full verb, or is used as auxiliary that our approach compares well with verb to build the future): wirt, wirdt, wirdet state-of-art approaches. vs wird; (Piotrowski, 2012) and 1 Introduction 2. the denomination of certain objects may re- The study of heritage documents has been one sult completely different from that used in of the regular sources of knowledge in the Hu- the modern language due to historical reasons manities, specially in history-related disciplines. (e.g. adoption of foreign language terms, The last years have witnessed an increased interest semantic shift). Consider, as an example, in approaches combining NLP and corpus-based the German historical/modern variants of the techniques in the Humanities (Piotrowski, 2012) word lemon (e.g. Limonie/Zitrone) or of the as they can provide new insights and/or a more word woman (e.g. Weib/Frau). consistent and reliable account of findings. Previous approaches to spelling normalization Until recently, research efforts have been fo- of historical texts have focused on the first sub- cused on building diachronic corpora (e.g. Old problem. Two main strategies that have been ap- Bailey Online project (Hitchcock et al., 2012) plied: and its follow-up, the Old Bailey Corpus (Huber, 2007), the Bonn Corpus of Early New High Ger- • a rule based strategy, in which the transla- man (Diel et al., 2002) or the GerManC (Scheible tion of historical variants into modern forms

84 Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 84–89, Sofia, Bulgaria, August 8 2013. c 2013 Association for Computational Linguistics is performed on the ground of manually writ- variants. Section 4 shows the results of a prelimi- ten or semi-automatically gathered rules (cf. nary evaluation. Finally, in Section 5 we conclude (Pilz et al., 2008), (Bollmann et al., 2011)); by discussing some final remarks.

• a string similarity strategy, in which a semi- 2 The Historical Comparable Corpus of automatic attempt is made to link histori- German Recipes cal variants with modern dictionary entries The text collection encoded in our corpus spans following string similarity (cf. (Giusti et two hundred years and includes samples from 14 al., 2007), (Kunstmann and Stein, 2007), cook books written in German between 1569 and (Dipper, 2010), (Hendrickx and Marquilhas, 1729. The core of the recipe corpus was compiled 2011), (Gotscharek et al., 2011)) or pho- as part of a PhD work in the field of Translation netic conflation strategies (cf. (Koolen et al., Studies (cf. (Wurm, 2007)). This corpus has been 2006), (Jurish, 2008) ). aligned resulting into two comparable corpora: These approaches have the disadvantage of end- • a historical comparable dataset aligned at ing up relying on a time-specific dictionary of vari- recipe level providing multiple versions of ants, e.g. they can cope with variants realized in the same dish across the time span of the core texts stemming from the same period of time for corpus; which they have been created but may result inap- propriate for texts belonging to other time spans. • a contemporary comparable dataset provid- Moreover, to our knowledge, there is currently ing contemporary German versions for each no approach to spelling normalization that can ad- recipe. dress successfully the second subproblem stated In order to produce the historical comparable com- above – the recognition of paraphrastic variations ponent we proceeded in the following way: realized as completely different strings or consist- ing of semantic shifts. • first, we classified the core recipes by main As it has been often noted, the problem of stan- ingredient and cooking method (e.g. chicken, dardizing diachronic variants can be understood as roast). These two parameters form the crite- a translation operation, where instead of translat- ria to consider the recipes aligned, then; ing between two different languages, translation • we collected as many as possible diachronic takes place between two diachronic varieties of the versions/variants of the same recipe by also same language. Inspired by experiments done for searching online resources providing collec- interlinguistic translation (Rapp et al., 2012), the tions of historical texts. idea is to use diachronic comparable corpora to automatically produce a dictionary of diachronic The historical component of the corpus (core and spelling variants even including semantic shifts, comparable) contains a total of 430 recipes and regardless of the historical variants at stake. about 45.000 tokens. This dataset constitutes the In short, we first build a comparable histori- object of study for subsequent research, provid- cal corpus made up of recipe repertoires published ing a representative sample of German during the in the German language during the Early Modern Early Modern Age in this specific domain. More- Age along with a contemporary comparable cor- over, language and genre evolution can be traced pus. Second, we address the problem of recog- thanks to its comparable nature. nizing and translating different variants by relay- Regarding the compilation of the contemporary ing on MT techniques based on string similarity as German comparable corpus, we collected a set of well as on semantic similarity measures. Finally, recipes belonging to the same register but repre- we automatically extract a diachronic dictionary senting contemporary German language. These of spelling and semantic variants which also pro- recipes were collected from Internet sources and vides a canonical form. filtered by geographical criteria (only the ones cat- This paper is organized as follows. Section egorized as belonging to the cuisine of German 2 presents the comparable corpus of German speaking regions were selected). The corpus con- recipes. Section 3 describes the approach used for tains around 1500 recipes and over 500.000 to- generating the dictionary of diachronic spelling kens, which have been also aligned with their

85