Toponym Resolution in Scientific Papers
Total Page:16
File Type:pdf, Size:1020Kb
SemEval-2019 Task 12: Toponym Resolution in Scientific Papers Davy Weissenbachery, Arjun Maggez, Karen O’Connory, Matthew Scotchz, Graciela Gonzalez-Hernandezy yDBEI, The Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA zBiodesign Center for Environmental Health Engineering, Arizona State University, Tempe, AZ 85281, USA yfdweissen, karoc, [email protected] zfamaggera, [email protected] Abstract Disambiguation of toponyms is a more recent task (Leidner, 2007). We present the SemEval-2019 Task 12 which With the growth of the internet, the public adop- focuses on toponym resolution in scientific ar- ticles. Given an article from PubMed, the tion of smartphones equipped with Geographic In- task consists of detecting mentions of names formation Systems and the collaborative devel- of places, or toponyms, and mapping the opment of comprehensive maps and geographi- mentions to their corresponding entries in cal databases, toponym resolution has seen an im- GeoNames.org, a database of geospatial loca- portant gain of interest in the last two decades. tions. We proposed three subtasks. In Sub- Not only academic but also commercial and open task 1, we asked participants to detect all to- source toponym resolvers are now available. How- ponyms in an article. In Subtask 2, given to- ponym mentions as input, we asked partici- ever, their performance varies greatly when ap- pants to disambiguate them by linking them plied on corpora of different genres and domains to entries in GeoNames. In Subtask 3, we (Gritta et al., 2018). Toponym disambiguation asked participants to perform both the de- tackles ambiguities between different toponyms, tection and the disambiguation steps for all like Manchester, NH, USA vs. Manchester, UK toponyms. A total of 29 teams registered, (Geo-Geo ambiguities), and between toponyms and 8 teams submitted a system run. We and other entities, such as names of people or summarize the corpus and the tools created daily life objects (Geo-NonGeo ambiguities). Ad- for the challenge. They are freely available at https://competitions.codalab. ditional linguistic challenges during the resolution org/competitions/19948. We also an- step may be metonymic usage of toponyms, “91% alyze the methods, the results and the errors of the US didn’t vote for either Hilary or Trump” made by the competing systems with a focus (a country does not vote, thus the toponym refers on toponym disambiguation. to the people living in the country), elliptical con- structions, “Lakeview and Harrison streets” (the 1 Introduction phrase refers to two street names Lakeview street Toponym resolution, also known as geoparsing, and Harrison street), or when the context simply geo-grounding or place name resolution, aims does not provide enough evidences for the resolu- to assign geographic coordinates to all location tion. names mentioned in documents. Toponym res- Although significant progress has been made in olution is usually performed in two independent the last decade on toponym resolution, it is still steps. First, toponym detection or geotagging, difficult to determine precisely the current state- where the span of place names mentioned in a doc- of-the-art performances (Leidner and Lieberman, ument is noted. Second, toponym disambiguation 2011). As emphasized by several authors (To- or geocoding, where each name found is mapped bin et al., 2010; Speriosu, 2013; Weissenbacher to latitude and longitude coordinates correspond- et al., 2015; Gritta et al., 2018; Karimzadeh and ing to the centroid of its physical location. To- MacEachren, 2019), the main obstacle is that few ponym detection has been extensively studied in corpora of large size exist or are freely available. named entity recognition: location names were Consequently, researchers create their own (lim- one of the first classes of named entities to be ited) corpora to evaluate their system, with the detected in text (Piskorski and Yangarber, 2013). known drawbacks and biases that this implies. 907 Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 907–916 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics Moreover, one corpus is not sufficient to evaluate taking the set of candidates that are the closest a toponym resolver thoroughly, as the domain of a to each other (Leidner, 2007). A recent heuris- corpus strongly impacts the performance of a re- tic computes from Wikipedia a network express- solver. A disambiguation strategy can be optimal ing important toponyms and their semantic rela- on one domain and damaging on another. In (Spe- tion with other entities. The network is then used riosu, 2013), Speriosu illustrates that toponyms to disambiguate jointly all toponyms in a doc- occurring in historical literature will tend to re- ument (Hoffart and Weikum, 2013; Spitz et al., solve within a local vicinity, whereas toponyms 2016). The last strategy is less frequently used occurring in international press news refer to the as it depends on metadata describing the doc- most prominent places by default. Otherwise ad- uments where toponyms are mentioned. These ditional information is provided to help the reso- metadata are of various kinds, but they all indi- lution (ex. Paris, the city in Texas). cate, directly or not, geographic areas to help in- In this article we first define the concept of to- terpret toponyms mentioned in documents. Such ponym and detail the subtasks of this challenge metadata can be geotagging of social media posts (Section3). Then, we summarize how we ac- (Zhang and Gelernter, 2014) or external databases quired and annotated our data (Section4). In Sec- structuring the information detailed in a document tion5, after describing the evaluation metrics, we (Weissenbacher et al., 2015). These three strate- briefly describe the resources and the baseline sys- gies are complementary and can be unified with tem provided to the participants. In the last Sec- machine learning algorithms as shown by (Santos tion6 we discuss the results obtained and the po- et al., 2015) or (Kamalloo and Rafiei, 2018). tential future direction for the task of toponym res- olution. 3 Task Description The definition of toponym is still in debate among 2 Related Work researchers. In its simpler definition, a toponym is a proper name of an existing populated place on The Entity Linking task aims to map a name of Earth. This definition can be extended to include a an entity with the ID of the corresponding en- place or geographical entity that is named, and can tity in a predefined Knowledge database (Bada, be designated by a geographical coordinate1. This 2014). Entity linking has been largely studied by encompasses cities and countries, but also lakes the community (Shen et al., 2015). Toponym res- or monuments. In this challenge we consider the olution is a special case of the entity linking task extended definition of toponyms and exclude all where strategies dedicated to toponyms can im- indirect mentions of places such as “30 km north prove overall performances. Three main strate- from Boston”, as well as metonymic usage and el- gies have been proposed in the literature. The first liptical constructions of toponyms. exploits the linguistic context where a toponym is mentioned in a document. The vicinity of the Subtask 1: Toponym Detection Toponym de- toponym often contains clues that help the read- tection consists of detecting the text boundaries of ers to interpret it. These clues can be other to- all toponym mentions in full PubMed articles. For ponyms (Tobin et al., 2010), other named entities example, given the sentence An H1N1 virus (Roberts et al., 2010), or even more generally, spe- was isolated in 2009 from a child cific topics associated more often with a particular hospitalized in Nanjing, China., a toponym than with others (Speriosu, 2013; Adams perfect detector, regardless how, would return two and McKenzie, 2013; Ju et al., 2016). The sec- pairs encoding the starting and ending positions ond strategy relies on the physical properties of of Nanjing and China, i.e. (64, 70) and (73, 77). the toponyms to disambiguate their mentions in Despite major progress, toponym detection is documents. The population heuristic or the min- still an open problem and it was evaluated in a imum distance heuristic are popular heuristics us- separate subtask since it determines the overall ing such properties. The population heuristics dis- performance of the resolution. Toponym mentions ambiguates toponyms by taking, among the am- missed during the detection cannot be disam- biguous candidates, the candidate with the largest biguated (False Negative, FN) and, inversely, population, whereas the minimum distance heuris- 1https://unstats.un.org/unsd/geoinfo/ tic disambiguates all toponyms in a document by UNGEGN/ 908 phrases wrongly detected as toponyms will re- 2017)3. The metadata provides the location of the ceived geocoordinates during the disambiguation infected host. With more than 3 million virus se- (False Positive, FP). Both FNs and FPs degrade quences4, GenBank provides abundant informa- the quality of the overall resolution. tion on viruses. However, previous work has sug- gested that geospatial metadata, when it is not sim- Subtask 2: Toponym Disambiguation The ply missing, can be too imprecise for local-scale second subtask focuses on the disambiguation of epidemiology (Scotch et al., 2011). In their ar- the toponyms only. In this subtask, all names of ticle Scotch et al., 2011 estimate that only 20% locations in articles are known by a disambigua- of GenBank records of zoonotic viruses contain tor but not their precise coordinates. The disam- detailed geospatial metadata such as a county or biguator has to select the GeoNames IDs corre- a town name (zoonotic viruses are viruses able sponding to the expected places among all pos- to infect naturally hosts of different species, like 2 sible candidates. GeoNames is a crowdsourced rabies). Most GenBank records provide generic database of geospatial locations and freely avail- information, such as Japan or Australia, without able.