et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON RECOGNITION 1 NameRec*: Highly Accurate and Fine-grained Person Name Recognition

Rui Zhang*, Yimeng Dai, and Shijie

Abstract—In this paper, we introduce the NameRec* task, which aims to do highly accurate and fine-grained person name recognition. Traditional Named Entity Recognition models have good performance in recognising well-formed person from text with consistent and complete syntax, such as news articles. However, there are rapidly growing scenarios where sentences are of incomplete syntax and names are in various forms such as user-generated contents and academic homepages. To address person name recognition in this context, we propose a fine-grained annotation scheme based on . To take full advantage of the fine-grained annotations, we propose a Co-guided Neural Network (CogNN) for person name recognition. CogNN fully explores the intra-sentence context and rich training signals of name forms. To better utilize the inter-sentence context and implicit relations, which are extremely essential for recognizing person names in long documents, we further propose Inter-sentence BERT Model (IsBERT). IsBERT has an overlapped input processor, and an inter-sentence encoder with bidirectional overlapped contextual embedding learning and multi-hop inference mechanisms. To derive benefit from different documents with a diverse abundance of context, we propose an advanced Adaptive Inter-sentence BERT Model (Ada-IsBERT) to dynamically adjust the inter-sentence overlapping ratio to different documents. We conduct extensive experiments to demonstrate the superiority of the proposed methods on both academic homepages and news articles.

Index Terms—Information extraction, fine-grained, BERT, inter-sentence context, hierarchical inference, adaptive overlapped embedding, joint learning. !

1 INTRODUCTION

Named Entity Recognition aims to locate and classify academic homepages, resumes, articles in online forums and named entities from unstructured text into predefined social media (cf. Figure 1(b)). These text often contain person categories such as person names, locations, organizations, names of various forms with incomplete syntax. quantities, time, medical codes, etc. Among the named In this paper, we introduce the NameRec* task, which entities, person names are one of the most important and aims to do highly accurate and fine-grained person name person name recognition serves as the basis for many recognition. Figure 2 shows an example of person name downstream applications in knowledge management and recognition in academic homepages. The biography section information management. Recognising person names from consists of complete sentences while the students section unstructured text is an important process for many online simply lists information in a line. The person names may academic mining system, such as AMiner [1] and CiteSeerX be in different forms. Figure 2 contains well-formed full [2]. The recognized person names plays an important name of the researcher ‘John Doe’ in the page header and role for knowledge base enrichment [3]. Person name abbreviated names in the publications section. Further, the recognition can also provide valuable insights for learning abbreviated names may have different abbreviation forms, the relationships between people and provides valuable e.g., ‘B.B. Bloggs’ vs. ‘Doe, J.’. insights for analysing their collaboration networks. [4], [5]. To better recognize person names from such a free-form

arXiv:2103.11360v2 [cs.CL] 23 Mar 2021 Traditional NER models [6], [7], [8] have achieve text, we exploit knowledge from anthroponymy [9] (i.e., good performance on recognising person names from the study of the names of human beings) and propose a well-formed text with consistent and complete syntax, such fine-grained annotation scheme that labels detailed name as news articles (cf. Figure 1(a) ). Person names in such text forms which distinguishes first, middle, or last name, and often have straightforward patterns, e.g., first name in full full names or initials (cf. Figure 2). Such fine-grained followed by last name in full. However, challenges remain annotations offer richer training signals to NER models for recognising person names from free form text. These to learn the patterns of person names in free-form text. may appear in many applications, such as user-generated However, fine-grained annotations also bring challenges because more label classes need to be learned. • Rui Zhang * Corresponding Author To take full advantage of the fine-grained annotations, URL https://ruizhang.info/ we propose a multi-task learning model called Co-guided [email protected] Neural Network (CogNN) for person name recognition. • Yimeng Dai CogNN consists of two sub-neural networks, which The University of Melbourne [email protected] are Bi-LSTM-CRF variants. One sub-network focuses on • Shijie Liu predicting whether a token is a name token, while the The University of Melbourne other focuses on predicting the name form of the token. [email protected] The intuition is that knowing whether a token is a part ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 2

a gated-fusion layer to balance the information learned from two sub-networks. This way, neither sub-network is overwhelmed by misleading signals that may be learned by the other sub-network. CogNN fully explores the intra-sentence context and rich training signals of name forms. However, the inter-sentence context and implicit relations, which are essential for recognizing person names in long documents, are not captured. For example, in Fig 2, the sentence ‘Dr John Doe is a Professor at the XYZ University’ in the academic homepage contains rich contextual information. The intra-sentence context indicates that the words ‘John Doe’ may be a person’s name, since the words appear after a ‘Dr’ and before a job occupation ‘a Professor’. However, in later part of the academic homepage, ‘John Doe’ appears with much less intra-sentence context and is harder to recognise. To address this issue, we propose a Inter-sentence BERT Model (IsBERT) to capture the inter-sentence context and implicit relations. A naive BERT model [10] only learns context-dependent word embeddings based on the context within a sentence of limited length, i.e., intra-sentence contextual word embeddings. Different from the naive BERT model, IsBERT learns the context-dependent word embeddings across different sentences, i.e., inter-sentence contextual word embeddings. The implicit relations among tokens are captured, passed and strengthen across different sentences. IsBERT mainly consists of an overlapped input processor and an inter-sentence encoder. Specifically, the input processor adds special tokens to better indicate boundaries and continuous in a long document. The input processor also overlaps each pair of adjacent sentences with a certain ratio of tokens to preserve inter-sentence context. To better capture the context information across different sentences in a long document, we further propose an inter-sentence encoder with the bidirectional overlapped contextual embedding learning and multi-hop inference mechanisms. The inter-sentence encoder contains multiple hop, where each hop learns the inter-sentence context-dependent word embeddings through a forward overlapped contextual embedding layer and a backward overlapped contextual embedding layer. The implicit relations among tokens, i.e., the inherent signals about whether certain words have been recognized in other context, are captured and passed Fig. 1. An example of a news with well-formed text and an across different sentences bidirectionally during learning. example of an academic resume with free-form text. All the person names are highlighted. Different hops encode the documents repeatedly to update the embeddings and strengthen the inherent signals. IsBERT takes full advantage of inter-sentence context of a name helps recognise the fine-grained name form in long documents, while for short documents with much class of the token, and vice versa. For example, if a less abundances of context, it may lose advantages. The token is not considered as a part of a name, then even predefined overlapping embeddings ratio, which works if it is a word initial, it should not be labelled as a well for long documents, may not help capture and pass name initial. However, the underlying correlation between meaningful signals in a much shorter document, but lead different annotations cannot be captured well when the two to many repeated context and redundant embeddings in sub-networks are trained together by simply minimising the model. To derive benefit from different documents the total loss. The reason is that the learning signals of with diverse abundances of context, we further propose an the two sub-neural networks are not shared sufficiently. To advanced Adaptive Inter-sentence BERT Model (Ada-IsBERT). better capture the underlying correlation between different Ada-IsBERT has similar architecture as IsBERT, while annotations, we share the learning signals of two sub-neural it has an adaptive bidirectional overlapping mechanism networks through a co-attention layer. Further, we use instead of using a predefined overlapped embedding ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 3

Fig. 2. An example of person name recognition in academic pages. All person names are highlighted (best viewed in color).

ratio. It dynamically adjust and update the inter-sentence conference paper [11]. In the conference paper, we overlapping ratio according to the length of different presented a fine-grained annotation scheme, a new dataset, documents during training. With the adaptive overlapping and a person name recognition model which fully mechanism, Ada-IsBERT is much robuster than IsBERT for explores the rich training signals of name forms and documents with different abundances of context. intra-sentence context. In this journal extension, we extend The main contributions of this paper are listed as follows. the techniques for person names recognition in long • C1: We propose a fine-grained annotation scheme documents by exploring the document-level inter-sentence based on anthroponymy. This fine-grained annotation context. To this end, we present the IsBERT model to scheme provides information on various forms of person utilize the document-level context and capture implicit names (Section 3.1). relations. We propose an overlapped input processor, and • C2: We create the first dataset consisting of diverse an inter-sentence encoder with bidirectional overlapped academic homepages where the person names are fully contextual embedding learning and multi-hop inference annotated under our proposed fine-grained annotation mechanisms. In addition, we present an Ada-IsBERT model scheme, called FinegrainedName, for the research of name with an adaptive bidirectional overlapping mechanism to recognition (Section 3.2). derive benefit from documents with diverse abundances • C3: We propose a Co-guided Neural Network (CogNN) of context. We also conduct more extensive experiments to model to recognise person names using the fine-grained demonstrate the effectiveness of our proposed methods for annotations. CogNN learns the different name form classes documents with different context. with two neural networks while fusing the learned The rest of the paper is organized as follows. signals through co-attention and gated fusion mechanisms Section 2 provides a review of the related work. Section 3 (Section 4.1). introduces the Fine-grained Annotation Scheme and our • C4: We also propose a Inter-sentence BERT Model FinegrainedName dataset annotated under this scheme. (IsBERT) to recognise fine-grained person names in long Section 4.1 presents our CogNN model that takes documents by capturing the inter-sentence context and advantages of the fine-grained annotations to recognise implicit relations. IsBERT consists of an overlapped input person names. Section 4.2 introduces the IsBERT model processor, and an inter-sentence encoder with bidirectional that utilizes the inter-sentence context in long documents overlapped contextual embedding learning and multi-hop to improve the performance of person name recognition. inference mechanisms (Section 4.2). Section 4.3 introduces the Ada-IsBERT model that adaptive • C5: We propose an Adaptive Inter-sentence BERT Model to documents with diverse abundances of context. Section 5 (Ada-IsBERT), which is robust for documents with diverse presents experiments in on academic homepages and news abundances of context. Ada-IsBERT dynamically adjust and articles. Section 6 concludes the paper. update the inter-sentence overlapping ratio according to the length of different documents during training (Section 4.3). ELATED ORK • C6: We empirically investigate the superiority of 2 R W our proposed models on both academic homepages and Named entity recognition (NER) aims to identify proper newswire texts compared with state-of-the-art NER models. names in text and classify them into different types, such Experimental results also show that our annotations can as person, organisation, and location [12]. Neural NER be utilised in different ways to improve the recognition models have shown excellent performance on long texts performance (Section 5). which follow strict syntactic rules, such as newswire and This article is an extended version of our earlier Wikipedia articles [6], [7], [8]. However, these NER models ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 4

are less attractive when applied to texts which may not many studies try to compress the model size for real-life have consistent and complete syntax [13], [14]. Recent applications and on resource-restricted devices, such as studies consider user-generated short texts from social TinyBERT [30]. However, these extensions have the same media platforms such as Twitter and Snapchat [15], [16]. limitation on input length as the vanilla BERT. They can However, there are few NER studies on free-form text with only capture the word embeddings based on intra-sentence incomplete syntax including person names with various context. Different from existing works, we explore the forms, such as academic homepages, academic resumes, inter-sentence context, and propose methods based on articles in online forums and social media. BERT for documents with different abundances of context. BIO and BIEO tagging schemes [17] are often used for named entity recognition in well-formed text, such as 3 ANNOTATION SCHEMEAND DATASET news articles in CoNLL-2003 [18] and wikipedia articles in WiNER [19]. However, such annotations for name spans We first present our fine-grained annotation scheme and cannot reflect the patterns of various name forms and brings introduce our FinegrainedName dataset annotated under challenges for recognising persons names in free-form text. this scheme. To the best of our knowledge, no other work has utilised anthroponymy [9] and fine-grained annotations to help 3.1 Fine-grained Annotations recognise person names. Fine-grained annotations are done to better capture the Information Extraction (IE) studies on academic person name form features in free-form texts. Annotating homepages and resumes usually treat the text content the name tokens with fine-grained forms offers more direct as a document, upon which traditional NER techniques training signals to NER models to learn the patterns of are applied. For example, [20] use a Bi-LSTM-CRF based person names. Thus, unlike traditional NER datasets, which hierarchical model to extract all the publication strings only label a name token with a PER (person) label, we from the text content of a given academic homepage. further provide fine-grained name form information for [21] capture the relationship between publication strings each name token based on anthroponymy [9]. and person names in academic homepages, and extract We label each name token using a three-dimensional them simultaneously. This technique does not apply to annotation scheme: our problem as we assume no pre-knowledge about the • BIE: Begin, Inside, or End of name, indicating the publication strings. position of a token in a person name, Person names are often recognised together with other • FML: First, Middle, or Last name, indicating whether a entities, such as locations and organisations [6], [7], [8], name token is used as the first, middle, or last name, [14]. [22] focus on extracting name from noisy OCR and data by combining rule based methods, the Maximum • FI: Full or Initial, indicating whether a name token is Entropy Markov Model, and the CRF model using a simple a full name word or an initial. voting-based ensemble. [23] extract person names from emails using CRF. They design email specific structural Using the three-dimensional annotation scheme above, features and exploit in-document repetition to improve the we can describe the fine-grained name form of a name extraction accuracy. [24] study person name recognition token. For example, in Figure 2, ‘John Doe’ can be labelled in Arabic using rule-based methods. To the best of our as Begin First Full End Last Full, while ‘Johnny der knowledge, no other work has taken into account name Doe’ can be labelled as Begin First Full Inside Last Full forms and uses deep learning based models. Inside Last Full End Last Full. Multi-task learning models, which train tasks in parallel and share representations between related tasks, have been 3.2 The FinegrainedName Dataset proposed to handle many NLP tasks. [25] propose to share FinegrainedName is a collection of academic homepages the hidden layers between tasks, while keeping several with person names fully annotated using the proposed task-specific output layers. [26] jointly learn POS tagging, annotation scheme. We use Selenium1, an open-source chunking and CCG supertagging by using successively automated rendering software, to render the webpages deeper layers. [27] propose a model for sentiment analysis and collect visible texts from the webpages. We download by jointly learn the character features and long distance academic homepages from universities and research dependencies through concatenation procedures. Rather institutes around the world and focus on English than directly sharing the representations or concatenating homepages. the representations of different tasks, our co-attention and FinegrainedName contains 2,087 subfolders and each gated fusion mechanisms allow our model to co-guide subfolder contains three files for a webpage: the jointly trained tasks without overwhelmed by the misleading signals. • An HTML file containing the page source. Bidirectional Encoder Representations from • A TXT file containing the visible text of the webpage, Transformers (BERT) [10] is a pre-trained language which is rendered by python’s Selenium2 package. model. BERT has an encoder and decoder architecture, • A JSON file containing name annotations. Figure 3 and learns to predict both the left and right context using shows the example format of the JSON files. a multi-layer transformer [28]. Many extensions of BERT has been proposed to solve domain-specific problems, 1. https://www.seleniumhq.org/ such as BioBERT [29] for biomedical text. There are also 2. https://selenium-python.readthedocs.io/ ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 5

each annotator to annotate six pages. We examine the results and provide guidance on how to improve the annotation quality. We highlighted the following at training:

• Any named entities such as places, buildings, organizations, prizes, honored or books, which are named after a person, should not be annotated as a person’s name. • Words connected with a or an apostrophe should not be split into multiple tokens. For example, both ‘Joon-gi’ and ‘O’Keeffe’ both have only one token. • Nobiliary particles3, e.g., ‘van’, ‘zu’ and ‘de’, should be annotated as last names.

Each academic homepage is annotated by two annotators using our annotation tool. Any pages with uncertain name labels is noted down in the comment field (cf. Figure 3). After their annotations, we make a decision on the disagreement between annotators and also check the uncertain pages and names. We send feedback when they annotate every 230 homepages. ur Annotation Analysis We summarise the disagreement between annotators. Fig. 3. Screenshot of an example JSON file

• Confidence: Only 3.64% of all the homepages contain Annotation Tool Annotation of homepages is annotations that are uncertain as flagged by the time-consuming, especially when a homepage contains annotators, while 78.08% of these pages are actually many names in complex forms. We developed a correctly labelled. This indicates that the annotators semi-automatic tool to assist the annotation, which have high confidence in their annotations. has five main functionalities: • Inter-annotator Agreement: We compute the inter-annotator agreement on name strings and • Group label: This functionality helps annotate a name forms using Cohen’s Kappa measurement. The group of names of the same form. For example, ‘Doe annotators have higher agreement on name strings J’ and ‘Joon-gi L’ have the same forms and can (κ = 0.63) and lower agreement on fine-grained be annotated at once. name forms (κ = 0.41). The disagreement is mainly • Index: This functionality helps find all positions of a in homepages with a long string of consecutive name string in the TXT file. tokens such that different annotators may disagree • Mask: This functionality helps annotators to on which tokens to form a name. The annotators proofread the text and find unlabelled names. It may also disagree on whether a name token is a first replaces all the names already annotated with a name, , or last name. This is difficult special token ‘ANNOTATED’. especially when the context is unclear. • Validate: This functionality runs a simple automated • Time: On average, it takes 16 minutes to annotate an quality check of the annotations. It checks: (1) academic homepage with our tool. whether the position indices of the names annotated in the JSON file are consistent with the names Dataset Analysis In total, the FinegrainedName contains appeared in the TXT file; and (2) whether each 2,087 English academic homepages from 286 institutes, i.e., annotated name comes with the name form under 7.29 pages per institute (standard deviation 7.27). A total the three-dimensional annotation schemes. of 34,880 names are annotated and 70,864 name position • Compare: This functionality locates disagreement indices are recorded. On average, a name appears twice in between two annotators’ labels on the same an academic homepage. Most names begin with last names homepage. It identifies the list of names with (64.73%) while the rest mostly begin with first names. Only inter-annotator disagreement. 13 names start with middle names. Most names contain at Annotators There are 6 annotators to annotate the dataset. least one initial (66.57%). The two most frequent name forms The annotators are postgraduate students who have taken are Begin Last Full End First Initial and Begin First Full machine learning subjects. We provide a one-hour training End Last Full. Table 1 summarises the annotation results to each annotator. and the dataset. We provide the annotators with an annotation scheme and two example pages that are already annotated. We ask 3. https://en.wikipedia.org/wiki/Nobiliary particle/ ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 6

Fig. 4. CogNN network structure.

TABLE 1 Our proposed model CogNN achieves this aim with the Summary of annotation and dataset. κ is the Cohens Kappa help of two Bi-LSTM-CRF based sub-networks: the name measurement. token network and the name form network, as illustrated in Figure 4. The name token network focuses on predicting Summary of Annotation whether a token is part of a name (the BIE dimension), Uncertain pages 3.64% Confidence while the name form network focuses on predicting the Accuracy on uncertain names 78.08% fine-grained name form class of the token (FML or FI Inter-annotator Names 0.63 dimensions). The intuition is that knowing whether a token agreement (κ) 0.41 Name forms is part of a name helps recognise the fine-grained name Time 16 min form class of the token, and vice versa. For example, if a Summary of Dataset token is not considered as part of a name, then even if it is a word initial, it should not be labelled as a name Total Homepages 2,087 Total Institutes 286 initial. To better capture the underlying correlation between Average Institutes 7.29 different annotations, we share the learning signals of two STD. Institutes 7.27 sub-neural networks through a co-attention layer. To avoid Total Names Indexes 70,864 being overwhelmed by possible misleading signals, we Total Names 34,880 further add a gated-fusion layer to balance the information Contain Initial 23,221 learned from two sub-networks. Begin with Last Name 22,581 In particular, an input token is represented by Begin with Middle Name 13 concatenating its word embedding and its letter case Begin with First Name 12,286 vector. We feed such representation of the input into Bi-LSTM to learn its hidden representation matrix, which is detailed in Section 4.1.1. Then, we use co-attention 4 PROPOSED MODELS and gated fusion mechanisms to co-guide the two 4.1 CogNN Model jointly trained sub-networks. Our co-attention mechanism updates the importance of each token learned from the The fine-grained annotations offer more direct training two sub-networks and records their correlations (Section signals to NER models but also bring challenges because 4.1.2). Our gated fusion mechanism helps decide whether more label classes need to be learned. In this section, we and how much to accept new signals from the other present our CogNN model that takes advantages of the sub-network (Section 4.1.3). The two sub-networks are 4 fine-grained annotations to recognise person names . trained simultaneously by minimising their total loss Given a sequence of input tokens X, where X = (Section 4.1.4). [x1, x2, ..., xn] and n is the length of the sequence, our aim 5 is to predict for each token xi whether it is a name token. 4.1.1 Capture: Hidden Feature Extraction

The name token network (denoted as NY ) and the name 4. A demonstration of our model will be available at http://www. N 0 ruizhang.info/namerec/ form network (denoted as Y ) have a similar structure. Y Y 0 Y 5. We use xi to denote both a token and its embedding vector as long They only differ in the target labels and . Here, as the context is clear. denotes the label sequence that indicates whether an input ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 7

token is part of a name, and Y 0 denotes the label sequence 4.1.3 Balance: Gated Fusion Mechanism that indicates the form class of each input token. We focus To avoid being overwhelmed by misleading learning signals our explanation on the name token network NY in the from the other sub-network NY 0 during co-attention, following discussion while the name form network works we dynamically balance the information learned from in a similar way. the (independent) hidden representation H and the An input token xi ∈ X is represented by concatenating corresponding new (dependent) hidden representation H˜ 0 0 its word embedding ei and its letter case vector . We use for NY (also H and H˜ for NY 0 ), and obtain a fused 0 GloVe [31] computed on our FinegrainedName corpus for representation matrix F (F for NY 0 ). the word embeddings ei. The letter case vector si indicates Inspired by the study on multi-modal fusion for images the letter case information of xi, which is an important and text [15], we add a gated fusion layer to balance the hint for recognising names. For example, the first letter of information from H and H˜ to obtain better representations. a name token is often in uppercase, and a name initial is We first transform each item in H and H˜ by: often formed by an uppercase letter plus a dot. Our letter h = tanh(W h˜ + b ) case vector is a three-dimensional binary vector where each h˜i h˜i i h˜i dimension represents: (i) whether the first character in the hhi = tanh(Whi hi + bhi ) token is in uppercase, (ii) whether all characters in the token where W ˜ and Wh are trainable parameters. are in uppercase, and (iii) whether any character in the token hi i is in uppercase. Then, our fusion gate, which decides whether and how We then use Bi-LSTM [32] to capture the hidden features much to accept the new information, is computed as: from the input sequence. The output hidden representation, g = σ(W (h ⊕ h )) t gt h˜i hi denoted as hi, summarises the context information of xi σ in X. Our hidden representation matrix H in NY can where is the element-wise sigmoid function to scale into d W be written as [h1, h2, ..., hn], where hi ∈ R and d is the range of (0,1) and gt are trainable parameters. the number of dimensions of the hidden representation. We fuse the two representations using the fusion gate 0 0 0 0 through: Similarly, H in NY 0 can be written as [h1, h2, ..., hn]. f = g h + (1 − g )h i t hi t h˜i 4.1.2 Share: Co-attention Mechanism The fused representation sequence F = [f1, f2, ..., fn] is Training the two sub-networks separately is suboptimal, trained to produce a label sequence Y . To enforce the since the underlying correlation among the name label structural correlations between labels, Y is passed to a CRF dimensions is lost. For example, a token recognised as layer to learn the correlations of the labels in neighborhood. Inside in NY is more possible to be Middle in NY 0 . To Let Y denotes the set of all possible label sequences for F . address this issue, we share the learning signals between Then, the the probability of the label sequence Y for a given the hidden representation matrices H and H0, and obtain fused representation sequence F can be written as : new hidden representation matrices H˜ and H˜0 for the two Q t ψt(yt−1, yt; F ) sub-networks, respectively. p(Y |F ,WY ) = P Q 0 0 Y 0∈Y t ψt(y t−1, y t; F ) Specifically, we use co-attention to take the learning 0 signals from two hidden representations into account by: where ψt(y , y; F ) is a potential function, WY is a set of parameters that defines the weight vector and bias 0 0 P = tanh(WhH ⊕ (Wh0 H + bh0 )) corresponding to label pair (y , y). Similarly, we can also compute the fused representation 0 0 0 k×d F p(Y |F ,W 0 ) where Wh and Wh0 ∈ R are trainable parameters, k sequence and Y . is dimensionality of the parameters, ⊕ is the concatenating operation, tanh is the activation function to scale into the 4.1.4 Joint Training range of (-1,1), and P ∈ R2k×n. The remaining question is how to train two networks 0 The co-attention distribution that records the importance simultaneously to produce label sequences Y and Y . We of each token after examining two hidden representation achieve this by joint optimisation. Specifically, we train the sequences can be obtained as: CogNN model end-to-end by minimising loss L, which is the sum of the loss of the two sub-networks:

A = softmax(WpP + bp) L = L(WY ) + L(WY 0 )

1×2k n where Wp ∈ R are trainable parameters and A ∈ R is where L(WY ) and L(WY ) are the negative log-likelihood an importance weight matrix. of the ground truth label sequences Yˆ and Yˆ0 for the input The new hidden representation h˜i can be computed by: sequences respectively, which are computed by:

˜ X X hi = aihi, ∈ A, hi ∈ H L(WY ) = − δ( = Yˆ ) log p(Yi|F ) i Yi X X 0 ˆ0 0 0 We thus obtain the new hidden representation sequences L(WY 0 ) = − δ(Yj = Y ) log p(Yj |F ) ˜ ˜ ˜ ˜ ˜0 ˜0 ˜0 ˜0 j 0 H = [h1, h2, ..., hn] and H = [h1, h2, ..., hn] for the two Yj sub-networks. ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 8

input processor converts input documents into chunks with overlapped context-independent embeddings (Section 4.2.1). The inter-sentence encoder contains multi-hop inference modules, where each inference module learns the inter-sentence context-dependent word embeddings through overlapped contextual embedding learning, and different inference modules encode the documents forward and backward alternately to update the embeddings and pass the inherent signals across sentences (Section 4.2.2). Finally, the output predictor produce the final predictions for each input token based on the learned inter-sentence contextual word embeddings (Section 4.2.3).

4.2.1 Overlapped Input Processor We focus on English documents. The documents are first tokenized by whitespace and punctuations followed by WordPiece tokenization [33]. While whitespace tokenizer breaks the input documents into tokens around the whitespace boundary, the WordPiece tokenizer uses longest prefix match to further break the tokens into sub-tokens. The resultant sub-tokens which from one token are assigned the same label. For example, if the token “zhenning” with label Begin First Full is tokenized into “z”, “##hen” and “##”, then all of the three sub-tokens are labelled as Begin First Full. For convenience’s sake, we will use “token” to refer to “resultant sub-tokens from WordPiece tokenizer” in the following discussion. After tokenizing the documents into a series of tokens, Fig. 5. The overall architecture of the IsBERT model. the most straightforward way to learns contextual word embeddings is to split the documents into different chunks sentence by sentence and feed them into the naive BERT 4.2 IsBERT Model model [10]. The sentences exceeding the input length limit are broken into several short sub-sentences before fed into CogNN fully explores the intra-sentence context and rich the model. training signals of name forms. However, the inter-sentence Such a straightforward method only capture the context and implicit relations, which are extremely essential intra-sentence context since each input to the model for recognizing person names in long documents, are not is a single distinct sentence, while the inter-sentence captured. In this section, we present our IsBERT that takes context and relationships are lost during learning. For advantages of inter-sentence context informations. applications which assume the independence between Given a sequence of sentences D = [T1,T2,T3, ..., TM ] different sentences, only capturing the intra-sentence from a document D, where Tj = [t1, t2, t3, .., tn], n is the context is acceptable. However, for applications which length of the sentence and ti is the i-th tokens in Tj, we aim strongly rely on document-level information, capturing the to predict whether ti is a name token and the corresponding inter-sentence context is important. fine-grained name type. To address this issue, we propose an overlapped Our IsBERT achieves this aim by learning inter-sentence chunking method to preserve inter-sentence information contextual word embeddings. The context information when learning the contextual word embedding using BERT. across different sentences are shared in a long document Two strategies are used in the overlapped chunking method through bidirectional overlapped embedding learning. compared with a naive pre-processing method in BERT: The implicit relations among tokens, i.e., the inherent 1) adding special tokens to indicate the boundaries and signals about whether certain words have been recognized continuous in document; 2) overlapping each pair of in other context, are captured, passed and strengthen adjacent chunks with predefined ratio. across different sentences during bidirectional overlapped Specifically, “[CLS]” is added at the beginning of the embedding learning and multi-hop inferences. The intuition first chunk of each document to indicate the document-level behind this is that some person name entities may appear boundary, while the remaining chunks of the document repeatedly in different context, while some context provide are started with the special token “$$” to indicate the strong signals about the entity type and others not. Passing continuation. The special token “[SEP]” is appended to the the inter-sentence context information can help recognize end of each sentence to specify the sentence-level boundary. more person names in different context. In addition to adding signals to indicate the boundaries Figure 5 illustrates the overall structure of the IsBERT and continuous, each pair of adjacent chunks from one model. The model consists of three parts: overlapped input document is overlapped with a certain ratio of tokens, processor, inter-sentence encoder and output predictor. The i.e., last k tokens of one chunk is the first k tokens of ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 9

Fig. 6. The forward and backward overlapped embedding learning in IsBERT.

To better learn the inter-sentence context-dependent word embeddings, we further propose an inter-sentence encoder. The chunks with context-independent word embeddings [W1,W2,W3, ..., WM ] from one document are fed into the inter-sentence encoder in . The inter-sentence encoder contains multi-hop inference modules, where each inference module learns the inter-sentence context-dependent word embeddings through two overlapped embedding learning layers, i.e., a forward overlapped embedding learning layer and a backward overlapped embedding learning layer. Fig. 7. An example of converting a document to token chunks Figure 6 illustrates the procedures or forward and backward overlapped embedding learning in IsBERT. the follow-up chunk. An example of the pre-processed In the forward overlapped embedding learning layer, for overlapped input documents is shown in Figure 7. each adjacent chunk pairs Wi−1 and Wi in a document, For sentence-level applications, different chunks need we take the output context-dependent word embeddings to be shuffled in the sentence-level after tokenization to of the overlapped tokens from Wi−1 as the input word avoid over-fitting during training. In our overlapped input embeddings of same tokens in Wi. Then, the corresponding −→ processor, the shuffling only happens in the document-level, forward context-dependent embedding Wi is represented while the order of different chunks from each document as: is maintained. Specifically, given a dataset U containing L −→ −−−→ Wi = BERT ([Wi−1]⊥k, Wi) (1) documents U = [D1,D2,D3, ...DL], where each document is pre-processed and splited into a list of chunks Dl = −−−→ where [Wi−1]⊥k is the last k numbers of the [T1,T2,T3, ..., TM ]. Then, after the shuffling of the dataset, −−−→ context-dependent embeddings of Wi−1. only the order of Dl in U should be changed, while the order of T in each D remains as before. Then, In the backward overlapped embedding learning layer, j l W W the tokens in each chunk are mapped to corresponding for each adjacent chunk pairs i and i+1 in a document, we feed the output context-dependent word embeddings of context-independent word embeddings and each document −→ W W D can be represented as [W1,W2,W3, ..., WM ], where Wj the overlapped tokens from i+1 back to i. represents all the context-independent word embeddings of Then, the forward and←→ backward (inter-sentence) all the tokens in chunk Tj. context-dependent embedding Wi, which carry the context information from both previous and succeeding chunks, is 4.2.2 Inter-sentence Encoder represented as: The input processor converts a document D into chunks ←→ −→ ←−−− with overlapped context-independent word embeddings. Wi = BERT (Wi, [Wi+1]>k) (2) ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 10 ←−−− where [W ]>k is the first k numbers of the i+1 ←−−− context-dependent embeddings of Wi+1. To strengthen the inter-sentence relations and strengthen the inherent signals in the context-dependent embeddings, we apply multi-hop inference modules and sum the learned inter-sentence context dependent embedding ←→ ←→ ←−→ [W1, W2, ..., WM ]t of the current inference module It with the inter-sentence context-dependent embeddings ←→ ←→ ←−→ [W1, W2, ..., WM ]t−1 from the previous inference module It−1 as the input for the next inference module It+1. Then, after multi-hop inferences, the last inter-sentence context-dependent embeddings can be represented as ←→ ←→ ←−→ Fig. 8. Training Dataset Label Distribution [W1, W2, ..., WM ]m, where m is the inference times.

4.2.3 Output Predictor The inter-sentence encoder converts chunks with overlapped context-independent word embeddings [W ,W ,W , ..., W ] into inter-sentence context-dependent 1 2 3 M←→ ←→ ←−→ word embeddings [W1, W2, ..., WM ]m. Then, the learned inter-sentence contextual word embeddings are passed through a linear layer to get final prediction. The predictions for wordpieces (sub-tokens) from the same token are inconsistent. For these cases, we apply the majority vote and random tiebreaker to decide the final predictions for these token. Fig. 9. Testing Dataset Label Distribution 4.3 Ada-IsBERT Model IsBERT takes full advantage of inter-sentence context in • The performance of the proposed models long documents, while for short documents with much against baseline models and variant models less abundances of context, it may lose advantages. The on recognising person names from academic predefined overlapping ratio k, which works well for long homepages (Section 5.1.2). documents, may not help capture and pass meaningful • The impact of using different overlapped ratios signals in a much shorter document. To derives benefit from and different numbers of inference module different documents with diverse abundances of context, we for recognising person names from academic introduce our Ada-IsBERT model in this section. homepages with different abundances of context Ada-IsBERT has similar architecture as IsBERT, while (Section 5.1.3). it has an adaptive overlapped embedding ratio k˜ instead • The applicability of the proposed models on of using a predefined overlapped embedding ratio k. It recognising person names from news articles dynamically choose the inter-sentence overlapping ratio (Section 5.2). from a list of ratios according to the length l of different • The cases study and error analysis of our proposed documents [D1,D2,D3, ...DL] during training and a list models (Section 5.3). of predefined length thresholds. Then, the inter-sentence ←→ contextual embedding Wi is represented as: 5.1 Effectiveness on Academic Homepages ←→ −→ ←−−− In this subsection, we study the performance of our W = BERT (W , [W ]>k˜) (3) i i i+1 proposed annotation scheme together with the CogNN ←−−− ˜ model on academic homepages. where [Wi+1]>k is the adaptive number of the ←−−− Dataset We use the FinegrainedName with the proposed context-dependent embeddings of Wi+1. fine-grained annotation scheme (Section 3), where 1,677 With the adaptive overlapping mechanism, Ada-IsBERT homepages are used for training and developing and 410 is much robuster than IsBERT for documents with different homepages are used for testing. The label distribution in abundances of context. two datasets are shown in Figure 8 and 9 respectively. Evaluation Recall (R), Precision (P) and F1-scores (F) are 5 EXPERIMENTAL STUDY used to measure the performance. We report the Token Level performance, which reflects the model capability to We explore the following five aspects of our approach by a recognise each person name token. We also report the Name comprehensive experimental study: Level performance, which reflects the model capability to • The impact of using fine-grained annotations in recognise a whole person name without missing any token. different ways for recognising person names from The reported improvements are statistically significant with academic homepages (Section 5.1.1). p < 0.05 as calculated using McNemar’s test. ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 11

size of 32, and an initial learning rate of 0.01 with a decay rate of 0.05. • CogNN: Our proposed model is implemented following the description in Section 4.1. Dropout is applied on the Bi-LSTM layers. We use a standard grid search to find the best hyperparameter values on the developing dataset. We choose the initial learning rate among [0.001, 0.01, 0.1], the decay rate among Fig. 10. Early, late, and in-network fusion. [0.05, 0.1], the dimension of hidden layer among [50, 100, 200], the dropout rate among [0.2, 0.5] and has a batch size of 32. The optimal hyperparameters Word Embedding For CogNN model, we use GloVe6 are highlighted above in bold. to learn word embeddings. For experiments on academic All above deep learning models are implemented using homepages, we train 100-dimensional word embeddings Theano9 and Lasagne10. All the optimal hyperparameters using GloVe on FinegrainedName, with a window size of of above deep learning models are obtained with standard 15, minimun vocabulary count of 5, full passes through grid search on the same developing dataset with stochastic cooccurrence matrix of 15, and an initial learning rate of gradient descent. We stop training if the accuracy does not 0.05. For experiments on newswire articles, we initialise improve in 10 epochs. word embeddings with GloVe pretrained 100-dimensional Table 2 shows the results. CRF achieves an up to 13.83% embeddings which are pretrained on English Gigaword improvement on F1 score when the fine-grained FML and Fifth Edition7 containing a comprehensive archive of FI annotations are provided without any fusion. The same newswire text data. trend can also be observed for Bi-LSTM-CRF. The reason 5.1.1 Effectiveness of the Annotation Scheme is that academic homepages contain person names with various forms and simple BIE annotations cannot reflect We first study the impact of using our fine-grained the patterns of name forms well. When examining the annotations with different fusion strategies (Figure 10) and performance of using early fusion, we find that it is much different models. Specifically, four fusion strategies are worse than no fusion. This is expected as early fusion of tested: different dimensions of name form leads to too many classes • No fusion: Training an independent model that to be predicted. Even for a two-token name, it may have learns to label the input sequence with the BIE, FML, (3 × 3 × 2)2 = 324 possible name form combinations. or FI label types but not a combination of any two Late fusion offers better performance than no fusion with types of the labels. up to 15.95% and 2.56% improvements in F1 score for • Early fusion: Training an independent model that Bi-LSTM-CRF and CRF, respectively, which indicate that learns to label the input sequence with a cartesian the separately trained models on different annotations have product of the BIE, FML, and FI label types, their own focuses. However, the underlying relationships e.g., to label ‘John Doe’ with Begin First Full among different name forms are not captured using late End Last Full. fusion. Our CogNN model, which uses in-network fusion, • Late fusion: Training sub-models each focusing on outperforms the best late fusion baseline by up to 5.35% in one label type and merging all the predicted labels F1-score. The reason is that our model can take advantage afterwards to yield the final prediction by using of the correlations between name form types when training every span of tokens with name label as a name. and gain higher prediction confidence. • In-network fusion: Training two sub-models each Overall, the fine-grained annotations can improve the focusing on one label type and sharing the learning performance of person name recognition on academic signals in the intermediate levels of the sub-models homepages using no fusion, late fusion and in-network (This is what our proposed CogNN model does). fusion strategies. The neural-based models perform better than non-neural models and the in-network fusion can Three models are tested: achieve the best results. • CRF: [34]. We use the Java implementation provided by the Stanford NLP group8. The software provides 5.1.2 Effectiveness of the Proposed Models a generic implementation of linear chain CRF model. Since there are no existing work study the fine-grained • Bi-LSTM-CRF: [6]. Specifically, the word person name recognition task, we compare with the embeddings are fed into a Bi-LSTM layer as following deep learning models and variants of our input. Dropout is applied to the output of Bi-LSTM proposed models: layer to avoid overfitting. The output is further fed into a linear chain CRF layer to predict the tokens • JointNN: We share the hidden layers between two labels. Specifically, it has a 100-dimensional hidden tasks while keeping two task-specific output layers, layer, a dropout layer with probability 0.5, a batch which is similar to [25]. Specifically, a Bi-LSTM layer is used to get the hidden representations of 6. https://nlp.stanford.edu/projects/glove/ 7. https://catalog.ldc.upenn.edu/LDC2011T07/ 9. http://deeplearning.net/software/theano/ 8. https://nlp.stanford.edu/software/CRF-NER.html 10. https://lasagne.readthedocs.io/ ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 12

TABLE 2 TABLE 3 Name level F1 score of using the fine-grained annotations in different Token-level and Name-level results of the models evaluated on ways with different models on FinegrainedName. FinegrainedName

Fusion Token-level Name-level Annotations F Strategies Models Models R P F F BIE 41.15 CRF FML 54.98 JointNN 88.02 89.82 88.91 81.04 No FI 50.32 DeepNN 87.84 89.44 88.63 80.12 Fusion BIE 80.89 ConcatNN 87.79 89.44 88.61 80.12 Bi-LSTM-CRF FML 82.11 FI 81.71 CoAttNN 89.23 90.54 89.88 84.26 Early CRF BIE × FML × FI 28.14 CogNN (proposed) 91.08 91.74 91.41 87.04 Fusion Bi-LSTM-CRF BIE × FML × FI 62.65 BERT 93.17 94.55 93.85 92.76 BIE ∪ FML 56.23 IsBERT no inference 94.98 95.00 94.99 94.78 BIE ∪ FI 56.01 CRF IsBERT (proposed) 95.27 96.11 95.69 95.56 FML ∪ FI 56.38 Late BIE ∪ FML ∪ FI 57.10 Fusion BIE ∪ FML 83.12 BIE ∪ FI 83.08 before perform predictions. And the two tasks are Bi-LSTM-CRF FML ∪ FI 83.29 trained jointly to minimize the total loss. Bi-LSTM BIE ∪ FML ∪ FI 83.45 has a 100-dimensional hidden layer, a dropout layer 87.04 with probability 0.5, a batch size of 32, and an initial [BIE, FML] In-network CogNN 87.34 learning rate of 0.01 with a decay rate of 0.05. (proposed) Fusion 88.26 • CoAttNN: A variant of our proposed CogNN [BIE, FI] 88.80 model with co-attention mechanism but not gated-fusion mechanism. The implementation, training procedures and hyperparameters are the the input. The hidden representations are passed same as those for CogNN. • CogNN two task-specific output layers to predict name : Our proposed model. • BERT forms and name span respectively. All the output : The vanilla version of BERT model. We is further fed into a linear chain CRF layer before choose the “bert-base-case” as the pre-trained model, perform predictions. And the two tasks are trained which is trained on cased English text and contains jointly to minimize the total loss. Bi-LSTM has a 12-layer, 768-hidden-features for each layer and 100-dimensional hidden layer, a dropout layer with 110M trainable parameters in total. We set the probability 0.5, a batch size of 32, and an initial maximum input length as 512. • IsBERT without multi-hop learning rate of 0.01 with a decay rate of 0.05. : A variant of our proposed model IsBERT with bidirectional • DeepNN: We use two successively deeper layers similar to [26] for predicting the name form classes overlapped contextual embedding but not and the name spans respectively. Specifically, two multi-hop inferences. The training procedures Bi-LSTM layers are stacked for predicting the name and hyperparameters are the same as those for form classes and the name spans respectively. The IsBERT. • IsBERT output of the first Bi-LSTM becomes the input of the : Our proposed model. The training second Bi-LSTM. All the output is further fed into a procedures and hyperparameters are the same as linear chain CRF layer to predict the tokens labels. those for BERT except that we set the inferences And the two tasks are trained jointly to minimize number as 2 and overlapped ratio as 0.5 (256 tokens). • Ada-IsBERT the total loss. Each Bi-LSTM has a 100-dimensional : Our proposed model. The training hidden layer, a dropout layer with probability 0.5, a procedures and hyperparameters are the same as batch size of 32, and an initial learning rate of 0.01 those for IsBERT except that we dynamically choose with a decay rate of 0.05. the overlapped ratio from 0, 0.10, 0.20, 0.30, 0.40 and 0.50. • ConcatNN: We use the concatenating procedure similar to [27] to fuse the output of one task-specific All the optimal hyperparameters of above non BERT layer with the input of another task-specific layer. based deep learning models are obtained with standard Specifically, a Bi-LSTM layer is used to get the grid search on the same developing dataset with stochastic hidden representations of the input. The hidden gradient descent. We stop training if the accuracy does representations are passed to the first task-specific not improve in 10 epochs. All the BERT based models are output layer to predict name form classes. Then the implemented using Pytorch and are trained using Adam output of the name form prediction layer as well optimizer and learning rate decay, starting from 10−5. as the initial idden representations from Bi-LSTM Table 3 shows the results. JointNN achieves slightly layer are concatenated as the input of the second better results at the name level compared with the no task-specific output layer to predict name spans. All fusion Bi-LSTM-CRF model in Table 2. However, DeepNN the output is further fed into a linear chain CRF layer and ConcatNN are worse than no fusion Bi-LSTM-CRF. ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 13

TABLE 4 TABLE 5 Token-level and Name-level results of IsBERT with different overlapped Token-level results of IsBERT and Ada-IsBERT on different ratio and different numbers of inference modules on FinegrainedName abundances of context

Para- Token-level Name-level Models R P F meters IsBERT 79.57 85.77 82.55 R P F F Short Docs Only Overlapped 0.25 94.79 95.61 95.20 93.97 Ada-IsBERT 84.27 85.51 84.88 Ratio IsBERT 96.41 96.81 96.61 0.5 95.27 96.11 95.69 95.56 Long Docs Only 1 94.79 95.61 95.20 93.97 Ada-IsBERT 96.41 96.81 96.61 Inference IsBERT 95.04 96.09 95.56 Modules 2 95.27 96.11 95.69 95.56 Mix Length Docs 3 93.31 96.46 94.76 94.80 Ada-IsBERT 96.17 95.53 95.85

The reason is that DeepNN and ConcatNN do not Using two inference modules in IsBERT is optimal have a good mechanism to filter the learning signals. for our task, which supports our hypothesis that the DeepNN uses successively deeper layers to connect two multi-hop inference method can strengthen the inherent tasks and ConcatNN utilises straightforward concatenating signals learned from bidirectional overlapped contextual procedures to share the informations. Noisy signals embedding. However, with the number of layers growing, may be introduced into the training process and the the performance of the model drops and is even lower than propagation of error may reduce the effectiveness of our the one without using multi-hop inferences. The possible annotations. Both our proposed model CogNN and its reason is overfitting since the model passes one document variant CoAttNN outperform these multi-task models. for too many times. CogNN outperforms the best baseline by up to 5.47% Table 5 shows the performance of IsBERT and and 6.06% in F1-score at the token level and name Ada-IsBERT on documents with different abundances of level, respectively. This verifies the effectiveness of our context. We select documents with not more than 256 tokens co-attention and gated fusion mechanisms for utilising from the FinegrainedName dataset to form the collection the fine-grained annotations. CogNN performs better than of documents with low abundances of context (Short Docs CoAttNN since the sub-networks can share the learning Only). We select documents with more than 256 tokens signals while neither sub-network is overwhelmed by from the FinegrainedName dataset to form the collection misleading signals. of documents with high abundances of context (Long Docs All the BERT based model yield better results than GloVe Only). We use the original FinegrainedName dataset as based methods. The vanilla BERT model outperforms the the the collection of documents with both high and low best joint learning model CogNN based on LSTM-CRF abundances of context (Mix Length Docs). The results by 2.44% and 5.72% in F1-score at the token-level and shows that both IsBERT and Ada-IsBERT experience huge name-level respectively. The transformers inside BERT can performance drop on Short Docs compared with on Long better capture the context information compared with the Docs. The reason is that our models do not have too much traditional LSTM-CRF based approaches. Our proposed context to utilize and to boost the performance. Ada-IsBERT IsBERT model achieves the highest F1-score in both performs much better on Short Docs and Mix Length Docs token-level and name-level, which outperforms the CogNN than IsBERT since it can adjust the overlapped ratio to adapt model by 4.28% and 8.52% in the field of token and name to documents with different length, and avoid creating level respectively. In particular, the IsBERT model achieves repeated context and redundant embeddings in the model. a significant improvement in the name-level and maintain roughly consistent performance in both levels. IsBERT also 5.2 Applicability on Newswire Articles outperforms the naive BERT and IsBERT without multi-hop inferences, which indicates the effectiveness of our proposed While not the focus of this study, we further show the bidirectional overlapped contextual embedding learning applicability of our models on traditional newswire texts, and multi-hop inferences. which are different from academic homepages and mostly have consistent and complete syntax. We use the CoNLL-2003 dataset which contains 1,393 5.1.3 Influence of Parameters annotated English newswire articles that focus on four Table 4 shows the performance of IsBERT under different types of named entities: person, location, organisation and overlapped ratios and different numbers of inference miscellaneous entity. We use the training, developing, and modules on long documents. The experiments results meet testing datasets in CoNLL-2003 to train Stanford NER and our expectation for long documents as a higher overlapped IsBERT models with different annotations. The reported ratio leads to better model performance. Compared with improvements are statistically significant with p < 0.05 as an overlapped ratio of 0.25, the F1-score improves about calculated using McNemar’s test. 0.39% and 1.69% in token-level and name-level with an This dataset does not come with fine-grained overlapped ratio of 0.5. The reason is that more tokens are annotations. We add annotations using the same shared between chunks and better inter-sentence context is method described in Section 3 and compare the following captured. combinations of annotations: ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 14

TABLE 6 Token level performance of person name recognition on CoNLL-2003 using different models.

Fig. 11. Stanford NER on short text Models Annotations RPF PER 85.29 94.75 89.77 FML 83.66 93.36 88.25 Stanford CoNLL 92.43 89.96 91.18 NER FML+CoNLL 90.19 89.04 89.61 PER 94.78 92.82 93.79 Fig. 12. Ada-IsBERT on short text FML 95.27 96.11 95.69 IsBERT CoNLL 94.78 92.82 93.79 (proposed) FML+CoNLL 92.34 94.11 93.23

• PER: Using only PERSON labels. • FML: Using only FML labels. • CoNLL: Using all original labels in CoNLL-2003. • FML+CoNLL: Replacing PERSON by FML labels in CoNLL-2003.

From Table 6, we see that neural models perform better than the non-neural model, which is consistent with the results in Section 5.1.1. When providing extra ORG, LOC, Fig. 13. Stanford NER on well-formatted text and MISC annotations apart from PER to Stanford NER, the recall increases while the precision decreases. This indicates that the extra annotations help recognise more named entity tokens but may also misguide the model. In comparison, IsBERT is not impacted. When providing extra FML annotations apart from PER to Stanford NER the performance does not improve while that of IsBERT improves. Our improvements mainly lie in the precision, with a 3.29% improvement at the token level compared with the best baseline, which indicates that IsBERT can well distinguish person name tokens from others. These results also indicate that only applying the fine-grained name form annotations on newswire data for the existing models is not enough. Our proposed model is essential to make use of the extra name form information. Fig. 14. Ada-IsBERT on well-formatted text Overall, our approach is also applicable on formal English newswire articles and is especially helpful for improving the precision. However, the advantage of our 5.3 Error Analysis approach is smaller than that on the academic homepages. Some typical errors of the proposed model are investigated The main reason is that the name forms in newswire and analyzed in this section. Model predictions are articles are less flexible compared with those in academic highlighted using the background colour, while the homepages, which reduces the benefits of adding extra underline colour shows the ground truths. False negative name form information. and false positive predictions are noted by red crossline. Figure 11, 12, 13, 14 show the results of Stanford Errors in organization names: Figure 15 shows a typical NER and Ada-IsBERT on different cases. Model predictions error that the model labels the name of an organization as are highlighted using the background colour, while the a person name. Even though the organization is named by underline colour shows the ground truths. False negative a person “Ronald O. Perelman”, we expecte that the model and false positive predictions are noted by red crossline. We could distinguish organization names from person names can see from the Figure 13, 14 that our proposed model by the context information, such as the “Department of can not only recognize more person names than Stanford Emergency Medicine”. It is challenging for the model to NER, our model also provide fine-grained name forms with perform well in this situation. To solve this problem, we higher accuracy. We also test two models on short text, could enhence our trainng dataset by addding more training Figure 11, 12 show that our model also perfroms well on examples that containing organizations named by person text without too much context, and it can distinguish the names labelled as negative. We leave this for future works. same tokens with different meanings. Errors in well-formatted text: Figure 16 shows a typical ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 15

name recognition. We also propose a IsBERT to recognise fine-grained person names in long documents by capturing the inter-sentence context and implicit relations via an overlapped input processor, and an inter-sentence encoder with bidirectional overlapped contextual embedding learning and multi-hop inference mechanisms. We also propose an Ada-IsBERT which is robust for documents with diverse abundances of context by dynamically adjusting and Fig. 15. Errors in organization names the inter-sentence overlapping ratio according to the length of different documents. Experiments on FinegrainedName dataset and Newswire dataset show that our annotations can be utilised in different ways to improve the person name recognition performance in many application cases with different document length. Our CogNN model outperforms other NER models and multi-tasks models on utilising the fine-grained name form information. Our IsBERT model outperforms other models on capturing the inter-sentence context in long documents. Our Ada-IsBERT model is robuster for documents with different abundances of Fig. 16. Errors in well-formatted text context. All our models outperforms state-of-the-art NER models for person name recognition and is especially advantageous in having high precision. For future work, we plan to investigate the fine-grained annotations and the proposed models on other languages rather than English. We also plan to evaluate the performance of our approach on other types of datasets such as online forums and social media.

REFERENCES [1] J. Tang, J. Zhang, L. Yao, J. , L. Zhang, and Z. Su, “Arnetminer: extraction and mining of academic social networks,” in Proceedings Fig. 17. Example with enhancement data of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, 2008, pp. 990–998. [2] A. G. Ororbia II, J. , M. Khabsa, K. Williams, and C. L. Giles, error that the model does not recognize some person names “Big scholarly data in citeseerx: Information extraction from the web,” in Proceedings of the 24th International Conference on World in the well-formatted text, such as in the introductions Wide Web, 2015, pp. 597–602. or in the career descriptions. The reason is that, in the [3] B. D. Trisedya, G. Weikum, J. , and R. Zhang, “Neural relation FinegrainedName dataset, most person names appear in extraction for knowledge base enrichment.” in Proceedings of the Annual Meeting of the Association for Computational Linguistics the publications and come with different formats as those (Volume 1: Long Papers), 2019, pp. 229–240. the introductions. Without sufficient training data that [4] P. Barrio, G. Simoes,˜ H. Galhardas, and L. Gravano, “Reel: A containing person names in the well-formatted text, the relation extraction learning framework,” in Proceedings of the 14th model tends to miss these person names. To solve this ACM/IEEE Joint Conference on Digital Libraries, 2014, pp. 455–456. [5] X. Liu, Y. , C. Guo, Y. Sun, and L. Gao, “Full-text based problem, we could enhence our trainng dataset by adding context-rich heterogeneous network mining approach for citation more well-formatted text articles such as CONLL-2003 recommendation,” in Proceedings of the 14th IEEE/ACM Joint dataset. We leave this for future works. To solve this Conference on Digital Libraries, 2014, pp. 361–370. problem, we add 200 annotated news articles from the [6] Z. , W. , and K. Yu, “Bidirectional lstm-crf models for sequence tagging,” arXiv preprint arXiv:1508.01991, 2015. CONLL-2003 dataset to the training set to balance the [7] J. Chiu and E. Nichols, “Named entity recognition with training data. Figure 17 shows an example which is tested bidirectional lstm-cnns,” Transactions of the Association of on the well-formatted text after the data enhancement. Computational Linguistics, vol. 4, no. 1, pp. 357–370, 2016. [8] X. Ma and E. Hovy, “End-to-end sequence labeling via IsBERT recognize all the person names correctly. bi-directional lstm-cnns-crf,” in Proceedings of Annual Meeting of the Association for Computational Linguistics, 2016, pp. 1064–1074. [9] O. Felecan, Name and naming: synchronic and diachronic perspectives. 6 CONCLUSION Cambridge Scholars Publishing, 2012. We studied the person name recognition problem. We [10] J. Devlin, M.-W. , K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language propose a new name annotation scheme which gives understanding,” in Proceedings of the 2019 Conference of the North fine-grained name annotations and a new model called American Chapter of the Association for Computational Linguistics: CogNN to take advantage of the fine-grained annotation Human Language Technologies (NAACL/HLT 2019). Association for scheme via co-attention and gated fusion. We have also Computational Linguistics, 2019, pp. 4171–4186. [11] Y. Dai, R. Zhang, and J. Qi, “Person name recognition with created the first dataset under the fine-grained annotation fine-grained annotation,” in Proceedings of the ACM/IEEE Joint scheme, called FinegrainedName, for the research of Conference on Digital Libraries in 2020, 2020, pp. 243–252. ZHANG et al.: NAMEREC*: HIGHLY ACCURATE AND FINE-GRAINED PERSON NAME RECOGNITION 16

[12] D. Nadeau and S. Sekine, “A survey of named entity recognition machine translation system: Bridging the gap between human and and classification,” Lingvisticae Investigationes, vol. 30, no. 1, pp. machine translation,” arXiv preprint arXiv:1609.08144, 2016. 3–26, 2007. [34] J. R. Finkel, T. Grenager, and C. Manning, “Incorporating [13] C. Li, A. Sun, J. , and Q. He, “Tweet segmentation and non-local information into information extraction systems by its application to named entity recognition,” IEEE Transactions on gibbs sampling,” in Proceedings of Annual Meeting of the Association Knowledge and Data Engineering, vol. 27, no. 2, pp. 558–570, 2015. for Computational Linguistics, 2005, pp. 363–370. [14] F. Dugas and E. Nichols, “Deepnnner: Applying blstm-cnns and extended lexicons to named entity recognition in tweets,” in Proceedings of the Workshop on Noisy User-generated Text, 2016, pp. 178–187. [15] D. , L. Neves, V. Carvalho, N. Zhang, and H. , “Visual attention model for name tagging in multimodal social media,” in Proceedings of Annual Meeting of the Association for Computational Linguistics, 2018, pp. 1990–1999. [16] S. Moon, L. Neves, and V. Carvalho, “Multimodal named entity disambiguation for noisy social media posts,” in Proceedings of Annual Meeting of the Association for Computational Linguistics, 2018, pp. 2000–2008. [17] A. Borthwick, J. Sterling, E. Agichtein, and R. Grishman, “Exploiting diverse knowledge sources via maximum entropy in named entity recognition,” in Workshop on Very Large Corpora, 1998. [18] E. F. Sang and F. De Meulder, “Introduction to the conll-2003 shared task: Language-independent named entity recognition,” arXiv preprint cs/0306050, 2003. [19] A. Ghaddar and P. Langlais, “Winer: A wikipedia annotated corpus for named entity recognition,” in Proceedings of the International Joint Conference on Natural Language Processing, 2017, pp. 413–422. [20] Y. Zhang, J. Qi, R. Zhang, and C. Yin, “Pubse: A hierarchical model for publication extraction from academic homepages,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2018, pp. 1005–1010. [21] Y. Dai, J. Qi, and R. Zhang, “Joint recognition of names and publications in academic homepages,” in Proceedings of the 13th International Conference on Web Search and Data Mining, 2020, pp. 133–141. [22] T. L. Packer, J. F. Lutes, A. P. Stewart, D. W. Embley, E. K. Ringger, K. D. Seppi, and L. S. Jensen, “Extracting person names from diverse and noisy ocr text,” in Workshop on Analytics for Noisy Unstructured Text Data, 2010, pp. 19–26. [23] E. Minkov, R. C. Wang, and W. W. Cohen, “Extracting personal names from email: Applying named entity recognition to informal text,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2005, pp. 443–450. [24] M. Aboaoga and M. J. Ab Aziz, “Arabic person names recognition by using a rule based approach,” Journal of Computer Science, vol. 9, no. 7, pp. 922–927, 2013. [25] R. Caruana, “Multitask learning,” Machine learning, vol. 28, no. 1, pp. 41–75, 1997. [26] A. Søgaard and Y. Goldberg, “Deep multi-task learning with low level tasks supervised at lower layers,” in Proceedings of Annual Meeting of the Association for Computational Linguistics, 2016, pp. 231–235. [27] D. Ma, S. Li, and H. Wang, “Joint learning for targeted sentiment analysis,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2018, pp. 4737–4742. [28] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all need,” in Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS 2017), 2017, pp. 5998–6008. [29] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. , “Biobert: a pre-trained biomedical language representation model for biomedical text mining,” Bioinformatics, vol. 36, no. 4, pp. 1234–1240, 2020. [30] X. , Y. Yin, L. , X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, “Tinybert: Distilling bert for natural language understanding,” arXiv preprint arXiv:1909.10351, 2019. [31] J. Pennington, R. Socher, and C. Manning, “Glove: Global vectors for word representation,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2014, pp. 1532–1543. [32] C. Dyer, M. Ballesteros, W. , A. Matthews, and N. A. Smith, “Transition-based dependency parsing with stack long short-term memory,” arXiv preprint arXiv:1505.08075, 2015. [33] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey et al., “Google’s neural