arXiv:1807.00571v1 [cs.CL] 2 Jul 2018 a nt,o lseigtcnqe o essand senses for techniques clustering or units, phraseologi- cal relations, and examples modeling glosses, and as such definitions , like direction, phenomena linguistic this in exist- attempts summarize ing will we Additionally, NLP. and resources topics lexical between these interplay the in concerning challenges open research present and will mainte- we problems tutorial and this In creation efforts. reduce nance effec- a to be becoming can leveraged is techniques tively process NLP this tech- practice. up standard language ease of to use nologies the less- and languages, domains resourced knowledge restricted most the in a even is and process, resources time-consuming lexical prohibitively of construction manual The Description 1 oeCmcoClao,Li Espinosa-Anke Luis Camacho-Collados, Jose l.Tesie ftettra r vial at available are tutorial the of slides mod- The and systems lexical els. their of improve knowledge would to the resources who from NLP benefit the to in on up like researchers and easing hand, curation; and/or other resource speeding of process of the goal end ac- the with becoming techniques, NLP in automatic re- with interested quainted language are on who working sources those from benefit tutorial: can this audience of types main lexi- applications. Two of NLP improving use for the resources cal and resources en- and lexical constructing meth- riching of process NLP the help of to use ods the perspectives: address method- ological two to from proposes goals tutorial complementary straightfor- these This a not task. is ward resources the lexical knowl- in At this edge storing and challenges. processing time, sev- and same with problems area, open research eral important cur- an is rently systems AI/NLP common into knowledge and sense world linguistic, Incorporating https://bitbucket.org/luisespinosa/lr-nlp/ colo optrSineadInformatics and Science Computer of School [email protected] AC 08Ttra h nepa ewe eia Resourc Lexical between Interplay The – Tutorial 2018 NAACL [email protected] adf University Cardiff Abstract n aua agaeProcessing Language Natural and . efcosadr eia eorefrEnglish, for ( the resource WordNet from lexical i.e., starting standard facto resources, de lexical current of future. near the towards in view scenarios a showpotential and far, to so interacted is have sources Re- tutorial Lexical and this Processing context, Language Natural of this how In motivation by main techniques. exploited the NLP better of and use resources enriched making these further time, be same downstream can the many the At in to obtained due applications. be challenge can that important gains an currently systems AI/NLP is into knowledge explicit Adding Motivation and Introduction 2.1 Outline 2 NLP downstream applications. improving en- for this architectures neu- into ral of resources integration lexical from the knowledge coded on present work briefly recent resources will some we lexical Finally, between bridge applications. and a as act techniques field flexible which as the embeddings, particu- sense present in lar will representations, we knowledge-based Moreover, of data. and resources text lexical connecting in role play decisive which a tasks these for knowledge-based methods supervised some and explain will re- We lexical from sources. knowledge of integration direct in enable a tasks also which key understanding as language natural Linking, Entity Dis- and Sense explain Word ambiguation in will challenges current we the of concerned, some is applications NLP of resources of integration the nature. different as well as topics, yst r n o h eorecnb viewed be can resource what the how showing and WordNet, are of synsets overview concise a sa nrdcinw rtpeeta overview an present first we introduction an As in resources lexical of integration the as far As oamdThrPilehvar Taher Mohammad agaeTcnlg Lab Technology Language nvriyo Cambridge of University [email protected] Fellbaum , 1998 .W provide We ). es as a . We then briefly dis- approaches exploiting linguistic knowledge cuss some of the limitations of WordNet and dis- (Hulth, 2003). cuss how these can be alleviated to some ex- tent with the help of collaboratively-constructed 2. Definition extraction. Techniques resources, such as Freebase (Bollacker et al., for extracting definitional text snip- 2008), Wikidata (Vrandeˇci´c, 2012) and Babel- pets from corpora (Navigli and Velardi, Net (Navigli and Ponzetto, 2012). As the main 2010; Boella and Di Caro, 2013; building block of these resources, we show Espinosa-Anke et al., 2015; Li et al., 2016; how collaboratively-constructed projects, such as Espinosa-Anke and Schockaert, 2018). Wikipedia1 and Wiktionary2, can serve as mas- sive multilingual sources of lexical information. 3. Automatic extraction of examples. De- The lexical resources session is concluded by scription of example extraction techniques a short introduction to the Paraphrase Database and designs on this direction, e.g., the (PPDB) (Ganitkevitch et al., 2013; Pavlick et al., GDEX criteria and their implementation 2015) and to a domain-specific : (Kilgarriff et al., 2008). SNOMED3, which is one of the major ontologies for the medical domain. 4. . Recent ap- The tutorial is then divided in two main blocks. proaches for extracting semantic relations First, we delve into NLP for Creation and Enrich- from text: NELL (Carlson et al., 2010), ment of Lexical Resources, where we address a ReVerb (Fader et al., 2011), PATTY range of NLP problems aimed specifically at im- (Nakashole et al., 2012), KB-Unify proving repositories of linguistically expressible (Delli Bovi et al., 2015). knowledge. Second, we cover different use cases in which Lexical Resources for NLP have been 5. Hypernym discovery and taxonomy leveraged successfully. The last part of the tutorial learning. Insights from recent SemEval focuses on lessons learned from work in which we tasks (Bordea et al., 2015, 2016) and re- tried to reconcile both worlds, as well as our own lated efforts on the automatic extraction view towards what the future holds for knowledge- of hypernymy relations from text corpora based approaches to NLP. (Velardi et al., 2013; Alfarone and Davis, 2015; Flati et al., 2016; Shwartz et al., 2016; 2.2 NLP for Lexical Resources Espinosa-Anke et al., 2016; Gupta et al., The application of language technologies to the 2016). automatic construction and extension of lexical re- 6. Topic/domain clustering techniques. sources has proven successful in that it has pro- Relevant techniques for filtering gen- vided various tools for optimizing this often pro- eral domain resources via topic grouping hibitively costly and expensive process. NLP tech- (Roget, 1911; Navigli and Velardi, 2004; niques provide end-to-end technologies that can Camacho-Collados and Navigli, 2017). tackle all challenges in the cre- ation and maintenance pipeline. In this tutorial 7. Alignment of lexical resources4. Align- we summarize existing efforts in this direction, in- ment of heterogeneous lexical resources cluding the extraction from text of linguistic phe- contributing to the creation of large re- nomena like terminology, definitions and glosses, sources containing different sources of examples and relations, as well as clustering tech- knowledge. We will present approaches niques for senses and topics. for the construction of such resources, 1. Terminology extraction. Measures for such as Yago (Suchanek et al., 2007), terminology extraction, the simple conven- UBY (Gurevych et al., 2012), BabelNet tional tf-idf (Sparck Jones, 1972), lexical (Navigli and Ponzetto, 2012) or Con- specificity (Lafon, 1980), and more recent ceptNet (Speer et al., 2017), as well as other works attempting to improve the 1https://www.wikipedia.org/ 2https://www.wiktionary.org/ 4Due to time constraints, items 7 and 8 were not presented 3https://www.snomed.org/ during the tutorial. automatic procedures to align lexical re- Camacho-Collados et al., 2016; Mancini et al., sources (Matuschek and Gurevych, 2013; 2017). Pilehvar and Navigli, 2014). Finally, we briefly present a few successful approaches integrating knowledge-based repre- 8. Ontology enrichment. Enriching lexical on- sentations into downstream tasks such as senti- tologies with novel concepts or with addi- ment analysis (Flekova and Gurevych, 2016), lex- tional relations (Jurgens and Pilehvar, 2016). ical substitution (Cocos et al., 2017) or visual ob- ject discovery (Young et al., 2017). As a case 2.3 Lexical Resources for NLP study, we present an analysis on the integra- In addition to the (semi)automatic efforts for eas- tion of knowledge-based embeddings into neu- ing the task of constructing and enriching lexi- ral architectures via WSD for text classification cal resources presented in the previous section, (Pilehvar et al., 2017), discussing its potential and we present NLP tasks in which lexical resources current open challenges. have shown an important contribution. Effec- tively leveraging linguistically expressible cues 2.4 Open problems and challenges with their associated knowledge remains a difficult In this last section we introduce some of the open task. Knowledge may be extracted from roughly problems and challenges for automatizing the re- three types of resource (Hovy et al., 2013): un- source creation and enrichment process as well as structured, e.g. text corpora; semistructured, for the integration of knowledge from lexical re- such as encyclopedic collaborative repositories sources into NLP applications. like Wikipedia, or structured, which include lex- icographic resources like WordNet. In this sec- Instructors tion we present some of the applications on which Jose Camacho Collados is a Research Asso- different kinds of lexical resource (including their ciate at Cardiff University. Previously he was a combination) play an important role. Google Doctoral Fellow and completed his PhD We begin this section by explaining some at Sapienza University of Rome. His research fo- of the problems and challenges in Word Sense cuses on Natural Language Processing and, more Disambiguation and Entity Linking, as key specifically, on the area of lexical and distribu- tasks in natural language understanding which tional semantics. Jose has experience in utiliz- enable the direct integration of knowledge from ing lexical resources for NLP applications, while lexical resources. We describe the most relevant enriching and improving these resources by ex- knowledge-based WSD systems, both based on tracting and processing knowledge from textual definitions (Lesk, 1986; Banerjee and Pedersen, data. On this area he has co-organized the Se- 2003; Basile et al., 2014) and graph-based mEval 2018 shared task on Hypernym Discovery. (Agirre et al., 2014; Moro et al., 2014); and super- Previously, he co-organized a workshop on Sense, vised, both linear models (Zhong and Ng, 2010; Concept and Entity Representations and their Ap- Iacobacci et al., 2016) and the most recent branch plications at EACL 2017 and a tutorial on the same exploiting neural networks (Melamud et al., 2016; topic at ACL 2016. His background education in- Raganato et al., 2017b). We present an analysis of cludes an Erasmus Mundus Master in Natural Lan- the main advantages and limitations of each kind guage Processing and Human Language Technol- of approach (Raganato et al., 2017a). ogy and a 5-year BSc degree in Mathematics. Then, we summarize the field of knowledge- based representations, in particular sense vectors Luis Espinosa Anke received his BA in English and embeddings, as flexible techniques con- Philology in 2006 (Univ. of Alicante, Spain), and necting lexical resources and downstream his PhD in Natural Language Processing in 2017 applications. We first present techniques which (Univ. Pompeu Fabra, Spain). He holds two MAs, leverage WordNet as main source of knowledge one in English-Spanish (Univ. of Al- (Chen et al., 2014; Rothe and Sch¨utze, 2015; icante), and an Erasmus Mundus MA in Natural Jauhar et al., 2015; Johansson and Pina, 2015; Language Processing (NLP) (Univ. of Wolver- Pilehvar and Collier, 2016) and also present other hampton and Univ. Autonoma de Barcelona). His techniques exploiting multilingual resources such research interests lie in the intersection between as Wikipedia or BabelNet (Iacobacci et al., 2015; structured representations of knowledge and NLP, specifically computational lexicography and dis- SIGMOD international conference on Management tributional semantics. He has co-organized the Se- of data, pages 1247–1250. ACM. mEval 2018 shared tasks on Hypernym Discovery Georgeta Bordea, Paul Buitelaar, Stefano Faralli, and Multilingual Emoji Prediction. Previously, he and Roberto Navigli. 2015. Semeval-2015 task co-organized the Spanish NLP conference (2014) 17: Taxonomy extraction evaluation (texeval). In and the Focused NER task (Open Knowledge Ex- SemEval@NAACL-HLT. traction challenge) at ESWC 2017. Georgeta Bordea, Els Lefever, and Paul Buitelaar. 2016. Semeval-2016 task 13: Taxonomy extrac- Mohammad Taher Pilehvar is a Research As- tion evaluation (texeval-2). In Proceedings of the sociate at the University of Cambridge. Taher’s 10th International Workshop on Semantic Evalua- research lies in lexical semantics, mainly focus- tion (SemEval-2016), pages 1081–1091. ing on semantic representation and similarity. In Jose Camacho-Collados and Roberto Navigli. 2017. the past, he has co-instructed three tutorials on BabelDomains: Large-Scale Domain Labeling of these topics (EMNLP 2015, ACL 2016, and EACL Lexical Resources. In Proceedings of EACL (2), Va- 2017) and co-organised three SemEval tasks. He lencia, Spain. has also co-authored several conference (includ- Jos´eCamacho-Collados, Mohammad Taher Pilehvar, ing two ACL best paper nominations, at 2013 and and Roberto Navigli. 2016. Nasari: Integrating ex- 2017) and journal papers, including different se- plicit knowledge and corpus statistics for a multilin- gual representation of concepts and entities. Artifi- mantic representation techniques based on hetero- cial Intelligence, 240:36–64. geneous lexical resources. Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R Hruschka Jr, and Tom M References Mitchell. 2010. Toward an architecture for never- ending language learning. In AAAI, volume 5, Eneko Agirre, Oier Lopez de Lacalle, and Aitor Soroa. page 3. Atlanta. 2014. Random walks for knowledge-based word sense disambiguation. Computational Linguistics, Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. 40(1):57–84. A unified model for word sense representation and disambiguation. In Proceedings of EMNLP, pages Daniele Alfarone and Jesse Davis. 2015. Unsupervised 1025–1035, Doha, Qatar. learning of an is-a taxonomy from a limited domain- specific corpus. In Proceedings of the 24th Inter- Anne Cocos, Marianna Apidianaki, and Chris Callison- national Joint Conference on Artificial Intelligence, Burch. 2017. Word sense filtering improves Proceed- pages 1434–1441. AAAI Press. embedding-based lexical substitution. In ings of the 1st Workshop on Sense, Concept and En- Satanjeev Banerjee and Ted Pedersen. 2003. Extended tity Representations and their Applications, pages gloss overlap as a measure of semantic relatedness. 110–119. Association for Computational Linguis- In Proceedings of the 18th International Joint Con- tics. ference on Artificial Intelligence, pages 805–810, Claudio Delli Bovi, Luis Espinosa-Anke, and Roberto Acapulco, Mexico. Navigli. 2015. Knowledge base unification via sense embeddings and disambiguation. In Proceed- Pierpaolo Basile, Annalina Caputo, and Giovanni Se- ings of EMNLP, pages 726–736. Association for meraro. 2014. An Enhanced Lesk Word Sense Dis- Computational Linguistics. ambiguation Algorithm through a Distributional Se- mantic Model. In Proceedings of COLING 2014, Luis Espinosa-Anke, Jose Camacho-Collados, Claudio the 25th International Conference on Computational Delli Bovi, and Horacio Saggion. 2016. Supervised Linguistics: Technical Papers, pages 1591–1600, distributional hypernym discovery via domain adap- Dublin, Ireland. tation. In Proceedings of EMNLP, pages 424–435. Guido Boella and Luigi Di Caro. 2013. Extracting def- Luis Espinosa-Anke, Horacio Saggion, and Francesco initions and hypernym relations relying on syntactic Ronzano. 2015. Weakly supervised definition ex- dependencies and support vector machines. In Pro- traction. In Proceedings of the International Con- ceedings of the 51st Annual Meeting of the Associa- ference Recent Advances in Natural Language Pro- tion for Computational Linguistics (Volume 2: Short cessing, pages 176–185. Papers), volume 2, pages 532–537. Luis Espinosa-Anke and Steven Schockaert. 2018. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Syntactically aware neural architectures for defini- Sturge, and Jamie Taylor. 2008. Freebase: a collab- tion extraction. In Proceedings of the 2018 Confer- oratively created graph database for structuring hu- ence of the North American Chapter of the Associ- man knowledge. In Proceedings of the 2008 ACM ation for Computational Linguistics: Human Lan- guage Technologies, Volume 2 (Short Papers), vol- Sujay Kumar Jauhar, Chris Dyer, and Eduard Hovy. ume 2, pages 378–385. 2015. Ontologically grounded multi-sense repre- sentation learning for semantic vector space models. Anthony Fader, Stephen Soderland, and Oren Etzioni. In Proceedings of NAACL, pages 683–693, Denver, 2011. Identifying relations for open information ex- Colorado. traction. In Proceedings of the conference on empir- ical methods in natural language processing, pages Richard Johansson and Luis Nieto Pina. 2015. Em- 1535–1545. Association for Computational Linguis- bedding a semantic network in a word space. In tics. Proceedings of NAACL, pages 1428–1433, Denver, Colorado. Christiane Fellbaum, editor. 1998. WordNet: An Elec- tronic Database. MIT Press, Cambridge, MA. David Jurgens and Mohammad Taher Pilehvar. 2016. Semeval-2016 task 14: Semantic taxonomy enrich- Tiziano Flati, Daniele Vannella, Tommaso Pasini, and ment. In Proceedings of the 10th International Roberto Navigli. 2016. Multiwibi: The multilin- Workshop on Semantic Evaluation (SemEval-2016), gual wikipedia bitaxonomy project. Artificial Intel- pages 1092–1102. ligence, 241:66–102. Adam Kilgarriff, Milos Hus´ak, Katy McAdam, Lucie Flekova and Iryna Gurevych. 2016. Supersense Michael Rundell, and Pavel Rychl`y. 2008. Gdex: embeddings: A unified model for supersense inter- Automatically finding good dictionary examples in pretation, prediction, and utilization. In Proceedings a corpus. In Proc. Euralex. of ACL. Pierre Lafon. 1980. Sur la variabilit´ede la fr´equence Juri Ganitkevitch, Benjamin Van Durme, and Chris des formes dans un corpus. Mots, 1:127–165. Callison-Burch. 2013. Ppdb: The paraphrase database. In Proceedings of NAACL-HLT, pages Michael Lesk. 1986. Automatic sense disambiguation 758–764. using machine readable dictionaries: How to tell a pine cone from an ice cream cone. In Proceedings Amit Gupta, Francesco Piccinno, Mikhail of the 5th Annual Conference on Systems Documen- Kozhevnikov, Marius Pasca, and Daniele Pighin. tation, Toronto, Ontario, Canada, pages 24–26. 2016. Revisiting taxonomy induction over wikipedia. In Proceedings of COLING 2016, the Qi Li, Tianshi Li, and Baobao Chang. 2016. Multi- 26th International Conference on Computational phase word sense embedding learning using a cor- Linguistics: Technical Papers, Osaka, Japan, pus and a lexical ontology. CoRR, abs/1606.04835. December 11-17 2016, EPFL-CONF-227401, pages 2300–2309. Massimiliano Mancini, Jose Camacho-Collados, Igna- cio Iacobacci, and Roberto Navigli. 2017. Embed- Iryna Gurevych, Judith Eckle-Kohler, Silvana Hart- ding and senses together via joint knowledge- mann, Michael Matuschek, Christian M Meyer, and enhanced training. In Proceedings of CoNLL, pages Christian Wirth. 2012. Uby: A large-scale unified 100–111, Vancouver, Canada. lexical-semantic resource based on lmf. In Proceed- ings of the 13th Conference of the European Chap- Michael Matuschek and Iryna Gurevych. 2013. ter of the Association for Computational Linguistics, Dijkstra-WSA: A graph-based approach to word pages 580–590. Association for Computational Lin- sense alignment. Transactions of the Association for guistics. Computational Linguistics (TACL), 1:151–164. Eduard H. Hovy, Roberto Navigli, and Simone Paolo Oren Melamud, Jacob Goldberger, and Ido Dagan. Ponzetto. 2013. Collaboratively built semi- 2016. context2vec: Learning generic context em- structured content and Artificial Intelligence: The bedding with bidirectional lstm. In Proceedings story so far. Artificial Intelligence, 194:2–27. of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 51–61, Berlin, Anette Hulth. 2003. Improved automatic keyword ex- Germany. traction given more linguistic knowledge. In Pro- ceedings of the 2003 conference on Empirical meth- Andrea Moro, Alessandro Raganato, and Roberto Nav- ods in natural language processing, pages 216–223. igli. 2014. Entity Linking meets Word Sense Disam- Association for Computational Linguistics. biguation: a Unified Approach. Transactions of the Association for Computational Linguistics (TACL), Ignacio Iacobacci, Mohammad Taher Pilehvar, and 2:231–244. Roberto Navigli. 2015. Sensembed: Learning sense embeddings for word and relational similarity. In Ndapandula Nakashole, Gerhard Weikum, and Fabian Proceedings of ACL, pages 95–105, Beijing, China. Suchanek. 2012. Patty: a taxonomy of relational patterns with semantic types. In Proceedings of Ignacio Iacobacci, Mohammad Taher Pilehvar, and the 2012 Joint Conference on Empirical Methods Roberto Navigli. 2016. Embeddings for word sense in Natural Language Processing and Computational disambiguation: An evaluation study. In Proceed- Natural Language Learning, pages 1135–1145. As- ings of ACL, pages 897–907, Berlin, Germany. sociation for Computational Linguistics. Roberto Navigli and Simone Paolo Ponzetto. 2012. Robert Speer, Joshua Chin, and Catherine Havasi. BabelNet: The automatic construction, evaluation 2017. Conceptnet 5.5: An open multilingual graph and application of a wide-coverage multilingual se- of general knowledge. In Proceedings of the AAAI mantic network. Artificial Intelligence, 193:217– Conference on Artificial Intelligence, pages 4444– 250. 4451. Roberto Navigli and Paola Velardi. 2004. Learn- Fabian M. Suchanek, Gjergji Kasneci, and Gerhard ing domain ontologies from document warehouses Weikum. 2007. YAGO: A core of semantic knowl- and dedicated websites. Computational Linguistics, edge. In Proc. of WWW-07, pages 697–706. 30(2). Paola Velardi, Stefano Faralli, and Roberto Navigli. Roberto Navigli and Paola Velardi. 2010. Learning 2013. Ontolearn reloaded: A graph-based algorithm Word-Class Lattices for definition and hypernym ex- for taxonomy induction. Computational Linguistics, traction. In Proceedings of ACL 2010, pages 1318– 39(3):665–707. 1327, Uppsala, Sweden. Denny Vrandeˇci´c. 2012. Wikidata: A New Platform Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, for Collaborative Data Collection. In Proceedings Benjamin Van Durme, and Chris Callison-Burch. of WWW, pages 1063–1064. 2015. Ppdb 2.0: Better paraphrase ranking, fine- grained entailment relations, word embeddings, and Jay Young, Lars Kunze, Valerio Basile, Elena Cabrio, style classification. In Proceedings of ACL(2), pages Nick Hawes, and Barbara Caputo. 2017. Semantic 425–430, Beijing, China. Association for Computa- web-mining and deep vision for lifelong object dis- tional Linguistics. covery. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 2774– Mohammad Taher Pilehvar, Jose Camacho-Collados, 2779. IEEE. Roberto Navigli, and Nigel Collier. 2017. Towards a Seamless Integration of Word Senses into Down- Zhi Zhong and Hwee Tou Ng. 2010. It Makes Sense: stream NLP Applications. In Proceedings of ACL, A wide-coverage Word Sense Disambiguation sys- Vancouver, Canada. tem for free text. In Proceedings of the ACL System Demonstrations, pages 78–83, Uppsala, Sweden. Mohammad Taher Pilehvar and Nigel Collier. 2016. De-conflated semantic representations. In Proceed- ings of EMNLP, pages 1680–1690, Austin, TX. Mohammad Taher Pilehvar and Roberto Navigli. 2014. A robust approach to aligning heterogeneous lexical resources. In Proceedings of ACL, pages 468–478. Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017a. Word sense disambigua- tion: A unified evaluation framework and empirical comparison. In Proceedings of EACL, pages 99– 110, Valencia, Spain. Alessandro Raganato, Claudio Delli Bovi, and Roberto Navigli. 2017b. Neural sequence learning models for word sense disambiguation. In Proceedings of the 2017 Conference on Empirical Methods in Nat- ural Language Processing, pages 1156–1167. Asso- ciation for Computational Linguistics. Peter M. Roget. 1911. Roget’s International Thesaurus (1st edition). Cromwell, New York, USA. Sascha Rothe and Hinrich Sch¨utze. 2015. Autoex- tend: Extending word embeddings to embeddings for synsets and lexemes. In Proceedings of ACL, pages 1793–1803, Beijing, China. Vered Shwartz, Yoav Goldberg, and Ido Dagan. 2016. Improving hypernymy detection with an inte- grated path-based and distributional method. arXiv preprint arXiv:1603.06076. Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11–21.