KLUE: Korean Language Understanding Evaluation

Total Page:16

File Type:pdf, Size:1020Kb

KLUE: Korean Language Understanding Evaluation KLUE: Korean Language Understanding Evaluation Sungjoon Park* Jihyung Moon* Sungdong Kim* Won Ik Cho* Upstage, KAIST Upstage NAVER AI Lab Seoul National University [email protected] [email protected] [email protected] [email protected] Jiyoon Hany Jangwon Park Chisung Song Junseong Kim Yonsei University [email protected] [email protected] Scatter Lab [email protected] [email protected] Yongsook Song Taehwan Ohy Joohong Lee Juhyun Ohy KyungHee University Yonsei University Scatter Lab Seoul National University [email protected] [email protected] [email protected] [email protected] Sungwon Lyu Younghoon Jeong Inkwon Lee Sangwoo Seo Dongjun Lee Kakao Enterprise Sogang University Sogang University Scatter Lab [email protected] [email protected] [email protected] [email protected] [email protected] Hyunwoo Kim Myeonghwa Lee Seongbo Jang Seungwon Do Sunkyoung Kim Seoul National University KAIST Scatter Lab [email protected] KAIST [email protected] [email protected] [email protected] [email protected] Kyungtae Lim Jongwon Lee Kyumin Park Jamin Shin Seonghyun Kim Hanbat National University [email protected] KAIST Riiid AI Research [email protected] [email protected] [email protected] [email protected] Lucy Park Alice Oh** Jung-Woo Ha** Kyunghyun Cho** Upstage KAIST NAVER AI Lab New York University [email protected] [email protected] [email protected] [email protected] Abstract We introduce Korean Language Understanding Evaluation (KLUE) benchmark. KLUE is a collection of 8 Korean natural language understanding (NLU) tasks, including Topic Classification, Semantic Textual Similarity, Natural Language Inference, Named Entity Recognition, Relation Extraction, Dependency Parsing, Machine Reading Comprehension, and Dialogue State Tracking. We build all of the tasks from scratch from diverse source corpora while respecting copyrights, to ensure accessibility for anyone without any restrictions. With ethical considerations in mind, we carefully design annotation protocols. Along with the benchmark tasks and data, we provide suitable evaluation metrics and fine-tuning recipes for pretrained language models for each task. We furthermore release arXiv:2105.09680v3 [cs.CL] 11 Jun 2021 the pretrained language models (PLM), KLUE-BERT and KLUE-RoBERTa, to help reproducing baseline models on KLUE and thereby facilitate future research. We make a few interesting ob- servations from the preliminary experiments using the proposed KLUE benchmark suite, already demonstrating the usefulness of this new benchmark suite. First, we find KLUE-RoBERTaLARGE outperforms other baselines, including multilingual PLMs and existing open-source Korean PLMs. Second, we see minimal degradation in performance even when we replace personally identifiable information from the pretraining corpus, suggesting that privacy and NLU capability are not at odds with each other. Lastly, we find that using BPE tokenization in combination with morpheme-level pre-tokenization is effective in tasks involving morpheme-level tagging, detection and generation. In addition to accelerating Korean NLP research, our comprehensive documentation on creating KLUE will facilitate creating similar resources for other languages in the future. KLUE is available at https://klue-benchmark.com/. *Equal Contribution. A description of each author’s contribution is available at the end of paper. **Corresponding Authors. yWork done at Upstage. Contents 1 Introduction 4 1.1 Summary . .4 2 Source Corpora 8 2.1 Corpora Selection Criteria . .8 2.2 Selected Corpora . .9 2.2.1 Potential Concerns . 11 2.3 Preprocessing . 11 2.4 Task Assignment . 12 3 KLUE Benchmark 13 3.1 Topic Classification (TC) . 13 3.1.1 Dataset Construction . 13 3.1.2 Evaluation Metric . 14 3.1.3 Related Work . 14 3.1.4 Conclusion . 15 3.2 Semantic Textual Similarity (STS) . 16 3.2.1 Dataset Construction . 16 3.2.2 Evaluation Metrics . 18 3.2.3 Related Work . 19 3.2.4 Conclusion . 19 3.3 Natural Language Inference (NLI) . 20 3.3.1 Dataset Construction . 20 3.3.2 Evaluation Metric . 23 3.3.3 Related Work . 23 3.3.4 Conclusion . 24 3.4 Named Entity Recognition (NER) . 25 3.4.1 Dataset Construction . 25 3.4.2 Evaluation Metrics . 26 3.4.3 Related Work . 27 3.4.4 Conclusion . 27 3.5 Relation Extraction (RE) . 28 3.5.1 Data Construction . 28 3.5.2 Evaluation Metrics . 31 3.5.3 Related Work . 32 3.5.4 Conclusion . 32 3.6 Dependency Parsing (DP) . 33 3.6.1 Dataset Construction . 33 3.6.2 Evaluation Metrics . 34 2 3.6.3 Related Work . 36 3.6.4 Conclusion . 36 3.7 Machine Reading Comprehension (MRC) . 37 3.7.1 Dataset Construction . 37 3.7.2 Evaluation Metrics . 40 3.7.3 Analysis . 41 3.7.4 Related Work . 42 3.7.5 Conclusion . 43 3.8 Dialogue State Tracking (DST) . 44 3.8.1 Dataset Construction . 44 3.8.2 Evaluation Metrics . 48 3.8.3 Analysis . 48 3.8.4 Related Work . 48 3.8.5 Conclusion . 49 4 Pretrained Language Models 50 4.1 Language Models . 50 4.2 Existing Language Models . 52 5 Fine-tuning Language Models 52 5.1 Task-Specific Architectures . 52 5.1.1 Single Sentence Classification . 53 5.1.2 Sentence Pair Classification / Regression . 53 5.1.3 Multiple-Sentence Slot-Value Prediction . 53 5.1.4 Sequence Tagging . 54 5.2 Fine-Tuning Configurations . 55 5.3 Evaluation Results . 55 5.4 Analysis of Models . 55 6 Ethical Considerations 57 6.1 Copyright and Accessibility . 57 6.2 Toxic Content . 57 6.3 Personally Identifiable Information . 58 7 Related Work 58 8 Discussion 59 9 Conclusion 60 Index 74 A Dev Set Results 76 3 1 Introduction A major factor behind recent success of pretrained language models, such as BERT [30] and its variants [82, 22, 49] as well as GPT-3 [110] and its variants [111, 76, 9], has been the availability of well-designed benchmark suites for evaluating their effectiveness in natural language understanding (NLU). GLUE [133] and SuperGLUE [132] are representative examples of such suites and were designed to evaluate diverse aspects of NLU, including syntax, semantics and pragmatics. The research community has embraced GLUE and SuperGLUE, and has made rapid progress in developing better model architectures as well as learning algorithms for NLU. The success of GLUE and SuperGLUE has sparked interest in building such a standardized benchmark suite for other languages, in order to better measure the progress in NLU in languages beyond English. Such efforts have been pursued along two directions. First, various groups in the world have independently created language-specific benchmark suites; a Chinese version of GLUE (CLUE [142]), a French version of GLUE (FLUE [72]), an Indonesian variant [137], an Indic version [57] and a Russian variant of SuperGLUE [125]. On the other hand, some have relied on both machine and human translation of existing benchmark suites for building multilingual version of the benchmark suites which were often created initially in English. These include for instance XGLUE [78] and XTREME [54]. Although the latter approach scales much better than the former does, the latter often fails to capture societal aspects of NLU and also introduces various artifacts arising from translation. To this end, we build a new benchmark suite for evaluating NLU in Korean which is the 13-th most used language in the world according to [34] but lacks a unified benchmark suite for NLU. Instead of starting from existing benchmark tasks or corpora, we build this benchmark suite from ground up by determining and collecting base corpora, identifying a set of benchmark tasks, designing appropriate annotation protocols and finally validating collected annotation. This allows us to preemptively address and avoid properties that may have undesirable consequences, such as copyright infringement, annotation artifacts, social biases and privacy violations. In the rest of this section, we summarize a series of decisions and principles that went behind creating KLUE. 1.1 Summary In designing the Korean Language Understanding Evaluation (KLUE), we aim to make KLUE; 1) cover diverse.
Recommended publications
  • Preparing for the Possibility of a North Korean Collapse
    CHILDREN AND FAMILIES The RAND Corporation is a nonprofit institution that EDUCATION AND THE ARTS helps improve policy and decisionmaking through ENERGY AND ENVIRONMENT research and analysis. HEALTH AND HEALTH CARE This electronic document was made available from INFRASTRUCTURE AND www.rand.org as a public service of the RAND TRANSPORTATION Corporation. INTERNATIONAL AFFAIRS LAW AND BUSINESS NATIONAL SECURITY Skip all front matter: Jump to Page 16 POPULATION AND AGING PUBLIC SAFETY SCIENCE AND TECHNOLOGY Support RAND Purchase this document TERRORISM AND HOMELAND SECURITY Browse Reports & Bookstore Make a charitable contribution For More Information Visit RAND at www.rand.org Explore the RAND National Security Research Division View document details Limited Electronic Distribution Rights This document and trademark(s) contained herein are protected by law as indicated in a notice appearing later in this work. This electronic representation of RAND intellectual property is provided for non-commercial use only. Unauthorized posting of RAND electronic documents to a non-RAND website is prohibited. RAND electronic documents are protected under copyright law. Permission is required from RAND to reproduce, or reuse in another form, any of our research documents for commercial use. For information on reprint and linking permissions, please see RAND Permissions. This report is part of the RAND Corporation research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for re- search quality and objectivity. Preparing for the Possibility of a North Korean Collapse Bruce W. Bennett C O R P O R A T I O N NATIONAL SECURITY RESEARCH DIVISION Preparing for the Possibility of a North Korean Collapse Bruce W.
    [Show full text]
  • Yun Mi Hwang Phd Thesis
    SOUTH KOREAN HISTORICAL DRAMA: GENDER, NATION AND THE HERITAGE INDUSTRY Yun Mi Hwang A Thesis Submitted for the Degree of PhD at the University of St Andrews 2011 Full metadata for this item is available in St Andrews Research Repository at: http://research-repository.st-andrews.ac.uk/ Please use this identifier to cite or link to this item: http://hdl.handle.net/10023/1924 This item is protected by original copyright This item is licensed under a Creative Commons Licence SOUTH KOREAN HISTORICAL DRAMA: GENDER, NATION AND THE HERITAGE INDUSTRY YUN MI HWANG Thesis Submitted to the University of St Andrews for the Degree of PhD in Film Studies 2011 DECLARATIONS I, Yun Mi Hwang, hereby certify that this thesis, which is approximately 80,000 words in length, has been written by me, that it is the record of work carried out by me and that it has not been submitted in any previous application for a higher degree. I was admitted as a research student and as a candidate for the degree of PhD in September 2006; the higher study for which this is a record was carried out in the University of St Andrews between 2006 and 2010. I, Yun Mi Hwang, received assistance in the writing of this thesis in respect of language and grammar, which was provided by R.A.M Wright. Date …17 May 2011.… signature of candidate ……………… I hereby certify that the candidate has fulfilled the conditions of the Resolution and Regulations appropriate for the degree of PhD in the University of St Andrews and that the candidate is qualified to submit this thesis in application for that degree.
    [Show full text]
  • A Topic-Aligned Multilingual Corpus of Wikipedia Articles for Studying Information Asymmetry in Low Resource Languages
    Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2373–2380 Marseille, 11–16 May 2020 c European Language Resources Association (ELRA), licensed under CC-BY-NC A Topic-Aligned Multilingual Corpus of Wikipedia Articles for Studying Information Asymmetry in Low Resource Languages Dwaipayan Roy, Sumit Bhatia, Prateek Jain GESIS - Cologne, IBM Research - Delhi, IIIT - Delhi [email protected], [email protected], [email protected] Abstract Wikipedia is the largest web-based open encyclopedia covering more than three hundred languages. However, different language editions of Wikipedia differ significantly in terms of their information coverage. We present a systematic comparison of information coverage in English Wikipedia (most exhaustive) and Wikipedias in eight other widely spoken languages (Arabic, German, Hindi, Korean, Portuguese, Russian, Spanish and Turkish). We analyze the content present in the respective Wikipedias in terms of the coverage of topics as well as the depth of coverage of topics included in these Wikipedias. Our analysis quantifies and provides useful insights about the information gap that exists between different language editions of Wikipedia and offers a roadmap for the Information Retrieval (IR) community to bridge this gap. Keywords: Wikipedia, Knowledge base, Information gap 1. Introduction other with respect to the coverage of topics as well as Wikipedia is the largest web-based encyclopedia covering the amount of information about overlapping topics.
    [Show full text]
  • Towards a Korean Dbpedia and an Approach for Complementing the Korean Wikipedia Based on Dbpedia
    Towards a Korean DBpedia and an Approach for Complementing the Korean Wikipedia based on DBpedia Eun-kyung Kim1, Matthias Weidl2, Key-Sun Choi1, S¨orenAuer2 1 Semantic Web Research Center, CS Department, KAIST, Korea, 305-701 2 Universit¨at Leipzig, Department of Computer Science, Johannisgasse 26, D-04103 Leipzig, Germany [email protected], [email protected] [email protected], [email protected] Abstract. In the first part of this paper we report about experiences when applying the DBpedia extraction framework to the Korean Wikipedia. We improved the extraction of non-Latin characters and extended the framework with pluggable internationalization components in order to fa- cilitate the extraction of localized information. With these improvements we almost doubled the amount of extracted triples. We also will present the results of the extraction for Korean. In the second part, we present a conceptual study aimed at understanding the impact of international resource synchronization in DBpedia. In the absence of any informa- tion synchronization, each country would construct its own datasets and manage it from its users. Moreover the cooperation across the various countries is adversely affected. Keywords: Synchronization, Wikipedia, DBpedia, Multi-lingual 1 Introduction Wikipedia is the largest encyclopedia of mankind and is written collaboratively by people all around the world. Everybody can access this knowledge as well as add and edit articles. Right now Wikipedia is available in 260 languages and the quality of the articles reached a high level [1]. However, Wikipedia only offers full-text search for this textual information. For that reason, different projects have been started to convert this information into structured knowledge, which can be used by Semantic Web technologies to ask sophisticated queries against Wikipedia.
    [Show full text]
  • Arxiv:2010.11856V3 [Cs.CL] 13 Apr 2021 Questions from Non-English Native Speakers to Rep- Information-Seeking Questions—Questions from Resent Real-World Applications
    XOR QA: Cross-lingual Open-Retrieval Question Answering Akari Asaiº, Jungo Kasaiº, Jonathan H. Clark¶, Kenton Lee¶, Eunsol Choi¸, Hannaneh Hajishirziº¹ ºUniversity of Washington ¶Google Research ¸The University of Texas at Austin ¹Allen Institute for AI {akari, jkasai, hannaneh}@cs.washington.edu {jhclark, kentonl}@google.com, [email protected] Abstract ロン・ポールの学部時代の専攻は?[Japanese] (What did Ron Paul major in during undergraduate?) Multilingual question answering tasks typi- cally assume that answers exist in the same Multilingual document collections language as the question. Yet in prac- (Wikipedias) tice, many languages face both information ロン・ポール (ja.wikipedia) scarcity—where languages have few reference 高校卒業後はゲティスバーグ大学へ進学。 (After high school, he went to Gettysburg College.) articles—and information asymmetry—where questions reference concepts from other cul- Ron Paul (en.wikipedia) tures. This work extends open-retrieval ques- Paul went to Gettysburg College, where he was a member of the Lambda Chi Alpha fraternity. He tion answering to a cross-lingual setting en- graduated with a B.S. degree in Biology in 1957. abling questions from one language to be an- swered via answer content from another lan- 生物学 (Biology) guage. We construct a large-scale dataset built on 40K information-seeking questions Figure 1: Overview of XOR QA. Given a question in across 7 diverse non-English languages that Li, the model finds an answer in either English or Li TYDI QA could not find same-language an- Wikipedia and returns an answer in English or L . L swers for. Based on this dataset, we introduce i i is one of the 7 typologically diverse languages.
    [Show full text]
  • The Korean Wave As a Localizing Process: Nation As a Global Actor in Cultural Production
    THE KOREAN WAVE AS A LOCALIZING PROCESS: NATION AS A GLOBAL ACTOR IN CULTURAL PRODUCTION A Dissertation Submitted to the Temple University Graduate Board In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY by Ju Oak Kim May 2016 Examining Committee Members: Fabienne Darling-Wolf, Advisory Chair, Department of Journalism Nancy Morris, Professor, Department of Media Studies and Production Patrick Murphy, Associate Professor, Department of Media Studies and Production Dal Yong Jin, Associate Professor, School of Communication, Simon Fraser University © Copyright 2016 by Ju Oak Kim All Rights Reserved ii ABSTRACT This dissertation research examines the Korean Wave phenomenon as a social practice of globalization, in which state actors have promoted the transnational expansion of Korean popular culture through creating trans-local hybridization in popular content and intra-regional connections in the production system. This research focused on how three agencies – the government, public broadcasting, and the culture industry – have negotiated their relationships in the process of globalization, and how the power dynamics of these three production sectors have been influenced by Korean society’s politics, economy, geography, and culture. The importance of the national media system was identified in the (re)production of the Korean Wave phenomenon by examining how public broadcasting-centered media ecology has control over the development of the popular music culture within Korean society. The Korean Broadcasting System (KBS)’s weekly show, Music Bank, was the subject of analysis regarding changes in the culture of media production in the phase of globalization. In-depth interviews with media professionals and consumers who became involved in the show production were conducted in order to grasp the patterns that Korean television has generated in the global expansion of local cultural practices.
    [Show full text]
  • QUARTERLY CHECK-IN Technology (Services) TECH GOAL QUADRANT
    QUARTERLY CHECK-IN Technology (Services) TECH GOAL QUADRANT C Features that we build to improve our technology A Foundation level goals offering B Features we build for others D Modernization, renewal and tech debt goals The goals in each team pack are annotated using this scheme illustrate the broad trends in our priorities Agenda ● CTO Team ● Research and Data ● Design Research ● Performance ● Release Engineering ● Security ● Technical Operations Photos (left to right) Technology (Services) CTO July 2017 quarterly check-in All content is © Wikimedia Foundation & available under CC BY-SA 4.0, unless noted otherwise. CTO Team ● Victoria Coleman - Chief Technology Officer ● Joel Aufrecht - Program Manager (Technology) ● Lani Goto - Project Assistant ● Megan Neisler - Senior Project Coordinator ● Sarah Rodlund - Senior Project Coordinator ● Kevin Smith - Program Manager (Engineering) Photos (left to right) CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 CTO 4.5 [LINK] ANNUAL PLAN GOAL: expand and strengthen our technical communities What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow? Program 4: Technical LAST QUARTER community building (none) Outcome 5: Organize Wikimedia Developer Summit NEXT QUARTER Objective 1: Developer Technical Collaboration Decide on event location, dates, theme, deadlines, etc. Summit web page and publicize the information published four months before the event (B) STATUS: OBJECTIVE IN PROGRESS Technology (Services) Research and Data July, 2017 quarterly
    [Show full text]
  • Living in Korea
    A Guide for International Scientists at the Institute for Basic Science Living in Korea A Guide for International Scientists at the Institute for Basic Science Contents ⅠOverview Chapter 1: IBS 1. The Institute for Basic Science 12 2. Centers and Affiliated Organizations 13 2.1 HQ Centers 13 2.1.1 Pioneer Research Centers 13 2.2 Campus Centers 13 2.3 Extramural Centers 13 2.4 Rare Isotope Science Project 13 2.5 National Institute for Mathematical Sciences 13 2.6 Location of IBS Centers 14 3. Career Path 15 4. Recruitment Procedure 16 Chapter 2: Visas and Immigration 1. Overview of Immigration 18 2. Visa Types 18 3. Applying for a Visa Outside of Korea 22 4. Alien Registration Card 23 5. Immigration Offices 27 5.1 Immigration Locations 27 Chapter 3: Korean Language 1. Historical Perspective 28 2. Hangul 28 2.1 Plain Consonants 29 2.2 Tense Consonants 30 2.3 Aspirated Consonants 30 2.4 Simple Vowels 30 2.5 Plus Y Vowels 30 2.6 Vowel Combinations 31 3. Romanizations 31 3.1 Vowels 32 3.2 Consonants 32 3.2.1 Special Phonetic Changes 33 3.3 Name Standards 34 4. Hanja 34 5. Konglish 35 6. Korean Language Classes 38 6.1 University Programs 38 6.2 Korean Immigration and Integration Program 39 6.3 Self-study 39 7. Certification 40 ⅡLiving in Korea Chapter 1: Housing 1. Measurement Standards 44 2. Types of Accommodations 45 2.1 Apartments/Flats 45 2.2 Officetels 46 2.3 Villas 46 2.4 Studio Apartments 46 2.5 Dormitories 47 2.6 Rooftop Room 47 3.
    [Show full text]
  • Social Changer Jae-Hee Technology Comfort Level
    Social Changer Jae-Hee Technology comfort level AGE: EDUCATION: 27 years old University Low High LOCATION: LANGUAGES: Seoul, South Korea Korean (fluent) Writing comfort level Japanese (proficient) OCCUPATION: English (proficient) Freelance graphic designer Low High Macbook Pro iPad iPhone 6S PRIMARY USE: Graphic design work, PRIMARY USE: Reading, looking for PRIMARY USE: Calling, messaging maintaining her website, writing, inspiration for her work, and quick friends on Kakao Talk, Twitter editing Wikipedia internet browsing when she’s not at home Background Jae-Hee graduated from university two years ago Japanese. While in university, she took a class on Experience Goals and currently works as a freelance graphic de- design and sustainability and became interested • To freely share her opinions and signer. She lives in the suburbs of Seoul, with her in environmental issues. Now she volunteers for knowledge without conflict or rebuke, parents and younger sister. In her spare time, she an environmental advocacy non-profit which ed- like she does elsewhere online works on her digital art and photography, which ucates people about living a sustainable lifestyle, she publishes on her personal website, through and shares similar lifestyle tips on her personal End Goals WordPress. She loves reading, particularly fantasy blog. and science fiction stories, in both Korean and • To raise awareness on environmental issues • To collaborate with other Tech Habits environmentalists She first started using a computer in grade school, environmental groups. She is well-known by her and got her first smartphone in high school. She online username, “jigu”. She learned basic HTML Challenges uses social media avidly, particularly Twitter to to run her website, and uses Adobe software for share her work and writing, and to follow other her graphic design and art.
    [Show full text]
  • Interview with Sue Gardner, Executive Director WMF October 1, 2009 510 Years from Now, What Is Your Vision?
    Interview with Sue Gardner, Executive Director WMF October 1, 2009 5-10 years from now, what is your vision? What's different about Wikimedia? Personally, I would like to see Wikimedia in the top five for reach in every country. I want to see a broad, rich, deep encyclopedia that's demonstrably meeting people's needs, and is relevant and useful for people everywhere around the world. In order for that to happen, a lot of things would need to change. The community of editors would need to be healthier, more vibrant, more fun. Today, people get burned out. They get tired of hostility and endless debates. Working on Wikipedia is hard, and it does not offer many rewards. Editors have intrinsic motivation not extrinsic, but even so, not much is done to affirm or thank or recognize them. We need to find ways to foster a community that is rich and diverse and friendly and fun to be a part of. That world would include more women, more newly-retired people, more teachers ± all different kinds of people. There would be more ways to participate, more affirmation, more opportunities to be social and friendly. We need a lot of tools and features to help those people be more effective. Currently, there are tons of hacks and workarounds that experienced editors have developed over time, but which new editors don©t know about, and would probably find difficult to use. We need to make those tools visible and easier to use, and we need to invent new ones where they are lacking.
    [Show full text]
  • 2020 Korean Books for Young Readers
    2020 Korean Books for Young Readers Korean Board on Books for Young People (IBBY Korea) About Contents KBBY and this Catalog KBBY(Korean Board on Books for Young People) was founded in 1995 7 Korean Nominees for the Hans Christian Andersen Awards 4 as the Korea national section of the International Board on Books for Young People (IBBY). Korean Nominations for the IBBY Honour List 2020 12 To fulfill IBBY’s mission, KBBY works as a network of professionals from both home and abroad, collecting and sharing information on Korean Nominations for BIB 2019 14 children’s and juvenile literature. KBBY also works in close partnership with the other national sections of IBBY to contribute to promoting Korean Nominations for Silent Books 2019 22 cross-cultural exchange in children’s literature. Recent Picture Books Recommended by KBBY Since 2017 25 KBBY organizes international book exhibitions in collaboration with library networks, in efforts to share with the Korean audience the in- formation on global books generated through the awards and activ- Recent Chapter Books and Novels Recommended by KBBY Since 2017 37 ities of IBBY. Moreover, KBBY is committed to providing information on outstanding Korean children’s and juvenile literature with readers Recent Non-fiction Recommended by KBBY Since 2017 50 across the world. This catalog presents the Korean nominees of the Hans Christian An- dersen Awards, who have made a lasting impact on children’s litera- ture not only at home but also to the world at large. Also included is a collection of the Korean children’s books recommended by the book selection committee of KBBY: Korean nominations for the IBBY Honour List, BIB, Silent Books; recent picutre books, chapter books & novels, and non-finction books.
    [Show full text]
  • A Study and Performance Analysis of Selected Art Songs by Un-Yeong Na" (2010)
    Graduate Theses, Dissertations, and Problem Reports 2010 A Study and Performance Analysis of Selected Art Songs by Un- Yeong Na Min Sue Kim West Virginia University Follow this and additional works at: https://researchrepository.wvu.edu/etd Recommended Citation Kim, Min Sue, "A Study and Performance Analysis of Selected Art Songs by Un-Yeong Na" (2010). Graduate Theses, Dissertations, and Problem Reports. 2991. https://researchrepository.wvu.edu/etd/2991 This Dissertation is protected by copyright and/or related rights. It has been brought to you by the The Research Repository @ WVU with permission from the rights-holder(s). You are free to use this Dissertation in any way that is permitted by the copyright and related rights legislation that applies to your use. For other uses you must obtain permission from the rights-holder(s) directly, unless additional rights are indicated by a Creative Commons license in the record and/ or on the work itself. This Dissertation has been accepted for inclusion in WVU Graduate Theses, Dissertations, and Problem Reports collection by an authorized administrator of The Research Repository @ WVU. For more information, please contact [email protected]. A Study and Performance Analysis of Selected Art Songs by Un-Yeong Na Min Sue Kim Research Project submitted to the College of Creative Arts at West Virginia University in partial fulfillment of the requirements for the degree of Doctor of Musical Arts in Performance: Voice Dr. Kathleen Shannon, Chair and Research Advisor Professor Robert Thieme Dr. Chris Wilkinson Dr. Peter Amstutz Dr. Georgia Narsavage Division of Music Morgantown, West Virginia 2010 Keywords: Un-Yeong Na, Korean Art Song, Korean Traditional Music Copyright 2010 Min Sue Kim ABSTRACT A Study and Performance Analysis of Selected Art Songs of Un-Yeong Na Min Sue Kim Korea‘s history spans over 5,000 years.
    [Show full text]