<p> Exploring Mutual Complementarity of Free-Text and Controlled-Vocabulary Collection-Level Subject Metadata in Large-Scale Digital Libraries: A Comparative Analysis OKSANA L. ZAVALINA College of Information, University of North Texas, Denton, Texas, USA [email protected] ABSTRACT Provision of high-quality subject metadata is crucial for organizing adequate subject access to rich content aggregated by digital libraries. A number of large-scale digital libraries worldwide are now generating subject metadata to describe not only individual objects but entire digital collections as integral whole. However, little research to date has been conducted to empirically evaluate the quality of this collection-level subject metadata. The study presented in this paper compares free- text and controlled-vocabulary collection-level subject metadata in three large-scale digital cultural heritage aggregations in the United States and the European Union. As observed by this study, the emerging best practices in creating rich collection- level subject metadata includes describing collection’s subject matter with mutually- complementary values in controlled-vocabulary and free-text subject metadata fields. INTRODUCTION Both cultural heritage institutions and funding agencies worldwide have invested intensively in digitization projects. Large-scale digital libraries now bring together hundreds of individual digital collections produced by these digitization projects. Metadata — “structured data about an object that supports functions associated with the designated object” (Greenberg, 2005, p. 1876) — is used in digital libraries to organize information for effective retrieval via search and browse functions. Metadata is subdivided into two distinct kinds: controlled-vocabulary metadata which draws values from formally-maintained list of terms, and free-text metadata which relies on natural language. The subject metadata – “information concerning what the resource is about and what it is relevant for” (Soergel, 2009) is crucial for providing subject access to information objects in digital collections and aggregations. To help achieve optimal recall and precision, it is recommended (e.g., ALCTS, 1999) to include Subject, Type, and Coverage elements in metadata records in digital libraries to accommodate different subject-related facets: topic, place, time period, language, etc. Metadata that describes collections as integral whole has long been applied in archival community. Many digital aggregations are now supplying collection-level metadata — “metadata providing a high-level description of an aggregation of individual items” (Macgregor, 2003, p. 248) — as means of providing context for the digital items harvested from distributed collections. However, virtually no research to date has evaluated and compared the collection-level metadata in digital aggregations. In discussions of metadata, the terms “richness,” “detailed description,” “level of description” or “quality” of metadata seem to be used interchangeably (e.g., Arms, 1998; Duval, Hodgins, Sutton, & Weibel, 2002). The three most important metadata quality criteria are accuracy, consistency, and completeness (Park, 2009; Park & Tosaka, 2010). Metadata accuracy is measured as the degree to which the metadata values match characteristics of the described object (e.g., Stvilia, Gasser, Twidale, & Smith, 2007). Metadata consistency is further subdivided into semantic and structural consistency (Park, 2009). Semantic consistency refers to an extent to which the same values or elements are used for representing similar concepts, while structural consistency is evaluated as a degree to which the same structure is followed in representing information in certain metadata elements (Bruce & Hillmann, 2004). Metadata completeness is evaluated as an extent to which objects are described using all applicable metadata elements to their full access capacity. Some of the assessment criteria used to evaluate metadata completeness (Moen, Stewart, & McClure, 1998) include the number of metadata elements per record, practice of presenting blank (i.e., nonpopulated but displayed) metadata elements, utilization and selected characteristics of mandatory and optional elements. While evaluation of metadata in digital libraries, which has not yet become a common practice, gains more and more importance to ensure metadata quality (Hillmann, 2008), almost no research to date has attempted to evaluate collection- level metadata. Zavalina, Palmer, Jackson, and Han (2008) started addressing this research gap by assessing collection-level metadata in the Digital Collections and Content registry of IMLS-funded digital collections. However, because that study focused on a single digital library, generalizabilty of its results is limited. To produce more generalizable results, Zavalina (2011) study examined and compared the free-text collection-level subject metadata (i.e., data values in Description metadata element) across multiple digital libraries, and found that a variety of information about a digital collection is included in free-text collection-level Description metadata element. This includes both subject-specific (topical, geographic and temporal coverage, and types/genres of objects in a digital collection) and non-subject-specific information: title, size, provenance, collection development, copyright, audience, navigation and functionality, language of items in a digital collection, frequency of additions, institutions that host a digital collection or contribute to it, funding sources, item creators, importance, uniqueness, and comprehensiveness of a digital collection. The study presented in this paper extends the comparative analysis reported in Zavalina (2011) study by comparing the data values in free-text Description and four controlled-vocabulary subject metadata fields in three large-scale digital libraries. DATA COLLECTION AND ANALYSIS Three large-scale cultural heritage digital libraries were selected for analysis: American Memory1 developed by the United States Library of Congress, Opening History2 developed by the University of Illinois at Urbana-Champaign, and The European Library3 that aggregates digital collections created by the national libraries in the European Union. Among these three digital libraries, only the Opening History displays its entire human-readable collection-level metadata records. American Memory and The European Library keep most of collection-level metadata (except for the Title and free-text Description elements) behind the scenes to support search and faceted browse functions. For this study, the XML files with complete collection metadata records were obtained from the developers of The European Library and American Memory. </p><p>1 http://memory.loc.gov. 2 http://imlsdcc.grainger.uiuc.edu/history. 3 http://www.theeuropeanlibrary.org.</p><p>2 The systematic sample of collection-level metadata records in the three digital libraries was analyzed: 39 records from American Memory, 33 records from Opening History, and 27 records from The European Library. The resulting 99 collection-level metadata records were subjected to detailed manual qualitative content analysis to determine how the data values in different collection-level subject metadata elements within a record relate to each other. These relations were categorized in three categories: one-way or two-way complementarity, and redundancy. The findings of the study are presented below. FINDINGS A significant proportion of collection metadata records in the sample included cases of one-way complementarity, when information in one collection-level subject metadata elements complemented information in one or more other metadata elements, by providing additional details absent elsewhere. The highest occurrence of one-way complementarity between collection-level subject metadata elements was observed in Opening History. In 76% of collection metadata records analyzed in this study it was the free-text Description metadata element that complemented information found in one or more of the controlled-vocabulary subject metadata elements: Subjects, Geographic Coverage, Temporal Coverage, and Objects. As seen in the Figure 1, the free-text Description metadata element data values most often (76% of records overall: 86% in American Memory, 76% in Opening History, and 70% in The European Library) complemented topical information found in the Subjects element. Representative examples include: “Spanish cartographer, … history, urbanism, public works and agriculture from a strictly geographic point of view” in Description vs. “900 History and geography, 911 Historical geography” in Subjects; “interior design, … homes of U.S. presidents” in Description, with these topics not mentioned in Subjects; “early developments in the National Park, … landscape and park facilities” in Description vs. “Great Basin, Social studies, State history” in Subjects.</p><p>FIGURE 1. Mutual complementarity between collection-level subject metadata elements Objects metadata element was the second most often complemented by object-type- or genre-specific information in Description field (49% overall: 70% in American Memory, 44% in The European Library, and 30% in Opening History). Representative examples included: “uniform books, ego documents, photographs and sketches” in Description vs. “images” in Objects; “digital pre-print originals and online publications” in Description while Objects field was missing; “historical photographs, … portraits, … aerial shots” in Description vs. “photographs/slides/negatives” in Objects; “rare books, government documents, manuscripts, maps, musical scores, plays, films, and recordings” in Description vs. “software, multimedia” in Objects. Data values in Temporal Coverage metadata element were also often complemented by Description (46% overall: 67% in Opening History, 51% in American Memory, and 15% in The European Library). Representative examples included: “16th century, 17th century, 18th century, 19th century, 20th century” in</p><p>3 Temporal Coverage vs. “Since the Eighty Years’ War” in Description; “from 1895- 1920s” in Description vs. “1850-1899, 1900-1929” in Temporal Coverage field. Geographic Coverage data values were complemented by Description metadata element the least often (29% overall: 39% in Opening History, 33% in The European Library, and 19% in American Memory). Representative examples included: “Hispanic America … Spanish territories in America and Oceania” in Description vs. “Hispanic America” in Geographic Coverage; “Hungary or the Central European region” in Description vs. machine-readable “hu” in Geographic Coverage; “American states, the District of Columbia, and London, England” in Description vs. “United States” in Geographic Coverage; “Baja California, Mexico in an area south- east of Ensenada” vs. “Mexico (nation)” in Geographic Coverage. In addition, the cases of data values in free-text Description metadata element complementing information contained in several controlled-vocabulary subject metadata elements in the same collection- level metadata record were observed. In the example in Figure 2, Description includes keywords that complement both Subjects and Objects with topical information (“foodways, religious traditions, Native American culture, maritime traditions, ethnic folk culture, material culture”), genre information (“children’s lore,” “occupational lore,” “performances,” “interviews,” “surveys”), and occupational subject information (“musicians, craftpersons, storytellers, folklife interpreters”), while also specifying the dates encoded in Temporal Coverage field. In fact, in 22% of collection metadata records in the sample (45% in Opening History, 11% in The European Library, and 10% in American Memory) Description field complemented two or more controlled-vocabulary subject metadata fields.</p><p>FIGURE 2. Example of multiple complementarities Data values in controlled-vocabulary subject metadata elements also complemented data values in free-text Description (Figure 1). For example, in this same collection metadata record (Figure 4), Geographic Coverage provided spatial information absent in Description (“United States (nation), Southern U.S. (general region), Florida (state)”), while Subjects listed additional topics (e.g., “Architecture”) not covered by Description. The Subjects metadata element was found to complement Description (Figure 1) the most often – in 52% of collection metadata records overall (70% in Opening History, 60% in The European Library, and 30% in American Memory). Representative examples included: “860 Spanish and Portuguese literatures” in Subjects when this topic was not mentioned at all in Description; “Tennessee Valley Authority, African Americans, forestry” in Subjects when these topics were not mentioned at all in Description; 15 specific subject strings (e.g., “North Carolina— African-Americans, North Carolina—Agriculture, North Carolina—Economics and Business” in Subjects vs. much broader topical and spatial coverage in Description (“North Carolina, … story of the Tar Heel State”). The Temporal Coverage metadata element was found to complement Description in 43% of collection metadata records (72 % in The European Library and 67% in Opening History, but only 3% in American Memory). Representative</p><p>4 examples included: “1400s-1699, 1700-1799, 1800-1849, 1850-1899, 1900-1929, 1930-1949, 1950-1969, 1970-1999, 2000 to present, Pre-1400” in Temporal Coverage when no time information was provided in Description; “1783-1789” in Temporal Coverage when no time information was provided in Description; “1200- 1900” in Temporal Coverage vs. “European age of chivalry” in Description. The Geographic Coverage metadata element was found to complement Description much more often than the Description complemented Geographic Coverage (Figure 1), or in 43% of collection metadata records overall (56% in The European Library, 55% in Opening History, and 24% in American Memory). Representative examples included: “Poland, Lithuania, Ukraine, Belarus” in Geographic Coverage vs. “Poland” in Description; “Germany” in Geographic Coverage when no geographic information was provided at all in Description; “Europe, Italy, Great Britain” in Geographic Coverage vs. “US and abroad” in Description; “United States (nation), Midwest U.S. (general region), Illinois (state), Randolph (county), Knox (county)” in Geographic Coverage vs. “Randolph County, Illinois” in Description. The Objects metadata element values also often complemented information found in Description in two digital libraries — Opening History (52%) and American Memory (14%) — for 23% of analyzed collection metadata records overall. No such trend was observed in The European Library, which can be explained by inconsistent application of Objects metadata element in this digital library: in 59% of collection metadata records in The European Library sample the Objects metadata element was blank or missing, while in the remaining 41% this field contained a broad single- word term (e.g., “images,” “maps”). Representative examples of the Objects metadata element values complementing Description included: “Film transparencies —Color, Cityscape photographs” in Objects vs. “photographs” in Description; “Gelatin silver prints, Safety film negatives, Nitrate negatives” in Objects vs. “original negatives and photographic prints” in Description; “books and pamphlets, photographs / slides / negatives, newspapers, posters and broadsides, periodicals, prints and drawings” in Objects vs. “manuscripts, photographs, ephemera and published materials” in Description. In addition, one-way complementarity between different controlled-vocabulary metadata elements was also observed. In particular, geographical subdivisions (as in “Japanese Americans—California—Manzanar”) and temporal qualifiers (as in “World War, 1914-1918”) in Subjects metadata element included information that complemented Temporal Coverage and Geographic Coverage values. In Opening History, Subjects complemented Geographic Coverage in 12% of collection metadata records and Temporal Coverage in 18% of the records in the sample. The cases of two-way complementarity between the two collection-level subject metadata elements were less numerous than cases of one-way complementarity. No cases of two-way complementarity were observed between the two or more controlled-vocabulary subject metadata elements. Two-way complementarity between the free-text (Description) and controlled-vocabulary subject metadata elements, in contrast, occurred in 40% of collection-level metadata records overall. Two-way complementarity was widespread in Opening History (79% of records), but occurred less often in The European Library (41%) and significantly less often in American Memory (8%). Most often two-way complemenarity was observed between Description and Subjects elements (29% of collection metadata</p><p>5 records overall: 58% in Opening History, 30% in The European Library, and 5% in American Memory). Two-way complementarity between Description and Temporal Coverage was observed only in Opening History (39% in Opening History or 13% overall). Two-way complementarity between Description and Geographic Coverage was observed in 11% of the records overall: 24% in Opening History, 11% in The European Library, but in none of the American Memory collection metadata records. The least overall two-way complementarity was observed between Description and Objects metadata element (7% overall: 18% in Opening History, 3% in American Memory and 0% in The European Library). Representative examples of two-way complementarity included:  “letters” in Description vs. “autograph albums” in Subjects (taken together, the values in two fields provide more comprehensive genre information).  “ dance instruction manuals, anti-dance manuals, histories, treatises on etiquette” in Description vs. “Ballroom dancing—United States” in Subjects (Subjects information specifies Description information from “dance” to “ballroom dancing” and adds geographic coverage information, while Description adds information on specific aspects of dancing — “etiquette” — and genre of materials in collection not covered by any other metadata field in this record).  “towns of Coal City, Braidwood, and Wilmington” in Description vs. “Illinois (state), Grundy (county)” in Geographic Coverage (state and county information in Geographic Coverage and town information in Description complement each other for a more specific geographic representation).  “ contemporary, … European age of chivalry, … prior to 1900” in Description vs. “1200-1900” in Temporal Coverage (while Temporal Coverage specifies the lower limit of the “prior to 1900” range of years — “1200” — and provides the time frame for “European age of chivalry,” Description introduces another — “contemporary” — time period not covered by Temporal Coverage).  “ newspaper photographs” in Description vs. “photographs/slides/negatives, archival finding aids” in Objects (Description specifies genre information in Objects from general “photographs” to “newspaper photographs, while Objects adds another genre not mentioned in Description — “archival finding aids”). Among the digital libraries examined in this study, only The European Library had a noticeable proportion (19%) of redundancy between the values in different collection-level subject metadata elements. Very little redundancy was observed in the Opening History and American Memory collection metadata records. Examples of redundancy include restating of identical geographic information (e.g., “Estonia,” “Netherlands,” “Ljubljana” in both Description and Geographic Coverage metadata element), temporal information (e.g., “1763” in both Description and Temporal Coverage), and genre information (e.g., “photographs” in both Description and Subjects). DISCUSSION AND CONCLUSIONS The findings presented in this paper demonstrate high level of mutual complementarity between free-text and controlled-vocabulary subject metadata in collection-level metadata records in three large-scale digital libraries that aggregate cultural heritage digital collections: American Memory and Opening History in the United States of America, and The European Library in Europe. Quite predictably,</p><p>6 the data values in free-text Description metadata element, due to its natural language values and higher length, often complemented information in controlled- vocabulary subject metadata elements. However, it was also observed in this study that data values in controlled-vocabulary subject metadata elements, especially Geographic Coverage, complemented information encoded in Description quite often. Both one-way complementarity and two-way complementarity was observed, with little redundancy. Results of this study empirically demonstrate that more detailed collection-level metadata records which include both free-text and controlled-vocabulary subject metadata allow more fully representing intellectual content of information objects and ultimately improving subject access for the users. Completeness of metadata records — an extent to which objects are described using all applicable metadata elements to their full access capacity — has long been emphasized as one of the most important metadata quality criteria (e.g., Moen, Stewart, & McClure, 1998; Bruce & Hillmann, 2004; Park, 2009; Park & Tosaka, 2010). Findings of the user studies conducted both decades ago, with card catalogs and early computerized library catalogs, (e.g., studies summarized by Krikelas, 1972), and more recently, with various online information retrieval systems, (e.g., Wang & Soergel,1998; Drori, 2003; Crystal & Greenberg, 2006; Smith- Yoshimura et al., 2010) demonstrate that users perceive both the free-text subject metadata (e.g., the data values in MARC 5XX fields or Dublin Core Description element) and controlled-vocabulary subject headings, such as the data values in 65X MARC fields or Dublin Core Subject and Coverage metadata elements, to be among the most useful metadata elements to judge the relevance of retrieved documents Item-level metadata records in digital libraries usually meet these user expectations by providing both free-text and controlled-vocabulary subject metadata. However, most newly-created digital libraries limit their collection-level metadata to free-text Title and Description elements for various reasons: lack of resources needed to create detailed collection-level metadata records, limitations introduced by the default settings in popular content management systems such as DSpace, or even a belief that full-text indexing and keyword searching make controlled- vocabulary subject metadata redundant. Lack of best practice guidelines for creation of collection-level metadata arguably contributes to this situation. Results of this study indicate that including mutually-complementary subject information in free-text and controlled-vocabulary collection-level metadata elements is already a common practice among some of the large-scale digital libraries, and possibly is recognized by digital library developers as a benchmark in crafting rich collection-level metadata. The findings of this study could be instrumental in developing best practice recommendations for creating collection-level metadata, including subject metadata, which are not currently available. These guidelines can be incorporated in the next edition of the Framework of Guidance for Building Good Digital Collections (NISO Framework Working Group, 2007) and/or the Guidelines for Digital Libraries that are currently being prepared by the IFLA working group jointly with the World Digital Library Project. This exploratory study focused on collection-level subject metadata in domain-specific digital libraries of one domain (aggregations of cultural heritage digital collections that are created for history scholars, educators, and enthusiasts) and of two scales – national and international. The task of developing best practice guidelines warrants more extensive content analysis of collection-level subject metadata, including those in domain-specific digital libraries with a subject focus</p><p>7 other than history (e.g., National Science Digital Library) or non-domain-specific digital libraries with wide subject coverage (e.g., IMLS Digital Collections and Content Collection Registry), with different scale (e.g., state-level aggregations such as Missouri Digital Heritage or regional-level aggregations such as Mountain West Digital Library, Documenting the American South etc.), and representing geographic areas beyond Europe and North America (e.g., New Zealand Digital Library). ACKNOWLEDGMENT The author wishes to thank developers of The European Library, American Memory, and Opening History digital libraries for providing collection-level metadata for this analysis. Special thanks to Drs. Carole L. Palmer, Allen Renear, and Kathryn La Barre at the University of Illinois (USA) and Dr. Dietmar Wolfram at the University of Wisconsin (USA) for valuable feedback on this study. BIOGRAPHICAL NOTE Dr. Oksana L. Zavalina is an Assistant Professor at the Department of Library and Information Sciences, College of Information, at the University of North Texas in the USA. She was born, raised and received undergraduate library and information science (LIS) training in Kiev, Ukraine. Dr. Zavalina’s two graduate degrees in LIS come from the University of Illinois at Urbana-Champaign (UIUC): MLIS in 2002 and Ph.D. in 2010. Her previous positions included being a bibliographer at the National Parliamentary Library of Ukraine, a solo librarian at the Kiev School of Economics, an intern at Yale University Sterling Memorial Library, an original cataloger at the UIUC Library (including with Google Digitization Project), and a research assistant with the Center for Informatics Research in Science and Scholarship in Illinois. Dr. Zavalina has been teaching graduate-level courses in library cataloging and classification since 2007, in metadata and in information organization since 2011. Her research focuses on subject metadata and its role in subject access in digital libraries. REFERENCES ALCTS/CCS/SAC/Subcommittee on Metadata and Subject Analysis. (1999). Subject Data in the Metadata Record: A Report from the ALCTS/CCS/SAC/Subcommittee on Metadata and Subject Analysis Working Draft. Retrieved from http://archive.ifla.org/VII/s12/mom/appendx3.htm Arms, C.R. (1996). Historical collections for the National Digital Library: Lessons and challenges at the Library of Congress. D-Lib Magazine, April-May 1996. Retrieved from http://www.dlib.org/dlib/april96/loc/04c-arms.html and http://www.dlib.org/dlib/may96/loc/05c- arms.html Bruce, T.R., & Hillmann, D.I. (2004). The continuum of metadata quality: defining, expressing, exploiting. In D.I. Hillmann (Ed.), Metadata in Practice. Chicago, IL: American Library Association. Crystal, A., & Greenberg, J. (2006). Relevance criteria identified by health information users during Web searches. Journal of the American Society for Information Science and Technology, 57(10), 1368-1382. Drori, O. (2003). How to display search results in digital libraries: user study. In P.T. Isaias, F. Sedes, J.C. Augusto, & U. Ultes-Nitsche (Eds.), Proceedings of the New Developments in Digital Libraries, (pp. 13-28). Angers, France: ICEIS Press. Retrieved from www.global- report.com//drori/?l=he&a=3330 Duval, E., Hodgins, W., Sutton, S., & Weibel, S. (2002). Metadata principles and practicalities. D-Lib Magazine, 8(4). Retrieved from http://www.dlib.org/dlib/april02/weibel/04weibel.html</p><p>8 Greenberg, J. (2005). Metadata and the World Wide Web. In Encyclopedia of Library and Information Science, pp. 1876-1888. New York: Marcel Dekker. Hillmann, D.I. (2008). Metadata quality: from evaluation to augmentation. Cataloging & Classification Quarterly, 46(1), 65-80. Krikelas, J. (1972). Catalog use studies and their implications. Advances in Librarianship, 3, 195-220. Macgregor, G. (2003). Collection-level descriptions: metadata of the future? Library Review, 52(6), 247-250. Miller, P. (2000). Collected wisdom: some cross-domain issues of collection-level description. D-Lib Magazine, 6(Sept.). Retrieved from http://www.dlib.org/dlib/september00/miller/09miller.html Moen, W.E., Stewart, E.L., & McClure, C.R. (1998). The Role of Content Analysis in Evaluating Metadata for the U.S. Government Information Locator Service (GILS): Results from an Exploratory Study. Retrieved from http://www.unt.edu/wmoen/publications/GILSMDContentAnalysis.htm NISO Framework Working Group. (2007). A Framework of Guidance for Building Good Digital Collections. (3rd ed.). Bethesda, MD: National Information Standards Organization. Retrieved from http://www.niso.org/publications/rp/framework3.pdf Park, J. (2009). Metadata quality in digital repositories: A survey of the current state of the art. Cataloging & Classification Quarterly, 47(3), 213-228. Park, J., & Tosaka, Y. (2010). Metadata quality control in digital repositories and collections: criteria, semantics, and mechanisms. Cataloging & Classification Quarterly, 48(8), 696-715. Smith-Yoshimura, K. et al. (2010). Implications of MARC Tag Usage in Library Metadata Practices. Dublin, Ohio: OCLC. Retrieved from http://www.oclc.org/research/publications/library/2010/2010- 06.pdf Soergel, D. (2009). Digital libraries and knowledge organization. In S.R. Kruk & B. McDaniel (Eds.), Semantic Digital Libraries, (pp. 9-39). Berlin: Springer. Stvilia, B., Gasser, L., Twidale M.B., & Smith L.C. (2007). A framework for information quality assessment. Journal of the American Society for Information Science and Technology, 58(12), 1720-1733. Wang, P., & Soergel, D. (1998). A cognitive model of document use during a research project. Study 1. Document selection. Journal of the American Society for Information Science and Technology, 49(2), 115-133. Zavalina, O.L., Palmer, C.L., Jackson, A.S., & Han, M.-J. (2008). Evaluating descriptive richness in collection-level metadata. Journal of Library Metadata, 8(4), 263-292. Zavalina, O.L. (2011). Free-text collection-level subject metadata in large-scale digital libraries: a comparative content analysis. In T. Baker, D.I. Hillmann & A. Isaac (Eds.), Proceedings of the International Conference on Dublin Core and Metadata Applications, (pp. 147-157). The Hague: Dublin Core Metadata Initiative.</p><p>9</p>