Empirical Evaluation of Metadata for Video Games and Interactive Media
Jin Ha Lee Information School, University of Washington, Mary Gates Hall, Suite 370, Seattle, WA 98195. E-mail: [email protected]
Rachel Ivy Clarke Information School, University of Washington, Mary Gates Hall, Suite 370, Seattle, WA 98195. E-mail: [email protected]
Andrew Perti Seattle Interactive Media Museum, 305 Harrison Street, Seattle, WA 98109. E-mail: [email protected]
Despite increasing interest in and acknowledgment of Introduction the significance of video games, current descriptive practices are not sufficiently robust to support search- Video games are of increasing importance to American ing, browsing, and other access behaviors from diverse society as objects of economic stimulus as well as cultural user groups. To address this issue, the Game Metadata heritage. Because of this increased interest in games for Research Group at the University of Washington consumer entertainment as well as historical, cultural, and Information School, in collaboration with the Seattle Interactive Media Museum, worked to create a standard- scientific study, many cultural heritage institutions have ized metadata schema. This metadata schema was established collections of video games and related media. empirically evaluated using multiple approaches— The Library of Congress collects, preserves, and offers collaborative review, schema testing, semi-structured access to two main types of video games: educational games user interview, and a large-scale survey. Reviewing and that support the Library’s initiatives and controversial games testing the schema revealed issues and challenges in sourcing the metadata for particular elements, determin- collected to support legislation related to sex and violence in ing the level of granularity for data description, and video games (Owens, 2012). In Great Britain, the National describing digitally distributed games. The findings Videogame Archive (NVA) focuses on collecting and pre- from user studies suggest that users value various serving hardware, original software, design documents, and subject and visual metadata, information about how marketing materials (Newman, 2009). Recently, the British games are related to each other, and data regarding game expansions/alterations such as additional content Library began collaborating with the NVA to archive video and networked features. The metadata schema was game-related websites, including screenshots and walk- extensively revised based on the evaluation results, and throughs of games (Crookes, 2012). This related informa- we present the new element definitions from the revised tion is valuable to scholars studying games in a social or schema in this article. This work will serve as a platform historical context. Other organizations, such as the Strong and catalyst for advances in the design and use of video game metadata. National Museum of Play in Rochester, New York, do not collect video games exclusively but recognize the impor- tance of sharing historical video game information. Their recent exhibit “Atari By Design: From Concept to Creation” used materials on loan from the library and archives at the International Center for the History of Electronic Games, which collects printed materials related to the history of Received April 9, 2014; revised May 27, 2014; accepted June 4, 2014 video games and the ways in which they affect how people © 2015 ASIS&T • Published online in Wiley Online Library learn, play, and connect. The exhibit placed design and (wileyonlinelibrary.com). DOI: 10.1002/asi.23357 development materials such as sketches and storyboards
JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY, ••(••):••–••, 2015 alongside original arcade game cabinets to let museum designed and whether the current version of the proposed visitors experience the entire process from ideation to metadata schema is capable of describing all games within gameplay. the video game domain. These and many other organizations collect, preserve, and circulate video games and interactive media, yet each institution has a unique way of describing them. Public and Relevant Work academic libraries shoehorn video games into their local Organization and Preservation of Video Games catalogs, using workarounds in metadata records to differ- entiate games from other media so that patrons might find In library and information science, a few projects share them more easily. In the Library of Congress, games are similar objectives of improved organization and preserva- cataloged in the Library’s moving images database using tion of video games and interactive media. The Preserving metadata designed for motion pictures and sound record- Virtual Worlds project was a collaborative research project ings; however, this solution is still in transition and raises conducted by the Rochester Institute of Technology, Stan- questions about challenges researchers might face in trying ford University, University of Maryland, and University of to find records for games in a database designed primarily Illinois at Urbana-Champaign as part of Preserving Creative for motion pictures that is only available on-site in the America, an initiative of the National Digital Information Library’s reading room. Additionally, current Library of Infrastructure and Preservation Program at the Library of Congress genre headings designed for literature and film are Congress (McDonough et al., 2010). This project focused on insufficient for video games; they do not adequately describe preserving older video games and software, and establishing games in a recognizable way for players or researchers best practices and strategies for game preservation. It laid (Owens, 2012). Archives that focus on hardware will also preliminary groundwork for basic metadata standards, and have metadata needs different from those of collections the final report specifically calls for future work in estab- focused on software or print materials. All organizations, lishing relationships and entities and states that the project regardless of focus, face the challenges of collecting, pre- barely scraped the surface for standardized ontologies serving, and offering access to digitally distributed games, in this domain (McDonough et al., 2010). Currently the games that have no physical component but exist only in project is in its second phase, focusing on determining sig- electronic bits accessed through download or streaming nificant properties for educational games. Our project from “the cloud.” Libraries, archives, and museums need a involves a wider range of games, from older games to recent common, shared metadata standard that covers all aspects of digitally downloadable games and game apps on tablets and video games—for players, researchers, and curators—that smartphones, as well as games for entertainment, and incor- can describe all types of video games, from historic arcade porates user behaviors and needs beyond preservation. The games in cabinets to game apps downloaded on smart- motivation for our project also stems from a very practical phones, in descriptive vocabulary terms that are relevant to problem: how best to organize and provide access to games and understandable to users. SIMM’s game collection. Thus, one of our future goals is to To address this issue, the GAMER (GAme MEtadata create a set of metadata records for a large and diverse game Research) Group at University of Washington Information collection. School, in partnership with the Seattle Interactive Media Stanford University Libraries and the National Institute of Museum (SIMM), has been working on a research project to Standards and Technology, funded by the National Software build a standardized metadata schema and controlled Reference Laboratory, have also launched a project on the vocabularies for video games since 2011. The objective of digital preservation of 15,000 software titles, including 1 our research is to create a robust, media-specific metadata games, in the Stanford University Libraries. Their goal is schema intended to describe a variety of games, from his- to preserve these titles for use by academic scholars for torical to contemporary, and to serve a variety of use cases to research purposes. Our project, however, targets a wider meet the needs of users of the SIMM or other similar cul- range of user personas, representing stakeholders beyond just tural heritage organizations. In particular, we seek to answer researchers (Lee, Tennis, Clarke, & Carpenter, 2013b). the following research questions: Twenty-four user interviews from earlier phases of this research investigated game information needs and search/ browse behaviors of real users such as players, parents of 1. What kinds of information about video games must be players, collectors, developers, and curators. The results provided to support users’ information-seeking activities from their perspective? directly informed the development of our initial metadata 2. Is the proposed metadata schema capable of describing a schema and encoding schemes. Stanford University Library, wide variety of games across different genres, formats, in partnership with the University of Santa Cruz, also recently platforms, and time periods? received funding from Institute of Museum and Library Services (IMLS) to develop a metadata scheme for digital This article reports our efforts to evaluate how well the schema represents and reflects the information needs, 1http://library.stanford.edu/news/2013/03/stanford-libraries-preserves- behaviors, and language of the user groups for whom it was historical-software-collection-federal-agency
2 JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY—•• 2015 DOI: 10.1002/asi games as well as a system for citing in-game events and game not enumerate lists of criteria but rather summarize quality states.2 We expect that their findings will augment our previ- as “fitness for purpose” (Guy, Powell, & Day, 2004); that is, ous and current research efforts. does the metadata accomplish the purpose it was designed to Megan Winget led an IMLS-supported video game pres- achieve? For traditional bibliographic collections, the ervation project centered on studying the creation process and purpose of metadata is typically to allow users to find, iden- artifacts of the video game industry through observation of tify, select, and obtain materials of interest (IFLA, 2009), and interviews with game developers (Winget & Sampson, and high-quality metadata must accomplish these goals. 2011). The project specifically focused on supporting the However, methods for measuring metadata quality based collection and preservation of massively multiplayer online on these criteria vary. Many studies focus on evaluation at (MMO) games, whereas our work involves all genres of video the metadata-creation stage, with a special focus on how to games for a variety of platforms (i.e., console, handheld, improve quality at the point of entry. Currier, Barton, mobile, PC). Rather than focusing on the creation process of O’Beirne, and Ryan (2004) found that inaccurate data entry a game, our work emphasizes the user experience and and inconsistent subject vocabularies affected resource dis- describes the game information that is most relevant to users. covery. Simple metadata input tools, such as drop-down This is evidenced by several metadata elements such as menus, can significantly improve consistency and therefore mood, visual style, plot, and theme, that describe the content improve metadata quality (Greenberg et al., 2001). To evalu- or subject of the games. ate metadata quality after creation, other methods may be Additionally, the International Center for the History of used. One typical method is to assess quality through expert Electronic Games is currently testing the functionality of review. However, this raises the question of who qualifies as approximately 7,000 games in their collection and video an expert. For many mature metadata standards, such as recording old games, supported by IMLS.3 The International those in traditional bibliographic description, longstanding Game Developers Association Game Preservation Special communities such as the Program for Cooperative Catalog- Interest Group has also advocated for and published on the ing (PCC) and the Online Audiovisual Catalogers (OLAC) importance of preserving video games since 2004 (Newman, have come to consensus over time about definitions of 2009). We believe that our research effort will augment the quality (Hillmann, 2008). Such communities also benefit impact of these preservation efforts by greatly enhancing from longevity, because experts develop over time. Unfor- access to collections through robust metadata. tunately, nascent metadata standards by their very nature do not have such resources. In any event, regardless of length of participation or amount of experience, a reviewer’s expertise Evaluation of Metadata Schemas may still be considered subjective. Another typical method Metadata evaluation is a longstanding topic in library to evaluate metadata quality after creation is to check for and information science, yet perspectives vary on how to consistency in application among multiple metadata cre- perform such evaluation. Most evaluation seeks to identify ators. Commonly referred to as “interindexer consistency” and improve metadata quality, but few attempt to define because of its origins in indexing studies, this method quan- concretely what “quality” entails. Without established con- titatively measures the degree of agreement between differ- ceptual and operational definitions, it is difficult, if not ent indexers indexing the same document (Hughes & impossible, to evaluate metadata quality (Moen, Steward, & Rafferty, 2011). When used to evaluate metadata creation, it McClure, 1997). In the absence of definitions of quality, may not be limited to consistency of subject indexing term scholars and organizations offer different principles that application but can address the values for any element(s) in constitute quality metadata. Bruce and Hillmann (2004) a schema. Historical studies show that agreement ranges compiled seven criteria for metadata quality: completeness, vary considerably, possibly resulting from a lack of an accuracy, provenance, conformance to expectation, logical agreed-upon definition of what constitutes an exact value consistency and coherence, timeliness, and accessibility. match and also the use of different measurement calcula- National Information Standards Organization (NISO) tions (Hughes & Rafferty, 2011). Although consistency was (2007) identifies “good” metadata as metadata that conforms put forth earlier as a criterion for metadata quality, and to community standards; supports interoperability; uses interindexer consistency has been looked upon as an indica- authority control and other content standards; includes clear tor of quality (Funk, Reid, & McGoogan, 1983), consistency conditions for and terms of use; supports long-term curation may also be detrimental (Bloomfield, 2001). After all, what and preservation; and possesses the qualities of authority, good is consistently applied metadata if the actual value is authenticity, archivability, persistence, and unique identifi- incorrect? cation. After reviewing multiple studies, Park (2009) notes It should be noted that these evaluation techniques—as that completeness, accuracy, and consistency are the most with most methods of metadata evaluation in general—were commonly identified criteria for metadata quality. Others do designed to evaluate descriptive metadata; that is, recorded descriptive values as opposed to elements of a schema or a schema as a whole. Even less is written about how to evalu- 2http://www.imls.gov/news/2013_ols_grant_announcement.aspx#CA ate a schema as a holistic entity. One technique is to compare 3http://blog.imls.gov/?p=4091 it with other schemas, such as Beak and Olson’s (2011)
JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY—•• 2015 3 DOI: 10.1002/asi comparative analysis of a standard bibliographic metadata previously described evaluation methods where applicable. schema with a metadata schema specifically designed for To ensure that we are reflecting as much domain expertise as children. Such an analysis was able to highlight missing possible, we conducted part of this research project in the elements as well as elements directly supporting children’s form of a graduate-level course. It is almost impossible for information-seeking behavior, thus increasing access points any single person to have a comprehensive understanding of for children in system implementation. Other methods video games and interactive media across multiple plat- evaluate the entirety of a metadata schema by implementing forms, genres, and time periods, so it was essential that we it in a functional system and testing it in situ. For example, involve as many video game experts and enthusiasts as pos- Shukair, Loutas, Peristeras, and Sklarß (2013) developed a sible in the research process. In total, 21 people, including federated metadata schema for e-government and sought students, domain experts, and metadata professionals, par- user feedback on the schema through its deployment via a ticipated in phase I of the project; 18 in phase II; and 15 in user interface portal. Although the study reports positive phase III. Table 1 outlines our previous and current research user responses, it is difficult to determine whether the efforts for this project. success was due to the schema itself or the design of the user In phase I, we conducted an extensive domain analysis interface through which it was presented. No metadata using empirical data currently describing video games (Lee, schema is likely to be used without some sort of interface Tennis, & Clarke, (2012). We used existing information layer, so the effect of the interface design must always be sources (i.e., a variety of video game-related websites and taken into consideration. catalog records from sources such as Mobygames, Giant- This study evaluates our metadata schema by employing bomb, Allgame, Gamefaqs, Gamespot, IGN, Wikipedia, two methods: (a) schema testing by creating sample meta- WorldCat, and Amazon) to understand how the domain has data records of a test collection of video games and (b) user been shaped, how it is currently described, and where gaps evaluation of the schema through multiple data collection appeared. We also developed six different personas— methods. These are discussed further in the Study Design archetypes representing the needs, behaviors, and goals of a and Methods section. particular group of users (Cooper, 1999)—epitomizing the most common types of people interested in games. The six personas developed were representing game player, parent Current Status of the Project of youth game player, nostalgic collector, academic scholar, Since the inception of the video game metadata schema game developer/designer, and curator/librarian (Lee et al., development by GAMER and SIMM in 2011, we have 2013b). Based on these personas and several use scenarios, attempted to incorporate the strengths of all of the we evaluated the 61 elements and identified the 16 core
TABLE 1. Overview of previous and current research efforts.
Timeline and goals Methods and activities Outcomes
Phase I (2011–2012) Domain analysis: Examined how video games are 16 CORE elements identified and defined Establish the core set of elements organized in current systems and collected video for a metadata schema game metadata elements Personas: Evaluated the collected metadata elements from domain analysis based on six personas Phase II (2012–2013) Domain analysis: Continued to explore more game 1. 46 REC (recommended) elements (including 1. Establish the recommended set of websites to collect additional elements and terms CORE16) identified and defined based on user data elements for a metadata schema that can be used for controlled vocabularies and facet analysis results 2. Develop encoding schemes User interview: Selected which metadata elements are 2. New controlled vocabularies created for 17 (controlled vocabularies) for important to users based on the in-depth interviews elements select elements of 24 gamers Facet analysis: Conducted a facet analysis on select elements and defined the facets and foci Phase III (2013–2014) Collaborative review: Collaboratively analyzed and 1. CORE and REC metadata sets thoroughly revised Review and evaluation of the revised the current metadata schema and controlled based on the additional user data and feedback from current schema vocabularies metadata record creators Schema testing: Created metadata records of 65 2. Controlled vocabularies for eight elements also sample games in order to test the applicability of the revised and new controlled vocabularies created for schema two additional elements User interview: Conducted 32 additional interviews focusing on personas other than gamers to gather feedback on the schema Survey: Conducted a large-scale survey collecting 1,257 responses to obtain more generalizable results
4 JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY—•• 2015 DOI: 10.1002/asi elements deemed to be most useful for all user personas. identifying and extracting the metadata values were written These elements were further evaluated and revised based for each element. Several controlled vocabularies were on cataloging sample games. More detailed information on revised, and two new vocabularies were created. our research activity in phase I can be found in Lee et al. (2013b). Schema testing. The schema and encoding schemes In phase II, we established the recommended set of ele- (syntax and vocabulary) were further evaluated by applying ments for a metadata schema (Lee, Cho, Fox, & Perti, them to examples of video games to create sample metadata 2013a) and began developing encoding schemes/controlled records of those games. These records were created by the vocabularies for select elements (Donovan, Cho, Magnifico, students participating in the course. The students had a wide & Lee, 2013; Lee, Karlova, Clarke, Thornton, & Perti, range of experience in cataloging (from novices to profes- 2014b). In addition to the core 16, 30 additional elements sionals) as well as gaming experience (from casual to dedi- were recommended to describe video games to the level of cated gamers). As a group, they cataloged a total of 65 thoroughness useful to the various user groups. As we began sample games which consisted of traditional console games to develop vocabularies for many of these elements, it for a variety of platforms, digitally distributed game apps, became evident that metadata from extant sources signifi- and flash-based games. The full list of sample games cata- cantly vary with regard to the types and granularity of the loged is provided in the Appendix (Table 6). Each game was terms. For example, most websites did not provide defini- cataloged twice according to the schema, once each by two tions for genre labels, and those that did were not consistent different students. The results from comparing the two meta- with definitions from other sites. We also found that infor- data records created for each game in conjunction with feed- mation sources varied widely in terms of reliability and back from the students allowed us to identify successes, availability. Dates and features varied, and descriptions of issues, and areas for improvement. features were often colored by subjective marketing propa- ganda. Definitions and differences between “developer” and User Evaluation of the Schema “publisher” were unclear. Differentiating among various editions of games was extremely challenging as a result of User interviews. In addition to the previous 24 interviews, releases of the same game in multiple regions or for multiple we interviewed 32 participants, all over 18 years old, who consoles or systems, as well as special “collector’s editions” find, play, purchase, collect, and recommend video games, or other limited special releases. Finding information on to investigate their current practices. Unlike the case for previous versions of digitally distributed games such as phase II, in which the researchers focused mostly on inter- game apps for smartphones or tablets was also exceedingly viewing gamers, in phase III the interviewees represented a difficult (Lee, Clarke, & Perti, 2014a). wider range of user groups: three casual gamers, five avid In the current phase of the study, phase III, we focused on gamers, four parents of youth players, six game collectors, revising and evaluating the current schema and controlled five game industry professionals, six curators/librarians vocabularies. There were two concurrent research streams: managing video game collections, and three scholars con- (a) refining the schema as a group and testing its applicabil- ducting game research. Interviewees were recruited by ity using sample games and (b) gathering user feedback on snowball sampling. Gamers were initially recruited via the the schema through interviews and a large-scale online SIMM’s public exhibition and librarians via the American survey. More detailed discussion on research activities from Library Association (ALA)’s annual conference and the phase III is provided in the following section. YoungAdult Library Services Association listserv. Scholars, employees, and parents were recruited via researchers’ per- sonal connections and references. Collectors were recruited Study Design and Methods via game-related forums. The interview protocol included specific questions about gaming experiences, game-related Refining and Testing the Schema information needs and search behaviors, feedback on the Collaborative review. The previous metadata work from current metadata schema, and so on. We were also interested phases I and II was revised through an iterative and collab- in exploring what participants had to say about games. Inter- orative expert review process. Each student independently views for some user groups had specific questions (e.g., wrote up his feedback on the initial list of personas, the librarians, people in game industry, parents, scholars). All elements and structure of the schema, and the controlled the interviews were conducted between July and October, vocabularies, guided by personal expertise and experience 2013, in person, over the telephone, or on Skype. Each with video games. To the method of expert review, we added interview lasted for approximately 45 minutes to 1 hour. a level of collaborative consensus, in which summary of the Each interviewee was compensated with a $20 Amazon gift feedback was reviewed by the group, and every aspect was card for their participation. discussed until a convergence of views was achieved. As a Interviews were fully transcribed and analyzed to obtain result, new elements were included to make the schema a detailed qualitative description of behavior surrounding more pertinent to digitally distributed games (Lee et al., video game use, purchases, recommendations, organiza- 2014a). All definitions were refined, and instructions for tional needs, and game seeking. The code book contained
JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY—•• 2015 5 DOI: 10.1002/asi codes for various aspects, such as user behaviors, game- completed the survey. Participants took an average of 20 related resources, and appeal factors. The code book minutes and 49 seconds to complete the survey. was created through an iterative coding process. Initially, we selected a subset of interview transcripts from which we Findings and Discussion developed a preliminary code book. Using this code book, two coders independently coded the transcripts once. A Overview series of meetings discussing the code revisions followed. This section provides a summary of our findings from Using the revised code book that resulted from these conducting a collaborative review, testing the schema, and meetings, the two coders reviewed and recoded the tran- analyzing the user feedback provided through an online scripts. Afterward, a third party reviewed all the coding survey and interviews. We describe each of the issues iden- work, identifying any issues and/or inconsistencies in apply- tified, followed by an explanation of the changes made to the ing the codes. The instances of questionable code applica- schema to address those issues. tions were discussed among all three (i.e., two coders and In the survey, we asked the users what information about the third-party reviewer) until consensus was reached. More games is most useful to them for games they are currently detailed description of the coding process as well as a full playing and also when they are looking for new games to discussion of findings is provided in another article in play. Table 2 shows the complete list of metadata elements4 preparation (Lee, Clarke, & Rossi, in preparation). The sorted by the proportion of positive responses. The count in current article will focus primarily on user feedback with the sixth column represents the number of responses stating regard to the metadata. that the particular element is useful for at least one of the two cases (i.e., for games currently playing or in seeking new Survey. We conducted a large-scale online survey asking games). 23 questions regarding users’ gaming experience and The elements