Usage and Usability Assessment: Library Practices and Concerns by Denise Troll Covey January 2002 Digital Library Federation Council on Library and Information Resources Washington, D.C. ii About the Author Denise Troll Covey is associate university librarian of arts, archives, and technology at Carnegie Mellon University. In 2000-2001 she was also a distinguished fellow in the Digital Library Federation, leading the initiative on usage, usability, and user support. Her professional work focuses on the research and development of digital library collections, services, and software; and assessment practices, copyright permissions, and change management as they relate to digital libraries. Covey has academic degrees in theology, philosophy, and rhetoric. Her graduate work emphasized the history of information storage and retrieval. Acknowledgments The author and the Digital Library Federation sincerely thank the 71 individuals who participated in the DLF telephone survey. Their time, experiences, concerns, and questions about library use and usability made this report possible. If the report facilitates discussion and research and encourages the development of benchmarks or best practices, it is because so many talented people shared their rewards and frustrations in trying to understand what is happening in their libraries and how to serve their users better. ISBN 1-887334-89-0 Published by: Digital Library Federation Council on Library and Information Resources 1755 Massachusetts Avenue, NW, Suite 500 Washington, DC 20036 Web site at http://www.clir.org Additional copies are available for $20 per copy. Orders must be placed online through CLIR’s Web site. 8 The paper in this publication meets the minimum requirements of the American National Standard for Information Sciences—Permanence of Paper for Printed Library Materials ANSI Z39.48-1984. Copyright 2002 by the Council on Library and Information Resources. No part of this publication may be reproduced or transcribed in any form without permission of the publisher. Requests for reproduction should be submitted to the Director of Communications at the Council on Library and Information Resources. iii Contents Preface ............................................................................................................................................. v 1. Introduction ................................................................................................................................1 1.1. Report Structure .......................................................................................................1 1.2. Summary of Challenges in Assessment ................................................................2 2. User Studies ................................................................................................................................5 2.1. Surveys (Questionnaires) ........................................................................................6 2.1.1. What Is a Survey Questionnaire? ..............................................................6 2.1.2. Why Do Libraries Conduct Surveys? .......................................................7 2.1.3. How Do Libraries Conduct Surveys? .......................................................8 2.1.4. Who Uses Survey Results? How Are They Used? ................................10 2.1.5. What Are the Issues, Problems, and Challenges With Surveys? .........12 2.1.5.1. The Costs and Benefits of Different Types of Surveys .............12 2.1.5.2. The Frequency of Surveys............................................................12 2.1.5.3. Composing Survey Questions .....................................................12 2.1.5.4. Lack of Analysis or Application ..................................................14 2.1.5.5. Lack of Resources or Comprehensive Plans .............................14 2.2. Focus Groups ..........................................................................................................15 2.2.1. What Is a Focus Group? ............................................................................15 2.2.2. Why Do Libraries Conduct Focus Groups? ...........................................17 2.2.3. How Do Libraries Conduct Focus Groups? ...........................................18 2.2.4. Who Uses Focus Group Results? How Are They Used? ......................21 2.2.5. What Are the Issues, Problems, and Challenges With Focus Groups? ..........................................................................22 2.2.5.1. Unskilled Moderators and Observers ........................................22 2.2.5.2. Interpreting and Using the Data .................................................22 2.3. User Protocols .........................................................................................................23 2.3.1. What Is a User Protocol? ...........................................................................23 2.3.2. Why Do Libraries Conduct User Protocols? ..........................................24 2.3.3. How Do Libraries Conduct User Protocols? ..........................................25 2.3.4. Who Uses Protocol Results? How Are They Used? ..............................28 2.3.5. What Are the Issues, Problems, and Challenges With User Protocols? .........................................................................29 2.3.5.1. Librarian Assumptions and Preferences ....................................29 2.3.5.2. Lack of Resources and Commitment .........................................29 2.3.5.3. Interpreting and Using the Data .................................................29 2.3.5.4. Recruiting Participants Who Can Think Aloud........................30 iv 2.4. Other Effective Research Methods.......................................................................30 2.4.1. Discount Usability Research Methods ....................................................30 2.4.1.1. Heuristic Evaluations ...................................................................30 2.4.1.2. Paper Prototypes and Scenarios..................................................32 2.4.2. Card-Sorting Tests ......................................................................................32 3. Usage Studies of Electronic Resources .................................................................................33 3.1. What Is Transaction Log Analysis? ......................................................................33 3.2. Why Do Libraries Conduct Transaction Log Analysis?....................................35 3.3. How Do Libraries Conduct Transaction Log Analysis? ...................................35 3.3.1. Web Sites and Local Digital Collections .................................................35 3.3.2. OPAC and Integrated Library Systems...................................................37 3.4. Who Uses the Results of Transaction Log Analysis? How Are They Used? .................................................................................37 3.4.1. Web Sites and Local Digital Collections .................................................37 3.4.2. OPAC and Integrated Library Systems...................................................39 3.4.3. Remote Electronic Resources ....................................................................40 3.5. What Are the Issues, Problems, and Challenges With Transaction Log Analysis? ...............................................................................................40 3.5.1. Getting the Right (Comparable) Data and Definitions .........................40 3.5.1.1. Web Sites and Local Digital Collections ....................................40 3.5.1.2. OPAC and Integrated Library Systems......................................41 3.5.1.3. Remote Electronic Resources .......................................................42 3.5.2. Analyzing and Interpreting the Data ......................................................43 3.5.3. Managing, Presenting, and Using the Data ...........................................43 4. General Issues and Challenges ..............................................................................................44 4.1. Issues in Planning a Research Project ..................................................................44 4.2. Issues in Implementing a Research Project ........................................................48 4.2.1. Issues in Sampling and Recruiting Research Subjects ..........................48 4.2.2. Issues in Getting Approval and Preserving Anonymity ......................51 5. Conclusions and Future Directions .......................................................................................53 APPENDIX A: References and Selected Bibliography ..........................................................61 APPENDIX B: Participating Institutions ................................................................................75 APPENDIX C: Survey Questions .............................................................................................76 APPENDIX D: Traditional Input, Output, and Outcome Measures ...................................77 v Preface Making library services available online is not only expensive; it is also very risky. The library’s roles there are not at all clear. Neither are its relationships with users or with other information services. There is little information about how library users behave in a network environment, how they react to online library services, and how they combine those services with others such as search engines like Google, bookstores