Usability of the Digital Library: an Evaluation Model
Total Page:16
File Type:pdf, Size:1020Kb
USABILITY OF THE DIGITAL LIBRARY: AN EVALUATION MODEL by JUDY H. JENG A Dissertation submitted to the Graduate School - New Brunswick Rutgers, The State University of New Jersey in partial fulfillment of the requirements for the degree of Doctor of Philosophy Graduate Program in Communication, Information, and Library Studies written under the direction of Professor Tefko Saracevic id approved by'" New Brunswick, New Jersey January, 2006 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. UMI Number: 3203382 Copyright 2006 by Jeng,Judy H. All rights reserved. INFORMATION TO USERS The quality of this reproduction is dependent upon the quality of the copy submitted. Broken or indistinct print, colored or poor quality illustrations and photographs, print bleed-through, substandard margins, and improper alignment can adversely affect reproduction. In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if unauthorized copyright material had to be removed, a note will indicate the deletion. ® UMI UMI Microform 3203382 Copyright 2006 by ProQuest Information and Learning Company. All rights reserved. This microform edition is protected against unauthorized copying under Title 17, United States Code. ProQuest Information and Learning Company 300 North Zeeb Road P.O. Box 1346 Ann Arbor, Ml 48106-1346 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. ®2006 Judy H. Jeng ALL RIGHTS RESERVED Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. ABSTRACT OF THE DISSERTATION Usability of the Digital Library: An Evaluation Model By JUDY H. JENG Dissertation Director: Professor Tefko Saracevic The main research goal of this dissertation is to develop a model and a suite of instruments to evaluate the usability of academic digital libraries. Usability in this study was examined from the perspectives of effectiveness, efficiency, satisfaction, and leamability. The model that was developed considers both quantifying elements such as accuracy rate, time, and number of steps required to complete a task, and the subjective criterion, satisfaction, which is further broken down into the areas of ease of use, organization of information, terminology and labeling, visual appearance, content, and mistake recovery. The model was tested at two academic digital libraries. It is a cross- institutional usability study. The study recruited a total of forty-one subjects and was divided into two stages to confirm findings. The study examined the interlocking relationships among effectiveness, efficiency, and satisfaction. It found that the relationships are statistically significant. The longer the time it took for a subject to perform a task, the more steps involved. The greater the number of steps involved in completing a task, the lower the satisfaction expressed by the subject. The longer the time spent on completing a task, the lower the satisfaction expressed by the subject. However, it should be noted that each criterion has its own emphasis and should be measured separately. ii Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. The research uncovered the user’s criteria for ease o f use, organization of information, terminology, visual attractiveness, and mistake recovery. The findings are helpful in enhancing the usability of the digital library. The issues of “user lostness” and navigation disorientation were examined, and the causes were identified. The study also examined the click cost issue. It found that 73% of the participants expected the click(s) to lead them eventually to the correct answer. Each click in the digital library should bring users closer to the answer. The study established that the demographic factors (gender, age, status, academic major, ethnic background, years at the institution, and frequency of using library Web sites) do not have a statistically significant impact on performance (i.e., effectiveness and efficiency). However, different attitudes among ethnic groups were found in rating satisfaction. Future research should recruit more subjects in each ethnic group to examine this cross-culture usability issue further. iii Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. ACKNOWLEDGMENTS I sincerely want to thank the following people for their assistance to me throughout this endeavor: Tefko Saracevic, Nick Belkin, Colleen Cool, Dan O’Connor, and Marija Dalbello. iv Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. DEDICATION This dissertation is dedicated with love to my parents, my husband, and my daughters. Without my family’s love, understanding, encouragement, and support, I could not have accomplished this. v Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. TABLE OF CONENTS Abstract ii Acknowledgments iv Dedication v Table of Contents vi List of Tables x List of Figures xiii CHAPTER 1 INTRODUCTION 1 1.1 Background 1 1.2 Problem Statement 2 1.3 Research Goals 3 1.4 Organization of This Dissertation 3 CHAPTER 2 LITERATURE REVIEW 5 2.1 Definitions of Digital Library 5 2.2 Dimensions of Usability 10 2.3 Techniques of Usability Evaluation 20 2.4 Selected Usability Tests of Digital Libraries 24 2.5 Synthesis of Findings from Digital Library Usability Tests 38 CHAPTER 3 USABILITY EVALUATION MODEL 45 CHAPTER 4 METHODS OF DATA COLLECTION 51 4.1 Sites 51 4.2 Stages 51 4.3 Subjects 52 4.4 Tasks 52 4.5 Instruments 53 4.6 Setting 54 4.7 Procedure 55 4.8 Pilot Study 55 CHAPTER 5 DATA ANALYSIS 56 5.1 Demographic Distribution 56 5.2 Effectiveness 59 5.3 Efficiency 64 5.4 Leamability 69 5.5 Satisfaction 73 5.5.1 Overall Satisfaction 73 5.5.2 Ease of Use 77 5.5.3 Organization of Information 81 5.5.4 Terminology 86 5.5.5 Attractiveness 88 5.5.6 Mistake Recovery 92 vi Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 5.5.7 Overall Reaction 94 5.6 Best Features 95 5.7 Worst Features 99 5.8 Desired Features 104 5.9 Lostness 108 5.10 Navigation 109 5.11 Click Cost 110 5.12 Relationships among Effectiveness, Efficiency, and Satisfaction 112 5.12.1 Effectiveness and Satisfaction 112 5.12.2 Efficiency and Satisfaction 113 5.12.3 Effectiveness and Efficiency 114 5.12.3a Effectiveness and Steps 114 5.12.3b Effectiveness and Time 116 5.13 Leamability and Ease of Use 118 5.14 Demographic Factors and Effectiveness 118 5.14.1 Gender and Effectiveness 119 5.14.2 Age and Effectiveness 120 5.14.3 Status and Effectiveness 120 5.14.4 Major and Effectiveness 120 5.14.5 Ethnic Background and Effectiveness 121 5.14.6 Institution Experience and Effectiveness 121 5.14.7 Frequency of Using Library Web Site and Effectiveness 122 5.14.8 Interaction Effects of Gender, Frequency of Use, and Years at the Institution on Effectiveness 122 5.15 Demographic Factors and Steps 123 5.15.1 Gender and Steps 123 5.15.2 Age and Steps 123 5.15.3 Status and Steps 124 5.15.4 Major and Steps 124 5.15.5 Ethnic Background and Steps 124 5.15.6 Institution Experience and Steps 125 5.15.7 Frequency of Using Library Web Site and Steps 125 5.16 Demographic Factors and Time 125 5.16.1 Gender and Time 125 5.16.2 Age and Time 126 5.16.3 Status and Time 126 5.16.4 Major and Time 127 5.16.5 Ethnic Background and Time 127 5.16.6 Institution Experience and Time 127 5.16.7 Frequency of Using Library Web Site and Time 128 5.17 Gender and Satisfaction 128 5.17.1 Gender and Ease of Use 128 5.17.2 Gender and Organization of Information 129 5.17.3 Gender and Terminology 130 5.17.4 Gender and Attractiveness 130 5.17.5 Gender and Mistake Recovery 131 vii Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 5.18 Ethnic Background and Satisfaction 131 5.18.1 Ethnic Background and Ease ofUse 131 5.18.2 Ethnic Background and Organization of Information 133 5.18.3 Ethnic Background and Terminology 136 5.18.4 Ethnic Background and Attractiveness 136 5.18.5 Ethnic Background and Mistake Recovery 138 5.19 Summary of Findings 139 5.19.1 Analysis by Task 139 5.19.2 Leamability 141 5.19.3 Criteria of Ease o f Use, Organization o f Information, Visual Terminology, Attractiveness, and Mistake Recovery from User’s Perspective 141 5.19.4 Relationships among Criteria 142 5.19.5 Demographic Factors and Criteria 143 CHAPTER 6 DISCUSSION 145 6.1 Criteria of Usability 146 6.2 Number of Subjects 146 6.3 Interlocking Relationships among Effectiveness, Efficiency, and Satisfaction 148 6.4 Ease of Learning and Ease of Use 149 6.5 User Lostness 151 6.6 Navigation 151 6.7 Click Cost 152 6.8 Demographic Factors on Performance 152 6.9 Cross-Cultural Usability 153 6.10 Gender Effect on Performance and Satisfaction 154 6.11 Terminology 155 6.12 Performance Data 156 6.13 Other Observations 158 6.14 Other Discussion 158 CHAPTER 7 CONCLUSIONS 161 7.1 Conclusions 161 7.2 Limitations 167 7.3 Future Research 168 7.4 Contributions 168 APPENDIX A-l Participant Information and Consent Form for Rutgers Subjects 170 A-2 Consent Form for Queens Subjects 171 B Pre-Test Questionnaire 174 C Usability Testing Questions 175 D-l Post-Test Questionnaire (1st Stage) 178 D-2 Post-Test Questionnaire (2nd Stage) 181 REFERENCES 184 viii Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. CURRICULUM VITA 200 ix Reproduced with permission of the copyright owner.