DEVISE A framework for the evaluation of Internet search engines by F C Johnson, J R Griffiths and R J Hartley Resource: The Council for Museums, Archives and Libraries 2001 Abstract This project investigates the feasibility of the use of user-satisfaction as a multidimensional evaluative construct of search engines. Search engine developments are reviewed to reveal a range of indexing and retrieval techniques that may assist casual users in the information retrieval task. Yet few evaluation studies have considered the impact of system features, in particular those with which the user interacts for search assistance. A broad review of retrieval system evaluation highlights the complex environment in which measures of both the utility of the search results and the usability of the system are sought from a user- perspective. Our proposed approach for a user-centered evaluation is based on a conceptual framework in which user-satisfaction is characterised as a variable dependent on system features and functions and expressed in a moderating context of user-task requirement. Towards this end, the research reported here focuses on the definition of the construct of user satisfaction on the multi dimensions of the retrieval process, an expression of what a typical user is trying to do. Empirical work was then undertaken to test the feasibility and potential value of the implementation of the framework for the evaluation of three search engines. Initial results are presented which provide a degree of understanding of how users are satisfied and on what criteria. This provides the basis on which we make recommendations for the refinement of the multidimensional framework and its use as a methodology for the evaluation of search engines from a user perspective. For further information or correspondence please contact the Project Director or the Project Research Fellow at the address below: Dr Frances Johnson Jillian Griffiths Project Director Research Fellow Centre for Research in Library & Information Management (CERLIM) Department of Information & Communications The Manchester Metropolitan University, Geoffrey Manton Building Manchester, M15 6LL Tel: +161-247-6156; Fax: +161-247-6351 [email protected] , [email protected] Ó Resource: The Council for Museums, Archives and Libraries 2001 The opinions expressed in this report are those of the authors and not necessarily those of Resource: The Council for Museums, Archives and Libraries Library and Information Commission Research Report 100 RE/085 ISBN 1 902394 61 5 ISSN 1466-2949 Library and Information Commission Research Reports are published by Resource: The Council for Museums, Archives and Libraries and may be purchased as photocopies or microfiche from the British Thesis Service, British Library Document Supply Centre, Boston Spa, Wetherby, West Yorkshire LS23 7BQ, UK. ii Content Page CHAPTER 1. INTRODUCTION ...............................................................................................................................1 1.1 RATIONALE FOR A USER-EVALUATION OF SEARCH ENGINES............................................................................2 1.1.1 Internet Search Engines .................................................................................................................................2 1.1.2 Evaluation........................................................................................................................................................3 1.1.3 Development of a framework for user-centered evaluation of search engines....................................6 CHAPTER 2. INTERNET SEARCH ENGINES: FACTORS WHICH AFFECT PERFORMANCE.8 2.1 SEARCH SERVICE COVERAGE ................................................................................................................................8 2.2 INDEXING STRATEGIES ...........................................................................................................................................9 2.3 SEARCH FEATURES (USER CONTROL OF SEARCH)...............................................................................................9 2.3.1 Users & usage of search features............................................................................................................... 10 2.4 SEARCH FEATURES (SYSTEM CONTROL OF QUERY).......................................................................................... 11 2.4.1 Query expansion........................................................................................................................................... 12 2.4.2 Query modification ...................................................................................................................................... 12 2.4.3 Query visualisation ...................................................................................................................................... 13 2.4.4 Popular queries ............................................................................................................................................. 13 2.5 RESULTS DISPLAY ................................................................................................................................................. 15 2.5.1 Ranking.......................................................................................................................................................... 15 2.6 CHAPTER SUMMARY............................................................................................................................................. 17 CHAPTER 3. A REVIEW OF IR SYSTEM EVALUATION........................................................................18 3.1 CRANFIELD STUDIES............................................................................................................................................. 18 3.1.1 Cranfield -type evaluations of search engines .......................................................................................... 19 3.1.1.1 Recall......................................................................................................................................................... 19 3.1.1.2 Dynamic database...................................................................................................................................... 20 3.1.1.3 Relevance criteria ...................................................................................................................................... 20 3.1.1.4 Queries....................................................................................................................................................... 21 3.2 USER-ORIENTED EVALUATION OF INTERACTIVE RETRIEVAL SYSTEMS........................................................ 22 3.2.1 Utility ............................................................................................................................................................. 23 3.2.1.1 Ranking..................................................................................................................................................... 24 3.2.2 Usability......................................................................................................................................................... 25 3.2.3 Searcher contexts .......................................................................................................................................... 26 3.2.4 User satisfaction........................................................................................................................................... 27 3.3 USER SATISFACTION BASED EVALUATIONS OF SEARCH ENGINES.................................................................. 28 3.4 CHAPTER SUMMARY............................................................................................................................................. 29 CHAPTER 4. DEVELOPMENT OF THE FRAMEWORK...........................................................................31 4.1 INTRODUCTION...................................................................................................................................................... 31 4.1.2 Measures........................................................................................................................................................ 35 4.2 TASK DIMENSIONS AND USER MEASURES........................................................................................................ 36 4.2.1 Context ........................................................................................................................................................... 40 4.3 IMPLEMENTATION................................................................................................................................................. 40 4.4 DATA ANALYSIS ................................................................................................................................................... 42 4.5 DISCUSSION AND CONCLUSIONS.......................................................................................................................48 REFERENCES ...............................................................................................................................................................50 List of Tables Table 1 Comparison of evaluation criteria and system/context parameters .............................................................5 Table 2 Search engine coverage ......................................................................................................................................8