SAVVYSEARCH: a Metasearch Engine That Learns Which Search

AI Magazine Volume 18 Number 2 (1997) (© AAAI) Articles SAVVYSEARCH A Metasearch Engine That Learns Which Search Engines to Query

Adele E. Howe and Daniel Dreilinger

■ Search engines are among the most successful single query to multiple search engines. applications on the web today. So many search The simplest metasearch engines are forms engines have been created that it is difficult for that allow the user to indicate which search users to know where they are, how to use them, engines should be contacted (for example, and what topics they best address. Metasearch ALL-IN-ONE [www.albany.net/allinone] and engines reduce the user burden by dispatching METASEARCH [members.gnn.com/infinet/ queries to multiple search engines in parallel. The meta.htm]). PROFUSION (Gauch, Wang, and SAVVYSEARCH metasearch engine is designed to efficiently query other search engines by carefully Gomez 1996; www.designlab.ukans.edu/pro- selecting those search engines likely to return fusion) gives the user the choice of selecting useful results and responding to fluctuating load search engines themselves or letting PROFUSION demands on the web. SAVVYSEARCH learns to iden- select three of six robot-based search engines tify which search engines are most appropriate using hand-built rules. METACRAWLER (Selberg for particular queries, reasons about resource and Etzioni 1995; metacrawler.cs.washing- demands, and represents an iterative parallel ton.edu:8080/home.html) significantly search strategy as a simple plan. enhances the output by downloading and analyzing the links returned by the search engines to prune out unavailable and irrele- ompanies, institutions, and individuals vant links. must have a presence on the web; each Metasearch engines reduce the burden on Care vying for the attention of millions the user. They make available search engines of people. Not too surprisingly then, the most that might have been unknown to the user. successful applications on the web to date are They handle the simultaneous submission of search engines: tools that assist users in queries; some direct the query to appropriate finding information on specific topics. engines, and some postprocess the results as A variety of search engines are available, well. They provide a single interface (on the from general, robot based (for example, down side, they might not support all the fea- ALTAVISTA [www.altavista.digital.com] and tures of the target search engines). WEBCRAWLER [webcrawler.com]) to topic or area Unfortunately, metasearch can lead to the specific (for example, FTPSEARCH [ftpsearch. tragedy of the commons problem from eco- unit.no/ftpsearch] and DEJANEWS [www. nomics in which an individual’s best interests dejanews.com]). Each uses different algo- run counter to society’s. Individual users rithms for collecting, indexing, and searching appear to be best served by simultaneously links; thus, each returns different results for searching every possible search engine on the similar queries. Empirical results indicate that web for desired information. However, the no single search engine is likely to return process might waste web resources: network more than 45 percent of the relevant results load and search-engine computation. (Selberg and Etzioni 1995). To find what they We believe that a metasearch system can be desire, users might need to query several a good web citizen (Eichmann 1994) by tar- search engines; metasearch engines automate geting those search engines likely to return this process by simultaneously submitting a useful results and responding to changing

load demands on the web. To provide this be present) or can be presented as an ordered function, we incorporated simple AI tech- phrase. Three aspects of the results display niques in a metasearch engine. Our can be varied: (1) the number of links metasearch engine learns to identify which returned, (2) the format of the links descrip- search engines are most appropriate for par- tion, and (3) the timing. By default, 10 links ticular queries, reasons about resource are displayed with the uniform resource loca- demands, and represents an iterative parallel tors (URLs) and descriptions when available, search strategy as a simple plan. and the results of each search engine are listed separately as they arrive. Alternatively, we could change the number of links to 50, Our Metasearch System: return less or more description of the links, SAVVYSEARCH and interleave the results of the separate search engines. The interface is also available S AVVYSEARCH is our metasearch system in 23 different languages.1 (Dreilinger and Howe 1997, 1996) available at guaraldi.cs.colostate.edu:2000. It runs on five Processing a Query machines (three Sun SPARCSTATIONs and two When a user submits the query, SAVVYSEARCH IBM RS 6000s) at Colorado State University. must make two decisions: (1) how many The system was first made available in March search engines to contact simultaneously and AVVYSEARCH 1995 and has undergone several revisions S (2) what order the search engines should be since the original design. At present, two ver- is designed to contacted in. The first decision requires rea- sions of the system are available: (1) the one soning about the available resources and the balance two described here and (2) an experimental inter- second about ranking the search engines. potentially face that is mentioned in the section entitled Retrospective and Prospective Views of the Resource Reasoning Each search engine conflicting SAVVYSEARCH Project. queried expends network and local computational resources. Thus, modifying concurren- goals: (1) SAVVYSEARCH is designed to balance two potentially conflicting goals: (1) maximizing cy (number of search engines queried in par- maximizing the likelihood of returning good links and (2) allel) is the best way to moderate resource the likelihood minimizing computational and web resource consumption. Concurrency is a function of consumption. The key to compromise is network load estimates, which are determined of returning knowing which search engines to contact for from a lookup table created from observations of the network load at this time of day good links specific queries at particular times. SAVVY- in the past, and local central processing unit and (2) SEARCH tracks long-term performance of search engines on specific query terms to load, which is computed using the UNIX minimizing determine which are appropriate and moni- uptime command. computa- tors recent performance of search engines to Concurrency has a base value of two; as determine whether it is even worth trying to many as two additional units are added for tional and contact them. each load estimate for periods of low load. Thus, the maximum concurrency value is six. web resource In this section, we describe SAVVYSEARCH from a user’s perspective. We follow a run- Ranking Search Engines SAVVYSEARCH consumption. ning example, indicating what a user sees includes both large robot-based search engines and what goes on behind the scenes in pro- and small specialized search engines in its set. cessing a search request. The large search engines are likely to return links for any query, but these links might not Submitting a Query be as appropriate as links returned by a spe- To find out about artificial intelligence con- cialized search engine for a query in its area. ferences, we enter the query, select the Inte- The purpose of ranking is to determine grate Results option, and click on the SAVVY- which search engines are most worthwhile to SEARCH! button, as shown in an image of the contact for a given query. Search engines are interface in figure 1. The search form, the ranked based on learned associations between query interface to SAVVYSEARCH, asks the user search engines and query terms (stored in a to specify a set of keywords (query terms) and metaindex) and recent data on search-engine options for the search. Users typically enter performance. two query terms. The metaindex—A compendium of The options cover the treatment of the search experience: The metaindex maintains terms, the display of results, and the interface associations between individual query terms language. Query terms can be combined with (simplified by stemming and case stripping) logical and (all query terms must be included and search engines as effectiveness values. in documents) or or (any query term should High positive values indicate excellent perfor-

20 AI MAGAZINE Articles

Figure 1. User Interface to SAVVYSEARCH for Entering a Query. mance of a search engine on queries contain- as some questionable responses. For each ing a speciﬁc term; high negative values indi- search, we collect two types of information: cate extremely poor performance. (1) no results, the search engine failed to The effectiveness values are derived from return links, and (2) visits, the number of two types of observation of the results of links that are explored by the user. No results users’ searches. We used observations (passive reduces conﬁdence that the search engine is measures) because we obtained a low rate of appropriate for the particular query, and Vis- response to requests for user feedback as well its indicates that the user found some

SUMMER 1997 21 Articles

returned links to be interesting and so and are likely to return something for any increases confidence. query, their weights will tend to grow larger SAVVYSEARCH uses a simple weight-adjust- and more quickly than the specialized search ment scheme for learning effectiveness val- engines. Ts mitigates the tendency toward ues. No results and Visits are treated as nega- their dominance, allowing more specialized tive and positive reinforcement, respectively, engines to be selected when appropriate. amortized by the number of terms in the The penalties (Ps,h and Ps,r) are accrued query. Thus, if a search engine returned noth- only if thresholds on the minimum number ing for the example query, the effectiveness of hits and the maximum response time are values for artificial, intelligence, and conferences exceeded. The thresholds have been set some- would each be reduced by one-third. what arbitrarily to an average of 1 hit and a Although simple, this scheme proved to be response time of 15 seconds. Once the effective (see Retrospective and Prospective thresholds are passed, the penalties increase Views of the SAVVYSEARCH Project for a brief quadratically to the worst-possible values: description of our evaluation of the learning). zero for hits and a time-out of 45 seconds for Tracking recent performance: Search response time. Details of these equations are engines occasionally are inaccessible or slow available in Dreilinger (1996). Just as to respond. Their network connections might primary be at fault, or the engines themselves might Dispatching a Query be experiencing problems or upgrades. SAVVY- The degree of parallelism determines how many search SEARCH monitors recent performance by search engines to query; the rank order deter- engines are recording the number of links returned (hits) mines which search engines should be queried. and the response time for the last five queries SAVVYSEARCH dispatches the query in parallel to now a submitted to each search engine. Given cur- each of the indicated search engines. For each resource for rent use of the system, 5 queries corresponds search engine, a specialized interface agent for- to about 45 seconds for the large, general mats the query according to the interface for metasearch search engines and about 15 minutes for the the search engine and submits it. The interface engines, so infrequently used search engines. agent waits a preset amount of time for a metasearch Calculating rank from experiences: For a response, handles errors that might occur, pars- given query (q), a search engine’s (s) rank es the result into a uniform format, and for- engines (Rq,s) is its query score reduced by a penalty wards it to another component for display. for recent poor performance on hits (h) and should response time (r): Presenting Results

become a Rq,s = Qq,s – (Ps,h + Ps,r) . For our example query, the three search means of Each term is normalized to a point between 0 engines contacted (WEBCRAWLER, LYCOS, and and 1. YELLOW PAGES) returned 30 different links with achieving similar names. The ﬁrst nine are shown in an The equation for Qq,s is based on a com- higher-level mon approach from information retrieval image of the result page in ﬁgure 2. These goals rather called term frequency times inverse document results were integrated; SAVVYSEARCH waited frequency (Witten, Moffat, and Bell 1994). The until all results had been received and then than an end query score, Qq,s, sums the metaindex values constructed a single ranked list. Results are in itself. for the terms in the query weighted by the integrated by normalizing the scores returned ubiquity of the query term and the search by search engines between 0 and 1.0 and engine: summing them for each link; links for search engines that did not return scores were arbi- M ∗ I = t,s t Qq,s ∑ trarily assigned a score of 0.5. Duplicate links ∈ T t q s are listed with the names of all the search

Mt,s is the weight from the metaindex of the engines that returned them. term t for search engine s. It is the inverse serv- The bottom of each results page displays er frequency of the term, which estimates the the search plan, as shown in ﬁgure 3. The ubiquity of the query term; frequently occur- query was issued during a time of moderately ring, common terms provide little informa- high demand; thus, each step includes only tion for distinguishing search-engine perfor- three search engines. Search plans can mance and so are discounted. Ts sums the include from two to six search engines for absolute values of all metaindex values for each step. The current version includes 11 search engine s; this term estimates the fre- search engines; the number fluctuates as quency of the overall use of the search search engines appear, disappear, and merge. engine. Because the most general search To supplement the results collected so far, engines are likely to be used more frequently the user can execute another step in the

22 AI MAGAZINE Articles

Figure 2. SAVVYSEARCH Display of Results of Artiﬁcial Intelligence Conferences Query, Interleaved Display. search plan by clicking on the one that looks Retrospective and Prospective the most promising. The ordering represents SAVVYSEARCH’s best guess about which search Views of the SAVVYSEARCH engines will return the best results; the user Project Results can easily override it by selecting a different step. In one long-term study, we found that The SAVVYSEARCH Project has been quite suc- users executed 1.42 steps in their search plans cessful. Currently, SAVVYSEARCH processes over on average. 20,000 queries each day and, based on elec-

SUMMER 1997 23 Articles

Figure 3. Search Plan for Artiﬁcial Intelligence Conferences Query, Which Allows Users to Continue Search if Current Results Are Inadequate.

tronic mail, has attracted a large well-satisfied and query terms): Visits averaged .36 in the user base. first 5 days and .42 in the last 5 days; No SAVVYSEARCH, as described here, is the result results averaged .142 in the first 5 days and of almost two years of work and a series of .135 in the last. Thus, this learning algorithm studies (Dreilinger and Howe 1997). The cur- leads to better selection of good search rent design of the resource reasoning and engines than the pruning of poor ones. learning algorithm resulted, in part, from the In a follow-up study, we examined how results of these studies (Dreilinger 1996). We much knowledge was needed to significantly studied the effects of the learning by starting improve performance using learning. We from a minimal metaindex (compiled from 2 found that as few as 10 uses of a query term days worth of data) and allowing it to accu- results in a halving of No Results from .18 to mulate experience over a 28-day period. We .09. Although Visits increases more slowly found that our simple learning algorithm with learning, an increase from .34 to .45 is improved performance on both Visits and No obtained with just 100 examples. By the end results overall (including all search engines of the study, the most used term (5000 uses)

24 AI MAGAZINE Articles had Visits of .76 and No results of .08. Given References the level of use of most search engines (mil- Dreilinger, D. 1996. Description and Evaluation of lions of queries a day), metasearch engines a Meta-Search Agent. Master’s thesis, Computer Sci- should have little difficulty in collecting ence Dept., Colorado State University. enough data to significantly improve perfor- Dreilinger, D., and Howe, A. 1997. Experiences mance through learning. with Selecting Search Engines Using Meta-Search. As web use and users becomes more sophis- ACM Transactions on Information Systems. Forthcom- ticated, metasearch will need to be able to be ing. personalized and embedded in other systems. Dreilinger, D., and Howe, A. 1996. An Information- Metasearch currently takes a “one-size-fits- Gathering Agent for Querying Web Search Engines, all” approach in which the knowledge under- Technical Report, TR 96-11, Computer Science lying query processing is shared by all users. Department, Colorado State University. A new experimental version of SAVVYSEARCH Eichmann, D. 1994. Ethical Web Agents. In Elec- (available on the main web page) takes a first tronic Proceedings of the Second World Wide Web step toward personalization by dividing Conference ‘94: MOSAIC and the Web, 17–20 Octo- ber, Chicago, Illinois. searches into eight categories; each category translates to a set of rules for creating a search Gauch, S.; Wang, G.; and Gomez, M. 1996. PROFU- SION: Intelligent Fusion from Multiple, Different plan for this type of search. Ideally, users Search Engines. Journal of Universal Computer Science themselves could create and store their own 2(9). stereotypical search plans or a system could Selberg, E., and Etzioni, O. 1995. Multi-Service infer them. Search and Comparison Using the METACRAWLER. Just as primary search engines are now a Presented at the Fourth International World Wide resource for metasearch engines, so meta- Web Conference, 11 to 14 December, Boston, Mas- search engines should become a means of sachusetts. achieving higher-level goals rather than an Witten, I. H.; Moffat, A.; and Bell, T. C. 1994. Man- end in itself. Intelligent-agent technology aging Gigabytes: Compressing and Indexing Documents promises to alleviate the tedium and frustra- and Images. New York: Van Nostrand Reinhold. tion of mundane tasks and navigate vast information spaces. Metasearch should be an Adele E. Howe is an assistant information-gathering tool for helping professor in the Computer Sci- human users and their intelligent agents find ence Department at Colorado what they need on the web. State University. Her research The web will continue to grow. Thus, areas include planning, agent architectures, information gather- search tools will continue to be critical for ing on the World Wide Web, and managing the information deluge. To keep evaluation and modeling of intel- pace with the expansion, the next generation ligent systems. She received her must include far more sophisticated AI tech- B.S.E. in computer science from the University of niques than the current but retain some of Pennsylvania and her M.S. and Ph.D. in computer the benefits of the current systems: be easy to science from the University of Massachusetts. She use, require little feedback from the users, can be reached at [email protected] or and be mindful of shared resources. www.cs.colostate.edu/~howe.

Acknowledgments Daniel Dreilinger received a B.A. The experiments and equipment were sup- in mathematics from the Univer- ported in part by Advanced Research Projects sity of California at San Diego in Agency–Air Force Office of Scientific Research 1992. In 1996, he received an contract F30602-93-C-0100, National Science M.S. in computer science from Colorado State University, where Foundation (NSF) Research Initiation Award he created SAVVYSEARCH. Currently, IRI-930857, and NSF Career Award IRI- he is a research assistant at the 9624058. We gratefully acknowledge the MIT Media Lab working on a sec- assistance of Lawrence Livermore Laboratory ond M.S. Research interests include information and the Computer Science Department at filtering and retrieval, intelligent agents, and elec- Colorado State University for providing tronic commerce. He can be reached at machines to run SAVVYSEARCH and Wayne Trzy- [email protected] or www.media.mit.edu/ na and his staff for patiently keeping the ~daniel/. machines running. Note 1. We thank the users who translated the interface for us.

SUMMER 1997 25 Diagrammatic Reasoning Cognitive & Computational Perspectives

Edited by Janice Glasgow, N. Hari Narayanan, and B. Chandrasekaran Foreword by Herbert Simon

“Understanding diagrammatic thinking will be of special importance to those who design human-computer interfaces, where the diagrams presented on computer screens must ﬁnd their way to the Mind’s Eye.… In a society that is preoccupied with ‘Information Superhighways,’ a deep understanding of diagrammatic reasoning will be essential to keep the trafﬁc moving.” – Herbert Simon

Diagrammatic reasoning—the understanding of concepts and ideas by the use of diagrams and imagery, as opposed to linguistic or algebraic representations—not only allows us to gain insight into the way we think but is a potential base for constructing representations of diagrammatic information that can be stored and processed by computers.

Diagrammatic Reasoning brings together nearly two dozen recent investigations into the cognitive, the logical, and particularly the computational characteristics of diagrammatic representations and the reasoning that can be done with them. Following a foreword by Herbert Simon (coauthor of one of the most influential papers on reasoning with diagrams, which is included here) and an introduction by the editors, chapters provide an overview of the recent history of the subject, survey and extend the underlying theory of diagrammatic representation, and provide numerous examples of diagrammatic reasoning (human and mechanical) that illustrate both its powers and its limitations. Each of the book’s four sections begins with an introduction by an eminent researcher who pro- vides an interesting personal perspective while he or she places the work in proper context.

ISBN 0-262-57112-9 800 pp., index. $50.00 softcover The AAAI Press • Distributed by The MIT Press Massachusetts Institute of Technology, Cambridge, Massachusetts 02142 To order, call toll free: (800) 356-0343 or (617) 625-8569. MasterCard and VISA accepted.