DOCSLIB.ORG

Intelligent Web Exploration by Pavel Kalinov

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Intelligent Web Exploration by Pavel Kalinov Intelligent Web Exploration Author Kalinov, Pavel Published 2012 Thesis Type Thesis (PhD Doctorate) School School of Information and Communication Technology DOI https://doi.org/10.25904/1912/635 Copyright Statement The author owns the copyright in this thesis, unless stated otherwise. Downloaded from http://hdl.handle.net/10072/365635 Griffith Research Online https://research-repository.griffith.edu.au Intelligent Web Exploration by Pavel Kalinov MSc Information Technology Institute for Integrated and Intelligent Systems (IIIS) School of Information and Communication Technology Science, Environment, Engineering and Technology Griffith University Submitted in fulfilment of the requirements of the degree of Doctor of Philosophy December 2011 Abstract The hyperlinked part of the internet known as \the Web" arose without much planning for a future of millions of publishers and countless pieces of online content. It has no in-built mechanism to find anything, so tools external to it were introduced: initially web directories and then search engines. Search engines are based on machine learning and have been extremely successful. However, they have some inherent limitations and cannot, by design, address some needs: they serve the \information locating" need only and not \information discovery". Search engine users have learned to accept them and in many cases do not realise how their search has been limited by shortcomings of the model. Before the advent of the search engine, web directories were the only information- finding tool on the web. They were manually built and could not compete economically with the efficiency of search engines. This lead to their virtual extinction, with the effect that the \information discovery" need of users is no longer served by any major information provider. Furthermore, none of the dominant information-finding models account for the person of the user in any meaningful way controllable by (or even visible to) the user. This work proposes a method to combine a search engine, a web directory and a personal information management agent into an intelligent Web Exploration Engine in a way which bridges the gaps between these seemingly unrelated tools. Our hybrid, for which we have developed a proof-of-concept prototype [Kalinov et al., 2010b], allows users to both locate specific data and to discover new information. Information discovery is served by a web directory which is built with the assistance of a dynamic hierarchical classifier we developed [Kalinov et al., 2010a]. The category structure achieved by it is also the basis of a large number of nested search engines, allowing information locating both in general (similar to a \standard" search engine) and in a variety of contexts selectable by the user. Personalisation in our model is distributed: user modelling happens where the user is, and is handled by a personal agent. The agent sends relevant information to the exploration engine at search time, allowing truly personalised search which accounts for the user's cognitive and current search context, while at the same time preserving privacy. This work is currently in the initial stages of commercialisation, and preliminary patent research has been done with view of patenting the concept of connecting distributed and independent personalisation agents to any number of search providers. i ii List of Publications 1. \Building a Dynamic Classifier for Large Text Data Collections", Pavel Kalinov, Bela Stantic and Abdul Sattar. In \Proceedings of the 21st Australasian Database Conference - ADC2010", Australian Computer Society, CRPIT Series, ISBN 978-1-920682-85-9 2. \Let's Trust Users - It is Their Search", Pavel Kalinov, Bela Stantic and Abdul Sattar. In \Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology - WI-IAT 2010", IEEE Computer Society, ISBN 978-0- 7695-4191-4 3. \Towards Real Intelligent Web Exploration", Pavel Kalinov, Bela Stantic and Abdul Sattar. 14th Asia-Pacific Web Conference - APWEB 2012. 4. \Proposed Model for Really Intelligent Web Exploration", Pavel Kalinov, Bela Stantic and Abdul Sattar. \Web Intelligence and Agent Systems" journal. (to be submitted) iii Acknowledgements Many thanks for the support, advice and ideas to: The School of ICT at Griffith University, and more specifically my supervisors Prof. Abdul Sattar and Dr. Bela Stantic for their guidance and technical help through this whole project, Dr. Sankalp Khanna (for listening to some of my more advanced ideas with a straight face), Dr. Michael Blumenstein (for pointing me to useful information and people), Dr. John Zakos (for first telling me I don't know anything yet, and later telling me I may have got it right this time; and for a coffee at Macy's) and Prof. Vladimir Estivill-Castro (for some handy pointers). Dr. Charles Gretton for having the patience to help me edit the first draft of my confir- mation paper, and Prof. Gabriella Pasi for her very informative and structured keynote talk at the Web Intelligence 2010 conference in Toronto which helped me greatly with my review of personalisation techniques. Last but not least, my wife Elena Vasileva and my parents, for putting up with me for so long. v Statement of Originality This work has not previously been submitted for a degree or diploma to any university. To the best of my knowledge and belief, the thesis contains no material previously published or written by another person except where due reference is made in the thesis itself. Signed: December 2011 vii Contents Abstract i List of Publications iii Contents ix List of Figures xiii List of Tables xv 1 Introduction 1 1.1 Introduction . .1 1.2 Contribution, Scope and Limitations . .5 1.3 This Document . .6 2 Background and Motivation 7 2.1 Summary . .7 2.2 Brief History . .7 2.3 Structure of the Web . .8 2.3.1 Physical Structure . .8 2.3.2 Logical Structure . .9 2.3.3 Document Interconnection (Hyperlinking): The Web . 12 2.4 Information Finding: Aspects and Approaches . 13 2.4.1 Information Locating . 14 2.4.2 Information Discovery . 25 2.5 Personalisation . 32 ix 2.5.1 Aspects of Personalisation . 32 2.5.2 Personalisation Solutions . 33 2.5.3 Inherent Problems of Personalised Solutions . 43 2.5.4 Avoidable Problems of Personalised Solutions . 44 2.5.5 Personal Web Assistants . 44 2.6 Issues Arising From Current Solutions . 45 2.6.1 Consequences of Some Approaches . 45 2.6.2 Research Issues . 51 3 Exploration Engine 53 3.1 Summary . 53 3.2 Research Question . 53 3.2.1 Research Sub-questions . 54 3.2.2 Assumptions and Limitations . 55 3.2.3 Practical Tasks . 56 3.3 General Outline of the Solution . 57 3.3.1 Hybrid Web Directory . 57 3.3.2 User-Controlled Ontology . 59 3.3.3 Personal to Global Ontology . 59 3.3.4 Prototype and Practical Considerations . 59 3.4 Advantages of the Proposed Solution . 60 3.4.1 Usable User Profile . 60 3.4.2 Improved General Usability . 62 3.4.3 Expressive Queries . 63 3.4.4 Exhaustive Exploration of a Topic . 63 3.4.5 User-Specified Search Context . 63 3.4.6 Busting the Filter Bubble . 64 3.4.7 Information Scent . 64 3.4.8 Recommendation Engine and Information Discovery . 64 3.5 Disadvantages of the Proposed solution . 65 3.5.1 Expensive Backend, Complexity . 65 3.5.2 Complicated Frontend, User Investment . 66 3.6 Specifics of the Exploration Engine . 66 3.6.1 Browsing the Directory . 66 3.6.2 Exploring the Directory . 68 3.6.3 Searching in the Directory . 69 3.6.4 Query and Query Expansion . 70 3.6.5 Enhancements . 73 4 Related Work 75 4.1 Summary . 75 4.2 Building the User Model . 75 4.2.1 Knowledge Representation and Ontology Matching . 78 4.3 Building the Web Directory . 79 4.3.1 Data Pre-processing . 79 4.3.2 Data Representation . 80 4.3.3 Indexing . 81 4.3.4 Dimensionality Reduction . 82 4.3.5 Unsupervised Clustering, SOM . 86 4.3.6 Classification . 90 5 Implementation and Practical Issues 93 5.1 Summary . 93 5.2 General Approach . 94 5.3 General Setup . 95 5.4 Selected Approaches to Tasks . 95 5.4.1 Building the User Model . 96 5.4.2 Knowledge Representation and Ontology Matching . 97 5.4.3 Data Acquisition and Processing . 100 5.4.4 Self-Organising Maps and Broken Dreams . 107 5.4.5 Classification . 113 5.5 Dataset and Experimental Study . 117 5.5.1 Test Setup . 117 5.5.2 Available Data . 118 5.5.3 Algorithm Testing and Experimental Results . 122 5.6 The Floating Query . 137 5.7 User Testing, or Lack Thereof . 138 6 Conclusion 141 6.1 State of the Art . 141 6.2 Proposed Solution . 143 6.3 Summary of Contributions . 144 6.4 Future Work . 147 Bibliography xvii Index of Terms xxix List of Figures 2.1 Context based information supply [Pasi, 2010]. Note the implied grouping of user profiling, context knowledge and web data on the remote side. 41 3.1 Context based information supply: our proposal. User profiling is on the user side, information-related activities are on the server side. A (partial) user profile is sent on to the server after being filtered through the current activity context. 57 3.2 User browsing the directory. Exploration has not been personalised yet: in the upper box (highlighted by bold rectangle) no relevance feedback has been added by the user. The keywords from the list in the bottom right box (highlighted) assist the user to formulate a search query and search either within the directory or at a number of external search providers. These keywords automatically emerge from the classifier data and are not supplied by the editor.
Load more

Recommended publications
  • Hacking the Master Switch? the Role of Infrastructure in Google's
    Hacking the Master Switch? The Role of Infrastructure in Google’s Network Neutrality Strategy in the 2000s by John Harris Stevenson A thesis submitteD in conformity with the requirements for the Degree of Doctor of Philosophy Faculty of Information University of Toronto © Copyright by John Harris Stevenson 2017 Hacking the Master Switch? The Role of Infrastructure in Google’s Network Neutrality Strategy in the 2000s John Harris Stevenson Doctor of Philosophy Faculty of Information University of Toronto 2017 Abstract During most of the decade of the 2000s, global Internet company Google Inc. was one of the most prominent public champions of the notion of network neutrality, the network design principle conceived by Tim Wu that all Internet traffic should be treated equally by network operators. However, in 2010, following a series of joint policy statements on network neutrality with telecommunications giant Verizon, Google fell nearly silent on the issue, despite Wu arguing that a neutral Internet was vital to Google’s survival. During this period, Google engaged in a massive expansion of its services and technical infrastructure. My research examines the influence of Google’s systems and service offerings on the company’s approach to network neutrality policy making. Drawing on documentary evidence and network analysis data, I identify Google’s global proprietary networks and server locations worldwide, including over 1500 Google edge caching servers located at Internet service providers. ii I argue that the affordances provided by its systems allowed Google to mitigate potential retail and transit ISP gatekeeping. Drawing on the work of Latour and Callon in Actor– network theory, I posit the existence of at least one actor-network formed among Google and ISPs, centred on an interest in the utility of Google’s edge caching servers and the success of the Android operating system.
    [Show full text]
  • Google Pagerank and Markov Chains
    COMP4121 Lecture Notes The PageRank, Markov chains and the Perron Frobenius theory LiC: Aleks Ignjatovic [email protected] THE UNIVERSITY OF NEW SOUTH WALES School of Computer Science and Engineering The University of New South Wales Sydney 2052, Australia Topic One: the PageRank Please read Chapter 3 of the textbook Networked Life, entitled \How does Google rank webpages?"; other references on the material covered are listed at the end of these lecture notes. 1 Problem: ordering webpages according to their importance Setup: Consider all the webpages on the entire WWW (\World Wide Web") as a directed graph whose nodes are the web pages fPi : Pi 2 WWW g, with a directed edge Pi ! Pj just in case page Pi points to page Pj, i.e. page Pi has a link to page Pj. Problem: Rank all the webpages of the WWW according to their \importance".1 Intuitively, one might feel that if many pages point to a page P0, then P0 should have a high rank, because if one includes on their webpage a link to page P0, then this can be seen as their recommendation of P0. However, this is not a good criterion for several reasons. For example, it can be easily manipulated to increase the rating of any webpage P0, simply by creating a lot of silly web pages which just point to P0; also, if a webpage \generously" points to a very large number of webpages, such \easy to get" recommendation is of dubious value. Thus, we need to refine our strategy how to rank webpages according to their \importance", by doing something which cannot be easily manipulated \locally" (i.e., by any group of people who might collude, even if such a group is sizeable).
    [Show full text]
  • 1. the Content Score Is the Resulting Score Provided by the Indexing Engine Avail
    INSTITUTO TECNOLÓGICO Y DE ESTUDIOS SUPERIORES DE MONTERREY CAMPUS MONTERREY PROGRAMA DE GRADUADOS EN ELECTRÓNICA, COMPUTACIÓN, INFORMACIÓN Y COMUNICACIONES TECNOLÓGICO DE MONTERREY CONCEPT BASED RANKING IN PERSONAL REPOSITORIES TESIS PRESENTADA COMO REQUISITO PARCIAL PARA OBTENER EL GRADO ACADÉMICO DE: MAESTRÍA EN CIENCIAS EN TECNOLOGÍA INFORMÁTICA POR ALBERTO JOAN SAAVEDRA CHOQUE MONTERREY, N. L., DICIEMBRE 2007 Concept Based Ranking in Personal Repositories by Alberto Joan Saavedra Choque Thesis Presented to the Gradúate Program in Electronics, Computing, Information and Communications School This thesis is a partial requirement to obtain the academic degree of Master of Science in Information Technology Instituto Tecnológico y de Estudios Superiores de Monterrey Campus Monterrey December 2007 A Gary, mi hermano Que me motiva a superarme cada día y a mirar las circunstancias desde diversos puntos de vista. A Yolanda, mi mamá Que me apoya incondicionalmente y me acompaño presencialmente durante un período de mi recorrido en este país que me acogió. A Alberto, mi papá Que me orienta cuando lo necesito y es mi punto de referencia para superar los obstáculos que se presentan en el camino. A Pilar, mi mejor amiga Que conoce mejor que nadie el sacrificio realizado durante esta aventura y me mantuvo enfocado en cumplir exitosamente la trayectoria iniciada. A mi abuelito, tíos y primos Que me hicieron sentir a la distancia, el respaldo y el apoyo de toda la familia. Reconocimientos Al Dr. Juan Carlos Lavariega, mi asesor Por su confianza y apoyo en la realización de esta investigación. Al Dr. Raúl Pérez, sinodal y coordinador de la maestría Por su disponibilidad constante y orientación para responder mis consultas Al Dr.
    [Show full text]
  • Google: the World's First Information Utility?
    BISE – STATE OF THE ART Google: The World’s First Information Utility? The idea of information utilities came to naught in the 70s and 80s, but the world wide web, the Internet, mobile access, and a plethora of access devices have now made information utilities within reach. Google’s business model, IT infrastructure, and innovation strategies have created the capabilities needed for Google to become the world’s first global information utility. But, the difficulty of monetizing utility services and concerns about privacy, which could bring government regulation, might stymie Google’s growth plans. DOI 10.1007/s12599-008-0011-6 a national scale (Sackmann and Nie 1970; the number of users to the point where it The Authors Sackman and Boehm 1972). Not to be left is the dominant search player in both the out, the idea was also promoted by the U.S. and the world. This IT-based strategy Rex Chen Computer Usage Development Institute has allowed Google to reach more than Prof. Kenneth L. Kraemer, PhD in Japan, the British Post Office in the UK, $16 billion in annual revenue, a stock price Prakul Sharma and Bell Canada and the Telecommunica- over $600 per share and a market capital- University of California, Irvine tions Board in Canada (Press 1974). ization of nearly $200 billion. Center for Research on Information The idea was so revolutionary at the Over the next decade, Google plans to Technology and Organization (CRITO) time that at least one critic called for a extend its existing services to the wire- 5251 California Ave., Ste.
    [Show full text]
  • The Commercial Search Engine Industry and Alternatives to The
    and Yahoo!), works successfully to hide an increasingly profitable information and advertising industry. Search engine companies, of which there are really only three, have morphed into advertising conglomerates and now serve advertisers, not users, in a mutual, rather delightful, relationship. The advertiser pays the search engine to be affiliated with certain key words; the search company The Commercial Search Engine Industry and provides the sponsored links, which users click on; and traffic is driven to the sites of advertisers. Indeed, users are now universally described as consumers in the Alternatives to the Oligopoly marketplace rhetoric of search engine enterprise; they are the pawns who the business world seeks to manipulate into clicking those links that will ultimately Bettina Fabos - Interactive Media Studies and Journalism, Miami lead to the most profits. University of Ohio In this arrangement, helping users find relevant information is a priority only in This essay details the search engine industry’s transformation into an that, like other commercial media systems (think radio or television), there has to advertising oligopoly. It discusses how librarians, educators, archivists, be some decent content to create a perception that Internet users matter. In fact, activists, and citizens, many of whom are the guardians of indispensable users only matter to the extent that they participate in the commercial system by noncommercial websites and portals, can band together against a sea of knowingly—or unknowingly—clicking on sponsored links.1 Keyword advertising advertising interests and powerful and increasingly overwhelming online generated an unprecedented $3.9 billion in 2004. By 2005 Google’s advertising marketing strategies.
    [Show full text]
  • Link Analysis Google
    Introduction to the Use of Link Analysis by Google Amy Langville Carl Meyer Department of Mathematics North Carolina State University Raleigh, NC SIAM Annual Meeting 7/15/04 Outline • Introduction to Information Retrieval (IR) • Link Analysis • PageRank Algorithm Short History of IR IR = search within doc. coll. for particular info. need (query) B. C. cave paintings 7-8th cent. A.D. Beowulf 12th cent. A.D. invention of paper, monks in scriptoriums 1450 Gutenberg’s printing press 1700s Franklin’s public libraries 1872 Dewey’s decimal system 1900s Card catalog 1940s-1950s Computer 1960s Salton’s SMART system 1989 Berner-Lee’s WWW the pre-1998 Web Yahoo • hierarchies of sites • organized by humans Best Search Techniques • word of mouth • expert advice Overall Feeling of Users • Jorge Luis Borges’ 1941 short story, The Library of Babel When it was proclaimed that the Library contained all books, the first impression was one of extravagant happiness. All men felt themselves to be the masters of an intact and secret treasure. There was no personal or world problem whose eloquent solution did not exist in some hexagon. ... As was natural, this inordinate hope was followed by an excessive depression. The certitude that some shelf in some hexagon held precious books and that these precious books were inaccessible, seemed almost intolerable. 1998 ... enter Link Analysis Change in User Attitudes about Web Search Today • “It’s not my homepage, but it might as well be. I use it to ego-surf. I use it to read the news. Anytime I want to find out anything, I use it.” - Matt Groening, creator and executive producer, The Simpsons • “I can’t imagine life without Google News.
    [Show full text]
  • Ambitious Form a General Theory of Visual Culture
    68 Art Ambitious Form A General Theory Giambologna, Ammanati, and Danti in Florence of Visual Culture Michael W. Cole Whitney Davis Ambitious Form describes the transformation of Italian What is cultural about vision—or visual about culture? sculpture during the neglected half-century between In this ambitious book, Whitney Davis provides new the death of Michelangelo and the rise of Bernini. The answers to these difficult and important questions by book follows the Florentine careers of three major presenting an original framework for understanding sculptors—Giambologna, Bartolomeo Ammanati, and visual culture. Grounded in the theoretical traditions Vincenzo Danti—as they negotiated the politics of the of art history, A General Theory of Visual Culture argues Medici court and eyed one another’s work, setting new that, in a fully consolidated visual culture, artifacts and aims for their art in the process. Only through a com- pictures have been made to be seen in a certain way; parative look at Giambologna and his contemporaries, what Davis calls “visuality” is the visual perspective it argues, can we understand them individually—or from which certain culturally constituted aspects of understand the period in which they worked. artifacts and pictures are visible to informed viewers. Michael Cole shows how the concerns of central In this book, Davis provides a systematic analysis of Italian artists changed during the last decades of the visuality and describes how it comes into being as a Cinquecento. Whereas their predecessors had focused historical form of vision. on specific objects and on the particularities of materi- Expansive in scope, A General Theory of Visual als, late sixteenth-century sculptors turned their atten- Culture draws on art history, aesthetics, the psychology tion to models and design.
    [Show full text]
  • Eric Miller Interview
    Volume 1 Issue 2 SIGSEMIS July 2004 Bulletin Theme: “SW Challenges http://www.sigsemis.org for KM” The Official Bimonthly Newsletter of AIS Special Interest Group on Semantic Web and Information Systems SIGSEMIS © 2004 Inside Bulletin EDITORIAL… Editorial................................1 For the second time AIS SIGSEMIS Bulletin is on air. We would like to thank SIGSEMIS Activities ...........2 Eric Miller Interview...........3 you for your warm receipt of the first issue which gives us a motive to IJSWIS Announcement ....11 continue our volunteer hard work towards more activities and services for our Dialogue Column..............18 research community. Our portal site at www.sigsemis.org had 1650 unique Special Issue Theme..........23 visitors (and same number of AIS SIGSEMIS Bulletin 1(1) downloads) from Research in Progress over 50 countries. Inside this issue you can find many interesting things. A Column...............................68 call for papers for our SIG Official Peer Reviewed Journal which will be Special Section: DERI........73 published by IDEA Group (inaugural issue 1/2005). I invite you to consider Chris Bussler Interview....74 this publication outlet as an interesting and high quality journal in which we Regular Columns ..............93 will put all of our efforts in order to meet your high standards for quality and Semantic Search to communicate high impact research. Technologies ......................93 SW Technologies ...............99 Methodologies for SW....103 The special theme that is discussed in this issue is Semantic Web Challenges SW Basics..........................113 for Knowledge Management: towards the Knowledge Web. We would like to SW Calendar ....................115 thank all the contributors of the short articles and especially the directors and SW Research Centers......119 Professors Christoph Bussler and Dieter Fensel of Digital Enterprise Research Projects Corner ................123 Institute (DERI) who contributed to the special section of DERI presentation.
    [Show full text]
  • Chenlamkraemer2007crito.Pdf
    INTRODUCTION Arguably the most popular search engine available today, Google is widely known for its unparalleled search engine technology, embodied in the web page ranking algorithm, PageRanki and running on an efficient distributed computer system. In fact, the verb “to Google” has ingrained itself in the vernacular as a synonym of “[performing] a web search.”1 The key to Google’s success has been its strategic use of both software and hardware information technologies. The IT infrastructure behind the search engine includes huge storage databases and numerous server farms to produce significant computational processing power. These critical IT components are distributed across multiple independent computers that provide parallel computing resources. This architecture has allowed Google’s business to reach a market capital over $100 billion and become one of the most respected and admirable companies in the world. MARKET ENVIRONMENTS Search Engine Internet search engines were first developed in the early 1990s to facilitate the sharing of information among researchers. The original effort to develop a tool for information search occurred simultaneously across multiple universities is shown in Table 1. Although functionalities of these systems were very limited, they provided the foundation for future web- based search engines. TABLE 1. Early Search Engines Search Engine Name University Year Inventor Archie McGill University 1990 Alan Emtage Veronica University of Nevada 1993 Many students WWW Wanderer Massachusetts Institute of Technology 1993 Matthew Gray Source: Battelle, 2005. Search Industry During the 1990s, the Internet experienced exponential growth with thousands of new web pages being created daily. Online document search became the chief method of navigating the ever- expanding World Wide Web, as Internet users sought useful information among the largely disorganized pages.
    [Show full text]
  • Ranking Pages and the Topology of The
    Ranking pages and the topology of the web Argimiro Arratia∗ Departament de Llenguatges i Sistemes Inform`atics, Universitat Polit`ecnica de Catalunya, Barcelona, Spain [email protected] Carlos Mariju´an † Departamento de Matem´atica Aplicada Universidad de Valladolid, Valladolid, Spain, [email protected] Abstract This paper presents our studies on the rearrangement of links in the structure of websites for the purpose of improving the valuation of a page or group of pages as established by a ranking function as Google’s PageRank. We build our topo- logical taxonomy starting from unidirectional and bidirectional rooted trees, and up to more complex hierarchical structures as cyclical rooted trees (obtained by closing cycles on bidirectional trees) and PR–digraph rooted trees (digraphs whose condensation digraph is a rooted tree that behave like cyclical rooted trees). We give different modifications on the structure of these trees and its effect on the valuation given by the PageRank function. We derive closed formulas for the PageRank of the root of various types of trees, and establish a hierarchy of these topologies in terms of PageRank. Keywords: PageRank, world wide web topology, link structure. AMS Math. Subject Classification: 05C05, 05C99, 05C40, 68R10, 94C15. 1 Introduction arXiv:1105.1595v2 [cs.DM] 23 Apr 2012 Google is still today’s most popular search engine for the World Wide Web, and the key to its success has been its PageRank algorithm [4], which ranks documents based primarily on the link structure of the web. Simply put, PageRank considers a link from a page H to another page J as a weighted vote from H in favour of the importance of J, where the weight of the vote of H is itself determined by the number of links (or voters) to H.
    [Show full text]
  • Link Analysis Web Search Engines
    Introduction to the Use of Link Analysis by Web Search Engines Amy Langville Carl Meyer Department of Mathematics North Carolina State University Raleigh, NC Mt. St. Mary’s College 4/19/04 Outline • Introduction to Information Retrieval (IR) • Link Analysis • HITS Algorithm • PageRank Algorithm Short History of IR IR = search within doc. coll. for particular info. need (query) B. C. cave paintings 7-8th cent. A.D. Beowulf 12th cent. A.D. invention of paper, monks in scriptoriums 1450 Gutenberg’s printing press 1700s Franklin’s public libraries 1872 Dewey’s decimal system Card catalog 1940s-1950s Computer 1960s Salton’s SMART system 1989 Berner-Lee’s WWW the pre-1998 Web Yahoo • hierarchies of sites • organized by humans Best Search Techniques • word of mouth • expert advice Overall Feeling of Users • Jorge Luis Borges’ 1941 short story, The Library of Babel When it was proclaimed that the Library contained all books, the first impression was one of extravagant happiness. All men felt themselves to be the masters of an intact and secret treasure. There was no personal or world problem whose eloquent solution did not exist in some hexagon. ... As was natural, this inordinate hope was followed by an excessive depression. The certitude that some shelf in some hexagon held precious books and that these precious books were inaccessible, seemed almost intolerable. 1998 ... enter Link Analysis Change in User Attitudes about Web Search Today • “It’s not my homepage, but it might as well be. I use it to ego-surf. I use it to read the news. Anytime I want to find out anything, I use it.” - Matt Groening, creator and executive producer, The Simpsons • “I can’t imagine life without Google News.
    [Show full text]
  • Discoveries That Will Change Our World
    discoveries that will change our world OFFICE OF TECHNOLOGY LICENSING ANNUAL REPORT 2002 - 2003 OFFICE OF TECHNOLOGY LICENSING HAT WILL LIFE BE LIKE WHEN OUR COVER BABY TURNS 21? WILL LIFE BE WEASIER, WITH UNIMAGINED CONVENIENCES, ENVIRONMENTALLY-FRIENDLY PRODUCTS, ENOUGH FOOD AND CLEAN WATER FOR ALL? ALL ABOUT US OR WILL IT BE A TIME OF GRAVE DECISIONS ABOUT HOW AND FOR WHOM AND FOR WHAT AVERAGE AGE 38 PURPOSE OUR LIMITED RESOURCES WILL BE USED, WHEN TECHNOLOGICAL ADVANCES HAVE AVERAGE YEARS AT OTL 7 COMPLICATED OUR LIVES AND WHEN THE TECHNOLOGICAL GAP BETWEEN “HAVES” AND CUMULATIVE EXPERIENCE 246.5 YEARS “HAVE-NOTS” IS AN UNCROSSABLE CHASM? NUMBER OF EMPLOYEES 24 OUR BABY’SFUTURE IS BEING CREATED NOW, BY RESEARCH AND INNOVATIONS AT STANFORD AND ELSEWHERE. MOST OF STANFORD’S INVENTIONS ARE EMBRYONIC, EARLY-STAGE DEVEL- OPMENTS FOR WHICH COMMERCIAL PRODUCTS ARE LIKELY TO BE 10-15 YEARS AWAY. YET THESE INNOVATIONS HAVE THE POTENTIAL TO CHANGE OUR LIVES IN WAYS WE CAN ONLY IMAGINE NOW. TAKE A PEAK INTO THE FUTURE…. DISCOVERIES THAT WILL CHANGE OUR WORLD | 1 IMAGINE A WORLD OF VISION FOR THE BLIND, WHERE IMAGINE IF DOCTORS COULD SEE CELLS AT THE MOLECULAR DISEASES SUCH AS MACULAR DEGENERATION CAN BE CURED. LEVEL, ENABLING THEM TO DIAGNOSE AND TREAT DISEASES Age-related macular degeneration is the leading cause of blind- SUCH AS CANCER AND HEART DISEASE. Molecular Imaging, or ness in people over the age of 65. Currently, there is no effec- MI, is a young research area, just 10 years old, that combines tive treatment for most patients with AMD, a disease that often contrast agents and/or pharmacologically active diagnostic results in permanent damage to photoreceptors.
    [Show full text]