School of Computing, Engineering and Information Sciences CG0174 Individual Computing Project, MSc Intelligent Search Optimization using Artificial Fuzzy Logics Student: Jai Manral Programme: Computer Science, MSc Supervisor: Prof. Alamgair Hossain Second Marker: Prof. Michael Brockway Session: 2011-2012 1 Declaration The copyright of this dissertation rests with the author. No part of it should be published without his prior written consent and any information derived from this report should be acknowledged. I certify that the work contained in this report is the sole work of the author except where indicated. All material that has been taken from other sources has been clearly acknowledged. Quotations from other sources have been clearly marked, using quotation marks or a block quote. Signed……Jai Manral……………….. Date………17/05/2012………………... 2 Acknowledgement The author expresses heartfelt gratitude to the following highly inspiring individuals, whose time, advice and guidance were extremely valuable and greatly appreciated, during the development of the project. • Professor Alamgir Hossain whose encouragement, supervision and support from the preliminary to the concluding level enabled me to develop an understanding of the subject. • Professor Michael Brockway who showed faith in the research and provides valuable insight in it. • Dr. Becky Strachan whose teaching and guidance helped me in writing this research work. • Shelagh Keogh for her kind words of encouragement that made an otherwise hard journey seems infinitely simple. I do like to thank all the interviewees and students who were patient while answering questions and reviewing product. Finally, to the much needed support and understanding from family and friends that made this day possible. 3 Abstract Information on the web is prodigious; searching relevant information is difficult making web users to rely on search engines for finding relevant information on the web. Search engines index and categorize web pages according to their contents using crawlers and rank them accordingly. For given user query they retrieve millions of webpages and display them to users according to web-page rank. Every search engine has their own algorithms based on certain parameters for ranking web-pages. Search Engine Optimization (SEO) is that technique by which webmasters try to improve ranking of their websites by optimizing it according to search engines ranking parameters. It is the aim of this research to identify the most popular SEO techniques used by search engines for ranking web-pages and to establish their importance for indexing and categorizing web data. The research tries to establish that using more SEO parameters in ranking algorithms helps in retrieving better search results thus increasing user satisfaction. In the accomplished research, a web based Meta search engine is proposed to aggregates search results from different search engines and rank web-pages based on new page ranking algorithm which will assign heuristic page rank to web-pages based on SEO parameters such as title tag, Meta description, sitemap etc. The research also provides insight into techniques which webmasters can use for better ranking their websites in Google and Bing. Initial results has shown that using certain SEO parameters in present ranking algorithm helps in retrieving more useful results for user queries. These results generated from Meta search engine outperformed existing search engines in terms of better retrieved search results and high precision. 4 Table of content List of Figures .............................................................................................................. viii List of Tables .................................................................................................................. ix List of Abbreviations .......................................................................................................x 1 Introduction .............................................................................................................2 1.1 Motivation .......................................................................................................... 2 1.2 Research Aims and Objective ............................................................................ 3 1.3 Work Done and Results ..................................................................................... 3 1.4 Structure of Thesis ............................................................................................. 4 2 Literature Review ................................................................................................... 5 2.1 Introduction ........................................................................................................ 5 2.2 Search Engines ................................................................................................... 5 2.2.1 Overview ......................................................................................................... 5 2.2.2 Evolution .................................................................................................... 6 2.2.3 Classification .............................................................................................. 7 2.2.4 Limitations .................................................................................................. 7 2.3 Meta search engine ............................................................................................ 8 2.3.1 Overview and Importance .......................................................................... 8 2.3.2 Previous work ............................................................................................. 9 2.4 Search Engine Optimization ............................................................................ 10 2.4.1 Overview .................................................................................................. 10 2.4.2 Previous Work .......................................................................................... 10 2.4.3 Optimizing Factors ................................................................................... 12 2.5 Web Mining ..................................................................................................... 17 2.5.1 Overview .................................................................................................. 17 2.5.2 Related Work ............................................................................................ 18 2.6 Summary .......................................................................................................... 18 3 Practical Work Undertaken ................................................................................ 20 3.1 Overview: ............................................................................................................. 20 3.2 Architecture of Meta search engine ...................................................................... 20 3.2.1 General Architecture...................................................................................... 20 3.2.2 Proposed iral Architecture ............................................................................. 22 3.3 Implementation and Design of Meta search engine-iral .................................. 24 3.3.1 Functional Requirements .......................................................................... 25 3.3.2 Detail level design .................................................................................... 26 3.3.3 Non-Functional Requirements .................................................................. 28 3.3.4 Data Source............................................................................................... 29 3.3.5 Alternate Design Approach ...................................................................... 31 3.3.6 Ranking Strategy ...................................................................................... 33 3.3.7 Design Constrains ..................................................................................... 35 3.3.8 Deployment Details .................................................................................. 35 3.4 Testing ............................................................................................................. 36 3.4.1 Overview .................................................................................................. 36 3.4.2 Test Cases ................................................................................................. 36 3.4.3 Summary ................................................................................................... 41 4 Results, Analysis and Evaluation ........................................................................ 42 4.1 SEO Techniques .............................................................................................. 42 4.1.1 Results ...................................................................................................... 42 4.1.2 SEO SERP Evaluation .............................................................................. 47 4.1.3 Summary ................................................................................................... 48 4.2 iral Meta Search Engine ................................................................................... 49 4.2.1 Overview .................................................................................................. 49 4.2.2 Precession and Recall ..............................................................................