DOCSLIB.ORG

Enterprise Search Technology Using Solr and Cloud Padmavathy Ravikumar Governors State University

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Governors State University OPUS Open Portal to University Scholarship All Capstone Projects Student Capstone Projects Spring 2015 Enterprise Search Technology Using Solr and Cloud Padmavathy Ravikumar Governors State University Follow this and additional works at: http://opus.govst.edu/capstones Part of the Databases and Information Systems Commons Recommended Citation Ravikumar, Padmavathy, "Enterprise Search Technology Using Solr and Cloud" (2015). All Capstone Projects. 91. http://opus.govst.edu/capstones/91 For more information about the academic degree, extended learning, and certificate programs of Governors State University, go to http://www.govst.edu/Academics/Degree_Programs_and_Certifications/ Visit the Governors State Computer Science Department This Project Summary is brought to you for free and open access by the Student Capstone Projects at OPUS Open Portal to University Scholarship. It has been accepted for inclusion in All Capstone Projects by an authorized administrator of OPUS Open Portal to University Scholarship. For more information, please contact [email protected]. ENTERPRISE SEARCH TECHNOLOGY USING SOLR AND CLOUD By Padmavathy Ravikumar Masters Project Submitted in partial fulfillment of the requirements For the Degree of Master of Science, With a Major in Computer Science Governors State University University Park, IL 60484 Fall 2014 ENTERPRISE SEARCH TECHNOLOGY USING SOLR AND CLOUD 2 Abstract Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, near real-time indexing, dynamic clustering, database in9tegration, rich document (e.g., Word, PDF) handling, and geospatial search. Solr is highly reliable, scalable and fault tolerant, providing distributed indexing, replication and load-balanced querying, automated failover and recovery, centralized configuration and more. Solr powers the search and navigation features of many of the world's largest internet sites. Databases and Solr have complementary strengths and weaknesses. SQL supports very simple wildcard-based text search with some simple normalization like matching upper case to lower case. The problem is that these are full table scans. In Solr all searchable words are stored in an "inverse index", which searches orders of magnitude faster. Solr is a standalone/cloud enterprise search server with a REST-like API. You put documents in it (called "indexing") via XML, JSON, CSV or binary over HTTP. You query it via HTTP GET and receive XML, JSON, CSV or binary results. The project will be implemented using Amazon/Azure cloud, Apache Solr, Windows/Linux, MS-SQL server and open source tools. The following are the software requirements of my project: Content Description OS Windows/ Linux/ Unix Database MS-SQL server 2012 Technologies Apache Solr Cloud Amazon, Azure Browser IE , Chrome, Firefox ENTERPRISE SEARCH TECHNOLOGY USING SOLR AND CLOUD 3 Contents LIST OF FIGURES ........................................................................................................................ 5 LIST OF TABLES .......................................................................................................................... 7 INTRODUCTION .......................................................................................................................... 8 OVERVIEW ............................................................................................................................... 8 Data and Information Technology .......................................................................................... 9 Searching and Search Engines ................................................................................................ 9 Database Approach to Searching .......................................................................................... 11 Apache Solr and Lucene Approach to Searching ................................................................. 12 FOCUS ...................................................................................................................................... 14 TOOLS ...................................................................................................................................... 16 CONCEPTS AND TERMINOLOGIES ................................................................................... 18 INDEXING ............................................................................................................................... 21 TOOLS OF THE TRADE ............................................................................................................ 25 LUCENE ................................................................................................................................... 25 Lucene Architecture .............................................................................................................. 25 Lucene Functions: ................................................................................................................. 27 SOLR ........................................................................................................................................ 27 Solr Features ......................................................................................................................... 28 ENTERPRISE SEARCH TECHNOLOGY USING SOLR AND CLOUD 4 Solr Concepts and Terminologies ......................................................................................... 30 Solr Cloud Architecture ........................................................................................................ 32 Solr/Lucene Architecture ...................................................................................................... 33 Solr History ........................................................................................................................... 34 Getting started with Solr ....................................................................................................... 35 CHAPTER 3: GETTING STARTED ........................................................................................... 38 VIEWING INSTANCES IN SERVER .................................................................................... 38 ACCESSING INSTANCES ..................................................................................................... 42 INDEXING DATA ................................................................................................................... 46 SEARCHING DATA................................................................................................................ 48 EXAMPLE 1 ................................................................................................................................. 51 AIRPORT APPLICATION ...................................................................................................... 51 REFERENCES: ............................................................................................................................ 60 ENTERPRISE SEARCH TECHNOLOGY USING SOLR AND CLOUD 5 LIST OF FIGURES Fig 1. Yahoo Website.............................................................................................12 Fig 2. Indexing and Searching............................................................................... 16 Fig 3. Indexing........................................................................................................22 Fig 4. Indexing Process..........................................................................................23 Fig 5. Querying.......................................................................................................23 Fig 6. Inverted Indexing..........................................................................................24 Fig 8. Lucene Architecture..................................…………………………………28 Fig 9. Solr................................................................................................................31 Fig 10. Solr Cloud Architecture................................................................................35 Fig 11. Lucene/Solr Architecture..............................................................................36 Fig 12. Solr History...................................................................................................37 Fig 13. Solr Master Slave Architecture.....................................................................37 Fig 14. Solr in Web Architecture..............................................................................40 Fig 15. Enterprise Search Architecture.....................................................................40 Fig 16. Amazon AWS EC2.......................................................................................42 Fig 17. View Instance...............................................................................................44 Fig 18. Instance Properties........................................................................................46 Fig 19. RedHat_Solr1............................................................................................... 46 Fig 20. Putty Key Generator..................................................................................... 47 Fig 21. Putty Configuratio.........................................................................................48 Fig 22. Terminal Window 1.......................................................................................49
Recommended publications
  • Oracle Data Sheet- Secure Enterprise Search

    Oracle Data Sheet- Secure Enterprise Search

    ORACLE SECURE ENTERPRISE SEARCH 11g DATA SHEET ORACLE SECURE ENTERPRISE SEARCH VERSION 11G R2 KEY FEATURES Oracle Secure Enterprise Search 11g (SES), a standalone product RELEASE 11.2.2.2 HIGHLIGHTS from Oracle, enables a high quality, secure search across all Facet Navigation enterprise information assets. Key SES features include: Push-based Content Indexing Multi-tier Install Search Result Tagging The ability to search and locate public, private and shared Unified Microsoft Sharepoint content across intranet web content, databases, files on local Connector, certified for MOSS 2010 version disk or file-servers, IMAP email, document repositories, New search result „hard sort‟ applications, and portals option Auto Suggestions “As You Type” Excellent search quality, with the most relevant items for a query Sitemap.org Support spanning diverse sources being shown first FACET NAVIGATION Full GUI support for facet Sub-second query performance creation, manipulation, and browsing. No programming required. Facet API support is Highly secure crawling, indexing, and searching also provided Facet navigation is secure. Integration with Desktop Search tools Facet values and counts are computed using only those documents that the search Ease of administration and maintenance – a „no-DBA‟ approach user is authorized to see Hierarchical- and range facets to Search. Date, number and string types Update facets without re- Information Uplift for the Intranet crawling As a result of search engines on the Internet, the power of effective search PUSH-BASED CONTENT technologies has become clear to everyone. Using the World Wide Web, consumers INDEXING have become their own information retrieval experts. But search within enterprises Allows customers to push differs radically from public Internet search.
  • Lucene in Action Second Edition

    Lucene in Action Second Edition

    Covers Apache Lucene 3.0 IN ACTION SECOND EDITION Michael McCandless Erik Hatcher , Otis Gospodnetic FOREWORD BY DOUG CUTTING MANNING www.it-ebooks.info Praise for the First Edition This is definitely the book to have if you’re planning on using Lucene in your application, or are interested in what Lucene can do for you. —JavaLobby Search powers the information age. This book is a gateway to this invaluable resource...It suc- ceeds admirably in elucidating the application programming interface (API), with many code examples and cogent explanations, opening the door to a fine tool. —Computing Reviews A must-read for anyone who wants to learn about Lucene or is even considering embedding search into their applications or just wants to learn about information retrieval in general. Highly recommended! —TheServerSide.com Well thought-out...thoroughly edited...stands out clearly from the crowd....I enjoyed reading this book. If you have any text-searching needs, this book will be more than sufficient equipment to guide you to successful completion. Even, if you are just looking to download a pre-written search engine, then this book will provide a good background to the nature of information retrieval in general and text indexing and searching specifically. —Slashdot.org The book is more like a crystal ball than ink on pape--I run into solutions to my most pressing problems as I read through it. —Arman Anwar, Arman@Web Provides a detailed blueprint for using and customizing Lucene...a thorough introduction to the inner workings of what’s arguably the most popular open source search engine...loaded with code examples and emphasizes a hands-on approach to learning.
  • Apache Lucene Searching the Web and Everything Else

    Apache Lucene Searching the Web and Everything Else

    Apache Lucene Searching the Web and Everything Else Daniel Naber Mindquarry GmbH ID 380 2 AGENDA > What's a search engine > Lucene Java – Features – Code example > Solr – Features – Integration > Nutch – Features – Usage example > Conclusion and alternative solutions 3 About the Speaker > Studied computational linguistics > Java developer > Worked 3.5 years for an Enterprise Search company (using Lucene Java) > Now at Mindquarry, creators on an Open Source Collaboration Software (Mindquarry uses Solr) 4 Question: What is a Search Engine? > Answer: A software that – builds an index on text – answers queries using that index “But we have a database already“ – A search engine offers Scalability Relevance Ranking Integrates different data sources (email, web pages, files, database, ...) 5 What is a search engine? (cont.) > Works on words, not on substrings auto != automatic, automobile > Indexing process: – Convert document – Extract text and meta data – Normalize text – Write (inverted) index – Example: Document 1: “Apache Lucene at Jazoon“ Document 2: “Jazoon conference“ Index: apache -> 1 conference -> 2 jazoon -> 1, 2 lucene -> 1 6 Apache Lucene Overview > Lucene Java 2.2 – Java library > Solr 1.2 – http-based index and search server > Nutch 0.9 – Internet search engine software > http://lucene.apache.org 7 Lucene Java > Java library for indexing and searching > No dependencies (not even a logging framework) > Works with Java 1.4 or later > Input for indexing: Document objects – Each document: set of Fields, field name: field content (plain text) > Input for searching: query strings or Query objects > Stores its index as files on disk > No document converters > No web crawler 8 Lucene Java Users > IBM OmniFind Yahoo! Edition > technorati.com > Eclipse > Furl > Nuxeo ECM > Monster.com > ..
  • Magnify Search Security and Administration Release 8.2 Version 04

    Magnify Search Security and Administration Release 8.2 Version 04

    Magnify Search Security and Administration Release 8.2 Version 04 April 08, 2019 Active Technologies, EDA, EDA/SQL, FIDEL, FOCUS, Information Builders, the Information Builders logo, iWay, iWay Software, Parlay, PC/FOCUS, RStat, Table Talk, Web390, WebFOCUS, WebFOCUS Active Technologies, and WebFOCUS Magnify are registered trademarks, and DataMigrator and Hyperstage are trademarks of Information Builders, Inc. Adobe, the Adobe logo, Acrobat, Adobe Reader, Flash, Adobe Flash Builder, Flex, and PostScript are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States and/or other countries. Due to the nature of this material, this document refers to numerous hardware and software products by their trademarks. In most, if not all cases, these designations are claimed as trademarks or registered trademarks by their respective companies. It is not this publisher's intent to use any of these names generically. The reader is therefore cautioned to investigate all claimed trademark rights before using any of these names other than to refer to the product described. Copyright © 2019, by Information Builders, Inc. and iWay Software. All rights reserved. Patent Pending. This manual, or parts thereof, may not be reproduced in any form without the written permission of Information Builders, Inc. Contents Preface ......................................................................... 7 Conventions ......................................................................... 7 Related Publications .................................................................

  • Unravel Data Systems Version 4.5

    UNRAVEL DATA SYSTEMS VERSION 4.5 Component name Component version name License names jQuery 1.8.2 MIT License Apache Tomcat 5.5.23 Apache License 2.0 Tachyon Project POM 0.8.2 Apache License 2.0 Apache Directory LDAP API Model 1.0.0-M20 Apache License 2.0 apache/incubator-heron 0.16.5.1 Apache License 2.0 Maven Plugin API 3.0.4 Apache License 2.0 ApacheDS Authentication Interceptor 2.0.0-M15 Apache License 2.0 Apache Directory LDAP API Extras ACI 1.0.0-M20 Apache License 2.0 Apache HttpComponents Core 4.3.3 Apache License 2.0 Spark Project Tags 2.0.0-preview Apache License 2.0 Curator Testing 3.3.0 Apache License 2.0 Apache HttpComponents Core 4.4.5 Apache License 2.0 Apache Commons Daemon 1.0.15 Apache License 2.0 classworlds 2.4 Apache License 2.0 abego TreeLayout Core 1.0.1 BSD 3-clause "New" or "Revised" License jackson-core 2.8.6 Apache License 2.0 Lucene Join 6.6.1 Apache License 2.0 Apache Commons CLI 1.3-cloudera-pre-r1439998 Apache License 2.0 hive-apache 0.5 Apache License 2.0 scala-parser-combinators 1.0.4 BSD 3-clause "New" or "Revised" License com.springsource.javax.xml.bind 2.1.7 Common Development and Distribution License 1.0 SnakeYAML 1.15 Apache License 2.0 JUnit 4.12 Common Public License 1.0 ApacheDS Protocol Kerberos 2.0.0-M12 Apache License 2.0 Apache Groovy 2.4.6 Apache License 2.0 JGraphT - Core 1.2.0 (GNU Lesser General Public License v2.1 or later AND Eclipse Public License 1.0) chill-java 0.5.0 Apache License 2.0 Apache Commons Logging 1.2 Apache License 2.0 OpenCensus 0.12.3 Apache License 2.0 ApacheDS Protocol
  • IBM Cloudant: Database As a Service Fundamentals

    IBM Cloudant: Database As a Service Fundamentals

    Front cover IBM Cloudant: Database as a Service Fundamentals Understand the basics of NoSQL document data stores Access and work with the Cloudant API Work programmatically with Cloudant data Christopher Bienko Marina Greenstein Stephen E Holt Richard T Phillips ibm.com/redbooks Redpaper Contents Notices . 5 Trademarks . 6 Preface . 7 Authors. 7 Now you can become a published author, too! . 8 Comments welcome. 8 Stay connected to IBM Redbooks . 9 Chapter 1. Survey of the database landscape . 1 1.1 The fundamentals and evolution of relational databases . 2 1.1.1 The relational model . 2 1.1.2 The CAP Theorem . 4 1.2 The NoSQL paradigm . 5 1.2.1 ACID versus BASE systems . 7 1.2.2 What is NoSQL? . 8 1.2.3 NoSQL database landscape . 9 1.2.4 JSON and schema-less model . 11 Chapter 2. Build more, grow more, and sleep more with IBM Cloudant . 13 2.1 Business value . 15 2.2 Solution overview . 16 2.3 Solution architecture . 18 2.4 Usage scenarios . 20 2.5 Intuitively interact with data using Cloudant Dashboard . 21 2.5.1 Editing JSON documents using Cloudant Dashboard . 22 2.5.2 Configuring access permissions and replication jobs within Cloudant Dashboard. 24 2.6 A continuum of services working together on cloud . 25 2.6.1 Provisioning an analytics warehouse with IBM dashDB . 26 2.6.2 Data refinement services on-premises. 30 2.6.3 Data refinement services on the cloud. 30 2.6.4 Hybrid clouds: Data refinement across on/off–premises. 31 Chapter 3.
  • Final Report CS 5604: Information Storage and Retrieval

    Final Report CS 5604: Information Storage and Retrieval

    Final Report CS 5604: Information Storage and Retrieval Solr Team Abhinav Kumar, Anand Bangad, Jeff Robertson, Mohit Garg, Shreyas Ramesh, Siyu Mi, Xinyue Wang, Yu Wang January 16, 2018 Instructed by Professor Edward A. Fox Virginia Polytechnic Institute and State University Blacksburg, VA 24061 1 Abstract The Digital Library Research Laboratory (DLRL) has collected over 1.5 billion tweets and millions of webpages for the Integrated Digital Event Archiving and Library (IDEAL) and Global Event Trend Archive Research (GETAR) projects [6]. We are using a 21 node Cloudera Hadoop cluster to store and retrieve this information. One goal of this project is to expand the data collection to include more web archives and geospatial data beyond what previously had been collected. Another important part in this project is optimizing the current system to analyze and allow access to the new data. To accomplish these goals, this project is separated into 6 parts with corresponding teams: Classification (CLA), Collection Management Tweets (CMT), Collection Management Webpages (CMW), Clustering and Topic Analysis (CTA), Front-end (FE), and SOLR. This report describes the work completed by the SOLR team which improves the current searching and storage system. We include the general architecture and an overview of the current system. We present the part that Solr plays within the whole system with more detail. We talk about our goals, procedures, and conclusions on the improvements we made to the current Solr system. This report also describes how we coordinate with other teams to accomplish the project at a higher level. Additionally, we provide manuals for future readers who might need to replicate our experiments.
  • Apache Lucene™ Integration Reference Guide

    Apache Lucene™ Integration Reference Guide

    Hibernate Search Apache Lucene™ Integration Reference Guide 3.3.0.Final Preface ............................................................................................................................ vii 1. Getting started ............................................................................................................. 1 1.1. System Requirements ......................................................................................... 1 1.2. Using Maven ...................................................................................................... 2 1.3. Configuration ...................................................................................................... 3 1.4. Indexing ............................................................................................................. 7 1.5. Searching ........................................................................................................... 7 1.6. Analyzer ............................................................................................................. 8 1.7. What's next ...................................................................................................... 10 2. Architecture ............................................................................................................... 11 2.1. Overview .......................................................................................................... 11 2.2. Back end .........................................................................................................
  • BI SEARCH and TEXT ANALYTICS New Additions to the BI Technology Stack

    BI SEARCH and TEXT ANALYTICS New Additions to the BI Technology Stack

    SECOND QUARTER 2007 TDWI BEST PRACTICES REPORT BI SEARCH AND TEXT ANALYTICS New Additions to the BI Technology Stack By Philip Russom TTDWI_RRQ207.inddDWI_RRQ207.indd cc11 33/26/07/26/07 111:12:391:12:39 AAMM Research Sponsors Business Objects Cognos Endeca FAST Hyperion Solutions Corporation Sybase, Inc. TTDWI_RRQ207.inddDWI_RRQ207.indd cc22 33/26/07/26/07 111:12:421:12:42 AAMM SECOND QUARTER 2007 TDWI BEST PRACTICES REPORT BI SEARCH AND TEXT ANALYTICS New Additions to the BI Technology Stack By Philip Russom Table of Contents Research Methodology and Demographics . 3 Introduction to BI Search and Text Analytics . 4 Defining BI Search . 5 Defining Text Analytics . 5 The State of BI Search and Text Analytics . 6 Quantifying the Data Continuum . 7 New Data Warehouse Sources from the Data Continuum . 9 Ramifications of Increasing Unstructured Data Sources . .11 Best Practices in BI Search . 12 Potential Benefits of BI Search . 12 Concerns over BI Search . 13 The Scope of BI Search . 14 Use Cases for BI Search . 15 Searching for Reports in a Single BI Platform Searching for Reports in Multiple BI Platforms Searching Report Metadata versus Other Report Content Searching for Report Sections Searching non-BI Content along with Reports BI Search as a Subset of Enterprise Search Searching for Structured Data BI Search and the Future of BI . 18 Best Practices in Text Analytics . 19 Potential Benefits of Text Analytics . 19 Entity Extraction . 20 Use Cases for Text Analytics . 22 Entity Extraction as the Foundation of Text Analytics Entity Clustering and Taxonomy Generation as Advanced Text Analytics Text Analytics Coupled with Predictive Analytics Text Analytics Applied to Semi-structured Data Processing Unstructured Data in a DBMS Text Analytics and the Future of BI .
  • Apache Lucene - Overview

    Apache Lucene - Overview

    Apache Lucene - Overview Table of contents 1 Apache Lucene...................................................................................................................2 2 The Apache Software Foundation..................................................................................... 2 3 Lucene News......................................................................................................................2 3.1 27 November 2011 - Lucene Core 3.5.0...................................................................... 2 3.2 26 October 2011 - Java 7u1 fixes index corruption and crash bugs in Apache Lucene Core and Apache Solr.............................................................................................................3 3.3 14 September 2011 - Lucene Core 3.4.0......................................................................3 3.4 28 July 2011 - WARNING: Index corruption and crashes in Apache Lucene Core / Apache Solr with Java 7.........................................................................................................4 3.5 1 July 2011 - Lucene Core 3.3.....................................................................................5 3.6 4 June 2011 - Lucene Core 3.2.................................................................................... 6 3.7 31 March 2011 - Lucene Core 3.1............................................................................... 6 3.8 3 December 2010 - Lucene Java 3.0.3 and 2.9.4 available.......................................... 7 3.9 18 June 2010 -
  • Hot Technologies” Within the O*NET® System

    Hot Technologies” Within the O*NET® System

    Identification of “Hot Technologies” within the O*NET® System Phil Lewis National Center for O*NET Development Jennifer Norton North Carolina State University Prepared for U.S. Department of Labor Employment and Training Administration Office of Workforce Investment Division of National Programs, Tools, & Technical Assistance Washington, DC April 4, 2016 www.onetcenter.org National Center for O*NET Development, Post Office Box 27625, Raleigh, NC 27611 Table of Contents Background ......................................................................................................................... 2 Hot Technologies Identification Procedure ...................................................................... 3 Mine data to collect the top technology related terms ................................................ 3 Convert the data-mined technology terms into O*NET technologies ......................... 3 Organize the hot technologies within the O*NET Tools & Technology Taxonomy ..... 4 Link the hot technologies to O*NET-SOC occupations .............................................. 4 Determine the display of occupations linked to a hot technology ............................... 4 Summary ............................................................................................................................. 5 Figure 1: O*NET Hot Technology Icon .............................................................................. 6 Appendix A: Hot Technologies Identified During the Initial Implementation ................ 7 National Center
  • Text Analysis: the Next Step in Search

    Text Analysis: the Next Step in Search

    eDiscovery & Information Management Text Analysis: The Next Step In Search ZyLAB White Paper Johannes C. Scholtes, Ph.D. Chief Strategy Officer, ZyLAB Contents Summary 3 Finding Without Knowing Exactly What to Look For 4 Beyond the Google Standard 4 Challenges Facing Text Analysis 6 Control of Unstructured Information 6 Different Levels of Semantic Information Extraction 7 Co-reference and Anaphora Resolution 11 Faceted Search and Information Visualization 12 Text Analysis on Non-English Documents 15 Content Analytics on Multimedia Files: Audio Search 16 A Prosperous Future for Text Analysis 17 About ZyLAB 19 Summary Text and content analysis differs from traditional search in that, whereas search requires a user to know what he or she is looking for, text analysis attempts to discover information in a pattern that is not known before- hand. One of the most compelling differences with regular (web) search is that typical search engines are optimized to find only the most relevant documents; they are not optimized to find all relevant documents. The majority of commonly-used search tools are built to retrieve only the most popular hits—which simply doesn’t meet the demands of exploratory legal search. This whitepaper will lead the reader beyond the Google standard, explore the limitations and possibilities of text analysis technology and show how text analysis becomes an essential tool to help process and analyze to- day’s enormous amounts of enterprise information in a timely fashion. 3 Finding Without Knowing Exactly What to Look For In general, text analysis refers to the process of extracting interesting and non-trivial information and knowledge from unstructured text.