Zebra - User’s Guide and Reference i Zebra - User’s Guide and Reference Copyright © 1994-2020 Index Data Zebra - User’s Guide and Reference iii COLLABORATORS TITLE : Zebra - User’s Guide and Reference ACTION NAME DATE SIGNATURE WRITTEN BY Adam Dickmeiss, November 5, 2020 Heikki Levanto, Marc Cromme, Mike Taylor, and Sebastian Hammer REVISION HISTORY NUMBER DATE DESCRIPTION NAME Zebra - User’s Guide and Reference v Contents 1 Introduction 1 1.1 Overview . .1 1.2 Zebra Features Overview . .2 1.2.1 Zebra Document Model . .2 1.2.2 Zebra Search Features . .2 1.2.3 Zebra Index Scanning . .2 1.2.4 Zebra Document Presentation . .2 1.2.5 Zebra Sorting and Ranking . .2 1.2.6 Zebra Live Updates . .2 1.2.7 Zebra Networked Protocols . .2 1.2.8 Zebra Data Size and Scalability . .2 1.2.9 Zebra Supported Platforms . .2 1.3 References and Zebra based Applications . .2 1.3.1 Koha free open-source ILS . .4 1.3.2 Kete Open Source Digital Library and Archiving software . .9 1.3.3 ReIndex.Net web based ILS . .9 1.3.4 DADS - the DTV Article Database Service . .9 1.3.5 ULS (Union List of Serials) . .9 1.3.6 Various web indexes . 10 1.4 Support . 10 2 Installation 11 2.1 UNIX . 11 2.2 GNU/Debian . 12 2.2.1 GNU/Debian Linux on amd64/i386 Platform . 12 2.2.2 GNU/Debian and Ubuntu on other architectures . 13 2.3 Windows . 13 2.4 Upgrading from Zebra version 1.3.x . 14 3 Tutorial 17 3.1 A first OAI indexing example . 17 3.2 Searching the OAI database by web service . 18 3.3 Presenting search results in different formats . 18 3.4 More interesting searches . 19 3.5 Investigating the content of the indexes . 20 3.6 Setting up a correct SRU web service . 20 3.7 Searching the OAI database by Z39.50 protocol . 21 4 Overview of Zebra Architecture 25 4.1 Local Representation . 25 4.2 Main Components . 25 4.2.1 Core Zebra Libraries Containing Common Functionality . 25 4.2.2 Zebra Indexer . 26 4.2.3 Zebra Searcher/Retriever . 26 4.2.4 YAZ Server Frontend . 26 4.2.5 Record Models and Filter Modules . 27 4.2.5.1 DOM XML Record Model and Filter Module . 27 4.2.5.2 ALVIS XML Record Model and Filter Module . 27 4.2.5.3 GRS-1 Record Model and Filter Modules . 28 4.2.5.4 TEXT Record Model and Filter Module . 28 4.3 Indexing and Retrieval Workflow . 28 4.4 Retrieval of Zebra internal record data . 29 5 Query Model 33 5.1 Query Model Overview . 33 5.1.1 Query Languages . 33 5.1.1.1 Prefix Query Format (PQF) . 33 5.1.1.2 Common Query Language (CQL) . 33 5.1.2 Operation types . 33 5.1.2.1 Explain Operation . 34 5.1.2.2 Search Operation . 34 5.1.2.3 Scan Operation . 34 5.2 RPN queries and semantics . 34 5.2.1 RPN tree structure . 34 Zebra - User’s Guide and Reference vii 5.2.1.1 Attribute sets . 35 5.2.1.2 Boolean operators . 35 5.2.1.3 Atomic queries (APT) . 36 5.2.1.4 Named Result Sets . 37 5.2.1.5 Zebra’s special access point of type ’string’ . 38 5.2.1.6 Zebra’s special access point of type ’XPath’ for GRS-1 filters . 38 5.2.2 Explain Attribute Set . 40 5.2.2.1 Use Attributes (type = 1) . 40 5.2.2.2 Explain searches with yaz-client . 40 5.2.3 BIB-1 Attribute Set . 41 5.2.3.1 Use Attributes (type 1) . 41 5.2.4 Zebra general Bib1 Non-Use Attributes (type 2-6) . 42 5.2.4.1 Relation Attributes (type 2) . 42 5.2.4.2 Position Attributes (type 3) . 43 5.2.4.3 Structure Attributes (type 4) . 44 5.2.4.4 Truncation Attributes (type = 5) . 45 5.2.4.5 Completeness Attributes (type = 6) . 46 5.3 Extended Zebra RPN Features . 47 5.3.1 Zebra specific retrieval of all records . 47 5.3.2 Zebra specific Search Extensions to all Attribute Sets . 48 5.3.2.1 Zebra Extension Embedded Sort Attribute (type 7) . 48 5.3.2.2 Zebra Extension Rank Weight Attribute (type 9) . 49 5.3.2.3 Zebra Extension Term Reference Attribute (type 10) . 49 5.3.2.4 Local Approximative Limit Attribute (type 11) . 49 5.3.2.5 Global Approximative Limit Attribute (type 12) . 50 5.3.3 Zebra specific Scan Extensions to all Attribute Sets . 50 5.3.3.1 Zebra Extension Result Set Narrow (type 8) . 50 5.3.3.2 Zebra Extension Approximative Limit (type 12) . 51 5.3.4 Zebra special IDXPATH Attribute Set for GRS-1 indexing . 51 5.3.4.1 IDXPATH Use Attributes (type = 1) . 51 5.3.5 Mapping from PQF atomic APT queries to Zebra internal register indexes . 52 5.3.5.1 Mapping of PQF APT access points . 53 5.3.5.2 Mapping of PQF APT structure and completeness to register type . 54 5.3.6 Zebra Regular Expressions in Truncation Attribute (type = 5) . 56 5.4 Server Side CQL to PQF Query Translation . 57 6 Administrating Zebra 59 6.1 Record Types . 59 6.2 The Zebra Configuration File . 60 6.3 Locating Records . 62 6.4 Indexing with no Record IDs (Simple Indexing) . 62 6.5 Indexing with File Record IDs . 63 6.6 Indexing with General Record IDs . 64 6.7 Register Location . 65 6.8 Safe Updating - Using Shadow Registers . 65 6.8.1 Description . 65 6.8.2 How to Use Shadow Register Files . 66 6.9 Relevance Ranking and Sorting of Result Sets . 67 6.9.1 Overview . 67 6.9.2 Static Ranking . 67 6.9.3 Dynamic Ranking . 68 6.9.3.1 Dynamically ranking using PQF queries with the ’rank-1’ algorithm . 68 6.9.3.2 Dynamically ranking CQL queries . 70 6.9.4 Sorting . ..