DOCSLIB.ORG

Near-Optimal Space Perfect Hashing Algorithms

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Near-Optimal Space Perfect Hashing Algorithms Fabiano Cupertino Botelho Supervisor - Nivio Ziviani Near-Optimal Space Perfect Hashing Algorithms PhD. dissertation presented to the Grad- uate Program in Computer Science of the Federal University of Minas Gerais as a par- tial requirement to obtain the PhD. degree in Computer Science. Belo Horizonte September 29, 2008 To my dear wife Jana´ına. To my dear parents Maria L´ucia and Jos´eV´ıtor. To my dear sisters Gleiciane and Cristiane. Acknowledgements To God for having granted me life and wisdom to realize a dream of childhood and for the great help in difficult moments. To my dear wife Jana´ına Marcon Machado Botelho for the love, understanding by several times when I could not give her the attention she deserves, companionship and en- couragement during moments in which I desired to give up everything. Jana thank you for sharing your life with me and the victories won during the entire doctorate. With the grace of God in our lives we will continue to be very happy. To my dear parents Maria L´ucia de Lima Botelho and Jos´eVitor Botelho for sacrifices made in the past that have given support for this achievement. To my dear sisters Cristiane Cupertino Botelho and Gleiciane Cupertino Botelho for the love of the best two sisters in the world. To my dear aunt M´arcia Novaes Alves and my dear uncle Sud´ario Alves for always welcome me with affection, giving me much support throughout my doctorate. To Prof. Nivio Ziviani for the excellent work of supervision and for being an example of professionalism and dedication to work. His extensive experience in academic research, and particularly in the areas of information retrieval and algorithms have been of extreme importance to realize this work. In addition, his excellent support, attention and encouragement were of great importance not only for completing the doctorate, but also for my academic and professional life. To Prof. Rasmus Pagh with whom I’ve learned a lot about techniques for designing and analyzing hashing algorithms, being crucial his participation in this thesis. To Prof. Yoshiharu Kohayakawa for the attention dedicated to the discussions that con- tributed to improve the quality of this work. Thanks also to receive me at the Institute of Mathematics and Statistics at the University of S˜ao Paulo and for all the support given to my work during the time I spent in S˜ao Paulo. To Prof. Edleno Silva de Moura for trusting on me and for always encouraging me. Thanks also to receive me at the Department of Computer Science at the Federal University of Amazonas during the time I spent in Manaus. To the other Professors that evaluated this thesis, namely, Gaston Gonnet, Antˆonio Al- fredo Loureiro, Wagner Meira Jr. and Jayme Luiz Szwarcfiter for having accepted to participate of the PhD. defense and for the relevant criticisms and suggestions. To Djamal Belazzougui for the intelligent suggestions and contributions made to this thesis and to the CMPH library. To Davi Reis for having conceived the idea of the CMPH library, which was fundamental to disseminate the results obtained in this thesis. To my colleague and friend Marco Antˆonio Pinheiro de Cristo for the fun moments we spent together during our English classes and for always encoraging me. To my colleague and friend Thierson Couto for his friendship, and to be always ready to cooperate. To my colleague and friend David Menotti for the discussions, suggestions and criticisms that contributed much in the beginning of this work. To my colleague and friend David Fernandes for having received me in your home during the time I spent in Manaus and for his endless friendship. To my colleagues and friends of our great and unforgettable soccer team Curucu and their wives for the friendship conquered during the period we spent together. Thanks Pedro Neto, Maur´ıcio Figueiredo, Eduardo Freire Nakamura, Ruiter Caldas, Andr´eLins, Jos´ePinheiro, Guillermo Camara Chavez, Martin Gomez Ravetti, David Patricio Viscarra del Pozo and David Menotti for the amazing and fun moments that served to relieve the stress of this difficult period of doctorate. To colleagues and friends from that period of our undergraduate course that, through the mailing list intrigas99, always supported me being close or distant. I thank also for all the good laughs that I gave when I was reading some posts of the list, which certainly helped a lot to ease the tension in difficult times. To my colleagues and friends of the Laboratory for Treating Information (LATIN) An´ısio Mendes Lacerda, Alvaro´ Pereira Jr., Charles Ornelas Almeida, Claudine Santos Badue, Daniel Galinkin, Denilson Pereira, Guilherme Vale Menezes, Hendrickson R. Langbehn, Humberto Mossri, Marco Antˆonio Pinheiro de Cristo, Marco Aur´elio Barreto Modesto, P´avel Calado and Wladmir Cardoso Brand˜ao for the criticism and suggestions provided during the defense preparation and for the climate of friendship we have established within LATIN. To Professors and employees of the Department of Computer Science at the Federal University of Minas Gerais that in various ways contributed to the completion of this work. To Professors and employees of the Department of Computer Engineering at the Federal Center for Technological Education of Minas Gerais for having received me so well and in a so respectful manner to integrate the department team. To the scholarships granted by CAPES (Coordination of Improvement of Higher Edu- cation) and CNPq (National Council for Scientific and Technological Development), which served as subsidy for the time dedicated to this thesis. Abstract A perfect hash function (PHF) h : S [0, m 1] for a key set S U of size n, where → − ⊆ m n and U is a key universe, is an injective function that maps the keys of S to unique ≥ values. A minimal perfect hash function (MPHF) is a PHF with m = n, the smallest possi- ble range. Minimal perfect hash functions are widely used for memory efficient storage and fast retrieval of items from static sets, such as words in natural languages, reserved words in programming languages or interactive systems, universal resource locations (URLs) in web search engines, or item sets in data mining techniques. In this thesis we present a simple, highly scalable and near-space optimal perfect hashing algorithm. Evaluation of a PHF on a given element of S requires constant time, and the dominating phase in the construction algorithm consists of sorting n fingerprints of O(log n) bits in O(n) time. The space usage depends on the relation between m and n. For m = n the space usage is in the range 2.62n to 3.3n bits, depending on the constants involved in the construction and in the evaluation phases. For m = 1.23n the space usage is in the range 1.95n to 2.7n bits. In all cases, this is within a small constant factor from the information theoretical minimum of approximately 1.44n bits for MPHFs and 0.89n bits for PHFs, something that has not been achieved by previous algorithms, except asymptotically for very large n. This small space usage opens up the use of MPHFs to applications for which they were not useful in the past. We demonstrate the scalability of our algorithm by constructing an MPHF for a set of 1.024 billion URLs from the World Wide Web of average length 64 characters in approx- imately 50 minutes, using a commodity PC. We also present a distributed and parallel implementation of the algorithm, which generates an MPHF for the same URL set, using a 14 computer cluster, in approximately 4 minutes, achieving an almost linear speedup. Also, for 14.336 billion 16-byte random integers distributed among the 14 participating ma- chines, the algorithm outputs an MPHF in approximately 50 minutes, with a performance degradation of 20%. Resumo Uma fun¸c˜ao hash perfeita (FHP) h : U [0, m 1] para um conjunto de chaves S U → − ⊆ de tamanho n, onde m n e U ´eum universo de chaves, ´euma fun¸c˜ao injetora que ≥ mapeia as chaves de S para valores ´unicos. Uma fun¸c˜ao hash perfeita m´ınima (FHPM) ´euma FHP com m = n, o menor intervalo poss´ıvel. Fun¸c˜oes hash perfeitas m´ınimas s˜ao amplamente utilizadas para armazenamento eficiente e recupera¸c˜ao r´apida de itens de con- juntos est´aticos, como palavras em linguagem natural, palavras reservadas em linguagens de programa¸c˜ao ou sistemas interativos, URLs (universal resource locations) em m´aquinas de busca, ou conjuntos de itens em t´ecnicas de minera¸c˜ao de dados. Nesta tese n´os apresentamos um algoritmo de hashing perfeito altamente escal´avel e de espa¸co quase ´otimo. A avalia¸c˜ao de uma FHP sobre um dado elemento de S requer tempo constante, e a fase dominante no algoritmo de constru¸c˜ao consiste da ordena¸c˜ao de n fingerprints de O(log n) bits em tempo O(n). A utiliza¸c˜ao de espa¸co depende da rela¸c˜ao entre m e n. Para m = n a utiliza¸c˜ao de espa¸co est´adentro do intervalo 2, 62n `a3, 3n bits, dependendo das constantes envolvidas nas fases de constru¸c˜ao e avalia¸c˜ao. Para m = 1, 23n a utiliza¸c˜ao de espa¸co est´adentro do intervalo 1, 95n `a2, 7n bits. Em todos os casos, isto est´adistante por um pequeno fator constante do m´ınimo te´orico de aproximadamente 1, 44n bits para FHPMs e 0, 89n bits para FHPs, uma coisa que n˜ao foi alcan¸cada por algoritmos anteriores, exceto assint´oticamente para valores de n muito grandes.
Load more

Recommended publications
  • CS 473: Algorithms, Fall 2019
    CS 473: Algorithms, Fall 2019 Universal and Perfect Hashing Lecture 10 September 26, 2019 Chandra and Michael (UIUC) cs473 1 Fall 2019 1 / 45 Today's lecture: Review pairwise independence and related constructions (Strongly) Universal hashing Perfect hashing Announcements and Overview Pset 4 released and due on Thursday, October 3 at 10am. Note one day extension over usual deadline. Midterm 1 is on Monday, Oct 7th from 7-9.30pm. More details and conflict exam information will be posted on Piazza. Next pset will be released after the midterm exam. Chandra and Michael (UIUC) cs473 2 Fall 2019 2 / 45 Announcements and Overview Pset 4 released and due on Thursday, October 3 at 10am. Note one day extension over usual deadline. Midterm 1 is on Monday, Oct 7th from 7-9.30pm. More details and conflict exam information will be posted on Piazza. Next pset will be released after the midterm exam. Today's lecture: Review pairwise independence and related constructions (Strongly) Universal hashing Perfect hashing Chandra and Michael (UIUC) cs473 2 Fall 2019 2 / 45 Part I Review Chandra and Michael (UIUC) cs473 3 Fall 2019 3 / 45 Pairwise independent random variables Definition Random variables X1; X2;:::; Xn from a range B are pairwise independent if for all 1 ≤ i < j ≤ n and for all b; b0 2 B, 0 0 Pr[Xi = b; Xj = b ] = Pr[Xi = b] · Pr[Xj = b ] : Chandra and Michael (UIUC) cs473 4 Fall 2019 4 / 45 Interesting case: n = m = p where p is a prime number Pick a; b uniformly at random from f0; 1; 2;:::; p − 1g Set Xi = ai + b Only need to store a; b.
    [Show full text]
  • Fundamental Data Structures Contents
    Fundamental Data Structures Contents 1 Introduction 1 1.1 Abstract data type ........................................... 1 1.1.1 Examples ........................................... 1 1.1.2 Introduction .......................................... 2 1.1.3 Defining an abstract data type ................................. 2 1.1.4 Advantages of abstract data typing .............................. 4 1.1.5 Typical operations ...................................... 4 1.1.6 Examples ........................................... 5 1.1.7 Implementation ........................................ 5 1.1.8 See also ............................................ 6 1.1.9 Notes ............................................. 6 1.1.10 References .......................................... 6 1.1.11 Further ............................................ 7 1.1.12 External links ......................................... 7 1.2 Data structure ............................................. 7 1.2.1 Overview ........................................... 7 1.2.2 Examples ........................................... 7 1.2.3 Language support ....................................... 8 1.2.4 See also ............................................ 8 1.2.5 References .......................................... 8 1.2.6 Further reading ........................................ 8 1.2.7 External links ......................................... 9 1.3 Analysis of algorithms ......................................... 9 1.3.1 Cost models ......................................... 9 1.3.2 Run-time analysis
    [Show full text]
  • Algorithms in a Nutshell 2E
    www.it-ebooks.info SECOND EDITION Algorithms in a Nutshell 2E George T. Heineman, Gary Pollice, and Stanley Selkow Boston www.it-ebooks.info Algorithms in a Nutshell 2E, Second Edition by George T. Heineman, Gary Pollice, and Stanley Selkow Copyright © 2010 George Heineman, Gary Pollice and Stanley Selkow. All rights re‐ served. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://safaribooksonline.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or [email protected]. Editor: Mary Treseler Indexer: FIX ME! Production Editor: FIX ME! Cover Designer: Karen Montgomery Copyeditor: FIX ME! Interior Designer: David Futato Proofreader: FIX ME! Illustrator: Rebecca Demarest January -4712: Second Edition Revision History for the Second Edition: 2015-07-27: Early release revision 1 See http://oreilly.com/catalog/errata.csp?isbn=0636920032885 for release details. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. !!FILL THIS IN!! and related trade dress are trade‐ marks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their prod‐ ucts are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.
    [Show full text]
  • Optimal Algorithms for Minimal Perfect Hashing
    Optimal algorithms for minimal perfect hashing George Havas and Bohdan S. Majewski Key Centre for Software Technology Department of Computer Science University of Queensland Queensland 4072 Australia Abstract Minimal perfect hash functions are used for memory e±cient storage and fast retrieval of items from static sets. We present an overview of previous solutions and analyze a new algorithm based on random graphs for generating order preserving minimal perfect hash functions. We show that the new algorithm is both time and space optimal in addition to being practical. The algorithm generates a minimal perfect hash function in two steps. First a special kind of function into a random graph is computed probabilistically. Then this function is re¯ned deterministically to a minimal perfect hash function. The ¯rst step uses linear random time, while the second runs in linear deterministic time. Key words: Data structures, probabilistic algorithms, analysis of algorithms, hashing, random graphs 1 Introduction Consider a set W of m words each of which is a ¯nite string of symbols over an ordered alphabet §. A hash function is a function h : W ! I that maps the set of words W into some given interval of integers I, say [0; k¡1], where k is an integer, and usually k ¸ m. The hash function, given a word, computes an address (an integer from I) for the storage or retrieval of that item. The storage area used to store items is known as a hash table. Words for which the same address is computed are called synonyms. Due to the existence of synonyms a situation called collision may arise in which two items w1 and w2 have the same address.
    [Show full text]
  • The Tree Model for Hashing: Lower and Upper Bounds
    The Tree Model for Hashing: Lower and Upper Bounds by Joseph Gil Friedhelm Meyer auf Der Heide and A vi Wigderson Technical Report 91-23 December 1991 Department of Computer Science University of British Columbia Rm 333 - 6356 Agricultural Road Vancouver, B.C. CANADA V6T 1Z2 The Tree Model for Hashing: Lower and Upper Bounds* Joseph Gil Friedhelm Meyer auf Der Heide The University of British Columbia Universitat Paderborn A vi Wigderson The Hebrew University of Jerusalem December 1991 Abstract We define a new simple general model which captures many natural (sequential and parallel) hashing algorithms. In a game against nature, the algorithm and coin­ tosses cause the evolution of a random tree, whose size corresponds to space (hash table size), and two notions of depth correspond to the longest probe sequences for insertion (parallel insertion time) and search of a key, respectively. We stu_dy these parameters of hashing schemes by analyzing the underlying stochas­ tic process, and derive tight lower and upper bounds on the relation between the amount of memory allocated to the hashing execution and the worst case insertion time. In particular, we show that if linear memory is used then not all key inser­ tions to the hash table can be done in constant time. Except for extremely unlikely events, every input set of size n will have members for which f!(lg lg n) applications of a hash function are required. From a parallel perspective it can be said that n processors need f!(lglgn) expected time to hash themselves into O(n) space, al­ though serial algorithms exist that achieve constant amortized time for insertion, as well as constant worst case search time [16).
    [Show full text]
  • Lecture 08 Hashing
    18.10.18 ADT – asscociative array • INSERT, SEARCH, DELETE Advanced Algorithmics (6EAP) • An associative array (also associative container, MTAT.03.238 map, mapping, dictionary, finite map, and in query- Hashing processing an index or index file) is an abstract data type composed of a collection of unique keys and a collection of values, where each key is associated Jaak Vilo with one value (or set of values). The operation of 2018 Fall finding the value associated with a key is called a lookup or indexing, and this is the most important operation supported by an associative array. Jaak Vilo 1 Why? • Speed! - O(1) • Space efficient 1 18.10.18 Keys • Integers • Strings • ID-codes • Floating point nr-s(?) • Usual assumption – keys are integers. • Step 1: Map keys to integers. 2 18.10.18 3 18.10.18 Primary clustering Implementation of open addressing How do we define h(k,i) ? Linear probing: Quadratic probing: Double hashing: 4 18.10.18 Quadratic Probing Quadratic Probing • Suppose that an element should appear in bin • If one of h + i2 falls into a cluster, this does not h: imply the next one will – if bin h is occupied, then check the following sequence of bins: h + 12, h + 22, h + 32, h + 42, h + 52, ... h + 1, h + 4, h + 9, h + 16, h + 25, ... • For example, with M = 17: • Secondary clustering – all k colliding in h(k) will cluster to same locations… 5 18.10.18 Birthday Paradox • In probability theory, the birthday problem, or birthday paradox[1] pertains to the probability that in a set of randomly chosen people some pair of them will have the same birthday.
    [Show full text]
  • Data Structures
    Data structures PDF generated using the open source mwlib toolkit. See http://code.pediapress.com/ for more information. PDF generated at: Thu, 17 Nov 2011 20:55:22 UTC Contents Articles Introduction 1 Data structure 1 Linked data structure 3 Succinct data structure 5 Implicit data structure 7 Compressed data structure 8 Search data structure 9 Persistent data structure 11 Concurrent data structure 15 Abstract data types 18 Abstract data type 18 List 26 Stack 29 Queue 57 Deque 60 Priority queue 63 Map 67 Bidirectional map 70 Multimap 71 Set 72 Tree 76 Arrays 79 Array data structure 79 Row-major order 84 Dope vector 86 Iliffe vector 87 Dynamic array 88 Hashed array tree 91 Gap buffer 92 Circular buffer 94 Sparse array 109 Bit array 110 Bitboard 115 Parallel array 119 Lookup table 121 Lists 127 Linked list 127 XOR linked list 143 Unrolled linked list 145 VList 147 Skip list 149 Self-organizing list 154 Binary trees 158 Binary tree 158 Binary search tree 166 Self-balancing binary search tree 176 Tree rotation 178 Weight-balanced tree 181 Threaded binary tree 182 AVL tree 188 Red-black tree 192 AA tree 207 Scapegoat tree 212 Splay tree 216 T-tree 230 Rope 233 Top Trees 238 Tango Trees 242 van Emde Boas tree 264 Cartesian tree 268 Treap 273 B-trees 276 B-tree 276 B+ tree 287 Dancing tree 291 2-3 tree 292 2-3-4 tree 293 Queaps 295 Fusion tree 299 Bx-tree 299 Heaps 303 Heap 303 Binary heap 305 Binomial heap 311 Fibonacci heap 316 2-3 heap 321 Pairing heap 321 Beap 324 Leftist tree 325 Skew heap 328 Soft heap 331 d-ary heap 333 Tries 335 Trie
    [Show full text]
  • Choosing Best Hashing Strategies and Hash Functions
    Choosing Best Hashing Strategies and Hash Functions Thesis submitted in partial fulfillment of the requirements for the award of Degree of Master of Engineering in Software Engineering By: Name: Mahima Singh Roll No: 80731011 Under the supervision of: Dr. Deepak Garg Assistant Professor, CSED & Mr. Ravinder Kumar Lecturer, CSED COMPUTER SCIENCE AND ENGINEERING DEPARTMENT THAPAR UNIVERSITY PATIALA – 147004 JUNE 2009 Certificate I hereby certify that the work which is being presented in the thesis report entitled, “Choosing Best Hashing Strategies and Hash Functions”, submitted by me in partial fulfillment of the requirements for the award of degree of Master of Engineering in Computer Science and Engineering submitted in Computer Science and Engineering Department of Thapar University, Patiala, is an authentic record of my own work carried out under the supervision of Dr. Deepak Garg and Mr. Ravinder Kumar and refers other researcher’s works which are duly listed in the reference section. The matter presented in this thesis has not been submitted for the award of any other degree of this or any other university. (Mahima Singh) This is to certify that the above statement made by the candidate is correct and true to the best of my knowledge. (Dr. Deepak Garg) Assistant Professor, CSED Thapar University Patiala And (Mr. Ravinder Kumar) Lecturer, CSED Thapar University Patiala Countersigned by: Dr. Rajesh Kumar Bhatia (Dr. R.K.SHARMA) Assistant Professor & Head Dean (Academic Affairs) Computer Science & Engineering. Department Thapar University, Thapar University Patiala. Patiala. i Acknowledgement No volume of words is enough to express my gratitude towards my guide, Dr. Deepak Garg Assistant Professor.
    [Show full text]