DOCSLIB.ORG

Efficient Data Structures for Internal Queries in Texts

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Efficient Data Structures for Internal Queries in Texts University of Warsaw Faculty of Mathematics, Informatics and Mechanics Tomasz Kociumaka Efficient Data Structures for Internal Queries in Texts PhD dissertation Supervisor prof. dr hab. Wojciech Rytter Institute of Informatics University of Warsaw October 2018 Author’s declaration: I hereby declare that this dissertation is my own work. October 26, 2018 . Tomasz Kociumaka Supervisor’s declaration: The dissertation is ready to be reviewed October 26, 2018 . prof. dr hab. Wojciech Rytter i Abstract This thesis is devoted to internal queries in texts, which ask to solve classic text-processing problems for substrings of a given text. More precisely, the task is to preprocess a static string T of length n (called the text) and construct a data structure answering certain questions about the substrings of T . The substrings involved in each query are specified in constant space by their occurrences in T , called fragments of T , identified by the start and the end positions. Components for internal queries often become parts of more complex data structures, and they are used in many algorithms for text processing. Longest Common Extension Queries, asking for the length of the longest common prefix of two substrings of the text T , are by far the most popular internal queries. They are used for checking if two fragments match (represent the same string) and for lexicographic comparison of substrings. Due to an optimal solution in the standard setting of texts over polynomially-bounded integer alphabets, with O(1)-time queries, O(n) size, and O(n) construction time, they have found numerous applications across stringology. In this dissertation, we provide the first optimal data structure for smaller alphabets of size σ n: it handles queries in O(1) time, takes O(n/ logσ n) space, and admits an O(n/ logσ n)-time construction (from the packed representation of T with Θ(logσ n) characters in each machine word). We then go back to alphabets of size σ polynomial in n and focus on more complex internal queries. Our first data structure supports Internal Pattern Matching Queries, which ask for the occurrences of one substring x within another substring y. After O(n)-time preprocessing of the text T , it answers these queries in time proportional to the quotient |y|/|x| of substrings’ lengths, which is required due to the information content of the output. We also use this data structure for Period Queries, asking for the periods of a given substring. Here, our logarithmic query time is also optimal by a similar information-theoretic argument. Further data structures are designed for Minimal Suffix and Minimal Rota- tion Queries, asking to compute the lexicographically smallest non-empty suffix and cyclic rotation of a given substring, respectively. They are answered in O(1) time after O(n)-time preprocessing. We also consider a more general problem of simulating the suffix array of a given substring (Substring Suffix Selection Queries, asking for the kth lexicographically smallest suffix of a substring) and its inverse suffix array (Substring Suffix Rank Queries, asking for the lexico- graphic rank of a substring’s suffix). Our data structure supports√ these queries in O(log n) time, takes O(n) space, and can be constructed in O(n log n) time. The tools developed in this dissertation additionally yield improved results for several kinds of Substring Compression Queries, which ask for the compressed representation of a given substring obtained using a specific method; we consider schemes based on the Lempel–Ziv parsing and the Burrows–Wheeler transform. Our results combine text-processing tools with combinatorics on words and state-of-the-art general-purpose data structures. The key technical contribution is a novel locally consistent symmetry-breaking scheme, formalized in terms of synchronizing functions, which is central to our solutions for Longest Common Extension Queries and Internal Pattern Matching Queries. 2012 ACM Subject Classification: Theory of computation → Pattern matching Keywords: Data structures, Longest common extension, Longest common prefix, Local consistency, Minimal Suffix, Period, Substring compression, Substring queries ii Streszczenie Rozprawa dotyczy zapytań wewnętrznych w tekstach, czyli rozwiązywania kla- sycznych problemów algorytmiki tekstów dla podsłów danego słowa. Formalnie, zadanie polega na wstępnym przetworzeniu statycznego słowa T długości n (tekstu), tak aby skonstruować strukturę danych odpowiadającą na pewne pytania dotyczące jego podsłów. Podsłowa te są identyfikowane w stałej pamięci za pomocą wystąpień w T , zwanych fragmentami tekstu i reprezentowanych przez pozycje początkową oraz końcową. Komponenty dla zapytań wewnętrznych są używane w wielu algorytmach tekstowych i wchodzą w skład bardziej rozbudowanych struktur danych. Zapytania o długość najdłuższego wspólnego prefiksu dwóch podsłów tekstu T stanowią zdecydowanie najpowszechniejszy problem tego rodzaju. Używa się ich między innymi do sprawdzania, czy dwa fragmenty pasują do siebie (reprezentują to samo słowo), i do porównywania podsłów w porządku leksykograficznym. Mnogość zastosowań wynika z istnienia optymalnego rozwiązania w standardowym modelu tekstów (nad alfabetem złożonym z liczb naturalnych wielkości wielomianowej ze względu na n), czyli struktury danych wielkości O(n), która odpowiada na zapytania w czasie O(1) i posiada procedurę konstrukcji w czasie O(n). W rozprawie przedstawiamy pierwszą optymalną strukturę danych dla małych alfabetów rozmiaru σ n: strukturę wielkości O(n/ logσ n) charakteryzującą się czasem zapytania O(1) i czasem konstrukcji O(n/ logσ n) (na podstawie spakowanej reprezentacji tekstu, gdzie każda komórka pamięci zawiera Θ(logσ n) symboli). W dalszej części rozprawy wracamy do tekstów nad alfabetem wielkości wie- lomianowej ze względu na n i skupiamy się na bardziej złożonych zapytaniach wewnętrznych. Nasza pierwsza struktura danych rozwiązuje problem wewnętrznego wyszukiwania wzorca, czyli zapytań o wystąpienia jednego podsłowa x w obrębie drugiego podsłowa y. Po przetworzeniu tekstu T w czasie O(n), odpowiada ona na takie zapytania w czasie proporcjonalnym do ilorazu |y|/|x| długości podsłów, który jest wymagany ze względu na pesymistyczną złożoność informacji zawartej w wyniku. Tej samej struktury danych używamy także do zapytań o wszystkie okresy wskazanego podsłowa; uzyskany tutaj czas logarytmiczny również jest optymalny. Kolejne struktury danych stworzone zostały do obsługi zapytań o najmniejszy leksykograficznie sufiks i najmniejszą leksykograficznie rotację cykliczną wskazanego podsłowa. Odpowiadamy na nie w czasie O(1) po przetworzeniu tekstu w czasie O(n). Rozważamy także bardziej ogólne pytania o k-ty leksykograficznie sufiks podsłowa (czyli o k-ty elementy tablicy sufiksowej podsłowa) oraz o leksykograficzną rangę wskazanego sufiksu podsłowa (czyli element odwrotnej tablicy sufiksowej). Nasza struktura danych przetwarza zapytania obydwu rodzajów w√ czasie O(log n), zajmuje O(n) komórek pamięci i można ją zbudować w czasie O(n log n). Stworzone w rozprawie narzędzia dają także lepsze od znanych dotychczas rozwiązania problemu kompresji podsłów, czyli zapytań o reprezentację podsłowa według ustalonego algorytmu kompresji; rozważamy w tym kontekście metody oparte na parsowaniu Lempela-Ziva i transformacie Burrowsa-Wheelera. Rezultaty rozprawy łączą narzędzia do przetwarzania tekstów z kombinatoryką słów i najnowszymi osiągnięciami w dziedzinie abstrakcyjnych struktur danych. Naszą kluczową technikę stanowi innowacyjna metoda lokalnie zgodnego łamania symetrii między pozycjami tekstu, która stoi za nowymi rozwiązaniami dla zapytań o długość najdłuższego wspólnego prefiksu i dla wewnętrznego wyszukiwania wzorca. Tytuł rozprawy w języku polskim: Efektywne struktury danych dla zapytań wewnętrznych w tekstach Słowa kluczowe: struktury danych, najdłuższy wspólny prefiks, lokalna zgodność, najmniejszy sufiks, okres, kompresja podsłów, zapytania o podsłowa iii Acknowledgements First, I wish to thank Wojciech Rytter, my advisor, for introducing me to text algorithms and guidance over the course of my undergraduate and graduate studies. I am also indebted to Jakub Radoszewski, who was the first to expose me to research problems and since then has become my closest collaborator. My further gratitude goes to Tomasz Waleń, with whom we have created a successful team at the University of Warsaw. Special thanks must be made to Paweł Gawrychowski, to whom I owe some of the most interesting problems I have been working on, as well as several short research visits in Saarbrücken, Haifa, and Wrocław. I am also obliged to Tatiana Starikovskaya and Maxim Babenko, collaboration with whom on Minimal Suffix Queries inspired most of the directions I followed preparing this thesis. I particularly appreciate numerous conversations with Tatiana and her warm welcome during my short stays in Moscow. Listed above are the coauthors of the papers that this dissertation is based on. However, a few other people have also made a significant impact on its contents. I have greatly benefited from Ely Porat, who has invited me to Bar-Ilan Univer- sity and offered me an early access to his results on local consistency and LCE Queries [24]. I wish to thank Moshe Lewenstein for drawing my attention to Generalized LZ Substring Compression Queries, as well as Djamal Belaz- zougui, Travis Gagie, and Simon J. Puglisi, who suggested applications related to the Burrows–Wheeler transform. I am also grateful to Dominik Kempa for proofreading the previously unpublished Chapter 4 of
Load more

Recommended publications
  • The Book Review Column1 by Frederic Green Department of Mathematics and Computer Science Clark University Worcester, MA 02465 Email: [email protected]
    The Book Review Column1 by Frederic Green Department of Mathematics and Computer Science Clark University Worcester, MA 02465 email: [email protected] In this column, the following books are reviewed: 1. Games and Mathematics: Subtle Connections, by David Wells. Reviewed by S. C. Coutinho. This is a (mostly non-technical) book that explores the mathematical nature of games. In addition, and more extensively, it discusses the game-like nature of mathematics through a variety of mathematical examples. 2. Jewels of Stringology, by Maxime Crochemore and Wojciech Rytter. Reviewed by Shoshana Marcus. This is a book about string algorithms, with an emphasis on those that are fundamental and elegant: i.e., “algorithmic jewels.” Written at the advanced undergraduate/beginning graduate level. 3. Algorithms on Strings, by Maxime Crochemore, Christophe Hancart and Thierry Lecroq. Reviewed by Matthias Galle.´ Another book on stringology, aimed more at the graduate level. Although there is some overlap with “Jewels” (e.g., algorithms and data structures for pattern matching), the two books have a substantial symmetric difference. 4. Polyhedral and Algebraic Methods in Computational Geometry, by Michael Joswig and Thorsten Theobald. Reviewed by Brittany Terese Fasy and David L. Millman. This book treats a wide array of topics, beginning from the basics (e.g., convex hulls) and proceeding through the fundamental tools of computational algebraic geometry (e.g., Grobner¨ bases), and containing a section on applications as well. 5. A Mathematical Orchard - Problems and Solutions, by Mark Krusemeyer, George Gilbert, and Loren Larson. Reviewed by S.V. Nagaraj. This is a collection of challenging mathematical problems, published by the MAA.
    [Show full text]
  • Text Searching
    Text Searching Thierry Lecroq [email protected] Laboratoire d’Informatique, du Traitement de l’Information et des Syst`emes. International PhD School in Formal Languages and Applications Tarragona, November 20th and 21st, 2006 Introduction Left-to-right scan — shift Right-to-left scan — shift Plan 1 Introduction 2 Left-to-right scan — shift 3 Right-to-left scan — shift Thierry Lecroq Text Searching 2/77 Introduction Left-to-right scan — shift Right-to-left scan — shift Plan 1 Introduction 2 Left-to-right scan — shift 3 Right-to-left scan — shift Thierry Lecroq Text Searching 3/77 Introduction Left-to-right scan — shift Right-to-left scan — shift Pattern matching text pattern - position of occurrence Problem Find all the occurrences of pattern x of length m inside the text y of length n Two types of solutions Fixed pattern preprocessing time O(m) Use of combinatorial properties searching time O(n) Static text preprocessing time O(n) Solutions based on indexes searching time O(m) Thierry Lecroq Text Searching 4/77 Introduction Left-to-right scan — shift Right-to-left scan — shift Searching for a fixed string String matching given a pattern x, find all its locations in any text y Pattern a string x of symbols, of length m t a t a Text a string y of symbols, of length n c acgtatatatgcgttataat Thierry Lecroq Text Searching 5/77 Introduction Left-to-right scan — shift Right-to-left scan — shift String matching Occurrences c acgtatatatgcgttataat t a t a t a t a t a t a Basic operation symbol comparison (=, 6=) Thierry Lecroq Text Searching 6/77 Introduction Left-to-right scan — shift Right-to-left scan — shift Interest Practical basic problem for search for various patterns lexical analysis approximate string matching comparisons of strings—alignments ..
    [Show full text]
  • {PDF} Jewels of Stringology: Text Algorithms
    JEWELS OF STRINGOLOGY: TEXT ALGORITHMS PDF, EPUB, EBOOK Maxime Crochemore, Wojciech Rytter | 320 pages | 01 Jan 2003 | World Scientific Publishing Co Pte Ltd | 9789810248970 | English | Singapore, Singapore Jewels Of Stringology: Text Algorithms PDF Book DPReview Digital Photography. Tell the Publisher! Matching by dueling and sampling. Top reviews from India. If you like books and love to build cool products, we may be looking for you. If you expect that there will be ready to use code, look for O'Reilly catalog. Be the first to ask a question about Jewels of Stringology. Introduction to equations on words. Read more Read less. We are all precious Jewels of God and we are to be light and salt in a darkened world. Audible Download Audio Books. Wojciech Rytter. Certainly, string matching and data compression are in the former class, while most problems related to symmetries and repetitions in texts are in the latter. Warning: this is not to "copy and paste" book. See all reviews. Gilbert Strang. This book deals with the most basic algorithms in the area. Myers' algorithm for edit distance algorithm used in 'diff' program from UNIX. Namespaces Article Talk. Presents algorithms that are very hard to find elsewhere. Cambridge: Cambridge University Press, To ask other readers questions about Jewels of Stringology , please sign up. This book fills the gap in the book literature on algorithms on words, and brings together the many results presently dispersed in the masses of journal articles. From Wikipedia, the free encyclopedia. Thanks for telling us about the problem. Verified Purchase. View 14 excerpts, cites methods.
    [Show full text]
  • Stringology Ewels of Stringology This Page Is Intentionally Left Blank .^ X
    ; Maxime Crochemore Wojciech Rytter ewels of. Stringology ewels of Stringology This page is intentionally left blank .^ X ^,,.-^""'''"•'"' •• •>•» \ ( 1 :i y „.,.,-<•-•• Maxime Crochemore -• r: Universite Marne-la- Vallee, France ^ Wojciech Rytter Warsaw University, Poland & University of Liverpool, UK ewels °f. Stringoh World Scientific New Jersey * London • Singapore • Hong Kong Published by World Scientific Publishing Co. Pte. Ltd. P O Box 128, Fairer Road, Singapore 912805 USA office: Suite IB, 1060 Main Street, River Edge, NJ 07661 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library. JEWELS OF STRINGOLOGY Copyright © 2002 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher. For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher. ISBN 981-02-4782-6 This book is printed on acid-free paper. Printed in Singapore by Mainland Press Preface The term stringology is a popular nickname for string algorithms as well as for text algorithms. Usually text and string have the same meaning. More formally, a text is a sequence of symbols. Text is one of the basic data types to carry information.
    [Show full text]
  • IMPROVED ALGORITHMS for STRING SEARCHING PROBLEMS Doctoral Dissertation
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Aaltodoc TKK Research Reports in Computer Science and Engineering A TKK-CSE-A1/09 Espoo 2009 IMPROVED ALGORITHMS FOR STRING SEARCHING PROBLEMS Doctoral Dissertation Leena Salmela Dissertation for the degree of Doctor of Science in Technology to be presented with due permission of the Faculty of Information and Natural Sciences for public examination and debate in Auditorium T2 at Helsinki University of Technology (Espoo, Finland) on the 1st of June, 2009, at 12 noon. Helsinki University of Technology Faculty of Information and Natural Sciences Department of Computer Science and Engineering Teknillinen korkeakoulu Informaatio- ja luonnontieteiden tiedekunta Tietotekniikan laitos Distribution: Helsinki University of Technology Faculty of Information and Natural Sciences Department of Computer Science and Engineering P.O. Box 5400 FI-02015 TKK FINLAND URL: http://www.cse.tkk.fi/ Tel. +358 9 451 3228 Fax +358 9 451 3293 e-mail: [email protected].fi c Leena Salmela c Cover photo: Teemu J. Takanen ISBN 978-951-22-9887-7 ISBN 978-951-22-9888-4 (PDF) ISSN 1797-6928 ISSN 1797-6936 (PDF) URL: http://lib.tkk.fi/Diss/2009/isbn9789512298884/ Multiprint Oy Espoo 2009 AB ABSTRACT OF DOCTORAL DISSERTATION HELSINKI UNIVERSITY OF TECHNOLOGY P. O. BOX 1000, FI-02015 TKK http://www.tkk.fi/ Author Leena Salmela Name of the dissertation Improved Algorithms for String Searching Problems Manuscript submitted 09.02.2009 Manuscript revised 11.05.2009 Date of the defence 01.06.2009 X Monograph Article dissertation (summary + original articles) Faculty Faculty of Information and Natural Sciences Department Department of Computer Science and Engineering Field of research Software Systems Opponent(s) Prof.
    [Show full text]
  • Faster Recovery of Approximate Periods Over Edit Distance
    Faster Recovery of Approximate Periods over Edit Distance Tomasz Kociumaka, Jakub Radoszewski,∗ Wojciech Rytter,y Juliusz Straszy´nski,∗ Tomasz Wale´n,and Wiktor Zuba Faculty of Mathematics, Informatics and Mechanics, University of Warsaw, Warsaw, Poland [kociumaka,jrad,rytter,jks,walen,w.zuba]@mimuw.edu.pl Abstract The approximate period recovery problem asks to compute all approximate word-periods of a given word S of length n: all primitive words P (jP j = p) which have a periodic extension at edit n distance smaller than τp from S, where τp = b (3:75+)·p c for some > 0. Here, the set of periodic extensions of P consists of all finite prefixes of P 1. We improve the time complexity of the fastest known algorithm for this problem of Amir et al. [Theor. Comput. Sci., 2018] from O(n4=3) to O(n log n). Our tool is a fast algorithm for Approximate Pattern Matching in Periodic Text. We consider only verification for the period recovery problem when the candidate approximate word-period P is explicitly given up to cyclic rotation; the algorithm of Amir et al. reduces the general problem in O(n) time to a logarithmic number of such more specific instances. 1 Introduction The aim of this work is computing periods of words in the approximate pattern matching model (see e.g. [8, 10]). This task can be stated as the approximate period recovery (APR) problem that was defined by Amir et al. [2]. In this problem, we are given a word; we suspect that it was initially periodic, but then errors might have been introduced in it.
    [Show full text]
  • Proceedings of the Prague Stringology Conference 2014
    Proceedings of the Prague Stringology Conference 2014 Edited by Jan Holub and Jan Zdˇ '´arek September 2014 PSC Prague Stringology Club http://www.stringology.org/ Conference Organisation Program Committee Amihood Amir (Bar-Ilan University, Israel) Gabriela Andrejkov´a (P. J. Saf´arikUniversity,ˇ Slovakia) Maxime Crochemore (King's College London, United Kingdom) Simone Faro (Universit`adi Catania, Italy) FrantiˇsekFranˇek (McMaster University, Canada) Jan Holub, Co-chair (Czech Technical University in Prague, Czech Republic) Costas S. Iliopoulos (King's College London, United Kingdom) Shunsuke Inenaga (Kyushu University, Japan) Shmuel T. Klein (Bar-Ilan University, Israel) Thierry Lecroq, Co-chair (Universit´ede Rouen, France) Boˇrivoj Melichar, Honorary chair (Czech Technical University in Prague, Czech Republic) Yoan J. Pinz´on (Universidad Nacional de Colombia, Colombia) Marie-France Sagot (INRIA Rh^one-Alpes, France) William F. Smyth (McMaster University, Canada) Bruce W. Watson (FASTAR Group (Stellenbosch University and University of Pretoria, South Africa)) Jan Zdˇ '´arek (Czech Technical University in Prague, Czech Republic) Organising Committee Miroslav Bal´ık, Co-chair Jan Janouˇsek Boˇrivoj Melichar Jan Holub, Co-chair Jan Zdˇ '´arek External Referees J´er´emy Barbay Juan Mendivelso Elise Prieur-Gaston Loek Cleophas Yuto Nakashima Ayumi Shinohara Arnaud Lefebvre iii Preface The proceedings in your hands contains the papers presented in the Prague Stringol- ogy Conference 2014 (PSC 2014) at the Czech Technical University in Prague, which organizes the event. The conference was held on September 1{3, 2014 and it focused on stringology and related topics. Stringology is a discipline concerned with algorith- mic processing of strings and sequences. The papers submitted were reviewed by the program committee.
    [Show full text]