DOCSLIB.ORG

More Fuzzy String Matching!

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
More Fuzzy String Matching! More fuzzy string matching! Steven Bedrick CS/EE 5/655, 12/1/14 Plan for today: Tries Simple uses of tries Fuzzy search with tries Levenshtein automata A trie is essentially a prefix tree: A: 15 i: 11 in: 5 inn: 9 to: 7 tea: 3 ted: 4 ten: 12 Simple uses of tries: Key lookup in O(m) time, predictably. (Compare to hash table: best-case O(1), worst-case O(n), depending on key) Fast longest-prefix matching IP routing table lookup For an incoming packet, find the closest next hop in a routing table. Simple uses of tries: Fast longest-prefix matching Useful for autocompletion: “All words/names/whatevers that start with XYZ...” The problem with tries: When the space of keys is sparse, the trie is not very compact: (One) Solution: PATRICIA Tries (One) Solution: PATRICIA Tries Key ideas: edges represent more than a single symbol; nodes with only one child get collapsed. One could explicitly represent edges with multiple symbols... ... but that would complicate matching. Instead, each internal node stores the offset for the next difference to look for: 1 e s 3 5 s t e i 6 8 sublease 7 a c t e i e i essence essential estimate estimation sublimate sublime subliminal Instead, each internal node stores the offset for the next difference to look for: 1 e s 3 5 s t e i 6 8 sublease 7 a c t e i e i essence essential estimate estimation sublimate sublime subliminal e s t i m a t i o n i 1 2 3 4 5 6 7 8 9 10 Instead, each internal node stores the offset for the next difference to look for: 1 e s 3 5 s t e i 6 8 sublease 7 a c t e i e i essence essential estimate estimation sublimate sublime subliminal ↓ e s t i m a t i o n i 1 2 3 4 5 6 7 8 9 10 Instead, each internal node stores the offset for the next difference to look for: 1 e s 3 5 s t e i 6 8 sublease 7 a c t e i e i essence essential estimate estimation sublimate sublime subliminal ↓ e s t i m a t i o n i 1 2 3 4 5 6 7 8 9 10 Instead, each internal node stores the offset for the next difference to look for: 1 e s 3 5 s t e i 6 8 sublease 7 a c t e i e i essence essential estimate estimation sublimate sublime subliminal ↓ e s t i m a t i o n i 1 2 3 4 5 6 7 8 9 10 Instead, each internal node stores the offset for the next difference to look for: 1 e s 3 5 s t e i 6 8 sublease 7 a c t e i e i essence essential estimate estimation sublimate sublime subliminal ↓ e s t i m a t i o n i 1 2 3 4 5 6 7 8 9 10 Instead, each internal node stores the offset for the next difference to look for: 1 e s 3 5 s t e i 6 8 sublease 7 a c t e i e i essence essential estimate estimation sublimate sublime subliminal ↓ e s t i m a t i o n i 1 2 3 4 5 6 7 8 9 10 Instead, each internal node stores the offset for the next difference to look for: 1 e s 3 5 s t e i 6 8 sublease 7 a c t e i e i essence essential estimate estimation sublimate sublime subliminal ↓ e s t i m a t i o n i 1 2 3 4 5 6 7 8 9 10 Instead, each internal node stores the offset for the next difference to look for: 1 e s 3 5 s t e i 6 8 sublease 7 a c t e i e i essence essential estimate estimation sublimate sublime subliminal Plan for today: Tries Simple uses of tries Fuzzy search with tries Levenshtein automata Fuzzy search with tries Problem: we want to search a dictionary for words similar to a query. Example: “Smyth” and “Zmith” should retrieve “Smith”, “Levenstien” should retrieve “Levenshtein”, etc. By “similar,” we mean “edit distance less than some threshold δ.” One solution: Compute pairwise edit distance between our query q and every word wi in our dictionary; Match if sim(q, wi) ≤ δ A (slightly) better solution: Speed up pairwise edit distance computation using prefix pruning. Prefix pruning’s key idea: If we only care whether strings r and s have an edit distance less than some threshold... j 0 1 2 3 4 2. TRIE-BASED FRAMEWORK i e b a y In this section, we first formalize the problem of string 0 0 1 2 3 4 similarity joins with edit-distance constraints and then in- troduce a trie-based framework for efficient similarity joins. 1 k 1 1 2 3 4 2 o 2 2 2 3 4 2.1 Problem Formulation 3 b 3 3 2 3 4 Given two sets of strings, a similarity join finds all simi- lar string pairs from the two sets. In this paper, we use edit 4 y 4 4 3 3 3 distance to quantify the similarity between two strings. For- Figure 1: Prefix pruning. Matrix for computing edit mally, the edit distance between two strings r and s,denoted ...wedist ancecan do of twoearly strings termination “ebay” of and our “koby computation”. Shaded as as ed(r, s), is the minimum number of single-character edit sooncells we denote exceedactive that entries threshold.for τ =1. operations (i.e., insertion, deletion, and substitution) needed Wang J, Feng J, Li G. Trie-join: efficient trie-based string similarity joins with edit-distance constraints. VLDB Endowment; 2010. to transform r to s.Forexample,ed(koby, ebay)=3. In Figure 2 shows a trie structure of a sample data set with this paper two strings are similar if their edit distance is no six strings. String “ebay”hasatrienodeIDof12andits larger than a given edit-distance threshold τ.Weformalize prefix “eb”hasatrienodeIDof10.Forsimplicity,anode the problem of string similarity joins as follows. is mentioned interchangeably with its corresponding string Definition 1 (String Similarity Joins). Given two in later text. For example, both node “ko”andstring“ko” sets of strings R and S,andanedit-distancethresholdτ,a refer to node 14, and node 14 also refers to string “ko”. similarity join finds all similar string pairs ⟨r, s⟩∈R × S Given a trie node n,let|n| denote its depth (the depth of such that ed(r, s) ≤ τ. the root node is 0). For example, |“ko”| =2. 2.2 Prefix Pruning 0 One na¨ıve solution to address this problem is all-pair ver- A sample data set 1 9 13 ification, which enumerates all string pairs ⟨r, s⟩∈R × S b e k SID String and computes their edit distances. However, this solution 25 10 14 s is rather expensive. In fact, in most cases to check whether a e b o 1 bag two strings are similar, we need not compute the edit dis- s2 ebay 3 4 6 11 15 tance between the two complete strings. Instead we can do g y a a b s3 bay an early termination in the dynamic-programming compu- s4 kobe 7 12 16 17 tation as follows [15]. g y e y s5 koby Given two strings r = r1r2 ...rn and s = s1s2 ...sm,let 8 s6 beagy D denote a matrix with n +1 rowsand m +1 columns,and y D(i, j)betheeditdistancebetweentheprefixr1r2 ...ri and Figure 2: Trie index of a sample data set the prefix s1s2 ...sj .Weusethedynamic-programmingal- gorithm to compute the matrix: D(0,j)=j for 0 ≤ j ≤ n, Note that many strings with same prefixes share the same and D(i, j)=min(D(i − 1,j)+1,D(i, j − 1) + 1,D(i − 1,j− ancestor nodes on the trie structure. Based on this property, we can extend the idea of prefix pruning to prune a group 1) + θ)whereθ =0ifri = sj ;otherwiseθ =1. D(i, j) is called an active entry if D(i, j) ≤ τ. Figure 1 shows the of strings. Given a trie and a string s,noden in the trie is matrix to compute the edit distance between “ebay”and called an active node of string s if ed(s, n) ≤ τ.Ifn is not “koby”. The shaded cells (e.g., D(1, 1)) denote active en- an active node for every prefix of string s,thenallthestrings tries for τ = 1. (For all running examples in the remainder under n cannot be similar to s.Thereasonisthefollowing. of this paper, we assume τ =1.)Tocheckwhetherr = For any string with prefix n in the trie, say r,inthedynamic- “ebay”ands =“koby” are similar, we first compute the programming algorithm, we can take r as the row and s entries in row D(0, ∗)(onlythoseentriescircledbythebold as the column. As the row D(|n|, ∗) has no active entry, lines). As D(0, 0) and D(0, 1) are active entries, we compute r cannot be similar to s based on prefix pruning.Basedon the entries in row D(1, ∗). Similarly, we compute the entries this observation, we propose a new pruning technique, called in row D(2, ∗). We find that D(2, 1), D(2, 2) and D(2, 3) subtrie pruning:Givenatrieandastrings,tocomputethe are not active entries.
Load more

Recommended publications
  • Dictionary Look-Up Within Small Edit Distance
    Dictionary Lo okUp Within Small Edit Distance Ab dullah N Arslan and Omer Egeciog lu Department of Computer Science University of California Santa Barbara Santa Barbara CA USA farslanomergcsucsbedu Abstract Let W b e a dictionary consisting of n binary strings of length m each represented as a trie The usual dquery asks if there exists a string in W within Hamming distance d of a given binary query string q We present an algorithm to determine if there is a memb er in W within edit distance d of a given query string q of length m The metho d d+1 takes time O dm in the RAM mo del indep endent of n and requires O dm additional space Intro duction Let W b e a dictionary consisting of n binary strings of length m each A dquery asks if there exists a string in W within Hamming distance d of a given binary query string q Algorithms for answering dqueries eciently has b een a topic of interest for some time and have also b een studied as the approximate query and the approximate query retrieval problems in the literature The problem was originally p osed by Minsky and Pap ert in in which they asked if there is a data structure that supp orts fast dqueries The cases of small d and large d for this problem seem to require dierent techniques for their solutions The case when d is small was studied by Yao and Yao Dolev et al and Greene et al have made some progress when d is relatively large There are ecient algorithms only when d prop osed by Bro dal and Venkadesh Yao and Yao and Bro dal and Gasieniec The small d case has applications
    [Show full text]
  • Approximate String Matching with Reduced Alphabet
    Approximate String Matching with Reduced Alphabet Leena Salmela1 and Jorma Tarhio2 1 University of Helsinki, Department of Computer Science [email protected] 2 Aalto University Deptartment of Computer Science and Engineering [email protected] Abstract. We present a method to speed up approximate string match- ing by mapping the factual alphabet to a smaller alphabet. We apply the alphabet reduction scheme to a tuned version of the approximate Boyer– Moore algorithm utilizing the Four-Russians technique. Our experiments show that the alphabet reduction makes the algorithm faster. Especially in the k-mismatch case, the new variation is faster than earlier algorithms for English data with small values of k. 1 Introduction The approximate string matching problem is defined as follows. We have a pat- tern P [1...m]ofm characters drawn from an alphabet Σ of size σ,atextT [1...n] of n characters over the same alphabet, and an integer k. We need to find all such positions i of the text that the distance between the pattern and a sub- string of the text ending at that position is at most k.Inthek-difference problem the distance between two strings is the standard edit distance where substitu- tions, deletions, and insertions are allowed. The k-mismatch problem is a more restricted one using the Hamming distance where only substitutions are allowed. Among the most cited papers on approximate string matching are the classical articles [1,2] by Esko Ukkonen. Besides them he has studied this topic extensively [3,4,5,6,7,8,9,10,11].
    [Show full text]
  • Approximate Boyer-Moore String Matching
    APPROXIMATE BOYER-MOORE STRING MATCHING JORMA TARHIO AND ESKO UKKONEN University of Helsinki, Department of Computer Science Teollisuuskatu 23, SF-00510 Helsinki, Finland Draft Abstract. The Boyer-Moore idea applied in exact string matching is generalized to approximate string matching. Two versions of the problem are considered. The k mismatches problem is to find all approximate occurrences of a pattern string (length m) in a text string (length n) with at most k mis- matches. Our generalized Boyer-Moore algorithm is shown (under a mild 1 independence assumption) to solve the problem in expected time O(kn( + m – k k)) where c is the size of the alphabet. A related algorithm is developed for the c k differences problem where the task is to find all approximate occurrences of a pattern in a text with ≤ k differences (insertions, deletions, changes). Experimental evaluation of the algorithms is reported showing that the new algorithms are often significantly faster than the old ones. Both algorithms are functionally equivalent with the Horspool version of the Boyer-Moore algorithm when k = 0. Key words: String matching, edit distance, Boyer-Moore algorithm, k mismatches problem, k differences problem AMS (MOS) subject classifications: 68C05, 68C25, 68H05 Abbreviated title: Approximate Boyer-Moore Matching 2 1. Introduction The fastest known exact string matching algorithms are based on the Boyer- Moore idea [BoM77, KMP77]. Such algorithms are “sublinear” on the average in the sense that it is not necessary to check every symbol in the text. The larger is the alphabet and the longer is the pattern, the faster the algorithm works.
    [Show full text]
  • Problem Set 7 Solutions
    Introduction to Algorithms November 18, 2005 Massachusetts Institute of Technology 6.046J/18.410J Professors Erik D. Demaine and Charles E. Leiserson Handout 25 Problem Set 7 Solutions Problem 7-1. Edit distance In this problem you will write a program to compute edit distance. This problem is mandatory. Failure to turn in a solution will result in a serious and negative impact on your term grade! We advise you to start this programming assignment as soon as possible, because getting all the details right in a program can take longer than you think. Many word processors and keyword search engines have a spelling correction feature. If you type in a misspelled word x, the word processor or search engine can suggest a correction y. The correction y should be a word that is close to x. One way to measure the similarity in spelling between two text strings is by “edit distance.” The notion of edit distance is useful in other fields as well. For example, biologists use edit distance to characterize the similarity of DNA or protein sequences. The edit distance d(x; y) of two strings of text, x[1 : : m] and y[1 : : n], is defined to be the minimum possible cost of a sequence of “transformation operations” (defined below) that transforms string x[1 : : m] into string y[1 : : n].1 To define the effect of the transformation operations, we use an auxiliary string z[1 : : s] that holds the intermediate results. At the beginning of the transformation sequence, s = m and z[1 : : s] = x[1 : : m] (i.e., we start with string x[1 : : m]).
    [Show full text]
  • 3. Approximate String Matching
    3. Approximate String Matching Often in applications we want to search a text for something that is similar to the pattern but not necessarily exactly the same. To formalize this problem, we have to specify what does “similar” mean. This can be done by defining a similarity or a distance measure. A natural and popular distance measure for strings is the edit distance, also known as the Levenshtein distance. 109 Edit distance The edit distance ed(A, B) of two strings A and B is the minimum number of edit operations needed to change A into B. The allowed edit operations are: S Substitution of a single character with another character. I Insertion of a single character. D Deletion of a single character. Example 3.1: Let A = Lewensteinn and B = Levenshtein. Then ed(A, B) = 3. The set of edit operations can be described with an edit sequence: NNSNNNINNNND or with an alignment: Lewens-teinn Levenshtein- In the edit sequence, N means No edit. 110 There are many variations and extension of the edit distance, for example: Hamming distance allows only the subtitution operation. • Damerau–Levenshtein distance adds an edit operation: • T Transposition swaps two adjacent characters. With weighted edit distance, each operation has a cost or weight, • which can be other than one. Allow insertions and deletions (indels) of factors at a cost that is lower • than the sum of character indels. We will focus on the basic Levenshtein distance. Levenshtein distance has the following two useful properties, which are not shared by all variations (exercise): Levenshtein distance is a metric.
    [Show full text]
  • Boyer Moore Pattern Matching Algorithm Example
    Boyer Moore Pattern Matching Algorithm Example Sonny still digitizes morganatically while lentic Axel revving that Switzers. Dressier Lewis tabularise some worshipfulness and obscure his neckband so pyramidically! Supernatant Matty dollop judicially while Randie always sang his silva unsaddles Tuesdays, he secularised so liturgically. Pada aplikasi media geometrical formulas for pattern matching algorithm Fast pattern matching algorithm is matched. Wikitechy Founder, Author, International Speaker, and Job Consultant. We consider their famous example pass string matching Boyer-Moore algorithm First and introduce Horspool's Algorithm which connect a simpler example. The current position of the good suffix in the pattern is based on its last character. Algorithms for finding patterns in strings. By one match a matching algorithms tested, boyer moore string is matched character matches while still starts searching, distinctive features of texts. The time complexity of Nave Pattern Search method is Omn. In this account, we debate discuss Boyer Moore pattern searching algorithm. Seek, Plunnge and more. Fluid dynamicist at a path, which group a string matching characters we set for randomly chosen because with an alignment of search algorithm is going to different. Best searching algorithm coding algorithms. Asking for boyer moore algorithm of patterns, there are matched. For example given by using the examples above picture below to find matches should review the term information for example and linear in. Not even a single meal pass, when authorities do is have to easy for efficient in month day to joint life, car keys, books, pen, mobile charger and dock not. If it finds the trumpet, it immediately returns the index.
    [Show full text]
  • Approximate String Matching Using Compressed Suffix Arrays
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Elsevier - Publisher Connector Theoretical Computer Science 352 (2006) 240–249 www.elsevier.com/locate/tcs Approximate string matching using compressed suffix arraysଁ Trinh N.D. Huynha, Wing-Kai Honb, Tak-Wah Lamb, Wing-Kin Sunga,∗ aSchool of Computing, National University of Singapore, Singapore bDepartment of Computer Science and Information Systems, The University of Hong Kong, Hong Kong Received 17 August 2004; received in revised form 23 September 2005; accepted 9 November 2005 Communicated by A. Apostolico Abstract Let T be a text of length n and P be a pattern of length m, both strings over a fixed finite alphabet A. The k-difference (k-mismatch, respectively) problem is to find all occurrences of P in T that have edit distance (Hamming distance, respectively) at most k from P . In this paper we investigate a well-studied case in which T is fixed and preprocessed into an indexing data structure so that any pattern k k query can be answered faster. We give a solution using an O(n log n) bits indexing data structure with O(|A| m ·max(k, log n)+occ) query time, where occ is the number of occurrences. The best previous result requires O(n log n) bits indexing data structure and k k+ gives O(|A| m 2 + occ) query time. Our solution also allows us to exploit compressed suffix arrays to reduce the indexing space to O(n) bits, while increasing the query time by an O(log n) factor only.
    [Show full text]
  • Levenshtein Distance Based Information Retrieval Veena G, Jalaja G BNM Institute of Technology, Visvesvaraya Technological University
    International Journal of Scientific & Engineering Research, Volume 6, Issue 5, May-2015 112 ISSN 2229-5518 Levenshtein Distance based Information Retrieval Veena G, Jalaja G BNM Institute of Technology, Visvesvaraya Technological University Abstract— In today’s web based applications information retrieval is gaining popularity. There are many advances in information retrieval such as fuzzy search and proximity ranking. Fuzzy search retrieves relevant results containing words which are similar to query keywords. Even if there are few typographical errors in query keywords the system will retrieve relevant results. A query keyword can have many similar words, the words which are very similar to query keywords will be considered in fuzzy search. Ranking plays an important role in web search; user expects to see relevant documents in first few results. Proximity ranking is arranging search results based on the distance between query keywords. In current research information retrieval system is built to search contents of text files which have the feature of fuzzy search and proximity ranking. Indexing the contents of html or pdf files are performed for fast retrieval of search results. Combination of indexes like inverted index and trie index is used. Fuzzy search is implemented using Levenshtein’s Distance or edit distance concept. Proximity ranking is done using binning concept. Search engine system evaluation is done using average interpolated precision at eleven recall points i.e. at 0, 0.1, 0.2…..0.9, 1.0. Precision Recall graph is plotted to evaluate the system. Index Terms— stemming, stop words, fuzzy search, proximity ranking, dictionary, inverted index, trie index and binning.
    [Show full text]
  • Efficient Privacy-Preserving General Edit Distance and Beyond
    Efficient Privacy-Preserving General Edit Distance and Beyond Ruiyu Zhu Yan Huang Indiana University Indiana University Email: [email protected] Email: [email protected] Abstract—Edit distance is an important non-linear metric that secure computation protocols such as weighted edit distance, has many applications ranging from matching patient genomes Needleman-Wunsch, longest common subsequence (LCS), and to text-based intrusion detection. Depends on the application, heaviest common subsequence (HCS), using all existing ap- related string-comparison metrics, such as weighted edit distance, Needleman-Wunsch distance, longest common subsequences, and plicable optimizations including fixed-key hardware AES [12], heaviest common subsequences, can usually fit better than the [15], Half-Gate garbling [13], free-XOR technique [11]. We basic edit distance. When these metrics need to be calculated on report the performance of these protocols in the “Best Prior” sensitive input strings supplied by mutually distrustful parties, row of TableI, as well as in the performance charts of Figure4, it is more desirable but also more challenging to compute 5 in Section V-A and use them as baselines to evaluate our them in privacy-preserving ways. In this paper, we propose efficient secure computation protocols for private edit distance as new approach. Note that our baseline performance numbers well as several generalized applications including weighted edit are already much better than any generic protocols we can distance (with potentially content-dependent weights), longest find in the literature, simply because we have, for the first common subsequence, and heaviest common subsequence. Our time, applied the most recent optimizations (such as Half- protocols run 20+ times faster and use an order-of-magnitude Gates, efficient AESNI-based garbling, and highly customized less bandwidth than their best previous counterparts.
    [Show full text]
  • A New Edit Distance for Fuzzy Hashing Applications
    326 Int'l Conf. Security and Management | SAM'15 | A New Edit Distance for Fuzzy Hashing Applications V. Gayoso Martínez1, F. Hernández Álvarez1, L. Hernández Encinas1, and C. Sánchez Ávila2 1Information Processing and Cryptography (TIC), Institute of Physical and Information Technologies (ITEFI) Spanish National Research Council (CSIC), Madrid, Spain 2Telecommunication Engineering School (ETSIT), Polytechnic University of Madrid (UPM), Madrid, Spain Abstract— Similarity preserving hashing applications, also similarity of the signatures. In order to do that, ssdeep im- known as fuzzy hashing functions, help to analyse the content plements an edit distance algorithm based on the Damerau- of digital devices by performing a resemblance comparison Levenshtein distance between two strings [4], [5]. That edit between different files. In practice, the similarity matching distance function compares the two strings and counts the procedure is a two-step process, where first a signature minimum number of operations needed to transform one associated to the files under comparison is generated, and into the other, where the allowed operations are insertions, then a comparison of the signatures themselves is performed. deletions, and substitutions of a single character, and trans- Even though ssdeep is the best-known application in positions of two adjacent characters [6], [7]. this field, the edit distance algorithm that ssdeep uses for Even though the success of ssdeep is quite remarkable, performing the signature comparison is not well-suited for its edit distance implementation has important limitations certain scenarios. In this contribution we present a new edit that prevent ssdeep from generating a score that reflects distance algorithm that better reflects the similarity of two the percentage of the bigger file that is also present in strings, and that can be used by fuzzy hashing applications the smaller file, which is the definition of similarity better in order to improve their results.
    [Show full text]
  • The String Edit Distance Matching Problem with Moves
    The String Edit Distance Matching Problem with Moves GRAHAM CORMODE AT&T Labs–Research and S. MUTHUKRISHNAN Rutgers University The edit distance between two strings S and R is defined to be the minimum number of character inserts, deletes and changes needed to convert R to S. Given a text string t of length n, and a pattern string p of length m, informally, the string edit distance matching problem is to compute the smallest edit distance between p and substrings of t. We relax the problem so that (a) we allow an additional operation, namely, sub- string moves, and (b) we allow approximation of this string edit distance. Our result is a near linear time deterministic algorithm to produce a factor of O(log n log∗ n) approximation to the string edit distance with moves. This is the first known significantly subquadratic algorithm for a string edit distance problem in which the distance involves nontrivial alignments. Our results are obtained by embed- ding strings into L1 vector space using a simplified parsing technique we call Edit Sensitive Parsing (ESP). Categories and Subject Descriptors: F.2.0 [Analysis of Algorithms and Problem Complex- ity]: General General Terms: Algorithms, Theory Additional Key Words and Phrases: approximate pattern matching, data streams, edit distance, embedding, similarity search, string matching 1. INTRODUCTION String matching has a long history in computer science, dating back to the first compilers in the sixties and before. Text comparison now appears in all areas of the discipline, from compression and pattern matching to computational biology Author’s addres: G.
    [Show full text]
  • Analysis of Algorithms and Data Structures for Text Indexing
    ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ FAKULTÄT FÜR INFORMATIK ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ TECHNISCHE UNIVERSITÄT MÜNCHEN Lehrstuhl für Effiziente Algorithmen Analysis of Algorithms and Data Structures for Text Indexing Moritz G. Maaß ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ FAKULTÄT FÜR INFORMATIK ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ TECHNISCHE UNIVERSITÄT MÜNCHEN Lehrstuhl für Effiziente Algorithmen Analysis of Algorithms and Data Structures for Text Indexing Moritz G. Maaß Vollständiger Abdruck der von der Fakultät für Informatik der Technischen Universität München zur Erlangung des akademischen Grades eines Doktors der Naturwissenschaften (Dr. rer. nat.) genehmigten Dissertation. Vorsitzender: Univ.-Prof. Dr. Dr. h.c. mult. Wilfried Brauer Prüfer der Dissertation: 1. Univ.-Prof. Dr. Ernst W. Mayr 2. Prof. Robert Sedgewick, Ph.D. (Princeton University, New Jersey, USA) Die Dissertation wurde am 12. April 2005 bei der Technischen Universität München eingereicht und durch die Fakultät für Informatik am 26. Juni 2006 angenommen. Abstract Large amounts of textual data like document collections, DNA sequence data, or the Internet call for fast look-up methods that avoid searching the whole corpus. This is often accomplished using tree-based data structures for text indexing such as tries, PATRICIA trees, or suffix trees. We present and analyze improved algorithms and index data structures for exact and error-tolerant search. Affix trees are a data structure for exact indexing. They are a generalization of suffix trees, allowing a bidirectional search by extending a pattern to the left and to the right during retrieval. We present an algorithm that constructs affix trees on-line in both directions, i.e., by augmenting the underlying string in both directions. An amortized analysis yields that the algorithm has a linear-time worst-case complexity. A space efficient method for error-tolerant searching in a dictionary for a pattern allowing some mismatches can be implemented with a trie or a PATRICIA tree.
    [Show full text]