DOCSLIB.ORG

Efficient Trie-Based String Similarity Joins with Edit-Distance Constraints

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Efficient Trie-Based String Similarity Joins with Edit-Distance Constraints Trie•Join: Efficient Trie•based String Similarity Joins with Edit•Distance Constraints Jiannan Wang Jianhua Feng Guoliang Li yDepartment of Computer Science, Tsinghua National Laboratory for Information Science and Technology, Tsinghua University, Beijing 10084, China [email protected]; [email protected]; [email protected] ABSTRACT study string similarity joins with edit-distance constraints, A string similarity join finds similar pairs between two col- which, given two sets of strings, find all similar string pairs lections of strings. It is an essential operation in many ap- from each set, such that the edit distance between each plications, such as data integration and cleaning, and has string pair is within a given threshold. The string simi- attracted significant attention recently. In this paper, we larity join has many real applications, such as finding near study string similarity joins with edit-distance constraints. duplicated queries in query log mining and correlating two Existing methods usually employ a filter-and-refine frame- sets of data (e.g., people name, place name, address). work and have the following disadvantages: (1) They are Existing studies, such as Part-Enum [1], All-Pairs-Ed [2], inefficient for the data sets with short strings (the average Ed-Join [18], usually employ a filter-and-refine framework. string length is no larger than 30); (2) They involve large In the filter step, they generate signatures for each string indexes; (3) They are expensive to support dynamic update and use the signatures to generate candidate pairs. In the of data sets. To address these problems, we propose a novel refine step, they verify the candidate pairs and output the framework called trie-join, which can generate results effi- final results. However, these approaches have the follow- ciently with small indexes. We use a trie structure to index ing disadvantages. Firstly, they are inefficient for the data the strings and utilize the trie structure to efficiently find sets with short strings (the average string length is no larger the similar string pairs based on subtrie pruning. We de- than 30), since they cannot select high-quality signatures for vise efficient trie-join algorithms and pruning techniques to short strings and thus they may generate a large number of achieve high performance. Our method can be easily ex- candidate pairs which need to be further verified. Secondly, tended to support dynamic update of data sets efficiently. they cannot support dynamic update of data sets. For ex- Experimental results show that our algorithms outperform ample, Ed-Join and All-Pairs-Ed need to select signatures state-of-the-art methods by an order of magnitude on three with higher weights. The dynamic update may change the real data sets with short strings. weights of signatures. Thus the two methods need to rese- lect signatures, rebuild indexes, and rerun their algorithms from scratch. Thirdly, they involve large index sizes as there 1. INTRODUCTION could be large numbers of signatures. The similarity join is an essential operation in many ap- To address above-mentioned problems, in this paper we plications, such as data integration and cleaning, near du- propose a new trie-based framework for efficient string sim- plicate object detection and elimination, and collaborative ilarity joins with edit-distance constraints. In comparison filtering. Recently it has attracted significant attention in with the filter-and-refine framework, our approach can ef- both academic and industrial community. For example, ficiently generate all similar string pairs without the refine SSJoin [4] proposed by Microsoft has been used in the data step. We use a trie structure to index strings which needs debugger project. A similarity join between two sets of much smaller space than existing methods, as the trie struc- objects finds all similar object pairs from each set. For ture can share many common prefixes of strings. To avoid example, given two sets of strings R = fkobe; ebay;::: g repeated computation, we propose subtrie pruning and dual and S = fbag; koby;::: g. We want to find all similar pairs subtrie pruning to improve performance. We devise efficient hr; si 2 R × S, such as hkobe; kobyi. trie-join-based algorithms and three pruning techniques to Many similarity functions have been proposed to quantify achieve high performance. Our method can be easily ex- the similarity between two objects, such as jaccard similar- tended to support dynamic update of data sets. ity, cosine similarity, and edit distance. In this paper, we To summarize, in this paper, we make the following contri- butions: (1) We propose a trie-based framework for efficient string similarity joins with edit-distance constraints. (2) We devise efficient trie-join-based algorithms and develop Permission to make digital or hard copies of all or part of this work for pruning techniques to achieve high performance. (3) We ex- personal or classroom use is granted without fee provided that copies are tend our method to support dynamic update of data sets not made or distributed for profit or commercial advantage and that copies efficiently. (4) Experimental results show that our method bear this notice and the full citation on the first page. To copy otherwise, to achieves high performance and outperforms existing algo- republish, to post on servers or to redistribute to lists, requires prior specific rithms by an order of magnitude on data sets with short permission and/or a fee. Technical Report of Department of Computer Science and Technology, strings (the average string length is no larger than 30). Tsinghua University, Beijing, China.. 1 The rest of this paper is organized as follows. Section 2 j 0 1 2 3 4 introduces the trie-based framework. We propose efficient i e b a y trie-join algorithms in Section 3 and develop three pruning 0 0 1 2 3 4 techniques in Section 4. Section 5 presents an incremental 1 k 1 1 2 3 4 algorithm for string similarity joins and Section 6 discuss how to extend our algorithm for two different string sets. 2 o 2 2 2 3 4 Experimental results are provided in Section 7. We review 3 b 3 3 2 3 4 related work in Section 8 and make a conclusion in Section 9. 4 y 4 4 3 3 3 2. TRIE•BASED FRAMEWORK Figure 1: Prefix pruning. Matrix for computing edit In this section, we first formalize the problem of string distance of two strings \ebay" and \koby". Shaded similarity joins with edit-distance constraints and then in- cells denote active entries for τ = 1. troduce a trie-based framework for efficient similarity joins. 2.3 Our Observations 2.1 Problem Formulation Observation 1 - Subtrie Pruning: As there are a large Given two sets of strings, a similarity join finds all simi- number of strings in the two sets and many strings share lar string pairs from the two sets. In this paper, we use edit prefixes, we can extend prefix pruning to prune a group of distance to quantify the similarity between two strings. For- strings. We use a trie structure to index all strings. Trie mally, the edit distance between two strings r and s, denoted is a tree structure where each path from the root to a leaf ed as (r, s), is the minimum number of single-character edit represents a string in the data set and every node on the operations (i.e., insertion, deletion, and substitution) needed path has a label of a character in the string. For instance, ed to transform r to s. For example, (koby, ebay)=3. In Figure 2 shows a trie structure of a sample data set with this paper two strings are similar if their edit distance is no six strings. String \ebay" has a trie node ID of 12 and its larger than a given edit-distance threshold τ. We formalize prefix \eb" has a trie node ID of 10. For simplicity, a node the problem of string similarity joins as follows. is mentioned interchangeably with its corresponding string Definition 1 (String Similarity Joins). Given two in later text. For example, both node \ko" and string \ko" sets of strings R and S, and an edit-distance threshold τ, a refer to node 14, and node 14 also refers to string \ko". similarity join finds all similar string pairs hr; si 2 R × S Given a trie node n, let jnj denote its depth (the depth of such that ed(r; s) ≤ τ. the root node is 0). For example, j\ko"j = 2. 2.2 Prefix Pruning 0 One na¨ıve solution to address this problem is all-pair ver- A sample data set ification, which enumerates all string pairs hr; si 2 R × S 1 9 13 b e k SID String and computes their edit distances. However, this solution s is rather expensive. In fact, in most cases to check whether 2 5 10 14 1 bag a e b o s two strings are similar, we need not compute the edit dis- 2 ebay 3 4 11 15 s tance between the two complete strings. Instead we can do g y 6 a a b 3 bay an early termination in the dynamic-programming compu- s4 kobe tation as follows [15]. 7 12 16 17 s g y e y 5 koby Given two strings r = r1r2 : : : rn and s = s1s2 : : : sm, let 8 s6 beagy D denote a matrix with n + 1 rows and m + 1 columns, and y D(i; j) be the edit distance between the prefix r1r2 : : : ri and Figure 2: Trie index of a sample data set the prefix s1s2 : : : sj .
Load more

Recommended publications
  • The One-Way Communication Complexity of Hamming Distance
    THEORY OF COMPUTING, Volume 4 (2008), pp. 129–135 http://theoryofcomputing.org The One-Way Communication Complexity of Hamming Distance T. S. Jayram Ravi Kumar∗ D. Sivakumar∗ Received: January 23, 2008; revised: September 12, 2008; published: October 11, 2008. Abstract: Consider the following version of the Hamming distance problem for ±1 vec- n √ n √ tors of length n: the promise is that the distance is either at least 2 + n or at most 2 − n, and the goal is to find out which of these two cases occurs. Woodruff (Proc. ACM-SIAM Symposium on Discrete Algorithms, 2004) gave a linear lower bound for the randomized one-way communication complexity of this problem. In this note we give a simple proof of this result. Our proof uses a simple reduction from the indexing problem and avoids the VC-dimension arguments used in the previous paper. As shown by Woodruff (loc. cit.), this implies an Ω(1/ε2)-space lower bound for approximating frequency moments within a factor 1 + ε in the data stream model. ACM Classification: F.2.2 AMS Classification: 68Q25 Key words and phrases: One-way communication complexity, indexing, Hamming distance 1 Introduction The Hamming distance H(x,y) between two vectors x and y is defined to be the number of positions i such that xi 6= yi. Let GapHD denote the following promise problem: the input consists of ±1 vectors n √ n √ x and y of length n, together with the promise that either H(x,y) ≤ 2 − n or H(x,y) ≥ 2 + n, and the goal is to find out which of these two cases occurs.
    [Show full text]
  • Fault Tolerant Computing CS 530 Information Redundancy: Coding Theory
    Fault Tolerant Computing CS 530 Information redundancy: Coding theory Yashwant K. Malaiya Colorado State University April 3, 2018 1 Information redundancy: Outline • Using a parity bit • Codes & code words • Hamming distance . Error detection capability . Error correction capability • Parity check codes and ECC systems • Cyclic codes . Polynomial division and LFSRs 4/3/2018 Fault Tolerant Computing ©YKM 2 Redundancy at the Bit level • Errors can bits to be flipped during transmission or storage. • An extra parity bit can detect if a bit in the word has flipped. • Some errors an be corrected if there is enough redundancy such that the correct word can be guessed. • Tukey: “bit” 1948 • Hamming codes: 1950s • Teletype, ASCII: 1960: 7+1 Parity bit • Codes are widely used. 304,805 letters in the Torah Redundancy at the Bit level Even/odd parity (1) • Errors can bits to be flipped during transmission/storage. • Even/odd parity: . is basic method for detecting if one bit (or an odd number of bits) has been switched by accident. • Odd parity: . The number of 1-bit must add up to an odd number • Even parity: . The number of 1-bit must add up to an even number Even/odd parity (2) • The it is known which parity it is being used. • If it uses an even parity: . If the number of of 1-bit add up to an odd number then it knows there was an error: • If it uses an odd: . If the number of of 1-bit add up to an even number then it knows there was an error: • However, If an even number of 1-bit is flipped the parity will still be the same.
    [Show full text]
  • Sequence Distance Embeddings
    Sequence Distance Embeddings by Graham Cormode Thesis Submitted to the University of Warwick for the degree of Doctor of Philosophy Computer Science January 2003 Contents List of Figures vi Acknowledgments viii Declarations ix Abstract xi Abbreviations xii Chapter 1 Starting 1 1.1 Sequence Distances . ....................................... 2 1.1.1 Metrics ............................................ 3 1.1.2 Editing Distances ...................................... 3 1.1.3 Embeddings . ....................................... 3 1.2 Sets and Vectors ........................................... 4 1.2.1 Set Difference and Set Union . .............................. 4 1.2.2 Symmetric Difference . .................................. 5 1.2.3 Intersection Size ...................................... 5 1.2.4 Vector Norms . ....................................... 5 1.3 Permutations ............................................ 6 1.3.1 Reversal Distance ...................................... 7 1.3.2 Transposition Distance . .................................. 7 1.3.3 Swap Distance ....................................... 8 1.3.4 Permutation Edit Distance . .............................. 8 1.3.5 Reversals, Indels, Transpositions, Edits (RITE) . .................... 9 1.4 Strings ................................................ 9 1.4.1 Hamming Distance . .................................. 10 1.4.2 Edit Distance . ....................................... 10 1.4.3 Block Edit Distances . .................................. 11 1.5 Sequence Distance Problems . .................................
    [Show full text]
  • Dictionary Look-Up Within Small Edit Distance
    Dictionary Lo okUp Within Small Edit Distance Ab dullah N Arslan and Omer Egeciog lu Department of Computer Science University of California Santa Barbara Santa Barbara CA USA farslanomergcsucsbedu Abstract Let W b e a dictionary consisting of n binary strings of length m each represented as a trie The usual dquery asks if there exists a string in W within Hamming distance d of a given binary query string q We present an algorithm to determine if there is a memb er in W within edit distance d of a given query string q of length m The metho d d+1 takes time O dm in the RAM mo del indep endent of n and requires O dm additional space Intro duction Let W b e a dictionary consisting of n binary strings of length m each A dquery asks if there exists a string in W within Hamming distance d of a given binary query string q Algorithms for answering dqueries eciently has b een a topic of interest for some time and have also b een studied as the approximate query and the approximate query retrieval problems in the literature The problem was originally p osed by Minsky and Pap ert in in which they asked if there is a data structure that supp orts fast dqueries The cases of small d and large d for this problem seem to require dierent techniques for their solutions The case when d is small was studied by Yao and Yao Dolev et al and Greene et al have made some progress when d is relatively large There are ecient algorithms only when d prop osed by Bro dal and Venkadesh Yao and Yao and Bro dal and Gasieniec The small d case has applications
    [Show full text]
  • Approximate String Matching with Reduced Alphabet
    Approximate String Matching with Reduced Alphabet Leena Salmela1 and Jorma Tarhio2 1 University of Helsinki, Department of Computer Science [email protected] 2 Aalto University Deptartment of Computer Science and Engineering [email protected] Abstract. We present a method to speed up approximate string match- ing by mapping the factual alphabet to a smaller alphabet. We apply the alphabet reduction scheme to a tuned version of the approximate Boyer– Moore algorithm utilizing the Four-Russians technique. Our experiments show that the alphabet reduction makes the algorithm faster. Especially in the k-mismatch case, the new variation is faster than earlier algorithms for English data with small values of k. 1 Introduction The approximate string matching problem is defined as follows. We have a pat- tern P [1...m]ofm characters drawn from an alphabet Σ of size σ,atextT [1...n] of n characters over the same alphabet, and an integer k. We need to find all such positions i of the text that the distance between the pattern and a sub- string of the text ending at that position is at most k.Inthek-difference problem the distance between two strings is the standard edit distance where substitu- tions, deletions, and insertions are allowed. The k-mismatch problem is a more restricted one using the Hamming distance where only substitutions are allowed. Among the most cited papers on approximate string matching are the classical articles [1,2] by Esko Ukkonen. Besides them he has studied this topic extensively [3,4,5,6,7,8,9,10,11].
    [Show full text]
  • Searching All Seeds of Strings with Hamming Distance Using Finite Automata
    Proceedings of the International MultiConference of Engineers and Computer Scientists 2009 Vol I IMECS 2009, March 18 - 20, 2009, Hong Kong Searching All Seeds of Strings with Hamming Distance using Finite Automata Ondřej Guth, Bořivoj Melichar ∗ Abstract—Seed is a type of a regularity of strings. Finite automata provide common formalism for many al- A restricted approximate seed w of string T is a fac- gorithms in the area of text processing (stringology), in- tor of T such that w covers a superstring of T under volving forward exact and approximate pattern match- some distance rule. In this paper, the problem of all ing and searching for borders, periods, and repetitions restricted seeds with the smallest Hamming distance [7], backward pattern matching [8], pattern matching in is studied and a polynomial time and space algorithm a compressed text [9], the longest common subsequence for solving the problem is presented. It searches for [10], exact and approximate 2D pattern matching [11], all restricted approximate seeds of a string with given limited approximation using Hamming distance and and already mentioned computing approximate covers it computes the smallest distance for each found seed. [5, 6] and exact covers [12] and seeds [2] in generalized The solution is based on a finite (suffix) automata ap- strings. Therefore, we would like to further extend the set proach that provides a straightforward way to design of problems solved using finite automata. Such a problem algorithms to many problems in stringology. There- is studied in this paper. fore, it is shown that the set of problems solvable using finite automata includes the one studied in this Finite automaton as a data structure may be easily imple- paper.
    [Show full text]
  • Approximate Boyer-Moore String Matching
    APPROXIMATE BOYER-MOORE STRING MATCHING JORMA TARHIO AND ESKO UKKONEN University of Helsinki, Department of Computer Science Teollisuuskatu 23, SF-00510 Helsinki, Finland Draft Abstract. The Boyer-Moore idea applied in exact string matching is generalized to approximate string matching. Two versions of the problem are considered. The k mismatches problem is to find all approximate occurrences of a pattern string (length m) in a text string (length n) with at most k mis- matches. Our generalized Boyer-Moore algorithm is shown (under a mild 1 independence assumption) to solve the problem in expected time O(kn( + m – k k)) where c is the size of the alphabet. A related algorithm is developed for the c k differences problem where the task is to find all approximate occurrences of a pattern in a text with ≤ k differences (insertions, deletions, changes). Experimental evaluation of the algorithms is reported showing that the new algorithms are often significantly faster than the old ones. Both algorithms are functionally equivalent with the Horspool version of the Boyer-Moore algorithm when k = 0. Key words: String matching, edit distance, Boyer-Moore algorithm, k mismatches problem, k differences problem AMS (MOS) subject classifications: 68C05, 68C25, 68H05 Abbreviated title: Approximate Boyer-Moore Matching 2 1. Introduction The fastest known exact string matching algorithms are based on the Boyer- Moore idea [BoM77, KMP77]. Such algorithms are “sublinear” on the average in the sense that it is not necessary to check every symbol in the text. The larger is the alphabet and the longer is the pattern, the faster the algorithm works.
    [Show full text]
  • B-Bit Sketch Trie: Scalable Similarity Search on Integer Sketches
    b-Bit Sketch Trie: Scalable Similarity Search on Integer Sketches Shunsuke Kanda Yasuo Tabei RIKEN Center for Advanced Intelligence Project RIKEN Center for Advanced Intelligence Project Tokyo, Japan Tokyo, Japan [email protected] [email protected] Abstract—Recently, randomly mapping vectorial data to algorithms intending to build sketches of non-negative inte- strings of discrete symbols (i.e., sketches) for fast and space- gers (i.e., b-bit sketches) have been proposed for efficiently efficient similarity searches has become popular. Such random approximating various similarity measures. Examples are b-bit mapping is called similarity-preserving hashing and approximates a similarity metric by using the Hamming distance. Although minwise hashing (minhash) [12]–[14] for Jaccard similarity, many efficient similarity searches have been proposed, most of 0-bit consistent weighted sampling (CWS) for min-max ker- them are designed for binary sketches. Similarity searches on nel [15], and 0-bit CWS for generalized min-max kernel [16]. integer sketches are in their infancy. In this paper, we present Thus, developing scalable similarity search methods for b-bit a novel space-efficient trie named b-bit sketch trie on integer sketches is a key issue in large-scale applications of similarity sketches for scalable similarity searches by leveraging the idea behind succinct data structures (i.e., space-efficient data structures search. while supporting various data operations in the compressed Similarity searches on binary sketches are classified
    [Show full text]
  • Problem Set 7 Solutions
    Introduction to Algorithms November 18, 2005 Massachusetts Institute of Technology 6.046J/18.410J Professors Erik D. Demaine and Charles E. Leiserson Handout 25 Problem Set 7 Solutions Problem 7-1. Edit distance In this problem you will write a program to compute edit distance. This problem is mandatory. Failure to turn in a solution will result in a serious and negative impact on your term grade! We advise you to start this programming assignment as soon as possible, because getting all the details right in a program can take longer than you think. Many word processors and keyword search engines have a spelling correction feature. If you type in a misspelled word x, the word processor or search engine can suggest a correction y. The correction y should be a word that is close to x. One way to measure the similarity in spelling between two text strings is by “edit distance.” The notion of edit distance is useful in other fields as well. For example, biologists use edit distance to characterize the similarity of DNA or protein sequences. The edit distance d(x; y) of two strings of text, x[1 : : m] and y[1 : : n], is defined to be the minimum possible cost of a sequence of “transformation operations” (defined below) that transforms string x[1 : : m] into string y[1 : : n].1 To define the effect of the transformation operations, we use an auxiliary string z[1 : : s] that holds the intermediate results. At the beginning of the transformation sequence, s = m and z[1 : : s] = x[1 : : m] (i.e., we start with string x[1 : : m]).
    [Show full text]
  • Large Scale Hamming Distance Query Processing
    Large Scale Hamming Distance Query Processing Alex X. Liu, Ke Shen, Eric Torng Department of Computer Science and Engineering Michigan State University East Lansing, MI 48824, U.S.A. {alexliu, shenke, torng}@cse.msu.edu Abstract—Hamming distance has been widely used in many a new solution to the Hamming distance range query problem. application domains, such as near-duplicate detection and pattern The basic idea is to hash each document into a 64-bit binary recognition. We study Hamming distance range query problems, string, called a fingerprint or a sketch, using the simhash where the goal is to find all strings in a database that are within a Hamming distance bound k from a query string. If k is fixed, algorithm in [11]. Simhash has the following property: if two we have a static Hamming distance range query problem. If k is documents are near-duplicates in terms of content, then the part of the input, we have a dynamic Hamming distance range Hamming distance between their fingerprints will be small. query problem. For the static problem, the prior art uses lots of Similarly, Miller et al. proposed a scheme for finding a song memory due to its aggressive replication of the database. For the that includes a given audio clip [20], [21]. They proposed dynamic range query problem, as far as we know, there is no space and time efficient solution for arbitrary databases. In this converting the audio clip to a binary string (and doing the paper, we first propose a static Hamming distance range query same for all the songs in the given database) and then finding algorithm called HEngines, which addresses the space issue in the nearest neighbors to the given binary string in the database prior art by dynamically expanding the query on the fly.
    [Show full text]
  • Parity Check Code Repetition Code 10010011 11100001
    Lecture Notes Charles Cusack Communication Errors Introduction Implementations of memory, storage, and communication are not perfect. Therefore, it is possible that when I store a number on a hard drive or send it over a network, a slightly different number is actually stored/received. There are techniques to deal with this. Definition An error detection (correction) code is a method of encoding data in so that if a certain number of errors has occurred during transmission, it can be detected (corrected). Definition The hamming distance between two bit strings is the number of bits where they differ. Example The hamming distance between 10010011 and 11100001 is 4 since 1001 00 11 these bit patterns differ in bits 2, 3, 4, and 7. 1110 00 01 Parity Check Code The simplest method of detecting errors is by using a parity bit . To use a parity bit, one simply appends one additional bit (to the beginning or end—it doesn’t matter which as long as you are consistent) to a bit string so that the number of 1s in the string is odd (or even). To detect an error you add the number of ‘1’ bits. If it is not odd (even) you know an error has occurred. Example The bit string 10010011 is encoded as 100100111 using an odd parity check code, and the bit string 11110001 is encoded as 111100010 using an odd parity check code. Question • How many errors can this method detect? What happens if more errors occur? • How many errors can this method correct? Repetition Code A repetition code encodes a bit string by appending k copies of the bit string.
    [Show full text]
  • Bit-Parallel Approximate String Matching Under Hamming Distance
    Aalto University School of Science Degree Programme in Computer Science and Engineering Tommi Hirvola Bit-parallel approximate string matching under Hamming distance Master's Thesis Espoo, July 21, 2016 Supervisors: Professor Jorma Tarhio Advisor: Professor Jorma Tarhio Aalto University School of Science ABSTRACT OF Degree Programme in Computer Science and Engineering MASTER'S THESIS Author: Tommi Hirvola Title: Bit-parallel approximate string matching under Hamming distance Date: July 21, 2016 Pages: 87 Major: Software engineering Code: T-106 Supervisors: Professor Jorma Tarhio Advisor: Professor Jorma Tarhio In many situations, an exact search of a pattern in a text does not find all in- teresting parts of the text. For example, exact matching can be rendered useless by spelling mistakes or other errors in the text or pattern. Approximate string matching addresses the problem by allowing a certain amount of error in occur- rences of the pattern in the text. This thesis considers the approximate matching problem known as string matching with k mismatches, in which k characters are allowed to mismatch. The main contribution of the thesis consists of new algorithms for approximate string matching with k mismatches based on the bit-parallelism technique. Ex- periments show that the proposed algorithms are competitive with the state-of- the-art algorithms in practice. A significant advantage of the novel algorithms is algorithmic simplicity and flexibility, which is demonstrated by deriving efficient algorithms to several problems involving approximate string matching under the k-mismatches model. A bit-parallel algorithm is presented for online approximate matching of circular strings with k mismatches, which is the first average-optimal solution that is not based on filtering.
    [Show full text]