DOCSLIB.ORG

Constraint Clustering and Parity Games

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Constraint Clustering and Parity Games Constrained Clustering Problems and Parity Games Clemens Rösner geboren in Ulm Dissertation zur Erlangung des Doktorgrades (Dr. rer. nat.) der Mathematisch-Naturwissenschaftlichen Fakultät der Rheinischen Friedrich-Wilhelms-Universität Bonn Bonn 2019 1. Gutachter: Prof. Dr. Heiko Röglin 2. Gutachterin: Prof. Dr. Anne Driemel Tag der mündlichen Prüfung: 05. September 2019 Erscheinungsjahr: 2019 Angefertigt mit Genehmigung der Mathematisch-Naturwissenschaftlichen Fakultät der Rheinischen Friedrich-Wilhelms-Universität Bonn Abstract Clustering is a fundamental tool in data mining. It partitions points into groups (clusters) and may be used to make decisions for each point based on its group. We study several clustering objectives. We begin with studying the Euclidean k-center problem. The k-center problem is a classical combinatorial optimization problem which asks to select k centers and assign each input point in a set P to one of the centers, such that the maximum distance of any input point to its assigned center is minimized. The Euclidean k-center problem assumes that the input set P is a subset of a Euclidean space and that each location in the Euclidean space can be chosen as a center. We focus on the special case with k = 1, the smallest enclosing ball problem: given a set of points in m-dimensional Euclidean space, find the smallest sphere enclosing all the points. We combine known results about convex optimization with structural properties of the smallest enclosing ball to create a new algorithm. We show that on instances with rational coefficients our new algorithm computes the exact center of the optimal solutions and has a worst-case run time that is polynomial in the size of the input. We use the new algorithm to show that we can solve the Euclidean k-center problem in polynomial time for constant k and dimension m. The general unconstrained clustering problems are mostly very well studied. The k-center problem for example allows for elegant 2-approximation algorithms [47, 53]. However, the situation becomes significantly more difficult when constraints are added to the problem. We first look at the fair clustering. The fairness constraint is motivated by the fact that the general process of computing a clustering may harm protected (minority) classes if the clustering algorithm does not adequately represent them in desirable clusters – especially if the data is already biased. At NIPS 2017, Chierichetti et al. [29] proposed a model for fair clustering requiring the representation in each cluster to (approximately) preserve the global fraction of each protected class. Restricting to two protected classes, they developed both a 4-approximation algorithm for the fair k-center problem and an O(t)-approximation algorithm for the fair k-median problem, where t is a parameter for the fairness model. For multiple protected classes, the best known result is a 14-approximation algorithm for fair k-center [90]. We extend and improve the known results. Firstly, we give a 5-approximation algorithm for the fair k-center problem with multiple protected classes. Secondly, we propose a relaxed fairness notion under which we can give bicriteria constant-factor approximation algorithms for the fair version of all of the classical clustering objectives (k-center, k- supplier, k-median, k-means and facility location). The latter approximation algorithms are achieved by a framework that takes an arbitrary existing unfair (integral) solution and a fair (fractional) LP solution and combines them into an essentially fair clustering with a weakly supervised rounding scheme. In this way, a fair clustering can be established belatedly, in a situation where for example the centers are already fixed. ii The second clustering constraint we study is privacy: Here, we are asked to only open a center when at least ` points will be assigned to it. We raise the question whether a general method can be derived to turn an approximation algorithm for a clustering problem with some constraints into an approximation algorithm that additionally respects privacy. We show how to combine privacy with several other constraints and obtain approximation algorithms for the k-center problem with several combinations of constraints. In this dissertation we also study parity games, a two player game played on a directed graph. We study the case in which one of the two players controls only a small number k of nodes and the other player controls the n − k other nodes of the game. Our main result is√ a fixed-parameter-tractable algorithm that solves bipartite√ parity games in time kO( k) · O(n3), and general parity games in time (p + k)O( k) · O(pnm), where p is the number of distinct priorities and m is the number of edges. For all games with k = o(n) this improves the previously fastest algorithm by Jurdziński, Paterson, and Zwick [64]. We also obtain novel kernelization results and an improved deterministic algorithm for parity games on graphs with small average node-degree. Zusammenfassung Clustering ist ein grundlegendes Werkzeug im Data Mining. Es unterteilt Punkte in Gruppen (Cluster) und kann verwendet werden, um Entscheidungen für jeden Punkt basierend auf seiner Gruppe zu treffen. Wir untersuchen verschiedene Clustering Modelle. Als erstes untersuchen wir das euklidische k-Center Problem. Das k-Center Problem ist ein klassisches kombinatorisches Optimierungsproblem, bei dem wir k Zentren auswählen und jeden Punkt aus der Eingabemenge P einem der Zentren zuweisen sollen, sodass die maximale Entfernung eines Punktes zu seinem zugewiesenen Zentrum minimiert wird. Das euklidische k-Center Problem geht davon aus, dass die Eingabemenge P eine Teilmenge eines euklidischen Raums ist und dass jeder mögliche Ort im euklidischen Raum als Zentrum gewählt werden darf. Wir konzentrieren uns auf den Sonderfall mit k = 1, das Smallest Enclosing Ball Problem: Gegeben sei eine Menge von Punkten im m-dimensionalen euklidischen Raum, finde die kleinste Kugel, die alle diese Punkte enthält. Wir kombinieren bekannte Ergebnisse der konvexen Optimierung mit strukturellen Eigenschaften des Smallest Enclosing Balls, um einen neuen Algorithmus zu entwerfen. Wir zeigen, dass unser neuer Algorithmus bei Instanzen mit rationalen Koeffizienten das exakte Zentrum der optimalen Lösungen berechnet und eine Worst-Case-Laufzeit aufweist, die polynomiell in der Eingabegröße ist. Wir verwenden den neuen Algorithmus, um zu zeigen, dass wir das euklidische k-Center Problem in Polynomialzeit für konstante k und Dimension m lösen können. Ohne Nebenbedingungen sind die allgemeinen Clustering-Probleme zumeist sehr gut untersucht. Für das k-Center Problem gibt es beispielsweise elegante 2-Approxima- tionsalgorithmen [47, 53]. Die Situation wird jedoch erheblich schwieriger, wenn wir fordern, dass zusätzliche Nebenbedingungen erfüllt werden. Als erstes betrachten wir faires Clustering. Die Fairness-Nebenbedingung wird durch die Tatsache motiviert, dass die allgemeine Berechnung guter Clusterings geschützte (Minderheits-) Klassen schädigen kann, wenn das Ergebnis sie nicht in den bevorzugten Clustern angemessen repräsentiert, insbesondere wenn die Daten bereits voreingenommen sind. Auf der NIPS 2017 haben Chierichetti et al. [29] ein Modell für faires Clustering vorgeschlagen, das einfordert, dass die Repräsentation jedes Clusters (ungefähr) den globalen Anteil jeder geschützten Klasse widerspiegelt. Sie beschränkten sich auf zwei geschützte Klassen und entwickelten sowohl einen 4-Approximationsalgorithmus für das faire k-Center Problem als auch einen O(t)-Approximationsalgorithmus für das faire k-Median Problem, wobei t ein Parameter des Fairness-Modells ist. Bei mehreren geschützten Klassen ist das beste bekannte Ergebnis ein 14-Approximationsalgorithmus für das faire k-Center Problem [90]. Wir erweitern und verbessern die bekannten Ergebnisse. Zum einen geben wir einen 5-Approximationsalgorithmus für das faire k-Center Problem mit mehreren geschützten Klassen an. Zum anderen schlagen wir eine abgeschwächte Fairness Bedingung vor, iv unter der wir für die faire Version von allen klassischen Clustering-Problemen (k- Center, k-Supplier, k-Median, k-Means und Facility Location) einen bikriteriellen Approximationsalgorithmus mit konstantem Approximationsfaktor angeben können. Die letzteren Approximationsalgorithmen werden durch ein Rahmenwerk erreicht, das eine beliebige gegebene unfaire (ganzzahlige) Lösung und eine faire (fraktionelle) LP- Lösung verwendet und diese mittels eines schwach überwachten Rundungsschema zu einem im Wesentlichen fairen Clustering kombiniert. Auf diese Weise kann Fairness nachträglich zu einem Clustering, bei dem zum Beispiel die Zentren bereits festgelegt sind, hinzugefügt werden. Die zweite untersuchte Nebenbedingung für Clustering-Probleme ist Privacy: Hier werden wir aufgefordert, ein Zentrum nur dann zu öffnen, wenn ihm mindestens ` Punkte zugewiesen werden. Wir stellen die Frage, ob es eine allgemeine Methode gibt, um einen Approximationsalgorithmus für ein Clustering Problem mit einigen Nebenbedingungen in einen Approximationsalgorithmus umzuwandeln, der zusätzlich Privacy erfüllt. Wir zeigen, wie man Privacy mit mehreren anderen Nebenbedingungen kombinieren kann und erhalten Approximationsalgorithmen für das k-Center Problem mit mehreren Kombinationen von Nebenbedingungen. In dieser Dissertation untersuchen wir auch Paritätsspiele, ein 2-Spieler-Spiel auf einem gerichteten Graph. Wir untersuchen die eingeschränkte Variante, bei der einer der beiden Spieler nur eine kleine Anzahl
Load more

Recommended publications
  • FOCS 2005 Program SUNDAY October 23, 2005
    FOCS 2005 Program SUNDAY October 23, 2005 Talks in Grand Ballroom, 17th floor Session 1: 8:50am – 10:10am Chair: Eva´ Tardos 8:50 Agnostically Learning Halfspaces Adam Kalai, Adam Klivans, Yishay Mansour and Rocco Servedio 9:10 Noise stability of functions with low influences: invari- ance and optimality The 46th Annual IEEE Symposium on Elchanan Mossel, Ryan O’Donnell and Krzysztof Foundations of Computer Science Oleszkiewicz October 22-25, 2005 Omni William Penn Hotel, 9:30 Every decision tree has an influential variable Pittsburgh, PA Ryan O’Donnell, Michael Saks, Oded Schramm and Rocco Servedio Sponsored by the IEEE Computer Society Technical Committee on Mathematical Foundations of Computing 9:50 Lower Bounds for the Noisy Broadcast Problem In cooperation with ACM SIGACT Navin Goyal, Guy Kindler and Michael Saks Break 10:10am – 10:30am FOCS ’05 gratefully acknowledges financial support from Microsoft Research, Yahoo! Research, and the CMU Aladdin center Session 2: 10:30am – 12:10pm Chair: Satish Rao SATURDAY October 22, 2005 10:30 The Unique Games Conjecture, Integrality Gap for Cut Problems and Embeddability of Negative Type Metrics Tutorials held at CMU University Center into `1 [Best paper award] Reception at Omni William Penn Hotel, Monongahela Room, Subhash Khot and Nisheeth Vishnoi 17th floor 10:50 The Closest Substring problem with small distances Tutorial 1: 1:30pm – 3:30pm Daniel Marx (McConomy Auditorium) Chair: Irit Dinur 11:10 Fitting tree metrics: Hierarchical clustering and Phy- logeny Subhash Khot Nir Ailon and Moses Charikar On the Unique Games Conjecture 11:30 Metric Embeddings with Relaxed Guarantees Break 3:30pm – 4:00pm Ittai Abraham, Yair Bartal, T-H.
    [Show full text]
  • Distance-Sensitive Hashing∗
    Distance-Sensitive Hashing∗ Martin Aumüller Tobias Christiani IT University of Copenhagen IT University of Copenhagen [email protected] [email protected] Rasmus Pagh Francesco Silvestri IT University of Copenhagen University of Padova [email protected] [email protected] ABSTRACT tight up to lower order terms. In particular, we extend existing Locality-sensitive hashing (LSH) is an important tool for managing LSH lower bounds, showing that they also hold in the asymmetric high-dimensional noisy or uncertain data, for example in connec- setting. tion with data cleaning (similarity join) and noise-robust search (similarity search). However, for a number of problems the LSH CCS CONCEPTS framework is not known to yield good solutions, and instead ad hoc • Theory of computation → Randomness, geometry and dis- solutions have been designed for particular similarity and distance crete structures; Data structures design and analysis; • Informa- measures. For example, this is true for output-sensitive similarity tion systems → Information retrieval; search/join, and for indexes supporting annulus queries that aim to report a point close to a certain given distance from the query KEYWORDS point. locality-sensitive hashing; similarity search; annulus query In this paper we initiate the study of distance-sensitive hashing (DSH), a generalization of LSH that seeks a family of hash functions ACM Reference Format: Martin Aumüller, Tobias Christiani, Rasmus Pagh, and Francesco Silvestri. such that the probability of two points having the same hash value is 2018. Distance-Sensitive Hashing. In Proceedings of 2018 International Confer- a given function of the distance between them. More precisely, given ence on Management of Data (PODS’18).
    [Show full text]
  • Arxiv:2102.08942V1 [Cs.DB]
    A Survey on Locality Sensitive Hashing Algorithms and their Applications OMID JAFARI, New Mexico State University, USA PREETI MAURYA, New Mexico State University, USA PARTH NAGARKAR, New Mexico State University, USA KHANDKER MUSHFIQUL ISLAM, New Mexico State University, USA CHIDAMBARAM CRUSHEV, New Mexico State University, USA Finding nearest neighbors in high-dimensional spaces is a fundamental operation in many diverse application domains. Locality Sensitive Hashing (LSH) is one of the most popular techniques for finding approximate nearest neighbor searches in high-dimensional spaces. The main benefits of LSH are its sub-linear query performance and theoretical guarantees on the query accuracy. In this survey paper, we provide a review of state-of-the-art LSH and Distributed LSH techniques. Most importantly, unlike any other prior survey, we present how Locality Sensitive Hashing is utilized in different application domains. CCS Concepts: • General and reference → Surveys and overviews. Additional Key Words and Phrases: Locality Sensitive Hashing, Approximate Nearest Neighbor Search, High-Dimensional Similarity Search, Indexing 1 INTRODUCTION Finding nearest neighbors in high-dimensional spaces is an important problem in several diverse applications, such as multimedia retrieval, machine learning, biological and geological sciences, etc. For low-dimensions (< 10), popular tree-based index structures, such as KD-tree [12], SR-tree [56], etc. are effective, but for higher number of dimensions, these index structures suffer from the well-known problem, curse of dimensionality (where the performance of these index structures is often out-performed even by linear scans) [21]. Instead of searching for exact results, one solution to address the curse of dimensionality problem is to look for approximate results.
    [Show full text]
  • Hierarchical Clustering with Global Objectives: Approximation Algorithms and Hardness Results
    HIERARCHICAL CLUSTERING WITH GLOBAL OBJECTIVES: APPROXIMATION ALGORITHMS AND HARDNESS RESULTS ADISSERTATION SUBMITTED TO THE DEPARTMENT OF COMPUTER SCIENCE AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Evangelos Chatziafratis June 2020 © 2020 by Evangelos Chatziafratis. All Rights Reserved. Re-distributed by Stanford University under license with the author. This work is licensed under a Creative Commons Attribution- Noncommercial 3.0 United States License. http://creativecommons.org/licenses/by-nc/3.0/us/ This dissertation is online at: http://purl.stanford.edu/bb164pj1759 ii I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Tim Roughgarden, Primary Adviser I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Moses Charikar, Co-Adviser I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Li-Yang Tan I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Gregory Valiant Approved for the Stanford University Committee on Graduate Studies. Stacey F. Bent, Vice Provost for Graduate Education This signature page was generated electronically upon submission of this dissertation in electronic format.
    [Show full text]
  • SIGMOD Flyer
    DATES: Research paper SIGMOD 2006 abstracts Nov. 15, 2005 Research papers, 25th ACM SIGMOD International Conference on demonstrations, Management of Data industrial talks, tutorials, panels Nov. 29, 2005 June 26- June 29, 2006 Author Notification Feb. 24, 2006 Chicago, USA The annual ACM SIGMOD conference is a leading international forum for database researchers, developers, and users to explore cutting-edge ideas and results, and to exchange techniques, tools, and experiences. We invite the submission of original research contributions as well as proposals for demonstrations, tutorials, industrial presentations, and panels. We encourage submissions relating to all aspects of data management defined broadly and particularly ORGANIZERS: encourage work that represent deep technical insights or present new abstractions and novel approaches to problems of significance. We especially welcome submissions that help identify and solve data management systems issues by General Chair leveraging knowledge of applications and related areas, such as information retrieval and search, operating systems & Clement Yu, U. of Illinois storage technologies, and web services. Areas of interest include but are not limited to: at Chicago • Benchmarking and performance evaluation Vice Gen. Chair • Data cleaning and integration Peter Scheuermann, Northwestern Univ. • Database monitoring and tuning PC Chair • Data privacy and security Surajit Chaudhuri, • Data warehousing and decision-support systems Microsoft Research • Embedded, sensor, mobile databases and applications Demo. Chair Anastassia Ailamaki, CMU • Managing uncertain and imprecise information Industrial PC Chair • Peer-to-peer data management Alon Halevy, U. of • Personalized information systems Washington, Seattle • Query processing and optimization Panels Chair Christian S. Jensen, • Replication, caching, and publish-subscribe systems Aalborg University • Text search and database querying Tutorials Chair • Semi-structured data David DeWitt, U.
    [Show full text]
  • Lower Bounds on Lattice Sieving and Information Set Decoding
    Lower bounds on lattice sieving and information set decoding Elena Kirshanova1 and Thijs Laarhoven2 1Immanuel Kant Baltic Federal University, Kaliningrad, Russia [email protected] 2Eindhoven University of Technology, Eindhoven, The Netherlands [email protected] April 22, 2021 Abstract In two of the main areas of post-quantum cryptography, based on lattices and codes, nearest neighbor techniques have been used to speed up state-of-the-art cryptanalytic algorithms, and to obtain the lowest asymptotic cost estimates to date [May{Ozerov, Eurocrypt'15; Becker{Ducas{Gama{Laarhoven, SODA'16]. These upper bounds are useful for assessing the security of cryptosystems against known attacks, but to guarantee long-term security one would like to have closely matching lower bounds, showing that improvements on the algorithmic side will not drastically reduce the security in the future. As existing lower bounds from the nearest neighbor literature do not apply to the nearest neighbor problems appearing in this context, one might wonder whether further speedups to these cryptanalytic algorithms can still be found by only improving the nearest neighbor subroutines. We derive new lower bounds on the costs of solving the nearest neighbor search problems appearing in these cryptanalytic settings. For the Euclidean metric we show that for random data sets on the sphere, the locality-sensitive filtering approach of [Becker{Ducas{Gama{Laarhoven, SODA 2016] using spherical caps is optimal, and hence within a broad class of lattice sieving algorithms covering almost all approaches to date, their asymptotic time complexity of 20:292d+o(d) is optimal. Similar conditional optimality results apply to lattice sieving variants, such as the 20:265d+o(d) complexity for quantum sieving [Laarhoven, PhD thesis 2016] and previously derived complexity estimates for tuple sieving [Herold{Kirshanova{Laarhoven, PKC 2018].
    [Show full text]
  • Approximate Nearest Neighbor: Towards Removing the Curse of Dimensionality
    THEORY OF COMPUTING, Volume 8 (2012), pp. 321–350 www.theoryofcomputing.org SPECIAL ISSUE IN HONOR OF RAJEEV MOTWANI Approximate Nearest Neighbor: Towards Removing the Curse of Dimensionality Sariel Har-Peled∗ Piotr Indyk † Rajeev Motwani ‡ Received: August 20, 2010; published: July 16, 2012. Abstract: We present two algorithms for the approximate nearest neighbor problem in d high-dimensional spaces. For data sets of size n living in R , the algorithms require space that is only polynomial in n and d, while achieving query times that are sub-linear in n and polynomial in d. We also show applications to other high-dimensional geometric problems, such as the approximate minimum spanning tree. The article is based on the material from the authors’ STOC’98 and FOCS’01 papers. It unifies, generalizes and simplifies the results from those papers. ACM Classification: F.2.2 AMS Classification: 68W25 Key words and phrases: approximate nearest neighbor, high dimensions, locality-sensitive hashing 1 Introduction The nearest neighbor (NN) problem is defined as follows: Given a set P of n points in a metric space defined over a set X with distance function D, preprocess P to efficiently answer queries for finding the point in P closest to a query point q 2 X. A particularly interesting case is that of the d-dimensional d Euclidean space where X = R under some `s norm. This problem is of major importance in several ∗Supported by NSF CAREER award CCR-0132901 and AF award CCF-0915984. †Supported by a Stanford Graduate Fellowship, NSF Award CCR-9357849 and NSF CAREER award CCF-0133849.
    [Show full text]
  • Implementation of Locality Sensitive Hashing Techniques
    Implementation of Locality Sensitive Hashing Techniques Project Report submitted in partial fulfillment of the requirement for the degree of Bachelor of Technology. in Computer Science & Engineering under the Supervision of Dr. Nitin Chanderwal By Srishti Tomar(111210) to Jaypee University of Information and TechnologyWaknaghat, Solan – 173234, Himachal Pradesh i Certificate This is to certify that project report entitled “Implementaion of Locality Sensitive Hashing Techniques”, submitted by Srishti Tomar in partial fulfillment for the award of degree of Bachelor of Technology in Computer Science & Engineering to Jaypee University of Information Technology, Waknaghat, Solan has been carried out under my supervision. This work has not been submitted partially or fully to any other University or Institute for the award of this or any other degree or diploma. Date: Supervisor’s Name: Dr. Nitin Chanderwal Designation : Associate Professor ii Acknowledgement I am highly indebted to Jaypee University of Information Technology for their guidance and constant supervision as well as for providing necessary information regarding the project & also for their support in completing the project. I would like to express my gratitude towards my parents & Project Guide for their kind co-operation and encouragement which help me in completion of this project. I would like to express my special gratitude and thanks to industry persons for giving me such attention and time. My thanks and appreciations also go to my colleague in developing the project and people who have willingly helped me out with their abilities Date: Name of the student: Srishti Tomar iii Table of Content S. No. Topic Page No. 1. Abstract 1 2.
    [Show full text]
  • Model Checking Large Design Spaces: Theory, Tools, and Experiments
    Iowa State University Capstones, Theses and Graduate Theses and Dissertations Dissertations 2020 Model checking large design spaces: Theory, tools, and experiments Rohit Dureja Iowa State University Follow this and additional works at: https://lib.dr.iastate.edu/etd Recommended Citation Dureja, Rohit, "Model checking large design spaces: Theory, tools, and experiments" (2020). Graduate Theses and Dissertations. 18304. https://lib.dr.iastate.edu/etd/18304 This Thesis is brought to you for free and open access by the Iowa State University Capstones, Theses and Dissertations at Iowa State University Digital Repository. It has been accepted for inclusion in Graduate Theses and Dissertations by an authorized administrator of Iowa State University Digital Repository. For more information, please contact [email protected]. Model checking large design spaces: Theory, tools, and experiments by Rohit Dureja A dissertation submitted to the graduate faculty in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Major: Computer Science Program of Study Committee: Kristin Y. Rozier, Co-major Professor Gianfranco Ciardo, Co-major Professor Samik Basu Robyn Lutz Hridesh Rajan The student author, whose presentation of the scholarship herein was approved by the program of study committee, is solely responsible for the content of this dissertation. The Graduate College will ensure this dissertation is globally accessible and will not permit alterations after a degree is conferred. Iowa State University Ames, Iowa 2020 Copyright © Rohit Dureja, 2020. All rights reserved. ii DEDICATION To my family. iii TABLE OF CONTENTS LIST OF FIGURES . vi LIST OF TABLES . .x ABSTRACT . xi CHAPTER 1. INTRODUCTION . .1 1.1 Motivation .
    [Show full text]
  • Detecting Near-Duplicates for Web Crawling
    WWW 2007 / Track: Data Mining Session: Similarity Search Detecting Near-Duplicates for Web Crawling ∗ Gurmeet Singh Manku Arvind Jain Anish Das Sarma Google Inc. Google Inc. Stanford University [email protected] [email protected] [email protected] ABSTRACT Documents that are exact duplicates of each other (due to Near-duplicate web documents are abundant. Two such mirroring and plagiarism) are easy to identify by standard documents differ from each other in a very small portion checksumming techniques. A more difficult problem is the that displays advertisements, for example. Such differences identification of near-duplicate documents. Two such docu- are irrelevant for web search. So the quality of a web crawler ments are identical in terms of content but differ in a small increases if it can assess whether a newly crawled web page portion of the document such as advertisements, counters is a near-duplicate of a previously crawled web page or not. and timestamps. These differences are irrelevant for web In the course of developing a near-duplicate detection sys- search. So if a newly-crawled page Pduplicate is deemed a tem for a multi-billion page repository, we make two research near-duplicate of an already-crawled page P , the crawl en- contributions. First, we demonstrate that Charikar's finger- gine should ignore Pduplicate and all its out-going links (in- printing technique is appropriate for this goal. Second, we tuition suggests that these are probably near-duplicates of P 1 present an algorithmic technique for identifying existing f- pages reachable from ). Elimination of near-duplicates bit fingerprints that differ from a given fingerprint in at most saves network bandwidth, reduces storage costs and im- k bit-positions, for small k.
    [Show full text]
  • Scalable Nearest Neighbor Search for Optimal Transport∗
    Scalable Nearest Neighbor Search for Optimal Transport∗ Arturs Backursy Yihe Dong Piotr Indyk Ilya Razenshteyn Tal Wagner TTIC Microsoft MIT Microsoft Research MIT September 30, 2020 Abstract The Optimal Transport (a.k.a. Wasserstein) distance is an increasingly popular similarity measure for rich data domains, such as images or text documents. This raises the necessity for fast nearest neighbor search algorithms according to this distance, which poses a substantial computational bottleneck on massive datasets. In this work we introduce Flowtree, a fast and accurate approximation algorithm for the Wasserstein- 1 distance. We formally analyze its approximation factor and running time. We perform extensive experimental evaluation of nearest neighbor search algorithms in the W1 distance on real-world dataset. Our results show that compared to previous state of the art, Flowtree achieves up to 7:4 times faster running time. 1 Introduction Given a finite metric space M = (X; dX ) and two distributions µ and ν on X, the Wasserstein-1 distance (a.k.a. Earth Mover's Distance or Optimal Transport) between µ and ν is defined as X W1(µ, ν) = min τ(x1; x2) · dX (x1; x2); (1) τ x1;x22X where the minimum is taken over all distributions τ on X × X whose marginals are equal to µ and ν.1 The Wasserstein-1 distance and its variants are heavily used in applications to measure similarity in structured data domains, such as images [RTG00] and natural language text [KSKW15]. In particular, [KSKW15] proposed the Word Mover Distance (WMD) for text documents. Each document is seen as a uniform distri- bution over the words it contains, and the underlying metric between words is given by high-dimensional word embeddings such as word2vec [MSC+13] or GloVe [PSM14].
    [Show full text]
  • SETH-Based Lower Bounds for Subset Sum and Bicriteria Path∗
    SETH-Based Lower Bounds for Subset Sum and Bicriteria Path∗ Amir Abboudy Karl Bringmannz Danny Hermelinx Dvir Shabtay{ Abstract 1 Introduction Subset Sum and k-SAT are two of the most extensively The field of fine-grained complexity is anchored around studied problems in computer science, and conjectures about their hardness are among the cornerstones of fine-grained certain hypotheses about the exact time complexity of a complexity. An important open problem in this area is to small set of core problems. Due to dozens of reductions, base the hardness of one of these problems on the other. we now know that the current algorithms for many Our main result is a tight reduction from k-SAT important problems are optimal unless breakthrough to Subset Sum on dense instances, proving that Bellman's 1962 pseudo-polynomial O∗(T )-time algorithm for Subset algorithms for the core problems exist. A central Sum on n numbers and target T cannot be improved to time challenge in this field is to understand the connections T 1−" · 2o(n) for any " > 0, unless the Strong Exponential and relative difficulties among these core problems. In Time Hypothesis (SETH) fails. As a corollary, we prove a \Direct-OR" theorem for this work, we discover a new connection between two Subset Sum under SETH, offering a new tool for proving core problems: a tight reduction from k-SAT to Subset conditional lower bounds: It is now possible to assume that Sum. deciding whether one out of N given instances of Subset In the first part of the introduction we discuss Sum is a YES instance requires time (NT )1−o(1).
    [Show full text]