Complex Queries and Complex Data: Challenges in Similarity Search Johannes Niedermayer M¨unchen2015 Complex Queries and Complex Data: Challenges in Similarity Search Johannes Niedermayer Dissertation an der Fakult¨atf¨urMathematik, Informatik und Statistik der Ludwig{Maximilians{Universit¨at M¨unchen vorgelegt von Johannes Niedermayer aus M¨unchen M¨unchen, den 21.07.2015 Erstgutachter: PD Dr. Peer Kr¨oger Zweitgutachter: Prof. Dr. Michael Gertz Tag der m¨undlichen Pr¨ufung:30.10.2015 Eidesstattliche Versicherung (Siehe Promotionsordnung vom 12.07.11, § 8, Abs. 2 Pkt. .5.) Hiermit erkläre ich an Eidesstatt, dass die Dissertation von mir selbstständig, ohne unerlaubte Beihilfe angefertigt ist. Name, Vorname Ort, Datum Unterschrift Doktorand/in Formular 3.2 To my future wife. To my parents. To my brother. Contents List of Figures xii List of Tables xv Table of Notations xix Summary xxii I Preliminaries 1 1 Introduction 3 2 Similarity Search 5 2.1 Mathematical Definitions . .5 2.2 Query Types . .6 2.3 Pipeline . .8 2.3.1 Feature Extraction . .9 2.3.2 Indexing and Query Processing . 11 2.4 Challenges . 16 2.4.1 Complex Data . 16 2.4.2 Complex Query Predicates . 18 2.4.3 Large Volumes . 19 3 Thesis Overview and Contributions 21 4 Incorporated Publications and Coauthorship 23 II The RkNN Join 25 5 Introduction 27 x CONTENTS 6 Preliminaries 31 6.1 Problem Definition . 31 6.2 Related Work . 33 6.3 Classification of Existing RkNN Joins . 34 7 Algorithms 37 7.1 The Mutual Pruning Algorithm . 37 7.1.1 General Idea . 37 7.1.2 The Algorithm joinEntry ................ 41 7.1.3 Refinement: The resolve-Routine . 42 7.2 A Self Pruning Approach . 42 7.2.1 General Idea . 42 7.2.2 Implementing the Self-kNN-Join . 45 7.2.3 Implementing the Varying-Range-Join . 47 7.3 Extension to Metric Spaces . 49 7.3.1 Adaptions of the Update List Approach . 50 7.3.2 Adaptions of the kNN-Based Approach . 51 8 Evaluation 53 8.1 Experiments on Synthetic Data . 56 8.2 Real Data Experiments . 62 8.3 Comparing CPU-Cost and IO-Cost . 63 9 Conclusion 65 III Nearest Neighbor Queries on Uncertain Spatio- Temporal Data 67 10 Introduction 69 11 Preliminaries 75 11.1 Problem Definition . 75 11.1.1 Uncertain Trajectory Model . 75 11.1.2 Nearest Neighbor Queries . 78 11.1.3 Probabilistic Reverse Nearest Neighbor Queries . 80 11.2 Related Work . 82 12 Nearest Neighbor Queries 85 12.1 Theoretical Analysis . 85 12.1.1 The P9NN Query . 85 12.1.2 The P8NN Query . 87 CONTENTS xi 12.1.3 The PCNN Query . 95 12.2 Sampling Possible Trajectories . 97 12.2.1 Traditional Sampling . 97 12.2.2 Efficient and Appropriate Sampling . 98 12.3 Spatial Pruning . 108 12.4 Experimental Evaluation . 111 12.4.1 Evaluation: P8NNQ and P9NNQ . 113 12.4.2 Continuous Queries . 119 13 Reverse Nearest Neighbor Queries 121 13.1 PRNN Query Processing . 121 13.1.1 Temporal and Spatial Filtering . 122 13.1.2 Verification . 125 13.2 Experiments . 125 13.2.1 Evaluation: P8RNNQ and P9RNNQ . 126 14 Conclusions 129 IV kNN Queries for Image Retrieval 131 15 Introduction 133 16 Preliminaries 139 16.1 Problem Definition . 140 16.2 Related Work . 142 16.2.1 Keypoint Reduction . 142 16.2.2 kNN Indexing . 143 16.2.3 kNN-based Matching . 144 16.2.4 Match Expansion . 144 17 Minimizing the Number of Matching Queries for Object Re- trieval 147 17.1 Pipeline . 147 17.1.1 Theory . 147 17.1.2 Practical Considerations . 151 17.2 Experiments . 154 17.2.1 Experimental Setup . 154 17.2.2 Experiments . 157 xii CONTENTS 18 Retrieval of Binary Features in Image Databases: A Study 163 18.1 Querying Binary Features with LSH . 164 18.2 Experimental Evaluation . 167 18.2.1 Nearest Neighbor Queries . 169 18.2.2 Range Queries and BoVW . 172 19 Conclusions 175 V Conclusions 177 Acknowledgements 194 List of Figures 2.1 Query Processing Pipeline . .9 2.2 The R-Tree . 12 2.3 The Filter-Refinement approach . 16 5.1 Application of RkNN join between two sets of products R and S for product (set) recommendation. 28 6.1 Visualization of the Monochromatic RkNN Query and the Monochromatic RkNN Join . 32 7.1 Spatial Domination. 38 7.2 Comparison of MS(eS) and CMBR(eS) in an (a) average, (b) worst, and (c) best case. 47 8.1 Performance (CPU time), synthetic dataset. Time is measured in seconds. 55 8.2 Performance (page accesses), synthetic dataset. 57 8.3 Given a set S of size jSj, the left figure visualizes the size of set Re for which TPL and the kNN-based joins have the same computational performance on our test sets R and S (note that this might vary with other datasets). The right figure visualizes the results for varying cache size. Note the logarithmic scale on the y-Axis. 60 8.4 Performance (CPU time in seconds, page accesses), real dataset (HSV). 61 8.5 Performance (CPU time, page accesses), real dataset (post office). 63 8.6 CPU and IO time in seconds when varying the cache size. Left: SSD, right: HDD . 63 10.1 Uncertainty in a Spatio-Temporal Context. 70 xiv LIST OF FIGURES 11.1 Model Visualization (best viewed in color) . 76 11.2 Example Setup for PNN Queries . 80 11.3 Example Setup for PRNN Queries . 82 12.1 An example instance of our mapping of the 3-SAT problem to a set of Markov chains. 86 12.2 P8NN: Violation of the Markov assumption . 95 12.3 Traditional MC-Sampling. 97 12.4 An overview over our forward-backward-algorithm. 99 12.5 Spatio-Temporal Pruning Example. 110 12.6 Examples of the models used for synthetic and real data. Black lines denote transition probabilities. Thicker lines de- note higher probabilities, thinner lines lower probabilities. The synthetic model consists of 10k states. 111 12.7 Varying the Number of States N ................ 113 12.8 Varying the Branching Factor b ................. 114 12.9 Varying the Number of Objects jDj ............... 115 12.10Real Data: Varying the Number of Objects . 115 12.11Efficiency of Sampling without Model Adaption. 116 12.12Effectiveness of Sampling, P8NN and P9NN . 117 12.13Real Data: Effectiveness of the Model Adaption . 118 12.14PCNN: Varying the Number of Objects . 119 12.15PCNN: Varying τ ......................... 120 13.1 Spatio-temporal filtering (only leaf nodes are shown) . 122 13.2 Synthetic Data, Varying jDj.................... 126 13.3 Synthetic Data, Varying b..................... 127 13.4 Synthetic Data, Varying N = jSj................. 128 13.5 Real Data, Varying jDj...................... 128 16.1 Object Recognition Pipeline . 139 17.1 Generation of additional match hypotheses. 149 17.2 Performance for varying k (Hessian-affine SIFT). Straight lines show the performance for 10 keypoints, dashed lines for 1000 keypoints. Equivalent approaches have equivalent colors. 161 18.1 LSH-based indexing . 165 18.2 Parameter Settings and Distance Distribution. 168 18.3 Populatation of buckets (log-log-space). 169 18.4 Varying # Tables (left) and Database Size (right) . 170 18.5 Varying # Probes, Recall (left) and Distance Calculations (right)171 LIST OF FIGURES xv 18.6 Varying # Bits . 172 18.7 Recall and False Hit Rate for Range Queries . 173 18.8 Applying the Hashing Functions to a BoVW-based Ranking . 173 xvi LIST OF FIGURES List of Tables 8.1 Values for the evaluated independent variables. Default values are denoted in bold. 54 12.1 Parameters varied during our experimental evaluation (syn- thetic data). Differing parameters for continuous experiments are denoted by a superscript c. Default values are denoted in bold. 112 17.1 Database Statistics . 155 17.2 Parameters for Match Expansion . 156 17.3 SIFT, Oxford5k, k=100 . 157 17.4 SIFT, Paris6k, k=100 . 157 17.5 SIFT, Holidays, k=10 . 158 17.6 BinBoost, Oxford5k, k=100 . 158 17.7 SIFT, Oxford 105k, k=100 . 159 17.8 BinBoost, Oxford 105k, k=100 . 160 xviii LIST OF TABLES Table of Notations General Notations Symbol Description M A metric space Rd The d-dimensional vector space Bd The space of d-dimensional binary vectors D A database of objects q A query object o 2 D A database object dist(x; y) A distance function, usually a metric − range(q; D; ) -range query kNN(q; D; k) kNN query RkNN(q; D; k) RkNN query MINDIST (A; B) Minimum distance between objects A and B MAXDIST (A; B) Maximum distance between object A and B Dom(A; B; C) Spatial domination function of the objects A, B, and C H A family of hash functions G A function family Q A queue Part II: The RkNN Join Symbol Description R A set of objects, if not denoted otherwise R ⊂ Rd. Corresponds to the left (query) set of the join. S A set of objects, usually S ⊂ Rd. Corresponds to the right (database) set of the join. r Element from R, r 2 R s Element from S, s 2 S R; S Index of set R and S, respectively xx TABLE OF NOTATIONS Part III: Nearest Neighbor Queries on Uncertain Spatio-Temporal Data Symbol Description T A time domain T = f0; : : : ; ng T A time interval t 2 T A single point in time S A spatial domain si 2 S A spatial location, i.e. a state θi 2 S A spatial observation Θo The representation of object o, i.e. a set of location-time tuples, w.l.o.g. ordered by the timesteps ti o o o o o o o Θ = fht1; θ1i; ht2; θ2i;:::; htjΘoj; θjΘojig o(t) The realization of the random variable represent- ing object o at time t M o(t) The transition matrix of object o at time t ~so(t) The probability distribution over spatial locations of object o at time t τ Probability threshold l A literal (9 query) c A clause (9 query) x A variable (9 query) F (t);R(t) Forward and backward model for sampling TABLE OF NOTATIONS xxi Part IV: kNN Queries for Image Retrieval Symbol Description Ij An image Ij 2 D i i i i i i i i i pj 2 Ij A keypoint, pj = (vj; xj; yj; sj; rj; σj;Aj) i vj Feature vector of feature i in image j i i xj, yj Interest point location of feature i in image j i sj Scale of feature i in image j i σj Response of feature i in image j i Aj Affine matrix of feature i in image j m(x) An arbitrary matching function Ψ ⊆ Ij A.