Dual Principal Component Pursuit and Filtrated Algebraic Subspace

Home , Linear subspace

Dual Principal Component Pursuit

and

Filtrated Algebraic Subspace Clustering

Manolis C. Tsakiris

A dissertation submitted to The Johns Hopkins University in conformity with the

requirements for the degree of Doctor of Philosophy.

Baltimore, Maryland

March, 2017

c Manolis C. Tsakiris 2017

Recent years have witnessed an explosion of data across scientific fields enabled by advances in sensor technology and distributed hardware systems. This has given rise to the challenge of efficiently processing such data for performing various tasks, such as face recognition in images and videos. Towards that end, two operations on the data are of fundamental importance, i.e., dimensionality reduction and clustering, and the idea of learning one or more linear subspaces from the data has proved a fruitful notion in that context. Nev- ertheless, state-of-the-art methods, such as Robust Principal Component Analysis (RPCA) or Sparse Subspace Clustering (SSC), operate under the hypothesis that the subspaces have dimensions that are small relative to the ambient dimension, and fail otherwise.

This thesis attempts to advance the state-of-the-art of subspace learning methods in the regime where the subspaces have high relative dimensions. The ﬁrst major contribution of this thesis is a single subspace learning method called Dual Principal Component Pursuit

(DPCP), which solves the robust PCA problem in the presence of outliers. Contrary to sparse and low-rank state-of-the-art methods, the theoretical guarantees of DPCP do not place any constraints on the dimension of the subspace. In particular, DPCP computes the

ii ABSTRACT

orthogonal complement of the subspace, thus it is particularly suited for subspaces of low

codimension. This is done by solving a non-convex cosparse problem on the sphere, whose

global minimizers are shown to be vectors normal to the subspace. Algorithms for solving

the non-convex problem are developed and tested on synthetic and real data, showing that

DPCP is able to handle higher subspace dimensions and larger amounts of outliers than

existing methods. Finally, DPCP is extended theoretically and algorithmically to the case

of multiple subspaces.

The second major contribution of this thesis is a subspace clustering method called

Filtrated Algebraic Subspace Clustering (FASC), which builds upon algebraic geometric

ideas of an older method, known as Generalized Principal Component Analysis (GPCA).

GPCA is naturally suited for subspaces of large dimension, but it suffers from two weak-

nesses: sensitivity to noise and large computational complexity. This thesis demonstrates

that FASC addresses successfully the robustness to noise. This is achieved through an

equivalent formulation of GPCA, which uses the idea of ﬁltrations of unions of subspaces.

An algebraic geometric analysis establishes the theoretical equivalence of the two meth-

ods, while experiments on synthetic and real data reveal that FASC not only dramatically

improves upon GPCA, but also upon existing methods on several occasions.

Primary Reader: Rene´ Vidal

Secondary Reader: Daniel P. Robinson

iii Dedication

This thesis is dedicated to my advisor, Rene´ Vidal, for marveling me with his genius, rigor, endurance and enthusiasm.

iv Acknowledgments

First of all, I am thankful to my advisor, Prof. Rene´ Vidal, whose inﬂuence has been enor- mous: working with him has been the best academic experience in my life. Secondly, I am thankful to Prof. Daniel P. Robinson, for many useful conversations, for always being supportive and friendly, and, quite importantly, for not being easily convinced: this led to the identiﬁcation of a few inaccuracies and the improvement of several arguments in this thesis. I also thank Prof. Aldo Conca of the mathematics department of the Uni- versity of Genova for enthusiastically answering many questions that i had regarding the

Castelnuovo-Mumford regularity of subspace arrangements, as well as Prof. Glyn Harman for pointing out a Koksma-Hlawka inequality for integration on the unit sphere.

Then I thank Miss Debbie Race of the ECE department for being extremely helpful and patient with several administrative issues. I thank the Center for Imaging Science (CIS) for being a cozy home for the last 4 years and the ECE department for always being supportive and admitting me as a PhD student to begin with.

I thank all people who were nice to me during my stay in Baltimore. Special thanks go to sensei Ebon Phoenix for passionately teaching me martial arts, and to Tony Hatzigeor-

v ACKNOWLEDGMENTS galis and Jose Torres for being like brothers to me. I also thank the JHU newbie, Christos

Sapsanis, from whom, remarkably, the JHU graduate community has a lot to beneﬁt. And

Guilherme Franca and Tao Xiong for being great friends.

Finally, i thank my best friend Dimitris Lountzis for being who he is, and most importantly i thank my family, Chris, Evi and Michalis, for their inﬁnite love.

vi Contents

Abstract ii

Acknowledgments v

List of Tables xiii

List of Figures xv

1 Introduction 1

1.1 Modeling data with linear subspaces ...... 1

1.1.1 Modeling data with a single subspace ...... 1

1.1.2 Modeling data with multiple subspaces ...... 3

1.2 Challenges and the role of dimension ...... 4

1.3 Contributions of this thesis ...... 5

1.3.1 Dual principal component pursuit (DPCP) ...... 6

1.3.2 Filtrated algebraic subspace clustering ...... 8

1.4 Notation ...... 10

vii CONTENTS

2 Prior Art 13

2.1 Learning a single subspace in the presence of outliers ...... 15

2.2 Learning multiple subspaces ...... 20

2.3 Challenges in high relative dimensions ...... 25

3 Dual Principal Component Pursuit 29

3.1 Introduction ...... 29

3.2 Single subspace learning with outliers via DPCP ...... 32

3.2.1 Problem formulation ...... 32

3.2.1.1 Data model ...... 32

3.2.1.2 Conceptual formulation ...... 33

3.2.1.3 Hyperplane pursuit by `1 minimization ...... 34

3.2.2 Theoretical analysis of the continuous problem ...... 37

3.2.2.1 The underlying continuous problem ...... 38

3.2.2.2 Conditions for global optimality and convergence . . . . 42

3.2.3 Theoretical analysis of the discrete problem ...... 48

3.2.3.1 Discrepancy bounds between continuous and discrete prob-

lems ...... 49

3.2.3.2 Conditions for global optimality of the discrete problem 56

3.2.3.3 Conditions for convergence of the discrete recursive al-

gorithm ...... 66

3.2.4 Algorithmic contributions ...... 71

viii CONTENTS

3.2.4.1 Relaxed DPCP and DPCA algorithms ...... 71

3.2.4.2 Relaxed and denoised DPCP ...... 74

3.2.4.3 Denoised DPCP ...... 76

3.2.4.4 DPCP via iteratively reweighted least-squares ...... 79

3.2.5 Experimental evaluation ...... 80

3.2.5.1 Computing a single dual principal component ...... 81

3.2.5.2 Outlier detection using synthetic data ...... 84

3.2.5.3 Outlier detection using real face and object images . . . . 90

3.3 Learning a hyperplane arrangement via DPCP ...... 96

3.3.1 Problem overview ...... 96

3.3.2 Data model ...... 97

3.3.3 Theoretical analysis of the continuous problem ...... 98

3.3.3.1 Derivation, interpretation and basic properties of the con-

tinuous problem ...... 99

3.3.3.2 The cases of i) two hyperplanes and ii) orthogonal hy-

perplanes ...... 102

3.3.3.3 The case of three equiangular hyperplanes ...... 105

3.3.3.4 Conditions of global optimality for an arbitrary hyper-

plane arrangement ...... 114

3.3.4 Theoretical analysis of the discrete problem ...... 120

3.3.5 Algorithms ...... 131

ix CONTENTS

3.3.5.1 Learning a hyperplane arrangement sequentially . . . . . 132

3.3.5.2 K-Hyperplanes via DPCP ...... 132

3.3.6 Experimental evaluation ...... 134

3.3.6.1 Synthetic data ...... 134

3.3.6.2 3D plane clustering of real Kinect data ...... 142

3.4 Conclusions ...... 148

4 Advances in Algebraic Subspace Clustering 155

4.1 Review of algebraic subspace clustering ...... 156

4.1.1 Subspaces of codimension 1 ...... 157

4.1.2 Subspaces of equal dimension ...... 160

4.1.3 Known number of subspaces of arbitrary dimensions ...... 161

4.1.4 Unknown number of subspaces of arbitrary dimensions ...... 165

4.1.5 Computational complexity and recursive ASC ...... 167

4.1.6 Instability in the presence of noise and spectral ASC ...... 168

4.1.7 The challenge ...... 170

4.2 Filtrated algebraic subspace clustering (FASC) ...... 171

4.2.1 Filtrations of subspace arrangements: geometric overview . . . . . 171

4.2.2 Filtrations of subspace arrangements: theory ...... 177

4.2.2.1 Data in general position in a subspace arrangement . . . 177

4.2.2.2 Constructing the ﬁrst step of a ﬁltration ...... 182

4.2.2.3 Deciding whether to take a second step in a ﬁltration . . . 185

x CONTENTS

4.2.2.4 Taking multiple steps in a ﬁltration and terminating . . . 188

4.2.2.5 The FASC algorithm ...... 193

4.3 Filtrated spectral algebraic subspace clustering ...... 195

4.3.1 Implementing robust ﬁltrations ...... 195

4.3.2 Combining multiple ﬁltrations ...... 198

4.3.3 The FSASC algorithm ...... 199

4.3.4 A distance-based afﬁnity ...... 200

4.3.5 Discussion on the computational complexity ...... 203

4.4 Experiments ...... 205

4.4.1 Experiments on synthetic data ...... 206

4.4.2 Experiments on real motion sequences ...... 217

4.5 Algebraic clustering of afﬁne subspaces ...... 219

4.5.1 Motivation ...... 219

4.5.2 Problem statement and traditional approach ...... 221

4.5.3 Algebraic geometry of unions of afﬁne subspaces ...... 224

4.5.3.1 Afﬁne subspaces as afﬁne varieties ...... 224

4.5.3.2 The projective closure of afﬁne subspaces ...... 227

4.5.4 Correctness theorems for the homogenization trick ...... 232

4.6 Conclusions ...... 238

4.7 Appendix ...... 239

4.7.1 Notions from commutative algebra ...... 239

xi CONTENTS

4.7.2 Notions from algebraic geometry ...... 241

4.7.3 Subspace arrangements and their vanishing ideals ...... 244

5 Conclusions 255

Bibliography 257

Vita 274

xii List of Tables

3.1 Mean running times in seconds, corresponding to the experiment of Figure 3.13 for data balancing parameter α =1...... 142 3.2 3D plane clustering error for a subset of the real Kinect dataset NYUdepthV2. n is the number of ﬁtted planes. GC(0) and GC(1) refer to clustering error without or with spatial smoothing, respectively...... 149

4.1 Mean subspace clustering error in % over 100 independent trials for synthetic data randomly generated in three random subspaces of R9 of dimensions (d1,d2,d3). There are 200 points associated to each subspace, which are corrupted by zero-mean additive white noise of standard deviation σ =0, 0.01 and support in the orthogonal complement of the subspace. 207 4.2 Mean subspace clustering error in % over 100 independent trials for synthetic data randomly generated in three random subspaces of R9 of dimensions (d1,d2,d3). There are 200 points associated to each subspace, which are corrupted by zero-mean additive white noise of standard deviation σ = 0.03, 0.05 and support in the orthogonal complement of the subspace...... 208 4.3 Mean intra-cluster connectivity over 100 independent trials for synthetic data randomly generated in three random subspaces of R9 of dimensions (d1,d2,d3). There are 200 points associated to each subspace, which are corrupted by zero-mean additive white noise of standard deviation σ and support in the orthogonal complement of each subspace...... 209 4.4 Mean inter-cluster connectivity in % over 100 independent trials for synthetic data randomly generated in three random subspaces of R9 of dimensions (d1,d2,d3). There are 200 points associated to each subspace, which are corrupted by zero-mean additive white noise of standard deviation σ and support in the orthogonal complement of each subspace...... 210

xiii LIST OF TABLES

4.5 Mean running time of each method in seconds over 100 independent trials for synthetic data randomly generated in three random subspaces of R9 of dimensions (d1,d2,d3). There are 200 points associated to each subspace, which are corrupted by zero-mean additive white noise of standard deviation σ =0.01 and support in the orthogonal complement of each subspace. The reported running time is the time required to compute the afﬁnity matrix, and it does not include the spectral clustering step. The experiment is run in MATLAB on a standard Macbook-Pro with a dual core 2.5GHz Processor and a total of 4GB Cache memory...... 211 4.6 Mean subspace clustering error in % over 100 independent trials for synthetic data randomly generated in four random subspaces of R9 of dimensions (8, 8, 5, 3). There are 200 points associated to each subspace, which are corrupted by zero-mean additive white noise of standard deviation σ = 0, 0.01, 0.03, 0.05 and support in the orthogonal complement of each subspace...... 215 4.7 Mean clustering error (E) in %, intra-cluster connectivity (C1), and inter- cluster connectivity (C2) in % for the Hopkins155 data set...... 218

xiv List of Figures

3.1 Various vectors and angles appearing in the proof of Theorem 3.4...... 44 3.2 Geometry of the optimality condition (3.98) and (3.114) for the case d = 1,D = 2,M = N = 5. The polytope Mob∗ + Conv( o ) + Span(b∗) ± 1 misses the point N Sign(xˆ>b∗)xˆ and so the optimality condition can not − be true for both b∗ = Span(xˆ) and φ large...... 63 3.3 Geometry of the optimality6⊥ S condition (3.98) and (3.114) for the case d = 1,D = 2,M = N = 5. A critical b∗ exists, but its angle from is 6⊥ S S small, so that the polytope Mob∗ +Conv( o )+Span(b∗) can contain the ± 1 point N Sign(xˆ>b∗)xˆ. However, b∗ can not be a global minimizer, since small− angles from yield large objective values...... 64 3.4 Geometry of the optimalityS condition (3.98) and (3.114) for the case d = 1,D = 2,N << M. Critical points b∗ do exist and moreover they can have large angle from . This is because6⊥ S N is small and so the poly- S tope Mob∗ +Conv( o1)+Span(b∗) contains the point N Sign(xˆ>b∗)xˆ. Moreover, such critical± points can be global minimizers.− Condition (3.97) of Theorem 3.12 prevents such cases from occuring...... 65 3.5 Various quantities associated to the performance of DPCP-r; see 3.2.5.1. Figure 3.5(a) shows whether condition (3.97) is true (white) or not§ (black). Figure 3.5(b) shows the angle from of nˆ 10 after 10 iterations of DPCP- r when (3.97) is true; the other casesS are mapped to black. Figure 3.5(c)

shows whether a random nˆ 0 satisﬁes φ0 >φ0∗, where φ0∗ is as in Thm. 3.15. Figure 3.5(d) shows the angle from of nˆ 10 for random nˆ 0. Figure 3.5(e) shows the angle from of the rightS singular vector of X˜ corresponding to the smallest singular value,S Figure 3.5(f) shows the corresponding angle of

nˆ 10, and Figure 3.5(g) plots φ0∗...... 83

xv LIST OF FIGURES

3.6 Outlier/Inlier separation in the absence of noise over 10 independent trials. The horizontal axis is the outlier ration defined as M/(N + M), where M is the number of outliers and N is the number of inliers. The vertical axis is the relative inlier subspace dimension d/D; the dimension of the ambient space is D = 30. Success (white) is declared by the existence of a threshold that, when applied to the output of each method, perfectly separates inliers from outliers...... 85 3.7 ROC curves as a function of noise standard deviation σ and outlier percentage R, for subspace dimension d = 25 in ambient dimension D = 30. The horizontal axis is False Positives ratio and the vertical axis is True Positives ratio. The number associated with each curve is the area above the curve; smaller numbers reflect more accurate performance...... 88 3.8 ROC curves as a function of noise standard deviation σ and outlier percentage R, for subspace dimension d = 29 in ambient dimension D = 30. The horizontal axis is False Positives ratio and the vertical axis is True Positives ratio. The number associated with each curve is the area above the curve; smaller numbers reflect more accurate performance...... 89 3.9 Average ROC curves and areas over the curves for different percentages R of outliers; see 3.2.5.3. Both inliers and outliers come from EYaleB. C1 means data are centered (C0 not centered), N1 means data are normalized (N0 not normalized)...... 91 3.10 Average ROC curves and areas over the curves for different percentages R of outliers; see 3.2.5.3. Inliers come from EYaleB, outliers from Cal- tech101. C1 means data are centered (C0 not centered), N1 means data are normalized (N0 not normalized)...... 92 3.11 ROC curves for three different projection dimensions, when there are 33% face outliers; data are centered but not normalized (C1 N0)...... 93 3.12 ROC curves for three different projection dimensions, when− there are 33% outliers from Caltech101; data are centered but not normalized (C1 N0). . 93 3.13 Clustering accuracy as a function of the number of hyperplanes n vs− relative dimension d/D vs data balancing (α). White corresponds to 1, black to 0...... 135 3.14 Clustering accuracy as a function of the number of hyperplanes n vs relative dimension d/D. Data balancing parameter is set to α =0.8...... 136 3.15 Clustering accuracy as a function of the number of hyperplanes n vs outlier ratio vs data balancing (α). White corresponds to 1, black to 0...... 137 3.16 Clustering accuracy as a function of the number of hyperplanes n vs outlier ratio. Data balancing parameter is set to α =0.8...... 138 3.17 Segmentation into planes of image 5 in dataset NYUdepthV2 without spatial smoothing. Numbers are segmentation errors...... 150 3.18 Segmentation into planes of image 5 in dataset NYUdepthV2 with spatial smoothing. Numbers are segmentation errors...... 151

xvi LIST OF FIGURES

3.19 Segmentation into planes of image 2 in dataset NYUdepthV2 without spatial smoothing. Numbers are segmentation errors...... 152 3.20 Segmentation into planes of image 2 in dataset NYUdepthV2 with spatial smoothing. Numbers are segmentation errors...... 153

4.1 A union of two lines and one plane in general position in R3...... 161

4.2 The geometry of the unique degree-2 polynomial p(x)=(b1>x)(f >x) that vanishes on 1 2 3. b1 is the normal vector to plane 1 and f is the normal vectorS to∪S the∪S plane spanned by lines and .S ...... 163 H23 S2 S3 4.3 (a): The plane spanned by lines 2 and 3 intersects the plane 1 at the line . (b): Intersection of the originalS subspaceS arrangement =S S4 A S1 ∪S2 ∪S3 with the intermediate ambient space (1), giving rise to the intermediate V1 subspace arrangement (1) = . (c): Geometry of the unique A1 S2 ∪S3 ∪S4 degree-3 polynomial p(x)=(b2>x)(b3>x)(b4>x) that vanishes on 2 3 (1) S ∪S ∪ 4 as a variety of the intermediate ambient space 1 . bi i, i =2, 3, 4. . 175 4.4 ClusteringS error ratios for both 2 and 3 motionsV in Hopkins155,⊥S ordered increasingly for each method. Errors start from the 90-th smallest error of each method...... 219

xvii Chapter 1

Introduction

1.1 Modeling data with linear subspaces

1.1.1 Modeling data with a single subspace

In many ﬁelds of science and engineering, datasets can naturally be viewed as subsets of

a coordinate vector space RD, with each data point being a vector of D coordinates. For

example, a digital grayscale image can be viewed as a coordinate vector having as many

coordinates as the number of pixels. It is then intuitively expected that if a collection of

images have similar content, then their representation in RD would not be arbitrary, rather it would exhibit a special structure, reﬂecting the fact that these points correspond to similar images. It is further expected, that the more similar the images are, the simpler the structure

1 CHAPTER 1. INTRODUCTION of their representation in RD will be. Then knowledge of this structure can potentially be used to simplify the representation of the dataset, i.e., for using less than D coordinates, or for comparing the dataset to another dataset, and so on.

Since RD is a geometric object naturally generalizing the three-dimensional space, it is reasonable, in our effort to formalize concepts such as structure and its attributes special or simpler, to consider geometric objects of RD itself. Among various possibilities, linear subspaces have proved both simple and effective. Indeed, the question of ﬁnding a linear subspace of dimension d < D, that passes as close as possible to a given collection of points of RD, is an old one, and its classical solution, known as Principal Component Analysis

(PCA) [50, 78], is still one of the most popular techniques in data analysis, in areas as diverse as engineering [74], economics and sociology [116], chemistry [57], physics [68], and genetics [79] to name a few; see [55] for more applications.

The success of PCA is largely due to two facts: First, the hypothesis that data lie close to a proper linear subspace of the ambient space is a valid one in many applications, e.g., face images of the same individual under ﬁxed pose but varying illumination conditions lie close to a 9-dimensional linear subspace [4], trajectories across video frames of image points that belong to the same rigid body lie close to a 3-dimensional afﬁne subspace [94], and point correspondences between two views of the same static scene lie close to an 8- dimensional linear subspace [47]. Second, the error metric used in PCA to penalize the modeling errors is the Euclidean distance. This allows for computation of the optimal subspace in closed form solely via linear algebraic algorithms, in particular via the Singular

2 CHAPTER 1. INTRODUCTION

Value Decomposition (SVD), for which very efﬁcient software has been developed through the years.

1.1.2 Modeling data with multiple subspaces

Even though the data model of a single linear subspace is a fundamental one, there are many occasions, where modeling with multiple subspaces is more precise. Intuitively, this is the case when there are more than one classes present in the data.

As a practical example, consider a moving surveillance camera taking a video of a moving car, with the rest of the background being fixed. As already mentioned, point trajectories corresponding to the car lie close to a 3-dimensional affine subspace, because they belong to the same rigid body motion. On the other hand, the motion of the camera induces a motion to the background, so that in fact there are two linear subspaces associated with the data: one corresponding to the motion of the car (original motion of the car superimposed with the motion of the camera), and one corresponding to the motion of the background. Thus, the right data model for this example is a union of two 3-dimensional affine subspaces. As another example, a collection of face images of n individuals under

ﬁxed pose but varying illumination, is naturally modeled by a union of n 9-dimensional linear subspaces [4].

As it turns out, there is a variety of applications across diverse areas of study, where a union of subspaces, also known as a subspace arrangement, is a more appropriate model than a single subspace. Important applications include computer vision problems, such as

3 CHAPTER 1. INTRODUCTION

motion segmentation [107, 110], structure from motion [46] and multiple view geometry

[47], face clustering [27], and 3D point cloud analysis [81], as well as genomics [103], document clustering [87], and system identification [71]. As a consequence, the problem of fitting a subspace arrangement to a given set of points of RD has gained significant attention over the past 15 years [9,15,27,28,30,31,42,64,66,69,93,102,106,109–111,119], giving rise to the field of subspace clustering [105], also known as Generalized Principal

Component Analysis [112].

1.2 Challenges and the role of dimension

There are many challenges associated with learning linear subspaces from a given dataset.

To begin with, suppose that we are given a dataset that perfectly lies in the union of n linear

subspaces of known dimensions, and suppose that the goal is to ﬁnd the subspaces and clus-

ter the data according to their dataset membership. If the subspaces themselves were given,

then we could simply cluster the data points based on their distance to the given subspaces

(if a point has zero distance from a subspace, then it belongs to that subspace). If, on the

other hand, the clustering of the data according to their subspace membership were known,

then we could estimate the subspaces by applying PCA to each cluster. Nevertheless, if

both the subspaces and the clustering are unknown, then this is an especially challenging

problem, since it is not clear how to set up a rigorous solution.

On top of the intrinsic difﬁculty of subspace learning, most real world datasets are

contaminated with outliers and noise, and they may have missing entries. Moreover, both

4 CHAPTER 1. INTRODUCTION the number of subspaces and their dimensions are usually unavailable, and they need to be estimated from the data.

When the underlying subspaces are low-dimensional, additional structures are present in the dataset, such as low-rank components [31, 64, 66, 106, 118] or sparse self-expressive patterns [27, 28, 30, 54, 77, 84, 120], which can be elegantly used to overcome many of the aforementioned challenges. As an example, when the data lie close to a low-dimensional subspace and they are corrupted in an entry-wise fashion, the robust PCA method of [12] detects and corrects these corruptions, by decomposing the observed data to the sum of a low-rank matrix and a sparse matrix. However, when the underlying subspaces have high relative dimensions (the relative dimension of a subspace is the quotient of the subspace dimension over the dimension of the ambient space), these additional structures disappear, which makes dealing with outliers, handling unknown a-priori knowledge of the subspace dimensions, and designing scalable algorithms even more challenging, and in fact these challenges are currently largely unsolved.

1.3 Contributions of this thesis

This thesis addresses several challenges associated with learning subspaces of high relative dimension. The ﬁrst part of the thesis is devoted to a new subspace learning method called

Dual Principal Component Pursuit, which is shown to advance the state-of-the-art in learning a single subspace of high relative dimension in the presence of outliers, as well as in clustering multiple hyperplanes (subspaces of maximal relative dimension). The second

5 CHAPTER 1. INTRODUCTION part of the thesis is devoted to a new subspace clustering method called Filtrated Algebraic

Subspace Clustering, which is shown to advance the state-of-the-art of Algebraic Subspace

Clustering (ASC) [109–111], the latter being one of the best theoretically justiﬁable methods for clustering data drawn from a union of subspaces of high relative dimension.

1.3.1 Dual principal component pursuit (DPCP)

The ﬁrst part of this thesis (Chapter 3) introduces a new subspace learning method called

Dual Principal Component Pursuit (DPCP) [96, 98, 99]. The adjective dual refers to the fact that DPCP searches for a basis for the orthogonal complement of a subspace. As such, DPCP is naturally suited for subspaces of maximal relative dimension, for which, the orthogonal complement is a low-dimensional subspace. Nevertheless, DPCP can in principle be applied to subspaces of any dimension.

In a noiseless setting, the basic idea of DPCP is to search for a hyperplane that contains as many points of the dataset as possible. Such a hyperplane will be called a maximal hyperplane. As it turns out, any maximal hyperplane tends to contain all points coming from the same subspace, a property that proves to be valuable for subspace learning, particularly in high relative dimensions. For example, if the dataset consists of points lying in a single hyperplane and is corrupted by many outliers, then, if the data points are in general position, there is a unique maximal hyperplane coinciding with the hyperplane associated to the inliers. On the other hand, if the data are drawn from a union of hyperplanes, and once again are in general position (inside their respective hyperplanes), then any maximal

6 CHAPTER 1. INTRODUCTION hyperplane must be one of the hyperplanes associated to the data. This concept generalizes to the case where the subspaces are not hyperplanes, in which case the objective is to compute a basis for the orthogonal complement of each of the underlying subspaces.

We cast the computational search for maximal hyperplanes as an `1 minimization problem on the sphere, whose global minimizer is ideally the normal vector to such a hyperplane. While the objective function is convex, the constraint is not, which renders the problem non-convex. An extensive mathematical analysis culminates in theorems that describe conditions, under which any global solution of the non-convex problem is orthogonal to the underlying subspace (or orthogonal to the dominant one if there are more than one subspaces), thus serving as a guarantee for a recursive computation of the orthogonal complement of the subspace.

As far as the non-convex optimization problem is concerned, the thesis investigates several different techniques for solving it. Among them, emphasis is placed on minimizing the `1 objective on an adaptively learned sequence of tangent spaces to the sphere, which computationally amounts to solving a recursion of linear programs. The mathematical analysis provides theorems that establish convergence of the recursion in a ﬁnite number of iterations to a vector orthogonal to the underlying subspace. Other scalable techniques for solving the non-convex problem are also investigated.

The thesis discusses extensive experiments on synthetic data, which demonstrate the superiority of DPCP to the state-of-the-art in learning a single subspace of high relative dimension in the presence of outliers, or clustering data drawn from a union of hyperplanes.

7 CHAPTER 1. INTRODUCTION

Experiments using real data show that DPCP is competitive to the state-of-the-art in ﬁtting

planes to 3D point clouds, or detecting outliers that corrupt a collection of face images of a

single individual.

1.3.2 Filtrated algebraic subspace clustering

The second part of this thesis (Chapter 4) is concerned with Algebraic Subspace Clustering

(ASC) [109–111], which is one of the most theoretically appropriate methods for clustering

data drawn from a union of subspaces of high relative dimension. Moreover, it has very

interesting connections to algebraic geometry [45,48] and commutative algebra [1,25,73],

which are fascinating branches of mathematics very rarely used in machine learning (see

also [67] for another recent such related method). As such, ASC is a very appealing method

to study and improve towards making progress in learning subspaces of high relative di-

mension.

The main idea behind ASC is that a (transversal) union of linear subspaces is uniquely

characterized by a set of vanishing polynomials, which can be estimated from the data.

Once the vanishing polynomials are available (or their estimates), a subspace associated to the data can be obtained as the linear subspace generated by the gradients of all the polynomials evaluated at any (non-singular) point in the subspace.

There are two main challenges that have been preventing ASC from being applicable to modern datasets. First, one needs to estimate the correct number of linearly independent vanishing polynomials of degree equal to the number of the subspaces. Numerically, this

8 CHAPTER 1. INTRODUCTION

corresponds to estimating the rank of a matrix, which is a precarious task in the presence of

even moderate amounts of noise, particularly, since, it is known that the clustering accuracy

is very sensitive to the estimation of this rank. The second main challenge of ASC is

that, even though the vanishing polynomials can be computed using the Singular Value

Decomposition (SVD), the SVD itself is to be applied on a multivariate Vandermode matrix

whose dimension is exponential in the number of subspaces and the ambient dimension. In

other words, ASC is characterized by an exponential computational complexity.

The main contribution of the second part of this thesis is a new algebraic algorithm,

called Filtrated Algebraic Subspace Clustering (FASC) [95, 101]. The key idea behind

FASC is to construct a nested sequence := of subspace arrangements A A0 ⊃ A1 ⊃ ··· , ,... (i.e., a descending ﬁltration), with each arrangement embedded in an inter- A0 A1 Ai

mediate ambient space i of strictly smaller dimension than i 1, where is the original V V − A union of subspaces and = RD is the original ambient space. Given a point in one of V0 the subspaces, say x , the method constructs a nested sequence of subspace ar- ∈S⊂A rangements with the following properties: (1) is contained in A0 ⊃ A1 ⊃···⊃Ac S each intermediate arrangement, i.e., ; (2) the sequence stabilizes at after a finite S⊂Ai S number of steps, i.e., = , and (3) the codimension of the subspace is equal to the num- S Ac ber of steps of the filtration, i.e., c = codim( ). As a consequence, one can identify the S subspace containing each data point by constructing a filtration at that point. The numerical implementation of FASC is based on using the filtration of two distinct points in the dataset to determine whether they lie in the same subspace or not. This is the basic concept behind

9 CHAPTER 1. INTRODUCTION

deﬁning a pairwise afﬁnity between all points in the dataset, and subsequently obtaining

the data clusters by means of standard spectral clustering. Interestingly, the resulting al-

gorithm, called Filtrated Spectral Algebraic Subspace Clustering (FSASC) [97], not only

dramatically improves upon the performance of earlier ASC methods, thus addressing the

challenge of robustness of ASC to noise, but also exhibits state-of-the-art performance in

the Hopkins155 dataset [94], which is a benchmark dataset for motion segmentation.

A second contribution of this second part of this thesis is a rigorous algebraic-geometric

study of the algebraic clustering of afﬁne subspaces [100]. In fact, very little theory exists

in the subspace clustering literature as far as dealing with afﬁne subspaces is concerned.

Usually, afﬁne subspaces are treated in the machine learning literature by reduction to the

case of linear subspaces, by means of appending an extra coordinate in the coordinate rep-

resentation of the data [104]. This is known as projectivization, homogenization or more

casually as the homogenization trick. Even though homogenization has been successful, it has been viewed only as a heuristic, not supported by any actual theory. This thesis establishes the theoretical correctness of dealing with afﬁne subspaces through homogeneous coordinates, when clustering is done algebraically.

1.4 Notation

We begin with some general notation. The notation ∼= stands for isomorphism in whatever category the objects lying to the left and right of the symbol belong to. The notation ' denotes approximation. For any positive integer n let [n] := 1, 2,...,n . For any positive { } 10 CHAPTER 1. INTRODUCTION number α let α denote the smallest integer that is greater than α. For sets , , the d e A B set is the set of all elements of that do not belong in . The right null space A\B A B of a matrix B is denoted by (B). If is a subspace of RD, then dim( ) denotes the N S S dimension of and π : RD is the orthogonal projection of RD onto . For vectors S S → S S D D b, b0 R we let ∠b, b0 be the angle between b and b0. If b is a vector of R and ∈ S a linear subspace of RD, the principal angle of b from is ∠b,π (b). The symbol S S ⊕ denotes direct sum of subspaces. The orthogonal complement of a subspace in RD is S D D ⊥. If y ,..., y are elements of R , we denote by Span(y ,..., y ) the subspace of R S 1 s 1 s D 1 D D spanned by these elements. S − denotes the unit sphere of R . For a vector w R we ∈ define wˆ := w/ w , if w = 0, and wˆ := 0 otherwise. With a mild abuse of notation k k2 6 we will be treating on several occasions matrices as sets, i.e., if X is D N and x a point × of RD, the notation x X signifies that x is a column of X . Similarly, if O is a D M ∈ × matrix, the notation X O signifies the points of RD that are common columns of X ∩ and O. Also, the shorthand RHS stands for Right-Hand-Side, and similarly for LHS. The notation Sign denotes the sign function Sign : R 1, 0, 1 defined as → {− }

x/ x if x =0, Sign(x)=  | | 6 (1.1)   0 if x =0.

 The subdifferential of the `1-norm 

D z =(z ,...,z )> z = z (1.2) 1 D 7→ k k1 | i| i=1 X

11 CHAPTER 1. INTRODUCTION is a set-valued function on RD deﬁned as

Sign(x) if x =0, Sgn(x)=  6 (1.3)   [ 1, 1] if x =0. −  Next, we establish some more specialized notation in support of the algebraic-geometric aspects of the thesis. We let R[x] = R[x1,...,xD] be the polynomial ring over the real numbers in D variables. We use x to denote the vector of variables x = (x1,...,xD),

D while we reserve x to denote a data point x = (χ1,...,χD) of R . We denote by R[x]`

1 the set of all homogeneous polynomials of degree ` and similarly R[x] ` the set of all ≤ homogeneous polynomials of degree less than or equal to `. R[x] is an inﬁnite dimensional real vector space, while R[x]` and R[x] ` are ﬁnite dimensional subspaces of R[x] of di- ≤

`+D 1 `+D mensions (D) := − and , respectively. We denote by R(x) the ﬁeld of M` ` ` all rational functions over R and variables x ,...,x . If p ,...,p is a subset of R[x], 1 D { 1 s} we denote by p ,...,p the ideal generated by p ,...,p (see Deﬁnition 4.49). If is a h 1 si 1 s A subset of RD, we denote by the vanishing ideal of , i.e., the set of all elements of R[x] IA A that vanish on and similarly ,` := R[x]` and , ` := R[x] `. Finally, for A IA IA ∩ IA ≤ IA ∩ ≤ D a point x R , and a set R[x] of polynomials, x is the set of gradients of all the ∈ I ⊂ ∇I| elements of evaluated at x. I

1A polynomial in many variables is called homogeneous if all monomials appearing in the polynomial have the same degree.

12 Chapter 2

Prior Art

The importance of modeling data with linear subspaces had been recognized more than 100 years ago with the seminal work of Pearson on “Lines and planes of closest ﬁt to systems of points in space” [78]. This led to the famous Principal Component Analysis (PCA) [50,

78], whose solution is achieved in closed form through the Singular Value Decomposition

D N D (SVD). Speciﬁcally, if the data matrix is X R × , consisting of N points in R , then ∈ the linear subspace of dimension d that passes closest to all points of X in the Euclidean sense, is the linear space spanned by the ﬁrst d left singular vectors of X 1.

D M However, the presence of even a few outliers O R × may signiﬁcantly bias the ∈ estimated subspace away from the true underlying subspace. Indeed, it is known that this

`2-optimal linear subspace is affected by the contribution of each and every point in the

1In principle, one should ﬁrst center the data to have zero mean, and then search for a linear subspace that passes close to them.

13 CHAPTER 2. PRIOR ART

˜ D (N+M) now corrupted dataset X = [X O]Γ R × , where Γ is an unknown permutation ∈ indicating that we do not know which point is an inlier and which point is an outlier. Over

the past 36 years many research efforts have been devoted to robustly learning a linear

subspace from the data, while mitigating the effect of outliers. Traditional methods, such as

RANSAC [34], Inﬂuence-based Detection, Multivariate Trimming and M-Estimators [52,

55], use techniques from robust statistics; such methods are usually based on non-convex optimization, are sensitive to initialization and admit limited theoretical guarantees. On other hand, over the past 10 years methods based on convex optimization [12, 62, 84, 118] have gained signiﬁcant popularity, due to their efﬁcient implementations and theoretical guarantees. The reader is referred to 2.1 for a brief review of single subspace learning § methods. This review is by no means intended to be exhaustive; rather it focuses on either highly popular methods, such as RANSAC [34], or methods that are conceptually distinct from other methods, and whose techniques are intellectually related with the techniques used in this thesis.2

When the outliers are themselves well modeled by a linear subspace, as is for example the case of background subtraction in video surveillance [53], or when there are multiple classes associated to the data, as is the case in motion segmentation [94], it is more precise to learn multiple subspaces from the data, instead of a single subspace. This has led to a research ﬁeld known as subspace clustering [105]. In its most fundamental form, the subspace clustering problem assumes that the data are drawn from n lin-

2 For an extensive literature review on single subspace learning the reader is referred to [61], or to [3, 33] and references therein for online methods.

14 CHAPTER 2. PRIOR ART

D ˜ D N ear subspaces ,..., of R , and so the data matrix X R × has the structure S1 Sn ∈

˜ D Ni X = [X X X ]Γ, where X R × are N points coming from subspace , 1 2 ··· n i ∈ i Si i =1,...,n, and Γ is an unknown permutation indicating that we do not know which point comes from which subspace. Then the goal is to ﬁnd a basis for each of the underlying subspaces , as well as cluster the data points X˜ according to their subspace membership. By Si now, a large variety of subspace clustering methods has appeared in the literature including algebraic [109–111], statistical [42, 93], information-theoretic [70], iterative [9, 102, 121], geometric and spectral techniques [6,15,17,27,28,30,31,56,64,66,106,114]; see 2.1 for § a brief review.

Overall, when the subspace associated to the data is low-dimensional, methods such as [84,118] are known to successfully detect outliers present in the dataset. Similarly, when the data lie close to a union of low-dimensional subspaces, methods such as [30] have been shown to cluster the data with high accuracy. On the other hand, the case of high relative dimensions is considerably harder and a satisfactory solution remains yet to be established.

This is due to a number of challenges associated to subspaces of high relative dimension, which are described in 2.3. §

2.1 Learning a single subspace in the presence of outliers

RANSAC. One of the oldest and most popular outlier detection methods in PCA is Ran- dom Sampling Consensus (RANSAC) [34]. The idea behind RANSAC is simple: alternate between randomly sampling a small subset of the dataset and computing a subspace model

15 CHAPTER 2. PRIOR ART

for this subset, until a model is found that maximizes the number of points in the entire

dataset that ﬁt to it within some error. RANSAC is usually characterized by high learning

performance. However, it requires a high computational time, since it is often the case

that exponentially many trials are required in order to sample outlier-free subsets, and thus

obtain reliable models. Additionally, RANSAC requires as input an estimate for the di-

mension of the subspace as well as a thresholding parameter, which is used to distinguish

outliers from inliers; naturally the performance of RANSAC is very sensitive to these two

parameters.

`2,1-RPCA. Contrary to the classic principles that underlie RANSAC, modern methods for outlier detection in PCA are primarily based on convex optimization. One of the earliest and most important such methods, to be referred to as `2,1-RPCA, is the method of [118], which is in turn inspired by the Robust PCA algorithm of [12]. `2,1-RPCA computes a

3 (` + `2,1)-norm decomposition of the data matrix, instead of the (` + `1)-decomposition ∗ ∗

in [12]. More speciﬁcally, `2,1-RPCA solves the optimization problem

min L + λ E 2,1 , (2.1) L,E: X˜=L+E k k∗ k k which attempts to decompose the data matrix X˜ = [X O]Γ into the sum of a low-rank matrix L, and a matrix E that has only a few non-zero columns. The idea is that L is associated with the inliers, having the form L = [X 0D M ]Γ, and E is associated with ×

the outliers, having the form E = [0D N O]Γ. The optimization problem (2.1) is convex × and admits theoretical guarantees and efﬁcient ADMM [7] implementations. However, it

3 Here `∗ denotes the nuclear norm, which is the sum of the singular values of the matrix. Also, `2,1 is deﬁned as the sum of the euclidean norms of the columns of a matrix.

16 CHAPTER 2. PRIOR ART

is expected to succeed only when the intrinsic dimension d of the inliers is small enough

(otherwise [X 0D M ] will not be low-rank), and the outlier ratio is not too large (otherwise ×

[0D N O] will not be column-sparse). Finally, notice that `2,1-RPCA does not require as × input the subspace dimension d, because it does not directly compute an estimate for the

subspace. Rather, the subspace can be obtained subsequently by doing classic PCA on L, and now one does need an estimate for d.

SE-RPCA. Another state-of-the-art method, referred to as SE-RPCA, is based on the self-expressiveness property of the data matrix, a notion popularized by the work of [29,30] in the area of subspace clustering [105]. More speciﬁcally, observe that if a column of X˜ is an inlier, then it can in principle be expressed as a linear combination of d other columns of X˜, which are inliers. If the column is instead an outlier, then it will in principle require

D other columns to express it as a linear combination. The self expressiveness matrix C can be obtained as the solution to the convex optimization problem

min C s.t. X˜ = X˜C, Diag(C)= 0. (2.2) C k k1

Having computed the matrix of coefﬁcients C, and under the hypothesis that d/D is small, ˜ a column of X is declared as an outlier, if the `1 norm of the corresponding column of C

is large; see [84] for an explicit formula. SE-RPCA admits theoretical guarantees [84] and

efﬁcient ADMM implementations [30]. However, as is clear from its description, it is ex-

pected to succeed only when the relative dimension d/D is sufﬁciently small. Nevertheless,

in contrast to `2,1-RPCA, which in principle fails in the presence of a very large number of

outliers, SE-RPCA is still expected to perform well, since the existence of sparse subspace-

17 CHAPTER 2. PRIOR ART

preserving self-expressive patterns does not depend on the number of outliers present. Also,

similarly to `2,1-RPCA, SE-RPCA does not directly require an estimate for the subspace dimension d. Nevertheless, knowledge of d is necessary if one wants to furnish an actual subspace estimate. This would entail removing the outliers (a judiciously chosen threshold would also be necessary here) and doing PCA on the remaining points.

REAPER. A recently proposed single subspace learning method that admits an interesting theoretical analysis is the REAPER [62], which is conceptually associated with the optimization problem

L min (I Π)x˜ , s.t. Π is an orthogonal projection, Trace (Π)= d, (2.3) Π k D − jk2 j=1 X ˜ where x˜j is the j-th column of X . The matrix Π appearing in (2.3) can be thought of as

D d the product Π = UU >, where U R × contains in its columns an orthonormal basis ∈ for a d-dimensional linear subspace . As (2.3) is non-convex, [62] relaxes it to the convex S semi-deﬁnite program

min (ID P )x˜j , s.t. 0 P ID, Trace (P )= d, (2.4) P k − k2 ≤ ≤ j=1 X whose global solution P ∗ is subsequently projected in an ` sense onto the space of rank-d ∗

orthogonal projectors. It is shown in [62] that the orthoprojector Π∗ obtained in this way

is within a neighborhood of the orthoprojector corresponding to the true underlying inlier

subspace. One advantage of REAPER with respect to `2,1-RPCA and SE-RPCA, is that

its theoretical conditions do not explicitly require the inlier dimension d to be small. On

the other hand, contrary to `2,1-RPCA, REAPER does require a-priori knowledge of the

18 CHAPTER 2. PRIOR ART

inlier dimension d. Moreover, the semi-deﬁnite program (2.4) may become prohibitively

expensive to solve even for moderate values of the ambient dimension D. As a consequence, [62] proposed an Iteratively Reweighted Least Squares (IRLS) scheme to obtain a numerical solution of (2.4). Interestingly, it was shown in [62] that the objective value of this IRLS scheme converges to a neighborhood of the optimal objective value of problem

(2.4); nevertheless no other properties of this scheme seem to be known.

R1-PCA. Another related method is the so-called R1-PCA [23], which attempts to solve the following problem:

˜ min X UV , s.t. U >U = I, (2.5) U,V − 2,1

where U is an orthonormal basis for the estimated subspace, and V contains in its columns the low-dimensional representations of the points. Besides for alternating minimization with a power iteration scheme that converges to a local minimum, little else is known about how to solve the non-convex problem (2.5) to global optimality.

L1-PCA∗. Finally, the method L1-PCA∗ of [11] works with the orthogonal complement

of the subspace, but it is slightly unusual in that it learns `1 hyperplanes, i.e., hyperplanes that minimize the `1 distance to the points, as opposed to the Euclidean distance, e.g., used by the classic PCA, R1-PCA, or REAPER. More speciﬁcally, an `1 hyperplane learned by the data is a hyperplane with normal vector b that solves the problem

L ˜ min xj yj s.t. yj>b =0, j [L], (2.6) b SD−1; y RD,j [L] − 1 ∀ ∈ ∈ j ∈ ∈ j=1 X

˜ where yj is the representation of point xj in the hyperplane. Overall, no theoretical guaran-

19 CHAPTER 2. PRIOR ART

tees seem to be known for L1-PCA∗, as far as the subspace learning problem is concerned.

In addition, L1-PCA∗ requires the solution to quadratically many linear programs of size equal to the ambient dimension, which makes it computationally expensive.

2.2 Learning multiple subspaces

Self-expressiveness-based methods. One of the most popular families of subspace clustering methods is based on applying spectral clustering [115] to an N N afﬁnity matrix × W built by exploiting the self-expressiveness property of the data. This latter property says

˜ ˜ D Ni that when the data matrix X has the structure X = [X X X ], where X R × 1 2 ··· n i ∈ are N points coming from subspace , i = 1,...,n, then every column x˜ of X˜ can i Si j ˜ be written as a linear combination x˜j = X cj of other points in the dataset, with the j-th entry of the vector of coefﬁcients cj being zero. Then the idea is that if the relationship

X˜ = X˜C, Diag(C) = 0 is enforced together with a suitable regularization on C, the points that every point x˜j selects will be from the same subspace. As it turns out, deﬁn- ing W jj0 to be the absolute value of the j0-th entry of cj has proved to be a simple yet powerful afﬁnity. Popular choices for the regularization on C are the `1-norm, the nuclear norm, the Frobenius norm, or combinations thereof, leading to what is known as Sparse

Subspace Clustering (SSC) [27, 28, 30], Low-Rank Subspace Clustering [31, 64, 66, 106],

Least-Squares Subspace Clustering [69], and Elastic Net Subspace Clustering [54,77,120].

Under the typical assumption that the subspaces are low-dimensional and sufﬁciently separated, this family of methods admits theoretical guarantees regarding the correctness of

20 CHAPTER 2. PRIOR ART

the afﬁnity W , and its robustness to outliers and noise. Moreover, efﬁcient algorithmic

implementations are available.

Spectral Curvature Clustering (SCC). Another yet conceptually distinct method from

the ones discussed so far is Spectral Curvature Clustering (SCC) [15], which is theoreti-

cally suitable for clustering subspaces of equal dimension d. The main idea of SCC is to

build a (d+1)-fold tensor as follows. For each (d+1)-tuple of distinct points in the dataset,

say xj1 ,..., xjd+1 , the value of the tensor is set to

2 c (x ,..., x ) A p j1 jd+1 (2.7) (j1,...,jd+1) = exp 2 , − 2σ !

where cp(xj1 ,..., xjd+1 ) is the polar curvature of the points xj1 ,..., xjd+1 (see [15] for an explicit formula) and σ is a tuning parameter. Intuitively, the polar curvature is a multiple of the volume of the simplex of the d +1 points, which becomes zero if the points lie in the same d-dimensional subspace, and the further the points lie from any such subspace the larger their volume becomes. SCC obtains the subspace clusters by unfolding the tensor

A to an afﬁnity matrix, upon which spectral clustering is applied. As with ASC, the main

N bottleneck of SCC is computational, since in principle all d+1 entries of the tensor need to be computed. Even though the combinatorial complexity of SCC can be reduced, this comes at the cost of signiﬁcant performance degradation.

RANSAC. On the other end of the spectrum, the classical method of RANSAC, described in 2.1, is often used to learn more than one linear subspaces. More speciﬁcally, § RANSAC attempts to identify a single subspace at a time, by treating points lying in Si other subspaces as outliers. Once a subspace estimate for one subspace, say , is ob- S1 21 CHAPTER 2. PRIOR ART

tained, points lying close to the subspace are removed and a second subspace is sought. As

in the case of a single subspace with outliers, RANSAC is sensitive to its thresholding pa-

rameter, and moreover, its efﬁciency depends on how big the probability is that d randomly

selected points lie close to the same underlying subpsace. In turn, this probability depends

on how large D is as well as how balanced or unbalanced the clusters are. If D is small,

then RANSAC is likely to succeed with few trials. The same is true if one of the clusters,

say X , is highly dominant, i.e., N >> N , i 2, since in such a case, identifying is 1 1 i ∀ ≥ S1 likely to be achieved with only a few trials. On the other hand, if D is large and the Ni are of the same order of magnitude, then exponentially many trials are required, and RANSAC becomes inefﬁcient.

K-Subspaces. Another classical method for clustering subspaces is the so-called K-

Subspaces, which was proposed in [9, 102]. K-Subspaces attempts to minimize the non-

convex objective function

n N 2 (B ,..., B ; s ,...,s ) := s (i) B>x , (2.8) JKS 1 n 1 N j i j 2 i=1 " j=1 # X X

where s : [n] 0, 1 is the subspace assignment of point x , i.e., s (i)=1 if and j → { } j j

D ci only if point x has been assigned to subspace i, and B R × is an orthonormal basis j i ∈ for the orthogonal complement of subspace .4 Because of the non-convexity of (2.8), Si the typical way to perform the optimization is by alternating between assigning points to

clusters, i.e., given the B assigning x to its closest subspace (in the euclidean sense), { i} j 4It should be noted here that an alternative formulation exists, where one searches directly for the basis of the subspaces together with the representation of each point to its closest subspace; the two formulations are equivalent, with the one chosen in the main text being the more efﬁcient in the case of high relative dimensions.

22 CHAPTER 2. PRIOR ART and ﬁtting subspaces, i.e., given the segmentation s , computing the best ` subspace for { j} 2 each cluster by means of Singular Value Decomposition on each cluster.

The theoretical guarantees of K-Subspaces are limited to convergence to a local minimum in a finite number of steps. Moreover, knowledge of the subspace dimensions di (or codimensions c = D d ) is required. Finally, even though the alternating minimization i − i in K-Subspaces is computationally efficient, in practice several restarts are typically used, in order to select the best among multiple local minima. In fact, the higher the ambient dimension D is, the more restarts are required, which significantly increases the computational burden of K-Subspaces. Moreover, K-Subspaces is robust to noise, but it is not robust to outliers, since the update of the subspaces is traditionally done in an `2 sense.

For this reason, it has been recently considered to replace the `2-norm with the `1-norm, leading to what is known as Median K-Flats [60, 121].

RANSAC/K-Subspaces Hybrids. In principle, any single subspace learning method can be used to perform subspace clustering, either via a RANSAC-style or a via a K-

Subspaces-style scheme or a combination of both. For example, if is a method that takes M a dataset and fits to it a linear subspace, for instance REAPER ( 2.1), then one can use § M to compute a first subspace, remove the points in the dataset lying close to it, then compute a second subspace and so on (RANSAC-style). Alternatively, one can start with a random guess for n subspaces, cluster the data according to their distance to these subspaces, and then use (instead of the classic SVD) to fit a new subspace to each cluster, and so on M (K-Subspaces-style). Generally speaking, such strategies can be powerful, provided that

23 CHAPTER 2. PRIOR ART

one has a good initialization in the case of K-Subspaces variants, but more importantly,

an accurate estimate for the subspace dimensions. In practice, this latter requirement is

usually a weakness, because estimating the subspace dimensions is a hard problem.

Algebraic Subspace Clustering. ASC was originally proposed in [110] with the goal of solving the subspace clustering problem in closed form, which was ﬁrst achieved in [110] for the case of a union of hyperplanes (which are subspaces of maximal relative dimension), and later in [111] for subspaces of any dimension.

The idea behind ASC is to ﬁt a polynomial p(x ,...,x ) R[x ,...,x ] of degree n 1 D ∈ 1 D to the data, where n is the number of hyperplanes, and x1,...,xD are polynomial indeter- minates. In the absence of noise, this polynomial can be shown to have the form

p(x)=(b>x) (b>x), x := [x , ,x ]> , (2.9) 1 ··· n 1 ··· D where bi is the normal vector to the i-th hyperplane. This reduces the problem of learning the n hyperplanes to that of factorizing p(x) to the product of linear factors; this was elegantly done in [110]. When the data are contaminated by noise, the ﬁtted polynomial need no longer be factorizable. This difﬁculty was circumvented in [111], where it was shown that the gradient of the polynomial evaluated at point xj is a good estimate for the normal vector of the hyperplane that xj lies closest to. This idea generalizes to the case

of subspaces of arbitrary dimensions, and an elegant theorem [20, 72, 111] assures that if

p1,...,ps is a basis of polynomials of degree n that vanish on the union of the subspaces

, and y is a point of , then = Span( p y , , p y )⊥; remarkably, S1 ∪···∪Sn i Si Si ∇ 1| i ··· ∇ s| i this formula is valid irrespectively of what the subspace dimensions are.

24 CHAPTER 2. PRIOR ART

Even though the theoretical guarantees of ASC are fascinating, its practical application

has been limited mainly due to two reasons. First, available implementations of ASC are

sensitive to noise or are suitable only for hyperplanes. Second, computing vanishing poly-

nomials of degree n is a task whose complexity is exponential in the number of subspaces n and ambient dimension D.

2.3 Challenges in high relative dimensions

When the subspaces associated to the data have low dimension, the task of learning them can be successfully done by existing methods [12, 30, 84, 118]. Despite this, comparably little is known about the high relative dimension setting, which only a few methods seem able to handle (mainly on an ad-hoc level). In this paragraph some of the key challenges in the high relative dimension setting are summarized.

1. Theoretical guarantees: Subspaces of high relative dimension tend to intersect,

e.g., two general hyperplanes in ambient dimension D always intersect in a (D 2)- − dimensional subspace. As a consequence, clustering points from the union of such

subspaces is intrinsically harder than in low relative dimensions, as the membership

of points lying close to the intersection is difﬁcult to be resolved. This fact vio-

lates a major assumption of most modern subspace clustering methods (e.g., sparse

and low-rank methods), which require the subspaces to be sufﬁciently separated

[27, 28, 30, 31, 54, 64, 66, 69, 77, 106, 120] in order to provide guarantees of correct-

25 CHAPTER 2. PRIOR ART

ness. On the other hand, methods that do admit theoretical guarantees for the high

relative dimension setting, such as Algebraic Subspace Clustering (ASC) [109–111]

and Spectral Curvature Clustering (SCC) [15], are plagued by non-scalable imple-

mentations. Finally, although the K-Subspaces [9, 102] method is applicable to the

high relative dimension setting and scales reasonably well, it is based on a non-

convex objective function, lacks sufﬁcient theoretical guarantees and is sensitive to

initialization.

2. Robustness to outliers: Informally, the average angle of a random point from a

randomly chosen subspace decreases as the subspace dimension increases. As a

consequence, distinguishing inliers from outliers becomes a challenge in the high

relative dimension regime. For example, state-of-the-art robust PCA methods [84,

118] perform well only when the subspace dimension is sufﬁciently small. Moreover,

RANSAC [34] can deal with high relative dimensions, but outliers are troublesome

unless the ambient dimension is small. Finally, ASC [109–111] and K-Subspaces

[9,102] are sensitive to outliers since they rely on SVD-based PCA, which is known

to lack robustness to outliers.

3. A priori knowledge of subspace dimensions: When the subspaces have low relative

dimension, the dataset exhibits additional structure, such as low-rank components

[31,64,66,106,118] or sparse self-expressive patterns [27,28,30,54,77,84,120]. This

additional structure makes it possible to detect outliers or cluster the data without

26 CHAPTER 2. PRIOR ART

needing to know the subspace dimensions in advance. On the other hand, when the

subspaces are of high relative dimension, this additional structure disappears. Thus,

methods such as RANSAC [34], K-Subspaces [9,102], SCC [15] and REAPER [62]

require a priori knowledge of the subspace dimensions. Remarkably, ASC [111] does

not require such knowledge, but this advantage comes at the cost of an exponential

computational complexity.

4. Non-convex optimization: Existing methods that are in principle suitable for sub-

spaces of high relative dimension and at the same time scale well, e.g., K-Subspaces

[9, 102] or Median K-Flats (MKF) [121], involve non-convex optimization. As a

consequence, the theoretical guarantees of such methods are usually limited to con-

vergence to a local minimum. In addition, one can empirically observe that their

performance is often very sensitive to initialization; this sensitivity becomes particu-

larly pronounced in large ambient dimensions.

5. Scalability: Some methods overcome several of the above challenges, but this comes

at the cost of high computational complexity. For example, ASC [109–111] has

strong theoretical guarantees but also exponential complexity, due to the ﬁtting of

polynomials of degree n to the data. RANSAC [34] is robust to outliers but may

require an exponentially large number of trials in order to sample outlier-free subsets

of the data. SCC [15] can, in theory, cluster subspaces of equal high relative di-

mension but it requires the computation of exponentially many relations between the

27 CHAPTER 2. PRIOR ART

data. On the other hand, K-Subspaces [9,102] scales fairly well, but it is sensitive to

initialization, outliers, and requires advanced knowledge of the subspace dimensions.

In summary, while there has been much progress in subspace learning and clustering, most of the existing work does not apply to the case of subspaces of high relative dimension.

The goal of this thesis is to address several of the fundamental challenges mentioned above, by introducing Dual Principal Component Pursuit (Chapter 3) and Filtrated Algebraic

Subspace Clustering (Chapter 4) .

28 Chapter 3

Dual Principal Component Pursuit

3.1 Introduction

This chapter contains the ﬁrst main contribution of this thesis, which is a method for ro-

bust subspace learning and clustering, called Dual Principal Component Pursuit (DPCP).

DPCP aims at learning a subspace by explicit computation of its orthogonal complement; this is a natural approach, when the subspace has high relative dimension, in which case its orthogonal complement is a low-dimensional linear subspace. Even though traditional methods such as PCA [50, 55, 78], RANSAC [34] or K-Subspaces [9, 60, 102, 121] can be conﬁgured to estimate the orthogonal complement of a subspace, the advantage of DPCP over such methods is that DPCP computes a basis for the orthogonal complement via solv-

29 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

ing a non-convex problem to global optimality.1

Consider for a moment the case of single subspace learning in the presence of outliers,

and suppose that there is no noise, i.e., that the inliers perfectly lie inside a linear subspace

of dimension d < D (here d can be anything). Then the key idea of DPCP comes from S

the fact that the inliers lie inside any hyperplane = Span(b )⊥ that contains the linear H1 1 subspace associated to the inliers. This suggests that, instead of attempting to ﬁt directly S a low-dimensional linear subspace to the entire dataset, as done e.g. in [118], we can search for a maximal hyperplane that contains as many points of the dataset as possible. When H1 the data are in general position2, such a maximal hyperplane will contain the entire set of inliers together with possibly a few outliers. After removing the points that do not lie in that hyperplane, the robust subspace learning problem is reduced to one with a potentially much smaller outlier percentage than in the original dataset. In fact, the number of outliers in the new dataset will be precisely D d 1 D 2, and this upper bound can be used − − ≤ − to dramatically facilitate the outlier detection process using existing methods.

As an example, suppose we have N = 1000 inliers lying in general position inside a

90-dimensional linear subspace of R100. Suppose that the dataset is corrupted by M =

1000 outliers lying in general position in R100. Then among all hyperplanes of R100, the hyperplanes that contain as many points as possible from the dataset (maximal hyperplanes) must contain all 1000 inliers. On the other hand, since the dimensionality of the inliers is

1A recent method that also works with the orthogonal complement of the subspace is [11], for which, however, no theoretical guarantees seem to be known; see 2.1. 2By general position we mean that there are no relations§ between the data points, other than those implied by the membership of the inliers to the subspace . S

30 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

90 and the dimensionality of the hyperplane is 99, there are only 99 90 = 9 linearly − independent directions left for the hyperplane to ﬁt, i.e., such a hyperplane will contain

exactly 9 outliers (it can not contain more outliers since this would violate the general

position hypothesis). If we remove the points of the dataset that do not lie in the hyperplane,

then we are left with 1000 inliers and only 9 outliers; interestingly, a simple application of

RANSAC will identify the remaining 9 outliers in only a few trials.

Alternatively, one can consider ﬁnding a sequence of c = D d orthogonal maximal −

hyperplanes = Span(b )⊥ with b b , i = j, thus leading to a Dual Principal Com- Hi i i ⊥ j 6 ponent Analysis (DPCA) of the dataset, in the sense that the inlier subspace is precisely S equal to c . Finally, when the data come from multiple subspaces, every maximal i=1 Hi hyperplaneT contains all points coming from one of the subspaces and the ideas presented

above can be adapted for the purpose of subspace clustering.

It follows from the above discussion that the problem of searching for maximal hyper-

planes is an important one. As it turns out, we can formalize this task as an `0 cosparsity-

type problem [75], which we will further relax to a non-convex `1 problem on the sphere. A large part of this chapter is devoted to showing that every global solution of this latter DPCP problem is a vector orthogonal to a subspace associated with the data. We further consider solving this problem by a recursion of convex relaxations, each of which is computationally equivalent to a linear program. We show that under mild conditions this recursion converges in a ﬁnite number of steps to a vector orthogonal to a subspace associated to the data. In particular, for the case of hyperplanes, the recursion converges to the global

31 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

minimum of the `1 non-convex problem. Alternative methods suitable for large-scale data

are also explored for solving the DPCP problem, and extensive experiments are performed

on both synthetic and real data, demonstrating the merit of DPCP.

3.2 Single subspace learning with outliers via DPCP

3.2.1 Problem formulation

In this section we formulate the problem of interest. We begin by establishing our data

model in 3.2.1.1, while in 3.2.1.2 we motivate the problem at a conceptual level. Finally, § § in 3.2.1.3 we formulate our problem as an optimization problem. §

3.2.1.1 Data model

We employ a deterministic noise-free data model, under which the given data is

˜ D L X =[X O]Γ =[x˜ ,..., x˜ ] R × , (3.1) 1 L ∈

D N D 1 where the inliers X =[x ,..., x ] R × lie in the intersection of the unit sphere S − 1 N ∈ with an unknown proper subspace of RD of unknown dimension 1 d D 1, and S ≤ ≤ − D M D 1 the outliers consist of M points O = [o ,..., o ] R × that lie on the sphere S − . 1 M ∈ The unknown permutation Γ indicates that we do not know which point is an inlier and which point is an outlier. Finally, we assume that the points X˜ are in general position, in the sense that there are no relations between the columns of X˜ except the ones implied by the inclusions X and X˜ RD. In particular, every D-tuple of columns of X˜ such ⊂ S ⊂ 32 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

that at most d points come frorm X is linearly independent. Notice that as a consequence

every d-tuple of inliers and every D-tuple of outliers are linearly independent, and also

X O = . Finally, to avoid degenerate situations we will assume that N d +1 and ∩ ∅ ≥ M D d.3 ≥ −

3.2.1.2 Conceptual formulation

Notice that we have made no assumption on the dimension of : indeed, can be anything S S from a line to a (D 1)-dimensional hyperplane. Ideally, we would like to be able to − partition the columns of X˜ into those that lie in and those that don’t. But under such S generality, this is not a well-posed problem since X lies inside every subspace that contains

, which in turn may contain some elements of O. In other words, given X˜ and without S any other a priori knowledge, it may be impossible to correctly partition X˜ into X and O.

Instead, it is meaningful to search for a linear subspace of RD that contains all of the inliers

and perhaps a few outliers. Since we do not know the intrinsic dimension d of the inliers, a natural choice is to search for a hyperplane of RD that contains all the inliers.

Problem 1. Given the dataset X˜ = [X O] Γ, ﬁnd a hyperplane that contains all the H inliers X .

Notice that hyperplanes that contain all the inliers always exist: any non-zero vector

b inside the orthogonal complement ⊥ of the linear subspace associated to the inliers S S 3If the number of outliers is less than D d, then the entire dataset is degenerate, in the sense that it lies in a proper hyperplane of the ambient space.− In such a case we can reduce the coordinate representation of the data and eventually satisfy the stated condition.

33 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

deﬁnes a hyperplane (with normal vector b) that contains all inliers X . Having such a hyperplane at our disposal, we can partition our dataset as X˜ = X˜ X˜ , where X˜ H1 1 ∪ 2 1 are the points of X˜ that lie in and X˜ are the remaining points. Then by deﬁnition of H1 2 , we know that X˜ will consist purely of outliers, in which case we can safely replace H1 2 ˜ ˜ our original dataset X with X 1 and reconsider the problem of robust subspace learning on

X˜ . We emphasize that X˜ will contain all the inliers X together with precisely D d 1 1 1 − − outliers 4, a number which may be dramatically smaller than the original number of outliers, especially when d is large. If d is small, then one may apply existing methods such as [118],

[84] or [34] to identify the remaining outliers. Alternatively, if d is known, one may repeat the above process c = codim = D d times, until c linearly independent hyperplanes S − ,..., have been found that contain all the inliers, in which case = c . H1 Hc S k=1 Hk T

3.2.1.3 Hyperplane pursuit by `1 minimization

In this section we propose an optimization framework for computing a hyperplane that contains all the inliers. To proceed, we need a deﬁnition.

Deﬁnition 3.1. A hyperplane of RD is called maximal with respect to the dataset X˜, if H ˜ it contains a maximal number of data points in X , i.e., if for any other hyperplane † of H D ˜ ˜ R we have that Card(X ) Card(X †). ∩H ≥ ∩H

In principle, hyperplanes that are maximal with respect to X˜ always solve Problem 1.

4This comes from the general position hypothesis.

34 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Proposition 3.2. Suppose that N d +1 and M D d, and let be a hyperplane that ≥ ≥ − H is maximal with respect to the dataset X˜. Then contains all the inliers X . H

Proof. By the general position hypothesis on X and O, any hyperplane that does not contain X can contain at most D 1 points from X˜. We will show that there exists a hyperplane − that contains more than D 1 points of X˜. Indeed, take d inliers and D d 1 outliers and − − − let be the hyperplane generated by these D 1 points. Denote the normal vector to that H − hyperplane by b. Since contains d inliers, b will be orthogonal to these inliers. Since X H is in general position, every d-tuple of inliers is a basis for Span(X ). As a consequence, b will be orthogonal to Span(X ), and in particular b X . This implies that X and so ⊥ ⊂H will contain N + D d 1 d +1+ D d 1 > D 1 points of X˜. H − − ≥ − − −

In view of Proposition 3.2, we may restrict our search for hyperplanes that contain all

the inliers X to the subset of hyperplanes that are maximal with respect to the dataset X˜.

The advantage of this approach is immediate: the set of hyperplanes that are maximal with

respect to X˜ is in principle computable, since it is precisely the set of solutions of the

following optimization problem

˜ min X >b s.t. b =0. (3.2) b 0 6

The idea behind (3.2) is that a hyperplane = Span(b)⊥ contains a maximal number of H columns of X˜ if and only if its normal vector b has a maximal cosparsity level with respect

to the matrix X˜>, i.e., the number of non-zero entries of X˜>b is minimal.

Since (3.2) is a combinatorial problem, which in general can not be solved efﬁciently,

35 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

we consider its natural relaxation

X˜> min b s.t. b 2 =1, (3.3) b 1 k k

which we will refer to as Dual Principal Component Pursuit (DPCP). A major question

that arises, to be answered in Theorem 3.12, is under what conditions every global solution

of (3.3) is orthogonal to the inlier subspace Span(X ). A second major question, raised

D 1 by the non-convexity of the constraint b S − , is how to efﬁciently solve (3.3) with ∈ theoretical guarantees for global optimality.

We emphasize here that the optimization problem (3.3) is far from new; interestingly, its earliest appearance in the literature that we are aware of is in [85], where the authors proposed to solve it by means of the recursion of convex problems given by 5

˜> nk+1 := argmin X b . (3.4) b>nˆ =1 1 k

Notice that the optimization problem in (3.4) is computationally equivalent to a linear program; this makes the recursion (3.4) a very appealing candidate for solving the non-convex

(3.3). Even though [85] proved the very interesting result that the sequence n generated { k} by (3.4) converges to a critical point of (3.3) in a ﬁnite number of steps (see Theorem 3.14), there is no reason to believe that in general this limit point is a global minimum of (3.3).

Interestingly, this thesis adopts the recursion (3.4) of [85] for solving (3.3), and shows that it converges to a normal vector to the subspace (Theorem 3.15), which, in the special case of d = D 1, implies convergence to the global minimum of (3.3). − 5Being unaware of the work of [85], we independently proposed the same recursion in [96].

36 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Other work in which optimization problem (3.3) appears are [80, 86, 88–91]. More speciﬁcally, [86] proposes to solve (3.3) by replacing the quadratic constraint b>b = 1 with a linear constraint b>w = 1 for some vector w. In [80, 89] (3.3) is approximately solved by alternating minimization, while a Riemannian trust-region approach is employed in [90]. Finally, we note that problem (3.3) is closely related to the non-convex problem

(2.3) associated with REAPER. To see this, suppose that the REAPER orthoprojector Π appearing in (2.3), represents the orthogonal projection to a hyperplane with unit-` H 2 normal vector b. In such a case I Π = bb>, and it readily follows that problem (2.3) D − becomes identical to problem (3.3).

3.2.2 Theoretical analysis of the continuous problem

In this section we begin our theoretical investigation of the non-convex problem (3.3) as well as the recursion of convex relaxations (3.4), by associating to these two problems certain underlying continuous problems ( 3.2.2.1). We will refer to (3.3) and (3.4) as dis- § crete problems, since they involve a ﬁnite set of inliers and outliers. In sharp contrast, the continuous problems depend on uniform distributions on the respective inlier and outlier spaces, which makes them easier to analyze. The analysis of this section reveals that both the continuous analogue of (3.3) as well the continuous recursion corresponding to (3.4) are naturally associated with vectors orthogonal to the inlier subspace ( 3.2.2.2). This S § suggests that under certain conditions on the distribution of the data, the same must be true for the discrete problem (3.3) and recursion (3.4) (to be established in 3.2.3). §

37 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

3.2.2.1 The underlying continuous problem

In this section we show that the problems of interest (3.3) and (3.4) can be viewed as dis-

crete versions of certain continuous problems, which we analyze. To begin with, consider

D 1 D 1 given outliers O = [o ,..., o ] S − and inliers X = [x ,..., x ] S − , 1 M ⊂ 1 N ⊂ S∩ and recall the notation X˜ = [X O]Γ, where Γ is an unknown permutation. Next, for any

D 1 D 1 b S − deﬁne the function fb : S − R by fb(z) = b>z . Deﬁne also discrete ∈ → D 1 measures µO and µX on S − associated with the outliers and inliers respectively, as

1 M 1 N µO(z)= δ(z o ) and µX (z)= δ(z x ), (3.5) M − j N − j j=1 j=1 X X D 1 where δ( ) is the Dirac function on S − , which is deﬁned through the property ·

g(z)δ(z z0)dµSD−1 = g(z0), (3.6) z SD−1 − Z ∈

D 1 D 1 for every g : S − R and every z S − ; where µ D−1 is the uniform measure on → 0 ∈ S D 1 S − .

With these deﬁnitions, we have that the objective function X˜>b appearing in (3.3) 1

and (3.4) is the sum of the weighted expectations of the function under the measures O fb µ

38 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

and µX , i.e.,

M N ˜ X >b = O>b + X >b = b>oj + b>xj (3.7) 1 1 1 j=1 j=1 X X M N = b>z δ(z oj)dµSD−1 + b>z δ(z xj)dµSD−1 (3.8) z SD−1 − z SD−1 − j=1 Z ∈ j=1 Z ∈ X X M N = b>z δ(z oj) dµSD−1 + b>z δ(z xj) dµSD−1 (3.9) z SD−1  −  z SD−1  −  Z ∈ j=1 Z ∈ j=1 X X     = M EµO (fb)+ N EµX (fb). (3.10)

Hence, the optimization problem (3.3), which we repeat here for convenience,

˜ min X >b s.t. b>b =1, (3.11) b 1

is equivalent to the problem

min [M EµO (fb)+ N EµX (fb)] s.t. b>b =1. (3.12) b

Similarly, the recursion (3.4), repeated here for convenience,

˜> nk+1 = argmin X b s.t. b>nˆ k =1, (3.13) b 1

is equivalent to the recursion

nk+1 = argmin [M EµO (fb)+ N EµX (fb)] s.t. b>nˆ k =1. (3.14) b

Now, the discrete measures µO,µX of (3.5), are discretizations of the continuous measures

D 1 µSD−1 , and µSD−1 respectively, where the latter is the uniform measure on S − . ∩S ∩S Hence, for the purpose of understanding the properties of the global minimizer of (3.12) and the limiting point of (3.14), it is meaningful to replace in (3.12) and (3.14) the discrete

39 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

measures µO and µX by their continuous counterparts µSD−1 and µSD−1 , and study the ∩S resulting continuous problems

min M Eµ (fb)+ N Eµ (fb) s.t. b>b =1, (3.15) b SD−1 SD−1∩S

n = argmin M E (f )+ N E (f ) s.t. b>nˆ =1. (3.16) k+1 µSD−1 b µSD−1∩S b k b It is important to note that if these two continuous problems have the geometric properties of interest, i.e., if every global solution of (3.15) is a vector orthogonal to the inlier subspace, and similarly, if the sequence of vectors n produced by (3.16) converges { k} to a vector nk∗ orthogonal to the inlier subspace, then this correctness of the continuous problems can be viewed as a ﬁrst theoretical veriﬁcation of the correctness of the discrete formulations (3.3) and (3.4). The objective of the rest of this section is to establish that this is precisely the case.

Before stating and proving our main two results in this direction, we note that the continuous objective function

(b)= M Eµ (fb)+ N Eµ (fb) (3.17) J SD−1 SD−1∩S can be re-written in a simple form. Writing b = b bˆ, and letting R be a rotation that k k2 takes bˆ to the ﬁrst standard basis vector e1, we see that the ﬁrst expectation in (3.17)

40 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

becomes equal to

D−1 EµSD−1 (fb)= fb(z)dµS (3.18) z SD−1 Z ∈

= b>z dµSD−1 (3.19) z SD−1 Z ∈

ˆ> = b 2 b z dµSD−1 (3.20) k k z SD−1 Z ∈

1 ˆ = b 2 z>R− Rb dµSD−1 (3.21) k k z SD−1 Z ∈

= b 2 z>e1 dµSD−1 (3.22) k k z SD−1 | | Z ∈

= b 2 z1 dµSD−1 = b 2 cD, (3.23) k k z SD−1 | | k k Z ∈ where z = (z1,...,zD)> is the coordinate representation of z, and cD is the mean height of the unit hemisphere of RD, given in closed form by

2 (D 2)!! π if D even, cD = −  (3.24) (D 1)!! ·  −  1 if D odd,  where the double factorial is deﬁned as 

k(k 2)(k 4) 4 2 if k even, k!! :=  − − ··· · (3.25)   k(k 2)(k 4) 3 1 if k odd. − − ··· ·  To see what the second expectation in (3.17) evaluates to, decompose b as b = π (b)+ S

π ⊥ (b), and note that because the support of the measure µSD−1 is contained in , we S ∩S S

41 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

must have that

E (f )= b>z dµ D−1 (3.26) µSD−1∩S b S z SD−1 ∩S Z ∈

= b>z dµSD−1 (3.27) z SD−1 ∩S Z ∈ ∩S

= (π (b))> z dµSD−1 (3.28) z SD−1 S ∩S Z ∈ ∩S

\ > = π (b) 2 π (b) z dµSD−1 . (3.29) k S k z SD−1 S ∩S Z ∈ ∩S

Writing z0 and b0 for the coordinate representation of z and π\( b) with respect to a basis S

of , and noting that µSD−1 = µSd−1 , we have that S ∩S ∼

\ > π (b) z dµSD−1 = z0>b0 dµSd−1 = cd, (3.30) z SD−1 S ∩S z0 Sd−1 Z ∈ ∩S Z ∈ d where now cd is the average height of the unit hemisphere of R . Finally, noting that

π (b) = b cos(φ), (3.31) k S k2 k k2

where φ [0,π/2] is the principal angle of b from the subspace , we have that ∈ S

Eµ (fb)= b cd cos(φ). (3.32) SD−1∩S k k2

Putting everything together, we arrive at the ﬁnal form of our continuous objective function:

(b)= M Eµ (fb)+ N Eµ (fb)= b (McD + Ncd cos(φ)) . (3.33) J SD−1 SD−1∩S k k2

3.2.2.2 Conditions for global optimality and convergence

We are now in a position to state and prove our main results about the continuous problems

(3.15) and (3.16).

42 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Theorem 3.3. Any global solution to problem (3.15) must be orthogonal to . S

Proof. Because of the constraint b>b = 1 in (3.15), and using (3.33), problem (3.15) can

be written as

min [McD + Ncd cos(φ)] s.t. b>b =1, (3.34) b where φ [0,π/2] is the principal angle of b from . It is then immediate that the global ∈ S minimum is equal to McD and it is attained if and only if φ = π/2, which corresponds to b . ⊥S

D 1 Theorem 3.4. Consider the sequence nk k 0 generated by recursion (3.16), nˆ 0 S − . { } ≥ ∈ Let φ be the principal angle of n from , and deﬁne α := Nc /Mc . Then, as long 0 0 S d D

as φ0 > 0, the sequence nk k 0 converges to a unit `2-norm element of ⊥ in a ﬁnite { } ≥ S

number k∗ of iterations, where k∗ = 0 if φ = π/2, k∗ = 1 if tan(φ ) 1/α, and 0 0 ≥ −1 tan (1/α) φ0 k∗ −1 − +1 otherwise. ≤ sin (α sin(φ0)) Proof. At iteration k the optimization problem associated with (3.16) is

min (b)= b 2 (McD + Ncd cos(φ)) s.t. b>nˆ k =1, (3.35) b RD J k k ∈

where φ [0,π/2] is the principal angle of b from the subspace . ∈ S Let φ be the principal angle of nˆ from (Figure 3.1), and let n be a global k k S k+1 minimizer of (3.35), with principal angle from equal to φ [0,π/2]. We show that S k+1 ∈

43 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

⊥ S

nˆ k⊥

nˆ k†

nˆ k ψk φ hˆ S k k

Figure 3.1: Various vectors and angles appearing in the proof of Theorem 3.4.

φ φ . To see this, note that the decrease in the objective function at iteration k is k+1 ≥ k

(nˆ ) (n ) :=Mc nˆ + Nc nˆ cos(φ ) J k −J k+1 D k kk2 d k kk2 k Mc n Nc n cos(φ ). (3.36) − D k k+1k2 − d k k+1k2 k+1

Since n> nˆ = 1, we must have that n 1 = nˆ . Now if φ < φ , k+1 k k k+1k2 ≥ k kk2 k+1 k then cos(φ ) > cos(φ ). But then (3.36) implies that (n ) > (nˆ ), which is a k+1 k J k+1 J k contradiction on the optimality of n . Hence it must be the case that φ φ , and so k+1 k+1 ≥ k the sequence φ is non-decreasing. In particular, since φ > 0 by hypothesis, we must { k}k 0 also have φ > 0, i.e., nˆ , k 0. k k 6∈ S ∀ ≥

Letting ψ be the angle of b from nˆ , the constraint b>nˆ =1 gives 0 ψ < π/2 and k k k ≤ k b =1/ cos(ψ ), and so we can write the optimization problem (3.35) equivalently as k k2 k

McD + Ncd cos(φ) min s.t. b>nˆ k =1. (3.37) b RD cos(ψ ) ∈ k

If nˆ is orthogonal to , i.e., φ = π/2, then (nˆ ) = Mc (b), b : b>nˆ = 1, k S k J k D ≤ J ∀ k with equality only if b = nˆ . As a consequence, n 0 = nˆ , k0 > k, and in particular if k k k ∀

φ0 = π/2, then k∗ =0.

44 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

So suppose that φk < π/2 and let nˆ k⊥ be the normalized orthogonal projection of nˆ k onto ⊥ (Figure 3.1). We will prove that every global minimizer of problem (3.37) must lie S in the two-dimensional plane := Span(nˆ , nˆ ⊥). To see this, let b have norm 1/ cos(ψ ) H k k k for some ψ [0,π/2). If ψ > π/2 φ , then such a b can not be a global minimizer of k ∈ k − k

(3.37), as the feasible vector nˆ ⊥/ sin(φ ) already gives a smaller objective, since k k ∈H

McD McD McD + Ncd cos(φ) (nˆ ⊥/ sin(φ )) = = < = (b). (3.38) J k k sin(φ ) cos(π/2 φ ) cos(ψ ) J k − k k

Thus, it must be the case that ψ π/2 φ . Denote by hˆ the normalized projection k ≤ − k k of nˆ onto and by nˆ † the vector that is obtained from nˆ by rotating it towards nˆ ⊥ by k S k k k

ψ (Figure 3.1). Note that both hˆ and nˆ † lie in . Letting Ψ [0,π] be the spherical k k k H k ∈

angle between the spherical arc formed by nˆ k, bˆ and the spherical arc formed by nˆ k, hˆ k,

and denoting by ∠b, hˆ k the angle between b and hˆ k, the ﬁrst spherical law of cosines gives

cos(∠b, hˆ k) = cos(φk)cos(ψk) + sin(φk)sin(ψk) cos(Ψk). (3.39)

Now, Ψ is equal to π if and only if nˆ , hˆ , b are coplanar, i.e., if and only if b . k k k ∈ H Suppose that b . Then Ψ <π, and so cos(Ψ ) > 1, which implies that 6∈ H k k −

cos(∠b, hˆ ) > cos(φ )cos(ψ ) sin(φ )sin(ψ ) = cos(φ + ψ ). (3.40) k k k − k k k k

This in turn implies that the principal angle φ of b from is strictly smaller than φ + ψ , S k k and so

McD + Ncd cos(φ) McD + Ncd cos(φk + ψk) (b)= > = (nˆ k† / cos(ψk)), (3.41) J cos(ψk) cos(ψk) J

45 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

i.e., the feasible vector nˆ † / cos(ψ ) gives strictly smaller objective than b. k k ∈H

To summarize, for the case where φk < π/2, we have shown that any global minimizer b of (3.37) must i) have angle ψ from nˆ less or equal to π/2 φ , and ii) it must lie in k k − k

Span(nˆ k, nˆ k⊥). Hence, we can rewrite (3.37) in the equivalent form

McD + Ncd cos(φk + ψ) min k(ψ) := , (3.42) ψ [ π/2+φk,π/2 φk] J cos(ψ ) ∈ − − k where now ψk takes positive values as b approaches nˆ k⊥ and negative values as it approaches hˆ . The function is continuous and differentiable in the interval [ π/2+ φ ,π/2 φ ], k Jk − k − k with derivative given by

∂ Mc sin(ψ) Nc sin(φ ) Jk = D − d k . (3.43) ∂ψ cos2(ψ)

Setting the derivative to zero gives

sin(ψ)= α sin(φk). (3.44)

If α sin(φ ) sin(π/2 φ ) = cos(φ ), or equivalently tan(φ ) 1/α, then is strictly k ≥ − k k k ≥ Jk decreasing in the interval [ π/2+φ ,π/2 φ ], and so it must attain its minimum precisely − k − k at ψ = π/2 φ , which corresponds to the choice n = nˆ ⊥/ sin(φ ). Then by an earlier − k k+1 k k argument we must have that nˆ 0 , k0 k +1. If, on the other hand, tan(φ ) < 1/α, k ⊥S ∀ ≥ k then the equation (3.44) deﬁnes an angle

1 ψ∗ := sin− (α sin(φ )) (0,π/2 φ ), (3.45) k k ∈ − k at which must attain its global minimum, since Jk 2 ∂ k 1 ∗ (3.46) J2 (ψk)= > 0. ∂ψ cos(ψk∗)

46 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

As a consequence, if tan(φk) < 1/α, then

1 φk+1 = φk + sin− (α sin(φk)) < π/2. (3.47)

We then see inductively that as long as tan(φk) < 1/α, φk increases by a quantity which is

1 bounded from below by sin− (α sin(φ0)). Thus, φk will keep increasing until it becomes greater than the solution to the equation tan(φ)=1/α, at which point the global minimizer will be the vector n = nˆ ⊥/ sin(φ ), and so nˆ 0 = nˆ , k0 k +1. Finally, under k+1 k k k k+1 ∀ ≥ 1 the hypothesis that φk < tan− (1/α), we have

k 1 − 1 1 φ = φ + sin− (α sin(φ )) φ + k sin− (α sin(φ )), (3.48) k 0 j ≥ 0 0 j=0 X from where it follows that the maximal number of iterations needed for φk to become

−1 1 tan (1/α) φ0 larger than tan− (1/α) is −1 − , at which point at most one more iteration will sin (α sin(φ0)) be needed to achieve orthogonality to . S

Notice the remarkable fact that according to Theorem 3.4, the continuous recursion

(3.16) converges to a vector orthogonal to the inlier subspace in a ﬁnite number of steps. S Moreover, if the relation

tan(φ ) 1/α, (3.49) 0 ≥ holds true, then this convergence occurs in a single step. One way to interpret (3.49) is to notice that as long as the angle φ0 of the initial estimate nˆ 0 from the inlier subspace is positive, and for any arbitrary but ﬁxed number of outliers M, there is always a sufﬁciently

large number N of inliers, such that (3.49) is satisﬁed and thus convergence occurs in

47 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

one step. Conversely, for any ﬁxed number of inliers N and outliers M, there is always a sufﬁciently large angle φ0 such that (3.49) is true, and thus (3.16) again converges in a

single step. More generally, even when (3.49) is not true, the larger φ0,N are, the smaller

the quantity

1 tan− (1/α) φ0 1 − (3.50) sin− (α sin(φ )) 0 is, and thus according to Theorem 3.3 the faster (3.16) converges.

3.2.3 Theoretical analysis of the discrete problem

In this section we analyze the discrete problem (3.3) and the associated discrete recursion

(3.4), where the adjective discrete refers to the fact that (3.3) and (3.4) depend on a ﬁnite set

˜ D 1 of points X =[X O]Γ sampled from the union of the space of outliers S − and the space

D 1 of inliers S − . In 3.2.2 we showed that these problems are discrete versions of the ∩S § continuous problems (3.15) and (3.16), for which we further showed that they possess the geometric property of interest, i.e., every global minimizer of (3.15) must be an element

D 1 of ⊥ S − (Theorem 3.3), and the recursion (3.16) produces a sequence of vectors S ∩ D 1 which converges to an element of ⊥ S − in a ﬁnite number of steps (Theorem 3.4). In S ∩ this section we show that under some deterministic conditions on the distribution of X =

[x1,..., xN ] and O = [o1,..., oM ], a similar statement holds for the discrete problems

(3.3) and (3.4). More speciﬁcally, in 3.2.3.1 we establish quantitative bounds that relate § the discrete objective function to its continuous counterpart, and in 3.2.3.2 and 3.2.3.3 § § we use these bounds to study the global minimizers of (3.3) and the limit point of (3.4).

48 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

3.2.3.1 Discrepancy bounds between continuous and discrete problems

The heart of our analysis framework is to bound the deviation of some underlying geometric

quantities, which we call the average outlier and the average inlier with respect to b, from

their continuous counterparts. To begin with, recall our discrete objective function

˜ discrete(b)= X >b = O>b + X >b (3.51) J 1 1 1

and its continuous counterpart

(b)= b (Mc + Nc cos(φ)) , (3.52) Jcontinuous k k2 D d

the latter derived in 3.2.2.1, equation (3.33). Now, notice that the term of the discrete § objective that depends on the outliers O can be written as

M M O>b o>b b> o>b o b> o (3.53) 1 = j = Sign( j ) j = M b, j=1 j=1 X X where Sign( ) is the sign function and · 1 M o := Sign(b>o )o (3.54) b M j j j=1 X D 1 D is the average outlier with respect to b. Deﬁning a vector valued function f : S − R b → D 1 f b by z S − Sign(b>z)z, we notice that ∈ 7−→ 1 M o = f (o ), (3.55) b M b j j=1 X and so ob is a discrete approximation to the integral z SD−1 f b(z)dµSD−1 . The value of ∈ that integral is given by the next Lemma. R

49 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

D 1 Lemma 3.5. For any b S − we have ∈

f b(z)dµSD−1 = Sign(b>z)zdµSD−1 = cD b, (3.56) z SD−1 z SD−1 Z ∈ Z ∈

where cD is deﬁned in (3.24).

Proof. Letting R be a rotation that takes b to the ﬁrst canonical vector e1, i.e., Rb = e1, we have that

Sign(b>z)zdµSD−1 = Sign(b>R>Rz)zdµSD−1 (3.57) z SD−1 z SD−1 Z ∈ Z ∈

= Sign(e1>z)R>zdµSD−1 (3.58) z SD−1 Z ∈

= R> Sign(z1)zdµSD−1 , (3.59) z SD−1 Z ∈

where z1 is the ﬁrst cartesian coordinate of z. Recalling the deﬁnition of cD in equation

(3.24), we see that

Sign(z1)z1dµSD−1 = z1 dµSD−1 = cD. (3.60) z SD−1 z SD−1 | | Z ∈ Z ∈ Moreover, for any i> 1, we have

Sign(z1)zidµSD−1 =0. (3.61) z SD−1 Z ∈ Consequently, the integral in (3.59) becomes

Sign(b>z)zdµSD−1 = R> Sign(z1)zdµSD−1 (3.62) z SD−1 z SD−1 Z ∈ Z ∈

= R> (cDe1)= cDb. (3.63)

50 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Now we deﬁne O to be the maximum among all possible approximation errors as b varies

D 1 on S − , i.e.,

O := max cD b ob 2 , (3.64) b SD−1 k − k ∈ and we establish that the more uniformly distributed O is the smaller O becomes.

D 1 The notion of uniformity of O = [o ,..., o ] S − that we employ here is a 1 M ⊂ deterministic one, and is captured by the spherical cap discrepancy of the set O, deﬁned

as [40, 41]

1 M SD(O) := sup I (oj) µSD−1 ( ) . (3.65) M C − C C j=1 X D 1 In (3.65) the supremum is taken over all spherical caps of the sphere S − , where a C D 1 D spherical cap is the intersection of S − with a half-space of R , and I ( ) is the indicator C · function of , which takes the value 1 inside and zero otherwise. The spherical cap C C discrepancy SD(O) is precisely the supremum among all errors in approximating integrals of indicator functions of spherical caps via averages of such indicator functions on the point set O. Intuitively, SD(O) captures how close the discrete measure µO (see equation (3.5))

associated with O is to the measure µSD−1 , and we will be referring to O as being uniformly

D 1 distributed on S − , when SD(O) is small.

Remark 3.6. As a function of the number of points M, SD(O) decreases with a rate

of [5,22]

1 1 log(M)M − 2 − 2(D−1) . (3.66) p 51 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

As a consequence, to show that uniformly distributed points O correspond to small O, it sufﬁces to bound the maximum integration error O from above by a quantity proportional

to the spherical cap discrepancy SD(O). Inequalities that bound from above the approximation error of the integral of a function in terms of the variation of the function and the discrepancy of a ﬁnite set of points (not necessarily the spherical cap discrepancy; there are several types of discrepancies) are widely known as Koksma-Hlawka inequalites [49, 58].

Even though such inequalities exist and are well-known for integration of functions on the unit hypercube [0, 1]D [44, 49, 58], similar inequalities for integration of functions on the

D 1 unit sphere S − seem not to be known in general [41], except if one makes additional as-

sumptions on the distribution of the ﬁnite set of points [10, 40]. Nevertheless, the function

fb : z b>z that is associated to O is simple enough to allow for a Koksma-Hlawka 7−→ | | inequality, as described in the next lemma.6

D 1 Lemma 3.7. Let O =[o1,..., oM ] be a ﬁnite subset of S − . Then

√ O O = max cDb ob 2 5SD( ), (3.67) b SD−1 k − k ≤ ∈

where cD, ob and SD(O) are deﬁned in (3.24), (3.54) and (3.65) respectively.

D 1 Proof. For any b S − we can write ∈

c b ob = ρ b + ρ ζ, (3.68) D − 1 2

D 1 2 2 for some vector ζ S − orthogonal to b, and so it is enough to show that ρ + ρ ∈ 1 2 ≤ p √5S (O). Let us ﬁrst bound from above ρ in terms of S (O). Towards that end, D | 1| D 6The author is grateful to Prof. Glyn Harman, who suggested that the proof of Theorem 1 in [44] can be easily adapted to establish the result of Lemma 3.7.

52 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT observe that

1 M ρ = b>(c b ob)= c b>o (3.69) 1 D − D − M j j=1 X 1 M = fb(z)dµSD−1 fb(oj), (3.70) D−1 − M z S j=1 Z ∈ X where the equality follows from the deﬁnition of cD in (3.24) and recalling that fb(z) =

D 1 b>z . In other words, ρ1 is the error in approximating the integral of fb on S − by the

average of fb on the point set O.

D 1 Now, notice that each super-level set z S − : fb(z) α for α [0, 1], is the ∈ ≥ ∈ union of two spherical caps, and also that

sup fb(z) inf fb(z)=1 0=1. (3.71) D z SD−1 − z S −1 − ∈ ∈

We these in mind, repeating the entire argument of the proof of Theorem 1 in [44] that lead to inequality (9) in [44], but now for a measurable function with respect to µSD−1 (that would be fb), leads directly to

ρ S (O). (3.72) | 1|≤ D

For ρ2 we have that

ρ = ζ> (c b) ζ>ob (3.73) 2 D − 1 M = Sign b>z ζ>zdµSD−1 Sign b>oj ζ>oj (3.74) z SD−1 − M Z ∈ j=1 X 1 M = gb,ζ(z)dµSD−1 gb,ζ(oj), (3.75) D−1 − M z S j=1 Z ∈ X

53 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

D 1 where gb ζ : S − R is deﬁned as gb ζ(z) = Sign b>z ζ>z. Then a similar argument , → , as for ρ1, with the difference that now

sup gb,ζ(z) inf gb,ζ(z)=1 ( 1)=2, (3.76) D z SD−1 − z S −1 − − ∈ ∈ leads to

ρ 2S (O). (3.77) | 2|≤ D

In view of (3.72), inequality (3.77) establishes that ρ2 + ρ2 √5S (O), which con- 1 2 ≤ D p cludes the proof of the lemma.

We now turn our attention to the inlier term X˜>b of the discrete objective function 1

(3.51), which is slightly more complicated than the outlier term. We have

N N X >b x>b b> x>b x b> x (3.78) 1 = j = Sign( j ) j = N b, j=1 j=1 X X where

1 N 1 N x := Sign(b>x )x = f (x ) (3.79) b N j j N b j j=1 j=1 X X

is the average inlier with respect to b. Thus, xb is a discrete approximation of the integral

x SD−1 f b(x)dµSD−1 , whose value is given by the next lemma. ∈ ∩S R D 1 Lemma 3.8. For any b S − we have ∈

f b(x)dµSD−1 = Sign(b>x)xdµSD−1 = cd vˆ, (3.80) x SD−1 x SD−1 Z ∈ ∩S Z ∈ ∩S

where cd is given by (3.24) after replacing D with d, and v is the orthogonal projection of

v onto . S 54 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Proof. Since x lies in , we have f (x)= f (x)= f (x), so that S b v vˆ

Sign(b>x)xdµSD−1 = Sign(vˆ>x)xdµSD−1 . (3.81) x SD−1 x SD−1 Z ∈ ∩S Z ∈ ∩S Now express x and vˆ on a basis of , use Lemma 3.5 replacing D with d, and then switch S back to the standard basis of RD.

Next, we deﬁne X to be the maximum among all possible approximation errors as b

D 1 varies on S − , which is the same as the maximum of all approximation errors as b varies

D 1 on S − , i.e., ∩S

\ X := max cd π (b) xb = max cd b xb 2 . (3.82) b SD−1 S − 2 b SD−1 k − k ∈ ∈ ∩S

Then an almost identical argument as the one that established Lemma 3.7 gives that

X √5S (X ), (3.83) ≤ d

where now the discrepancy Sd(X ) of the inliers X is deﬁned exactly as in (3.65) with the

D 1 d 1 only difference that the supremum is taken over all spherical caps of S − = S − . ∩S ∼

As a consequence of our deﬁnitions of O and X we obtain lower and upper bounds of our discrete objective function (3.51) in terms of its continuous counterpart (3.52), that will be repeatedly used in the sequel.

D 1 Lemma 3.9. Let b S − ⊥, and let φ [0,π/2) be its principal angle from . Then ∈ \S ∈ S

M(c + O) O>b M(c O), (3.84) D ≥ 1 ≥ D −

N(c + X ) cos( φ) X >b N(c X ) cos(φ). (3.85) d ≥ 1 ≥ d −

55 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Proof. We only prove the second inequality as the ﬁrst is even simpler. Let v = 0 be the 6

orthogonal projection of b onto . By deﬁnition of X , there exists a vector ξ of ` S ∈ S 2 norm less or equal to X , such that

1 N x = x = Sign(b>x )x = c vˆ + ξ. (3.86) v b N j j d j=1 X Taking inner product of both sides with b gives

1 X >b = c cos(φ)+ b>ξ. (3.87) N 1 d

Now, the result follows by noting that b>ξ X cos(φ), since the principal angle of b ≤ from Span(ξ) can not be less then φ.

3.2.3.2 Conditions for global optimality of the discrete problem

In this section we give our main theorem regarding the global minimizers of (3.3). We

begin with a deﬁnition.

D 1 Deﬁnition 3.10. Given a set Y = [y ,..., y ] S − and an integer K L, deﬁne 1 L ⊂ ≤

Y to be the maximum circumradius among all polytopes of the form R ,K

K α y : α [ 1, 1] , (3.88) ji ji ji ∈ − ( i=1 ) X

where j1,...,jK are distinct integers in [L], and the circumradius of a bounded subset of

RD is the inﬁmum over the radii of all Euclidean balls of RD that contain that subset.

Next, we state a result already known in [85], and for completeness we also give the proof.

56 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Proposition 3.11 ( [85]). Let Y = [y ,..., y ] be a D L matrix of rank D. Then any 1 L × global solution b∗ to

Y >b (3.89) min 1 , b>b=1

must be orthogonal to (D 1) linearly independent columns of Y . −

Proof. Let b∗ be an optimal solution of (3.89). Then b∗ must satisfy the ﬁrst order optimality relation

0 Y Sgn(Y >b∗)+ λb∗, (3.90) ∈ where λ is a scalar Lagrange multiplier parameter, and Sgn is the sub-differential of the

`1 norm. Without loss of generality, let y1,..., yK be the columns of Y to which b∗ is orthogonal. Then equation (3.90) implies that there exist real numbers α ,...,α 1 K ∈ [ 1, 1] such that − K L 0 αjyj + Sign(yj>b∗)yj + λb∗ = . (3.91) j=1 j=K+1 X X Now, suppose that the span of y ,..., y is of dimension less than D 1. Then there exists 1 K − D 1 a unit norm vector ζ S − that is orthogonal to all y ,..., y , b∗, and multiplication of ∈ 1 K

(3.91) from the left by ζ> gives

Sign(yj>b∗)ζ>yj =0. (3.92) j=K+1 X Furthermore, we can choose a sufﬁciently small ε> 0, such that

Sign(y>b∗ + εy>ζ) = Sign(y>b∗), j [L]. (3.93) j j j ∀ ∈

57 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

The above equation is trivially true for all j such that yj>b∗ = 0, because in that case y>ζ =0 by the deﬁnition of ζ. On the other hand, if y>b∗ =0, then a small perturbation j j 6

will not change the sign of yj>b∗. Consequently, we can write

y>(b∗ + εζ) = y>b∗ + ε Sign(y>b∗)y>ζ, j [L] (3.94) j j j j ∀ ∈

and so

L Y > b∗ ζ Y >b∗ y>b∗ ζ>y Y >b∗ (3.95) ( + ε ) 1 = 1 + ε Sign( j ) j = 1 . j=K+1 X

However,

b∗ + εζ = √1+ ε2 > 0, (3.96) k k2

and normalizing b∗ + εζ to have unit `2 norm, we get a contradiction on b∗ being a global

solution.

We are now ready to establish the main theorem of this section. Theorem 3.12 is the

discrete analogue of Theorem 3.3, and it says that if both inliers and outliers are sufﬁciently

uniformly distributed, then every global solution of (3.3) must be orthogonal to the inlier

subspace . More precisely, S

Theorem 3.12. Suppose that the condition

M c X c X ( O + X ) /N γ := < min d − , d − − R ,K1 R ,K2 , (3.97) N 2 O O

holds for all non-negative integers K ,K such that K + K = D 1,K d 1. Then 1 2 1 2 − 2 ≤ − any global solution b∗ to (3.3) must be orthogonal to Span(X ).

58 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Proof. Let b∗ be an optimal solution of (3.3). Then b∗ must satisfy the ﬁrst order optimality

relation

˜ ˜ 0 X Sgn(X >b∗)+ λb∗, (3.98) ∈

where λ is a scalar Lagrange multiplier parameter, and Sgn is the sub-differential of the `1 norm. For the sake of contradiction, suppose that b∗ . If b∗ , then using Lemma 6⊥ S ∈ S 3.9 we have

O O>b O>b∗ X >b∗ McD + M min 1 1 + 1 ≥ b ,b>b=1 ≥ ⊥S

Mc MO + Nc NX , (3.99) ≥ D − d −

which violates hypothesis (3.97). Hence, we can assume that b∗ . 6∈ S

Now, by the general position hypothesis as well as Proposition 3.11, b∗ will be orthog-

onal to precisely D 1 points, among which K points belong to O, say o ,..., o , and − 1 1 K1 0 K d 1 points belong to X , say x ,..., x . Then there must exist real numbers ≤ 2 ≤ − 1 K2 1 α , β 1, such that − ≤ j j ≤ K1 M K2 N 0 αjoj + Sign(oj>b∗)oj + βjxj + Sign(xj>b∗)xj + λb∗ = . (3.100) Xj=1 j=XK1+1 Xj=1 j=XK2+1

Since Sign(o>b∗)=0, j K and similarly Sign(x>b∗)=0, j K , we can equiva- j ∀ ≤ 1 j ∀ ≤ 2 lently write

K1 M K2 N 0 αjoj + Sign(oj>b∗)oj + βjxj + Sign(xj>b∗)xj + λb∗ = (3.101) j=1 j=1 j=1 j=1 X X X X or more compactly

ξO + M ob∗ + ξX + N xvˆ∗ + λb∗ = 0, (3.102)

59 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

where

K1 K2

ξO := αjoj and ξX := βjxj, (3.103) j=1 j=1 X X vˆ∗ is the normalized projection of b∗ onto (which is nonzero since b∗ by hypothesis), S 6⊥ S and

1 M 1 N o ∗ := Sign(o>b∗)o , x ∗ := Sign(x>v∗)x . (3.104) b M j j v N j j j=1 j=1 X X

Now, from the deﬁnitions of O and X in (3.64) and (3.82) respectively, we have that

ob∗ = c b∗ + ηO, ηO O (3.105) D || ||2 ≤

xv∗ = c vˆ∗ + ηX , ηX X , (3.106) ˆ d || ||2 ≤

and so (3.102) becomes

ξO + McD b∗ + M ηO + ξX + Ncd vˆ∗ + N ηX + λb∗ = 0. (3.107)

Since b∗ , we have that b∗, v∗ are linearly independent. Deﬁne := Span(b∗, vˆ∗) and 6∈ S U project (3.107) onto to get U

π (ξO)+ McD b∗ + Mπ (ηO)+ π (ξX )+ Ncd vˆ∗ + Nπ (ηX )+ λb∗ = 0. U U U U (3.108)

Notice that every vector u in the image of π can be written as a linear combination of U

60 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

b∗ and vˆ∗: u =[u]b∗ b∗ +[u]vˆ∗ vˆ∗. Using this decomposition, we can write (3.108) as

[π (ξO)]b∗ b∗ +[π (ξO)]vˆ∗ vˆ∗ + McD b∗ U U

+ M [π (ηO)]b∗ b∗ + M [π (ηO)]vˆ∗ vˆ∗ U U

+[π (ξX )]b∗ b∗ +[π (ξX )]vˆ∗ vˆ∗ + Ncd vˆ∗ U U 0 + N [π (ηX )]b∗ b∗ + N [π (ηX )]vˆ∗ vˆ∗ + λb∗ = . (3.109) U U

Since is a two-dimensional space, there exists a vector ζˆ that is orthogonal to b∗ but U ∈U not orthogonal to vˆ∗. Projecting the above equation onto the line spanned by ζˆ, we obtain the one-dimensional equation

ˆ> ([π (ξO)]vˆ∗ + M [π (ηO)]vˆ∗ +[π (ξX )]vˆ∗ + Ncd + N [π (ηX )]vˆ∗ ) ζ vˆ∗ = 0. (3.110) U U U U ·

Since ζˆ is not orthogonal to vˆ∗, the above equation implies that

[π (ξO)]vˆ∗ + M [π (ηO)]vˆ∗ +[π (ξX )]vˆ∗ + Ncd + N [π (ηX )]vˆ∗ =0, (3.111) U U U U

which, together with ξO O , ξX X , ηO O and ηX X , k k2 ≤ R ,K1 k k2 ≤ R ,K2 k k2 ≤ k k2 ≤ implies that

Nc O + MO + X + NX , (3.112) d ≤R ,K1 R ,K2 which violates the hypothesis (3.97). Consequently, the initial hypothesis of the proof that b∗ can not be true, and the theorem is proved. 6⊥ S

A Geometric View of the Proof of Theorem 3.12. Let us provide some geometric intuition that underlies the proof of Theorem 3.12. It is instructive to begin our discussion by

61 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

considering the case d = 1,D = 2, i.e. the inlier space is simply a line and the ambient

space is a 2-dimensional plane. Since all points have unit `2-norm, every column of X will

be of the form xˆ or xˆ for a ﬁxed vector xˆ 1 that spans the inlier space . In this − ∈ S S

setting, let us examine a global solution b∗ of the optimization problem (3.3). We will start by assuming that such a b∗ is not orthogonal to , and intuitively arrive at the conclusion S that this can not be the case as long as there are sufﬁciently many inliers.

We will argue on an intuitive level that if b∗ , then the principal angle φ of b∗ from 6⊥ S

needs to be small. To see this, suppose b∗ ; then b∗ will be non-orthogonal to every S 6⊥ S inlier, and by Proposition 3.11 orthogonal to D 1=1 outlier, say o . The optimality − 1 condition (3.98) specializes to

M N 0 α1o1 + Sign(oj>b∗)oj + Sign(xj>b∗)xj + λb∗ = , (3.113) j=1 j=1 X X Mob∗

where 1 α 1. Notice| that{z the third} term is simply N Sign(xˆ>b∗)xˆ, and so − ≤ 1 ≤

α o + M ob∗ + λb∗ = N Sign(xˆ>b∗)xˆ. (3.114) 1 1 −

Now, what (3.114) says is that the point N Sign(xˆ>b∗)xˆ must lie inside the set −

Conv( o )+ Mob∗ + Span(b∗)= α o + Mob∗ + λb∗ : α 1,λ R , ± 1 { } { 1 1 | 1|≤ ∈ } (3.115) where the + operator on sets is the Minkowski sum. Notice that the set Conv( o )+Mob∗ ± 1 is the translation of the line segment (polytope) Conv( o ) by Mob∗ . Then (3.114) says ± 1 that if we draw all afﬁne lines that originate from every point of Conv( o )+ Mob∗ and ± 1 62 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

have direction b∗, then one of these lines must meet the point N Sign(xˆ>b∗)xˆ. Let us −

illustrate this for the case where M = N = 5 and say it so happens that b∗ has a rather

large angle φ from , say φ = 45◦. Recall that ob∗ is concentrated around c b∗ and for S D 2 the case D = 2 we have cD = π . As illustrated in Figure 3.2, because φ is large, the

Mob∗ + Conv( o ) + Span(b∗) ± 1

Mob∗

Mob∗ + Conv( o ) ± 1 o1 b∗ φ N Sign(xˆ>b∗)xˆ S −

Figure 3.2: Geometry of the optimality condition (3.98) and (3.114) for the case d =

1,D = 2,M = N = 5. The polytope Mob∗ + Conv( o ) + Span(b∗) misses the point ± 1

N Sign(xˆ>b∗)xˆ and so the optimality condition can not be true for both b∗ = − 6⊥ S Span(xˆ) and φ large.

unbounded polytope Mob∗ + Conv( o ) + Span(b∗) misses the point N Sign(xˆ>b∗)xˆ ± 1 − thus making the optimality equation (3.114) infeasible. This indicates that critical vectors b∗ having large angles from are unlikely to exist. 6⊥ S S

On the other hand, critical points b∗ may exist, but their angle φ from needs 6⊥ S S to be small, as illustrated in Figure 3.3. However, such critical points can not be global

63 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Mob∗ + Conv( o ) + Span(b∗) ± 1 o1 o ∗ b∗ M b N Sign(xˆ>b∗)xˆ S −

Mob∗ + Conv( o ) ± 1

Figure 3.3: Geometry of the optimality condition (3.98) and (3.114) for the case d =

1,D = 2,M = N = 5. A critical b∗ exists, but its angle from is small, so that 6⊥ S S the polytope Mob∗ + Conv( o ) + Span(b∗) can contain the point N Sign(xˆ>b∗)xˆ. ± 1 −

However, b∗ can not be a global minimizer, since small angles from yield large objective S values. minimizers, because small angles from yield large objective values.7 S

Hence the only possibility that critical points b∗ that are also global minimizers 6⊥ S do exist is that the number of inliers is signiﬁcantly less than the number of outliers, i.e.

N <

We should note here that the picture for the general setting is analogous to what we described above, albeit harder to visualize: with reference to equation (3.259), the optimality condition says that every feasible point b∗ must have the following property: there 6⊥ S exist 0 K d 1 inliers x ,..., x and 0 K D 1 K outliers o ,..., o ≤ 2 ≤ − 1 K2 ≤ 1 ≤ − − 2 1 K1

K1 to which b∗ is orthogonal, and two points ξO α o : α [ 1, 1] + ob∗ and ∈ j=1 j j j ∈ − n o 7On a more techincal level, it can be veriﬁed that if suchP a critical point is a global minimizer, then its angle φ must be large in the sense that cos(φ) 2O, contradicting the necessary condition that φ be small in the ﬁrst place. ≤

64 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Mob∗ + Conv( o ) + Span(b∗) ± 1

Mob∗

Mob∗ + Conv( o ) ± 1 o1 b∗

N Sign(xˆ>b∗)xˆ −

Figure 3.4: Geometry of the optimality condition (3.98) and (3.114) for the case d =

1,D =2,N <b∗)xˆ. Moreover, such critical points can be global mini- − mizers. Condition (3.97) of Theorem 3.12 prevents such cases from occuring.

K2 ξX α x : α [ 1, 1] + xb∗ that are joined by an afﬁne line that is parallel to ∈ j=1 j j j ∈ − nP o the line spanned by b∗. In fact in our proof of Theorem 3.12 we reduced this general case to

the case d =1,D =2 described above: this reduction is precisely taking place in equation

(3.108), where we project the optimality equation onto the 2-dimensional subspace . The U arguments that follow this projection consist of nothing more than a technical treatment of

the intuition given above.

Discussion of Theorem 3.12. Towards interpreting Theorem 3.12, consider ﬁrst the asymp-

totic case where we allow N and M to go to inﬁnity, while keeping the ratio γ constant,

and notice that the quantities O , X are always bounded from above by D 1. R ,K1 R ,K2 −

65 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Assuming that both inliers and outliers are perfectly well distributed in the limit, i.e., under the hypothesis that limN Sd(X )=0 and limM SD(O)=0, Lemma 3.7 and →∞ →∞ inequality (3.83) give that limN X = 0 and limM O = 0, in which case (3.97) is →∞ →∞ satisﬁed. This suggests the interesting fact that (3.3) is possible to give a normal to the inliers even for arbitrarily many outliers, and irrespectively of the subspace dimension d.

Along the same lines, for a given γ and under the point set uniformity hypothesis, we can always increase the number of inliers and outliers (thus decreasing X and O), while keeping γ constant, until (3.97) is satisﬁed, once again indicating that (3.3) is possible to yield a normal to the space of inliers irrespectively of their intrinsic dimension. Notice that the intrinsic dimension d of the inliers manifests itself through the quantity cd, which we recall is a decreasing function of d. Consequently, the smaller d is the larger the RHS of (3.97) becomes, and so the easier it is to satisfy (3.97).

3.2.3.3 Conditions for convergence of the discrete recursive algorithm

In this section we study the discrete recursion (3.4). We begin by stating and proving two results already known in [85].

Lemma 3.13 ( [85]). Let Y = [y ,..., y ] be a D L matrix of rank D. Then problem 1 L × min > Y >b admits a computable solution n that is orthogonal to (D 1) b nˆ k=1 1 k+1 − linearly independent points of Y .

Proof. Let n be a solution to min > Y >b that is orthogonal to less than D 1 k+1 b nˆ k=1 1 − linearly independent points of Y . Then we can ﬁnd a unit norm vector ζ that is orthogonal

66 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT to the same points of Y that n is orthogonal to, and moreover ζ n . In addition, k+1 ⊥ k+1 by similar arguments as in the proof of Proposition 3.11, we can ﬁnd a sufﬁciently small

ε> 0 such that

Y >(nk+1 + εζ) = Y >nk+1 + ε Sign(yj>nk+1)ζ>yj, (3.116) 1 1 j: nk+1 yj X6⊥ where

Sign(yj>nk+1)ζ>yj =0. (3.117) j: nk y X+16⊥ j

Since nk+1 is optimal, it must be the case that

Sign(yj>nk+1)ζ>yj =0, (3.118) j: nk y X+16⊥ j and so

Y > n ζ Y >n (3.119) ( k+1 + ε ) 1 = k+1 1 .

By (3.119) we see that as we vary ε the objective remains unchanged. Notice also that varying ε preserves all zero entries appearing in the vector Y >nk+1. Furthermore, because of (3.118), it is always possible to either decrease or increase ε until an additional zero is achieved, i.e., until nk+1 + εζ becomes orthogonal to a point of Y that nk+1 is not orthogonal to. Then we can replace nk+1 with nk+1 + εζ and repeat the process, until we get some n that is orthogonal to D 1 linearly independent points of Y . k+1 −

Theorem 3.14 ( [85]). Let Y = [y ,..., y ] be a D L matrix of rank D. Suppose 1 L × that for each problem min > Y >b , a solution n is chosen such that n is b nˆ k=1 1 k+1 k+1

67 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

orthogonal to D 1 linearly independent points of Y , in accordance with Lemma 3.13. −

Then the sequence n converges to a critical point of problem min > Y >b in a { k} b b=1 1

ﬁnite number of steps.

Proof. If nk+1 = nˆ k, then inspection of the ﬁrst order optimality conditions of the two problems, reveals that nˆ is a critical point of min > Y >b . If n = nˆ , then k b b=1 1 k+1 6 k

n > 1, and so Y >nˆ < Y >nˆ . As a consequence, if n = nˆ , then k k+1k2 k+1 1 k 1 k+1 6 k nˆ k can not arise as a solution for some k0 > k. Now, because of Lemma 3.13, for each k,

there is a ﬁnite number of candidate directions nk+1. These last two observations imply

that the sequence n must converge in a ﬁnite number of steps to a critical point of { k}

> Y >b . minb b=1 1

We are now ready for the main theorem of this section, Theorem 3.15. Before stating

and proving the theorem, note that according to Theorem 3.3, the continuous recursion

converges in a ﬁnite number of iterations to a vector that is orthogonal to Span(X ) = , S as long as the initialization nˆ does not lie in (equivalently φ > 0). Intuitively, one 0 S 0 should expect that in passing to the discrete case, the conditions for the discrete recursion

(3.4) to converge to an element of ⊥ should be at least as strong as the conditions of S

Theorem 3.12, and strictly stronger than the condition φ0 > 0 of Theorem 3.4. Our next result formalizes this intuition.

Theorem 3.15. Suppose that condition (3.97) holds true and consider the vector sequence

nˆ k k 0 generated by the recursion (3.4). Let φ0 be the principal angle of nˆ 0 from { } ≥

68 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Span(X ) and suppose that

1 cd X 2 γO φ0 > cos− − − . (3.120) c + X d

Then after a ﬁnite number of iterations the sequence nˆ k k 0 converges to a unit `2-norm { } ≥ vector that is orthogonal to Span(X ).

Proof. We start by establishing that nˆ does not lie in the inlier space . For the sake of k S contradiction suppose that nˆ for some k > 0. Note that k ∈S

˜> ˜> ˜> ˜> X nˆ 0 X n1 X nˆ 1 X nˆ k . (3.121) 1 ≥ 1 ≥ 1 ≥···≥ 1

Suppose ﬁrst that nˆ . Then (3.121) gives 0 ⊥S

O>nˆ O>nˆ + X >vˆ , (3.122) 0 1 ≥ k 1 k 1

where vˆ is the normalized projection of nˆ onto (and since nˆ , these two are k k S k ∈ S equal). Using Lemma 3.9, we take an upper bound of the LHS and a lower bound of the

RHS of (3.122), and obtain

Mc + MO Mc MO + Nc NX , (3.123) D ≥ D − d − or equivalently

c X γ d − , (3.124) ≥ 2 O which contradicts (3.97). Consequently, nˆ . Then (3.121) implies that 0 6⊥ S

O>nˆ + X >nˆ O>nˆ + X >nˆ , (3.125) 0 1 0 1 ≥ k 1 k 1

69 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

or equivalently

O>nˆ + cos(φ ) X >vˆ O>nˆ + X >vˆ , (3.126) 0 1 0 0 1 ≥ k 1 k 1

where vˆ is the normalized projection of nˆ onto . Once again, Lemma 3.9 is used to 0 0 S furnish an upper bound of the LHS and a lower bound of the RHS of (3.126), and yield

Mc + MO +(Nc + NX )cos(φ ) Mc MO + Nc NX , (3.127) D d 0 ≥ D − d −

which contradicts (3.120).

Now let us complete the proof of the theorem. We know by Theorem 3.14 that the

sequence n converges to a critical point n ∗ of problem (3.3) in a ﬁnite number of { k} k

steps, and we have already shown that n ∗ (see the beginning of the current proof). k 6∈ S

Then an identical argument as in the proof of Theorem 3.12 (with nk∗ in place of b∗) shows

that n ∗ must be orthogonal to . k S

Discussion of Theorem 3.15. First note that if (3.97) is true, then the expression of (3.120)

always deﬁnes an angle between 0 and π/2. Moreover, Theorem 3.15 can be interpreted

using the same asymptotic arguments as Theorem 3.12; notice in particular that the lower

bound on the angle φ0 tends to zero as M,N go to inﬁnity with γ constant, i.e., the more uniformly distributed inliers and outliers are, the closer n0 is allowed to be to Span(X ).

We also emphasize that Theorem 3.15 asserts the correctness of the linear programming recursions (3.4) as far as recovering a vector n ∗ orthogonal to := Span(X ) is k S concerned. Even though this was our initial motivation for posing problem (3.3), Theorem

3.15 does not assert in general that nk∗ is a global minimizer of problem (3.3). However,

70 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT this is indeed the case, when the inlier subspace is a hyperplane, i.e., d = D 1. This is S − D 1 because, up to a sign, there is a unique vector b S − that is orthogonal to (the normal ∈ S vector to the hyperplane), which, under conditions (3.97) and (3.120), is the unique global minimizer of (3.3), as well as the limit point nk∗ of Theorem 3.15.

3.2.4 Algorithmic contributions

In this section we discuss algorithmic formulations based on the ideas presented so far.

Speciﬁcally, 3.2.4.1 contains the basic Dual Principal Component Pursuit and Analysis § algorithms, which can be implemented by linear programming, while 3.2.4.2- 3.2.4.4 dis- § § cuss alternative DPCP algorithms suitable for noisy or high-dimensional data.

3.2.4.1 Relaxed DPCP and DPCA algorithms

Theorem 3.15 suggests a mechanism for obtaining an element b of ⊥, where = 1 S S Span(X ): run the sequence of linear programs (3.4) until the sequence nˆ converges { k} and identify the limit point with b1. Due to computational constraints, in practice one usu-

˜> ally terminates the recursions when the objective value X nˆ k converges within some 1

small ε, or a maximal number Tmax of recursions is reached, and obtains an approximate normal b1. The resulting Algorithm 3.1 is referred to as DPCP-r, standing for relaxed

DPCP.

We emphasize that step 6 of Algorithm 3.1 can be canonically solved by linear pro-

71 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Algorithm 3.1 Relaxed Dual Principal Component Pursuit ˜ 1: procedure DPCP-R(X ,ε,Tmax)

2: k 0;∆ ; ← J ← ∞

X˜> 3: nˆ 0 argmin b =1 b ; ← k k2 2

4: while kε do max J 5: k k +1; ←

X˜> 6: nk argminb>nˆ =1 b ; ← k−1 1

7: nˆ n / n ; k ← k k kk2

˜> ˜> ˜> 9 8: ∆ X nˆ k 1 X nˆ k / X nˆ k 1 + 10− ; J ← − 1 − 1 − 1

9: end while

10: return nˆ k;

11: end procedure gramming. More speciﬁcally, we can rewrite it in the form

u+ min 1 1   , such that (3.128) b,u+,u− 1 N 1 N × × u−     u+ ˜> IN IN X   0N 1 − − × +   u =   , u , u− 0, (3.129)  − ≥ 0 0    1 N 1 N nˆ k>     1   × ×         b        and solve it efﬁciently with an optimized general purpose linear programming solver, such as Gurobi [43]. Having computed a b1 with Algorithm 3.1, there are two possibilities: either is a hyperplane of dimension D 1 or dim < D 1. In the ﬁrst case we can S − S −

72 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Algorithm 3.2 Relaxed Dual Principal Component Analysis ˜ 1: procedure DPCA-R(X ,c,ε,Tmax)

2: ; B ← ∅ 3: for i =1: c do

4: k 0;∆ ; ← J ← ∞

X˜> 5: nˆ 0 argminb b ; ← ⊥B 2

6: while k T and ∆ >ε do ≤ max J 7: k k +1; ← ˜ 8: n > X >b ; k argminb nˆ k =1,b ← −1 ⊥B 1

9: nˆ n / n ; k ← k k kk2

˜> ˜> ˜> 9 10: ∆ X nˆ k 1 X nˆ k / X nˆ k 1 + 10− ; J ← − 1 − 1 − 1

11: end while

12: b nˆ ; i ← k 13: b ; B ← B ∪ { i} 14: end for

15: return ; B 16: end procedure

identify our subspace model with the hyperplane deﬁned by the normal b1. If on the other hand dim < D 1, we can proceed to ﬁnd a b b that is approximately orthogonal S − 2 ⊥ 1 to , and so on; this naturally leads to the relaxed Dual Principal Component Analysis of S Algorithm 3.2.

73 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

In Algorithm 3.2, c is an estimate for the codimension D d of the inlier subspace − Span( ). If c is rather large, then in the computation of each b , it is more efﬁcient to be X i reducing the coordinate representation of the data to D (i 1) coordinates, by project- − −

ing the data orthogonally onto Span(b1,..., bi 1)⊥ and solving the linear program in the − projected space.

Notice further how the algorithm initializes n0: This is precisely the right singular vector of X˜> that corresponds to the smallest singular value, after projection of X˜ onto

Span(b1,..., bi 1)⊥. As it will be demonstrated in section 3.2.5.1, this choice has the −

effect that the angle of n0 from the inlier subspace is typically large; more precisely, it

is often larger than the smallest initial angle (3.120) required for the success of the linear

programming recursions.

3.2.4.2 Relaxed and denoised DPCP

When the data are corrupted by noise, one no longer expects the product X˜>b to be sparse, even if b is orthogonal to the underlying inlier space. Instead, our expectation is that, for such a b, X˜>b should be the sum of a sparse vector with a dense vector of small euclidean norm. This motivates us to replace the optimization problem

˜> min X b , s.t. b>nˆ k 1 =1, (3.130) b 1 −

which appears in Algorithm 3.1, with the denoised optimization problem

1 2 X˜> min τ y 1 + y b , s.t. b>nˆ k 1 =1, (3.131) y,b k k 2 − 2 −

74 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

where τ is a positive parameter. Observe, that if the optimal b were known, then y would ˜ be given by the element-wise soft thresholding S (X >b), where the function S : R R τ τ → is deﬁned by S (α) = Sign(α) max(0, α τ). If on the other hand y were known, then τ · | |−

b would be given by b = nˆ k 1 + U k 1z, where U k 1 is a D (D 1) matrix containing − − − × −

in its columns an orthonormal basis for the orthogonal complement of nˆ k 1, and z is the − solution to the standard least-squares problem

2 ˜> ˜> min y X nˆ k 1 X U k 1z . (3.132) z RD−1 − − − − 2 ∈

Observe that solving problem (3.132) requires solving a linear system of equations with

˜> > ˜> coefﬁcient matrix X U k 1 X U k 1. The dependence of this matrix on the iteration − − index k, may become a computational issue when D is large. This dependence can be

circumvented as follows. First, we treat the computation of b given y as a constrained

problem

2 ˜> min y X b , , s.t. b>nˆ k 1 =1, (3.133) b − 2 −

and thinking in terms of a Lagrange multiplier λ associated to the constraint b>nˆ k 1 = 1, − we see that λ, b must satisfy the relation

˜ ˜ ˜> X y X X b λnˆ k 1 =0. (3.134) − − −

Noting that X˜X˜> is always invertible under our data model, we must have that

1 1 ˜ ˜> − ˜ ˜> − b = X X y λ X X nˆ k 1. (3.135) − −

75 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Now, multiplying the above equation from the left with nˆ k> 1, we obtain −

1 1 X˜X˜> − X˜X˜> − 1= nˆ k> 1 y λnˆ k> 1 nˆ k 1, (3.136) − − − − or equivalently

1 X˜X˜> − nˆ k> 1 y 1 λ = − − , (3.137) 1 X˜X˜> − nˆ k> 1 nˆ k 1 − − which upon substitution into (3.135) gives our ﬁnal formula for b:

1 X˜X˜> − 1 nˆ k> 1 y 1 1 ˜ ˜> − − − ˜ ˜> − b = X X y X X nˆ k 1. (3.138)  1  − − X˜X˜> − nˆ k> 1 nˆ k 1  − −    1 1 ˜ ˜> − ˜ ˜> − Notice that the quantities X X y and X X nˆ k 1 can be obtained as solutions − to linear systems of equations with common coefﬁcient matrix X˜X˜>, which themselves can be solved very efﬁciently by backward and forward substitution, assuming that a precomputed Cholesky factorization of X˜X˜> is available.

To summarize, we have shown how to approximately solve problem (3.130) by a very efﬁcient alternating minimization scheme, which involves soft-thresholding and forward- backward substitution; we will be referring to the resulting DPCP algorithm as DPCP-r-d, which stands for relaxed and denoised DPCP.

3.2.4.3 Denoised DPCP

Interestingly, DPCP-r-d is very closely related to the scheme proposed in [80], where the authors study problem (3.3) in the very different context of dictionary learning. To approx-

76 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT imately solve (3.3), the authors of [80] solve its denoised version

1 2 X˜> min τ y 1 + y b , (3.139) b,y: b 2=1 k k 2 − 2 || || by alternating minimization. Given b, the optimal y is given by X˜>b , where is the Sτ Sτ soft-thresholding operator applied element-wise on the vector X˜>b. Given y the optimal b is a solution to the quadratically constrained least-squares problem

2 X˜> min y b , s.t. b 2 =1. (3.140) b RD − 2 k k ∈

In the context of [80], the coefﬁcient matrix of the least-squares problem (X˜> in our notation) has orthonormal columns. As a consequence, the solution to (3.140) is obtained in closed form by projecting the solution of the unconstrained least-squares problem

˜ min y X >b (3.141) b RD − 2 ∈ onto the unit sphere. However, in our context the assumption that X˜> has orthonormal columns is strongly violated, so that the optimal b is no longer available in closed form.

In fact, problem (3.140) is well known in the literature [26,35,38], and the standard way to solve it is by means of Lagrange multipliers. This involves solving a non-linear equation for the Lagrange multiplier, which is known to be challenging [26]. For this reason we leave exact approaches for solving (3.140) to future investigations, and we instead propose to obtain an approximate b as in [80]. We will call the resulting Algorithm 3.3 DPCP-d, which stands for denoised DPCP.

Notice that DPCP-d is very efﬁcient, since the least-squares problems that appear in the various iterations have the same coefﬁcient matrix X˜X˜>, a factorization of which can be

77 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Algorithm 3.3 Denoised Dual Principal Component Pursuit ˜ 1: procedure DPCP-D(X ,ε,Tmax,δ,τ)

˜ ˜> 2: Compute a Cholesky factorization LL> = X X + δID;

3: k 0;∆ ; ← J ← ∞

X˜> 4: b argminb RD: b =1 b ; ← ∈ k k2 2

˜> 5: 0 τ X b ; J ← 1

6: while kε do max J 7: k k +1; ← ˜ 8: y X >b ; ←Sτ ˜ 9: b solution of LL>ξ = X y by backward/forward propagation; ← 10: b b/ b ; ← k k2 2 1 X˜> 11: k τ y 1 + 2 y b ; J ← k k − 2

9 12: ∆ ( k 1 k) / ( k 1 + 10− ); J ← J − −J J − 13: end while

14: return (y, b);

15: end procedure

precomputed 8. As we will see in section 3.2.5 , the performance of DPCP-d is remarkably close to that of DPCP-r, for which we have guarantees of global optimality, suggesting that

DPCP-d converges to a global minimum. We leave theoretical investigations of DPCP-d to future research. 8The parameter δ in Algorithm 3.3 is a small positive number, typically 10−6, which helps avoiding solving ill-conditioned linear systems.

78 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Finally, notice that Algorithm 3.3 computes a single normal vector b1. As with DPCP-r, it is trivial to adjust Algorithm 3.3 to compute a second normal vector b2, since one only needs to incorporate the linear constraint b>b1 =0, and so on. We will again refer to such an algorithm that computes c 1 normals as DPCP-d. ≥

3.2.4.4 DPCP via iteratively reweighted least-squares

Even though DPCP-d and DPCP-r-d are very attractive from a computational point of view,

they only produce approximate normal vectors (even in the absence of noise), and have the

additional disadvantage that their performance depends on the parameter τ, whose tuning

is not well understood. On the other hand, DPCP-r was shown to produce exact normal

vectors, yet solving linear programs of the form (3.128)-(3.129) can be inefﬁcient when the

data are high-dimensional. This motivates us to propose a direct IRLS algorithm for solving

the DPCP problem (3.3). In fact, since we are interested in obtaining an orthonormal basis

for the orthogonal complement of the inlier subspace, we propose an Iteratively Reweighted

Least-Squares (IRLS) scheme for solving problem

˜> min X B , s.t. B>B = Ic, (3.142) B RD×c 1,2 ∈

which is a generalization of the DPCP problem for multiple normal vectors. More speciﬁ-

cally, given a D c orthonormal matrix Bk 1, we deﬁne for each point x˜j a weight × −

1 wj,k := , (3.143) max δ, Bk> 1x˜j − 2

79 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

where δ > 0 is a small constant that prevents division by zero. Then we obtain Bk as the solution to the quadratic problem

L 2 min wj,k B>x˜j , s.t. B>B = Ic, (3.144) B RD×c 2 ∈ j=1 X which is readily seen to be the c right singular vectors corresponding to the c smallest

˜> singular values of the weighted data matrix W kX , where W is a diagonal matrix with

√wj,k at position (j,j). We refer to the resulting Algorithm 3.4 as DPCP-IRLS; a study

of its theoretical properties is deferred to future work. We note here that the technique

of solving an optimization problem (convex or non-convex) through IRLS is a common

one. In fact, REAPER [62] solves through IRLS a convex relaxation of precisely problem

(3.142). Other prominent instances of IRLS schemes from compressed sensing are [13,14,

19].

3.2.5 Experimental evaluation

In this section we evaluate the proposed algorithms experimentally. In 3.2.5.1 we inves- § tigate the performance of our principal algorithmic proposal, i.e., DPCP-r, described in

Algorithm 3.1. In 3.2.5.2 we compare DPCP-r, DPCP-d, DPCP-r-d 9 and DPCP-IRLS § with state-of-the-art robust PCA algorithms using synthetic data, and similarly in 3.2.5.3 § using real images.

9To avoid confusion, we will slightly abuse our terminology and refer to DPCA-r (DPCA-d, DPCA-r-d) as DPCP-r (DPCP-d, DPCP-r-d), even when c> 1. The distinction of whether a single versus multiple dual principal components are being computed will be clear from the context.

80 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Algorithm 3.4 Dual Principal Component Pursuit via Iteratively Reweighted Least Squares ˜ 1: procedure DPCP-IRLS(X ,c,ε,Tmax,δ)

2: k 0;∆ ; ← J ← ∞

X˜> 3: B0 argminB RD×c B , s.t. B>B = c; ← ∈ 2 I

4: while kε do max J 5: k k +1; ← X˜ 6: wx˜ 1/ max δ, Bk> 1x˜ , x˜ ; ← − 2 ∈ 2 7: B D×c ˜ B>x s.t. B>B ; k argminB R x˜ X wx˜ ˜ 2 = c ← ∈ ∈ I

˜> P ˜> ˜> 9 8: ∆ X Bk 1 X Bk / X Bk 1 + 10− ; J ← − 1 − 1 − 1

9: end while

10: return Bk;

11: end procedure

3.2.5.1 Computing a single dual principal component

We begin by investigating the behavior of the DPCP-r Algorithm 3.1 in the absence of

noise, for random subspaces of varying dimensions d =1:1:29 and varying outlier S percentages R := M/(M + N)=0.1:0.1:0.9. We ﬁx the ambient dimension D = 30,

D 1 sample N = 200 inliers uniformly at random from S − and M outliers uniformly S ∩ D 1 3 at random from S − . We set = 10− and Tmax = 10 in Algorithm 3.1. Our main interest is in examining the ability of DPCP-r in recovering a single normal vector to the subspace (c =1). The results over 10 independent trials are shown in Figure 3.5, in which

81 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

the vertical axis denotes the relative dimension of the subspace, deﬁned as d/D.

Figure 3.5(a) shows whether the theoretical condition (3.97) is satisﬁed (white) or not

(black). In checking this condition, we estimate the abstract quantities

O,X , O , X (3.145) R ,K1 R ,K2

by Monte-Carlo simulation. Whenever the condition is true, we choose nˆ 0 in a controlled fashion, so that its angle φ0 from the subspace is larger than the minimal angle φ0∗ of (3.120); then we run DPCP-r. If on the other hand (3.97) is not true, we do not run DPCP-r and report a 0 (black). Fig 3.5(b) shows the angle of nˆ 10 from the subspace. We see that whenever

(3.97) is true, DPCP-r returns a normal after only 10 iterations. Fig 3.5(c) shows that if we initialize randomly nˆ 0, then its angle φ0 from the subspace tends to become less than the

minimal angle φ0∗, as d increases. Even so, Figure 3.5(d) shows that DPCP-r still yields a

numerical normal, except for the regime where both d and R are very high. Notice that this

is roughly the regime where we have no theoretical guarantees, according to Figure 3.5(a).

˜> Figure 3.5(e) shows that if we initialize nˆ 0 as the right singular vector of X corresponding

to the smallest singular value, then φ0 > φ0∗ is true for most cases, and the corresponding

performance of DPCP-r in Figure 3.5(f) improves further. Finally, Figure 3.5(g) plots φ . 0∗ We see that for very low d this angle is almost zero, i.e. DPCP-r does not depend on the

initialization, even for large R. As d increases though, so does φ0∗, and in the extreme case

o of the upper rightmost regime, where d and R are very high, φ0∗ is close to 90 , verifying

our expectation that DPCP-r will succeed only if nˆ is very close to ⊥. 0 S

82 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

0.97 0.97 0.97 0.83 0.83 0.83

0.67 0.67 0.67

0.5 0.5 0.5

0.33 0.33 0.33 relative dimension relative dimension 0.17 relative dimension 0.17 0.17 0.03 0.03 0.03 0.1 0.5 0.9 0.1 0.5 0.9 0.1 0.5 0.9 outlier ratio outlier ratio outlier ratio

∗ ∗ (a) eq. (3.97) (b) DPCP-r(φ0 >φ0) (c) φ0 >φ0

0.97 0.97 0.97 0.83 0.83 0.83

0.67 0.67 0.67

0.5 0.5 0.5

0.33 0.33 0.33

relative dimension 0.17 relative dimension 0.17 relative dimension 0.17 0.03 0.03 0.03 0.1 0.5 0.9 0.1 0.5 0.9 0.1 0.5 0.9 outlier ratio outlier ratio outlier ratio

∗ (d) DPCP-r(random φ0) (e) φ0,SVD >φ0 (f) DPCP-r(φ0,SVD)

0.97 0.83

0.67

0.5

0.33

relative dimension 0.17 0.03 0.1 0.5 0.9 outlier ratio

∗ (g) φ0

Figure 3.5: Various quantities associated to the performance of DPCP-r; see 3.2.5.1. Fig- § ure 3.5(a) shows whether condition (3.97) is true (white) or not (black). Figure 3.5(b) shows the angle from of nˆ after 10 iterations of DPCP-r when (3.97) is true; the other S 10 cases are mapped to black. Figure 3.5(c) shows whether a random nˆ 0 satisﬁes φ0 > φ0∗,

where φ∗ is as in Thm. 3.15. Figure 3.5(d) shows the angle from of nˆ for random nˆ . 0 S 10 0 Figure 3.5(e) shows the angle from of the right singular vector of X˜ corresponding to S

the smallest singular value, Figure 3.5(f) shows the corresponding angle of nˆ 10, and Figure

3.5(g) plots φ . 0∗ 83 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

3.2.5.2 Outlier detection using synthetic data

In this section we begin by using the same synthetic experimental set-up as in 3.2.5.1 § (except that now N = 300) to demonstrate the behavior of several methods relative to each other under uniform conditions, in the context of outlier rejection in single subspace learning. In particular, we test DPCP-r, DPCP-d, DPCP-r-d, DPCP-IRLS, the IRLS version of REAPER [62], RANSAC [34], SE-RPCA [84], and `2,1-RPCA [118]; see Chapter 2 for details on existing methods.

For the methods that require an estimate of the subspace dimension d, such as REAPER,

RANSAC, and all DPCP variants, we provide as input the true subspace dimension. The

3 convergence accuracy of all methods is set to 10− . For REAPER we set the regularization

6 parameter δ equal to 10− and the maximal number of iterations equal to 100. For DPCP-

r we set τ = 1/√N + M as suggested in [80] and maximal number of iterations 1000.

3 For RANSAC we set its thresholding parameter equal to 10− , and for fairness, we do not let it terminate earlier than the running time of DPCP-r. Both SE-RPCA and `2,1-

RPCA are implemented with ADMM, with augmented Lagrange parameters 1000 and 100 respectively. For `2,1-RPCA λ is set to 3/(7√M), as suggested in [118]. DPCP variants are initialized as in Algorithm 3.2, and the parameters of DPCP-r are as in 3.2.5.1. § Absence of noise. We investigate the potential of each of the above methods to perfectly distinguish outliers from inliers in the absence of noise 10. Note that each method

10We do not include the results of DPCP-d and DPCP-r-d for this experiment, since they only approximately solve the DPCP optimization problem, and hence they can not be expected to perfectly separate inliers from outliers, even when there is no noise.

84 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

0.97 0.97 0.97 0.83 0.83 0.83

0.67 0.67 0.67

0.5 0.5 0.5

0.33 0.33 0.33 relative dimension relative dimension relative dimension 0.17 0.17 0.17 0.03 0.03 0.03 0.1 0.3 0.5 0.7 0.9 0.1 0.3 0.5 0.7 0.9 0.1 0.3 0.5 0.7 0.9 outlier ratio outlier ratio outlier ratio

(a) REAPER (b) RANSAC (c) SE-RPCA

0.97 0.97 0.97 0.83 0.83 0.83

0.67 0.67 0.67

0.5 0.5 0.5

(d) `2,1-RPCA (e) DPCP-IRLS (f) DPCP-r

Figure 3.6: Outlier/Inlier separation in the absence of noise over 10 independent trials.

The horizontal axis is the outlier ration deﬁned as M/(N + M), where M is the number

of outliers and N is the number of inliers. The vertical axis is the relative inlier subspace

dimension d/D; the dimension of the ambient space is D = 30. Success (white) is declared

by the existence of a threshold that, when applied to the output of each method, perfectly

separates inliers from outliers.

returns a signal α RN+M , which can be thresholded for the purpose of declaring outliers ∈ +

and inliers. For SE-RPCA, α is the `1-norm of the columns of the coefﬁcient matrix C,

while for `2,1-RPCA it is the `2-norm of the columns of E. Since REAPER, RANSAC,

DPCP-r and DPCP-IRLS directly return subspace models, for these methods α is simply

the distances of all points to the estimated subspace.

85 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

In Figure 3.6 we depict success (white) versus failure (black), where success is inter-

preted as the existence of a threshold on α that perfectly separates outliers and inliers.

First observe that, as expected, SE-RPCA and `2,1-RPCA succeed only when the subspace dimension d is small. In particular, the more outliers are present the lower the dimension of the subspace needs to be for the methods to succeed. The same is true for RANSAC, except when there are only very few outliers (10%), in which case the probability of sampling outlier-free points is high. Finally notice that SE-RPCA is the best method among these three in dealing with large percentages of outliers (> 70%). This is not a surprise, because the theoretical guarantees of SE-RPCA do not place an explicit upper bound on the number of outliers, in contrast to `2,1-RPCA and RANSAC. Next, notice that REAPER performs uniformly better than RANSAC, SE-RPCA and `2,1-RPCA. In particular REAPER can handle higher dimensions and higher outlier percentages; for example it succeeds over all trials for hyperplanes when there are 10% outliers.

In summary, none of REAPER, RANSAC, SE-RPCA, and `2,1-RPCA can deal with hyperplanes with more than 20% outliers or with subspaces of medium relative dimension

(d > 13) for as many as 90% outliers. This gap is ﬁlled by the two proposed methods DPCP-r and DPCP-IRLS. Notice that DPCP-r is the only method that succeeds irrespectively of subspace dimension with almost 70% outliers, while DPCP-IRLS is the only method that succeeds when d 19 and R = 90%. ≤ Presence of Noise. Next, we keep D = 30 and investigate the performance of the

methods, adding DPCP-d and DPCP-r-d in the mix, in the presence of varying levels of

86 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

noise and outliers for two cases of high-dimensional subspaces, i.e., d = 25 and d =

29. The inliers are corrupted by additive white gaussian noise of zero mean and standard deviation σ = 0.02, 0.06, 0.1, with support in the orthogonal complement of the inlier subspace, while the percentage of outliers varies as R = 20%, 33%, 50%. Finally, for

DPCP-d and DPCP-r-d we set τ = max σ, 1/√N + M , while for RANSAC we set its threshold equal to σ.

We evaluate the performance of each method by its corresponding ROC curve. Each

point of an ROC curve corresponds to a certain value of a threshold, with the vertical

coordinate of the point giving the percentage of inliers being correctly identiﬁed as inliers

(True Positives), and the horizontal coordinate giving the number of outliers erroneously

identiﬁed as inliers (False Positives). As a consequence, an ideal ROC curve should be

concentrated to the top left of the ﬁrst quadrant, i.e., the area over the curve should be zero.

The ROC curves for the case d = 25 are given in Figure 3.7, where for each curve we also report the area over the curve. As expected, the low-rank methods RANSAC,

SE-RPCA and `2,1-RPCA perform poorly with RANSAC being the worst method for 50% outliers and SE-RPCA being the weakest method otherwise. On the other hand REAPER,

DPCP-d, DPCP-IRLS, DPCP-r-d and DPCP-r perform almost perfectly well, with DPCP-

IRLS actually giving zero error across all cases. Notice the interesting fact that DPCP-r performs slightly better across all cases than both DPCP-d and DPCP-r-d, despite the fact that DPCP-d and DPCP-r-d are intuitively more suitable for noisy data than DPCP-r.

The ROC curves for the case d = 29 are given in Figure 3.8. As expected, the low-rank

87 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

1 1 1

0.8 0.8 0.8 RANSAC(0.038) (0.147) (0.401) ℓ 0.6 21-RPCA(0.067) 0.6 (0.132) 0.6 (0.202) SE-RPCA(0.109) (0.173) (0.230) REAPER(0) (0.000) (0.010) 0.4 DPCP-r(0) 0.4 (0) 0.4 (0.001) DPCP-r-d(0) (0.000) (0.012) 0.2 DPCP-d(0) 0.2 (0) 0.2 (0.007) DPCP-IRLS(0) (0) (0) 0 0.2 0.4 0.6 0.8 0 0.2 0.4 0.6 0.8 0 0.2 0.4 0.6 0.8 1

(a) σ = 0.02,R = 20% (b) σ = 0.02,R = 33% (c) σ = 0.02,R = 50%

1 1 1

0.8 0.8 0.8 (0.054) (0.143) (0.269) (0.069) 0.6 0.6 (0.138) 0.6 (0.201) (0.112) (0.179) (0.233) (0) (0.000) (0.009) 0.4 (0) 0.4 (0) 0.4 (0.000) (0) (0.000) (0.006) 0.2 (0) 0.2 (0) 0.2 (0.002) (0) (0) (0.000) 0 0.2 0.4 0.6 0.8 0 0.2 0.4 0.6 0.8 0 0.2 0.4 0.6 0.8 1

(d) σ = 0.06,R = 20% (e) σ = 0.06,R = 33% (f) σ = 0.06,R = 50%

1 1 1 0.8 0.8 (0.077) 0.8 (0.163) (0.278) 0.6 (0.080) (0.139) 0.6 (0.207) (0.125) 0.6 (0.181) (0.236) (0.000) (0.001) (0.013) 0.4 0.4 (0.000) 0.4 (0.000) (0.005) (0.000) (0.001) (0.015) (0.000) (0.007) 0.2 0.2 (0.000) 0.2 (0.000) (0.000) (0.000) 0 0.2 0.4 0.6 0.8 0 0.2 0.4 0.6 0.8 0 0.2 0.4 0.6 0.8 1

(g) σ = 0.1,R = 20% (h) σ = 0.1,R = 33% (i) σ = 0.1,R = 50%

Figure 3.7: ROC curves as a function of noise standard deviation σ and outlier percentage

R, for subspace dimension d = 25 in ambient dimension D = 30. The horizontal axis

is False Positives ratio and the vertical axis is True Positives ratio. The number associ-

ated with each curve is the area above the curve; smaller numbers reﬂect more accurate

performance.

88 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

1 1 1

0.8 0.8 0.8 RANSAC(0.276) (0.417) (0.468) ℓ (0.406) 0.6 21-RPCA(0.355) 0.6 0.6 (0.427) SE-RPCA(0.381) (0.421) (0.436) REAPER(0.033) (0.111) (0.175) 0.4 DPCP-r(0.016) 0.4 (0.014) 0.4 (0.016) DPCP-r-d(0.017) (0.019) (0.058) 0.2 DPCP-d(0.016) 0.2 (0.015) 0.2 (0.018) DPCP-IRLS(0.018) (0.016) (0.020) 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

(a) σ = 0.02,R = 20% (b) σ = 0.02,R = 33% (c) σ = 0.02,R = 50%

1 1 1

0.8 0.8 0.8 (0.302) (0.388) (0.448) (0.424) 0.6 (0.365) 0.6 (0.397) 0.6 (0.387) (0.409) (0.430) (0.053) (0.111) (0.190) 0.4 (0.039) 0.4 (0.042) 0.4 (0.045) (0.038) (0.047) (0.085) 0.2 (0.038) 0.2 (0.040) 0.2 (0.048) (0.040) (0.045) (0.052) 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

(d) σ = 0.06,R = 20% (e) σ = 0.06,R = 33% (f) σ = 0.06,R = 50%

1 1 1

0.8 0.8 0.8 (0.312) (0.392) (0.456) 0.6 (0.369) 0.6 (0.404) 0.6 (0.428) (0.392) (0.416) (0.434) (0.086) (0.129) (0.188) 0.4 (0.071) 0.4 (0.072) 0.4 (0.077) (0.072) (0.077) (0.105) 0.2 (0.068) 0.2 (0.069) 0.2 (0.077) (0.073) (0.075) (0.084) 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

(g) σ = 0.1,R = 20% (h) σ = 0.1,R = 33% (i) σ = 0.1,R = 50%

Figure 3.8: ROC curves as a function of noise standard deviation σ and outlier percentage

R, for subspace dimension d = 29 in ambient dimension D = 30. The horizontal axis

is False Positives ratio and the vertical axis is True Positives ratio. The number associ-

ated with each curve is the area above the curve; smaller numbers reﬂect more accurate

performance.

89 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

methods RANSAC, SE-RPCA and `2,1-RPCA fail even for low noise (σ =0.02) and mod-

erate outliers (20%). Notice that for 50% outliers and for any threshold, there is roughly an

equal chance of identifying a point as inlier or as outlier, i.e., the performance of the meth-

ods is almost the same as that of a random guess. On the other hand, DPCP-r, DPCP-d,

and DPCP-IRLS are very robust to variations of the noise level and outlier percentages and

are the best methods. DPCP-r-d performs almost identically with these methods, except in

the case of 50% outliers, where it performs less accurately. Finally, notice that the perfor-

mance of REAPER degrades signiﬁcantly as soon as the outlier percentage exceeds 20%,

indicating that REAPER is not the best method for subspaces of very low codimension.

3.2.5.3 Outlier detection using real face and object images

In this section we use the Extended-Yale-B real face dataset [36] as well as the real image

dataset Caltech101 [32] to compare the proposed algorithms DPCP-r, DPCP-d, DPCP-r-d

and DPCP-IRLS to REAPER, RANSAC, `2,1-RPCA, and SE-RPCA. We recall that the

Extended-Yale-B dataset contains 64 face images for each of 38 distinct individuals. We

use the ﬁrst 19 individuals from the Extended-Yale-B dataset and the ﬁrst half images of

each category in Caltech101 for gaining intuition and tuning the parameters of each method

(training set), while the remaining part of the datasets serve as a test set.

In the Extended-Yale-B dataset, all face images correspond to the same ﬁxed pose,

while the illumination conditions vary. Such images are known to lie in a 9-dimensional

linear subspace, with each individual having its own corresponding subspace [4]. In this

90 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

1 1 1

0.8 0.8 0.8 RANSAC(0.256) (0.250) (0.260) ℓ -RPCA(0.043) 0.6 21 0.6 (0.069) 0.6 (0.101) SE-RPCA(0.078) (0.095) (0.153) REAPER(0.051) (0.061) (0.072) 0.4 DPCP-r(0.070) 0.4 (0.068) 0.4 (0.080) DPCP-r-d(0.049) (0.065) (0.087) 0.2 DPCP-d(0.049) 0.2 (0.065) 0.2 (0.087) DPCP-IRLS(0.068) (0.069) (0.074) 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

(a) R = 20%,C0 N0 (b) R = 33%,C0 N0 (c) R = 50%,C0 N0 − − −

1 1 1

0.8 0.8 0.8 (0.233) (0.238) (0.239) 0.6 (0.137) 0.6 (0.157) 0.6 (0.197) (0.165) (0.182) (0.238) (0.125) (0.137) (0.165) 0.4 (0.166) 0.4 (0.151) 0.4 (0.172) (0.127) (0.136) (0.173) 0.2 (0.141) 0.2 (0.145) 0.2 (0.176) (0.149) (0.141) (0.164) 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

(d) R = 20%,C0 N1 (e) R = 33%,C0 N1 (f) R = 50%,C0 N1 − − −

1 1 1

0.8 0.8 0.8 (0.136) (0.133) (0.161) 0.6 (0.045) 0.6 (0.060) 0.6 (0.091) (0.050) (0.083) (0.150) (0.055) (0.050) (0.064) 0.4 (0.069) 0.4 (0.052) 0.4 (0.075) (0.050) (0.050) (0.076) 0.2 (0.050) 0.2 (0.050) 0.2 (0.076) (0.084) (0.055) (0.067) 0 0.2 0.4 0.6 0 0.2 0.4 0.6 0 0.2 0.4 0.6 0.8

(g) R = 20%,C1 N0 (h) R = 33%,C1 N0 (i) R = 50%,C1 N0 − − −

1 1 1

0.8 0.8 0.8 (0.129) (0.133) (0.141) 0.6 (0.051) 0.6 (0.065) 0.6 (0.088) (0.047) (0.078) (0.146) (0.059) (0.062) (0.064) 0.4 (0.070) 0.4 (0.065) 0.4 (0.071) (0.063) (0.060) (0.075) 0.2 (0.072) 0.2 (0.062) 0.2 (0.078) (0.079) (0.066) (0.068) 0 0.2 0.4 0.6 0 0.2 0.4 0.6 0 0.2 0.4 0.6 0.8

(j) R = 20%,C1 N1 (k) R = 33%,C1 N1 (l) R = 50%,C1 N1 − − −

Figure 3.9: Average ROC curves and areas over the curves for different percentages R of outliers; see 3.2.5.3. Both inliers and outliers come from EYaleB. C1 means data are centered (C0 not centered), N1 means data are normalized (N0 not normalized). 91 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

1 1 1

0.8 0.8 0.8 RANSAC(0.184) (0.152) (0.090) ℓ 0.6 21-RPCA(0.008) 0.6 (0.003) 0.6 (0.048) SE-RPCA(0.337) (0.028) (0.033) REAPER(0.059) (0.011) (0.006) 0.4 DPCP-r(0.170) 0.4 (0.036) 0.4 (0.039) DPCP-r-d(0.045) (0.007) (0.043) 0.2 DPCP-d(0.045) 0.2 (0.007) 0.2 (0.043) DPCP-IRLS(0.165) (0.034) (0.011) 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8

(a) R = 20%,C0 N0 (b) R = 33%,C0 N0 (c) R = 50%,C0 N0 − − −

1 1 1

0.8 0.8 0.8 (0.212) (0.198) (0.189) 0.6 (0.047) 0.6 (0.100) 0.6 (0.224) (0.395) (0.178) (0.177) (0.093) (0.126) (0.143) 0.4 (0.150) 0.4 (0.156) 0.4 (0.172) (0.090) (0.134) (0.185) 0.2 (0.114) 0.2 (0.143) 0.2 (0.187) (0.118) (0.138) (0.147) 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

(d) R = 20%,C0 N1 (e) R = 33%,C0 N1 (f) R = 50%,C0 N1 − − −

1 1 1

0.8 0.8 0.8 (0.093) (0.067) (0.042) (0.002) (0.004) (0.040) 0.6 0.6 0.6 (0.003) (0.019) (0.030) (0.052) (0.012) (0.004) 0.4 (0.108) 0.4 (0.037) 0.4 (0.033) (0.035) (0.008) (0.035) 0.2 (0.035) 0.2 (0.008) 0.2 (0.035) (0.159) (0.043) (0.009) 0 0.1 0.2 0.3 0 0.05 0.1 0.15 0 0.05 0.1 0.15

(g) R = 20%,C1 N0 (h) R = 33%,C1 N0 (i) R = 50%,C1 N0 − − −

1 1 1

0.8 0.8 0.8 (0.089) (0.060) (0.043) 0.6 (0.025) 0.6 (0.013) 0.6 (0.082) (0.014) (0.018) (0.043) (0.022) (0.018) (0.012) 0.4 (0.086) 0.4 (0.031) 0.4 (0.064) (0.031) (0.022) (0.070) 0.2 (0.045) 0.2 (0.023) 0.2 (0.070) (0.079) (0.021) (0.016) 0 0.2 0.4 0 0.05 0.1 0.15 0.2 0 0.05 0.1 0.15 0.2 0.25

(j) R = 20%,C1 N1 (k) R = 33%,C1 N1 (l) R = 50%,C1 N1 − − −

Figure 3.10: Average ROC curves and areas over the curves for different percentages R of outliers; see 3.2.5.3. Inliers come from EYaleB, outliers from Caltech101. C1 means data are centered (C0 not centered), N1 means data are normalized (N0 not normalized). 92 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

1 1 1

0.8 0.8 0.8 RANSAC(0.211) (0.133) (0.145) ℓ 0.6 21-RPCA(0.165) 0.6 (0.060) 0.6 (0.492) SE-RPCA(0.181) (0.083) (0.078) REAPER(0.069) (0.050) (0.055) 0.4 DPCP-r(0.113) 0.4 (0.052) 0.4 (0.059) DPCP-r-d(0.137) (0.050) (0.053) 0.2 DPCP-d(0.137) 0.2 (0.050) 0.2 (0.053) DPCP-IRLS(0.100) (0.055) (0.060) 0 0.2 0.4 0.6 0.8 0 0.2 0.4 0.6 0 0.2 0.4 0.6 0.8 1

(a) D = 15 (b) D = 50 (c) D = 150

Figure 3.11: ROC curves for three different projection dimensions, when there are 33% face outliers; data are centered but not normalized (C N ). 1 − 0

1 1 1

0.8 0.8 0.8 RANSAC(0.078) (0.067) (0.059) ℓ21-RPCA(0.089) (0.004) (0.435) 0.6 0.6 0.6 SE-RPCA(0.123) (0.019) (0.008) REAPER(0.003) (0.012) (0.011) 0.4 DPCP-r(0.064) 0.4 (0.037) 0.4 (0.033) DPCP-r-d(0.077) (0.008) (0.008) 0.2 DPCP-d(0.077) 0.2 (0.008) 0.2 (0.008) DPCP-IRLS(0.039) (0.043) (0.046) 0 0.1 0.2 0.3 0.4 0 0.05 0.1 0.15 0 0.2 0.4 0.6 0.8

(a) D = 15 (b) D = 50 (c) D = 150

Figure 3.12: ROC curves for three different projection dimensions, when there are 33% outliers from Caltech101; data are centered but not normalized (C N ). 1 − 0 experiment we use the 42 48 cropped images that were also used in [30]. Thus, the images × 42 48 2016 of each individual lie approximately in a 9-dimensional linear subspace of R × ∼= R . It is then natural to take as inliers all the images of one individual. For outliers, we consider two possibilities: either the outliers consist of images randomly chosen from the rest of the individuals, or they consist of images randomly chosen from Caltech101; we recall here that Caltech101 is a database of 101 image categories, such as images of airplanes or

93 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

images of brains. We will consider three different levels of outliers: 20%, 33% and 50%.

There are three important preprocessing steps that one may or may not chose to perform. The first such step is dimensionality reduction. In our case, we choose to use projection of the data matrix X˜ onto its first D = 50 principal components. This choice is justified by noting that, since the inliers and outliers are each at most 64, the data matrix is at most of rank 128. The choice d = 50 avoids situations where the entire data matrix is of low-rank to begin with, or cases where the outliers themselves span a low-dimensional subspace; e.g., this would be true if we were working with the original dimensionality of 2016.

These instances would be particularly unfavorable for methods such as SE-RPCA and `2,1-

RPCA. On the other hand, methods such as REAPER, DPCP-IRLS, DPCP-d, DPCP-r-d and DPCP-r work with the orthogonal complement of the subspace, and so the lower the codimension the more efﬁcient these methods are. We will shortly see how the methods behave for various projection dimensions.

Another pre-processing step is that of centering the data, i.e., forcing the entire dataset ˜ X to have zero mean; we will be writing C0 to denote that no centering takes place and C1 otherwise. Note here that we do not consider centering the inliers separately, as was done in [62]; this is unrealistic, since it requires knowledge of the ground truth, i.e., which image is an inlier and which is an outlier. Finally, one may normalize the columns of X˜ to have unit norm or not. We will denote these two possibilities as N1 and N0 respectively.

Various possibilities for the parameters of all algorithms were considered by experi- menting with the training set, and the ones that minimize the average area under the cor-

94 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

responding ROC curves were chosen, for the case of 33% outliers, for 50 independent experimental instances (which individual plays the role of the inlier subspace is a random event). The results for the two different choices of outliers and all four pre-processing

C N ,C N ,C N and C N , over 50 independent trials on the test set, are 0 − 0 0 − 1 1 − 0 1 − 1 reported in Figs. 3.9 and 3.10.

Our ﬁrst observation is that, on the average, all methods perform about the same, with

REAPER being the best method and RANSAC the worse. Evidently, normalizing the data

without centering them, leads to uniform performance degradation for all methods, for both

outlier scenarios. On the other hand, all methods seem to be robust to the remaining three

combinations of centering and normalization, with perhaps C N being the best for 1 − 0 this experiment. A second observation is that the ROC curves are better for all methods,

when the outliers come from Caltech101. This is indeed expected, since, in that case, not

only are the outliers chosen from a different dataset, but also their content is very different

from that of the inliers. On the other hand, when the outliers are face images themselves,

it is intuitively expected that the inlier/outlier separation problem becomes harder, simply

because the outliers are of similar nature with the inliers.

Notice also the interesting phenomenon of all methods behaving worse when the out-

liers are minimal. For example, when R = 20%, the area over the curves for all methods in

Figure 3.10(a) is bigger than when R = 33% (Figure 3.10(b)). In fact, REAPER, RANSAC and DPCP-IRLS are becoming better as the number of outliers increases from 20% to 50%

(Figs. 3.10(a)-3.10(c)). This phenomenon is partially explained by our theory: separating

95 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

inliers from outliers is easier when the outliers are uniformly distributed in the ambient

space, and this latter condition is more easily achieved when the outliers are large in num-

ber.

Finally, ﬁgures 3.11 and 3.12 show how the methods behave if we vary the projection

dimension to D = 15 or D = 150, without adjusting any parameters. Evidently, REAPER is once again the most robust method, while `2,1-RPCA is the least robust. Interestingly, when going from D = 50 to D = 150 for the case of face outliers, only SE-RPCA shows a slight improvement; the rest of the methods become slightly worse.

3.3 Learning a hyperplane arrangement via DPCP

3.3.1 Problem overview

In 3.2 we saw that when the data consist of a set of points uniformly distributed in a § deterministic sense on the unit sphere of a linear subspace , together with a set of outliers S uniformly distributed on the unit sphere of the ambient space, then the DPCP problem

(3.3) has the remarkable property that every global solution is a vector b orthogonal to the subspace . A natural question that arises is whether the DPCP problem has the same S property when the data comes from a subspace arrangement, i.e., whether every global solution of the DPCP problem is orthogonal to one of the subspaces of the arrangement. If this were to be true, then DPCP could prove a useful tool for clustering subspaces of high relative dimension, in particular for learning arrangements of hyperplanes.

96 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Hyperplane arrangements arise in several applications, with important examples being

projective motion segmentation [108,113], 3D point cloud analysis [81] and hybrid system

identiﬁcation [2]. In such applications the dataset X consists of N points of RD, such

D Ni that N points of X , say X R × , lie close to a hyperplane = x : b>x =0 , i i ∈ Hi i where bi is the normal vector to the hyperplane. It is then of interest to learn the underlying hyperplane arrangement, i.e., learn a set of n normal vectors b1,..., bn, and cluster the

data points X according to their hyperplane membership. Clearly, this is a special case of

the subspace clustering problem, which we will be referring to as hyperplane clustering.

The purpose of this section is to establish both theoretically and experimentally that

DPCP a useful new tool for clustering hyperplanes.11 Indeed, a large part of this section

is devoted to understanding under what conditions is every solution of the DPCP problem

the normal vector to one of the hyperplanes associated to the data. As we will see, the

conditions entail the hyperplanes to be sufﬁciently separated or one of the hyperplanes to

be sufﬁciently dominant. The rest of the section explores hyperplane clustering algorithms

using synthetic as well as real 3D data.

3.3.2 Data model

The data structure of interest in this section is an arrangement = n RD of A i=1 Hi ⊂ D D S D 1 n hyperplanes of R , given by = x R : x>b =0 , i [n], where b S − Hi ∈ i ∈ i ∈ is the normal vector to hyperplane . We assume that we are given a collection of N Hi 11The general case is left to future investigations.

97 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

D N D 1 data points X = [x ,..., x ] R × that are in general position in S − . By 1 N ∈ A∩ general position we mean two things. First, we mean that there are no relations among the

points other than the ones induced by their membership to the hyperplanes; in particular,

every (D 1) points of X are linearly independent. Second, we mean that the points − X uniquely deﬁne the arrangement , in the sense that is the only arrangement of A A n hyperplanes that contains X . We assume that for every i [n], precisely N points ∈ i of X , denoted by X = [x(i),..., x(i)], belong to , with n N = N. With that i 1 Ni Hi i=1 i P notation, X = [X 1,..., X n]Γ, where Γ is an unknown permutation matrix, indicating

that the hyperplane membership of the points is unknown. Finally, we assume an ordering

N N N , and we refer to as the dominant hyperplane. 1 ≥ 2 ≥···≥ n H1

3.3.3 Theoretical analysis of the continuous problem

Similarly to the case of data from a single subspace corrupted with outliers ( 3.2.2.1), § certain important insights regarding the DPCP problem (3.3) with respect to hyperplane

clustering can be gained by examining an associated continuous problem. As it turns out,

this problem has a fascinating geometric interpretation, that is interesting in its own right,

and this section is devoted to its study. In 3.3.3.1 we derive the continuous problem, § discuss its geometric interpretation and give a basic property of its global minimizers. In

3.3.3.2 we completely characterize the global minimizers of the continuous problem for § the special cases of i) two hyperplanes and ii) orthogonal hyperplanes, while in 3.3.3.3 § we completely characterize the global minimizers of an arrangement of three equiangular

98 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

hyperplanes. Finally, in 3.3.3.4 we give optimality conditions for an arbitrary hyperplane § arrangement.

3.3.3.1 Derivation, interpretation and basic properties of the continuous problem

To see what is that the continuous problem associated with DPCP for the case of hyperplane

D 1 D 1 arrangements, let ˆ = S − , and note ﬁrst that for any b S − we have Hi Hi ∩ ∈

Ni 1 (i) b>x b>x (3.146) j dµ ˆi , Ni ' x ˆi H j=1 Z H X 1 where the LHS of (3.146) is precisely X >b and can be viewed as a discretization Ni i 1

via the point set X of the integral on the RHS of (3.146), with denoting the uniform i µ ˆi H measure on ˆ .12 Letting θ be the principal angle between b and b , for any x we Hi i i ∈ Hi have

ˆ > b>x = b>π i (x)=(π i (b))> x = hi,>bx = sin(θi) hi,bx. (3.147) H H

Hence,

b>x hˆ > x (3.148) dµ î = i,b dµ î sin(θi) x î H x î H Z ∈H Z ∈H

= x1 dµSD−2 sin(θi) (3.149) x SD−2 | | Z ∈

= c sin(θi), (3.150)

D 1 where c is the average height of the unit hemisphere of R − (in the notation of equation

(3.24) we have c = cD 1). As a consequence, we can view the objective function of (3.3), − 12See 3.2.2.1 for a detailed measure-theoretic discussion of 3.146 in the context of data from a single subspace§ corrupted with outliers. Similar arguments apply here but are omitted for the sake of brevity.

99 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

which is given by

n n Ni 1 (i) X >b = X >b = N b>x , (3.151) 1 i 1 i N j i=1 i=1 i j=1 ! X X X

as a discretization via the point set X of the function

n n (3.150) b b>x (3.152) ( ) := Ni dµ ˆi = Ni c sin(θi). J x ˆi H i=1 Z ∈H i=1 X X

In that sense, the continuous counterpart of problem (3.3) is precisely

D 1 min (b)= N1 c sin(θ1)+ + Nn c sin(θn), s.t. b S − , (3.153) b J ··· ∈ where now the integer N is interpreted as a positive weight assigned to hyperplane , i Hi penalizing the distance of b from bi.

Remark 3.16. Geometrically, a solution b∗ to problem (3.153) can be interpreted as a weighted median of the lines spanned by b1,..., bn. Medians in Riemmannian manifolds, and in particular in the Grassmannian manifold, are an active subject of research [24,37].

However, we are not aware of any work in the literature that deﬁnes a median by means of

(3.153), nor any work that studies (3.153).

The advantage of working with (3.153) instead of (3.3), is that the solution set of the continuous problem (3.153) depends solely on the weights N N N assigned 1 ≥ 2 ≥···≥ n to the hyperplane arrangement, as well as on the geometry of the arrangement, captured by the principal angles φij between bi and bj. In contrast, the solutions of the discrete problem (3.3) may also depend on the distribution of the points X . From that perspective, understanding when problem (3.153) has a unique solution that coincides with the normal

100 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

b to the dominant hyperplane , is essential for understanding the potential of (3.3) ± 1 H1 for hyperplane clustering. Towards that end, we provide a series of results pertaining to the

continuous problem (3.153).

To begin with, we note that the objective function of (3.153) is everywhere differen-

tiable except at the points b ,..., b , where its partial derivatives do not exist. For any ± 1 ± n D 1 b S − distinct from b , the gradient at b is given by ∈ ± i n bi>b b = 1 bi. (3.154) ∇ J − 2 i=1 1 (b>b)2 X − i

Now let b∗ be a global solution of (3.153) and suppose that b∗ = b , i [n]. Then b∗ 6 ± i ∀ ∈ must satisfy the ﬁrst order optimality condition

b b∗ + λ∗ b∗ = 0, (3.155) ∇ J| where λ∗ is a Lagrange multiplier. Equivalently, we have

n 1 2 − 2 N b>b∗ 1 b>b∗ b + λ∗ b∗ = 0, (3.156) − i i − i i i=1 X which implies that

n 1 n 1 2 2 − 2 2 − 2 N b>b∗ 1 b>b∗ b∗ = N b>b∗ 1 b>b∗ b , (3.157) i i − i i i − i i i=1 i=1 X X from which the next lemma follows.

Lemma 3.17. Let b∗ be a global solution of (3.153). Then b∗ Span(b ,..., b ). ∈ 1 n

Proof. If b∗ is equal to some b , then the statement of the lemma is certainly true. If ± i b∗ = b , i [n], then b∗ satisﬁes (3.157), from which again the statement is true. 6 ± i ∀ ∈ 101 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

3.3.3.2 The cases of i) two hyperplanes and ii) orthogonal hyperplanes

In this section we characterize the global minimizers of the continuous problem (3.153) for two special cases of hyperplane arrangements. The ﬁrst conﬁguration that we examine is that of two hyperplanes. As it turns out in that case, the weighted geometric median of the two lines spanned by the normals to the hyperplanes, always corresponds to one of the normals, as the next theorem shows.

Theorem 3.18. Let b , b be an arrangement of two hyperplanes in RD, with weights N 1 2 1 ≥

N2. Then the set B∗ of global minimizers of (3.153) satisﬁes:

1. If N = N , then B∗ = b , b . 1 2 {± 1 ± 2}

2. If N >N , then B∗ = b . 1 2 {± 1}

Proof. By Lemma 3.17 any global solution must lie in the plane Span(b1, b2), and so our problem becomes planar, i.e., we may as well assume that the hyperplane arrangement b , b is a line arrangement of R2. Note that b , b S1 partition S1 in two arcs, and 1 2 1 2 ∈ among these, only one arc has length φ strictly less than π; we denote this arc by a. Next, recall that the continuous objective function for two hyperplanes can be written as

1 1 2 2 2 2 1 (b)= N 1 (b>b) + N 1 (b>b) , b S . (3.158) J 1 − 1 2 − 2 ∈ Let b∗ be a global solution, and suppose that b∗ a. If b∗ a, then we can replace b , b 6∈ − ∈ 1 2 by b , b , an operation that does not change neither the arrangement nor the objective. − 1 − 2

After this replacement, we have that b∗ a. Finally suppose that neither b∗ nor b∗ are ∈ − 102 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

inside a. Then replacing either b with b or b with b , leads to b∗ a. Consequently, 1 − 1 2 − 2 ∈

without loss of generality we may assume that b∗ lies in a. Moreover, subject to a rotation

and perhaps exchanging b1 with b2, we can assume that b1 is aligned with the positive x-axis and that the angle φ between b1 and b2, measured counter-clockwise, lies in (0,π).

Then b∗ is a global solution to

1 1 2 2 2 2 1 (b)= N 1 (b>b) + N 1 (b>b) , b S a. (3.159) J 1 − 1 2 − 2 ∈ ∩ Now, for any vector b S1 a, let θ ,θ = φ θ be the angle between b and b , b ∈ ∩ 1 2 − 1 1 2 respectively. Then our objective can be written as

(b)= ˜(θ )= N sin(θ )+ N sin(φ θ ), θ [0,φ]. (3.160) J J 1 1 1 2 − 1 1 ∈

Taking ﬁrst and second derivatives, we have

∂ ˜ J = N1 cos(θ1) N2 cos(φ θ1) (3.161) ∂θ1 − − ∂2 ˜ J2 = N1 sin(θ1) N2 sin(φ θ1). (3.162) ∂θ1 − − −

Since the second derivative is everywhere negative on [0,φ], ˜(θ ) is strictly concave on J 1

[0,φ] and so its minimum must be achieved at the boundary θ1 =0 or θ1 = φ. This means that either b∗ = b1 or b∗ = b2.

Notice that when N1 > N2, problem (3.153) recovers the normal b1 to the dominant

hyperplane, irrespectively of how separated the two hyperplanes are, since, according to

Proposition 3.18, the principal angle φ1,2 between b1, b2 does not play a role. The continu-

ous problem (3.153) is equally favorable in recovering normal vectors as global minimizers

103 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

in another extreme situation, where the arrangement consists of up to D perfectly separated

(orthogonal) hyperplanes, as asserted by the next theorem.

Theorem 3.19. Let b ,..., b be an orthogonal hyperplane arrangement, i.e, φ = π/2, i = 1 n ij ∀ 6 D j, of R , with n D, and weights N N N . Then the set B∗ of global mini- ≤ 1 ≥ 2 ≥···≥ n mizers of (3.153) can be characterized as follows:

1. If N = N , then B∗ = b ,..., b . 1 n {± 1 ± n}

2. If N = = N >N N , for some ` [n 1], then B∗ = b ,..., b . 1 ··· ` `+1 ≥··· n ∈ − {± 1 ± `}

Proof. For the sake of simplicity we assume n = 3, the general case follows in a similar

2 fashion. Letting x := b>b and y := 1 x , (3.157) can be written as i i i − i 2 2 2 p x1 x2 x3 x1 x2 x3 N + N + N b∗ = N b + N b + N b . (3.163) 1 y 2 y 3 y 1 y 1 2 y 2 3 y 3 1 2 3 1 2 3

Taking inner products of (3.163) with b1, b2, b3 we respectively obtain

2 2 2 x1 x2 x3 x1 x2 x3 N + N + N x = N + N (b>b )+ N (b>b ), (3.164) 1 y 2 y 3 y 1 1 y 2 y 1 2 3 y 1 3 1 2 3 1 2 3 2 2 2 x1 x2 x3 x1 x2 x3 N + N + N x = N (b>b )+ N + N (b>b ), (3.165) 1 y 2 y 3 y 2 1 y 2 1 2 y 3 y 2 3 1 2 3 1 2 3 2 2 2 x1 x2 x3 x1 x2 x3 N + N + N x = N (b>b )+ N (b>b )+ N . (3.166) 1 y 2 y 3 y 3 1 y 3 1 2 y 3 2 3 y 1 2 3 1 2 3

Since by Lemma 3.17 b∗ is a linear combination of b1, b2, b3, we can assume that D = 3.

Suppose that b∗ = b , i [n]. Now, suppose that x =0. Then we can not have either 6 ± i ∀ ∈ 3 x = 0 or x = 0, otherwise b∗ = b or b∗ = b respectively. Hence x ,x = 0. Then 1 2 2 1 1 2 6 equations (3.164)-(3.166) imply that

N1 N2 2 2 = and x1 + x2 =1. (3.167) y1 y2

104 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

2 2 Taking into consideration the relations xi + yi =1, we deduce that

N1 N2 y1 = , y2 = . (3.168) 2 2 2 2 N1 + N2 N1 + N2 p p Then

2 2 (b∗)= N y + N y + N y = N + N + N > (b )= N + N , (3.169) J 1 1 2 2 3 3 1 2 3 J 1 2 3 q which is a contradiction on the optimality of b∗. Similarly, none of the x1,x2 can be zero, i.e. x ,x ,x =0. Then equations (3.164)-(3.166) imply that 1 2 3 6

2 2 2 N1 N2 N3 x1 + x2 + x3 =1, = = , (3.170) y1 y2 y3 which give

Ni√2 yi = , i =1, 2, 3. (3.171) 2 2 2 N1 + N2 + N3

2 p2 2 But then (b∗) = 2(N + N + N ) > (b ) = N + N . This contradiction shows J 1 2 3 J 1 2 3 p that our hypothesis b∗ = b , i [n] is not valid, i.e., B∗ b , b , b . The rest 6 ± i ∀ ∈ ⊂ {± 1 ± 2 ± 3} of the theorem follows by comparing the values (b ), i [3]. J i ∈

3.3.3.3 The case of three equiangular hyperplanes

Theorems 3.18 and 3.19 were not hard to prove, since for two hyperplanes the objective

function is strictly concave, while for orthogonal hyperplanes the objective function is sep-

arable. In contrast, the problem becomes considerably harder for n > 2 non-orthogonal

hyperplanes. Even when n =3, characterizing the global minimizers of (3.153) as a func-

tion of the geometry and the weights seems hard. Nevertheless, when the three hyperplanes

105 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

are equiangular and their weights are equal, the symmetry of the conﬁguration allows us to

fully characterize the geometric median as a function of the angle of the arrangement.

To begin with, without loss of generality, we can describe an equiangular arrangement

of three hyperplanes of RD, with an equiangular arrangement of three planes of R3, with

normals b1, b2, b3 given by

> b1 := µ 1+ α α α (3.172) > b2 := µ α 1+ α α (3.173) > b3 := µ α α 1+ α (3.174) 1 µ := (1 + α)2 +2α2 − 2 , (3.175) with α a positive real number that determines the angle φ (0,π/2] of the arrangement, ∈ given by

2α(1 + α)+ α2 2α +3α2 cos(φ) := = . (3.176) (1 + α)2 +2α2 1+2α +3α2

Since N1 = N2 = N3, so our objective function essentially becomes

1 1 1 2 2 2 2 2 2 2 (b)= 1 (b>b) + 1 (b>b) + 1 (b>b) , b S (3.177) J − 1 − 2 − 3 ∈ = sin(θ1) + sin(θ2) + sin(θ3), (3.178)

where θi is the principal angle of b from bi. The next lemma shows that any global minimizer b∗ must have equal principal angles from at least two of the b1, b2, b3.

106 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

3 Lemma 3.20. Let b1, b2, b3 be an arrangement of equiangular planes in R , with angle

φ and weights N1 = N2 = N3. Let b∗ be a global minimizer of (3.153) and let xi :=

2 b>b∗, y = 1 x i =1, 2, 3. Then either y = y or y = y or y = y . i i − i 1 2 1 3 2 3 p Proof. If b∗ is one of b , b , b , then the statement clearly holds, since if say b∗ = b , ± 1 ± 2 ± 3 1

then y = y = sin(φ). So suppose that b∗ = b , i [3]. Then x ,y must satisfy 2 3 6 ± i ∀ ∈ i i 2 2 equations (3.164)-(3.166), together with xi + yi = 1. Allowing for yi to take the value

zero, the xi,yi must satisfy

p := x y y y + x y [z x x ]+ x y [z x x ]=0, (3.179) 1 1 1 2 3 2 3 − 1 2 3 2 − 1 3 p := x y [z x x ]+ x y y y + x y [z x x ]=0, (3.180) 2 1 3 − 1 2 2 1 2 3 3 1 − 2 3 p := x y [z x x ]+ x y [z x x ]+ x y y y =0 (3.181) 3 1 2 − 1 3 2 1 − 2 3 3 1 2 3

q := x2 + y2 1, (3.182) 1 1 1 − q := x2 + y2 1, (3.183) 2 2 2 − q := x2 + y2 1, (3.184) 3 3 3 − where z := cos(φ). Viewing the above system of equations as polynomial equations in

the variables x1,x2,x3,y1,y2,y3,z, standard Groebner basis computations reveal that the

polynomial

g := (1 z)(y2 y2)(y2 y2)(y2 y2)(y + y + y ) (3.185) − 1 − 2 1 − 3 2 − 3 1 2 3

lies in the ideal generated by pi,qi, i = 1, 2, 3. In simple terms, this means that b∗ must

satisfy g(xi,yi,z = cos(φ))=0. However, the yi are by construction non-negative and can

107 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

not be all zero. Moreover, φ> 0 so 1 z =0. This implies that − 6

(y2 y2)(y2 y2)(y2 y2)=0, (3.186) 1 − 2 1 − 3 2 − 3

which in view of the non-negativity of the yi implies

(y y )(y y )(y y )=0. (3.187) 1 − 2 1 − 3 2 − 3

The next lemma says that a global minimizer of (b) is not far from the arrangement. J

3 Lemma 3.21. Let b1, b2, b3 be an arrangement of equiangular planes in R , with angle

φ and weights N1 = N2 = N3. Let Ci be the spherical cap with center bi and radius φ.

Then any global minimizer of (3.178) must lie (up to a sign) either on the boundary or the

interior of C C C . 1 ∩ 2 ∩ 3

Proof. First of all notice that b , b , b lie on the boundary of C C C . Let b∗ be 1 2 3 1 ∩ 2 ∩ 3

a global minimizer. If φ = π/2, we have already seen in Theorem 3.19 that b∗ has to

be one of the vertices b1, b2, b3 (up to a sign); so suppose that φ < π/2. Let θi∗ be the

principal angle of b∗ from bi. Then at least two of θ1∗,θ2∗,θ3∗ must be less or equal to φ; for if say θ1∗,θ2∗ > φ, then b3 would give a smaller objective than b∗. Hence, without loss of generality we may assume that θ∗,θ∗ φ. In addition, because of Lemma 3.20, we can 1 2 ≤ further assume without loss of generality that θ1∗ = θ2∗. Let ζ be the vector in the small arc

1 that joins 1 and b and has angle from b , b equal to θ∗. Since (b∗) (ζ), it must √3 3 1 2 1 J ≤J

be the case that the principal angle θ3∗ is less or equal to φ (because the angle of ζ from b3

108 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

is φ). We conclude that θ∗,θ∗,θ∗ φ. Consequently, there exist i = j such that up to a ≤ 1 2 3 ≤ 6

sign b∗ C C . Let us assume without loss of generality that b∗ C C , i.e., θ∗,θ∗ ∈ i ∩ j ∈ 1 ∩ 2 1 2

are the angles of b∗ from b1, b2 (notice that now it may no longer be the case that θ1∗ = θ2∗).

Notice that the boundaries of C1 and C2 intersect at two points: b3 and its reﬂection b˜3 with respect to the plane spanned by b , b . In fact, divides C C in two halves, H1,2 1 2 H1,2 1 ∩ 2 , ˜, with being the reﬂection of ˜ with respect to . Letting C˜ be the spherical cap Y Y Y Y H1,2 3

of radius φ around b˜3, we can write

C C =(C C C ) (C C C˜ ). (3.188) 1 ∩ 2 1 ∩ 2 ∩ 3 ∪ 1 ∩ 2 ∩ 3

If b∗ C C C we are done, so let us assume that b∗ C C C˜ . Let b˜∗ be the ∈ 1 ∩ 2 ∩ 3 ∈ 1 ∩ 2 ∩ 3

reﬂection of b∗ with respect to . This reﬂection preserves the angles from b and b . H1,2 1 2

˜∗ ˜ We will show that b has a smaller principal angle θ3∗ from b3 than b∗. In fact the spherical

˜∗ ˜ ˜ angle of b from b3 is θ3∗ itself, and this is precisely the angle of b∗ from b3. Denote by H3,3˜

˜ ¯∗ the plane spanned by b3 and b3, b the spherical projection of b∗ onto H3,3˜, γ the angle

between b¯∗ and b∗, α the angle between b¯∗ and b3, and α˜ the angle between b¯∗ and b˜3.

Then the spherical law of cosines gives

˜ cos(θ3∗) = cos(˜α)cos(γ), (3.189)

cos(θ3∗) = cos(α)cos(γ). (3.190)

Letting 2ψ be the angle between b3 and b˜3, we have that

α = ψ +(ψ α˜). (3.191) −

109 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

By hypothesis α<ψ˜ and so α>ψ. If 2ψ π/2, then α is an acute angle and cos(˜α) > ≤ cos(α). If 2ψ >π/2, then cos(˜α) cos(α) only when π (2ψ α˜) α˜ ψ π/2. ≤ − − ≤ ⇔ ≥ But by construction ψ π/2 and equality is achieved only when φ = π/2. Hence, we ≤ conclude that cos(˜α) > cos(α) , which implies that cos(θ˜ ) > cos(θ ) . This in turn | | 3 | 3 | means that (b˜∗) < (b∗), which is a contradiction. J J

The next lemma ensures a every global minimizer satisﬁes a certain symmetry property.

3 Lemma 3.22. Let b1, b2, b3 be an arrangement of equiangular planes in R , with angle

φ and weights N1 = N2 = N3. Let b∗ be a global minimizer of (3.153) and let xi :=

bi>b∗, i =1, 2, 3. Then either x1,x2,x3 are all non-negative or they are all non-positive.

Proof. By Lemma 3.21, we know that either b∗ C C C or b∗ C C C . In ∈ 1 ∩ 2 ∩ 3 − ∈ 1 ∩ 2 ∩ 3

the ﬁrst case, the angles of b∗ from b , b , b are less or equal to φ π/2. 1 2 3 ≤

Finally, we arrive at our main result about three equiangular hyperplanes.

Theorem 3.23. Let b , b , b be an equiangular hyperplane arrangement of RD, D 3, 1 2 3 ≥ with φ = φ = φ = φ (0,π/2] and weights N = N = N . Let B∗ be the set of 1,2 13 23 ∈ 1 2 3 global minimizers of (3.153). Then B∗ satisﬁes the following phase transition:

1. If φ> 60◦, then B∗ = b , b , b . {± 1 ± 2 ± 3}

1 2. If φ = 60◦, then B∗ = b , b , b , 1 . ± 1 ± 2 ± 3 ± √3 n o 1 3. If φ< 60◦, then B∗ = 1 . ± √3 n o

110 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Proof. Lemmas 3.20 and 3.22 show that b∗ is a global minimizer of problem

2 2 2 min 1 (b1>b) + 1 (b2>b) + 1 (b3>b) (3.192) b S2 − − − ∈ q q q if and only if it is a global minimizer of problem

2 2 2 min 1 (b1>b) + 1 (b2>b) + 1 (b3>b) , (3.193) b S2 − − − ∈ q q q

s.t. b>b = b>b, for some i = j [3]. (3.194) i j 6 ∈

So suppose without loss of generality that b∗ is a global minimizer of (3.194) corresponding to indices i =1,j =2. Then b∗ lives in the vector space

1 0     = Span , , (3.195) V1,2 1 0             0 1         which consists of all vectors that have equal angles from b1 and b2. Taking into considera-

2 tion that b∗ also lies in S , we have the parametrization

v 1   b∗ = . (3.196) √ 2 2 v  2v + w       w     The choice v = 0, corresponding to b∗ = e3 (the third standard basis vector), can be excluded, since b3 always results in a smaller objective: moving b from e3 to b3 while staying in the plane results in decreasing angles of b from b , b , b . Consequently, we V1,2 1 2 3 can assume v =1, and our problem becomes an unconstrained one, with objective

2[(2 + w2)(1+2α +3α2) (αw +2α + 1)2]1/2 + √2 a aw +1 (w)= − | − |. (3.197) J [(2 + w2)(1+2α +3α2)]1/2

111 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Now, it can be shown that:

The following quantity is always positive •

u := (2+ w2)(1+2α +3α2) (αw +2α + 1)2. (3.198) −

The choice w =1+1/α corresponds to b∗ = b , and that is precisely the only point • 3 where (w) is non-differentiable. J

1 The choice w =1 corresponds to b∗ = 1. • √3

The choice α =1/3 corresponds to φ = 60◦. •

(b )= 1 1 precisely for α =1/3. • J 3 J √3 Since for α = 0 the theorem has already been proved (orthogonal case), we will assume that α> 0. We proceed by showing that for α (0, 1/3) and for w =1+1/a, it is always ∈ 6 the case that (w) > (1+1/a). Expanding this last inequality, we obtain J J 1/2 2 (2 + w2)(1 + 2α + 3α2) (αw + 2α + 1)2 + √2 a aw + 1 2√1 + 4α + 6α2 − | − | > , 2 2 1/2 α α2 [(2 + w )(1 + 2α + 3α )] 1 + 2 + 3 (3.199) which can be written equivalently as

p < 4√2u1/2 α αw +1 (1+2α +3α2), where (3.200) 1 | − | p := 4(2+ w2)(1+4α +6α2) (1+2α +3α2) 4u + 2(α αw + 1)2 , (3.201) 1 − − and u has been deﬁned in (3.198). Viewing p1 as a polynomial in w, p1 has two real roots

given by

1 7α + α2 + 15α3 r(1) :=1+1/α > r(2) := − − . (3.202) p1 p1 α(7+22α + 15α2)

112 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Since the leading coefﬁcient of p1 is always a negative function of α (for α > 0), (3.200)

(2) (1) will always be true for w [rp ,rp ], in which interval p is strictly negative. Conse- 6∈ 1 1 1 (2) (1) quently, we must show that as long as α (0, 1/3), (3.200) is true for every w [rp ,rp ). ∈ ∈ 1 1

For such w, p1 is non-negative and by squaring (3.200), we must show that

p >0, w [r(2),r(1)), α (0, 1/3), (3.203) 2 ∀ ∈ p1 p1 ∀ ∈ p :=32u(α αw + 1)2(1+2α +3α2) p2. (3.204) 2 − − 1

Interestingly, p2 admits the following factorization

p = 4( 1 α + αw)2p , (3.205) 2 − − − 3 p := 7 18α 49α2 204α3 441α4 162α5 + 81α6 3 − − − − − − + (30α + 238α2 + 612α3 + 468α4 162α5 162α6)w − − +( 8 48α 111α2 12α3 + 270α4 + 324α5 + 81α6)w2 (3.206) − − − −

The discriminant of p3 is the following 10-degree polynomial in α:

∆(p ) = 32( 7 60α 226α2 312α3 + 782α4 + 5160α5 + 13500α6+ 3 − − − − + 21816α7 + 22761α8 + 14580α9 + 4374α10). (3.207)

By Descartes rule of signs, ∆(p3) has precisely one positive root. In fact this root is equal to

1/3. Since the leading coefﬁcient of ∆(p ) is positive, we must have that ∆(p ) < 0, α 3 3 ∀ ∈

(0, 1/3), and so for such α, p3 has no real roots, i.e. it will be either everywhere negative

or everywhere positive. Since p (α = 1/4,w = 1) = 80327/4096, we conclude that as 3 − long as α (0, 1/3), p is everywhere negative and as long as w =1+1/α, p is positive, ∈ 3 6 2 113 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

i.e. we are done. Moving on to the case α =1/3, we have

128 p (α =1/3,w)= (w 4)2(w 1)2, (3.208) 2 9 − −

which shows that for such α the only global minimizers are b and 1 1. In a similar ± 3 ± √3 fashion, we can proceed to show that (w) > 1 1 , for all w = 1 and all α J J √3 6 ∈ (1/3, ). However, the roots of the polynomials that arise are more complicated functions ∞ of α and establishing the inequality (w) > 1 1 analytically, seems intractable; this J J √3 can be done if one allows for numeric computation of polynomial roots.

3.3.3.4 Conditions of global optimality for an arbitrary hyperplane arrangement

Theorem 3.23 suggests that when the hyperplanes are sufﬁciently separated, then only nor-

mals can be global minimizers, otherwise the only global minimizer lies at the center of the arrangement. This is in striking similarity with the results regarding the Fermat point of a planar or even spherical triangle [37]. We note that when the symmetry of the arrangement in Theorem 3.23 is removed, either by not requiring the principal angles φij

to be equal or/and by not requiring the weights Ni to be equal, then our proof technique

no longer applies, and the problem seems even harder. However, under sufﬁciently strong

conditions (which are not too strong), b is the unique solution of (3.153). To see what ± 1 these conditions are, we need two lemmas.

D 1 Lemma 3.24. Let b1,..., bn be vectors of S − , with pairwise principal angles φij. Then

1/2 2 max N1 b1>b + + Nn bn>b Ni +2 NiNj cos(φij) . (3.209) b>b=1 ··· ≤ " i i=j # X X6

114 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Proof. Let b† be a maximizer of N b>b + + N b>b . Then b† must satisfy the ﬁrst 1 1 ··· n n order optimality condition, which is

λ†b† = Ni Sgn(bi>b†)bi, (3.210) i X where λ† is a Lagrange multiplier and Sgn(bi>b†) is the subdifferential of bi>b† . Then

λ†b† = N s†b + + N s† b , (3.211) 1 1 1 ··· n n n where s† = Sign(b>b†), if b>b† =0, and s† [ 1, 1] otherwise. Recalling that b† =1, i i i 6 i ∈ − 2 and taking equality of 2-norms on both sides of (3.212), we get

N1s1†b1 + + Nnsn† bn b† = ··· . (3.212) N1s1†b1 + + Nnsn† bn ··· 2

Now

> Ni bi>b† = Ni bi>b† = Nisi†bi>b† = b† Nisi†bi † †  †  i i:b bi i:b bi i:b bi X X6⊥ X6⊥ X6⊥  

= b† > N s†b + N s†b = b† > N s†b  i i i i i i i i i † † ! i:b bi i:b bi i X6⊥ X⊥ X   (3.212) = N1s1†b1 + + Nnsn† bn ··· 2 1/2 2

= si†Ni +2 NiNjsi†sj†bi>bj " i i=j # X X6 1/2 N 2 +2 N N cos(φ ) . (3.213) ≤ i i j ij " i i=j # X X6

115 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

D Lemma 3.25. Let b1,..., bn be a hyperplane arrangement of R with integer weights

D 1 N ,...,N assigned. For b S − , let θ be the principal angle between b and b . Then 1 n ∈ i i

2 min Ni sin(θi) Ni σmax [NiNj cos(φij)]i,j , (3.214) b SD−1 ≥ − ∈ r X X where σ [N N cos(φ )] denotes the maximal eigenvalue of the n n matrix, whose max i j ij i,j × (i, j) entry is N N cos(φ ) and 1 i, j n. i j ij ≤ ≤

Proof. For any vector ξ we have that ξ ξ . Let ψ [0, 180◦] be the angle between k k1 ≥ k k2 i ∈ b and bi. Then

1 2 N sin(θ )= N sin(ψ ) k·k ≥k·k N 2 sin2(ψ ) (3.215) i i i | i | ≥ i i X X qX = N 2 N 2 cos2(ψ ). (3.216) i − i i qX X 2 2 Hence Ni sin(θi) is minimized when Ni cos (ψi) is maximized. But P 2 2 2 Ni cos (ψi)= b> Ni bibi> b, (3.217) X X 2 2 and the maximum value of Ni cos (ψi) is equal to the maximal eigenvalue of the matrix P 2 > Ni bibi> = N b N b N b N b , (3.218) 1 1 ··· n n 1 1 ··· n n X which is the same as the maximal eigenvalue of the matrix

> N1b1 Nnbn N1b1 Nnbn =[NiNj cos(ψij)]i,j , (3.219) ··· ··· where ψ is the angle between b , b . Now, if A is a matrix and we denote by A the ij i j | | matrix that arises by taking absolute values of each element in the matrix A, then it is known that σ ( A ) σ (A). Hence the result follows by recalling that cos(ψ ) = max | | ≥ max | ij | cos(φij).

116 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

The next result gives sufﬁcient conditions under which there is a unique global mini-

mizer of the continuous problem, equal to the normal vector of the dominant hyperplane.

Theorem 3.26. Let b ,..., b be an arrangement of n 3 hyperplanes in RD, with pair- 1 n ≥ wise principal angles φ . Let N N N be positive integer weights assigned ij 1 ≥ 2 ≥···≥ n to the arrangement. Suppose that N1 is large enough, in the sense that

2 2 N1 > α + β , where (3.220) p

α := N sin(φ ) N 2 σ [N N cos(φ )] 0, (3.221) i 1,i − i − max i j ij i,j>1 ≥ i>1 s i>1 X X 2 β := Ni +2 NiNj cos(φij), (3.222) i>1 i=j, i,j>1 sX 6 X with σ [N N cos(φ )] denoting the maximal eigenvalue of the (n 1) (n 1) max i j ij i,j>1 − × − matrix, whose (i 1,j 1) entry is N N cos(φ ) and 1 < i, j n. Then any global − − i j ij ≤ minimizer b∗ of problem (3.153) must satisfy b∗ = b , for some i [n]. If in addition, ± i ∈

γ := min Ni sin(φi0,i) Ni sin(φ1,i) > 0, (3.223) i0=1 − 6 i=i0 i>1 X6 X then problem (3.153) admits a unique up to sign global minimizer b∗ = b . ± 1

Proof. Let b∗ be a global solution of (3.153). Suppose for the sake of a contradiction that b∗ , i [n], i.e., b∗ = b , i [n]. Consequently, is differentiable at b∗ and so 6⊥ Hi ∀ ∈ 6 ± i ∀ ∈ J b∗ must satisfy (3.156), which we repeat here for convenience:

n 1 2 − 2 N b>b∗ 1 b>b∗ b + λ∗ b∗ = 0. (3.224) − i i − i i i=1 X 117 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Projecting (3.224) orthogonally onto the hyperplane ∗ deﬁned by b∗ we get H n 1 2 − 2 0 Ni bi>b∗ 1 bi>b∗ π ∗ (bi)= . (3.225) − − H i=1 X Since b∗ = bi, i [n], it will be the case that hi := π ∗ (bi) = 0, i [n]. Since 6 ∀ ∈ H 6 ∀ ∈ 1 2 2 π ∗ (bi) = 1 bi>b∗ > 0, (3.226) k H k2 − equation (3.225) can be written as

n ˆ 0 Ni bi>b∗ hi = , (3.227) i=1 X which in turn gives

ˆ N1 b1>b∗ Ni bi>b∗ hi (3.228) ≤ i>1 2 X

N b>b∗ (3.229) ≤ i i i>1 X

max Ni bi>b (3.230) ≤ b>b=1 i>1 X Lem.3.24 β. (3.231) ≤

Since by hypothesis N1 > β, we can deﬁne an angle θ1† by

β cos(θ1†) := , (3.232) N1 and so (3.231) says that θ can not drop below θ†. Hence (b∗) can be bounded from 1 1 J below as follows:

(b∗)= N1 sin(θ1∗)+ Ni sin(θi∗) N1 sin(θ1†)+ min Ni sin(θi) (3.233) J ≥ b>b=1 i>1 i>1 X X Lem.3.25 2 N sin(θ†)+ N σ [N N cos(φ )] . (3.234) ≥ 1 1 i − max i j ij i,j>1 s i>1 X

118 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

By the optimality of b∗, we must also have (b ) (b∗), which in view of (3.234) gives J 1 ≥J

2 N sin(φ ) N sin(θ†)+ N σ [N N cos(φ )] . (3.235) i 1i ≥ 1 1 i − max i j ij i,j>1 i>1 s i>1 X X Now, a little algebra reveals that this latter inequality is precisely the negation of hypothesis

N > α2 + β2. This shows that b∗ has to be b , for some i [n]. For the last 1 ± i ∈ p statement of the theorem, notice that condition γ > 0 is equivalent to saying that (b ) < J 1 (b ), i> 1. J i ∀

Let us provide some intuition about the meaning of the quantities α, β and γ in Theorem

3.26. To begin with, the first term in α is precisely equal to (b ), while the second J 1 term in α is a lower bound on the objective function N sin(θ )+ + N sin(θ ), if one 2 2 ··· n n β discards hyperplane 1. Moving on, the quantity admits a nice geometric interpretation: H N1 1 β cos− is a lower bound on how small the principal angle of a critical point b† from b N1 1 can be, if b† = b . Interestingly, the larger N is, the larger this minimum angle is, which 6 ± 1 1 shows that critical hyperplanes † (i.e., hyperplanes defined by critical points b†) that are H distinct from , must be sufficiently separated from . Finally, the second term in γ is H1 H1 (b ), while the first term is the smallest objective value that corresponds to b = b ,i> 1, J 1 i and so (3.223) simply guarantees that (b ) < (b ), i> 1. J 1 J i ∀

2 2 Next, notice that condition N1 > α + β of Theorem 3.26 is easier to satisfy when p is close to the rest of the hyperplanes (which leads to small α), while the rest of the H1 hyperplanes are sufﬁciently separated (which leads to small α and small β). Regardless,

119 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

one can show that

√2 N α2 + β2, (3.236) i ≥ i>1 X p and so if

N1 > √2 Ni, (3.237) i>1 X then any global minimizer of (3.153) has to be one of the normals, irrespectively of the φij.

Finally, condition (3.223) is consistent with condition (3.220) in that it requires to be H1 close to ,i> 1 and , to be sufficiently separated for i, j > 1. Once again, (3.223) Hi Hi Hj can always be satisfied irrespectively of the φij, by choosing N1 sufficiently large, since only the positive term in the definition of γ depends on N1.

3.3.4 Theoretical analysis of the discrete problem

We now turn our attention to the discrete problem of hyperplane clustering via DPCP, i.e., to

problems (3.3) and (3.4), for the case where X =[X 1,..., X n]Γ, with X i being Ni points

in , as described in section 3.2.1.1. As we did in the case of single subspace learning Hi D 1 with outliers ( 3.2.3), for any i [n] and b S − , we write the quantity X >b as § ∈ ∈ || i ||1

Ni Ni (i) (i) (i) X >b b>x b> b>x x b>x (3.238) i 1 = j = Sign j j = Ni i,b, j=1 j=1 X X

where

Ni 1 (i) (i) x := Sign b>x x (3.239) i,b N j j i j=1 X

120 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

X is the average point of i with respect to the orthogonal projection hi,b := π i (b) of b H onto . Then by similar arguments as the ones that established Lemma 3.8 and inequality Hi

(3.83), we have that xi,b is a discrete approximation to an integral that evaluates to chi,b, which leads us to deﬁne b

i := max xi,b c hi,b (3.240) b SD−1 − 2 ∈ b D 1 as the maximum approximation error as b varies on S − . In turn, i is bounded above as

i √5SD 1(X i), (3.241) ≤ −

where SD 1(X i) is the spherical cap discrepancy of X i; see the deﬁning equation (3.65) − and the discussion surrounding (3.83), which adapts it for points that lie in a proper linear subspace. As a consequence, the more uniformly distributed the points X i are, the smaller

SD 1(X i) is (by definition), and hence the same is true for the uniformity parameter i. − Before stating our main result regarding the properties of problem (3.3) when the data lie in a union of hyperplanes, we need a definition analogous to Definition 3.10.

D 1 Deﬁnition 3.27. For a set Y =[y ,..., y ] − and integer K L, deﬁne Y to 1 L ⊂S ≤ R ,K be the maximum circumradius among the circumradii of all polytopes of the form

K α y : α [ 1, 1] , (3.242) ji ji ji ∈ − ( i=1 ) X where j1,...,jK are distinct integers in [L]. Using this notation, we now deﬁne

:= max i,Ki . (3.243) R K1+ +Kn=D 1 RX 0 ···Ki D 2− i=1 ≤ ≤ − X 121 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

We note that it is always the case that i,Ki Ki, with this upper bound achieved when RX ≤ contains K colinear points. Combining this fact with the constraint K = D 1 Xi i i i − in (3.243), we get that D 1, and the more uniformly distributedP are the points X R ≤ − inside the hyperplanes, the smaller is (even though does not go to zero). R R The theorem that follows is the discrete counterpart of Theorem 3.26, and it says that if 1) the dominant hyperplane is sufﬁciently dominant, 2) the remaining hyperplanes are sufﬁciently separated, and 3) the points are uniformly distributed in their respective hyperplanes, then there is a unique up to sign global minimizer of problem (3.3), the normal vector to the dominant hyperplane.

Theorem 3.28. Let b∗ be a solution of (3.3) with X = [X 1,..., X n]Γ, and suppose that c> √21. If

2 2 N1 > α¯ + β¯ , where (3.244) q 1 α¯ := α + c− 1N1 +2 iNi , and (3.245) i>1 ! X 1 β¯ := β + c− + N , (3.246) R i i X

with α, β as in Theorem 3.26, then b∗ = b for some i [n]. Furthermore, b∗ = b , if ± i ∈ ± 1

1 γ¯ := γ c− N + N +2 N > 0. (3.247) − 1 1 2 2 i i i>2 ! X (1) Proof. Let us ﬁrst derive an upper bound θmax on how large θ1∗ can be. Towards that end,

we derive a lower bound on the objective function (b) in terms of θ : For any vector J 1

122 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

D 1 b S − we can write ∈

(b)= X >b = X >b = N b>x b (3.248) J 1 i 1 i i, X X = cN sin( θ )+ N b>η , η (3.249) i i i i,b i,b 2 ≤ i X X c N sin(θ ) N (3.250) ≥ i i − i i X X = cN sin(θ )+ c N sin(θ ) N (3.251) 1 1 i i − i i i>1 X X

cN1 sin(θ1)+ c min Ni sin(θi) iNi (3.252) ≥ b>b=1 − " i>1 # X X Lem.3.25 cN sin(θ )+ c N 2 σ [N N cos(φ )] N . (3.253) ≥ 1 1 i − max i j ij i,j>1 − i i s i>1 X X Next, we derive an upper bound on (b ): J 1

(b )= X >b = N b>x b (3.254) J 1 i 1 1 i 1 i, 1 i>1 i>1 X X

= cN sin(φ )+ N b>η , η (3.255) i 1i i 1 i,b1 i,b1 2 ≤ i i>1 i>1 X X

c N sin(φ )+ N . (3.256) ≤ i 1i i i i>1 i>1 X X Since any vector b for which the corresponding lower bound (3.253) on (b) is strictly J larger than the upper bound (3.256) on (b ), can not be a global minimizer (because it J 1 (1) gives a larger objective than b1), θ1∗ must be bounded above by θmax, where the latter is deﬁned, in view of (3.244), by

1 α + c− 1N1 +2 iNi sin θ(1) := i>1 , (3.257) max N 1 P where α is as in Theorem 3.28 . Now let b∗ be a global minimizer, and suppose for the sake

(1) of contradiction that b∗ , i [n]. We will show that there exists a lower bound θ 6⊥ Hi ∀ ∈ min 123 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

(1) (1) on θ1, such that θmin >θmax, which is of course a contradiction. Towards that end, the ﬁrst

order optimality condition for b∗ can be written as

0 X Sgn(X >b∗)+ λb∗, (3.258) ∈

where λ is a Lagrange multiplier and Sgn(α) = Sign(α) if α =0 and Sgn(0) = [ 1, 1], is 6 − the subdifferential of the function . Since the points X are general, any hyperplane of |·| H RD spanned by D 1 points of X such that at most D 2 points come from X , i [n], − − i ∀ ∈

does not contain any of the remaining points of X . Consequently, by Proposition 3.11 b∗

X will be orthogonal to precisely D 1 points ξ1,..., ξD 1 , from which at most − − ⊂ K D 2 lie in . Thus, we can write relation (3.258) as i ≤ − Hi

D 1 n − 0 αjξj + Ni xi,b∗ + λb∗ = , (3.259) j=1 i=1 X X for real numbers 1 α 1, j [D 1]. Using the deﬁnition of , we can write − ≤ j ≤ ∀ ∈ − i

x b∗ = c hˆ b∗ + η ∗ , i [n], (3.260) i, i, i,b ∀ ∈

with η ∗ . Note that since b∗ , i [n], we have hˆ b∗ = 0. Substituting i,b 2 ≤ i 6⊥ Hi ∀ ∈ i, 6

(3.260) in (3.259) we get

D 1 n n − ˆ 0 αjξj + c Ni hi,b∗ + Ni ηi,b∗ + λb∗ = , (3.261) j=1 i=1 i=1 X X X and projecting (3.261) onto the hyperplane b∗ with normal b∗, we obtain H

D 1 n n − ˆ ∗ 0 π b∗ αjξj + c Niπ b∗ hi,b + Ni π b∗ ηi,b∗ = . (3.262) H H H j=1 ! i=1 i=1 X X X

124 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

ˆ ∗ Let us analyze the term π b∗ hi,b . We have H π i (b∗) b∗ bi>b∗ bi ˆ ∗ π b∗ hi,b = π b∗ H = π b∗ − (3.263) H H H π i (b∗) 2 b∗ bi>b∗ bi ! k H k − 2 2 b∗ bi>b∗ bi b∗ bi>b∗ bi 1 cos (θi) = π b∗ − = − − b∗ (3.264) H sin(θ ) sin(θ ) − sin(θ ) i ! i i b>b∗ b>b∗ b∗ b i i i ˆ = − = bi>b∗ ζi, ζi = π b∗ (bi). (3.265) sin(θi) − H Using (3.265), (3.262) becomes

D 1 n n − ˆ 0 π b∗ αjξj Ni c bi>b∗ ζi + Ni π b∗ ηi,b∗ = . (3.266) H − H j=1 ! i=1 i=1 X X X Isolating the term that depends on i = 1 to the LHS and moving everything else to the

RHS, and taking norms, we get

n cN cos(θ ) cN cos(θ )+ 1 1 ≤ i i i>1 X K n

+ π b∗ αjξj + Ni π b∗ ηi,b∗ . (3.267) H H 2 j=1 ! i=1 X 2 X K Since ηi,b∗ i, we have that π b∗ ηi,b∗ i. Next, the quantity αjξj 2 ≤ H 2 ≤ j=1 P can be decomposed along the index i, based on the hyperplane membership of the ξj.

For instance, if ξ , then replace the term α ξ with α(1)ξ(1), where the superscript 1 ∈ H1 1 1 1 1 (1) denotes association to hyperplane . Repeating this for all ξ and after a possible · H1 j re-indexing, we have

D 1 n Ki − (i) (i) αjξj = αj ξj . (3.268) j=1 i=1 j=1 X X X Now, by Deﬁnition 3.27 we have that

Ki (i) (i) αj ξj i,Ki . (3.269) ≤R j=1 2 X

125 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

As a consequence, the upper bound (3.267) can be extended to

n cN cos(θ ) cN cos(θ )+ N + . (3.270) 1 1 ≤ i i i i R i>1 i X X Finally, Lemma 3.24 provides a bound

n N cos(θ ) β, (3.271) i i ≤ i>1 X where β is as in Theorem 3.26. In turn, this can be used to extend (3.270) to

1 β + c− ( + iNi) (1) cos(θ1) R =: cos θmin . (3.272) ≤ N1 P (1) ¯ Note that the angle θmin of (3.272) is well-deﬁned, since by hypothesis N1 > β, and that

(1) what (3.272) effectively says, is that θ1 never drops below θmin. It is then straightforward

2 ¯2 (1) (1) to check that hypothesis N1 > α¯ + β implies θmin > θmax, which is a contradiction. p In other words, b∗ must be equal up to sign to one of the bi, which proves the ﬁrst part of the theorem. The second part follows from noting that condition γ¯ > 0 guarantees that

(b ) < min (b ). J 1 i>1 J i

2 2 Notice the similarity of conditions N1 > α¯ + β¯ , γ¯ > 0 of Theorem 3.28 with

2 2 p conditions N1 > α + β ,γ > 0 of Theorem 3.26. In fact α¯ >α, β¯ >β, γ<γ¯ , which p implies that the conditions of Theorem 3.28 are strictly stronger than those of Theorem

3.26. This is no surprise since, as we have already remarked, the solution set of (3.3) depends not only on the geometry (φij) and the weights (Ni) of the arrangement, but also on the distribution of the data points (parameters and ). i R

We note that in contrast to condition (3.220) of Theorem 3.26, N1 now appears in both sides of condition (3.244) of Theorem 3.28 . Nevertheless, under the assumption

126 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

c > √21, (3.244) is equivalent to the positivity of a quadratic polynomial in N1, whose leading coefficient is positive, and hence it can always be satisfied for sufficiently large N1.

Another interesting connection of Theorem 3.26 to Theorem 3.28 , is that the former can be seen as a limit version of the latter: dividing (3.244) and (3.247) by N1, letting

N ,...,N go to inﬁnity while keeping each ratio N /N ﬁxed, and recalling that 0 1 n i 1 i → as N and D 1, we recover the conditions of Theorem 3.26. i → ∞ R≤ − Next, we consider the linear programming recursion (3.4). At a conceptual level, the main difference between the linear programming recursion in (3.4) and the continuous and discrete problems (3.153) and (3.3), respectively, is that the behavior of (3.4) depends highly on the initialization nˆ 0. Intuitively, the closer nˆ 0 is to b1, the more likely the recursion will converge to b1, with this likelihood becoming larger for larger N1. The precise technical statement is as follows.

Theorem 3.29. Let nˆ be the sequence generated by the linear programming recursion { k} D 1 (3.4) by means of the simplex method, where nˆ S − is an initial estimate for b , with 0 ∈ 1 principal angle from b equal to θ . Suppose that c> √5 , and let φ(1) = min φ . i i,0 1 min i>1 { 1i}

If θ1,0 is small enough, i.e.,

(1) 1 2 1 sin(θ ) < min sin φ 2 , 1 (c ) 2c− , (3.273) 1,0 min − 1 − − 1 − 1 n p o

and N1 is large enough in the sense that

ν + ν2 +4ρτ N1 > max µ, , where (3.274) ( p2τ )

127 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

1 1 i>1 Ni sin(θi,0)+ c− jNj + i=1,j Ni 2c− i sin(φij) µ := max 6 − , (3.275) j=1 sin(φ ) sin(θ ) 2c 1 6 (P 1j − P1,0 − − 1 )

1 1 1 ν :=2c− 1 β + c− + c− iNi R ! Xi>1 1 1 + 2 sin(θ1,0) + 2c− 1 α + 2c− iNi , (3.276) i>1 ! 2 X 2 1 1 1 ρ := α + 2c− iNi + β + c− + c− iNi , (3.277) ! R ! Xi>1 Xi>1 2 1 1 2 τ := cos (θ ) 4c− sin(θ ) 5(c− ) , (3.278) 1,0 − 1 1,0 − 1

with α, β as in Theorem 3.26, then n converges to b in a ﬁnite number of steps. { k} ± 1

Proof. First of all, it follows from the theory of the simplex method, that if nk+1 is obtained via the simplex method, then it will satisfy the conclusion of Lemma 3.13. Then Theorem

3.14 guarantees that n converges to a critical point of problem (3.3) in a ﬁnite number { k} of steps; denote that point by nk∗ . In other words, nk∗ will satisfy equation (3.224) and it will have unit ` norm. Now, if n ∗ = b for some j > 1, then 2 k ± j

(nˆ ) (b ), (3.279) J 0 ≥J j or equivalently

N nˆ >x n N b>x b . (3.280) i 0 i,ˆ 0 ≥ i j i, j i=j X X6 Substituting the concentration model

\ xi,nˆ 0 = cπ i (n0)+ ηi,0, ηi,0 i, (3.281) H 2 ≤

\ xi,bj = cπ i (bj)+ ηij, ηij i, (3.282) H 2 ≤

128 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

into (3.280), we get

N c sin(θ )+ N nˆ >η N c sin(φ )+ N b>η . (3.283) i i,0 i 0 i,0 ≥ i ij i j ij i=j X X X6 X Bounding the LHS of (3.283) from above and the RHS from below, we get

N c sin(θ )+ N N c sin(φ ) N . (3.284) i i,0 i i ≥ i ij − i i i=j X X X6 X But this very last relation is contradicted by hypothesis N > µ, i.e., none of the b for 1 ± j

j > 1 can be n ∗ . We will show that n ∗ has to be b . So suppose for the sake of a k k ± 1

contradiction that that n ∗ is not colinear with b , i.e., n ∗ , i [n]. Since n ∗ k 1 k 6⊥ Hi ∀ ∈ k satisﬁes (3.224), we can use part of the proof of Theorem 3.28 , according to which the

(1) (1) principal angle θ1, of nk∗ from b1 does not become less than θmin, where θmin is as in ∞ (3.272). Consequently, and using once again the concentration model, we obtain

Ni c sin(θi,0)+ iNi (nˆ 0) (nk∗ ) Ni c sin(θi, ) iNi ≥J ≥J ≥ ∞ − X X X X N c sin θ(1) + c N 2 σ [N N cos(φ )] N . ≥ 1 min i − max i j ij i,j>1 − i i s i>1 X X (3.285)

A little algebra reveals that the outermost inequality in (3.285) contradicts (3.274).

The quantities appearing in Theorem 3.29 are harder to interpret than those of Theorem

3.28 , but we can still give some intuition about their meaning. To begin with, the two

inequalities in (3.274) represent two distinct requirements that we enforced in our proof,

which when combined, guarantee that the limit point of (3.4) is b . ± 1

129 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

The ﬁrst requirement is that no b can be the limit point of (3.4) for i > 1; this is ± i captured by a linear inequality of the form

µN1 +(terms not depending on N1) > 0, (3.286)

which is satisfied either for N1 sufficiently large (if µ > 0) or for N1 sufficiently small (if

µ < 0). To avoid pathological situations where N1 is required to be negative or less than

D 1, it is natural to enforce µ to be positive. This is precisely achieved by inequality − sin(θ ) < sin φ(1) 2 in (3.273), which is a quite natural condition itself: the initial 1,0 min − 1 estimate nˆ 0 needs to be closer to b1 than any other normal bi for i > 1, and the more well-distributed the data X are inside (smaller ), the further nˆ can be from b . 1 H1 1 0 1 The second requirement that we employed in our proof is that the limit point of (3.4) is one of the b ,..., b ; this is captured by requiring that a certain quadratic polynomial ± 1 ± n

p(N ) := τN 2 νN ρ (3.287) 1 1 − 1 −

in N1 is positive. To avoid situations where the positivity of this polynomial contradicts the relation N1 > µ, it is important that we ask its leading coefﬁcient τ to be positive, so that the second requirement is satisﬁed for N1 large enough, and thus is compatible with

N1 > µ. As it turns out, τ is positive only if the data X 1 are sufﬁciently well distributed in , which is captured by condition c > √5 of Theorem 3.29. Even so, this latter H1 1

1 2 1 condition is not sufﬁcient; instead sin(θ ) < 1 (c ) 2c− is needed (as in 1,0 − − 1 − 1 p (3.273)), which is once again very natural: the more well-distributed the data X 1 are inside

(smaller ), the further nˆ from b can be. H1 1 0 1 130 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Next, notice that the conditions of Theorem 3.29 are not directly comparable to those

of Theorem 3.28 . Indeed, it may be the case that b is not a global minimizer of the ± 1

non-convex problem (3.3), yet the recursions (3.4) do converge to b1, simply because nˆ 0 is close to b1. In fact, by (3.273) nˆ 0 must be closer to b1 than bi to b1 for any i > 1, i.e.,

(1) φmin > θ1,0. Similarly to Theorems 3.26 and 3.28 , the more separated the hyperplanes

, are for i, j > 1, the easier it is to satisfy condition (3.274). In contrast, needs Hi Hj H1 to be sufﬁciently separated from for i > 1, since otherwise µ becomes large. This has Hi an intuitive explanation: the less separated is from the rest of the hyperplanes, the less H1

resolution the linear program (3.4) has in distinguishing b1 from bi,i> 1. To increase this

resolution, one needs to either select nˆ 0 very close to b1, or select N1 very large. The acute

reader may recall that the quantity α appearing in (3.277) becomes larger when becomes H1 separated from ,i> 1. Nevertheless, there are no inconsistency issues in controlling Hi the size of µ and ρ. This is because α is always bounded from above by i>1 Ni, i.e., α P does not increase arbitrarily as the φ1i increase. Another way to look at the consistency

of condition (3.274), is that its RHS does not depend on N1; hence one can always satisfy

(3.274) by selecting N1 large enough.

3.3.5 Algorithms

There are at least two ways in which DPCP can be used to learn a hyperplane arrangement;

either through a sequential (RANSAC-style) scheme, or through a parallel (K-Subspaces-

style) scheme. These two cases are described next.

131 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

3.3.5.1 Learning a hyperplane arrangement sequentially

Since at its core DPCP is a single subspace learning method, we may as well use it to learn

n hyperplanes in the same way that RANSAC [34] is used: learn one hyperplane from

the entire dataset, remove the points close to it, then learn a second hyperplane and so on.

The main weakness of this technique is well known, and consists of its sensitivity to the

thresholding parameter, which is necessary in order to remove points.

To alleviate the need of knowing a good threshold, we propose to replace the process of

removing points by a process of appropriately weighting the points. In particular, suppose

we solve the DPCP problem (3.3) on the entire dataset X and obtain a unit `2-norm vector b1. Now, instead of removing the points of X that are close to the hyperplane with normal vector b1 (which would require a threshold parameter), we weight each and every point xj

X of by its distance b1>xj from that hyperplane. Then to compute a second hyperplane

with normal b2 we apply DPCP on the weighted dataset b1>xj xj . To compute a third hyperplane, the weight of point xj is chosen as the smallest distance of xj from the already

computed two hyperplanes, i.e., DPCP is now applied to mini=1,2 bi>xj xj . After n hyperplanes have been computed, the clustering of the points is obtained based on their

distances to the n hypeprlanes. The resulting scheme is listed in Algorithm 3.5.

3.3.5.2 K-Hyperplanes via DPCP

Another way to do hyperplane clustering via DPCP, is to modify the classic K-Subspaces

[9,102,121] (see also Chapter 2) by computing the normal vector of each cluster by DPCP;

132 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Algorithm 3.5 Sequential Hyperplane Learning via DPCP D N 1: procedure SHL-DPCP(X =[x , x ,..., x ] R × ,n) 1 2 N ∈ 2: i 0; ← 3: w 1, j =1,...,N; j ← 4: for i =1: n do

5: Y [w x w x ]; ← 1 1 ··· N N

6: bi argminb RD Y >b , s.t. b>b =1 ; ← ∈ 1 7: w min b> x , j =1,...,N; j ← k=1,...,i k j

8: end for

9: C x X : i = argmin b>x , i =1,...,n; i ← j ∈ k=1,...,n k j n 10: return (b , C ) ; { i i }i=1 11: end procedure

see Algorithm 3.6. It is worth noting that since DPCP minimizes the `1-norm of the distances of the points to a hyperplane, consistency dictates that the stopping criterion for KH-

DPCP be governed by the sum over all points of the distance of each point to its assigned hyperplane (instead of the traditional sum of squares [9, 102]); in other words the global objective function minimized by KH-DPCP is the same as that of Median K-Flats [121].

Finally, it is important to note that K-Hyperplanes algorithms require in principle a good initialization. Interestingly, such initialization may be provided by the output of Alg. 3.5.

133 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

Algorithm 3.6 K-Hyperplanes via Dual Principal Component Pursuit

1: procedure KH-DPCP(X =[x1, x2,..., xN ] , b1,..., bn,ε,Tmax)

2: , ∆ , t 0; Jold ← ∞ J ← ∞ ← 3: while tε do max J 4: 0, t = t +1; Jnew ←

5: C x X : i = argmin b>x , i =1,...,n; i ← j ∈ k=1,...,n k j n 6: b>x ; new = i=1 xj Ci i j J ∈ P P 9 7: ∆ ( )/ ( + 10− ), ; J ← Jold −Jnew Jold Jold ←Jnew

8: b argmin C>b , s.t. b>b =1 , i =1,...,n; i ← b i 1 9: end while

n 10: return (b , C ) ; { i i }i=1 11: end procedure

3.3.6 Experimental evaluation

In this section we evaluate experimentally Algorithms 3.5 and 3.6 using both synthetic

( 3.3.6.1) and real data ( 3.3.6.2). § §

3.3.6.1 Synthetic data

Dataset design. We begin by evaluating experimentally Algorithm 3.5 using synthetic data. The coordinate dimension D of the data is inspired by major applications where hyperplane arrangements appear. In particular, we recall that

134 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

0.97 0.97 0.97

0.89 0.89 0.89 d/D d/D d/D

0.75 0.75 0.75

2 3 4 2 3 4 2 3 4 n n n

(a) RANSAC, α = 1 (b) RANSAC, α = 0.8 (c) RANSAC, α = 0.6

0.97 0.97 0.97

0.89 0.89 0.89 d/D d/D d/D

0.75 0.75 0.75

2 3 4 2 3 4 2 3 4 n n n

(d) REAPER, α = 1 (e) REAPER, α = 0.8 (f) REAPER, α = 0.6

0.97 0.97 0.97

0.89 0.89 0.89 d/D d/D d/D

0.75 0.75 0.75

2 3 4 2 3 4 2 3 4 n n n

(g) DPCP-r, α = 1 (h) DPCP-r, α = 0.8 (i) DPCP-r, α = 0.6

0.97 0.97 0.97

0.89 0.89 0.89 d/D d/D d/D

0.75 0.75 0.75

2 3 4 2 3 4 2 3 4 n n n

(j) DPCP-IRLS, α = 1 (k) DPCP-IRLS, α = 0.8 (l) DPCP-IRLS, α = 0.6

Figure 3.13: Clustering accuracy as a function of the number of hyperplanes n vs relative dimension d/D vs data balancing (α). White corresponds to 1, black to 0.

135 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

0.97 0.97 0.97

0.89 0.89 0.89 d/D d/D d/D

0.75 0.75 0.75

2 3 4 2 3 4 2 3 4 n n n

(a) RANSAC (b) REAPER (c) DPCP-r

0.97 0.97 0.97

0.89 0.89 0.89 d/D d/D d/D

0.75 0.75 0.75

2 3 4 2 3 4 2 3 4 n n n

(d) DPCP-r-d (e) DPCP-d (f) DPCP-IRLS

Figure 3.14: Clustering accuracy as a function of the number of hyperplanes n vs relative dimension d/D. Data balancing parameter is set to α =0.8.

In 3D point cloud analysis, the coordinate dimension is 3, but since the various planes • do not necessarily pass through a common origin, i.e., they are afﬁne, one may work

with homogeneous coordinates, which increases the coordinate dimension of the data

by 1 (see 4.5), i.e., D =4.

In two-view geometry one works with correspondences between pairs of 3D points. • Each such correspondence is treated as a point itself, equal to the tensor product of

the two 3D corresponding points, thus having coordinate dimension D =9.

136 CHAPTER 3. DUAL PRINCIPAL COMPONENT PURSUIT

0.5 0.5 0.5 ) ) ) M M M