DOCSLIB.ORG

Sparse Recovery Guarantees

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Sparse Recovery Guarantees Sparse Optimization Lecture: Sparse Recovery Guarantees Instructor: Wotao Yin July 2013 Note scriber: Zheng Sun online discussions on piazza.com Those who complete this lecture will know • how to read different recovery guarantees • some well-known conditions for exact and stable recovery such as Spark, coherence, RIP, NSP, etc. • indeed, we can trust `1-minimization for recovering sparse vectors 1 / 35 The basic question of sparse optimization is: Can I trust my model to return an intended sparse quantity? That is • does my model have a unique solution? (otherwise, different algorithms may return different answers) • is the solution exactly equal to the original sparse quantity? • if not (due to noise), is the solution a faithful approximate of it? • how much effort is needed to numerically solve the model? This lecture provides brief answers to the first three questions. 2 / 35 What this lecture does and does not cover It covers basic sparse vector recovery guarantees based on • spark • coherence • restricted isometry property (RIP) and null-space property (NSP) as well as both exact and robust recovery guarantees. It does not cover the recovery of matrices, subspaces, etc. Recovery guarantees are important parts of sparse optimization, but they are not the focus of this summer course. 3 / 35 Examples of guarantees Theorem (Donoho and Elad [2003], Gribonval and Nielsen [2003]) m×n For Ax = b where A 2 R has full rank, if x satisfies 1 −1 kxk0 ≤ 2 (1 + µ(A) ), then `1-minimization recovers this x. Theorem (Candes and Tao [2005]) If x is k-sparse and A satisfies the RIP-based condition δ2k + δ3k < 1, then x is the `1-minimizer. Theorem (Zhang [2008]) m×n IF A 2 R is a standard Gaussian matrix, then with probability at least 1 − exp(−c0(n − m)) `1-minimization is equivalent to `0-minimization for all x: c2 m kxk < 1 0 4 1 + log(n=m) where c0; c1 > 0 are constants independent of m and n. 4 / 35 How to read guarantees Some basic aspects that distinguish different types of guarantees: • Recoverability (exact) vs stability (inexact) • General A or special A? • Universal (all sparse vectors) or instance (certain sparse vector(s))? • General optimality? or specific to model / algorithm? • Required property of A: spark, RIP, coherence, NSP, dual certificate? • If randomness is involved, what is its role? • Condition/bound is tight or not? Absolute or in order of magnitude? 5 / 35 Spark First questions for finding the sparsest solution to Ax = b 1. Can sparsest solution be unique? Under what conditions? 2. Given a sparse x, how to verify whether it is actually the sparsest one? Definition (Donoho and Elad [2003]) The spark of a given matrix A is the smallest number of columns from A that are linearly dependent, written as spark(A). rank(A) is the largest number of columns from A that are linearly independent. In general, spark(A) 6= rank(A) + 1; except for many randomly generated matrices. Rank is easy to compute (due to the matroid structure), but spark needs a combinatorial search. 6 / 35 Spark Theorem (Gorodnitsky and Rao [1997]) If Ax = b has a solution x obeying kxk0 < spark(A)=2, then x is the sparsest solution. • Proof idea: if there is a solution y to Ax = b and x − y 6= 0, then A(x − y) = 0 and thus kxk0 + kyk0 ≥ kx − yk0 ≥ spark(A) or kyk0 ≥ spark(A) − kxk0 > spark(A)=2 > kxk0. • The result does not mean this x can be efficiently found numerically. m×n • For many random matrices A 2 R , the result means that if an algorithm returns x satisfying kxk0 < (m + 1)=2, the x is optimal with probability 1. • What to do when spark(A) is difficult to obtain? 7 / 35 General Recovery - Spark Rank is easy to compute, but spark needs a combinatorial search. However, for matrix with entries in general positions, spark(A) = rank(A) + 1. m×n For example, if matrix A 2 R (m < n) has entries Aij ∼ N (0; 1), then rank(A) = m = spark(A) − 1 with probability 1. m×n In general, 8 full rank matrix A 2 R (m < n), any m + 1 columns of A is linearly dependent, so spark(A) ≤ m + 1 = rank(A) + 1: 8 / 35 Coherence Definition (Mallat and Zhang [1993]) The (mutual) coherence of a given matrix A is the largest absolute normalized inner product between different columns from A. Suppose A = [a1 a2 ··· an]. The mutual coherence of A is given by ja>a j µ(A) = max k j : k;j;k6=j kakk2 · kaj k2 • It characterizes the dependence between columns of A • For unitary matrices, µ(A) = 0 • For matrices with more columns than rows, µ(A) > 0 • For recovery problems, we desire a small µ(A) as it is similar to unitary matrices. • For A = [Φ Ψ] where Φ and Ψ are n × n unitary, it holds n−1=2 ≤ µ(A) ≤ 1 • µ(A) = n−1=2 is achieved with [I F], [I Hadamard], etc. m×n −1=2 • if A 2 R where n > m, then µ(A) ≥ m . 9 / 35 Coherence Theorem (Donoho and Elad [2003]) spark(A) ≥ 1 + µ−1(A): Proof sketch: • A¯ A with columns normalized to unit 2-norm • p spark(A) • B a p × p minor of A¯ >A¯ P • jBiij = 1 and j6=i jBij j ≤ (p − 1)µ(A) −1 P • Suppose p < 1 + µ (A) ) 1 > (p − 1)µ(A) ) jBiij > j6=i jBij j; 8i • ) B 0 (Gershgorin circle theorem) ) spark(A) > p. Contradiction. 10 / 35 Coherence-base guarantee Corollary −1 If Ax = b has a solution x obeying kxk0 < (1 + µ (A))=2, then x is the unique sparsest solution. Compare with the previous Theorem If Ax = b has a solution x obeying kxk0 < spark(A)=2, then x is the sparsest solution. m×n −1 p For A 2 R where m < n, (1 + µ (A)) is at most 1 + m but spark can be 1 + m. spark is more useful. Assume Ax = b has a solution with kxk0 = k < spark(A)=2. It will be the unique `0 minimizer. Will it be the `1 minimizer as well? Not necessarily. −1 However, kxk0 < (1 + µ (A))=2 is a sufficient condition. 11 / 35 Coherence-based `0 = `1 Theorem (Donoho and Elad [2003], Gribonval and Nielsen [2003]) If A has normalized columns and Ax = b has a solution x satisfying 1 kxk < 1 + µ−1(A) ; 0 2 then this x is the unique minimizer with respect to both `0 and `1. Proof sketch: • Previously we know x is the unique `0 minimizer; let S := supp(x) • Suppose y is the `1 minimizer but not x; we study e := y − x • e must satisfy Ae = 0 and kek1 ≤ 2keS k1 > −1 • A Ae = 0 ) jej j ≤ (1 + µ(A)) µ(A)kek1; 8j • the last two points together contradict the assumption p Result bottom line: allow kxk0 up to O( m) for exact recovery 12 / 35 The null space of A P p1=p • Definition: kxkp := i jxij . • Lemma: Let 0 < p ≤ 1. If k(y − x)S¯kp > k(y − x)S kp then kxkp < kykp. Proof: Let e := y − x. p p p p kykp = kx + ekp = kxS + eS kp + keS¯kp = p p p p p p kxkp + (keS¯kp − keS kp) + (kxS + eS kp − kxS kp + keS kp). Last term is nonnegative for 0 < p ≤ 1. p p So, a sufficient condition is keS¯kp > keS kp. • If the condition holds for 0 < p ≤ 1, it also holds for q 2 (0; p]. • Definition (null space property NSP(k; γ)). Every nonzero e 2 N (A) satisfies keS k1 < γkeS¯k1 for all index sets S with jSj ≤ k. 13 / 35 The null space of A Theorem (Donoho and Huo [2001], Gribonval and Nielsen [2003]) o Basis pursuit minfkxk1 : Ax = bg uniquely recovers all k-sparse vectors x from measurements b = Axo if and only if A satisfies NSP(k; 1). Proof. Sufficiency. Pick any k-sparse vector xo. Let S := supp(xo) and S¯ = Sc. For any non-zero h 2 N (A), we have A(xo + h) = Axo = b and 0 0 kx + hk1 = kxS + hS k1 + khS¯k1 0 ≥ kxS k1 − khS k1 + khS¯k1 (1) 0 = kx k1 + (khS¯k1 − khS k1) : 0 0 o NSP(k; 1) of A guarantees kx + hk1 > kx k1, so x is the unique solution. o Necessity. The inequality (1) holds with equality if sign(xS ) = −sign(hS ) and hS has a sufficiently small scale. Therefore, basis pursuit to uniquely recovers all k-sparse vectors xo, NSP(k; 1) is also necessary. 14 / 35 The null space of A • Another sufficient condition (Zhang [2008]) for kxk1 < kyk1 is 2 1 ky − xk1 kxk0 < : 4 ky − xk2 Proof: p p p keS k1 ≤ jSjkeS k2 ≤ jSjkek2 = kxk0kek2: Then, the above sufficient condition ky − xk1 > 2k(y − x)S k1 is given the above inequality. 15 / 35 Null space Theorem (Zhang [2008]) Given x and b = Ax, min kxk1 s.t. Ax = b recovers x uniquely if 2 1 kek1 kxk0 < min 2 : e 2 N (A) n f0g : 4 kek2 Comments: p • We know 1 ≤ kek1=kek2 ≤ n for all e 6= 0. The ratio is small for sparse p vectors but we want it large, i.e., close to n and away from 1. • Fact: in most subspaces, the ratio is away from 1 • In particular, Kashin, Garvaev, and Gluskin showed that a randomly drawn (n − m)-dimensional subspace V satisfies p kek1 c1 m ≥ p ; e 2 V; e 6= 0 kek2 1 + log(n=m) with probability at least 1 − exp(−c0(n − m)), where c0; c1 > 0 are independent of m and n.
Load more

Recommended publications
  • Arxiv:1205.2081V6 [Math.OC]
    THE COMPUTATIONAL COMPLEXITY OF RIP, NSP, AND RELATED CONCEPTS IN COMPRESSED SENSING 1 The Computational Complexity of the Restricted Isometry Property, the Nullspace Property, and Related Concepts in Compressed Sensing Andreas M. Tillmann and Marc E. Pfetsch Abstract This paper deals with the computational complexity of conditions which guarantee that the NP-hard problem of finding the sparsest solution to an underdetermined linear system can be solved by efficient algorithms. In the literature, several such conditions have been introduced. The most well-known ones are the mutual coherence, the restricted isometry property (RIP), and the nullspace property (NSP). While evaluating the mutual coherence of a given matrix is easy, it has been suspected for some time that evaluating RIP and NSP is computationally intractable in general. We confirm these conjectures by showing that for a given matrix A and positive integer k, computing the best constants for which the RIP or NSP hold is, in general, NP-hard. These results are based on the fact that determining the spark of a matrix is NP-hard, which is also established in this paper. Furthermore, we also give several complexity statements about problems related to the above concepts. Index Terms Compressed Sensing, Computational Complexity, Sparse Recovery Conditions I. INTRODUCTION CENTRAL problem in compressed sensing (CS), see, e.g., [1], [2], [3], is the task of finding a sparsest solution to an A underdetermined linear system, i.e., min x s.t. Ax = b, (P ) k k0 0 m n for a given matrix A R × with m n, where x 0 denotes the ℓ0-quasi-norm, i.e., the number of nonzero entries in x.
    [Show full text]
  • High-Girth Matrices and Polarization
    High-Girth Matrices and Polarization Emmanuel Abbe Yuval Wigderson Prog. in Applied and Computational Math. and EE Dept. Department of Mathematics Princeton University Princeton University Email: [email protected] Email: [email protected] Abstract—The girth of a matrix is the least number of linearly requirement to be asymptotic, requiring rank(A) ∼ girth(A) 1 dependent columns, in contrast to the rank which is the largest when the number of columns in A tends to infinity. If F = F2, number of linearly independent columns. This paper considers the Gilbert-Varshamov bound provides a lower-bound on the the construction of high-girth matrices, whose probabilistic girth maximal girth (conjectured to be tight by some). Namely, is close to their rank. Random matrices can be used to show for a uniformly drawn matrix A with n columns, with high the existence of high-girth matrices. This paper uses a recursive probability, construction based on conditional ranks (inspired by polar codes) to obtain a deterministic and efficient construction of high- girth matrices for arbitrary relative ranks. Interestingly, the rank(A) = nH(girth(A)=n) + o(n); (3) construction is agnostic to the underlying field and applies to both finite and continuous fields with the same binary matrix. The where H is the binary entropy function. construction gives in particular the following: (i) over the binary For F = Fq, the bound in (1) is a restatement of the field, high-girth matrices are equivalent to capacity-achieving Singleton bound for linear codes and expressed in terms of the codes, and our construction turns out to match exactly the BEC co-dimension of the code.
    [Show full text]
  • The Road to Deterministic Matrices with the Restricted Isometry Property
    Air Force Institute of Technology AFIT Scholar Faculty Publications 2013 The Road to Deterministic Matrices with the Restricted Isometry Property Afonso S. Bandeira Matthew C. Fickus Air Force Institute of Technology Dustin G. Mixon Percy Wong Follow this and additional works at: https://scholar.afit.edu/facpub Part of the Mathematics Commons Recommended Citation Bandeira, A. S., Fickus, M., Mixon, D. G., & Wong, P. (2013). The Road to Deterministic Matrices with the Restricted Isometry Property. Journal of Fourier Analysis and Applications, 19(6), 1123–1149. https://doi.org/10.1007/s00041-013-9293-2 This Article is brought to you for free and open access by AFIT Scholar. It has been accepted for inclusion in Faculty Publications by an authorized administrator of AFIT Scholar. For more information, please contact [email protected]. THE ROAD TO DETERMINISTIC MATRICES WITH THE RESTRICTED ISOMETRY PROPERTY AFONSO S. BANDEIRA, MATTHEW FICKUS, DUSTIN G. MIXON, AND PERCY WONG Abstract. The restricted isometry property (RIP) is a well-known matrix condition that provides state-of-the-art reconstruction guarantees for compressed sensing. While random matrices are known to satisfy this property with high probability, deterministic constructions have found less success. In this paper, we consider various techniques for demonstrating RIP deterministically, some popular and some novel, and we evaluate their performance. In evaluating some techniques, we apply random matrix theory and inadvertently find a simple alternative proof that certain random matrices are RIP. Later, we propose a particular class of matrices as candidates for being RIP, namely, equiangular tight frames (ETFs). Using the known correspondence between real ETFs and strongly regular graphs, we investigate certain combinatorial implications of a real ETF being RIP.
    [Show full text]
  • Equiangular Tight Frames with Simplices and with Full Spark in Rd
    ISSN 1995-0802, Lobachevskii Journal of Mathematics, 2021, Vol. 42, No. 1, pp. 154–165. c Pleiades Publishing, Ltd., 2021. Equiangular Tight Frames with Simplices and with Full Spark in Rd S. Ya. Novikov1* (SubmittedbyA.M.Elizarov) 1Samara University, Samara, 443011 Russia Received May 23, 2020; revised August 19, 2020; accepted August 26, 2020 Abstract—An equiangular tight frame (ETF) is an equal norm tight frame with the same sharp angles between the vectors. This work is an attempt to create a brief review with complete proofs and calculations of two directions of research on the equiangular tight frames (ETF): bounds of the spark of the ETF, namely the smallest number of the vectors from ETF that are linearly dependent, and the existence of a regular simplex inside ETF. Tracing these two directions, we go through the case of equality in the Welch estimate, see the connection between RIP (restricted isometry property) and the spark of an ETF,construct a regular simplex using the technique of Naimark complements. We show the connection between equality in the lower estimate of the spark and the presence of a simplex inside ETF. Gram matrix and the matrix of the synthesis operator are calculated for the frame with 10 elements in the space R5. This frame contains a regular simplex and its spark is equal to 4, i.e. is not full. On the other hand you may see an example of the ETF with 6 vectors in R3 without a simplex, but with the spark equal to 4. In such cases the term “full spark” is used, so we have an example of the full spark ETF (FSETF).
    [Show full text]
  • An Improved Lower Bound of the Spark with Application
    AN IMPROVED LOWER BOUND OF THE SPARK WITH APPLICATION Wajeb Gharibi College of Computer Science and Information Systems Jazan University, Kingdom of Saudi Arabia [email protected] ABSTRACT Spark plays a great role in studying uniqueness of sparse solutions of the underdetermined linear equations. In this article, we derive a new lower bound of spark. As an application, we obtain a new criterion for the uniqueness of sparse solutions of linear equations. Keywords: Underdetermined, Linear equations, Sparse solution, Spark. 1. Introduction Recent theoretical developments have generated a great deal of interest in sparse signal representation. The setup assumes a given dictionary of “elementary” signals, and models an input signal as a linear combination of dictionary elements, with the provision that the representation is sparse, i.e., involves only a few of the dictionary elements. Finding sparse representations ultimately requires solving for the sparsest solution of an underdetermined system of linear equations. Such models arise often in signal processing, image processing, and digital communications. Given an full-rank matrix with no zero columns bRn , the linear system Ax b has infinitely many solutions when the system is underdetermined. Depending on the nature of source problems, we are often interested in finding a particular solution, and thus optimization methods come into a play through certain merit functions that measure the desired special structure of the solution. One of the recent interests is to find the sparsest solution of an underdetermined linear system, which has found many applications in signal and image processing [1-3]. To find a sparsest solution of , perhaps the ideal merit function is the cardinality of a vector, denoted by x i.e., the number of nonzero components of x .
    [Show full text]
  • Compressed Sensing Linear Algebra Review
    EE 520: Topics { Compressed Sensing Linear Algebra Review Notes scribed by Kevin Palmowski, Spring 2013, for Namrata Vaswani's course Notes on matrix spark courtesy of Brian Lois More notes added by Namrata Vaswani Notes based primarily on Horn and Johnson, Matrix Analysis, 1e and 2e, as well as Dr. Namrata Vaswani's in-class lectures. Chapter 0 { Miscellaneous Preliminaries Unless otherwise noted, all vectors are elements of Rn, although results extend to complex vector spaces. k n Let S = fvig1 ⊆ C . We define the span of S by ( k ) n k X span fSg = x 2 C j 9fαig1 ⊆ C with x = αivi : 1 Pk The set S is linearly independent if 1 αivi = 0 if and only if αi = 0 for all i. S is a spanning set for the vector space V if span fSg = V . A linearly independent spanning set for a vector space V is called a basis. The dimension of a vector space V , dim(V ), is the size of the smallest spanning set for V . The rank of a matrix A 2 Rm×n, rank(A), is the size of the largest linearly independent set of columns of A. Rank satisfies rank(A) ≤ minfm; ng. For matrices A 2 Rm×k and B 2 Rk×n, we have rank(A) + rank(B) − k ≤ rank(AB) ≤ minfrank(A); rank(B)g: For two matrices of the same size, we have rank(A + B) ≤ rank(A) + rank(B). P The trace of a matrix is the sum of its main diagonal entries, that is, trace(A) = aii.
    [Show full text]
  • FULL SPARK FRAMES 1. Introduction a Frame Over a Hilbert Space H Is A
    FULL SPARK FRAMES BORIS ALEXEEV, JAMESON CAHILL, AND DUSTIN G. MIXON Abstract. Finite frame theory has a number of real-world applications. In applications like sparse signal processing, data transmission with robustness to erasures, and reconstruction without phase, there is a pressing need for deterministic constructions of frames with the following property: every size-M subcollection of the M-dimensional frame elements is a spanning set. Such frames are called full spark frames, and this paper provides constructions using Vandermonde matrices and discrete Fourier transforms. Later, we prove that full spark Parseval frames are dense in the entire set of Parseval frames, meaning full spark frames are abundant, even if one imposes an additional tightness constraint. Finally, we prove that testing whether a given matrix is full spark is hard for NP under randomized polynomial-time reductions, indicating that deterministic full spark constructions are particularly significant because they guarantee a property which is otherwise difficult to check. 1. Introduction A frame over a Hilbert space H is a collection of vectors ffigi2I ⊆ H with frame bounds 0 < A ≤ B < 1 such that 2 X 2 2 Akxk ≤ jhx; fiij ≤ Bkxk 8x 2 H: (1) i2I N For a finite frame ffngn=1 in an M-dimensional Hilbert space HM , the upper frame bound B which satisfies (1) trivially exists. In this finite-dimensional setting, the notion of being a frame solely depends on the lower frame bound A being strictly positive, which is equivalent N to having the frame elements ffngn=1 span the Hilbert space. N In practice, frames are chiefly used in two ways.
    [Show full text]
  • Algorithms to Find the Girth and Cogirth of a Linear Matroid by John David Arellano a Thesis Submitted in Partial Fulfillment of the Requirements for the Degree Ph.D
    RICE UNIVERSITY Algorithms to Find the Girth and Cogirth of a Linear Matroid by John David Arellano A Thesis Submitted in Partial Fulfillment of the Requirements for the Degree Ph.D. Thesis Approved, Thesis Committee: Illya V. Hicks, Chair Associate Professor of Computational and Applied Mathematics Richard A. Tapia University Professor Maxfield-Oshman Professor of Engineering Professor of Computational and Applied Mathematics Wotao Yin Professor of Mathematics, UCLA Richard G. Baraniuk Victor E. Cameron Professor of Electrical and Computer Engineering Houston, Texas April, 2014 ABSTRACT Algorithms to Find the Girth and Cogirth of a Linear Matroid by John David Arellano In this thesis, I present algorithms to find the cogirth and girth, the cardinality of the smallest cocircuit and circuit respectively, of a linear matroid. A set covering problem (SCP) formulation of the problems is presented. The solution to the lin- ear matroid cogirth problem provides the degree of redundancy of the corresponding sensor network, and allows for the evaluation of the quality of the network. Hence, addressing the linear matroid cogirth problem can lead to significantly enhancing the design process of sensor networks. The linear matroid girth problem is related to reconstructing a signal in compressive sensing. I provide an introduction to ma- troids and their relation to the degree of redundancy problem as well as compressive sensing. I also provide an overview of the methods used to address linear matroid cogirth/girth problems, the SCP, and reconstructing a signal in compressive sensing. Computational results are provided to validate a branch-and-cut algorithm that ad- dresses the SCP formulation as well as an algorithm which uses branch decompositions and dynamic programming to find the girth of a linear matroid.
    [Show full text]
  • Multiarray Signal Processing: Tensor Decomposition Meets Compressed Sensing
    C. R. Mecanique 338 (2010) 311–320 Contents lists available at ScienceDirect Comptes Rendus Mecanique www.sciencedirect.com Multiarray signal processing: Tensor decomposition meets compressed sensing Traitement du signal multi-antenne : Les décompositions tensorielles rejoignent l’échantillonnage compressé ∗ Lek-Heng Lim a, Pierre Comon b, a Department of Mathematics, University of California, Berkeley, CA 94720-3840, United States b Laboratoire I3S, CNRS UMR6070, University of Nice, 06903, Sophia-Antipolis, France article info abstract Article history: We discuss how recently discovered techniques and tools from compressed sensing can be Received 16 December 2009 used in tensor decompositions, with a view towards modeling signals from multiple arrays Accepted after revision 29 June 2010 of multiple sensors. We show that with appropriate bounds on a measure of separation Available online 24 July 2010 between radiating sources called coherence, one could always guarantee the existence and Presented by Odile Macchi uniqueness of a best rank-r approximation of the tensor representing the signal. We also deduce a computationally feasible variant of Kruskal’s uniqueness condition, where the Keywords: coherence appears as a proxy for k-rank. Problems of sparsest recovery with an infinite Signal processing continuous dictionary, lowest-rank tensor representation, and blind source separation are Blind source separation treated in a uniform fashion. The decomposition of the measurement tensor leads to Blind channel identification simultaneous localization
    [Show full text]
  • Computing the Spark: Mixed-Integer Programming for the (Vector) Matroid Girth Problem
    Computing the Spark: Mixed-Integer Programming for the (Vector) Matroid Girth Problem Andreas M. Tillmann∗ (draft, last updated March 7, 2018) Abstract We investigate the NP-hard problem of computing the spark of a matrix (i.e., the smallest number of linearly dependent columns), a key parameter in compressed sensing and sparse signal recovery. To that end, we identify polynomially solvable special cases, gather upper and lower bounding procedures, and propose several exact (mixed-)integer programming models and linear programming heuristics. In particular, we develop a branch & cut scheme to determine the girth of a matroid, focussing on the vector matroid case, for which the girth is precisely the spark of the representation matrix. Extensive numerical experiments demonstrate the effectiveness of our specialized algorithms compared to general- purpose black-box solvers applied to several mixed-integer programming models. 1 Introduction The spark of a matrix A 2 Rm×n is defined as the smallest number of linearly dependent columns, i.e., spark(A) := minfkxk0 : Ax = 0; x 6= 0g; (1) where kxk0 := jfj : xj 6= 0gj. In the context of compressed sensing (cf. [31, 36]) and the sparse recovery problem min kxk0 s.t. Ax = b; (2) the spark is essential regarding solution uniqueness; in particular, every k-sparse vector x^ uniquely solves (2) with b := Ax^ if and only if 2k < spark(A). The spark is also a key parameter for identifying ambiguities in linear sensor arrays (see, e.g., [63]) or existence of unique tensor decompositions (by the relation to the Kruskal rank, which equals spark(A) − 1, see, e.g., [57, 56, 83]), verification of k-stability in the context of matrix completion (holds if and only if spark(A) = n − k + 1, see [84]), and other applications.
    [Show full text]
  • From Sparse Solutions of Systems of Equations to Sparse Modeling Of
    SIAM REVIEW c 2009 Society for Industrial and Applied Mathematics Vol. 51,No. 1,pp. 34–81 From Sparse Solutions of Systems of Equations to Sparse ∗ Modeling of Signals and Images Alfred M. Bruckstein† David L. Donoho‡ Michael Elad§ Abstract. A full-rank matrix A ∈ Rn×m with n<mgenerates an underdetermined system of linear equations Ax = b having infinitely many solutions. Suppose we seek the sparsest solution, i.e., the one with the fewest nonzero entries. Can it ever be unique? If so, when? As opti- mization of sparsity is combinatorial in nature, are there efficient methods for finding the sparsest solution? These questions have been answered positively and constructively in recent years, exposing a wide variety of surprising phenomena, in particular the existence of easily verifiable conditions under which optimally sparse solutions can be found by con- crete, effective computational methods. Such theoretical results inspire a bold perspective on some important practical problems in signal and image processing. Several well-known signal and image processing problems can be cast as demanding solutions of undetermined systems of equations. Such problems have previously seemed, to many, intractable, but there is considerable evidence that these problems often have sparse solutions. Hence, ad- vances in finding sparse solutions to underdetermined systems have energized research on such signal and image processing problems—to striking effect. In this paper we review the theoretical results on sparse solutions of linear systems, empirical results on sparse mod- eling of signals and images, and recent applications in inverse problems and compression in image processing. This work lies at the intersection of signal processing and applied mathematics, and arose initially from the wavelets and harmonic analysis research commu- nities.
    [Show full text]
  • Mixed-Integer Programming for the (Vector) Matroid Girth Problem
    Computing the Spark: Mixed-Integer Programming for the (Vector) Matroid Girth Problem Andreas M. Tillmann∗ March 7, 2018 Abstract We investigate the NP-hard problem of computing the spark of a matrix (i.e., the smallest number of linearly dependent columns), a key parameter in compressed sensing and sparse signal recovery. To that end, we identify polynomially solvable special cases, gather upper and lower bounding procedures, and propose several exact (mixed-)integer programming models and linear programming heuristics. In particular, we develop a branch & cut scheme to determine the girth of a matroid, focussing on the vector matroid case, for which the girth is precisely the spark of the representation matrix. Extensive numerical experiments demonstrate the effectiveness of our specialized algorithms compared to general- purpose black-box solvers applied to several mixed-integer programming models. 1 Introduction The spark of a matrix A 2 Rm×n is defined as the smallest number of linearly dependent columns, i.e., spark(A) := minfkxk0 : Ax = 0; x 6= 0g; (1) where kxk0 := jfj : xj 6= 0gj. In the context of compressed sensing (cf. [31, 36]) and the sparse recovery problem min kxk0 s.t. Ax = b; (2) the spark is essential regarding solution uniqueness; in particular, every k-sparse vector x^ uniquely solves (2) with b := Ax^ if and only if 2k < spark(A). The spark is also a key parameter for identifying ambiguities in linear sensor arrays (see, e.g., [63]) or existence of unique tensor decompositions (by the relation to the Kruskal rank, which equals spark(A) − 1, see, e.g., [57, 56, 83]), verification of k-stability in the context of matrix completion (holds if and only if spark(A) = n − k + 1, see [84]), and other applications.
    [Show full text]