Circulant and Toeplitz Matrices in Compressed Sensing Holger Rauhut Hausdorff Center for Mathematics & Institute for Numerical Simulation University of Bonn Conference on Time-Frequency Strobl, June 15, 2009 Institute for Numerical Simulation Rheinische Friedrich-Wilhelms-Universität Bonn Holger Rauhut Hausdorff Center for Mathematics & Institute for NumericalCirculant and Simulation Toeplitz University Matrices inof CompressedBonn Sensing Overview • Compressive Sensing (compressive sampling) • Partial Random Circulant and Toeplitz Matrices • Recovery result for `1-minimization • Numerical experiments • Proof sketch (Non-commutative Khintchine inequality) Holger Rauhut Circulant and Toeplitz Matrices 2 Recovery requires the solution of the underdetermined linear system Ax = y: Idea: Sparsity allows recovery of the correct x. Suitable matrices A allow the use of efficient algorithms. Compressive Sensing N Recover a large vector x 2 C from a small number of linear n×N measurements y = Ax with A 2 C a suitable measurement matrix. Usual assumption x is sparse, that is, kxk0 := j supp xj ≤ k where supp x = fj; xj 6= 0g. Interesting case k < n << N. Holger Rauhut Circulant and Toeplitz Matrices 3 Compressive Sensing N Recover a large vector x 2 C from a small number of linear n×N measurements y = Ax with A 2 C a suitable measurement matrix. Usual assumption x is sparse, that is, kxk0 := j supp xj ≤ k where supp x = fj; xj 6= 0g. Interesting case k < n << N. Recovery requires the solution of the underdetermined linear system Ax = y: Idea: Sparsity allows recovery of the correct x. Suitable matrices A allow the use of efficient algorithms. Holger Rauhut Circulant and Toeplitz Matrices 3 n×N For suitable matrices A 2 C , `0 recovers every k-sparse x provided n ≥ 2k. Problem: `0 is NP hard in general. Need efficient alternative algorithms! We will consider `1-minimization `0-minimization Natural approach min kxk0 subject to Ax = y Holger Rauhut Circulant and Toeplitz Matrices 4 Problem: `0 is NP hard in general. Need efficient alternative algorithms! We will consider `1-minimization `0-minimization Natural approach min kxk0 subject to Ax = y n×N For suitable matrices A 2 C , `0 recovers every k-sparse x provided n ≥ 2k. Holger Rauhut Circulant and Toeplitz Matrices 4 Need efficient alternative algorithms! We will consider `1-minimization `0-minimization Natural approach min kxk0 subject to Ax = y n×N For suitable matrices A 2 C , `0 recovers every k-sparse x provided n ≥ 2k. Problem: `0 is NP hard in general. Holger Rauhut Circulant and Toeplitz Matrices 4 `0-minimization Natural approach min kxk0 subject to Ax = y n×N For suitable matrices A 2 C , `0 recovers every k-sparse x provided n ≥ 2k. Problem: `0 is NP hard in general. Need efficient alternative algorithms! We will consider `1-minimization Holger Rauhut Circulant and Toeplitz Matrices 4 `1-minimization Convex relaxation of `0 problem: N X min kxk1 = jxj j subject to Ax = y j=1 Efficient convex optimization methods apply. Idea: The minimizers of `1 and `0 coincide under appropriate conditions on the matrix A and on the sparsity k of x. Holger Rauhut Circulant and Toeplitz Matrices 5 Theorem (Cand´es,Romberg, Tao, '06,'08; Foucart, Lai '08) p n×N 2(3− 2) IfA 2 C is a matrix satisfying δ2k < 7 ≈ 0:4531 then `1-minimization stably recovery everyk-sparsex fromy = Ax. Restricted Isometry Property (RIP) The restricted isometry constant δk of a matrix A is the smallest constant δ such that 2 2 2 (1 − δ)kxk2 ≤ kAxk2 ≤ (1 + δ)kxk2 for all k-sparse x: All column submatrices of A with at most k columns are well-conditioned. Holger Rauhut Circulant and Toeplitz Matrices 6 Restricted Isometry Property (RIP) The restricted isometry constant δk of a matrix A is the smallest constant δ such that 2 2 2 (1 − δ)kxk2 ≤ kAxk2 ≤ (1 + δ)kxk2 for all k-sparse x: All column submatrices of A with at most k columns are well-conditioned. Theorem (Cand´es,Romberg, Tao, '06,'08; Foucart, Lai '08) p n×N 2(3− 2) IfA 2 C is a matrix satisfying δ2k < 7 ≈ 0:4531 then `1-minimization stably recovery everyk-sparsex fromy = Ax. Holger Rauhut Circulant and Toeplitz Matrices 6 Random Partial Fourier matrix: (n rows of the discrete Fourier matrix selected at random) n ≥ Cδ−2k log4(N) log(−1): RIP for random matrices n×N A Bernoulli random matrix A 2 R with independent entries p1 having values ± n with equal probability; or a Gaussian random n×N matrix A 2 R having independent normal distributed entries with mean zero and variance 1=n satisfies δk ≤ δ with probability at least1 − provided n ≥ Cδ−2(k log(N=k) + log(−1)): Holger Rauhut Circulant and Toeplitz Matrices 7 RIP for random matrices n×N A Bernoulli random matrix A 2 R with independent entries p1 having values ± n with equal probability; or a Gaussian random n×N matrix A 2 R having independent normal distributed entries with mean zero and variance 1=n satisfies δk ≤ δ with probability at least1 − provided n ≥ Cδ−2(k log(N=k) + log(−1)): Random Partial Fourier matrix: (n rows of the discrete Fourier matrix selected at random) n ≥ Cδ−2k log4(N) log(−1): Holger Rauhut Circulant and Toeplitz Matrices 7 Circulant and Toeplitz matrices N Circulant matrix: For b = (b0; b1;:::; bN−1) 2 R let b N×N S = S 2 R be the matrix with entries Si;j = bj−i mod N , 0 1 b0 b1 ······ bN−1 BbN−1 b0 b1 ··· bN−2C Sb = B C : B . C @ . A b1 b2 ··· bN−1 b0 2N−1 Toeplitz matrix: For c = (c−N−1; c−N−2;:::; cN−1) 2 R let c N×N T = T 2 R be the matrix with entries Ti;j = ci−j , 0 1 c0 c1 ······ cN−1 B c−1 c0 c1 ··· cN−2C T c = B C : B . C @ . A c−N−1 c−N−2 ··· c−1 c0 Holger Rauhut Circulant and Toeplitz Matrices 8 N 2N−1 We will always choose the vectors b 2 R ; c 2 R at random. b c Performance of SΩ, TΩ in compressed sensing? Partial random circulant and Toeplitz matrices LetΩ ⊂ f1;:::; Ng arbitrary of cardinality n. N RΩ: operator that restricts a vector x 2 R to its entries in Ω. Restrict Sb and T c to the rows indexed by Ω: b b n×N Partial circulant matrix: SΩ = RΩS 2 R c c n×N Partial Toeplitz matrix: TΩ = RΩT 2 R Convolution followed by subsampling: b y = RΩS x = RΩ(x ∗ b) Holger Rauhut Circulant and Toeplitz Matrices 9 Partial random circulant and Toeplitz matrices LetΩ ⊂ f1;:::; Ng arbitrary of cardinality n. N RΩ: operator that restricts a vector x 2 R to its entries in Ω. Restrict Sb and T c to the rows indexed by Ω: b b n×N Partial circulant matrix: SΩ = RΩS 2 R c c n×N Partial Toeplitz matrix: TΩ = RΩT 2 R Convolution followed by subsampling: b y = RΩS x = RΩ(x ∗ b) N 2N−1 We will always choose the vectors b 2 R ; c 2 R at random. b c Performance of SΩ, TΩ in compressed sensing? Holger Rauhut Circulant and Toeplitz Matrices 9 Why Circulant and Toeplitz matrices? • Structure: Fast algorithms via the FFT • Smaller amount of randomness than for Gaussian / Bernoulli matrices • Natural appearance in applications such as system identification, radar, channel estimation in wireless communication, ... Holger Rauhut Circulant and Toeplitz Matrices 10 Previous results Bajwa et al.(2007/2008): Ω = f1;:::; ng, the entries of b and c are either independent Bernoulli ±1 or standard normal random variables. 2 p1 b p1 c If n ≥ Cδk log(N=k) then n SΩ and n TΩ satisfy RIP, δ2k ≤ δ In N particular, `1-minimization recovers k-sparse vectors x 2 R . Romberg (2008): Ω is chosen at random, b = F −1b^, where F is the Fourier transform and the entries of b^ are essentially chosen at random from a uniform distribution on the torus (such that Sb is a real matrix). 3 Assume n ≥ maxfC1k log(N/); C2 log (N/)g. Fix a support set S and choose the signs of the non-zero coefficients of x on S independently and uniformly at random. Then `1-minimization recovers x with probability at least1 − . Holger Rauhut Circulant and Toeplitz Matrices 11 Main Result Theorem Let Ω ⊂ f1; 2;:::; Ng be an arbitrary (deterministic) set of N cardinality n. Letx 2 R bek-sparse such that the signs of its non-zero entries are Bernoulli ±1 random variables. Choose N b 2 R to be a random vector whose entries are ±1 Bernoulli b n random variables. Lety = SΩx 2 R . There exists a constant C > 0 such that n ≥ Ck log3(N/) implies that with probability at least 1 − the `1-minimizer coincides withx. c b The same statement holds withT Ω in place of SΩ where 2N−1 c 2 R is a random vector with Bernoulli ±1 entries. Improvements with respect to previous results: Theorem removes randomness in Ω, n scales essentially linear in k c and it works also for TΩ. Holger Rauhut Circulant and Toeplitz Matrices 12 Numerical experiments Partial Circulant and Toeplitz matrix generated by Bernoulli vector, N = 500, n = 100. Empirical Recovery Rate Empirical Recovery Rate 1 1 0.9 0.9 0.8 0.8 0.7 0.7 0.6 0.6 0.5 0.5 0.4 0.4 0.3 0.3 0.2 0.2 0.1 0.1 0 0 0 5 10 15 20 25 30 35 40 0 5 10 15 20 25 30 35 40 Sparsity Sparsity (a) Circulant (b) Toeplitz Holger Rauhut Circulant and Toeplitz Matrices 13 Numerical experiments Partial Circulant and Toeplitz matrix generated by Gaussian vector, N = 500, n = 100.