An Overview of Compressed Sensing

What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered

An Overview of Compressed sensing

M. Vidyasagar FRS

Cecil & Ida Green Chair, The University of Texas at Dallas [email protected], www.utdallas.edu/∼m.vidyasagar Distinguished Professor, IIT Hyderabad [email protected]

Ba;a;ga;va;tua;l .=+a;ma mUa; a;tRa and Ba;a;ga;va;tua;l Za;a:=+d;a;}ba Memorial Lecture University of Hyderabad, 16 March 2015

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Outline

1 What is Compressed Sensing?

2 Main Results

3 Construction of Measurement Matrices Deterministic Approaches Probabilistic Approaches A Case Study

4 Some Topics Not Covered

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Outline

1 What is Compressed Sensing?

2 Main Results

3 Construction of Measurement Matrices Deterministic Approaches Probabilistic Approaches A Case Study

4 Some Topics Not Covered

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Compressed Sensing: Basic Problem Formulation

n Suppose x ∈ R is known to be k-sparse, where k n. That is, |supp(x)| ≤ k n, where supp denotes the support of a vector. However, it is not known which k components are nonzero. Can we recover x exactly by taking m n linear measurements of x?

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Precise Problem Statement: Basic

Deﬁne the set of k-sparse vectors

n Σk = {x ∈ R : |supp(x)| ≤ k}.

m×n Do there exist an integer m, a “measurement matrix” A ∈ R , m n and a “demodulation map” ∆ : R → R , such that

∆(Ax) = x, ∀x ∈ Σk?

Note:

Measurements are linear but demodulation map could be nonlinear. The algorithm is universal – the same A and ∆ need to work for every vector x, and nonadaptive – one has to choose all m rows of A at the outset.

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Further Considerations

What if:

The vector x is not exactly sparse, but only nearly sparse? The measurement is corrupted by noise, and equals Ax + η The vector x is (nearly) sparse in some other basis, and not the canonical basis (e.g., time and frequency)?

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Some Deﬁnitions

n Given a norm k · k on R , for each integer k < n, deﬁne

σk(x, k · k) := min kx − zk, z∈Σk the k-sparsity index of the vector x.

x ∈ Σk (x is k-sparse) if and only if σk(x, k · k) = 0 for every norm.

If x 6∈ Σk, then σk(x, k · k) depends on the speciﬁc norm.

The k-sparsity index w.r.t. to an `p-norm is easy to compute. Let Λ0 denote the index set of the k largest components by magnitude of x. Then

σ (x, k · k ) = kx c k . k p Λ0 p

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Precise Problem Statement: Final

Given integers n, k n and a real number > 0, do there exist an m×n m n integer m, a matrix A ∈ R , and a map ∆ : R → R such that k∆(Ax + η) − xk2 ≤ C1σk(x, k · kp) + C2, m whenever η ∈ R satisﬁes kηk2 ≤ ? Here C1,C2 are “universal” constants that do not depend on x or η. If so the pair (A, ∆) is said to display near-ideal signal recovery. This formulation combines several desirable features into one.

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Interpretation

Suppose x is k-sparse, and suppose y = Ax + η. If an “oracle” knows the support set of x, call it J, then the standard least-squares estimate is

t −1 t t −1 t xˆ = (AJ AJ ) AJ y = x + (AJ AJ ) AJ η,

t −1 t kxˆ − xk2 = k(AJ AJ ) AJ ηk2 ≤ C0kηk2, t −1 t where C0 is the induced norm of (AJ AJ ) AJ . With a pair (A, ∆) that achieves near-ideal signal recovery, we have

kxˆ − xk2 ≤ C2kηk2.

So, if (A, ∆) achieves near-ideal signal recovery, then the estimation error is a constant multiple of that achievable by an “oracle,” but without knowing the support set of x.

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Interpretation (Cont’d)

If x is not sparse, but measurements are noise-free (η = 0), then the estimate xˆ = ∆(Ax) satisﬁes

kxˆ − xk2 ≤ C1σk(x, k · kp).

So the estimate is within a “universal constant” times the k-sparsity index of x. If x is k-sparse and measurements are noise-free, we get exact signal recovery.

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Outline

1 What is Compressed Sensing?

2 Main Results

3 Construction of Measurement Matrices Deterministic Approaches Probabilistic Approaches A Case Study

4 Some Topics Not Covered

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Restricted Isometry Property

Deﬁnition m×n A matrix A ∈ R is said to satisfy the restricted isometry property (RIP) or order k with constant δk if

2 2 2 (1 − δk)kuk2 ≤ kAuk2 ≤ (1 + δk)kuk2, ∀u ∈ Σk.

t If J ⊆ {1, . . . , n} and |J| ≤ k, then the spectrum of AJ AJ lies in the interval [1 − δk, 1 + δk].

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered

Cande`es-Tao Result on `1-Norm Minimization

Theorem Suppose√ A satisﬁes the RIP of order 2k with constant n δ2k < 2 − 1. Given x ∈ R , deﬁne the demodulation map ∆ by

∆(y) =x ˆ := arg min kzk1 s.t. Az = y. z Then ∆(Ax) = x, ∀x ∈ Σk.

In plain English, any k-sparse vector can be recovered exactly by −1 minimizing kzk1 for z in A (Ax). Note that this is a linear programming problem.

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered A More General Result

Theorem m×n Suppose√ A ∈ R satisﬁes the RIP of order 2k with constant δ2k < 2 − 1, and that y = Ax + η where kηk2 ≤ . Deﬁne

xˆ = arg min kzk1 s.t. ky − Azk2 ≤ . z Then σk(x, k · k1) kxˆ − xk2 ≤ C0 √ + C2, k where √ √ 1 + ( 2 − 1)δ2k 4 1 + δ2k C0 = 2 √ ,C2 = √ . 1 − ( 2 + 1)δ2k 1 − ( 2 + 1)δ2k

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Some Observations

n Bounds are valid for all vectors x ∈ R – no assumptions of sparsity. The smaller we can make the RIP constant δ, the tighter the bounds. We suspect that making δ smaller requires larger values of m (more measurements). Indeed this is so, as we shall discover next.

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Deterministic Approaches Main Results Probabilistic Approaches Construction of Measurement Matrices A Case Study Some Topics Not Covered Outline

1 What is Compressed Sensing?

2 Main Results

3 Construction of Measurement Matrices Deterministic Approaches Probabilistic Approaches A Case Study

4 Some Topics Not Covered

Challenge: Given integers n (dimension of√ the vector), k (desired level of sparsity), and real number δ ∈ (0, 2 − 1), choose an m×n integer m and a matrix A ∈ R such that A has the RIP of order 2k with constant δ. m×n Refresher: A matrix A ∈ R is said to satisfy the restricted isometry property (RIP) or order k with constant δk if

2 2 2 (1 − δk)kuk2 ≤ kAuk2 ≤ (1 + δk)kuk2, ∀u ∈ Σk.

Two distinct classes of approaches: deterministic and probabilistic.

1 What is Compressed Sensing?

2 Main Results

3 Construction of Measurement Matrices Deterministic Approaches Probabilistic Approaches A Case Study

4 Some Topics Not Covered

m×n Given a matrix A ∈ R , assume w.l.o.g. that it is column-normalized, that is each column has `2-norm of one. Deﬁnition

The one-column coherence µ1(A) is deﬁned as

µ1(A) := max max |hai, aji|. i∈[n] j∈[n]\{i}

The k-column coherence µk(A) is deﬁned as X µk(A) := max max |hai, aji|. i∈[n] S⊆[n]\{i},|S|≤k j∈S

Coherence quantiﬁes how “nearly orthonormal” the columns are.

Easy consequence of the Gerschgorin circle theorem: Lemma m×n Suppose A ∈ R is column-normalized. Let k ≤ m be a ﬁxed integer. Suppose S ⊆ {1, . . . , n} and that |S| ≤ k ≤ m. Then

t spec(ASAS) ∈ [1 − µk−1(A), 1 + µk−1(A)].

Consequently A satisﬁes the RIP of order k with constant δk = µk−1(A).

The following result is known as the “Welch bound.” Lemma m×n Suppose A ∈ R is column-normalized; then r n − m 1 µ (A) ≥ ≈ √ , 1 m(n − 1) m

r n − m k µ (A) ≥ k ≈ √ , k m(n − 1) m √ In view of the Welch bound, any matrix with µk(A) ≈ k/ m can be thought of as having “optimal coherence.”

Due to DeVore (2007). Let p be a prime, and let a be a polynomial over the finite field Z/(p). Define the p × p matrix M(a) by 1 if j = a(i), [M(a)] = ij 0 if j 6= a(i).

Note that each column of M(a) has one 1 and the rest 0. Define 2 the p × 1 column vector ua by concatenating the p columns of M(a), and note that ua has exactly p ones and the rest are zero. Now define A0 ∈ {0, 1}p2×pr+1 by lining up the columns M(a), as a varies over all pr+1 polynomials of degree r or less with 0 √ coefficients in Z/(p). Finally, define A = A / p.

Theorem p2×pr+1 The matrix A ∈ R has coherence kr µ (A) ≤ . k p

Note that p is the square root of the number of rows of A. So this construction is within a factor of r of an “optimally coherent” matrix. The matrix A is very sparse, with only 1/p elements being nonzero. √ The nonzero elements are all equal to 1/ p; so this matrix is “multiplication free.”

Given n, k, δ, we need to choose a prime number p such that

(2k − 1)r (2k − 1)r ≤ δ, n ≤ pr+1 ⇐⇒ p ≥ max , n1/(r+1) . p δ

We can choose r as we wish. Then m = p2. Example 1: Let n = 10, 000, k = 5, δ = 0.4. Choosing r = 3 gives p ≥ 67.5 =⇒ p = 71 and m = p2 = 5, 041. Choosing r = 2 gives p ≥ 45 =⇒ p = 47 and m = p2 = 2, 209. Example 2: Let n = 106, k = 10, δ = 0.4. Choosing r = 3 gives p ≥ 142.5 =⇒ p = 149 and m = p2 = 22, 201. Choosing r = 2 gives p ≥ 100 =⇒ p = 101 and m = p2 = 10, 201. In general we get m ≈ n2/3 unless k is very large.

1 What is Compressed Sensing?

2 Main Results

3 Construction of Measurement Matrices Deterministic Approaches Probabilistic Approaches A Case Study

4 Some Topics Not Covered

Let X0 be a random variable with zero mean and standard 0 √ m×n deviation of one. Define X = X / m, and define Φ ∈ R to consist of nm independent realizations of X. Then it is easy to see that 2 2 n E[kΦuk2] = kuk2, ∀u ∈ R . 2 If the r.v. kΦuk2 is also “highly concentrated” around its expected value, then “with high probability” the matrix Φ satisfies the RIP.

A r.v. X is said to be sub-Gaussian if there exist constants α, β such that Pr{|X| > t} ≤ α exp(−βt2), ∀t > 0. A normal r.v. satisﬁes the above with α = 2, β = 0.5. For later use, deﬁne β2 c = 4α + 2β

Set-up: Given a constant δ, choose any < δ (preferably very close to δ), and choose any r such that

r 1 + r ≤ 1 − . 1 + δ

Theorem Suppose X0 is sub-Gaussian with constants α, β, and define c = β2/(4α + 2β). Define Φ as nm independent realizations of √ X0/ m. Then Φ satisfies the RIP of order k with constant δ with probability at least equal to 1 − ζ, where

enk 3 k ζ = 2 exp(−mc2). k k

Suppose k, δ are given, and choose , r as above. To ensure that Φ satisﬁes the RIP of order k with constant δ with probability ≥ 1 − ζ, it suﬃces to choose

1 2 en 3 m ≥ log + k log + k log c2 ζ k r

samples of X. Note that k m ≈ log n cδ2 plus other terms.

Choose X to be a Gaussian, so that α = 2, β = 0.5, and c = β2/(4α + 2β) = 1/44. Let ζ = 10−6. Example 1: Let n = 10, 000, k = 5, δ = 0.4. Then m = 21, 683 > n. Compare with m = 2, 209 for the deterministic approach. Example 2: Let n = 106, k = 10, δ = 0.4. Then m = 49, 863. Compare with m = 10, 201 for the deterministic approach.

In the deterministic approach, m ≈ n2/(r+1) for some integer r, whereas in the probabilistic approach, m = O(k log n). But the O symbol hides a huge constant! In both cases, the probabilistic approach requires more measurements than the deterministic approach! Moreover, the deterministic approach leads to a highly sparse matrix, whereas the probabilistic approach leads to a matrix where every element is nonzero, with probability one. Open Problem: Find a deterministic approach that leads to m = O(k log n) measurements.

1 What is Compressed Sensing?

2 Main Results

3 Construction of Measurement Matrices Deterministic Approaches Probabilistic Approaches A Case Study

4 Some Topics Not Covered

Whether a signal is “sparse” depends on the basis used. For example, a vector x denoting time samples of a signal may not be sparse, but its discrete cosine transform (or discrete Fourier transform) may be sparse. The use of the DFT requires measurement matrices with complex elements, but the theory works just the same. In particular, suppose M is the n × n discrete cosine transform matrix which is real and orthogonal, or the n × n discrete Fourier transform matrix which is complex and unitary. Transposes of “randomly selected” rows of these matrices satisfy the RIP.

This example is due to Cleve Moler (2010). Suppose x(t) = sin(1394πt) + sin(3296πt), and we sample at 40KHz for 0.2 seconds (8,000 samples). The three frequencies involved (1394 Hz, 3296 Hz and 40,000 Hz) are not commensurate.

This signal is not sparse in the time domain! However, it is sparse in the frequency domain. For this purpose we employ the discrete cosine transform (dct). N Given a vector x ∈ R , its discrete cosine transform (dct) N y ∈ R is given by

N X π(2n − 1)(k − 1) y(k) = w(k) x(n) cos , k = [N], 2N n=1 where the weight vector w is deﬁned by  q  1 , k = 1, w(k) N q 2  N , k = 2,...,N.

N Given a vector y ∈ R , its inverse discrete cosine transform (idct) is given by

N X π(2n − 1)(k − 1) x(n) = w(k)y(k) cos , n = [N], 2N k=1 where the weight vector is as before. Both the dct and idct correspond to multiplying the vector by an orthogonal matrix.

The dct of x(t) = sin(1394πt) + sin(3296πt) is shown below. It is highly concentrated around two frequencies, as expected.

Spectra of Original and Approximated Signals 60 Original Approximated

0 Coefficient

−20

−40

−60 0 100 200 300 400 500 600 700 800 900 1000 Frequency

There are 8,000 samples of the signal x(·). Now we will choose 500 samples at random from these 8,000 samples, and use those to reconstruct the signal. Note: When we generate 500 integers at random between 1 and 8,000, ony 485 distinct integers result. The next slide shows some of the samples.

The ﬁgure below shows the actual samples and some of the randomly chosen samples.

Original Signal and Random Samples 2

1.5

0.5

0 Signal Value −0.5

−1

−1.5

−2 0 0.002 0.004 0.006 0.008 0.01 0.012 0.014 Time in Seconds

Signal Reconstruction via `1-Norm Minimization

Let n = 8000, k = 485. Let S denote the index set of the randomly selected samples; note that |S| = 485 = k. Deﬁne D to be the dct of the n × n identity matrix. Thus the j-th column of k×n D is the dct of the j-th elementary vector. Deﬁne A ∈ R to consist of the rows of D corresponding to the randomly selected samples; that is, A equals the projection of D onto the rows in S. k Finally, let b ∈ R denote the randomly selected samples. n We will reconstruct the original signal x ∈ R by setting

xˆ = arg min kzk1 s.t. Az = b.

The ﬁgure below shows the original and reconstructed signal for small values of t.

Reconstruction of a Signal with 8000 Samples Using 500 Frequencies 2 Original Reconstructed 1.5

0.5

−0.5

−1 Original and Reconstructed Signals

−1.5

−2 0 0.005 0.01 0.015 0.02 0.025 Time

The ﬁgure below shows the original and reconstructed signal for slightly larger values of t. The reconstruction is indistinguishable from the original signal.

Reconstruction of a Signal with 8000 Samples Using 500 Frequencies 2 Original Reconstructed 1.5

0.5

−0.5

−1

Original and Reconstructed Signals −1.5

−2

−2.5 0.025 0.03 0.035 0.04 0.045 0.05 Time M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Deterministic Approaches Main Results Probabilistic Approaches Construction of Measurement Matrices A Case Study Some Topics Not Covered Reconstructed Signal in the Frequency Domain

The ﬁgure below shows the dcts of the original and reconstructed signals.

Spectra of Original and Approximated Signals 60 Original Approximated

0 Coefficient

−20

−40

−60 0 100 200 300 400 500 600 700 800 900 1000 Frequency

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Outline

1 What is Compressed Sensing?

2 Main Results

3 Construction of Measurement Matrices Deterministic Approaches Probabilistic Approaches A Case Study

4 Some Topics Not Covered

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Group Sparsity

Partition the index set {1, . . . , n} into g disjoint groups G1,...,Gg. n A vector x ∈ R is “group k-sparse” if its support contains elements from very few groups, and |supp(x)| ≤ k. “Group” analogs of all previous results exist, e.g. group RIP (GRIP), and both exact recovery of group k-sparse vectors, as well as approximate recovery of nearly group k-sparse vectors.

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered

Alternatives to the `1-Norm

What is so special about the `1-norm? It turns out: Nothing! There are inﬁnitely many norms that permit exact recovery of sparse or group-sparse vectors.

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered One-Bit Compressed Sensing

In standard compressed sensing, the measurement vector is i y = Ax, or yi = ha , xi, for i = 1, . . . , m. i What if yi = sign(ha , xi), for i = 1, . . . , m? This is called one-bit compressed sensing. The subject is still in its infancy. Perhaps the problem can be eﬀectively analyzed using probably approximately correct (PAC) learning theory.

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Low Rank Matrix Recovery

l×s Suppose M ∈ R has low rank, say ≤ k. By randomly sampling just m ls elements of M, is it possible to recover M exactly? Results to similar to vector recovery. Norm of a vector is replaced by “nuclear norm,” which is the sum of the singular values of a matrix.

M. Vidyasagar FRS Overview of Compressed Sensing What is Compressed Sensing? Main Results Construction of Measurement Matrices Some Topics Not Covered Questions?

M. Vidyasagar FRS Overview of Compressed Sensing