Sobolev Duals of Random Frames
C. Sinan Gunt¨ urk,¨ 1 Mark Lammers,2 Alex Powell,3 Rayan Saab,4 Ozg¨ ur¨ Yılmaz 4 1Courant Institute of Mathematical Sciences, New York University, NY, USA. 2University of North Carolina, Wilmington, NC, USA. 3Vanderbilt University, Nashville, TN, USA. 4University of British Columbia, Vancouver BC, Canada.
Abstract—Sobolev dual frames have recently been proposed as Not all Σ∆ quantization schemes are presented (or imple- optimal alternative reconstruction operators that are specifically mented) in the above canonical form, but they all can be tailored for Sigma-Delta (Σ∆) quantization of frame coefficients. rewritten as such for an appropriate choice of r and u. For the While the canonical dual frame of a given analysis (sampling) u frame is optimal for the white-noise type quantization error of noise shaping principle to work, it is crucial that the solution remains bounded. The smaller the size of the alphabet gets Pulse Code Modulation (PCM), the Sobolev dual offers significant A reduction of the reconstruction error for the colored-noise of relative to r, the harder it is to guarantee this property. The Σ∆ quantization. However, initial quantitative results concerning extreme case is 1-bit quantization, i.e., =2, which is also |A| the use of Sobolev dual frames required certain regularity the most challenging setting. In this paper, we will assume that assumptions on the given analysis frame in order to deduce Q is such that the sequence u remains uniformly bounded improvements of performance on reconstruction that are similar A to those achieved in the standard setting of bandlimited functions. for all inputs y such that y µ for some µ>0, i.e., that # #∞ ≤ In this paper, we show that these regularity assumptions can be there exists a constant C(µ, r, Q ) < such that lifted for (Gaussian) random frames with high probability on A ∞ the choice of the analysis frame. Our results are immediately uj C(µ, r, Q ), j. (3) | |≤ A ∀ applicable in the traditional oversampled (coarse) quantization scenario, but also extend to compressive sampling of sparse In this case, we say that the Σ∆ scheme or the quantization signals. rule is stable. A standard quantization rule is the so called “greedy rule” I.INTRODUCTION which minimizes uj given uj 1, . . . , uj r and yj, i.e., A. Background on Σ∆ quantization | | − − r i 1 r Methods of quantization have long been studied for over- qj = arg min ( 1) − uj i + yj a . (4) sampled data conversion. Sigma-Delta (Σ∆) quantization has a − i − − ∈A ! i=1 # $ ! been one of the dominant methods in this setting, such ! " ! The greedy rule is not! always stable for large values! of r. as A/D conversion of audio signals [9], and has received However, it is stable if is sufficiently big. It can be checked significant interest from the mathematical community in recent A that if years [5], [6]. Oversampling is typically exploited to employ (2r 1)δ/2+µ δ/2, (5) very coarse quantization (e.g., 1 bit/sample), however, the − ≤|A| working principle of Σ∆ quantization is applicable to any then u δ/2. However, this requires that contain at # #∞ ≤ A quantization alphabet. In fact, it is more natural to consider least 2r levels. With more stringent, fixed size quantization Σ∆ quantization as a “noise shaping” method, for it seeks alphabets, the best constant C(µ, r, Q ) in (3) has to be A a quantized signal (qj) by a recursive procedure to push the significantly larger than this bound. In fact, it is known that quantization error signal y q towards an unoccupied portion for any quantization rule with a 1-bit alphabet, C(µ, r, Q ) is − A of the signal spectrum. In the case of bandlimited signals, this Ω(rr), e.g., see [5], [6]. spectrum corresponds to high-frequency bands. The significance of stability in a Σ∆ quantization scheme The governing equation of a standard rth order Σ∆ quan- has to do with the reconstruction error analysis. In standard tization scheme with input y =(yj) and output q =(qj) is oversampled quantization, it is shown that the reconstruction r error incurred after low-pass filtering of the quantized output is (∆ u)j = yj qj,j=1, 2,..., (1) r − mainly controlled by u λ− , where λ is the oversampling # #∞ where ∆ stands for the difference operator defined by ratio, i.e., the ratio of the sampling frequency to the bandwidth (∆u)j := uj uj 1, and the qj are chosen according to of the reconstruction filter [5]. This error bound relies on the − − ∈A some quantization rule that incorporates quantities available specific structure of the space of bandlimited functions and in the recursion; the quantization rule typically takes the form the associated sampling theorem.
qj = Q (uj 1,uj 2, . . . , yj,yj 1,... ). (2) A − − − B. Σ∆ quantization and finite frames is called the quantization alphabet and is typically a finite Various authors have explored analogies of Σ∆ quantization A arithmetic progression of a given spacing δ, and symmetric and the above error bound in the setting of finite frames, about the origin. especially tight frames [1], [2], [4]. Recall that in a finite dimensional inner product space, a frame is simply a spanning frame variation defined by set of (typically not linearly independent) vectors. In this m setting, if a frame of consisting of m vectors is employed in V (F ) := f f , (10) # j − j+1#2 a space of k dimensions, then a corresponding oversampling j=1 ratio can be defined as λ = m/k. If a Σ∆ quantization " algorithm is used to quantize frame coefficients, then it would where one defines fm+1 =0. Higher-order frame variations to be desirable to obtain reconstruction error bounds that have be used with higher-order Σ∆ schemes are defined similarly, see [1], [2]. It is clear that for the frame variation to be small, the same nature as in the specific infinite dimensional setting k of bandlimited functions. the frame vectors must follow a smooth path in R . Frames (analysis as well as synthesis) that are obtained via uniform In finite dimensions, the equations of sampling, Σ∆ quan- k tization, and reconstruction can all be phrased using matrix sampling a smooth curve in R (so-called frame path) are equations, which we shall describe next. For the simplicity typical in many settings. However, the above “frame variation of our analysis, it will be convenient for us to set the bound” is useful in finite dimensions when the frame path terminates smoothly. Otherwise, it does not necessarily provide initial condition of the recursion in (1) equal to zero. With r higher-order reconstruction accuracy (i.e., of the type λ− ) u r+1 = = u0 =0, and j =1, . . . , m, the difference − ··· equation (1) can be rewritten as a matrix equation due to the presence of boundary terms. On the other hand, designing smoothly terminating frames can be technically Dru = y q, (6) challenging, e.g., [4]. − where D is defined by C. Sobolev duals Recently, a more straightforward approach was proposed 1 0 0 0 0 ··· for the design of (alternate) duals of finite frames for Σ∆ 1 1 0 0 0 − ··· quantization [8], [3]. Here, one instead considers the operator 0 1 1 0 0 norm of FDr on #m and the corresponding bound − ··· 2 D = . . . (7) ...... x xˆ FDr u . 2 op 2 (11) # − # ≤# # # # 00 1 1 0 ···− (Note that this bound is not available in the infinite dimen- 00 0 11 ··· − sional setting of bandlimited functions due to the fact that u is typically not in #2(N).) With this bound, it is now natural to Suppose that we are given an input signal x and an analysis r m k minimize FD op over all dual frames of a given analysis frame (ei)1 of size m in R . We can represent the frame # # frame E. These frames have been called Sobolev duals, in vectors as the rows of an an m k matrix E, the sampling × analogy with classical L -type Sobolev (semi)norms. operator. The sequence y will simply be the frame coefficients, 2 Σ∆ quantization algorithms are normally designed for ana- i.e., y = Ex. Similarly, let us consider a corresponding m log circuit operation, so they control u , which would synthesis frame (fj)1 . We stack these frame vectors along # #∞ control u only in a suboptimal way. However, it turns the columns of a k m matrix F , the reconstruction operator, # #2 × out that there are important advantages in working with the which is then a left inverse of E, i.e., FE = I. The Σ∆ # norm in the analysis. The first advantage is that Sobolev quantization algorithm will replace the coefficient sequence y 2 duals are readily available by an explicit formula. The solution with its quantization given by q, which will then represent F of the optimization problem the coefficients of an approximate reconstruction xˆ using sob,r xˆ = Fq x = Fy r the synthesis frame. Hence, . Since , the min FD op subject to FE = I (12) reconstruction error is given by F # # is given by the matrix equation x xˆ = FDru. (8) − r r Fsob,rD =(D− E)†, (13) With this expression, x xˆ can be bounded for any norm # − # simply as where † stands for the Moore-Penrose inversion operator, #·# 1 which, in our case, is given by E† := (E∗E)− E∗. Note m r that for r =0(i.e., no noise-shaping, or PCM), one simply x xˆ u (FD )j . (9) # − # ≤ # #∞ # # obtains F = E†, the canonical dual frame of E. j=1 " As for the reconstruction error bound, plugging (13) into r r Here (FD )j is the jth column of FD . In fact, when suitably (11), it immediately follows that stated, this bound is also valid in infinite dimensions, and r 1 has been used extensively in the mathematical treatment of x xˆ 2 (D− E)† op u 2 = u 2, (14) # − # ≤# # # # σ (D rE)# # oversampled A/D conversion of bandlimited functions. min − r For r =1, and = 2, the summation term on the where σmin(D− E) stands for the smallest singular value of #·# #·# r right hand side of (9) motivated the study of the so-called D− E. D. Main result We have seen that the main object of interest for the r The main advantage of the Sobolev dual approach is that reconstruction error bound is σmin(D− E) for a random frame highly developed methods are present for spectral norms of E. Let H be a square matrix. The first observation we make matrices, especially in the random setting. Minimum singular is that when E is i.i.d. Gaussian, the distribution of Σ(HE) values of random matrices with i.i.d. entries have been studied is the same as the distribution of Σ(Σ(H)E). To see this, let extensively in the mathematical literature. For an m k random UΣ(H)V ∗ be the singular value decomposition of H where × matrix E with i.i.d. entries sampled from a sub-Gaussian U and V are unitary matrices. Then HE = UΣ(H)V ∗E. distribution with zero mean and unit variance one has Since the unitary transformation U does not alter singular values, we have Σ(HE)=Σ(Σ(H)V ∗E), and because of the σmin(E) √m √k (15) unitary invariance of the i.i.d. Gaussian measure, the matrix ≥ − E˜ := V E has the same distribution as E, hence the claim. with high probability [11]. However, note that D rE would ∗ − Therefore it suffices to study the singular values of Σ(H)E. not have i.i.d. entries. A naive approach would be to split; r r r In our case, H = D− and we first need information on the σmin(D− E) is bounded from below by σmin(D− )σmin(E). r r deterministic object Σ(D− ). However (see Lemma II.1), σmin(D− ) satisfies r r A. Singular values of D− σmin(D− ) r 1, (16) * It turns out that we only need an approximate estimate of r and therefore this naive product bound yields no improvement the singular values of D− : in r on the reconstruction error for Σ∆-quantized measure- r Lemma II.1. Let r be any positive integer and D be as in ments. As can be expected, the true behavior of σ (D− E) min c (r) c (r) turns out to be drastically different, and is described as part (7). There are positive numerical constants 1 and 2 , m of Theorem A, our main result in this paper. independent of , such that r r m r m Theorem A. Let E be an m k random matrix whose c1(r) σj(D− ) c2(r) ,j=1, . . . , m. × j ≤ ≤ j entries are i.i.d. Gaussian random variables with zero mean , - , - (19) and unit variance. For any α (0, 1) and positive in- ∈ Here we shall only give a simple heuristic argument. First, teger r, there are constants c, c%,c%% > 0 such that if r 1/(1 α) it is convenient to work with the singular values of D instead, λ := m/k c(log m) − , then with probability at least r r ≥ α r because D is a banded (Toeplitz) matrix whereas D− is full. 1 exp( c%mλ− ), the smallest singular value of D− E − − satisfies Note that because of our convention of descending ordering r α(r 1 ) of singular values, we have σ (D− E) c%%λ − 2 √m. (17) min ≥ r 1 In this event, if a stable rth order Σ∆ quantization scheme is σj(D− )= r ,j=1, . . . , m. (20) σm+1 j(D ) used in connection with the rth order Sobolev dual frame of − E, then the resulting reconstruction error is bounded by For r =1, an explicit formula is available [12]. Indeed, we have α(r 1 ) x xˆ 2 ! λ− − 2 . (18) πj # − # σ (D) = 2 cos ,j=1, . . . , m, (21) j 2m +1 Previously, the only setting in which this type of approxi- # $ mation accuracy could be achieved (with or without Sobolev which implies duals) was the case of highly structured frames (e.g. when 1 1 the frame vectors are found by sampling along a piecewise σj(D− )= ,j=1, . . . , m. (22) π(j 1/2) smooth frame path). Theorem A shows that such accuracy is 2 sin 2(m−+1/2) generically available, provided the reconstruction is done via For r>1, the first observation, is- that σ (Dr) and (σ (D))r Sobolev duals. j j are different, because D and D∗ do not commute. However, II.SKETCHOF PROOFOF THEOREM A this becomes insignificant as the size m . In fact, the →∞ asymptotic distribution of (σ (Dr))m as m is rather Below we give a sketch of the proof of Theorem A as well j j=1 →∞ easy to find using standard results in the theory of Toeplitz as describe the main ideas in our approach. The full argument matrices: D is a banded Toeplitz matrix whose symbol is is given in [7]. f(θ) = 1 eiθ, hence the symbol of Dr is (1 eiθ)r. It In what follows, σj(A) will denote the jth largest singular − − then follows by Parter’s extension of Szego’s¨ theorem [10] value of the matrix A. Similarly, λ (B) will denote the jth j that for any continuous function ψ, we have largest eigenvalue of the Hermitian matrix B. Hence, we have m π σj(A)= λj(A∗A). We will also use the notation Σ(A) 1 r 1 r for the diagonal matrix of singular values of A, with the lim ψ(σj(D )) = ψ( f(θ) ) dθ. (23) m m 2π π | | + →∞ j=1 .− convention (Σ(A))jj = σj(A). All matrices in our discussion " will be real valued and the Hermitian conjugate reduces to the We have f(θ) = 2 sin θ /2 for θ π, hence the | | r m | | | |≤ transpose. distribution of (σj(D ))j=1 is asymptotically the same as r that of 2r sin (πj/2m), and consequently, we can think of Proof: Consider a ρ-net Q˜ of the unit sphere of Rk with 1 k r r r − ˜ 2 σj(D− ) roughly as 2 sin (πj/2m) which behaves, up #Q ρ +1 where the value of ρ< 1 will be chosen later. r ≤ to r-dependent constants, as (m/j) . Moreover, we know that ˜ ˜ r 1/2 ˜ r r /r 0 r r Let (Q/) be the0 event SEq 2 c1Λ − , q Q . By D D 2 , hence σ (D− ) 2− . E # # ≥ ∀ ∈ # #op ≤# #op ≤ min ≥ Lemma II.3, we know that In [7], the above heuristic is turned into a formal statement 1 2 k r/2 via a standard perturbation theorem (namely, Weyl’s theorem) c 2 60m m(r 1/2)/Λ P ˜(Q˜) +1 e− − . (27) on the spectrum of Hermitian matrices. Here, the important E ≤ ρ Λ r r r # $ # $ observation is that D∗ D is a perturbation of (D∗D) of , - Let be the event in Lemma II.2 with Θ= 4. Let E be any rank at most 2r. E given matrix in the event ˜(Q˜). For each x 2 =1, there r E∩ E # # B. Lower bound for σmin(D− E) is q Q˜ with q x ρ, hence by Lemma II.2, we have ∈ # − #2 ≤ In light of the above discussion, the distribution of r r SE(x q) 2 4 s x q 2 4c2m ρ. σmin(D− E) is the same as that of # − # ≤ # #∞# − # ≤ r Choose inf Σ(D− )Ex . r 1/2 r 2 (24) c1Λ − c1 Λ x 2=1 # # ' ' ρ = r = . r 8c2m 8c2√Λ m We replace Σ(D− ) with an arbitrary diagonal matrix S with , - Sjj =: sj > 0. For the next set of results, we will work with Hence Gaussian variables with variance equal to 1/m. SEx 2 SEq 2 SE(x q) 2 # # ≥# r #1/2−# r− # Lemma II.2. Let E be an m k random matrix whose entries c1Λ − 4c2m ρ 1 × ≥ − are i.i.d. (0, m ). For any Θ > 1, consider the event 1 r 1/2 N = c Λ − . 2 1 √ := SE $k $m 2 Θ s . E # # 2 → 2 ≤ # #∞ This shows that ˜(Q˜) . Clearly, ρ 1/8 by our choice 1 2 E∩E ⊂F ≤ Then of parameters and hence 2 +1 17 . The result follows by c k m/2 (Θ 1)m/2 ρ ≤ 8ρ P ( ) 5 Θ e− − . using the probability bounds of Lemma II.2 and (27). E ≤ The above lemma follows easily from the bound S E The following theorem is now a direct corollary of the above # #·# # on SE and the corresponding standard concentration esti- estimate. # # mates for E . Likewise, the proof of the next lemma also Theorem II.5. Let E be an m k random matrix whose # # × follows from standard methods. entries are i.i.d. (0, 1 ), r be a positive integer, D be the N m 1 difference matrix defined in (7), and the constant c1 = c1(r) Lemma II.3. Let ξ (0, Im), r be a positive integer, ∼N m be as in Lemma II.1. Let 0 <α< 1 be any number. Assume and c1 > 0 be such that r that m m 1/(1 α) s c ,j=1, . . . , m. (25) λ := c3(log m) − , (28) j ≥ 1 j k ≥ # $ Then for any Λ 1 and m Λ, where c3 = c3(r) is an appropriate constant. Then ≥ ≥ 1 α α r α(r 1/2) c4m − k m P σmin(D− E) c1λ − 1 2e− (29) 2 2 2 2r 1 r/2 m(r 1/2)/Λ ≥ ≥ − P s ξ
4 10 3 10
3 1 10 E N(0, ) ∼ m 2 1 10 r σmin(D− E) 2 1 10 r σmin(D− E) 1 10 1 10
0 10 0 10
−1 −1 10 10
−2 −2 10 10 0 1 0 1 10 10 10 10 λ = m/k λ = m/k
r r Fig. 1. Numerical behavior (in log-log scale) of 1/σmin(D− E) as a Fig. 2. Numerical behavior (in log-log scale) of 1/σmin(D− E) as a function of λ = m/k, for r =0, 1, 2, 3, 4. In this figure, k = 50 and function of λ = m/k, for r =0, 1, 2, 3, 4. In this figure, k = 10 and 1 r ≤ 1 λ 20. For each problem size, the largest value of 1/σmin(D− E) λ 20. For each r and problem size (m, k), a random m k submatrix E of among≤ 50≤ realizations of a random m k matrix E sampled from the Gaussian the≤N N Discrete Cosine Transform matrix, where N =× 10m, was selected. 1 × × r ensemble (0, m Im) was recorded. After 50 such random selections, the largest value of 1/σmin(D− E) was N recorded. then the same probabilistic estimate on σ (D rE) holds for min − ACKNOWLEDGMENT every m k submatrix E of Φ. This extension has powerful × implications on compressed sensing. The details are given in The authors would like to thank Ronald DeVore for value- [7]. able discussions, and the American Institute of Mathematics and the Banff International Research Station for hosting two III.NUMERICALEXPERIMENTSFOR GAUSSIANAND meetings where this work was initiated. OTHERRANDOMENSEMBLES REFERENCES In order to test the accuracy of Theorem II.5, our first [1] J.J. Benedetto, A.M. Powell, and O.¨ Yılmaz. Sigma-delta (Σ∆) quan- numerical experiment concerns the minimum singular value tization and finite frames. IEEE Transactions on Information Theory, of D rE as a function of λ = m/k. In Figure 1, we plot 52(5):1990–2005, May 2006. − [2] J.J. Benedetto, A.M. Powell, and O.¨ Yılmaz. Second order sigma–delta the worst case (the largest) value, among 50 realizations, of (Σ∆) quantization of finite frame expansions. Appl. Comput. Harmon. r 1/σmin(D− E) for the range 1 λ 20, where we have Anal, 20:126–148, 2006. ≤ ≤ ¨ kept k = 50 constant. As predicted by this theorem, we find [3] J. Blum, M. Lammers, A.M. Powell, and O. Yılmaz. Sobolev duals in frame theory and Sigma-Delta quantization. Journal of Fourier Analysis that the negative slope in the log-log scale is roughly equal to and Applications. Published online in October 2009. r 1/2, albeit slightly less, which seems in agreement with [4] B.G. Bodmann, V.I. Paulsen, and S.A. Abdulbaki. Smooth Frame-Path − the presence of our control parameter α. As for the size of the Termination for Higher Order Sigma-Delta Quantization. 13(3):285– r r+1/2 307, 2007. r-dependent constants, the function 5 λ− seems to be a [5] I. Daubechies and R. DeVore. Approximating a bandlimited function reasonably close numerical fit, which also explains why we using very coarsely quantized data: A family of stable sigma-delta observe the separation of the individual curves after λ> 5. modulators of arbitrary order. Ann. of Math., 158(2):679–710, September 2003. It is natural to consider other random matrix ensembles [6] C.S. Gunt¨ urk.¨ One-bit sigma-delta quantization with exponential accu- when constructing random frames and their Sobolev duals. racy. Comm. Pure Appl. Math., 56(11):1608–1630, 2003. [7] C.S. Gunt¨ urk,¨ A. Powell, R. Saab, and O.¨ Yılmaz, Sobolev Duals for In the setting of compressed sensing, the restricted isometry Random Frames and Sigma-Delta Quantization of Compressed Sensing property has been extended to measurement matrices sampled Measurements. arXiv:1002.0182. from the Bernoulli and general sub-Gaussian distributions. [8] M. Lammers, A.M. Powell, and O.¨ Yılmaz. Alternative dual frames for digital-to-analog conversion in Sigma-Delta quantization. Advances in Structured random matrices, such as random Fourier samplers, Computational Mathematics, Vol. 32 (1) pp. 73-102, 2008. have also been considered now in some detail. Our numerical [9] S.R. Norsworthy, R.Schreier, and G.C. Temes, editors. Delta-Sigma findings for the Bernoulli case are almost exactly the same Data Converters. IEEE Press, 1997. [10] S.V. Parter. On the distribution of the singular values of Toeplitz as those for the Gaussian case, hence we omit. For structured matrices. Linear Algebra Appl., 80:115–130, 1986. random matrices, we give the results of one of our numerical [11] M. Rudelson and R. Vershynin. Smallest singular value of a random experiments in Figure 2. Here E is a randomly chosen m k rectangular matrix. Comm. Pure Appl. Math., 62(12):1595–1739, 2009. × submatrix of a much larger N N Discrete Cosine Transform [12] G. Strang. The discrete cosine transform. SIAM review, pages 135–147, × 1999. matrix. We find the results somewhat similar, but as expected, perhaps less reliable than the Gaussian ensemble.