1 Polarization of the Renyi´ Information Dimension with Applications to Compressed Sensing Saeid Haghighatshoar, Member, IEEE, Emmanuel Abbe, Member, IEEE Abstract—In this paper, we show that the Hadamard matrix in diverse areas in probability theory and signal processing acts as an extractor over the reals of the Renyi´ information such as signal quantization [4], rate-distortion theory [5], and dimension (RID), in an analogous way to how it acts as fractal geometry [6]. More recently, the operational aspect of an extractor of the discrete entropy over finite fields. More precisely, we prove that the RID of an i.i.d. sequence of mixture RID has reappeared in applications as varied as lossless analog random variables polarizes to the extremal values of 0 and 1 compression [7–9], Compressed Sensing of sparse signals [10– (corresponding to discrete and continuous distributions) when 12], and characterization of the degrees-of-freedom (DoFs) of transformed by a Hadamard matrix. Further, we prove that vector interference channels [13, 14], which has recently been the polarization pattern of the RID admits a closed form of significant importance in wireless communication. expression and follows exactly the Binary Erasure Channel (BEC) polarization pattern in the discrete setting. We also extend the In this paper, motivated by [3], we first extend the definition results from the single- to the multi-terminal setting, obtaining a of RID as an information measure from scalar random Slepian-Wolf counterpart of the RID polarization. We discuss variables to a family of vector random variables over which applications of the RID polarization to Compressed Sensing the RID is well-defined. We also extend the definition to the of i.i.d. sources. In particular, we use the RID polarization to joint and the conditional RIDs and provide a closed-form construct a family of deterministic ˘1-valued sensing matrices for Compressed Sensing. We run numerical simulations to expression for computing them. Using these, we investigate compare the performance of the resulting matrices with that the high-dimensional behavior of the RID of i.i.d. mixture of random Gaussian and random Hadamard matrices. The random variables when transformed by a Hadamard matrix. results indicate that the proposed matrices afford competitive We prove that the conditional RIDs of almost all the resulting performances while being explicitly constructed. random variables polarize to the extremal values of 0 and Index Terms—Renyi´ Information Dimension, Polarization 1. We also obtain a formula for computing those conditional Theory, Slepian-Wolf coding, Compressed Sensing. RIDs and their polarization pattern using the Binary Erasure Channel (BEC) polarization in the discrete case [15]. This I.I NTRODUCTION gives a natural extension of the polarization phenomenon for Let X be a real-valued random variable. We denote the q-ary the entropy over the finite fields to the RID over the reals. tqXu We study some of the potential applications of the quantization of X by xXyq “ q ; where for a real number r, we denote by tru the largest integer less than or equal to r. new polarization result in Compressed Sensing [16–19]. In The upper and the lower Renyi´ Information Dimension (RID) particular, motived by the recent results on the operational of X are defined by aspect of RID in Compressed Sensing [7, 10] and inspired by the success of polar codes in achieving information theoretic HpxXyqq limits [15, 20], we exploit the RID polarization to design dpXq “ lim sup ; (1) qÑ8 log2pqq deterministic partial Hadamard matrices for Compressed Sensing of i.i.d. sparse signals. We compare the performance HpxXyqq dpXq “ lim inf ; (2) of the resulting matrices with that of other traditional qÑ8 log2pqq arXiv:1301.6388v2 [cs.IT] 23 Aug 2017 matrices in Compressed Sensing such as random Gaussian where HpxXyqq denotes the Shannon entropy of the discrete and random Hadamard matrices. Numerical simulations random variable xXyq obtained from the quantization. If the provide evidence that the constructed matrices together with limits coincide, we define dpXq :“ dpXq “ dpXq.Renyi´ in recovery algorithms such as l1-norm minimization provide a his paper [3] proved that if the random variable X is discrete, low-complexity Compressed Sensing and recovery procedure continuos, or a mixture thereof, the upper and the lower for the sparse signals. The use of polarization techniques for RID are equal, thus, dpXq is well-defined. He also provided Compressed Sensing was also investigated independently in an example of a singular random variable for which these [11], approaching noiseless Compressed Sensing via a duality two limits do not coincide. Apart from being an information with analog channel coding. measure, the RID appears as the fundamental operational limit A. Notation This paper is the updated version of the paper [1], which was partially We use R for the reals, R` for the positive reals, Z for the presented in ISIT 2013, Istanbul, Turkey [2]. Saeid Haghighatshoar is with the Communications and Information Theory integers, Z` for the set of positive integers, and N for the set of Group, Technische Universitat¨ Berlin ([email protected]). strictly positive integers. We denote sets by calligraphic letters Emmanuel Abbe has a joint position in Applied and Computational such as A and their cardinality by |A|. We use capital letters Mathematics and Electrical Engineering at Princeton University, New Jersey, USA ([email protected]). This work was started when both the authors for random variables and small letters for their realizations, were with the Information Processing Group (IPG), EPFL, Switzerland. e.g., x is a realization of the random variable X. We denote 2 n the distribution of a random variable X by pX . For N “ 2 , using a new Entropy Power Inequality (EPI) for integer-valued we denote by HN the standard Hadamard matrix of order N. random variables [28], a novel polarization result was proved We use rns for the set of integers t1; 2; : : : ; nu. We denote by for integer-valued sources under the conventional arithmetic j T Xi the column vector rXi;Xi`1;:::;Xjs , where the vector over Z. The result was used to construct deterministic is empty when i ¡ j. Vectors are denoted by boldface letters, partial Hadamard matrices for almost lossless encoding of n n e.g., X “ X1 and x “ x1 denote an n-dim vector of random integer-valued signals with a vanishing measurement rate of variables and its realization. For two sequences f; h : Ñ , opNq for large block-lengths N. The more general case of : N R we say fq “ hq if and only if real-valued sources, however, was left open in [12]. In this paper, we extend this result to real-valued sources and provide fq ´ hq lim “ 0: (3) a polarization theory for infinite-alphabet signals. qÑ8 log2pqq We denote matrices by capital letters, e.g., A. For an m ˆ n II.R ENYI´ INFORMATION DIMENSION matrix A, and a subset of its columns C Ď rns, we denote by Let X be a random variable with a probability distribution AC the m ˆ |C| submatrix of A obtained by selecting those pX over R. The upper and the lower RID of this random columns of A belonging to C. In a similar way, we denote by variable were defined in (1) and (2) respectively. Let m P N ArRs the |R| ˆ n submatrix obtained by selecting the rows and suppose X P r0; 1s almost surely. It is not difficult to see of A belonging to R Ď rms. For two matrices A and B, we that if X “ 0:X1X2 ::: is the m-ary expansion of the random denote by rA; Bs a matrix obtained by putting the rows of A k variable X with Xi P t0; 1; : : : ; m ´ 1u, then for q “ m , we on top of the rows of B and by rA; Bs the matrix obtained by have HpxXyqq “ HpX1;X2;:::;Xkq, where H denotes the putting the columns of B to the right of the columns of A, discrete entropy in basis m. From (1) and (2), we have provided that the resulting matrices are well-defined. dpXq ¤ H ptX u8 q ¤ H8ptX u8 q ¤ dpXq; B. Reminder on Polar Codes 8 i i“1 i i“1 (4) HpX1;X2;:::;Xkq Polar codes were introduced by Arikan in his seminal paper where H8 “ lim infkÑ8 k denotes the lower 8 [15]. They are the first class of efficient codes that provably entropy rate of the stochastic process tXiui“1 (with a similar achieve channel capacity on all binary input symmetric expression for H8 by replacing lim inf with lim sup). As channels. Recent research on polar codes has illustrated a special case, when X is uniformly distributed over r0; 1s, 8 their theoretical optimality for other classical problems in the random variables tXiui“1 are i.i.d. each having a uniform information theory such as lossless and lossy source coding distribution over t0; 1; : : : ; m ´ 1u. Thus, the upper and lower [20, 21], coding over multiple-access channels (MAC) [22], RID are equal to dpXq “ dpXq “ 1. Also, dpXq “ dpXq “ 0 Wyner-Ziv and Gelfand-Pinsker problem [23], and coding for for any discrete random variable X with HpXq ă 8. secrecy over the wiretap channel [24, 25]. By Lebesgue decomposition theorem [29], any probability The underlying structure behind all these applications of distribution pX over R can be written as a convex combination polar codes is the polarization phenomenon. To explain briefly, of a continuous part pc, a singular part ps, and a discrete part we will mainly focus on the source coding aspect, which is pd (with the latter two being singular with respect to Lebesgue N n more relevant to our work.