Principles of Communications ECS 332
Total Page：16
File Type：pdf, Size：1020Kb
Load more
Recommended publications
-
Data Compression: Dictionary-Based Coding 2 / 37 Dictionary-Based Coding Dictionary-Based Coding
Dictionary-based Coding already coded not yet coded search buffer look-ahead buffer cursor (N symbols) (L symbols) We know the past but cannot control it. We control the future but... Last Lecture Last Lecture: Predictive Lossless Coding Predictive Lossless Coding Simple and effective way to exploit dependencies between neighboring symbols / samples Optimal predictor: Conditional mean (requires storage of large tables) Affine and Linear Prediction Simple structure, low-complex implementation possible Optimal prediction parameters are given by solution of Yule-Walker equations Works very well for real signals (e.g., audio, images, ...) Efficient Lossless Coding for Real-World Signals Affine/linear prediction (often: block-adaptive choice of prediction parameters) Entropy coding of prediction errors (e.g., arithmetic coding) Using marginal pmf often already yields good results Can be improved by using conditional pmfs (with simple conditions) Heiko Schwarz (Freie Universität Berlin) — Data Compression: Dictionary-based Coding 2 / 37 Dictionary-based Coding Dictionary-Based Coding Coding of Text Files Very high amount of dependencies Affine prediction does not work (requires linear dependencies) Higher-order conditional coding should work well, but is way to complex (memory) Alternative: Do not code single characters, but words or phrases Example: English Texts Oxford English Dictionary lists less than 230 000 words (including obsolete words) On average, a word contains about 6 characters Average codeword length per character would be limited by 1 -
Minimax Redundancy for Markov Chains with Large State Space
Minimax redundancy for Markov chains with large state space Kedar Shriram Tatwawadi, Jiantao Jiao, Tsachy Weissman May 8, 2018 Abstract For any Markov source, there exist universal codes whose normalized codelength approaches the Shannon limit asymptotically as the number of samples goes to infinity. This paper inves- tigates how fast the gap between the normalized codelength of the “best” universal compressor and the Shannon limit (i.e. the compression redundancy) vanishes non-asymptotically in terms of the alphabet size and mixing time of the Markov source. We show that, for Markov sources (2+c) whose relaxation time is at least 1 + √k , where k is the state space size (and c> 0 is a con- stant), the phase transition for the number of samples required to achieve vanishing compression redundancy is precisely Θ(k2). 1 Introduction For any data source that can be modeled as a stationary ergodic stochastic process, it is well known in the literature of universal compression that there exist compression algorithms without any knowledge of the source distribution, such that its performance can approach the fundamental limit of the source, also known as the Shannon entropy, as the number of observations tends to infinity. The existence of universal data compressors has spurred a huge wave of research around it. A large fraction of practical lossless compressors are based on the Lempel–Ziv algorithms [ZL77, ZL78] and their variants, and the normalized codelength of a universal source code is also widely used to measure the compressibility of the source, which is based on the idea that the normalized codelength is “close” to the true entropy rate given a moderate number of samples. -
Mathematics of Data Science
SIAM JOURNAL ON Mathematics of Data Science Volume 2 • 2020 Editor-in-Chief Tamara G. Kolda, Sandia National Laboratories Section Editors Mark Girolami, University of Cambridge, UK Alfred Hero, University of Michigan, USA Robert D. Nowak, University of Wisconsin, Madison, USA Joel A. Tropp, California Institute of Technology, USA Associate Editors Maria-Florina Balcan, Carnegie Mellon University, USA Vianney Perchet, ENSAE, CRITEO, France Rina Foygel Barber, University of Chicago, USA Jonas Peters, University of Copenhagen, Denmark Mikhail Belkin, University of California, San Diego, USA Natesh Pillai, Harvard University, USA Robert Calderbank, Duke University, USA Ali Pinar, Sandia National Laboratories, USA Coralia Cartis, University of Oxford, UK Mason Porter, University of Califrornia, Los Angeles, USA Venkat Chandrasekaran, California Institute of Technology, Maxim Raginsky, University of Illinois, USA Urbana-Champaign, USA Patrick L. Combettes, North Carolina State University, USA Bala Rajaratnam, University of California, Davis, USA Alexandre d’Aspremont, CRNS, Ecole Normale Superieure, Philippe Rigollet, MIT, USA France Justin Romberg, Georgia Tech, USA Ioana Dumitriu, University of California, San Diego, USA C. Seshadhri, University of California, Santa Cruz, USA Maryam Fazel, University of Washington, USA Amit Singer, Princeton University, USA David F. Gleich, Purdue University, USA Marc Teboulle, Tel Aviv University, Israel Wouter Koolen, CWI, the Netherlands Caroline Uhler, MIT, USA Gitta Kutyniok, University of Munich, Germany -
Digital Communication Systems 2.2 Optimal Source Coding
Digital Communication Systems EES 452 Asst. Prof. Dr. Prapun Suksompong [email protected] 2. Source Coding 2.2 Optimal Source Coding: Huffman Coding: Origin, Recipe, MATLAB Implementation 1 Examples of Prefix Codes Nonsingular Fixed-Length Code Shannon–Fano code Huffman Code 2 Prof. Robert Fano (1917-2016) Shannon Award (1976 ) Shannon–Fano Code Proposed in Shannon’s “A Mathematical Theory of Communication” in 1948 The method was attributed to Fano, who later published it as a technical report. Fano, R.M. (1949). “The transmission of information”. Technical Report No. 65. Cambridge (Mass.), USA: Research Laboratory of Electronics at MIT. Should not be confused with Shannon coding, the coding method used to prove Shannon's noiseless coding theorem, or with Shannon–Fano–Elias coding (also known as Elias coding), the precursor to arithmetic coding. 3 Claude E. Shannon Award Claude E. Shannon (1972) Elwyn R. Berlekamp (1993) Sergio Verdu (2007) David S. Slepian (1974) Aaron D. Wyner (1994) Robert M. Gray (2008) Robert M. Fano (1976) G. David Forney, Jr. (1995) Jorma Rissanen (2009) Peter Elias (1977) Imre Csiszár (1996) Te Sun Han (2010) Mark S. Pinsker (1978) Jacob Ziv (1997) Shlomo Shamai (Shitz) (2011) Jacob Wolfowitz (1979) Neil J. A. Sloane (1998) Abbas El Gamal (2012) W. Wesley Peterson (1981) Tadao Kasami (1999) Katalin Marton (2013) Irving S. Reed (1982) Thomas Kailath (2000) János Körner (2014) Robert G. Gallager (1983) Jack KeilWolf (2001) Arthur Robert Calderbank (2015) Solomon W. Golomb (1985) Toby Berger (2002) Alexander S. Holevo (2016) William L. Root (1986) Lloyd R. Welch (2003) David Tse (2017) James L. -
Randomized Lempel-Ziv Compression for Anti-Compression Side-Channel Attacks
Randomized Lempel-Ziv Compression for Anti-Compression Side-Channel Attacks by Meng Yang A thesis presented to the University of Waterloo in fulfillment of the thesis requirement for the degree of Master of Applied Science in Electrical and Computer Engineering Waterloo, Ontario, Canada, 2018 c Meng Yang 2018 I hereby declare that I am the sole author of this thesis. This is a true copy of the thesis, including any required final revisions, as accepted by my examiners. I understand that my thesis may be made electronically available to the public. ii Abstract Security experts confront new attacks on TLS/SSL every year. Ever since the compres- sion side-channel attacks CRIME and BREACH were presented during security conferences in 2012 and 2013, online users connecting to HTTP servers that run TLS version 1.2 are susceptible of being impersonated. We set up three Randomized Lempel-Ziv Models, which are built on Lempel-Ziv77, to confront this attack. Our three models change the determin- istic characteristic of the compression algorithm: each compression with the same input gives output of different lengths. We implemented SSL/TLS protocol and the Lempel- Ziv77 compression algorithm, and used them as a base for our simulations of compression side-channel attack. After performing the simulations, all three models successfully pre- vented the attack. However, we demonstrate that our randomized models can still be broken by a stronger version of compression side-channel attack that we created. But this latter attack has a greater time complexity and is easily detectable. Finally, from the results, we conclude that our models couldn't compress as well as Lempel-Ziv77, but they can be used against compression side-channel attacks. -
Marconi Society - Wikipedia
9/23/2019 Marconi Society - Wikipedia Marconi Society The Guglielmo Marconi International Fellowship Foundation, briefly called Marconi Foundation and currently known as The Marconi Society, was established by Gioia Marconi Braga in 1974[1] to commemorate the centennial of the birth (April 24, 1874) of her father Guglielmo Marconi. The Marconi International Fellowship Council was established to honor significant contributions in science and technology, awarding the Marconi Prize and an annual $100,000 grant to a living scientist who has made advances in communication technology that benefits mankind. The Marconi Fellows are Sir Eric A. Ash (1984), Paul Baran (1991), Sir Tim Berners-Lee (2002), Claude Berrou (2005), Sergey Brin (2004), Francesco Carassa (1983), Vinton G. Cerf (1998), Andrew Chraplyvy (2009), Colin Cherry (1978), John Cioffi (2006), Arthur C. Clarke (1982), Martin Cooper (2013), Whitfield Diffie (2000), Federico Faggin (1988), James Flanagan (1992), David Forney, Jr. (1997), Robert G. Gallager (2003), Robert N. Hall (1989), Izuo Hayashi (1993), Martin Hellman (2000), Hiroshi Inose (1976), Irwin M. Jacobs (2011), Robert E. Kahn (1994) Sir Charles Kao (1985), James R. Killian (1975), Leonard Kleinrock (1986), Herwig Kogelnik (2001), Robert W. Lucky (1987), James L. Massey (1999), Robert Metcalfe (2003), Lawrence Page (2004), Yash Pal (1980), Seymour Papert (1981), Arogyaswami Paulraj (2014), David N. Payne (2008), John R. Pierce (1979), Ronald L. Rivest (2007), Arthur L. Schawlow (1977), Allan Snyder (2001), Robert Tkach (2009), Gottfried Ungerboeck (1996), Andrew Viterbi (1990), Jack Keil Wolf (2011), Jacob Ziv (1995). In 2015, the prize went to Peter T. Kirstein for bringing the internet to Europe. Since 2008, Marconi has also issued the Paul Baran Marconi Society Young Scholar Awards. -
MIMO Wireless Communications Ezio Biglieri, Robert Calderbank, Anthony Constantinides, Andrea Goldsmith, Arogyaswami Paulraj and H
Cambridge University Press 978-0-521-13709-6 - MIMO Wireless Communications Ezio Biglieri, Robert Calderbank, Anthony Constantinides, Andrea Goldsmith, Arogyaswami paulraj and H. Vincent Poor Frontmatter More information MIMO Wireless Communications Multiple-input multiple-output (MIMO) technology constitutes a breakthrough in the design of wireless communication systems, and is already at the core of several wireless standards. Exploiting multi-path scattering, MIMO techniques deliver significant performance enhancements in terms of data transmission rate and interference reduction. This book is a detailed introduction to the analysis and design of MIMO wireless systems. Beginning with an overview of MIMO technology, the authors then examine the fundamental capacity limits of MIMO systems. Transmitter design, including precoding and space–time coding, is then treated in depth, and the book closes with two chapters devoted to receiver design. Written by a team of leading experts, the book blends theoretical analysis with physical insights, and highlights a range of key design challenges. It can be used as a textbook for advanced courses on wireless communications, and will also appeal to researchers and practitioners working on MIMO wireless systems. Ezio Biglieri is a professor in the Department of Technology at the Universitat Pompeu Fabra, Barcelona. Robert Calderbank is a professor in the Departments of Electrical Engineering and Mathematics at Princeton University, New Jersey. Anthony Constantinides is a professor in the Department of Electrical and Electronic Engineering at Imperial College of Science, Technology and Medicine, London. Andrea Goldsmith is a professor in the Department of Electrical Engineering at Stanford University, California. Arogyaswami Paulraj is a professor in the Department of Electrical Engineering at Stanford University, California. -
![Arxiv:1306.1586V4 [Quant-Ph]](https://docslib.b-cdn.net/cover/3349/arxiv-1306-1586v4-quant-ph-643349.webp)
Arxiv:1306.1586V4 [Quant-Ph]
Strong converse for the classical capacity of entanglement-breaking and Hadamard channels via a sandwiched R´enyi relative entropy Mark M. Wilde∗ Andreas Winter†‡ Dong Yang†§ May 23, 2014 Abstract A strong converse theorem for the classical capacity of a quantum channel states that the probability of correctly decoding a classical message converges exponentially fast to zero in the limit of many channel uses if the rate of communication exceeds the classical capacity of the channel. Along with a corresponding achievability statement for rates below the capacity, such a strong converse theorem enhances our understanding of the capacity as a very sharp dividing line between achievable and unachievable rates of communication. Here, we show that such a strong converse theorem holds for the classical capacity of all entanglement-breaking channels and all Hadamard channels (the complementary channels of the former). These results follow by bounding the success probability in terms of a “sandwiched” R´enyi relative entropy, by showing that this quantity is subadditive for all entanglement-breaking and Hadamard channels, and by relating this quantity to the Holevo capacity. Prior results regarding strong converse theorems for particular covariant channels emerge as a special case of our results. 1 Introduction One of the most fundamental tasks in quantum information theory is the transmission of classical data over many independent uses of a quantum channel, such that, for a fixed rate of communica- tion, the error probability of the transmission decreases to zero in the limit of many channel uses. The maximum rate at which this is possible for a given channel is known as the classical capacity of the channel. -
Channel Coding
1 Channel Coding: The Road to Channel Capacity Daniel J. Costello, Jr., Fellow, IEEE, and G. David Forney, Jr., Fellow, IEEE Submitted to the Proceedings of the IEEE First revision, November 2006 Abstract Starting from Shannon’s celebrated 1948 channel coding theorem, we trace the evolution of channel coding from Hamming codes to capacity-approaching codes. We focus on the contributions that have led to the most significant improvements in performance vs. complexity for practical applications, particularly on the additive white Gaussian noise (AWGN) channel. We discuss algebraic block codes, and why they did not prove to be the way to get to the Shannon limit. We trace the antecedents of today’s capacity-approaching codes: convolutional codes, concatenated codes, and other probabilistic coding schemes. Finally, we sketch some of the practical applications of these codes. Index Terms Channel coding, algebraic block codes, convolutional codes, concatenated codes, turbo codes, low-density parity- check codes, codes on graphs. I. INTRODUCTION The field of channel coding started with Claude Shannon’s 1948 landmark paper [1]. For the next half century, its central objective was to find practical coding schemes that could approach channel capacity (hereafter called “the Shannon limit”) on well-understood channels such as the additive white Gaussian noise (AWGN) channel. This goal proved to be challenging, but not impossible. In the past decade, with the advent of turbo codes and the rebirth of low-density parity-check codes, it has finally been achieved, at least in many cases of practical interest. As Bob McEliece observed in his 2004 Shannon Lecture [2], the extraordinary efforts that were required to achieve this objective may not be fully appreciated by future historians. -
IEEE Information Theory Society Newsletter
IEEE Information Theory Society Newsletter Vol. 59, No. 3, September 2009 Editor: Tracey Ho ISSN 1059-2362 Optimal Estimation XXX Shannon lecture, presented in 2009 in Seoul, South Korea (extended abstract) J. Rissanen 1 Prologue 2 Modeling Problem The fi rst quest for optimal estimation by Fisher, [2], Cramer, The modeling problem begins with a set of observed data 5 5 5 c 6 Rao and others, [1], dates back to over half a century and has Y yt:t 1, 2, , n , often together with other so-called 5 51 c26 changed remarkably little. The covariance of the estimated pa- explanatory data Y, X yt, x1,t, x2,t, . The objective is rameters was taken as the quality measure of estimators, for to learn properties in Y expressed by a set of distributions which the main result, the Cramer-Rao inequality, sets a lower as models bound. It is reached by the maximum likelihood (ML) estima- 5 1 u 26 tor for a restricted subclass of models and asymptotically for f Y|Xs; , s . a wider class. The covariance, which is just one property of u5u c u models, is too weak a measure to permit extension to estima- Here s is a structure parameter and 1, , k1s2 real- valued tion of the number of parameters, which is handled by various parameters, whose number depends on the structure. The ad hoc criteria too numerous to list here. structure is simply a subset of the models. Typically it is used to indicate the most important variables in X. (The traditional Soon after I had studied Shannon’s formal defi nition of in- name for the set of models is ‘likelihood function’ although no formation in random variables and his other remarkable per- such concept exists in probability theory.) formance bounds for communication, [4], I wanted to apply them to other fi elds – in particular to estimation and statis- The most important problem is the selection of the model tics in general. -
Jacob Wolfowitz 1910–1981
NATIONAL ACADEMY OF SCIENCES JACOB WOLFOWITZ 1910–1981 A Biographical Memoir by SHELEMYAHU ZACKS Any opinions expressed in this memoir are those of the author and do not necessarily reflect the views of the National Academy of Sciences. Biographical Memoirs, VOLUME 82 PUBLISHED 2002 BY THE NATIONAL ACADEMY PRESS WASHINGTON, D.C. JACOB WOLFOWITZ March 19, 1910–July 16, 1981 BY SHELEMYAHU ZACKS ACOB WOLFOWITZ, A GIANT among the founders of modern Jstatistics, will always be remembered for his originality, deep thinking, clear mind, excellence in teaching, and vast contributions to statistical and information sciences. I met Wolfowitz for the first time in 1957, when he spent a sab- batical year at the Technion, Israel Institute of Technology. I was at the time a graduate student and a statistician at the building research station of the Technion. I had read papers of Wald and Wolfowitz before, and for me the meeting with Wolfowitz was a great opportunity to associate with a great scholar who was very kind to me and most helpful. I took his class at the Technion on statistical decision theory. Outside the classroom we used to spend time together over a cup of coffee or in his office discussing statistical problems. He gave me a lot of his time, as though I was his student. His advice on the correct approach to the theory of statistics accompanied my development as statistician for many years to come. Later we kept in touch, mostly by correspondence and in meetings of the Institute of Mathematical Statistics. I saw him the last time in his office at the University of Southern Florida in Tampa, where he spent the last years 3 4 BIOGRAPHICAL MEMOIRS of his life. -
Information Theory and Statistics: a Tutorial
Foundations and Trends™ in Communications and Information Theory Volume 1 Issue 4, 2004 Editorial Board Editor-in-Chief: Sergio Verdú Department of Electrical Engineering Princeton University Princeton, New Jersey 08544, USA [email protected] Editors Venkat Anantharam (Berkeley) Amos Lapidoth (ETH Zurich) Ezio Biglieri (Torino) Bob McEliece (Caltech) Giuseppe Caire (Eurecom) Neri Merhav (Technion) Roger Cheng (Hong Kong) David Neuhoff (Michigan) K.C. Chen (Taipei) Alon Orlitsky (San Diego) Daniel Costello (NotreDame) Vincent Poor (Princeton) Thomas Cover (Stanford) Kannan Ramchandran (Berkeley) Anthony Ephremides (Maryland) Bixio Rimoldi (EPFL) Andrea Goldsmith (Stanford) Shlomo Shamai (Technion) Dave Forney (MIT) Amin Shokrollahi (EPFL) Georgios Giannakis (Minnesota) Gadiel Seroussi (HP-Palo Alto) Joachim Hagenauer (Munich) Wojciech Szpankowski (Purdue) Te Sun Han (Tokyo) Vahid Tarokh (Harvard) Babak Hassibi (Caltech) David Tse (Berkeley) Michael Honig (Northwestern) Ruediger Urbanke (EPFL) Johannes Huber (Erlangen) Steve Wicker (GeorgiaTech) Hideki Imai (Tokyo) Raymond Yeung (Hong Kong) Rodney Kennedy (Canberra) Bin Yu (Berkeley) Sanjeev Kulkarni (Princeton) Editorial Scope Foundations and Trends™ in Communications and Information Theory will publish survey and tutorial articles in the following topics: • Coded modulation • Multiuser detection • Coding theory and practice • Multiuser information theory • Communication complexity • Optical communication channels • Communication system design • Pattern recognition and learning • Cryptology