IEEE Information Theory Society
Total Page:16
File Type:pdf, Size:1020Kb

Load more
Recommended publications
-
Data Compression: Dictionary-Based Coding 2 / 37 Dictionary-Based Coding Dictionary-Based Coding
Dictionary-based Coding already coded not yet coded search buffer look-ahead buffer cursor (N symbols) (L symbols) We know the past but cannot control it. We control the future but... Last Lecture Last Lecture: Predictive Lossless Coding Predictive Lossless Coding Simple and effective way to exploit dependencies between neighboring symbols / samples Optimal predictor: Conditional mean (requires storage of large tables) Affine and Linear Prediction Simple structure, low-complex implementation possible Optimal prediction parameters are given by solution of Yule-Walker equations Works very well for real signals (e.g., audio, images, ...) Efficient Lossless Coding for Real-World Signals Affine/linear prediction (often: block-adaptive choice of prediction parameters) Entropy coding of prediction errors (e.g., arithmetic coding) Using marginal pmf often already yields good results Can be improved by using conditional pmfs (with simple conditions) Heiko Schwarz (Freie Universität Berlin) — Data Compression: Dictionary-based Coding 2 / 37 Dictionary-based Coding Dictionary-Based Coding Coding of Text Files Very high amount of dependencies Affine prediction does not work (requires linear dependencies) Higher-order conditional coding should work well, but is way to complex (memory) Alternative: Do not code single characters, but words or phrases Example: English Texts Oxford English Dictionary lists less than 230 000 words (including obsolete words) On average, a word contains about 6 characters Average codeword length per character would be limited by 1 -
Download PDF Call for Papers
2020 IEEE International Symposium on Information Theory Los Angeles, CA, USA |June 21-26, 2020 The Westin Bonaventure Hotel & Suites, Los Angeles Call for Papers Interested authors are encouraged to submit previously unpublished contributions in any area related to information theory, including but not limited to the following topic areas: Topics ➢ Communication and Storage Coding ➢ Distributed Storage ➢ Network Information Theory ➢ Coding Theory ➢ Emerging Applications of IT ➢ Pattern Recognition and ML ➢ Coded and Distributed Computing ➢ Information Theory and Statistics ➢ Privacy in Information Processing ➢ Combinatorics and Information Theory ➢ Information Theory in Biology ➢ Quantum Information Theory ➢ Communication Theory ➢ Information Theory in CS ➢ Shannon Theory ➢ Compressed Sensing and Sparsity ➢ Information Theory in Data Science ➢ Signal Processing ➢ Cryptography and Security ➢ Learning Theory ➢ Source Coding and Data Compression ➢ Detection and Estimation ➢ Network Coding and Applications ➢ Wireless Communication ➢ Deep Learning for Networks ➢ Network Data Analysis Researchers working in emerging fields of information theory or on novel applications of information theory are especially encouraged to submit original findings. The submitted work and the published version are limited to 5 pages in the standard IEEE format, plus an optional extra page containing references only. Information about paper formatting and submission policies can be found on the conference web page, noted below. The paper submission deadline is Sunday, January 12, 2020, at 11:59 PM, Eastern Time (New York, USA). Acceptance notifications will be sent out by Friday, March 27, 2020. We look forward to your participation in ISIT 2020! General Chairs: Salman Avestimehr, Giuseppe Caire, and Babak Hassibi TPC Chairs: Young-Han Kim, Frederique Oggier, Greg Wornell, and Wei Yu https://2020.ieee-isit.org . -
Digital Communication Systems 2.2 Optimal Source Coding
Digital Communication Systems EES 452 Asst. Prof. Dr. Prapun Suksompong [email protected] 2. Source Coding 2.2 Optimal Source Coding: Huffman Coding: Origin, Recipe, MATLAB Implementation 1 Examples of Prefix Codes Nonsingular Fixed-Length Code Shannon–Fano code Huffman Code 2 Prof. Robert Fano (1917-2016) Shannon Award (1976 ) Shannon–Fano Code Proposed in Shannon’s “A Mathematical Theory of Communication” in 1948 The method was attributed to Fano, who later published it as a technical report. Fano, R.M. (1949). “The transmission of information”. Technical Report No. 65. Cambridge (Mass.), USA: Research Laboratory of Electronics at MIT. Should not be confused with Shannon coding, the coding method used to prove Shannon's noiseless coding theorem, or with Shannon–Fano–Elias coding (also known as Elias coding), the precursor to arithmetic coding. 3 Claude E. Shannon Award Claude E. Shannon (1972) Elwyn R. Berlekamp (1993) Sergio Verdu (2007) David S. Slepian (1974) Aaron D. Wyner (1994) Robert M. Gray (2008) Robert M. Fano (1976) G. David Forney, Jr. (1995) Jorma Rissanen (2009) Peter Elias (1977) Imre Csiszár (1996) Te Sun Han (2010) Mark S. Pinsker (1978) Jacob Ziv (1997) Shlomo Shamai (Shitz) (2011) Jacob Wolfowitz (1979) Neil J. A. Sloane (1998) Abbas El Gamal (2012) W. Wesley Peterson (1981) Tadao Kasami (1999) Katalin Marton (2013) Irving S. Reed (1982) Thomas Kailath (2000) János Körner (2014) Robert G. Gallager (1983) Jack KeilWolf (2001) Arthur Robert Calderbank (2015) Solomon W. Golomb (1985) Toby Berger (2002) Alexander S. Holevo (2016) William L. Root (1986) Lloyd R. Welch (2003) David Tse (2017) James L. -
Randomized Lempel-Ziv Compression for Anti-Compression Side-Channel Attacks
Randomized Lempel-Ziv Compression for Anti-Compression Side-Channel Attacks by Meng Yang A thesis presented to the University of Waterloo in fulfillment of the thesis requirement for the degree of Master of Applied Science in Electrical and Computer Engineering Waterloo, Ontario, Canada, 2018 c Meng Yang 2018 I hereby declare that I am the sole author of this thesis. This is a true copy of the thesis, including any required final revisions, as accepted by my examiners. I understand that my thesis may be made electronically available to the public. ii Abstract Security experts confront new attacks on TLS/SSL every year. Ever since the compres- sion side-channel attacks CRIME and BREACH were presented during security conferences in 2012 and 2013, online users connecting to HTTP servers that run TLS version 1.2 are susceptible of being impersonated. We set up three Randomized Lempel-Ziv Models, which are built on Lempel-Ziv77, to confront this attack. Our three models change the determin- istic characteristic of the compression algorithm: each compression with the same input gives output of different lengths. We implemented SSL/TLS protocol and the Lempel- Ziv77 compression algorithm, and used them as a base for our simulations of compression side-channel attack. After performing the simulations, all three models successfully pre- vented the attack. However, we demonstrate that our randomized models can still be broken by a stronger version of compression side-channel attack that we created. But this latter attack has a greater time complexity and is easily detectable. Finally, from the results, we conclude that our models couldn't compress as well as Lempel-Ziv77, but they can be used against compression side-channel attacks. -
Principles of Communications ECS 332
Principles of Communications ECS 332 Asst. Prof. Dr. Prapun Suksompong (ผศ.ดร.ประพันธ ์ สขสมปองุ ) [email protected] 1. Intro to Communication Systems Office Hours: Check Google Calendar on the course website. Dr.Prapun’s Office: 6th floor of Sirindhralai building, 1 BKD 2 Remark 1 If the downloaded file crashed your device/browser, try another one posted on the course website: 3 Remark 2 There is also three more sections from the Appendices of the lecture notes: 4 Shannon's insight 5 “The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point.” Shannon, Claude. A Mathematical Theory Of Communication. (1948) 6 Shannon: Father of the Info. Age Documentary Co-produced by the Jacobs School, UCSD- TV, and the California Institute for Telecommunic ations and Information Technology 7 [http://www.uctv.tv/shows/Claude-Shannon-Father-of-the-Information-Age-6090] [http://www.youtube.com/watch?v=z2Whj_nL-x8] C. E. Shannon (1916-2001) Hello. I'm Claude Shannon a mathematician here at the Bell Telephone laboratories He didn't create the compact disc, the fax machine, digital wireless telephones Or mp3 files, but in 1948 Claude Shannon paved the way for all of them with the Basic theory underlying digital communications and storage he called it 8 information theory. C. E. Shannon (1916-2001) 9 https://www.youtube.com/watch?v=47ag2sXRDeU C. E. Shannon (1916-2001) One of the most influential minds of the 20th century yet when he died on February 24, 2001, Shannon was virtually unknown to the public at large 10 C. -
Marconi Society - Wikipedia
9/23/2019 Marconi Society - Wikipedia Marconi Society The Guglielmo Marconi International Fellowship Foundation, briefly called Marconi Foundation and currently known as The Marconi Society, was established by Gioia Marconi Braga in 1974[1] to commemorate the centennial of the birth (April 24, 1874) of her father Guglielmo Marconi. The Marconi International Fellowship Council was established to honor significant contributions in science and technology, awarding the Marconi Prize and an annual $100,000 grant to a living scientist who has made advances in communication technology that benefits mankind. The Marconi Fellows are Sir Eric A. Ash (1984), Paul Baran (1991), Sir Tim Berners-Lee (2002), Claude Berrou (2005), Sergey Brin (2004), Francesco Carassa (1983), Vinton G. Cerf (1998), Andrew Chraplyvy (2009), Colin Cherry (1978), John Cioffi (2006), Arthur C. Clarke (1982), Martin Cooper (2013), Whitfield Diffie (2000), Federico Faggin (1988), James Flanagan (1992), David Forney, Jr. (1997), Robert G. Gallager (2003), Robert N. Hall (1989), Izuo Hayashi (1993), Martin Hellman (2000), Hiroshi Inose (1976), Irwin M. Jacobs (2011), Robert E. Kahn (1994) Sir Charles Kao (1985), James R. Killian (1975), Leonard Kleinrock (1986), Herwig Kogelnik (2001), Robert W. Lucky (1987), James L. Massey (1999), Robert Metcalfe (2003), Lawrence Page (2004), Yash Pal (1980), Seymour Papert (1981), Arogyaswami Paulraj (2014), David N. Payne (2008), John R. Pierce (1979), Ronald L. Rivest (2007), Arthur L. Schawlow (1977), Allan Snyder (2001), Robert Tkach (2009), Gottfried Ungerboeck (1996), Andrew Viterbi (1990), Jack Keil Wolf (2011), Jacob Ziv (1995). In 2015, the prize went to Peter T. Kirstein for bringing the internet to Europe. Since 2008, Marconi has also issued the Paul Baran Marconi Society Young Scholar Awards. -
Arxiv:2010.10473V2 [Cs.LG] 1 Feb 2021
Proceedings of Machine Learning Research vol 134:1{22, 2021 Regret-optimal control in dynamic environments Gautam Goel [email protected] California Insititute of Technology Babak Hassibi [email protected] California Insititute of Technology Abstract We consider control in linear time-varying dynamical systems from the perspective of regret minimization. Unlike most prior work in this area, we focus on the problem of designing an online controller which minimizes regret against the best dynamic sequence of control actions selected in hindsight (dynamic regret), instead of the best fixed controller in some specific class of controllers (policy regret). This formulation is attractive when the environ- ment changes over time and no single controller achieves good performance over the entire time horizon. We derive the state-space structure of the regret-optimal controller via a novel reduction to H1 control and present a tight data-dependent bound on its regret in terms of the energy of the disturbance. Our results easily extend to the model-predictive setting where the controller can anticipate future disturbances and to settings where the controller only affects the system dynamics after a fixed delay. We present numerical experiments which show that our regret-optimal controller interpolates between the performance of the H2-optimal and H1-optimal controllers across stochastic and adversarial environments. Keywords: dynamic regret, robust control 1. Introduction The central question in control theory is how to regulate the behavior of an evolving system with state x that is perturbed by a disturbance w by dynamically adjusting a control action u. Traditionally, this question has been studied in two distinct settings: in the H2 setting, we assume that the disturbance w is generated by a stochastic process and seek to select the control u so as to minimize the expected control cost, whereas in the H1 setting we assume the noise is selected adversarially and instead seek to minimize the worst-case control cost. -
INFORMATION and CODING THEORY Exercise Sheet 4
INFO-H-422 2017-2018 INFORMATION AND CODING THEORY Exercise Sheet 4 Exercise 1. Lempel-Ziv code. (a) Consider a source with alphabet A, B, C,_ . Encode the sequence AA_ABABBABC_ABABC with the Lempel-Ziv code. What is the numberf of bitsg necessary to transmit the encoded sequence? What happens at the last ABC? What would happen if the sequence were AA_ABABBABC_ACABC? (b) Consider a source with alphabet A, B . Encode the sequence ABAAAAAAAAAAAAAABB with the Lempel-Ziv code. Give the numberf of bitsg necessary to transmit the encoded sequence and compare it with a naive encoding. (c) Consider a source with alphabet A, B . Encode the sequence ABAAAAAAAAAAAAAAAAAAAA with the Lempel-Ziv code. Give the numberf ofg bits necessary to transmit the sequence and compare it with a naive encoding. (d) The sequence below is encoded by the Lempel-Ziv code. Reconstruct the original sequence. (0 , A), (1 , B), (2 , C), (0 , _), (2 , B), (0 , B), (6 , B), (7 , B), (0 , .). Exercise 2. We are given a set of n objects. Each object in this set can either be faulty or intact. The random variable Xi takes the value 1 if the i-th object is faulty and 0 otherwise. We assume that the variables X1, X2, , Xn are independent, with Prob Xi = 1 = pi and p1 > p2 > > pn > 1=2. The problem is to determine··· the set of all faulty objects withf an optimalg method. ··· (a) How to find the optimum sequence of yes/no-questions that identifies all faulty objects? (b) – What is the last question that has to be asked in the worst case (i.e., in the case when one has to ask the most number of questions)? – Which two sets can be distinguished with this question? Exercise 3. -
Information Theory and Statistics: a Tutorial
Foundations and Trends™ in Communications and Information Theory Volume 1 Issue 4, 2004 Editorial Board Editor-in-Chief: Sergio Verdú Department of Electrical Engineering Princeton University Princeton, New Jersey 08544, USA [email protected] Editors Venkat Anantharam (Berkeley) Amos Lapidoth (ETH Zurich) Ezio Biglieri (Torino) Bob McEliece (Caltech) Giuseppe Caire (Eurecom) Neri Merhav (Technion) Roger Cheng (Hong Kong) David Neuhoff (Michigan) K.C. Chen (Taipei) Alon Orlitsky (San Diego) Daniel Costello (NotreDame) Vincent Poor (Princeton) Thomas Cover (Stanford) Kannan Ramchandran (Berkeley) Anthony Ephremides (Maryland) Bixio Rimoldi (EPFL) Andrea Goldsmith (Stanford) Shlomo Shamai (Technion) Dave Forney (MIT) Amin Shokrollahi (EPFL) Georgios Giannakis (Minnesota) Gadiel Seroussi (HP-Palo Alto) Joachim Hagenauer (Munich) Wojciech Szpankowski (Purdue) Te Sun Han (Tokyo) Vahid Tarokh (Harvard) Babak Hassibi (Caltech) David Tse (Berkeley) Michael Honig (Northwestern) Ruediger Urbanke (EPFL) Johannes Huber (Erlangen) Steve Wicker (GeorgiaTech) Hideki Imai (Tokyo) Raymond Yeung (Hong Kong) Rodney Kennedy (Canberra) Bin Yu (Berkeley) Sanjeev Kulkarni (Princeton) Editorial Scope Foundations and Trends™ in Communications and Information Theory will publish survey and tutorial articles in the following topics: • Coded modulation • Multiuser detection • Coding theory and practice • Multiuser information theory • Communication complexity • Optical communication channels • Communication system design • Pattern recognition and learning • Cryptology -
Network Information Theory
Network Information Theory This comprehensive treatment of network information theory and its applications pro- vides the first unified coverage of both classical and recent results. With an approach that balances the introduction of new models and new coding techniques, readers are guided through Shannon’s point-to-point information theory, single-hop networks, multihop networks, and extensions to distributed computing, secrecy, wireless communication, and networking. Elementary mathematical tools and techniques are used throughout, requiring only basic knowledge of probability, whilst unified proofs of coding theorems are based on a few simple lemmas, making the text accessible to newcomers. Key topics covered include successive cancellation and superposition coding, MIMO wireless com- munication, network coding, and cooperative relaying. Also covered are feedback and interactive communication, capacity approximations and scaling laws, and asynchronous and random access channels. This book is ideal for use in the classroom, for self-study, and as a reference for researchers and engineers in industry and academia. Abbas El Gamal is the Hitachi America Chaired Professor in the School of Engineering and the Director of the Information Systems Laboratory in the Department of Electri- cal Engineering at Stanford University. In the field of network information theory, he is best known for his seminal contributions to the relay, broadcast, and interference chan- nels; multiple description coding; coding for noisy networks; and energy-efficient packet scheduling and throughput–delay tradeoffs in wireless networks. He is a Fellow of IEEE and the winner of the 2012 Claude E. Shannon Award, the highest honor in the field of information theory. Young-Han Kim is an Assistant Professor in the Department of Electrical and Com- puter Engineering at the University of California, San Diego. -
Linear Estimation in Krein Spaces-- Part II: Applications
34 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL 41, NO. 1, JANUARY 1996 stimation in Krein Spaces- art 11: Applications Babak Hassibi, Ali H. Sayed, Member, IEEE, and Thomas Kailath, Fellow, IEEE Abstract-We show that several interesting problems in Rw - by constructing the corresponding Krein-space "stoc filtering, quadratic game theory, and risk sensitive control and problems for which the Kalman-filter solutions can be written estimation follow as special cases of the Krein-space linear esti- down immediately; moreover, the conditions for a minimum mation theory developed in [l]. We show that a11 these problems can be cast into the problem of calculating the stationary point can also be expressed in terms of quantities easily related to the of certain second-order forms, and that by considering the ap- basic Riccati equations of the Kalman filter. This approach also propriate state space models and error Gramians, we can use the explains the many similarities between, say, the H" solutions ~re~n-spa~eestimation theory to calculate the stationary points and ~e classical LQ solutions and in addition marks out their and study their properties. The approach discussed here allows key differences. for interesting generalizations, such as finite memory adaptive filtering with varying sliding patterns. The paper is organized as follows. In Section I1 we introduce the H" estimation problem, state the conventional solution, and discuss its similarities with and differences from the I. INTRODUCTION Conventional Kalman filter. In Section I11 we reduce the H" LASSICAL results in linear least-squares estimation and estimation problem to guaranteeing the positivity of a certain Kalman filtering are based on an LZ or HZ criterion indefinite quadratic form. -
Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalization
Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalization Navid Azizan1, Sahin Lale2, Babak Hassibi2 1 Department of Computing and Mathematical Sciences 2 Department of Electrical Engineering California Institute of Technology {azizan,alale,hassibi}@caltech.edu Abstract Most modern learning problems are highly overparameterized, meaning that there are many more parameters than the number of training data points, and as a result, the training loss may have infinitely many global minima (parameter vectors that perfectly interpolate the training data). Therefore, it is important to understand which interpolating solutions we converge to, how they depend on the initialization point and the learning algorithm, and whether they lead to different generalization performances. In this paper, we study these questions for the family of stochastic mirror descent (SMD) algorithms, of which the popular stochastic gradient descent (SGD) is a special case. Our contributions are both theoretical and experimental. On the theory side, we show that in the overparameterized nonlinear setting, if the initialization is close enough to the manifold of global minima (something that comes for free in the highly overparameterized case), SMD with sufficiently small step size converges to a global minimum that is approximately the closest one in Bregman divergence. On the experimental side, our extensive experiments on standard datasets and models, using various initializations, various mirror descents,