A Conversation with Robert Fano, by Sergio Verdú
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Digital Communication Systems 2.2 Optimal Source Coding
Digital Communication Systems EES 452 Asst. Prof. Dr. Prapun Suksompong [email protected] 2. Source Coding 2.2 Optimal Source Coding: Huffman Coding: Origin, Recipe, MATLAB Implementation 1 Examples of Prefix Codes Nonsingular Fixed-Length Code Shannon–Fano code Huffman Code 2 Prof. Robert Fano (1917-2016) Shannon Award (1976 ) Shannon–Fano Code Proposed in Shannon’s “A Mathematical Theory of Communication” in 1948 The method was attributed to Fano, who later published it as a technical report. Fano, R.M. (1949). “The transmission of information”. Technical Report No. 65. Cambridge (Mass.), USA: Research Laboratory of Electronics at MIT. Should not be confused with Shannon coding, the coding method used to prove Shannon's noiseless coding theorem, or with Shannon–Fano–Elias coding (also known as Elias coding), the precursor to arithmetic coding. 3 Claude E. Shannon Award Claude E. Shannon (1972) Elwyn R. Berlekamp (1993) Sergio Verdu (2007) David S. Slepian (1974) Aaron D. Wyner (1994) Robert M. Gray (2008) Robert M. Fano (1976) G. David Forney, Jr. (1995) Jorma Rissanen (2009) Peter Elias (1977) Imre Csiszár (1996) Te Sun Han (2010) Mark S. Pinsker (1978) Jacob Ziv (1997) Shlomo Shamai (Shitz) (2011) Jacob Wolfowitz (1979) Neil J. A. Sloane (1998) Abbas El Gamal (2012) W. Wesley Peterson (1981) Tadao Kasami (1999) Katalin Marton (2013) Irving S. Reed (1982) Thomas Kailath (2000) János Körner (2014) Robert G. Gallager (1983) Jack KeilWolf (2001) Arthur Robert Calderbank (2015) Solomon W. Golomb (1985) Toby Berger (2002) Alexander S. Holevo (2016) William L. Root (1986) Lloyd R. Welch (2003) David Tse (2017) James L. -
Principles of Communications ECS 332
Principles of Communications ECS 332 Asst. Prof. Dr. Prapun Suksompong (ผศ.ดร.ประพันธ ์ สขสมปองุ ) [email protected] 1. Intro to Communication Systems Office Hours: Check Google Calendar on the course website. Dr.Prapun’s Office: 6th floor of Sirindhralai building, 1 BKD 2 Remark 1 If the downloaded file crashed your device/browser, try another one posted on the course website: 3 Remark 2 There is also three more sections from the Appendices of the lecture notes: 4 Shannon's insight 5 “The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point.” Shannon, Claude. A Mathematical Theory Of Communication. (1948) 6 Shannon: Father of the Info. Age Documentary Co-produced by the Jacobs School, UCSD- TV, and the California Institute for Telecommunic ations and Information Technology 7 [http://www.uctv.tv/shows/Claude-Shannon-Father-of-the-Information-Age-6090] [http://www.youtube.com/watch?v=z2Whj_nL-x8] C. E. Shannon (1916-2001) Hello. I'm Claude Shannon a mathematician here at the Bell Telephone laboratories He didn't create the compact disc, the fax machine, digital wireless telephones Or mp3 files, but in 1948 Claude Shannon paved the way for all of them with the Basic theory underlying digital communications and storage he called it 8 information theory. C. E. Shannon (1916-2001) 9 https://www.youtube.com/watch?v=47ag2sXRDeU C. E. Shannon (1916-2001) One of the most influential minds of the 20th century yet when he died on February 24, 2001, Shannon was virtually unknown to the public at large 10 C. -
4 Classical Error Correcting Codes
“Machines should work. People should think.” Richard Hamming. 4 Classical Error Correcting Codes Coding theory is an application of information theory critical for reliable communication and fault-tolerant information storage and processing; indeed, Shannon channel coding theorem tells us that we can transmit information on a noisy channel with an arbitrarily low probability of error. A code is designed based on a well-defined set of specifications and protects the information only for the type and number of errors prescribed in its design. Agoodcode: (i) Adds a minimum amount of redundancy to the original message. (ii) Efficient encoding and decoding schemes for the code exist; this means that information is easily mapped to and extracted from a codeword. Reliable communication and fault-tolerant computing are intimately related to each other; the concept of a communication channel, an abstraction for a physical system used to transmit information from one place to another and/or from one time to another is at the core of communication, as well as, information storage and processing. It should however be clear 355 Basic Concepts Linear Codes Polynomial Codes Basic Concepts Linear Codes Other Codes Figure 96: The organization of Chapter 4 at a glance. that the existence of error-correcting codes does not guarantee that logic operations can be implemented using noisy gates and circuits. The strategies to build reliable computing systems using unreliable components are based on John von Neumann’s studies of error detection and error correction techniques for information storage and processing. This chapter covers topics from the theory of classical error detection and error correction and introduces concepts useful for understanding quantum error correction, the subject of Chapter 5. -
IEEE Information Theory Society Newsletter
IEEE Information Theory Society Newsletter Vol. 53, No.4, December 2003 Editor: Lance C. Pérez ISSN 1059-2362 The Shannon Lecture Hidden Markov Models and the Baum-Welch Algorithm Lloyd R. Welch Content of This Talk what the ‘running variable’ is. The lectures of previous Shannon Lecturers fall into several Of particular use will be the concept of conditional probabil- categories such as introducing new areas of research, resusci- ity and recursive factorization. The recursive factorization tating areas of research, surveying areas identified with the idea says that the joint probability of a collection of events can lecturer, or reminiscing on the career of the lecturer. In this be expressed as a product of conditional probabilities, where talk I decided to restrict the subject to the Baum-Welch “algo- each is the probability of an event conditioned on all previous rithm” and some of the ideas that led to its development. events. For example, let A, B, and C be three events. Then I am sure that most of you are familiar with Markov chains Pr(A ∩ B ∩ C) = Pr(A)Pr(B | A)Pr(C | A ∩ B) and Markov processes. They are natural models for various communication channels in which channel conditions change Using the bracket notation, we can display the recursive fac- with time. In many cases it is not the state sequence of the torization of the joint probability distribution of a sequence of model which is observed but the effects of the process on a discrete random variables: signal. That is, the states are not observable but some func- tions, possibly random, of the states are observed. -
Information Theory and Statistics: a Tutorial
Foundations and Trends™ in Communications and Information Theory Volume 1 Issue 4, 2004 Editorial Board Editor-in-Chief: Sergio Verdú Department of Electrical Engineering Princeton University Princeton, New Jersey 08544, USA [email protected] Editors Venkat Anantharam (Berkeley) Amos Lapidoth (ETH Zurich) Ezio Biglieri (Torino) Bob McEliece (Caltech) Giuseppe Caire (Eurecom) Neri Merhav (Technion) Roger Cheng (Hong Kong) David Neuhoff (Michigan) K.C. Chen (Taipei) Alon Orlitsky (San Diego) Daniel Costello (NotreDame) Vincent Poor (Princeton) Thomas Cover (Stanford) Kannan Ramchandran (Berkeley) Anthony Ephremides (Maryland) Bixio Rimoldi (EPFL) Andrea Goldsmith (Stanford) Shlomo Shamai (Technion) Dave Forney (MIT) Amin Shokrollahi (EPFL) Georgios Giannakis (Minnesota) Gadiel Seroussi (HP-Palo Alto) Joachim Hagenauer (Munich) Wojciech Szpankowski (Purdue) Te Sun Han (Tokyo) Vahid Tarokh (Harvard) Babak Hassibi (Caltech) David Tse (Berkeley) Michael Honig (Northwestern) Ruediger Urbanke (EPFL) Johannes Huber (Erlangen) Steve Wicker (GeorgiaTech) Hideki Imai (Tokyo) Raymond Yeung (Hong Kong) Rodney Kennedy (Canberra) Bin Yu (Berkeley) Sanjeev Kulkarni (Princeton) Editorial Scope Foundations and Trends™ in Communications and Information Theory will publish survey and tutorial articles in the following topics: • Coded modulation • Multiuser detection • Coding theory and practice • Multiuser information theory • Communication complexity • Optical communication channels • Communication system design • Pattern recognition and learning • Cryptology -
IEEE Information Theory Society Newsletter
IEEE Information Theory Society Newsletter Vol. 63, No. 3, September 2013 Editor: Tara Javidi ISSN 1059-2362 Editorial committee: Ioannis Kontoyiannis, Giuseppe Caire, Meir Feder, Tracey Ho, Joerg Kliewer, Anand Sarwate, Andy Singer, and Sergio Verdú Annual Awards Announced The main annual awards of the • 2013 IEEE Jack Keil Wolf ISIT IEEE Information Theory Society Student Paper Awards were were announced at the 2013 ISIT selected and announced at in Istanbul this summer. the banquet of the Istanbul • The 2014 Claude E. Shannon Symposium. The winners were Award goes to János Körner. the following: He will give the Shannon Lecture at the 2014 ISIT in 1) Mohammad H. Yassaee, for Hawaii. the paper “A Technique for Deriving One-Shot Achiev - • The 2013 Claude E. Shannon ability Results in Network Award was given to Katalin János Körner Daniel Costello Information Theory”, co- Marton in Istanbul. Katalin authored with Mohammad presented her Shannon R. Aref and Amin A. Gohari Lecture on the Wednesday of the Symposium. If you wish to see her slides again or were unable to attend, a copy of 2) Mansoor I. Yousefi, for the paper “Integrable the slides have been posted on our Society website. Communication Channels and the Nonlinear Fourier Transform”, co-authored with Frank. R. Kschischang • The 2013 Aaron D. Wyner Distinguished Service Award goes to Daniel J. Costello. • Several members of our community became IEEE Fellows or received IEEE Medals, please see our web- • The 2013 IT Society Paper Award was given to Shrinivas site for more information: www.itsoc.org/honors Kudekar, Tom Richardson, and Rüdiger Urbanke for their paper “Threshold Saturation via Spatial Coupling: The Claude E. -
Network Information Theory
Network Information Theory This comprehensive treatment of network information theory and its applications pro- vides the first unified coverage of both classical and recent results. With an approach that balances the introduction of new models and new coding techniques, readers are guided through Shannon’s point-to-point information theory, single-hop networks, multihop networks, and extensions to distributed computing, secrecy, wireless communication, and networking. Elementary mathematical tools and techniques are used throughout, requiring only basic knowledge of probability, whilst unified proofs of coding theorems are based on a few simple lemmas, making the text accessible to newcomers. Key topics covered include successive cancellation and superposition coding, MIMO wireless com- munication, network coding, and cooperative relaying. Also covered are feedback and interactive communication, capacity approximations and scaling laws, and asynchronous and random access channels. This book is ideal for use in the classroom, for self-study, and as a reference for researchers and engineers in industry and academia. Abbas El Gamal is the Hitachi America Chaired Professor in the School of Engineering and the Director of the Information Systems Laboratory in the Department of Electri- cal Engineering at Stanford University. In the field of network information theory, he is best known for his seminal contributions to the relay, broadcast, and interference chan- nels; multiple description coding; coding for noisy networks; and energy-efficient packet scheduling and throughput–delay tradeoffs in wireless networks. He is a Fellow of IEEE and the winner of the 2012 Claude E. Shannon Award, the highest honor in the field of information theory. Young-Han Kim is an Assistant Professor in the Department of Electrical and Com- puter Engineering at the University of California, San Diego. -
President's Column
IEEE Information Theory Society Newsletter Vol. 53, No. 3, September 2003 Editor: Lance C. Pérez ISSN 1059-2362 President’s Column Han Vinck This message is written after re- tion of the electronic library, many turning from an exciting Informa- universities and companies make tion Theory Symposium in this product available to their stu- Yokohama, Japan. Our Japanese dents and staff members. For these colleagues prepared an excellent people there is no direct need to be technical program in the beautiful an IEEE member. IEEE member- and modern setting of Yokohama ship reduced in general by about city. The largest number of contri- 10% and the Society must take ac- butions was in the field of LDPC tions to make the value of member- coding, followed by cryptography. ship visible to you and potential It is interesting to observe that new members. This is an impor- quantum information theory gets tant task for the Board and the more and more attention. The Jap- Membership Development com- anese hospitality contributed to IT Society President Han Vinck playing mittee. The Educational Commit- the general success of this year’s traditional Japanese drums. tee, chaired by Ivan Fair, will event. In spite of all the problems contribute to the value of member- with SARS, the Symposium attracted approximately 650 ship by introducing new activities. The first action cur- participants. At the symposium we announced Bob rently being undertaken by this committee is the McEliece to be the winner of the 2004 Shannon Award. creation of a web site whose purpose is to make readily The Claude E. -
Ieee Information Theory Society
IEEE INFORMATION THEORY SOCIETY The Information Theory Society is an organization, within the framework of the IEEE, of members with principal professional interests in information theory. All members of the IEEE are eligible for membership in the Society and will receive this TRANSACTIONS upon payment of the annual Society membership fee of $30.00. For information on joining, write to the IEEE at the address below. Member copies of Transactions/Journals are for personal use only. BOARD OF GOVERNORS President Secretary Treasurer Transactions Editor Conference Committee Chair GIUSEPPE CAIRE NATASHA DEVROYE NIHAR JINDAL HELMUT BÖLCSKEI BRUCE HAJEK EE Dept. Dept. ECE Dept. Elec. Eng. Comp. Sci. Dept. IT & EE Dept. EECS and Univ. of Southern California Univ. of Illinois Univ. of Minnesota ETH Zurich Coordinated Science Lab. Los Angeles, CA 90089 USA Chicago, IL 60607 USA Minneapolis, MN 55455 USA Zurich, Switzerland Univ. of Illinois Urbana, IL 61801 USA First Vice President Second Vice President Junior Past President Senior Past President MURIEL MÉDARD GERHARD KRAMER FRANK R. KSCHISCHANG ANDREA J. GOLDSMITH HELMUT BÖLCSKEI (2011) ALEX GRANT (2011) MURIEL MÉDARD (2012) EMINA SOLJANIN (2011) MARTIN BOSSERT (2012) ROLF JOHANNESSON (2012) PRAKASH NARAYAN (2012) DAVID TSE (2013) MAX H. M. COSTA (2012) GERHARD KRAMER (2011) L. PING (2012) ALEXANDER VARDY (2013) MICHELLE EFFROS (2013) J. NICHOLAS LANEMAN (2013) AMIN SHOKROLLAHI (2013) SERGIO VERDÚ (2011) ABBAS EL GAMAL (2011) HANS-ANDREA LOELIGER (2012) PAUL SIEGEL (2011) EMANUELE VITERBO (2013) IEEE TRANSACTIONS ON INFORMATION THEORY HELMUT BÖLCSKEI, Editor-in-Chief Executive Editorial Board G. DAVID FORNEY,JR.SHLOMO SHAMAI (SHITZ)ALEXANDER VARDY SERGIO VERDÚ CYRIL MÉASSON, Publications Editor PREDRAG SPASOJEVIc´, Publications Editor ERDAL ARIKAN ELZA ERKIP NAVIN KASHYAP DANIEL P. -
Sharp Threshold Rates for Random Codes∗
Sharp threshold rates for random codes∗ Venkatesan Guruswami1, Jonathan Mosheiff1, Nicolas Resch2, Shashwat Silas3, and Mary Wootters3 1Carnegie Mellon University 2Centrum Wiskunde en Informatica 3Stanford University September 11, 2020 Abstract Suppose that P is a property that may be satisfied by a random code C ⊂ Σn. For example, for some p 2 (0; 1), P might be the property that there exist three elements of C that lie in some Hamming ball of radius pn. We say that R∗ is the threshold rate for P if a random code of rate R∗ + " is very likely to satisfy P, while a random code of rate R∗ − " is very unlikely to satisfy P. While random codes are well-studied in coding theory, even the threshold rates for relatively simple properties like the one above are not well understood. We characterize threshold rates for a rich class of properties. These properties, like the example above, are defined by the inclusion of specific sets of codewords which are also suitably \symmetric." For properties in this class, we show that the threshold rate is in fact equal to the lower bound that a simple first-moment calculation obtains. Our techniques not only pin down the threshold rate for the property P above, they give sharp bounds on the threshold rate for list-recovery in several parameter regimes, as well as an efficient algorithm for estimating the threshold rates for list-recovery in general. arXiv:2009.04553v1 [cs.IT] 9 Sep 2020 ∗SS and MW are partially funded by NSF-CAREER grant CCF-1844628, NSF-BSF grant CCF-1814629, and a Sloan Research Fellowship. -
Urban Flooding Centre for Neuroscience Solid Waste
Connect ISSN 2454-6232 WITH THE INDIAN INSTITUTE OF SCIENCE Urban Flooding Causes and answers Centre for Neuroscience From brain to behaviour Solid Waste Management February 2016 Volume 3 • Issue 1 Towards zero waste CONTRIBUTORS Nithyanand Rao is a Consultant Editor at the Archives N Apurva Ratan Murty is a PhD student at the Centre for and Publications Cell Neuroscience Sudhi Oberoi is a Project Trainee at the Archives and Soumyajit Bhar is a PhD student at ATREE Publications Cell Akanksha Dixit is a PhD student in the Department of Science Media Center is a joint initiative of IISc and Biochemistry Gubbi Labs Sakshi Gera is a PhD student in the Department of Debaleena Basu is a PhD student in the Centre for Molecular Reproduction, Development and Genetics Neuroscience Disha Mohan is a PhD student in the Molecular Karthik Ramaswamy is the Editor, CONNECT, at the Biophysics Unit Archives and Publications Cell Sayantan Khan is an undergraduate student Manbeena Chawla is a Research Associate at the Centre Ankit Ruhi is a PhD student in the Department of for Infectious Disease Research Mathematics Manu Rajan is a Technical Officer at the Archives and Megha Prakash is a Consultant Editor at the Archives and Publications Cell Publications Cell Nisha Meenakshi is a PhD student in the Department of Raghunath Joshi is a PhD student in the Department of Electrical Engineering Materials Engineering Bharti Dharapuram is a PhD student in the Centre for Sridevi Venkatesan is an undergraduate student Ecological Sciences Navin S is a Master’s student -
Reflections on ``Improved Decoding of Reed-Solomon and Algebraic-Geometric Codes
Reflections on \Improved Decoding of Reed-Solomon and Algebraic-Geometric Codes" Venkatesan Guruswami∗ Madhu Sudany March 2002 n A t-error-correcting code over a q-ary alphabet Fq is a set C ⊆ Fq such that for any received n vector r 2 Fq there is at most one vector c 2 C that lies within a Hamming distance of t from r. The minimum distance of the code C is the minimum Hamming distance between any pair of distinct vectors c1; c2 2 C. In his seminal work introducing these concepts, Hamming pointed out that a code of minimum distance 2t + 1 is a t-error-correcting code. It also pointed out the obvious fact that such a code is not a t0-error-correcting code for any t0 > t. We conclude that a code can correct half as many errors as its distance and no more. The mathematical correctness of the above statements are indisputable, yet the interpretation is quite debatable. If a message encoded with a t-error-correcting code ends up getting corrupted in t0 > t places, the decoder may simply throw it hands up in the air and cite the above paragraph. Or, in an alternate notion of decoding, called list decoding, proposed in the late 1950s by Elias [10] and Wozencraft [43], the decoder could try to output a list of codewords within distance t0 of the received vector. If t0 is not much larger than t and the errors are caused by a probabilistic (non-malicious) channel, then most likely this list would have only one element | the transmitted codeword.