itNL1208.qxd 11/26/08 9:09 AM Page 1 IEEE Information Theory Society Newsletter Vol. 58, No. 4, December 2008 Editor: Daniela Tuninetti ISSN 1059-2362 Source Coding and Simulation XXIX Shannon Lecture, presented at the 2008 IEEE International Symposium on Information Theory, Toronto Canada Robert M. Gray Prologue Source coding/compression/quantization A unique aspect of the Shannon Lecture is the daunting fact that the lecturer has a year to prepare for (obsess over?) a single lec- source reproduction ture. The experience begins with the comfort of a seemingly infi- { } - bits- - Xn encoder decoder {Xˆn} nite time horizon and ends with a relativity-like speedup of time as the date approaches. I early on adopted a few guidelines: I had (1) a great excuse to review my more than four decades of Simulation/synthesis/fake process information theoretic activity and historical threads extending simulation even further back, (2) a strong desire to avoid repeating the top- - - ics and content I had worn thin during 2006–07 as an interconti- random bits coder {X˜n} nental itinerant lecturer for the Signal Processing Society, (3) an equally strong desire to revive some of my favorite topics from the mid 1970s—my most active and focused period doing Figure 1: Source coding and simulation. unadulterated information theory, and (4) a strong wish to add something new taking advantage of hindsight and experience, The goal of source coding [1] is to communicate or transmit preferably a new twist on some old ideas that had not been pre- the source through a constrained, discrete, noiseless com- viously fully exploited. The result was a new look at an old munication or storage channel to a receiver, whose job is to problem—the connections between classic Shannon source cod- reconstruct or reproduce the original source as accurately as ing subject to a fidelity criterion and the “simulation problem” possible given constraints on the channel. As is common, we formulated as part of a proof of the source coding theorem for will focus on the situation where the channel sends binary trellis encoding using a time-invariant trellis. symbols or bits of information, say R bits per input symbol. For simplicity the special case of R = 1 will be emphasized The problem and several questions are naturally motivated by here. The goal of simulation [2] is to simulate the source the contiguous diagrams of the two systems in Figure 1, and using a system of a particular structure, for example using a the discussion provides a tour of operational distortion-rate sequence of independent fair coin flips to drive a time- functions, block and sliding-block coding, process distortion invariant or stationary coder to produce a random process and distance measures, and some fundamental shared ideas with distribution closely matching that of the original source of information theory and ergodic theory. X in some precise sense. There are several reasons for con- sidering a stationary operation on the input bits rather than Source Coding and Simulation some form of block coding: Both systems shown in Figure 1 concern an information • A stationary mapping of coin flips (or any iid source) source X ={Xn; n ∈ Z}, a discrete-time stationary ergodic will produce a process that is stationary and ergodic, as random process described by a process distribution μ. The is the target source X we are trying to emulate. A block random variables Xn take values in an alphabet AX, which mapping will not be stationary or ergodic; it will be might be discrete, continuous, or mixed. The focus here is on block stationary, but not necessarily block ergodic. Even stationary and ergodic, but many of the results generalize to if “stationarized” by a random start, the resulting nonergodic sources (using the ergodic decomposition) and to process will retain in general periodicities not present in asymptotically mean stationary sources (sources for which the original source. sample averages converge). continued on page 5 itNL1208.qxd 11/26/08 9:09 AM Page 2 2 From the Editor Daniela Tuninetti Dear IT society members, by Emmanuel Candes and Terence Tao. The winner of the 2008 Joint IT/ComSoc Paper Award is a Communications Transactions paper I trust you all had a good start of the fall on Accumulate-Repeat-Accumulate Codes by Aliazam Abbasfar, semester. By the time this issue arrives on Dariush Divsalar and Kung Yao. Congratulations to all the authors your desk, you will already be preparing on their paper award. The summaries of the paper awards will be finals and planning winter vacation. followed by the summaries of the plenary talks: “Golay, Heisenberg Although summer may appear long gone, and Weyl” by A. Robert Calderbank, “Building Quantum this issue will bring back warm memories of Computers” by Emanuel H. Knill, and “Randomness - A ISIT 2008 in Toronto, Canada. I would like to Computational Complexity View” by Avi Wigderson. express our thanks to co-chairs Frank R. Kschischang and En-hui Yang, and their You will also enjoy an update on the activities of the student committee chaired by Aylin team, for the organization of a great ISIT. Yener, on the newly formed outreach committee chaired by Muriel Medard, and an account Before I give you a taste of what you will find of the International Symposium on Advances in Communications on the occasion of Vijay K. in this issue, I would also like to thank our Bhargava's 60th birthday. Congratulations Vijay on your birthday and all your achievements. president Dave Forney, whose term will end in December 2008, and welcome our new Last but not the least, you will find our NSF guest column by Program Manager Sirin president, Andrea Goldsmith, whose term Tekinay. This will be Sirin's last column as her term at NSF ended in September 2008. She will start in January 2009. Please join me in did a terrific job as champion for our community. congratulating Dave and welcoming Andrea. Please help to make the Newsletter as interesting and informative as possible by offer- This issue opens with a summary of Robert M. ing suggestions and contributing news. The deadlines for the next few issues of the Gray's Shannon Lecture “Source Coding and Newsletter are as follows: Simulation.” Then our regular columns by our president Dave Forney, our historian Anthony Issue Deadline Ephremides, and our creative puzzle maker March 2009 January 10, 2009 Sol Golomb will follow. You will then read first June 2009 April 10, 2009 hand from Frank R. Kschischang and En-hui Yang about the major ISIT 2008 events. Electronic submission in Ascii, LaTeX and Word formats is encouraged. Potential authors should not worry about layout and fonts of their contributions. Our IEEE professionals take You will find the reflections on the paper care of formatting the source files according to the IEEE Newsletter style. Electronic photos and award announced at the Award Luncheon in graphs should be in high resolution and sent in as separate file. Toronto. The winners of the 2008 IT Paper Award are two 2006 IT Transactions papers on I may be reached at the following email address: [email protected]. Compressed Sensing by David Donoho, and I wish everyone happy winter holidays and all the best for the New Year, IEEE Daniela Tuninetti Information Theory Society Newsletter Table of Contents IEEE Information Theory Society Newsletter From the Editor . .2 (USPS 360-350) is published quarterly by the President’s Column . .3 Information Theory Society of the Institute of Call for Nominations . .3 Electrical and Electronics Engineers, Inc. The Historian’s Column . .4 Workshop Report: ISIT 2008 . .12 Headquarters: 3 Park Avenue, 17th Floor, Reflections on Compressed Sensing, by E. Candes and T. Tao . .14 New York, NY 10016-5997. Reflections on Compressed Sensing, by D. Donoho . .18 Cost is $1.00 per member per year (included Reflections on Accumulate Repeat Accumulate Codes . .24 in Society fee) for each member of the Golay, Heisenberg and Weyl . .28 Information Theory Society. Printed in the Building Quantum Computers . .32 U.S.A. Periodicals postage paid at New York, NY and at additional mailing offices. Randomness—A Computational Complexity Perspective . .36 Recent Activities of the IT Student Committee . .41 Postmaster: Send address changes to IEEE IT Society’s Outreach Efforts . .42 Information Theory Society Newsletter, Workshop Report: Vijayfest . .43 IEEE, 445 Hoes Lane, Piscataway, NJ 08854. In Memoriam of Marvin K. Simon . .45 © 2008 IEEE. Information contained in this Golomb’s Puzzle Column: Proofs by Dissections, Tiling, Etc. .46 newsletter may be copied without permission Golomb’s Puzzle Column: Some Problems About Primes Solutions . .47 provided that the copies are not made or dis- Guest Column: News from the Communications Program at the National Science Foundation . .48 tributed for direct commercial advantage, and the title of the publication and its date appear. Call for Papers . .50 Conference Calendar . .52 IEEE Information Theory Society Newsletter December 2008 itNL1208.qxd 11/26/08 9:09 AM Page 3 3 President’s Column David Forney It has been a great honor and privilege to be does not speak to today’s real needs, or the future’s. Yet President of the IEEE Information Theory Society work proceeds that will have the cumulative effect over during 2008. It is a classy and successful organi- the course of time of completely changing the outlines zation, full of enjoyable colleagues, which always of the field, its practice, and our understanding of it. aims for the highest standards. Which work falls in which category is not always immediately clear. In January 1973, when I was Editor of the IEEE Transactions on Information Theory, I published Without great daring, one can make a few predictions, the Editorial below in connection with a series of valid now or any time.