Unreliable and Resource-Constrained Decoding by Lav R

Unreliable and Resource-Constrained Decoding by Lav R. Varshney B.S., Electrical and Computer Engineering Cornell University, 2004 S.M., Electrical Engineering and Computer Science Massachusetts Institute of Technology, 2006 Electrical Engineer Massachusetts Institute of Technology, 2008 Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Electrical Engineering and Computer Science at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY June 2010 ⃝c Massachusetts Institute of Technology 2010. All rights reserved. Author............................................................................ Department of Electrical Engineering and Computer Science March 25, 2010 Certified by........................................................................ Vivek K Goyal Esther and Harold Edgerton Career Development Associate Professor of Electrical Engineering and Computer Science Thesis Supervisor Certified by........................................................................ Sanjoy K. Mitter Professor of Electrical Engineering and Engineering Systems Thesis Supervisor Accepted by....................................................................... Terry P. Orlando Graduate Officer 2 Unreliable and Resource-Constrained Decoding by Lav R. Varshney Submitted to the Department of Electrical Engineering and Computer Science on March 25, 2010, in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Electrical Engineering and Computer Science Abstract Traditional information theory and communication theory assume that decoders are noiseless and operate without transient or permanent faults. Decoders are also tradi- tionally assumed to be unconstrained in physical resources like materiel, memory, and energy. This thesis studies how constraining reliability and resources in the decoder limits the performance of communication systems. Five communication problems are investigated. Broadly speaking these are communication using decoders that are wiring cost-limited, that are memory-limited, that are noisy, that fail catastrophically, and that simultaneously harvest information and energy. For each of these problems, fundamental trade-offs between communication system performance and reliability or resource consumption are established. For decoding repetition codes using consensus decoding circuits, the optimal trade- off between decoding speed and quadratic wiring cost is defined and established. Designing optimal circuits is shown to be NP-complete, but is carried out for small circuit size. The natural relaxation to the integer circuit design problem is shown to be a reverse convex program. Random circuit topologies are also investigated. Uncoded transmission is investigated when a population of heterogeneous sources must be categorized due to decoder memory constraints. Quantizers that are optimal for mean Bayes risk error, a novel fidelity criterion, are designed. Human decision making in segregated populations is also studied with this framework. The ratio between the costs of false alarms and missed detections is also shown to fundamentally affect the essential nature of discrimination. The effect of noise on iterative message-passing decoders for low-density parity- check (LDPC) codes is studied. Concentration of decoding performance around its average is shown to hold. Density evolution equations for noisy decoders are derived. Decoding thresholds degrade smoothly as decoder noise increases, and in certain cases, arbitrarily small final error probability is achievable despite decoder noisiness. Precise information storage capacity results for reliable memory systems constructed from unreliable components are also provided. Limits to communicating over systems that fail at random times are established. Communication with arbitrarily small probability of error is not possible, but schemes that optimize transmission volume communicated at fixed maximum message error probabilities are determined. System state feedback is shown not to improve performance. 3 For optimal communication with decoders that simultaneously harvest information and energy, a coding theorem that establishes the fundamental trade-off between the rates at which energy and reliable information can be transmitted over a single line is proven. The capacity-power function is computed for several channels; it is non-increasing and concave. Thesis Supervisor: Vivek K Goyal Title: Esther and Harold Edgerton Career Development Associate Professor of Electrical Engineering and Computer Science Thesis Supervisor: Sanjoy K. Mitter Title: Professor of Electrical Engineering and Engineering Systems 4 Acknowledgments It has been a great pleasure to have studied under Sanjoy Mitter and Vivek Goyal for the past several years and it is a high honor that I will be known as their student for time hereafter. Not only did they provide a warm and supportive environment in which to develop expertise, but also encouraged `walking around'|both physically and metaphysically|so as to expose me to a wide range of epistemic and working cultures; this thesis is a product of both a rarefied home and a collection of ideas encountered elsewhere. Moreover, they have each been tireless advocates on my behalf. The contributions that Vivek and Sanjoy have made to this thesis and to my general education are very much appreciated. I am grateful for the encouragement and suggestions provided by Dave Forney and Vladimir Stojanovic, the other members of my doctoral committee. Research, at least the way I enjoy doing it, is an interactive endeavor. As such, the content of this thesis has been influenced by many besides my doctoral committee. The work in Chapter 3 benefited from conversations with JoséMoura, Mitya Chklovskii, Soummya Kar, Jo~aoXavier, Alex Olshevsky, and Pablo Parrilo. The work in Chapter 4 was carried out in large part with Kush Varshney and was improved through interactions with Alan Willsky, Daron Acemoglu, and Sendhil Mullainathan. The central ideas of Chapter 5 were developed through several enlightening discus- sions with RüdigerUrbanke and Emre Telatar. The initial ideas for the problems studied in Chapters 6 and 7 came to mind when sitting at a synthetic biology bench with Team Biogurt and when taking an STS class with David Mindell, respectively. Figure 7-2 was created with assistance from Justin Dauwels. Although works stemming from collaborations with Julius Kusuma, Vinith Misra, Mitya Chklovskii, Ha Nguyen, and Ram Srinivasan do not appear in this thesis, the interactions have also greatly enhanced my research experience. An important part of graduate school for me was talking to fellow students. Be- sides those already mentioned, I would like to single out Adam Zelinski, John Sun, Demba Ba, Ahmed Kirmani, Dan Weller, Aniruddha Bhargava, and Joong Bum Rhim in STIR; Mukul Agarwal, Peter Jones, Barı¸sNakibo˘glu,Mesrob Ohannessian, and Shashi Borade in LIDS; Urs Niesen, Charles Swannack, and Ashish Khisti in SIA; Pat Kreidl, Sujay Sanghavi, Emily Fox, Walter Sun, Vincent Tan, and Matt Johnson in SSG; Dinkar Vasudevan, Etienne Perron, and Shrinivas Kudekar at EPFL; Krish Eswaran and Bobak Nazer at Berkeley; and Quan Wen at CSHL. Rachel Cohen and Eric Strattman provided unmatched administrative assistance. Friends and family have provided immeasurable joy throughout this journey. Kush, who is not only my brother but also my friend, collaborator, and colleague, has influenced me and this thesis in myriad ways. My parents have given me strength and supported me every step of the way. B • Financial support provided by the National Science Foundation Graduate Re- search Fellowship Program, Grant 0325774, Grant 0836720, and Grant 0729069. 5 6 Contents 1 Introduction 15 1.1 Models and Fundamental Limits in Engineering Theory . 17 1.2 Outline and Contributions . 18 2 Background 25 2.1 The Problem of Reliable Communication . 25 2.2 Bayes Risk and Error Probabilities . 26 2.3 Codes and Optimal Decoders . 28 2.4 Information-Theoretic Limits . 36 3 Infrastructure Costs|Wires 41 3.1 Consensus Decoding of Repetition Codes . 42 3.2 Convergence Speeds of Decoding Circuits . 45 3.3 Wiring Costs of Decoding Circuits . 46 3.4 Functionality-Cost Trade-off . 48 3.5 Optimal Decoding Circuits with Costly Wires . 49 3.5.1 Optimization is NP-Hard . 49 3.5.2 Small Optimal Decoding Circuits . 51 3.5.3 Natural Relaxation is Reverse Convex Minimization . 56 3.6 Random Decoding Circuits . 60 3.7 Discussion . 63 3.A Review of Algebraic Graph Theory . 64 4 Infrastructure Costs|Memory 67 4.1 Population Decoding . 68 4.1.1 Mismatch and Bayes Risk Error . 69 4.2 Optimal Quantization Design . 72 4.2.1 Local Optimality Conditions . 73 4.2.2 Design using the Lloyd-Max Algorithm . 75 4.2.3 Design using Dynamic Programming . 77 4.3 High-Rate Quantization Theory . 78 4.4 Optimal Quantizers . 79 4.5 Implications on Human Decision Making . 84 4.5.1 Multiple Populations . 91 7 4.5.2 Social Discrimination . 92 4.5.3 The Price of Segregation . 95 4.6 Discussion . 98 5 Operation Reliability|Transient Faults 99 5.1 Transient Circuit Faults: Causes and Consequences . 100 5.2 Message-Passing Decoders . 102 5.3 Density Evolution Concentration Results . 103 5.3.1 Restriction to All-One Codeword . 104 5.3.2 Concentration around Ensemble Average . 105 5.3.3 Convergence to the Cycle-Free Case . 106 5.3.4 Density Evolution . 106 5.4 Noisy Gallager A Decoder . 107 5.4.1 Density Evolution Equation . 108 5.4.2 Performance

Load more