Discrete Generalizations of the Nyquist-Shannon Sampling Theorem
Total Page:16
File Type:pdf, Size:1020Kb
DISCRETE SAMPLING: DISCRETE GENERALIZATIONS OF THE NYQUIST-SHANNON SAMPLING THEOREM A DISSERTATION SUBMITTED TO THE DEPARTMENT OF ELECTRICAL ENGINEERING AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY William Wu June 2010 © 2010 by William David Wu. All Rights Reserved. Re-distributed by Stanford University under license with the author. This work is licensed under a Creative Commons Attribution- Noncommercial 3.0 United States License. http://creativecommons.org/licenses/by-nc/3.0/us/ This dissertation is online at: http://purl.stanford.edu/rj968hd9244 ii I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Brad Osgood, Primary Adviser I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Thomas Cover I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. John Gill, III Approved for the Stanford University Committee on Graduate Studies. Patricia J. Gumport, Vice Provost Graduate Education This signature page was generated electronically upon submission of this dissertation in electronic format. An original signed hard copy of the signature page is on file in University Archives. iii Abstract 80, 0< 11 17 æ 80, 1< 5 81, 0< 81, 1< 12 83, 1< 84, 0< 82, 1< 83, 4< 82, 3< 15 7 6 83, 2< 83, 3< 84, 8< 83, 6< 83, 7< 13 3 æ æ æ æ æ æ æ æ æ æ 9 -5 -4 -3 -2 -1 1 2 3 4 5 84, 4< 84, 13< 84, 14< 1 HIS DISSERTATION lays a foundation for the theory of reconstructing signals from T finite-dimensional vector spaces using a minimal number of samples. We call this theory Discrete Sampling. Suppose the signals we wish to reconstruct are drawn from a known k-dimensional subspace of Cn, denoted by Y. Then the two problems that Discrete Sampling seeks to address are: n 1. Interpolating System Problem: Find an index set 2(k) and basis vectors := I ∈ U u : i such that { i ∈ I} f Y : f = f[i]u . ∀ ∈ i i X∈I A pair ( , ) satisfying this equation is called an interpolating system for Y. I U 2. Orthogonal Interpolating System Problem: Find an interpolating system for Y in which the ui’s are orthogonal. iv In the first problem, the pairing of a sampling sequence and a interpolating basis I that can reconstruct all signals in Y is dubbed an interpolating system. These are the U basic objects of interest that we aim to study. The second problem asks for a more refined version of this object, in which the interpolating basis is orthogonal. The Nyquist-Shannon sampling theorem provides an example of an orthogonal interpolating system. There, the 1 1 1 vector space is the Paley-Wiener space Y = − [ , ], the sampling sequence is = Z, F − 2 2 I and the orthogonal basis vectors are = sinc(t i) : i Z . U { − ∈ } In the first half of this dissertation, we provide a simple theoretical framework for iden- tifying and constructing interpolating systems (IS) and orthogonal interpolating systems (OIS) in general finite dimensional vector spaces. This can be thought of as a generaliza- tion of Shannon’s sampling theorem in the finite dimensional setting. Which vector spaces will admit these systems? What algebraic properties must be satisfied? While it can be proven that every vector space has an interpolating system, not all vector spaces have an orthogonal interpolating system. When does an OIS exist, and how can we construct it? In the second half of this dissertation, we investigate interpolating systems in finite- dimensional bandlimited spaces, which are parameterized by their DFT support. We pro- vide algorithms for constructing all orthogonal interpolating systems for such spaces, yield- ing a sampling dictionary. The dictionaries exhibit many patterns, making connections with group theory, number theory, and graph theory, specifically: orbit counting, vanishing sums of roots of unity, primes, modular arithmetic, difference sets, cliques, and perfect graphs. All algorithms described have been implemented and can be run online for further investi- gation at the webMathematica-powered website http://www.discretesampling.com. The remainder of the dissertation deals with smaller topics. We briefly analyze sequency- limited spaces, which are defined by their supports in a Discrete Haar Wavelet Transform domain, and give a classification theorem for all orthogonal interpolating systems in terms of posets and Hasse diagrams. Lastly, we draw connections between our work and coding v theory, and provide thoughts for future research. LINEAR ALGEBRA NUMBER dot products THEORY circulancy projections eigenvalues mod arithmetic COMBINA- coin changing TORICS Vandermonde null space squarefree RREF primality difference sets enumerationnecklacesposets of DISCRETE clique problem SAMPLING perfect graphs adjacency matrix DFT cyclic convolution GRAPH dihedral reproducingwavelets kernel THEORY Polya counting Nyquist-Shannon irreducible poly. vanishing sums of roots of unity HARMONIC ANALYSIS ABSTRACT ALGEBRA vi Acknowledgments ELOW ARE THE MAIN PEOPLE AND ORGANIZATIONS that I would like to thank B for helping me through my graduate school career. Brad Osgood Frank H. and Eva Buck Foundation John Gill Jiehua Chen Thomas Cover Hsin and Frances Wu David Wayne Tillay Ever since my first interaction with him, my principal advisor Brad Osgood has always been extremely supportive, encouraging, and optimistic at all times. Although I initially lacked the maturity to make a dent in the problems he gave me, Brad trained me, and taught me to organize my thoughts in a disciplined fashion. By following his example, keeping meticulous notebooks, and systematically writing to myself in a conversational style, I learned to think about long-term problems in an incremental and patient fashion. In addition, Brad has been extremely generous in sharing his time, his mental energy, and his ideas. I recall once, in hope of deciphering the underlying pattern behind a problem, Brad typed out fifty pages of special cases! Brad also envisioned the foundational theory of Discrete Sampling, inspired by the possibility of sampling more efficiently in medical imaging applications. I must also thank Brad for introducing me to Mathematica, which vii has been crucially helpful for doing research. These lessons in writing, discipline, collabo- ration, and mathematical programming that Brad gave me have deeply changed the way I think. Living up to Brad’s example will be an eternal challenge. I am very appreciative to John Gill for all his support over the years. John gave me guidance both before and after the qualifying exams, and kindly helped me in so many ways, including sharing a lifetime’s worth of research problems, encouraging me to learn Perl, and even recovering data from two broken laptops. I have always been impressed by John’s razor sharp mathematical intuition and computer science know-how, having spent many hours in his office watching him at work. More impressively, John demonstrates how to be both humble and indispensable, and I find that very inspiring. Another personal hero of mine is Thomas Cover, who personifies everything that I once fantasized about becoming as a child – a gambler possessing deep mathematical insights, and also being able to explain them with wit, clarity, style, and tremendous humor. We share a love for clever things. I thank him for helping me get into Stanford, rooting for me during the quals, always offering a haven to share ideas and puzzles on Wednesdays, and for the memorable opportunity to serve as his information theory teaching assistant. I want to thank David Wayne Tillay for inspiring me as a middle school student to pursue math and science. In a town without many opportunities for high academic achieve- ment, “Dr. Tillay” was a shiny anomaly. He gave us sophisticated lab equipment, infused humor into the learning process, and created the only science fair our city ever had. With- out Dr. Tillay, I might have ended up as a starving cartoonist instead of a starving graduate student. I am deeply indebted to the Frank H. and Eva Buck Foundation, which somehow se- lected me as a recipient of its scholarship in 1999, most likely due to an accounting error. My entire tertiary education has been funded through their insane generosity and the stew- ardship of Ms. Gloria Brown. viii I thank a number of researchers in related areas for their consultation and/or giving me public speaking opportunities. Joe Sawada at the University of Guelph was very help- ful in explaining his efficient algorithms for necklace enumeration and identification. Re- searchers who discussed applications with me included Daniel Ensign, Kendall Fruchey, and Daniel Rosenfeld from Stanford’s chemistry department, Daniel M. Spielman from Stanford’s radiological sciences lab, and Babak Hassibi from Caltech’s EE department. Kannan Soundarajan at Stanford lent his expertise in modular arithmetic. I graciously thank Maria Chudnovsky at Columbia University for patiently analyzing the difference graphs with me, and giving me the opportunity to speak in her discrete math seminar. Wal- ter Neumann at Barnard College agreed to an out-of-the-blue discussion with me about cyclotomic polynomials. Sinan Gunturk invited me to give a talk at the Courant Institute, which was very helpful; I also thank Mark Tygert and Ron Peled at the Courant for their insightful discussions. Other academics that invited me to give talks include Julius Smith at CCRMA, Jon Hamkins at JPL, Thierry Klein at Bell Labs, and Amit Singer at Princeton, all providing useful feedback.