Foundations of Signal Processing Martin Vetterli, Jelena Kovačević and Vivek K Goyal Frontmatter More Information

Cambridge University Press 978-1-107-03860-8 - Foundations of Signal Processing Martin Vetterli, Jelena Kovačević and Vivek K Goyal Frontmatter More information

Foundations of Signal Processing

This comprehensive and engaging textbook introduces the basic principles and techniques of signal processing, from the fundamental ideas of signals and systems theory to real-world applications. • Introduces students to the powerful foundations of modern signal processing, including the basic geometry of Hilbert space, the mathematics of Fourier transforms, and essentials of sampling, interpolation, approximation, and compression. • Discusses issues in real-world use of these tools such as effects of truncation and quantization, limitations on localization, and computational costs. • Includes over 160 homework problems and over 220 worked examples, speciﬁcally designed to test and expand students’ understanding of the fundamentals of signal processing. • Accompanied by extensive online materials designed to aid learning, including Mathematica resources and interactive demonstrations.

MARTIN VETTERLI is a Professor of Computer and Communication Sciences at the Ecole´ Polytechnique Fed´ erale´ de Lausanne, and the President of the Swiss National Science Foundation. He has formerly held positions at Columbia University and the University of California, Berkeley, and has received the IEEE Signal Processing Society Technical Achievement Award (2001) and Society Award (2010). He is a Fellow of the ACM, EURASIP, and the IEEE, and is a Thomson Reuters Highly Cited Researcher in Engineering.

JELENA KOVACEVIˇ C´ is the David Edward Schramm Professor and Head of Electrical and Computer Engineering, and a Professor of Biomedical Engineering, at Carnegie Mellon University. She has been awarded the Belgrade October Prize (1986), the E. I. Jury Award (1991) from Columbia University, and the 2010 Philip L. Dowd Fellowship at Carnegie Mellon University. She is a former Editor-in-Chief of IEEE Transactions on Image Processing and a Fellow of the IEEE and EURASIP.

VIVEK KGOYAL is an Assistant Professor of Electrical and Computer Engineering at Boston University, and a former Esther and Harold E. Edgerton Associate Professor of Elec- trical Engineering at the Massachusetts Institute of Technology. He has been awarded the IEEE Signal Processing Society Magazine Award (2002) and the Eliahu Jury Award (1998) from the University of California, Berkeley, for outstanding achievement in systems, communications, control, and signal processing. He is a Fellow of the IEEE.

© in this web service Cambridge University Press www.cambridge.org Cambridge University Press 978-1-107-03860-8 - Foundations of Signal Processing Martin Vetterli, Jelena Kovačević and Vivek K Goyal Frontmatter More information

“Foundations of Signal Processing by Martin Vetterli, Jelena Kovacevic, and Vivek K Goyal lives up to its title by providing a thorough tour of the subject matter based on selected tools from real analysis which allow sufficient generality to develop the foundations of the classical Fourier methods along with modern wavelet approaches. A key distinction of the book is the use of Hilbert space ideas to provide a geometric interpretation and intuition that enhances both the classic and modern approaches by providing a unified view of their similarities and relative merits in the most important special cases. Many of the specific examples of signal processing considered can be viewed as examples of projections onto subspaces, which yield immediate properties and descriptions from the underlying fundamentals. The development is both pedagogically and theoretically sound, proceeding from the underlying mathematics through discrete-time systems to the more complicated continuous time systems into a wonderfully general and enlightening treatment of sampling and interpolation operations connecting discrete and continuous time. All of the important signal classes are considered and their basic properties and interrelations developed and summarized. The book then develops important topics not ordinarily found in signal processing texts – the accuracy of approximations involving the truncation of series expansions and the quantization of series coefficients, and the localization of signals in the time-frequency plane. The completeness of the book results in a lengthy volume of roughly 800 pages, but it is easy to navigate to extract portions of interest while saving the many byways and special topics for future reference. The chapter introductions are particularly good at setting the stage in a simple but informative context and then sketching the details to come for the remainder of the chapter. Each chapter closes with a ‘Chapter at a glance’ section highlighting the primary ideas and results. The book will be a welcome addition to the library of students, practitioners, and researchers in signal processing for learning, reviewing, and referencing the broad array of tools and properties now available to analyze, synthesize, and understand signal processing systems.” Robert M. Gray, Stanford University and Boston University

“Finally a wonderful and accessible book for teaching modern signal processing to undergraduate students.” Stephane´ Mallat, Ecole´ Normale Superieure´

“This is a major book about a serious subject – the combination of engineering and mathematics that goes into modern signal processing: discrete time, continuous time, sampling, ﬁltering, and compression. The theory is beautiful and the applications are so important and widespread.” Gil Strang, Massachusetts Institute of Technology

“This book (FSP) and its companion (FWSP) bring a refreshing new, and comprehensive approach to teaching the fundamentals of signal processing, from analysis and decompositions, to multi- scale representations, approximations, and many other aspects that have a tremendous impact in modern information technology. Whereas classical texts were usually written for students in electrical or communication engineering programs, FSP and FWSP start from basic concepts in algebra and geometry, with the beneﬁt of being easily accessible to a much broader set of readers, and also help those readers develop strong abstract reasoning and intuition about signals and processing operators. A must-read!” Rico Malvar, Microsoft Research

“This is a wonderful book that connects together all the elements of modern signal processing. From functional analysis and probability theory, to linear algebra and computational methods, it’s all here and seamlessly integrated, along with a summary of history and developments in the ﬁeld. A real tour-de-force, and a must-have on every signal processor’s shelf!” Robert D. Nowak, University of Wisconsin–Madison

“Most introductory signal processing textbooks focus on classical transforms, and study how these can be used. Instead, Foundations of Signal Processing encourages readers to think of signals ﬁrst. It develops a ’signal-centric’ view, one that focuses on signals, their representation and approximation, through the introduction of signal spaces. Unlike most entry-level signal processing texts, this general view, which can be applied to many different signal classes, is introduced right at the beginning. From this, starting from basic concepts, and placing an emphasis on intuition, this book develops mathematical tools that give the readers a fresh perspective on classical results, while providing them with the tools to understand many state-of-the-art signal representation techniques.” Antonio Ortega, University of Southern California

“Foundations of Signal Processing by Vetterli, Kovaceviˇ c,´ and Goyal, is a pleasure to read. Draw- ing on the authors’ rich experience of research and teaching of signal processing and signal representations, it provides an intellectually cohesive and modern view of the subject from the geometric point of view of vector spaces. Emphasizing Hilbert spaces, where ﬁne technicalities can be relegated to backstage, this textbook strikes an excellent balance between intuition and mathematical rigor, that will appeal to both undergraduate and graduate engineering students. The last two chapters, on sampling and interpolation, and on localization and uncertainty, take full advan- tage of the machinery developed in the previous chapters to present these two very important topics of modern signal processing, that previously were only found in specialized monographs. The explanations of advanced topics are exceptionally lucid, exposing the reader to the ideas and thought processes behind the results and their derivation. Students will learn not only a substantial body of knowledge and techniques, but also why things work, at a deep level, which will equip them for independent further reading and research. I look forward to using this text in my own teaching.” Yoram Bresler, University of Illinois at Urbana-Champaign

Foundations of Signal Processing

MARTIN VETTERLI École Polytechnique Fédérale de Lausanne, Switzerland University of California, Berkeley, USA

JELENA KOVACEVIˇ C´ Carnegie Mellon University, USA

VIVEK K GOYAL Boston University, USA Massachusetts Institute of Technology, USA

University Printing House, Cambridge CB2 8BS, United Kingdom

Cambridge University Press is part of the University of Cambridge. It furthers the University’s mission by disseminating knowledge in the pursuit of education, learning and research at the highest international levels of excellence.

www.cambridge.org Information on this title: www.cambridge.org/9781107038608 c M. Vetterli, J. Kovaceviˇ c´ & V. K. Goyal 2014 This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published 2014 Printed and bound in the United Kingdom by TJ International Ltd. Padstow Cornwall A catalogue record for this publication is available from the British Library ISBN 978-1-107-03860-8 Hardback Additional resources for this publication at www.cambridge.org/vetterli and www.fourierandwavelets.org Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party internet websites referred to in this publication, and does not guarantee that any content on such websites is, or will remain, accurate or appropriate.

To Marie-Laure, for her ∞ patience and many other qualities, Thomas and No´emie, whom I might still convince of the beauty of this material, and my parents, who gave me all the opportunities one can wish for. —MV

To Danica and Giovanni, who make life beautiful. To my parents, who made me who I am. —JK

To Allie, Sundeep, and my family, who encourage me unceasingly, and to the educators who made me want to be one of them. —VKG

The cover illustration captures an experiment ﬁrst described by Isaac Newton in Opticks in 1730, showing that white light can be split into its color components and then synthesized back into white light. It is a physical implementation of a decomposition of white light into its Fourier components – the colors of the rainbow – followed by a synthesis to recover the original.

Contents

Quick reference page xvii

Acknowledgments xxiii

Preface xxv

1Onrainbowsandspectra 1

2 From Euclid to Hilbert 9 2.1 Introduction 10 2.2 Vector spaces 18 2.2.1 Deﬁnition and properties 18 2.2.2 Inner product 23 2.2.3 Norm 27 2.2.4 Standard spaces 30 2.3 Hilbert spaces 35 2.3.1 Convergence 36 2.3.2 Completeness 37 2.3.3 Linear operators 40 2.4 Approximations, projections, and decompositions 50 2.4.1 Projection theorem 51 2.4.2 Projection operators 54 2.4.3 Direct sums and subspace decompositions 60 2.4.4 Minimum mean-squared error estimation 63 2.5 Bases and frames 69 2.5.1 Bases and Riesz bases 69 2.5.2 Orthonormal bases 76 2.5.3 Biorthogonal pairs of bases 86 2.5.4 Frames 101 2.5.5 Matrix representations of vectors and linear operators 109 2.6 Computational aspects 119 2.6.1 Cost, complexity, and asymptotic notations 120 2.6.2 Precision 123 2.6.3 Conditioning 126 2.6.4 Solving systems of linear equations 129 2.A Elements of analysis and topology 135 2.A.1 Basic deﬁnitions 135

xiiContents

2.A.2 Convergence 136 2.A.3 Interchange theorems 138 2.A.4 Inequalities 139 2.A.5 Integration by parts 140 2.B Elements of linear algebra 141 2.B.1 Basic deﬁnitions and properties 141 2.B.2 Special matrices 147 2.C Elements of probability 151 2.C.1 Basic deﬁnitions 151 2.C.2 Standard distributions 154 2.C.3 Estimation 155 2.D Basis concepts 159 Chapter at a glance 161 Historical remarks 162 Further reading 162 Exercises with solutions 163 Exercises 169

3 Sequences and discrete-time systems 181 3.1 Introduction 182 3.2 Sequences 185 3.2.1 Infinite-length sequences 185 3.2.2 Finite-length sequences 192 3.2.3 Two-dimensional sequences 193 3.3 Systems 195 3.3.1 Discrete-time systems and their properties 195 3.3.2 Difference equations 202 3.3.3 Linear shift-invariant systems 205 3.4 Discrete-time Fourier transform 216 3.4.1 Definition of the DTFT 216 3.4.2 Existence and convergence of the DTFT 218 3.4.3 Properties of the DTFT 221 3.4.4 Frequency response of filters 227 3.5 z-transform 233 3.5.1 Definition of the z-transform 234 3.5.2 Existence and convergence of the z-transform 235 3.5.3 Properties of the z-transform 240 3.5.4 z-transform of filters 249 3.6 Discrete Fourier transform 252 3.6.1 Definition of the DFT 252 3.6.2 Properties of the DFT 255 3.6.3 Frequency response of filters 259 3.7 Multirate sequences and systems 264 3.7.1 Downsampling 265 3.7.2 Upsampling 268 3.7.3 Combinations of downsampling and upsampling 270 3.7.4 Combinations of downsampling, upsampling, and filtering 272 3.7.5 Polyphase representation 278 3.8 Stochastic processes and systems 285 3.8.1 Stochastic processes 285

Contents xiii

3.8.2 Systems 288 3.8.3 Discrete-time Fourier transform 292 3.8.4 Multirate sequences and systems 294 3.8.5 Minimum mean-squared error estimation 300 3.9 Computational aspects 303 3.9.1 Fast Fourier transforms 303 3.9.2 Convolution 307 3.9.3 Multirate operations 311 3.A Elements of analysis 313 3.A.1 Complex numbers 313 3.A.2 Diﬀerence equations 315 3.A.3 Convergence of the convolution sum 316 3.A.4 Dirac delta function 316 3.B Elements of algebra 318 3.B.1 Polynomials 318 3.B.2 Vectors and matrices of polynomials 321 3.B.3 Kronecker product 324 Chapter at a glance 325 Historical remarks 328 Further reading 328 Exercises with solutions 329 Exercises 336

4 Functions and continuous-time systems 343 4.1 Introduction 344 4.2 Functions 345 4.2.1 Functions on the real line 345 4.2.2 Periodic functions 351 4.3 Systems 351 4.3.1 Continuous-time systems and their properties 352 4.3.2 Differential equations 355 4.3.3 Linear shift-invariant systems 355 4.4 Fourier transform 359 4.4.1 Definition of the Fourier transform 359 4.4.2 Existence and inversion of the Fourier transform 360 4.4.3 Properties of the Fourier transform 365 4.4.4 Frequency response of filters 373 4.4.5 Regularity and spectral decay 374 4.4.6 Laplace transform 379 4.5 Fourier series 380 4.5.1 Definition of the Fourier series 381 4.5.2 Existence and convergence of the Fourier series 383 4.5.3 Properties of the Fourier series 385 4.5.4 Frequency response of filters 394 4.6 Stochastic processes and systems 395 4.6.1 Stochastic processes 395 4.6.2 Systems 397 4.6.3 Fourier transform 399 Chapter at a glance 401 Historical remarks 403

xiv Contents

Further reading 403 Exercises with solutions 404 Exercises 406

5 Sampling and interpolation 411 5.1 Introduction 412 5.2 Finite-dimensional vectors 420 5.2.1 Sampling and interpolation with orthonormal vectors 421 5.2.2 Sampling and interpolation with nonorthogonal vectors 425 5.3 Sequences 429 5.3.1 Sampling and interpolation with orthonormal sequences 430 5.3.2 Sampling and interpolation for bandlimited sequences 437 5.3.3 Sampling and interpolation with nonorthogonal sequences 442 5.4 Functions 447 5.4.1 Sampling and interpolation with orthonormal functions 449 5.4.2 Sampling and interpolation for bandlimited functions 452 5.4.3 Sampling and interpolation with nonorthogonal functions 470 5.5 Periodic functions 477 5.5.1 Sampling and interpolation with orthonormal periodic functions 477 5.5.2 Sampling and interpolation for bandlimited periodic functions 481 5.6 Computational aspects 489 5.6.1 Projection onto convex sets 491 Chapter at a glance 496 Historical remarks 498 Further reading 498 Exercises with solutions 499 Exercises 503

6 Approximation and compression 507 6.1 Introduction 508 6.2 Approximation of functions on ﬁnite intervals by polynomials 513 6.2.1 Least-squares approximation 514 6.2.2 Lagrange interpolation: Matching points 517 6.2.3 Taylor series expansion: Matching derivatives 520 6.2.4 Hermite interpolation: Matching points and derivatives 522 6.2.5 Minimax polynomial approximation 523 6.2.6 Filter design 529 6.3 Approximation of functions by splines 537 6.3.1 Splines and spline spaces 538 6.3.2 Bases for uniform spline spaces 541 6.3.3 Strang–Fix condition for polynomial representation 548 6.3.4 Continuous-time operators in spline spaces implemented with discrete- time processing 554 6.4 Approximation of functions and sequences by series truncation 560 6.4.1 Linear and nonlinear approximations 560 6.4.2 Linear approximation of random vectors and stochastic processes 566 6.4.3 Linear and nonlinear diagonal estimators 571 6.5 Compression 576 6.5.1 Lossless compression 577 6.5.2 Scalar quantization 579

Contents xv

6.5.3 Transform coding 584 6.6 Computational aspects 591 6.6.1 Huﬀman algorithm for lossless code design 591 6.6.2 Iterative design of quantizers 593 6.6.3 Estimating from quantized samples 594 Chapter at a glance 597 Historical remarks 598 Further reading 599 Exercises with solutions 600 Exercises 606

7 Localization and uncertainty 615 7.1 Introduction 616 7.2 Localization for functions 619 7.2.1 Localization in time 620 7.2.2 Localization in frequency 622 7.2.3 Uncertainty principle for functions 624 7.3 Localization for sequences 627 7.3.1 Localization in time 629 7.3.2 Localization in frequency 630 7.3.3 Uncertainty principle for sequences 633 7.3.4 Uncertainty principle for ﬁnite-length sequences 637 7.4 Tiling the time–frequency plane 638 7.4.1 Localization for structured sets of functions 638 7.4.2 Localization for structured sets of sequences 640 7.5 Examples of local Fourier and wavelet bases 645 7.5.1 Local Fourier and wavelet bases for functions 645 7.5.2 Local Fourier and wavelet bases for sequences 651 7.6 Recap and a glimpse forward 656 7.6.1 Tools 657 7.6.2 Adapting tools to real-world problems 658 7.6.3 Music analysis, communications, and compression 659 Chapter at a glance 666 Historical remarks 667 Further reading 667 Exercises with solutions 668 Exercises 671

Image and quote attribution 673

References 675

Index 681

Quick reference

Abbreviations AR Autoregressive ARMA Autoregressive moving average AWGN Additive white Gaussian noise BIBO Bounded input, bounded output CDF Cumulative distribution function DCT Discrete cosine transform DFT Discrete Fourier transform DTFT Discrete-time Fourier transform DWT Discrete wavelet transform FFT Fast Fourier transform FIR Finite impulse response i.i.d. Independent and identically distributed IIR Inﬁnite impulse response KLT Karhunen–Lo`eve transform LMMSE Linear minimum mean-squared error LPSV Linear periodically shift-varying LSI Linear shift-invariant MA Moving average MAP Maximum a posteriori probability ML Maximum likelihood MMSE Minimum mean-squared error MSE Mean-squared error PDF Probability density function PMF Probability mass function POCS Projection onto convex sets rad Radians ROC Region of convergence SNR Signal-to-noise ratio SVD Singular value decomposition WSCS Wide-sense cyclostationary WSS Wide-sense stationary

Abbreviations used in tables and captions but not in the text FT Fourier transform FS Fourier series LT Laplace transform

xvii

xviii Quick reference

Sets natural numbers N 0, 1, ... integers Z ..., −1, 0, 1, ... positive integers Z+ 1, 2, ... rational numbers Q p/q, p, q ∈ Z,q=0 real numbers R (−∞, ∞) positive real numbers R+ (0, ∞) complex numbers C a + jb or rejθ with a, b, r, θ ∈ R a generic index set I a generic vector space V a generic Hilbert space H closure of set S S

Real and complex analysis

sequence xn argument n is an integer, n ∈ Z function x(t)argumentt is continuous-valued, t ∈ R ordered sequence (xn)n set containing xn {xn}n vector x with xn as elements [xn]

Kronecker delta sequence δn δn =1forn =0;δn =0otherwise ∞ Dirac delta function δ(t) x(t) δ(t) dt = x(0) for x continuous at 0 −∞

indicator function of interval I 1I 1I (t)=1fort ∈ I;1I (t)=0otherwise integration by parts udv = uv − vdu

complex number za+ jb, rejθ, a, b ∈ R, r ∈ [0, ∞), θ ∈ [0, 2π) conjugation z∗ a − jb, re−jθ real part of ( · ) (a + jb)=a, a, b ∈ R imaginary part of ( · ) (a + jb)=b, a, b ∈ R ∗ ∗ conjugation of coeﬃcients X∗(z) X (z ) −j2π/N principal root of unity WN e

Asymptotic notation

big O x ∈ O(y)0≤ xn ≤ γyn for all n ≥ n0; some n0 and γ>0 little o x ∈ o(y)0≤ xn ≤ γyn for all n ≥ n0; some n0,anyγ>0 Omega x ∈ Ω(y) xn ≥ γyn for all n ≥ n0; some n0 and γ>0 Theta x ∈ Θ(y) x ∈ O(y)andx ∈ Ω(y) asymptotic equivalence x y limn→∞ xn/yn =1

Quick reference xix

Standard vector spaces Hilbert space of square-summable 2(Z) x : Z → C |x |2 < ∞ with sequences n n ∗ inner product x, y = n xnyn Hilbert space of square-integrable 2 2 L (R) x : R → C |x(t)| dt < ∞ with functions inner product x, y = x(t) y(t)∗ dt normed vector space of sequences p(Z) x : Z → C |x |p < ∞ with with ﬁnite p norm, 1 ≤ p<∞ n n | |p 1/p norm x p = n xn normed vector space of functions Lp R R → C | |p ∞ Lp ≤ ∞ ( ) x : x(t) dt < with with ﬁnite norm, 1 p< p 1/p norm x p = |x(t)| dt normed vector space of bounded ∞ (Z) x : Z → C sup |xn| < ∞ with sequences with supremum norm n | | norm x ∞ =supn xn normed vector space of bounded ∞ L (R) x : R → C ess sup |x(t)| < ∞ with functions with supremum norm t | | norm x ∞ = ess supt x(t)

Bases and frames for sequences

standard basis {ek} ek,n =1forn = k; ek,n =0otherwise vector, element of basis or frame ϕ when applicable, a column vector basis or frame Φ set of vectors {ϕk} operator Φ concatenation of {ϕk} in a linear operator: [ϕ0 ϕ1 ... ϕN−1] vector, element of dual basis or frame ϕ when applicable, a column vector Φ set of vectors {ϕk} operator Φ concatenation of {ϕk} in a linear operator: [ϕ0 ϕ1 ... ϕN−1] expansion with a basis or frame x =ΦΦ∗x

xx Quick reference

Discrete-time signal processing

Sequence xn signal, vector

Convolution linear h ∗ x xkhn−k k∈Z N−1 circular h x x h − (N-periodic sequences) k (n k)modN k=0 (h ∗ x)n convolution result at n

Eigensequence vn eigenvector jωn jω inﬁnite time vn = e h ∗ v = H(e ) v j2πkn/N ﬁnite time vn = e h v = Hkv

Frequency response eigenvalue corresponding to vn jω −jωn inﬁnite time H(e ) hne n∈Z N−1 N−1 −j2πkn/N kn ﬁnite time Hk hne = hnWN n=0 n=0 Continuous-time signal processing Function x(t)signal

Convolution ∞ linear h ∗ x x(τ) h(t − τ) dτ −∞ circular T/2 h x x(τ) h(t − τ) dτ (T -periodic functions) −T/2 (h ∗ x)(t) convolution result at t

Eigenfunction v(t)eigenvector inﬁnite time v(t)=ejωt h ∗ v = H(ω) v j2πkt/T ﬁnite time v(t)=e h v = Hkv

Frequency response eigenvalue corresponding to v(t)

∞ inﬁnite time H(ω) h(t) e−jωt dt −∞ T/2 −j2πkτ/T ﬁnite time Hk h(τ) e dτ −T/2

Quick reference xxi

Spectral analysis ∞ Fourier transform x(t) ←→FT X(ω) X(ω)= x(t)e−jωt dt −∞ 1 ∞ inverse x(t)= X(ω) ejωt dω 2π −∞ T/2 1 − Fourier series coeﬃcients x(t) ←→FS X X = x(t) e j(2π/T)kt dt k k T −T/2 j(2π/T)kt reconstruction x(t)= Xke k∈Z DTFT jω jω −jωn discrete-time Fourier transform xn ←→ X(e ) X(e )= xne n∈Z π 1 jω jωn inverse xn = X(e ) e dω 2π −π N−1 DFT kn discrete Fourier transform xn ←→ Xk Xk = xnWN n=0 N−1 1 − inverse x = X W kn n N k N n=0 ZT −n z-transform xn ←→ X(z) X(z)= xnz n∈Z

Acknowledgments

This book would not exist without the help of many people, whom we attempt to list below. We apologize for any omissions and welcome corrections and suggestions. We are grateful to Professor Libero Zuppiroli of EPFL and Christiane Grimm for the photograph that graces the cover; Professor Zuppiroli proposed an experiment from Newton’s treatise, Opticks [70], as emblematic of the book, and Ms. Grimm beautifully photographed the apparatus that he designed. Fran¸coise Behn, Jocelyne Plantefol, and Jacqueline Aeberhard typed parts of the manuscript, Eric Strattman assisted with some of the figures, Krista Van Guilder designed and implemented the initial book web site, and Jorge Albaladejo Pomares designed and implemented the book blog. We thank them for their diligence and patience. S. Grace Chang and Yann Barbotin helped organize and edit the problem companion, while Patrick Vandewalle designed and implemented a number of MATLABR problems available on the book website.1 We thank them for their expertise and insight. We are indebted to George Beck for his MathematicaR editorial comments on Wolfram Demonstrations inspired by this book. We are grateful to Giovanni Pacifici for the many useful scripts for automating various book-writing processes. We thank John Cozzens of the US National Science Foundation for his unwavering support over the last two decades. At Cambridge University Press, Steven Holt held us to high standards in our writing and typesetting through many corrections and suggestions; Elizabeth Horne and Jonathan Ratcliffe coordinated production expertly; and, last but not least, Phil Meyler repeatedly (and most gracefully) re- minded us of our desire to finish the book. Many instructors have gamely tested pre-alpha, alpha, and beta versions of this manuscript. Of these, Amina Chebira, Filipe Condessa, Mihailo Kolundˇzija, Yue M. Lu, Truong-Thao Nguyen, Reza Parhizkar, and Jayakrishnan Unnikrishnan have done far more than their share in providing invaluable comments and suggestions. We also thank Robert Gray for reviewing the manuscript and providing many important suggestions, in particular a better approach to covering the Dirac delta function; Thierry Blu for, among other things, providing a simple proof for a particular case of the Strang–Fix theorem; Matthew Fickus for consulting on some finer mathematical points; Michael Unser for his notes and teaching approach; and Zoran Cvetković, Minh N. Do, and Philip Schniter for teaching with the manuscript and providing many constructive comments. Useful comments have also been provided

1http://www.fourierandwavelets.org

xxiii

xxiv Acknowledgments

by Pedro Aguilar, Ganesh Ajjanagadde, A. Avudainayagam, Jerry Bauch, Anirud- dha Bhargava, Ozan Çaˇglayan, André Tomaz de Carvalho, Gustavo Corach, Todd Doucet, S. Esakkirajan, Germán González, Alexandre Haehlen, Mina Karzand, Ull- rich Mönich, Juan Pablo Muszkats, Hossein Rouhani, Noah D. Stein, John Z. Sun, Christophe Tournery, and Lav R. Varshney. Martin Vetterli thanks current and former EPFL graduate students and post- docs who helped develop the material, solve problems, catch typos, and suggest im- provements, among other things. They include Florence Bénézit, Amina Chebira, Minh N. Do, Ivan Dokmanić, Pier Luigi Dragotti, Ali Hormati, Ivana Jovanović, Mihailo Kolundˇzija, Jérôme Lebrun, Yue M. Lu, Pina Marziliano, Fritz Menzer, Reza Parhizkar, Paolo Prandoni, Juri Ranieri, Olivier Roy, Rahul Shukla, Jayakr- ishnan Unnikrishnan, Patrick Vandewalle and Vladan Velisavljević. He gratefully acknowledges support from EPFL, the Swiss NSF through awards 2000-063664, 200020-103729, and 200021-121935, and the European Research Council through award SPARSAM 247006. In addition, the support from Qualcomm, in particular from Dr. Chong Lee, is gratefully acknowledged. Jelena Kovaˇcević thanks her present and past graduate students Ramamurthy Bhagavatula, Amina Chebira, Kuan-Chieh Jackie Chen, Siheng Chen, Filipe Con- dessa, Charles Jackson, Xindian Long, Michael T. McCann, Anupama Kuruvilla, Thomas E. Merryman, Vivek Oak, Aliaksei Sandryhaila, and Gowri Srinivasa, many of whom served as TAs for her classes at CMU, together with Pablo Hennings Yeo- mans. Thanks are due also to all the students in the following classes at CMU: 42-431/18-496, 42-540/42-698, 42-703/18-799, 42-731/18-795, and 42-732/18-790, taught from 2003 to 2013. She gratefully acknowledges support from the US NSF through awards 1017278, 1130616, 515152, 633775, and 331657; the NIH through awards EB008870, EB009875, and 1DC010283; the Carnegie Institute of Tech- nology at CMU through an Infrastructure for Large-Scale Biomedical Computing award; the Pennsylvania State Tobacco Settlement, the Kamlet–Smith Bioinfor- matics Grant; and the Philip and Marsha Dowd Teaching Fellowship. Vivek Goyal thanks TAs and students at MIT and BU for suggestions, in particular Giuseppe Bombara, Baris Erkmen, Ying-zong Huang, Zahi Karam, Ahmed Kirmani, Srilalitha Kumaresan, Ranko Sredojević, Ramesh Sridharan, Watchara- pan Suwansantisuk, Vincent Tan, Archana Venkataraman, Adam Zelinski, and Ser- hii Zhak. He gratefully acknowledges support from the US NSF through awards 0643836, 0729069, 1101147, 1115159, and 1161413; Texas Instruments through its Leadership University Program; Hewlett–Packard; Qualcomm; Siemens; MIT through the Esther and Harold E. Edgerton Career Development Chair; and the Centre Bernoulli of EPFL.

Preface

Our main goals in this book and its companion volume, Fourier and Wavelet Signal Processing (FWSP) [57], are to enable an understanding of state-of-the-art signal processing methods and techniques, as well as to provide a solid foundation for those hoping to advance the theory and practice of signal processing. We believe that the best way to grasp and internalize the fundamental concepts in signal processing is through the geometry of Hilbert spaces, as this leverages the great innate human capacity for spatial reasoning. While using geometry should ultimately simplify the subject, the connection between signals and geometry is not innate. The reader will have to invest effort to see signals as vectors in Hilbert spaces before reaping the benefits of this view; we believe that effort to be well placed. Many of the results and techniques presented in the two volumes, while rooted in classic Fourier techniques for signal representation, first appeared during a flurry of activity in the 1980s and 1990s. New constructions of local Fourier transforms and orthonormal wavelet bases during that period were motivated both by theoretical interest and by applications, multimedia communications in particular. New bases with specified time–frequency behavior were found, with impact well beyond the original fields of application. Areas as diverse as computer graphics and numerical analysis embraced some of the new constructions – no surprise given the pervasive role of Fourier analysis in science and engineering. Many of these new tools for signal processing were developed in the applied harmonic analysis community. The resulting high level of mathematical sophistica- tion was a barrier to entry for many signal processing practitioners. Now that the dust has settled, some of what was new and esoteric has become fundamental; we want to bring these new fundamentals to a broader audience. The Hilbert space formalism gives us a way to begin with the classical Fourier analysis of signals and systems and reach structured representations with time–frequency locality and their varied applications. Whenever possible, we use explanations rooted in elementary analysis over those that would require more advanced background (such as measure theory). We hope to have balanced the competing virtues of accessibility to the student, rigor, and adequate analytical power to reach important conclusions. The book can be used as a self-contained text on the foundations of signal processing, where discrete and continuous time are treated on equal footing. All the necessary mathematical background is included, with examples illustrating the applicability of the results. In addition, the book serves as a precursor to FWSP, which relies on the framework built here; the two books are thus integrally related.

xxv

xxvi Preface

Foundations of Signal Processing This book covers the foundations for an in- depth understanding of modern signal processing. It contains material that many readers may have seen before scattered across multiple sources, but without the Hilbert space interpretations, which are essential in signal processing. Our aim is to teach signal processing with geometry, that is, to extend Euclidean geometric insights to abstract signals; we use Hilbert space geometry to accomplish that. With this approach, fundamental concepts – such as properties of bases, Fourier representations, sampling, interpolation, approximation, and compression – are often unified across finite dimensions, discrete time, and continuous time, thus making it easier to point out the few essential differences. Unifying results geometrically helps generalize beyond Fourier-domain insights, pushing the understanding farther, faster. Chapter 2, From Euclid to Hilbert, is our main vehicle for drawing out unifying commonalities; it develops the basic geometric intuition central to Hilbert spaces, together with the necessary tools underlying the constructions of bases and frames. The next two chapters cover signal processing on discrete-time and continuous- time signals, specializing general concepts from Chapter 2. Chapter 3, Sequences and discrete-time systems, is a crash course on processing signals in discrete time or discrete space together with spectral analysis with the discrete-time Fourier transform and discrete Fourier transform. Chapter 4, Functions and continuous-time systems, is its continuous-time counterpart, including spectral analysis with the Fourier transform and Fourier series. Chapter 5, Sampling and interpolation, presents the critical link between discrete and continuous domains given by sampling and interpolation theorems. Chap- ter 6, Approximation and compression, veers from exact representations to approx- imate ones. The final chapter in the book, Chapter 7, Localization and uncertainty, considers time–frequency behavior of the abstract representation objects studied thus far. It also discusses issues arising in applications as well as ways of adapting the previously introduced tools for use in the real world.

Fourier and Wavelet Signal Processing The companion volume focuses on signal representations using local Fourier and wavelet bases and frames. It covers the two-channel ﬁlter bank in detail, and then uses it as the implementation vehicle for all sequence representations that follow. The local Fourier and wavelet methods are presented side-by-side, without favoring any one in particular; the truth is that each representation is a tool in the toolbox of the practitioner, and the problem or application at hand ultimately determines the appropriate one to use. We end with examples of state-of-the-art signal processing and communication problems, with sparsity as a guiding principle.

Teaching points Our aim is to present a synergistic view of signal representations and processing, starting from basic mathematical principles and going all the way to actual constructions of bases and frames, always with an eye on concrete applications. While the beneﬁt is a self-contained presentation, the cost is a rather sizable manuscript. Referencing in the main text is sparse; pointers to the bibliography are given in Further reading at the end of each chapter.

Preface xxvii

The material grew out of teaching signal processing, wavelets, and applications in various settings. Two of us (MV and JK) authored a graduate textbook, Wavelets and Subband Coding (originally with Prentice Hall in 1995, now open access2), which we and others used to teach graduate courses at various US and European institutions. With the maturing of the field and the interest arising from and for these topics, the time was right for the three of us to write entirely new texts geared toward a broader audience. We and others have taught with these books, in their entirety or in parts, a number of times and to a number of different audiences: from senior undergraduate to graduate level, and from engineering to mixes that include life-science students. The books and their website3 provide a number of features for teaching and learning: • Exercises are an integral part of the material and come in two forms: solved exercises with explicit solutions within the text, and regular exercises that allow students to test their knowledge. Regular exercises are marked with (1), (2),or(3) in increasing order of difficulty. • Numerous examples illustrate concepts throughout the book. • An electronic version of the text is provided with the printed copy. It includes PDF hyperlinks and an additional color to enhance interpretation of figures. • A free electronic version of the text without PDF hyperlinks, exercises or solved exercises, and with figures in grayscale, is available at the book website. • A MathematicaR companion, which contains the code to produce all numerical figures in the book, is provided with the printed version. • Several interactive MathematicaR demonstrations using the free CDF player are available at the book website. • Additional material, such as lecture slides, is available at the book website. • To instructors, we provide a Solutions Manual, with solutions to all regular exercises in the book.

Notational points To traverse the book efficiently, it will help to know the various numbering conventions that we have employed. In each chapter, a single counter is used for definitions, theorems, and corollaries (which are all shaded) and another for examples (which are slightly indented). Equations, figures, and tables are also numbered within each chapter. A prefix En.m– in the number of an equation, figure, or table indicates that it is part of Solved Exercise n.m in Chapter n.The letter P is used similarly for statements of regular exercises.

Martin Vetterli, Jelena Kovaˇcevi´c, and Vivek K Goyal

2http://waveletsandsubbandcoding.org/ 3http://www.fourierandwavelets.org/