Wasyl Wasylkiwskyj
Signals and Transforms in Linear Systems Analysis Signals and Transforms in Linear Systems Analysis
Wasyl Wasylkiwskyj
Signals and Transforms in Linear Systems Analysis
123 Wasyl Wasylkiwskyj Professor of Engineering and Applied Science The George Washington University Washington, DC, USA
ISBN 978-1-4614-3286-9 ISBN 978-1-4614-3287-6 (eBook) DOI 10.1007/978-1-4614-3287-6 Springer New York Heidelberg Dordrecht London
Library of Congress Control Number: 2012956318
© Springer Science+Business Media, LLC 2013 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.
Printed on acid-free paper
Springer is part of Springer Science+Business Media (www.springer.com) Preface
This book deals with aspects of mathematical techniques and models that con- stitute an important part of the foundation for the analysis of linear systems. The subject is classical and forms a significant component of linear systems theory. These include Fourier, Z-transforms, Laplace, and related transforms both in their continuous and discrete versions. The subject is an integral part of electrical engineering curricula and is covered in many excellent textbooks. In light of this, an additional book dealing with the same topics would appear su- perfluous. What distinguishes this book is that the same topics are viewed from a distinctly different perspective. Rather than dealing with different transforms essentially in isolation, a methodology is developed that unifies the classical portion of the subject and permits the inclusion of topics that usually are not considered part of the linear systems theory. The unifying principle here is the least mean square approximation, the normal equations, and their extensions to the continuum. This approach gives equal status to expansions in terms of special functions (that need not be orthogonal), Fourier series, Fourier integrals, and discrete transforms. As a by-product one also gains new insights. For ex- ample, the Gibbs phenomenon is a general property of LMS convergence at step discontinuities and is not limited to Fourier series. This book is suitable for a first year graduate course that provides a tran- sition from the level the subject is presented in an undergraduate course in signals and systems to a level more appropriate as a prerequisite for graduate work in specialized fields. The material presented here is based in part on the notes used for a similar course taught by the author in the School of Electri- cal and Computer Engineering at The George Washington University. The six chapters can be covered in one semester with sufficient flexibility in the choice of topics within each chapter. The exception is Chap. 1 which, in the spirit of the intended unity, sets the stage for the remainder of the book. It includes the mathematical foundation and the methodology applied in the chapters to follow. The prerequisites for the course are an undergraduate course in signals and systems, elements of linear algebra, and the theory of functions of a complex variable. Recognizing that frequently the preparation, if any, in the latter is sketchy, the necessary material is presented in the Appendix.
Wasyl Wasylkiwskyj
V
Contents
1 Signals and Their Representations 1 1.1 Signal Spaces and the Approximation Problem ...... 1 1.2 Inner Product, Norm and Representations by Finite Sums of Elementary Functions ...... 4 1.2.1 Inner Product and Norm ...... 4 1.2.2 Orthogonality and Linear Independence ...... 7 1.2.3 Representations by Sums of Orthogonal Functions ...... 12 1.2.4 Nonorthogonal Expansion Functions and Their Duals ...... 14 1.2.5 Orthogonalization Techniques ...... 16 1.3 The LMS Approximation and the Normal Equations ...... 19 1.3.1 The Projection Theorem ...... 19 1.3.2 The Normal Equations ...... 21 1.3.3 Generalizations of the Normal Equations ...... 22 1.3.4 LMS Approximation and Stochastic Processes* ...... 25 1.4 LMS Solutions via the Singular Value Decomposition ...... 27 1.4.1 Basic Theory Underlying the SVD ...... 27 1.4.2 Solutions of the Normal Equations Using the SVD ...... 30 1.4.3 Signal Extraction from Noisy Data ...... 32 1.4.4 The SVD for the Continuum ...... 35 1.4.5 Frames ...... 37 1.4.6 Total Least Squares ...... 39 1.4.7 Tikhonov Regularization ...... 43 1.5 Finite Sets of Orthogonal Functions ...... 44 1.5.1 LMS and Orthogonal Functions ...... 44 1.5.2 Trigonometric Functions ...... 45 1.5.3 Orthogonal Polynomials [1] ...... 47 1.6 Singularity Functions ...... 52 1.6.1 The Delta Function ...... 52 1.6.2 Higher Order Singularity Functions ...... 59
VII VIII Contents
1.6.3 Idealized Signals ...... 61 1.6.4 Representation of Functions with Step Discontinuities ...... 63 1.6.5 Delta Function with Functions as Arguments ...... 65 1.7 Infinite Orthogonal Systems ...... 66 1.7.1 Deterministic Signals ...... 66 1.7.2 Stochastic Signals: Karhunen–Loeve Expansion∗ ..... 68
2 Fourier Series and Integrals with Applications to Signal Analysis 75 2.1 Fourier Series ...... 75 2.1.1 Pointwise Convergence at Interior Points for Smooth Functions ...... 75 2.1.2 Convergence at Step Discontinuities ...... 78 2.1.3 Convergence at Interval Endpoints ...... 82 2.1.4 Delta Function Representation ...... 84 2.1.5 The Fejer Summation Technique ...... 86 2.1.6 Fundamental Relationships Between the Frequency and Time Domain Representations ...... 92 2.1.7 Cosine and Sine Series ...... 94 2.1.8 Interpolation with Sinusoids ...... 98 2.1.9 Anharmonic Fourier Series ...... 104 2.2 The Fourier Integral ...... 107 2.2.1 LMS Approximation by Sinusoids Spanning a Continuum ...... 107 2.2.2 Transition to an Infinite Observation Interval: The Fourier Transform ...... 108 2.2.3 Completeness Relationship and Relation to Fourier Series ...... 109 2.2.4 Convergence and the Use of CPV Integrals ...... 111 2.2.5 Canonical Signals and Their Transforms ...... 114 2.2.6 Basic Properties of the FT ...... 117 2.2.7 Convergence at Discontinuities ...... 128 2.2.8 Fejer Summation ...... 128 2.3 Modulation and Analytic Signal Representation ...... 132 2.3.1 Analytic Signals ...... 132 2.3.2 Instantaneous Frequency and the Method of Stationary Phase ...... 134 2.3.3 Bandpass Representation ...... 140 2.3.4 Bandpass Representation of Random Signals* ...... 143 2.4 Fourier Transforms and Analytic Function Theory ...... 148 2.4.1 Analyticity of the FT of Causal Signals ...... 148 2.4.2 Hilbert Transforms and Analytic Functions ...... 149 2.4.3 Relationships Between Amplitude and Phase ...... 152 2.4.4 Evaluation of Inverse FT Using Complex Variable Theory ...... 154 Contents IX
2.5 Time-Frequency Analysis ...... 159 2.5.1 The Uncertainty Principle ...... 159 2.5.2 The Short-Time Fourier Transform ...... 163 2.6 Frequency Dispersion ...... 168 2.6.1 Phase and Group Delay ...... 168 2.6.2 Phase and Group Velocity ...... 171 2.6.3 Effects of Frequency Dispersion on Pulse Shape ...... 173 2.6.4 Another Look at the Propagation of a Gaussian Pulse When β (ω0)=0...... 180 2.6.5 Effects of Finite Transmitter Spectral Line Width* ....182 2.7 Fourier Cosine and Sine Transforms ...... 185
3LinearSystems 191 3.1 Fundamental Properties ...... 191 3.1.1 Single-valuedness, Reality, and Causality ...... 191 3.1.2 Impulse Response ...... 193 3.1.3 Step Response ...... 196 3.1.4 Stability ...... 196 3.1.5 Time-invariance ...... 197 3.2 Characterizations in terms of Input/Output Relationships ....199 3.2.1 LTI Systems ...... 199 3.2.2 Time-varying Systems ...... 201 3.3 Linear Systems Characterized by Ordinary Differential Equations ...... 207 3.3.1 First-Order Differential Equations ...... 207 3.3.2 Second-Order Differential Equations ...... 213 3.3.3 N-th Order Differential Equations ...... 225
4 Laplace Transforms 235 4.1 Single-Sided Laplace Transform ...... 235 4.1.1 Analytic Properties ...... 235 4.1.2 Singularity Functions ...... 239 4.1.3 Some Examples ...... 240 4.1.4 Inversion Formula ...... 241 4.1.5 Fundamental Theorems ...... 243 4.1.6 Evaluation of the Inverse LT ...... 248 4.2 Double-Sided Laplace Transform ...... 260 4.2.1 Definition and Analytic Properties ...... 260 4.2.2 Inversion Formula ...... 261 4.2.3 Relationships Between the FT and the Unilateral LT . . . 267
5 Bandlimited Functions Sampling and the Discrete Fourier Transform 271 5.1 Bandlimited Functions ...... 271 5.1.1 Fundamental Properties ...... 271 5.1.2 The Sampling Theorem ...... 274 X Contents
5.1.3 Sampling Theorem for Stationary Random Processes* ...... 276 5.2 Signals Defined by a Finite Number of Samples ...... 278 5.2.1 Spectral Concentration of Bandlimited Signals ...... 281 5.2.2 Aliasing ...... 283 5.3 Sampling ...... 286 5.3.1 Impulse Sampling ...... 286 5.3.2 Zero-Order Hold Sampling and Reconstruction ...... 287 5.3.3 BandPass Sampling ...... 290 5.3.4 Sampling of Periodic Signals ...... 293 5.4 The Discrete Fourier Transform ...... 297 5.4.1 Fundamental Definitions ...... 297 5.4.2 Properties of the DFT ...... 300
6 The Z-Transform and Discrete Signals 311 6.1 The Z-Transform ...... 311 6.1.1 From FS to the Z-Transform ...... 311 6.1.2 Direct ZT of Some Sequences ...... 316 6.1.3 Properties ...... 317 6.2 Analytical Techniques in the Evaluation of the Inverse ZT ....320 6.3 Finite Difference Equations and Their Use in IIR and FIR Filter Design ...... 327 6.4 Amplitude and Phase Relations Using the Discrete Hilbert Transform ...... 331 6.4.1 Explicit Relationship Between Real and Imaginary Parts of the FT of a Causal Sequence ...... 331 6.4.2 Relationship Between Amplitude and Phase of a Transfer Function ...... 333 6.4.3 Application to Design of FIR Filters ...... 334
A Introduction to Functions of a Complex Variable 337 A.1 Complex Numbers and Complex Variables ...... 337 A.1.1 Complex Numbers ...... 337 A.1.2 Function of a Complex Variable ...... 341 A.2 Analytic Functions ...... 342 A.2.1 Differentiation and the Cauchy–Riemann Conditions ...... 342 A.2.2 Properties of Analytic Functions ...... 344 A.2.3 Integration ...... 345 A.3 Taylor and Laurent Series ...... 349 A.3.1 The Cauchy Integral Theorem ...... 349 A.3.2 The Taylor Series ...... 351 A.3.3 Laurent Series ...... 354 A.4 Singularities of Functions and the Calculus of Residues ...... 356 A.4.1 Classification of Singularities ...... 356 A.4.2 Calculus of Residues ...... 361 Contents XI
Bibliography 369
Index 371
*The subsections marked with* are supplements and not parts of the main text
Introduction
Although the book’s primary purpose is to serve as a text, the basic nature of the subject and the selection of topics should make it also of interest to a wider audience. The general idea behind the text is two fold. One is to close the gap that usually exists between the level of student’s preparation in transform calculus acquired in undergraduate studies and the level needed as preparation for graduate work. The other is to broaden the student’s intellectual horizon. The approach adopted herein is to exclude many of the topics that are usually covered in undergraduate linear systems texts, select those that in the opinion of the author serve as the common denominator for virtually all electrical engi- neering disciplines and present them within the unifying framework of the gen- eralized normal equations. The selected topics include Fourier analysis, both in its discrete and continuous formats, its ramifications to time- frequency analysis, frequency dispersion and its ties to linear systems theory, wherein equal statues is accorded the LTI and time-varying systems . The Laplace and Z-transforms are presented with special emphasis on their connection with Fourier analysis. The book begins within a rather abstract mathematical framework that could be discouraging for a beginner. Nevertheless, to pave the path to the material in the following chapters, I could not find a simpler approach. The introductory mathematics is largely contained in the first chapter. The following is the synopsis. Starting on familiar ground, a signal is defined as a piecewise differentiable functions of time and the system input/output relation as a mapping by an operator from its domain unto its range. Along with the restriction to linear operators the representation of a signal as a sum of canonical expansion functions is introduced. The brief discussion of error criteria with focus on the LMS (Least Mean Squared approximation) is followed by an examination of the basic linear algebra concepts: norm, inner products linear independence and orthogonality. To emphasize the conceptual unity of the subject, analogue signals and their discrete counterparts are given equal status. The LMS problem is viewed from the standpoint of the normal equations. The regularization of ill conditioned matrices is studied using the SVD (singular value decomposition). Its noise suppression attributes are examined via numer- ical examples. The TLS (Total Least Square) solution technique is discussed briefly as is the Tikhonov regularization.
XIII XIV Introduction
The normal equations present us with three algebraically distinct but con- ceptually identical representations. The first is the discrete form which solves the problem of minimizing the MS error in the solution of an overdetermined system: