Adaptive Digital Filters Branko Kovacˇevic´ • Zoran Banjac Milan Milosavljevic´

Adaptive Digital Filters

Academic Mind University of Belgrade - School of Electrical Engineering University Singidunum, Belgrade Springer-Verlag Berlin Heidelberg Branko Kovacˇevic´ Milan Milosavljevic´ University of Belgrade University of Belgrade and Singidunum Belgrade University Serbia Belgrade Serbia Zoran Banjac School of Electrical and Computing Engineering of Applied Studies Belgrade Serbia

ISBN 978-3-642-33560-0 ISBN 978-3-642-33561-7 (eBook) DOI 10.1007/978-3-642-33561-7 Springer Heidelberg New York Dordrecht London

Library of Congress Control Number: 2013935402 ISBN of Academic Mind : 978-86-7466-434-6

Ó Academic Mind Belgrade and Springer-Verlag Berlin Heidelberg 2013 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer and Academic Mind. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.

Printed on acid-free paper

Academic Mind is official publisher of University of Belgrade, School of Electrical Engineering (www.akademska-misao.rs) Springer is part of Springer Science+Business Media (www.springer.com) Preface

The book Adaptive Digital Filters appeared as a result of years of cooperation between the Department for Signal Processing within the Institute of Applied Mathematic and Electronics, Belgrade, Serbia and the Division for Automation that has later evolved to the present Department for Signals and Systems within the School of Electrical Engineering, University of Belgrade. This cooperation started back in mid-1970s, with a goal to research the phenomenon of speech and has continued up until present days. Among important results of joint science research in the fields of modeling, analysis, processing, recognition, and transmission of speech signals are, besides mathematical algorithms, program packages, technical solutions, and electronic instruments, also numerous research papers, either pub- lished in leading international science journals or presented in proceedings of prestigious international science conferences. Since the very inception of their cooperation the Institute and the School introduced a custom to inform a wider circle of domestic researchers and experts, as well as the students of mathematics, electrical engineering, computing, and related areas about the most important results of their joint projects. This goal was usually achieved by publishing science monographs in Serbian language, and this was the way two science monographs appeared, ‘‘Speech signal processing and recognition’’ (a group of authors from the Institute and the School, published by the Center of High Military Schools, Bel- grade, 1993) and ‘‘Robust digital processing of speech signals’’ (published by Academic Mind, Belgrade, 2000). It is important to mention here that the quoted publications were preceded by a number of master theses and doctoral disserta- tions. In this way, this book represents a continuation of the good practice intro- duced by the Institute and the School and presents the results of joint research within the past decade. The youngest author of this book, Dr. Zoran Banjac, finished his master’s thesis and doctoral dissertation in the course of his work within these projects. The aim and the stance of this book was maybe best defined by its reviewers, professors at the School of Electrical Engineering, the University of Belgrade, professors Ljiljana Milic´ and Dušan Drajic´, who wrote in the con- clusion of their review: ‘‘The monograph Adaptive Digital Filters presents to our scientific and expert community an important discipline which has been under- represented prior to the appearance of this book. The book first makes the reader acquainted with the basic terms of filtering and adaptive filtering, to further

v vi Preface introduce the reader into the field of advanced modern algorithms, some of which represent a contribution of the authors of the book. The work in the field of adaptive signal processing requires the use of a complex mathematical apparatus. The manner of exposition in this book presumes a detailed presentation of the mathematical models, a task done by the authors in a clear and consistent way. The chosen approach enables everyone with a college level of mathematics knowledge to successfully follow the mathematical derivations and descriptions of algorithms in the book. The algorithms are presented by flow charts, which facilitates their practical implementation. The book gives many experimental results and treats the aspects of practical application of adaptive filtering in real systems. The book will be useful both to students of undergraduate and graduate studies, and to all of those who did not have an opportunity to master this important science field during their formal education’’. The authors would like to express their gratitude to the referees for their useful suggestions and advices which contributed significantly to the quality of the book. The text of the book is divided into six chapters. The first, introductory chapter, considers generally three most often used theoretical approaches to the design of linear filters—the conventional approach, the optimal filtration, and the adaptive filtration. The further text analyzes only the third approach, i.e., the adaptive filtration. Chapter 2 presents the basic structures of adaptive filters. It also considers the criterion function for the optimization of the parameters of adaptive filters and analyzes the two basic numerical methods for the determination of the minimum of the criterion function: the Newton method and the method of steepest descent. After presenting the basic concept of adaptive filtering, it overviews the standard and the derived adaptive algorithms of the Least Mean Square Error—LMS type and Recursive Least Square—RLS algorithm, for the sake of further analysis and estimation of the possibilities to modify them in order to improve the character- istics of the mentioned adaptive algorithms. Also, potential advantages of the Infinite Impulse Response—IIR filters impose a need for their more intensive use, as well as for the analysis of the adaptation of the proposed solution for the systems with Finite Impulse Response—FIR systems, as well as for the IIR sys- tems. This is the reason why in the second chapter care has been given to this problem too. An analysis of the ability of adaptive algorithms to follow nonstationary changes in the system, together with the synthesis of efficient algorithms based on variable forgetting factor, is presented in Chap. 3. A comparison has been made among a number of strategies for the choice of forgetting factor (extended pre- diction error, parallel adaptation, and Fortecue-Kershenbaum-Ydstie algorithm) against their ability to follow nonstationary changes and the complexity of the implementation of algorithms. The most convenient strategies for the choice of variable forgetting factor from the practical point of view were emphasized. Chapter 4 presents an original approach to the design of an FIR-type adaptive algorithm with a goal to increase the convergence speed in the parameter esti- mation process. The approach is based on an optimum approach to the Preface vii construction of input signal, which belongs to the D-class of optimal experiment planning. The properties of the proposed algorithm were subsequently analyzed through the practical problem of local echo cancellation in scrambling systems. Besides that, a possibility has been shown to apply this approach in nonstationary environments through the application of a convenient strategy for the choice of variable forgetting factor. Robustification of adaptive algorithms against impulsive nonstationary noise in the desired response is considered in Chap. 5. An original robust algorithm based on the LMS approach is presented and an analysis is given of the possibility to apply D-optimal input to the robust recursive least squares algorithm in order to improve the convergence speed. After that a robust RLS algorithm is introduced with a recursive estimation of scaling factors for the case when besides the impulsive noise sudden changes of the system dynamics also occur. Besides that, another novel algorithm is presented which besides its robust properties against impulse noise also has the ability to track nonstationary changes of the values of estimated parameters. Contrary to the previous algorithm, the proposed approach is based on the design of robust detector of impulse noise, based on the design of robust median filter and the application of either robust or standard RLS-type procedure to the estimation of filter parameters, depending on the detection result. Chapter 6 is dedicated to the analysis of the possibility to apply the proposed adaptive digital filters for signal echo cancellation in telecommunication networks. Let us notice at the end that the algorithms and solutions considered in this book may find application in the wider area of adaptive signal processing, like adaptive cancellation of echo signal, adaptive noise cancellation, adaptive equalization, as well as in processing of signals with various physical nature (speech and image signals, biomedical signals, signals from radars, sonars, satellites, and other intelligent sensors.

Belgrade, May 2012 The Authors Contents

1 Introduction ...... 1 1.1 Conventional Approach to the Design of Digital Filters ...... 1 1.2 Optimal Filters ...... 9 1.2.1 Wiener Filter ...... 9 1.2.2 Kalman Filter ...... 16 1.3 Adaptive Filters...... 27

2 Adaptive Filtering ...... 31 2.1 Introduction ...... 31 2.2 Structures of Digital Filters...... 31 2.2.1 Filters with Infinite Impulse Response (IIR Filters). . . . . 32 2.2.2 Filters with Finite Impulse Response (FIR Filters) . . . . . 34 2.3 Criterion Function for the Estimation of FIR Filter Parameters . . . 36 2.3.1 Mean Square Error (Risk) Criterion: MSE Criterion . . . . 37 2.3.2 Minimization of the Criterion of Mean Square Error (Risk) ...... 39 2.4 Adaptive Algorithms for the Estimation of Parameters of FIR Filters ...... 45 2.4.1 Least Mean Square (LMS) Algorithm ...... 46 2.4.2 Least Squares Algorithm (LS Algorithm)...... 49 2.4.3 Recursive Least Squares (RLS) Algorithm ...... 51 2.4.4 Weighted Recursive Least Squares (WRLS) Algorithm with Exponential Forgetting Factor ...... 53 2.5 Adaptive Algorithms for the Estimation of the Parameters of IIR Filters...... 59 2.5.1 Recursive Prediction Error Algorithm (RPE Algorithm)...... 67 2.5.2 Pseudo-Linear Regression (PLR) Algorithm ...... 72

ix x Contents

3 Finite Impulse Response Adaptive Filters with Variable Forgetting Factor...... 75 3.1 Choice of Variable Forgetting Factor ...... 75 3.1.1 Choice of Forgetting Factor Based on the Extended Prediction Error ...... 76 3.1.2 Fortescue–Kershenbaum–Ydstie Algorithm ...... 78 3.1.3 Parallel Adaptation Algorithm (PA-RLS Algorithm) . . . . 86 3.1.4 Generalized Weighted Least Squares Algorithm with Variable Forgetting Factor ...... 93 3.1.5 Modified Generalized Likelihood Ratio: MGLR Algorithm ...... 96 3.2 Experimental Analysis ...... 101 3.2.1 Comparative Analysis of Recursive Algorithms for the Estimation of Variable Forgetting Factor (Analysis of RLS Algorithm with EGP, FKY and PA Strategy for the Calculation of Variable Forgetting Factor) ...... 101

4 Finite Impulse Response Adaptive Filters with Increased Convergence Speed ...... 109 4.1 Definition of the Parameter Identification Problem ...... 110 4.2 Finite Impulse Response Adaptive Filters with Optimal Input. . . 112 4.3 Convergence Analysis of Adaptive Algorithms ...... 115 4.4 Application of Recursive Least Squares Algorithm with Optimal Input for Local Echo Cancellation in Scrambling Systems ...... 131 4.4.1 Definition of the Local Echo Cancellation Problem in Scrambling Systems ...... 133 4.4.2 Experimental Analysis ...... 134 4.5 Application of Variable Forgetting Factor to Finite Impulse Response Adaptive Filter with Optimal Input ...... 139

5 Robustification of Finite Impulse Response Adaptive Filters ..... 147 5.1 Robust Least Mean Square Algorithm ...... 149 5.1.1 Robustification of Least Mean Square Algorithm: Robust LMS Algorithm ...... 152 5.1.2 Stability Analysis of Robust Estimators ...... 155 5.1.3 Simulation-Based Experimental Analysis ...... 158 5.2 Robust Recursive Least Squares Algorithm with Optimal Output ...... 162 5.2.1 Experimental Analysis ...... 168 5.3 Adaptive Estimation of the Scaling Factor in Robust Algorithms...... 170 5.3.1 Experimental Analysis ...... 177 Contents xi

5.4 Robust Recursive Least Squares Algorithm with Variable Forgetting Factor and with Detection of Impulse Noise ...... 180 5.4.1 Experimental Analysis ...... 184

6 Application of Adaptive Digital Filters for Echo Cancellation in Telecommunication Networks ...... 187 6.1 Echo: Causes and Origins...... 189 6.1.1 Echo in Speech Transmission ...... 189 6.1.2 Acoustic Echo ...... 191 6.1.3 Echo in Data Transfer ...... 192 6.1.4 Basic Principles of Adaptive Echo Cancellation ...... 193 6.2 Mathematical Model of an Echo Cancellation System ...... 196 6.3 Analysis of the Influence of Excitation Signal to the Performance of Echo Cancellation System for Speech Signal Transmission...... 197

References ...... 205

Index ...... 209 Abbreviations

AECM Asymptotic error covariance matrix APE Adaptive echo suppression BP Band pass filter BS Band stop filter EC Echo canceller EE Equation error EGP Extended prediction error EPE Extended prediction error ERLE Echo return loss enhancement FF Forgetting factor FIR Finite impulse response FKY Fortescue-Kershenbaum-Ydstie HP High pass filter IIR Infinite impulse response LAD Least absolute deviation LMS Least mean square LP Low pass filter LS Least square MAD Median absolute deviation MIMO Multiple input–multiple output ML Maximum likelihood MGLR Modified generalized likelihood ratio MLMS Median least mean square MSE Mean square error NEE Normalized estimation error NLMS Normalized least mean square ODE Ordinary differential equation OE Output error PA-RLS Parallel adaptation recursive least square PLR Pseudo linear regression RLMS Robust least mean square RLS Recursive least square RMN Robust mixed norm

xiii xiv Abbreviations

RPE Recursive prediction error RRLS Robust recursive least square RRLSO RRLS algorithm with optimal input RW Random walk SISO Single input–single output SNR Signal to noise ratio VFF Variable forgetting factor WRLS Weighted recursive least square