996 PROCEEDINGSVOL. IEEE, OF 9, THE NO. 66, SEPTEMBER 1978 Relationships Between Digital Signal Processing and Control and Estimation Theory
Invited Paper
Akmcr-The purpose of this paper is to explore several current far deeper level than is possible in this initial effort. To aid le&aeh directions in the fields of digitrl signal procesing and modem others who may wish to follow up on some of the directions cant101 and estimation theory. We examine topics such as stability theory, tineu prediction, and pmuneter identifiition, system synthesis developedin thispaper; an extensivebibliography has been and implementation, twodimensionnl filtering, decentralized control included.In addition, theinterested reader is referred to and estimation, and image processbrg, in order to uncover some of the [ 1621 in which all of these research directions are explored brsic similarities and differences in thegods, techniques, and philosophy in substantially greater breadth and detail. of the two disciplines. Nowhere in thepaper have I madea direct attemptto INTRODUCTION define the fields of digital signalprocessing and control and HE WRITING of this paper was motivated by the belief estimation.Rather, I hopethat by examiningmany of the that the fields of digital signalprocessing and control issues of importance to workers in these fields, the reader will andestimation theory possess enough similarities and be able to piece together a picture of the disciplines and their differencesT in philosophy, goals, and analytical techniques to relationship to each other. As a preface to our examination, merita detailed joint examination. In orderto explore the let me mention several points concerning each field. relationship between these two fields, I found it essential to In digital signalprocessing, one of the crucial problems is concentrate on several specific research directions to provide the design of an implementable system meeting certain given a focus for my investigations. The results of this study were a design specifications such as an ideal frequency response. Here talk delivered during the 1976 IEEE Arden House Workshop the emphasis often is on the word implementable, with a fair on Digital Signal Processing, the present paper, and a far more amount of attention paid to issues such as the structure of the comprehensive manuscript [ 1621. digital filter, its complexity in terms of architecture and com- Although the paper consists of discussions of several specific putation time, the effect of finite wordlength on performance, research directions, the primary emphasis of this paper is not etc. Muchof this attention is motivated by the need for ex- on results. Rather, I have been far m.ore interested in under- tremely efficient systems to perform complex signal processing standing thegoals of the research and the methods and approachtasks (e.g., the implementation of high-order recursive or non- used by workers in both fields. Understanding the goals may recursive filters) at very high data rates (for example, sampling help us to see why the techniques used in the two disciplines rates encountered in speech processing run on the order of 10 differ. Inspecting the methods and approaches may allow one kHz, and higher rates arethe rule in video and radar systems). to see areasin which concepts in one field maybe usefully In control and estimation, the emphasis has been far less on applied in theother. In summary,the primary goal of this implementationand more on developing methodsfor deter- study is to providea basis for future collaboration among mining system design specifications for estimation or control researchers in bothfields. systems. At one level these specifications are just a particular It is hoped that the above comments will help explain the class of design guidelines which can then be used to construct spirit in whichthis paper has been written. In readingthis an implementable digital system.However, there aremajor paper,the reader may find many comments that are either differences between the systems arising in the control context partially ortotally unsubstantiated. These points havebeen and the typical digital processing application. For one thing, included in keeping with the speculative nature of the study. the data rates for control systems are often far lower (e.g., in However, I have attemptedto providebackground for the aircraft control systems sampling rates on the order of 0.1 kHz speculationand have limitedthese comments toquestions are oftenencountered, although much higherrates can be which I feel representexciting opportunities for interaction found in certain casessuch as video-directed systems). More andcollaboration. Clearlythese issues must be studiedat a fundamentally, however, the signal processing to be done in a control system cannot be judged by itself, as can other signal processing systems, since it is part of a feedback loop, and the Manuscript received January 16,1978;revised May 8, 1978. Thiswork effect of the processing must be studied in the context of its wassupported in part by NASA Ames under Grant NGL-22-009-124 closed loop effects. and by NSF under Grant GK-41647, and in part by a Senior Visiting Fellowship to the Department of Computingand Control, Imperial Also, many modern control and estimation techniquesinvolve College, awarded by the Science Research Council of Great Britain. the use of a state-space formulation,as opposed to input-output Theauthor is withthe Department of ElectricalEngineering and Computer Science, Massachusetts Institute of Technology, Cambridge, descriptionswhich are usually encountered in digital signal MA 02 139. processing applications. Some of the reasons for this difference
0018-9219/78/0900-0996$00.750 1978 IEEE WILLSKY:DIGITAL SIGNAL PROCESSING AND CONTROL AND ESTIMATION THEORY 997 will be made clear in the following sections, but one implica- A. State-Space Realizationsand State-SpaceDesign Techniques tion is immediately evident. The use of a state-space descrip- The basicrealization problem (for linearsystems) isas tion implies that the system under consideration is causal. In follows: we are given a (possibly time-varying) description of standardfeedback control problems this is clearly the case, the input-output behavior of a system and thus state-space formulations make a great deal of sense. As we shallsee, there are digital signal processing problems involving noncausal systems or systems in which the indepen- (1) dent variable hasnothing to dowith time and for which causalityhas no intrinsicmeaning. Thus, while wewill find where u and y mayboth be vectors.In the time-invariant several places in which statespace concepts fit in naturally case we have that the sequence of impulse response matrices in the digital signal processing context, we’ll also find others satisfies in which that is decidedly not the case. The preceding comments were made in order to provide the T(k, i)= T(k - i, 0) p Tk-i reader with some insight into the perspective I have taken in and in this case we may be given an alternative input-output writing this paper. With this as background, let us begin our description in the transform domain examination of research topics in the twofields. 00 I. DESIGN, REALIZATION,AND IMPLEMENTATION Y(z)= G(z)U(z), G(z)= Tiz-’. (3 1 In this section we investigate one subject area in which some i=O of the differences in perspective between the two disciplines The realization problem consists of finding a state space model aremost apparent. Specifically, we consider thequestion of design. However, our discussion will not deal very much with x(k + 1) = A(k)x(k)+ B(k)u(k) design methods but rather with the question of trying to pin- y(!c) = C(k)x(k)+ D(k)u(k) (4) point what researchers in the twodisciplines mean by “design” and what sorts of problems their techniques are equipped to that yields the desired input-output behavior (( 1) or (3)) when handle. x( 0) = 0. Perhaps the most obvious difference between the fields‘is in Therealization problem has been studied in detail in the the type of system representations used. In digital signal pro- controlliterature, and one aspect that has received agreat cessing, the emphasis is heavily on input-output descriptions, deal of attention is that of determining minimal realizations- while in control and estimation the emphasis is more on state- Le., models as in (4) with the dimension df x as small as pos- spacemodels. The reasons for this difference stem from the sible. The basic idea here is that a minimal realization has no differentquestions addressed by researchers in the two dis- superfluous states that either cannot be affected by inputs or ciplines.In digital signal processing one isinterested in the do not affect the output. These concepts lead directly to the issue of implementation of a system with aspecified input- notions of controllabilityand observability. In the time: output behavior(hence the need foran input-output de- invariant case, one obtains a rather complete description. Spe- scription).Questions such as efficientimplementation and cifically, we find that the system (3), has a finite-dimensional numberof bits needed to achievethe desired levelof ac- realization if and only if G(z) is rational with each element curacy are of great importance. having nomore zeroes than poles. Furthermore,any con- Onthe other hand, in controland estimation theory the trollable,and observable time-invariant realizationis of minimal issue of implementation is not considered to nearly the same dimension, and any such minimal realization can be obtained extent.Realization techniques do address thequestion of from aparticular one by changeof basis (see, forexample, constructing a state-space realization that leads to a specified [4l, 151, [271, [301). input-output behavior. However, as discussed in the following The algorithm ofHo [ 291 and that ofSilverman and Meadows subsections, such techniques do not address many of the major [ 51 provide methods for extracting minimal constant realiza- issues involved in implementation,and, in fact,state space tionsfrom the Hankel matrix determined by the Ti (see realizations, when viewed as implementable algorithms, don’t Section 11-C and the references for details of these results). include some of the most important system structures that are Thus, if we are given a design specification or plant description used in digital system design. Nevertheless, state-space models in terms of a rational G(z),we can readily determine a minimal do playan important role in control and estimation system realization. On the other hand, ifwe are given G in the form design. Specifically, a state space model for a given physical (3) as opposed to in rational form,partial realization algorithms system, is a necessary ingredient in the application of a number must be used. We will discuss such algorithms in Section 11-C. of techniques for the analysis of system performance and for State-spacerealization algorithms can, in principle,solve the design of feedback control or estimation systems (Le., the certainquestions related to systemimplementation. Specifi- specification of the desired input-output behavior of a control cally, the computation of a minimal realization allows us to or estimation system). determine the minimal number of delays required in imple- Thus, wesee somefundamental differences between the menting a system with a given impulse or frequency response perspectives of researchers in the two disciplines. There also function,and this clearly hasan impact on storage require- clearly exist several areas for interaction between the fields-to ments. Also, one of the most important aspects of the state- develop useful multiinput-multioutput structures (a marriage ofspace approach is that it allows one to consider multiinput- digital implementation and multivariable realization concepts), multioutputsystems and time-varying systems.Since any to utilizestate-space techniques to analyze the performance minimal state-space realization can be obtained from a given of digital filter structures, and to consider the digital imple- oneby change of coordinates, clearlyrealization theory mentation of state-space control andestimation system de- allowssome flexibility in designing good digitalfilter struc- signs. All of these issues are discussed in detail in this section. tures. But it is far from the complete answer, as wewill see 998 PROCEEDINGS OF THE IEEE, VOL. 66, NO. 9, SEPTEMBER 1978 in the next subsection. Not only is memory becoming cheaper + + (thus reducing the importance of minimality), but there are -@ z: other implementation issues besides storage that are of impor- + JQ+ J tance, and one also runs into limitations in interpreting state Unit Delq Unit Delay space realizations as filter structures (see Section I-B). A more important aspect of state-space realizations comes from the fact that they play an extremely important part in a number of control and estimation design problems,where one uses state-spacerealizations to model the system to be Fig. 1. A cascade of two fist-order filters. controlledor the signals to be filtered. By doingthis, one canbring into playextremely powerful state-space tech- niquesfor compensator design 121, [ 61, decoupling of the arebased. But what aboutrealizations from the point of effects ofdifferent input channels [7], etc.,and werefer view of systemsynthesis and implementation? Aswe shall the reader to the specialissue of the IEEE Transactionson see, state-space realizations can play some role, but they are AutomaticControl [27]for anoverview of many design far from providing the entire solution. methodsthat havebeen developed. These design algorithms A widevariety of digital filterdesign methods havebeen allow oneto consider a variety of extremelycomplicated developed to dealwith thesecond issue. Onefactor that multivariablesystem problems within a single framework, does enter into this design question is the number of .storage andthis ability to handle many variables atonce is atthe elements(delays) in thefilter structure, and thusthe issue heart of the value of state-space concepts. of minimality isof some importance. Of course, in dealing One important aspect of some of these techniques is that withsingle-input single-output transferfunctions, one can they allow one to solve quantitativeoptimization problems. readoff the order of a canonic structure andcan construct Thelinear-quadratic optimal control problem is anexample several quite easily bysimple inspection of the specified of this, as is the design of a Wiener filteras a steady-state transfer function. The determination of the order of a canonic Kalman filter [ 81, [ 271. In this case, we begin by modelling realization and the ability to construct several minimal realiza- the observed signal as theadditive white-noise-corrupted tions without much difficulty barely scratches the surface of output of a linear state-space model (a shaping filter) driven the structures problem, however. As pointed out in [ 11, the bywhite noise. Having solved thisrealization problem, the variousfilter structures available may be equivalent from an determination of theoptimal Kalmanfilter is reduced to input-output viewpoint if one did not have to worry about solving a time-varying Riccati equation or a nonlinearalgebraic computation time, the complexity of the digital architecture Riccatiequation for the steady-state (Wiener)filter. Algo- or algorithm required to implement a given structure, the ef- rithms for solving this algebraic equation essentially solve the fect of finite precision in representing filter coefficients, or the Wiener spectral factorization problem. effects of overflow and quantization. These are the issues that In addition to providing a framework for the specification motivate much of the study of various filter structures [I], of designs, the state-spaceframework allows one to analyze [IO], [Ill. the performance characteristics of the overall system after it Letus examine some of theseissues in thecontext of a has been implemented. For example, the techniques described particularlyimportant structure, the cascade form, obtained in Section 111 can be used to study the stability characteristics by factoring a transfer function H(z) as a product of lower of the system. Another analytical tool used to study system order transfer functions. Consider the example performance is covariance analysis. Consider the model z2 +(b+d)z+bd- (1 +bz-')(I +dz-') H(z)= - . (8) x(k + 1) = Ax(k)+ w(k),y(k)= Cx(k)+ u(k) (5) z2 - (a + c)z +ac (1 - UZ-')(I - cz-'1 where w and u are zero mean, independent white noises, with In Fig. 1 we haverealized thisfilter as the cascade of two variances Q and R, respectively.These noises may represent first-order filters. Note that the overall filter is minimal. actual noisesources or the effects of smallnonlinearities, InSection 111 we consider the effects on digital filterper- unmodeledphenomena, etc. A simplecalculation yields an formance of quantizationand overflow on system stability. equation for thecovariances P(k) and S(k) of x(k) and y(k): An alternative,approximate method for evaluating the ef- P(k + 1) = AP(k)A' + Q, S(k) = CP(k)C'+ R. (6) fect of finite word length on system performance is to model eachquantization as if it introduced noise intothe system If A is a stable matrix, we can evaluate the steady-state co- [ 11. By assuming independence of thesevarious sources-a variances P and S by solving the Lyapunov equation ratherstrong and sometimes unjustified assumption (as the existence of periodic effects, i.e., limit cycles, indicates)-one APA' - P = (7) -Q. can in principle evaluate the overallnoise power at the out- put, and thus obtain a measure of the size of the quantization B. The Implementation of Digital Systems and Filters effectsfor any given filterstructure. We can then use this As discussed in [ I], the design of digital systems consists of information to comparealternative designs (see [ 11, [ IO], several parts, including the specification of the desired input- [ 121 for detailed discussions of the uses of such noise analysis output relationshipand theimplementation, usingfinite techniques). As an example,consider the case [ 11of fixed- precision arithmetic, of a system thatapproximates this pointarithmetic and roundoff quantization in which the desiredbehavior. Fromthis point of view, themethods of quantizationinterval 4 is 2-b.In this case, the quantization the precedingsection deal with the first issue. Realization error e introduced by a single multiplication takes on a value proceduresplay an indirectrole in thesetechniques in pro- between +0.5q. If onemakes the assumption that e is uni- viding the state-spacemodels on which the design methods formly distributed, we find that it has zero mean and variance WILLSKY:DIGITAL SIGNAL PROCESSING AND CONTROL AND ESTIMATION THEORY 999 q2/12. Then, for example, in the cascade filter of Fig. 1, one trol and estimation system designs such as those mentioned in couldadd one such noise sourcefollowing each of the four Section I-A. Also, recall thatstate-space realization tech- multiplications. We will returnto this topic shortly to see niques allow one to determine minimal realizations for systems how state-space methods canbe used to analyze noise effects. with multiple inputs and outputs. It is possible that this fact, Anotherextremely important issuein filter designis the combined with a thorough understanding of the relationship sensitivityof filter performance to variation in coefficients. betweenstate-space realizations andvarious digital system This is quitecentral an issue,since onecan only represent structures will lead to the development of useful filter struc- coefficients up to a finite degree of accuracy, and hence one tures for multivariable systems. cannot obtainfilters with arbitrary pole and zero locations. Also, as mentioned in the precedingsubsection, the state- The allowable poles and zeroes and thesensitivity to variations space framework is particularly useful for the analysis of the in parametersdepend quite significantly on the particular propertiesof dynamical systems. Thus,it seems natural to structureunder consideration. For example, parallel and ask if thesetechniques might be usefulin the analysis of cascadestructures are often used because theperturbations various filter structures. In Section 111 we discuss this question in the poles are isolated from one another. withrespect to stabilityanalysis techniques. Also, it is pos- For the remainder of this section, wewish to examine the sible that state-space sensitivity techniques [ 91 could be useful relationship of state-space techniques and concepts to some of in the study of thesensitivity of various digital filterstructures, the questions in digital filter design. Let us first examine the but this awaits further study. use ofstate-space techniques to determinefilter structures. Finally, let us examine the utility of state-space techniques Consider thetransfer function (8). In this case,state-space inthe analysisof theeffect of quantization noise onfilter techniques yield a variety of minimal realizations of the form performance. We do thisby example, although it should be clear that this approach extends to arbitrary structures. Con- sider the cascade structure in Fig. 1 where we add quantization noiseafter each mu!tiplication. A state-spacerepresentation of this system can be written down by inspection:
x(k + 1 = px(k)+ gu(k) + rN(k) If we interpret (9) as analgorithm, we must computethe various products fiixi(k), giu(k), hixi(k) (i,j = 1, 2), and per- y(k)= h‘x(k)+ u(k)+ \kN(k) (1 1) form the appropriate additions. Note that in general, there are eight multiplications and six additions required. where F, g, and h are given in (1O),N(k) is the four-dimensional Nowconsider the cascade structure ofFig. 1. Interpreting noise vector whose components are the noises contributed by it as an algorithm (a and b multiply xl(k),c and d multiply the multiplicationsby a, b, c, and d, respectively.Then x2(k), and we perform the required additions), we see that we \k=.(l, 1, 1, 1),and require four multiplications and four additions, but this is not 1000 themost important difference between the two algorithms, since it is possible to obtain realizations (9) with some zero r=[1 1 1 01 elementsin (F,g, h). However, the crucialdifference is the Ifwe makethe usual independenceassumptions concerning following: if one interprets Q state-space realization QS deter- the components.and time-behavior of N, we can directly apply mining an algorithm of thetype indicated, then there is no the covariance analysis equations (6), (7) to determinethe waythat the Cascade algorithm is of this type! This is not effectof quantization noise on x and y. Notethat (6), (7) to say thatone cannot find astate-space description of the cascade realization. In fact yield the effect of noise throughout the network. The utility of an approach such as this for digital network analysis needs to be examined more carefully, but it appears that it may be computationally superior to other methods, such as those that usesignal flowgraph techniques [ 121 or thatrequire com- ~(k)= [(a + b), (c + d)l x(k)+ u(k) (10) puting a number of partialtransfer functions [3]. We nqte that Parkerand Girard [ 151 used Lyapunov-typeequations is such a realization. and analysis quite similar to our development for the evalua- Thepoint made above may, at first glance, seem to be tionof output noise power dueto correlated quantization trivial, butit is not, since is pointsout that although any errors. In addition, similar analyses have been undertaken by (infiniteprecision) algorithm can be describeddynamically Hwang [ 171, Mullis and Roberts [ 181, and Sripad and Snyder in state-spaceterms, direct interpretation of astate-space [ 191, [ 201. Hwang uses Lyapunov state-space equations to descriptionas analgorithm does not allow one to consider studythe effects of possible structuretransformations and all possible algorithms. That is, it is relatively easy to go from state-amplitude scalings.Mullis andRoberts have obtained an algorithm to a state-space description, but it is not at all some significant results for digital filter design using a frame- natural or clear how to gothe other way, and hindsight is work similar to Hwang’s to studywhat they call“minimal needed in order to interpret a realization of the form of (1 0) noise realizations (see [ 3 1] for further developments). Sripad as a cascade structure. andSnyder develop conditions under which quantization Thus, we see that state-space models have limitations when errorsare in fact white, and they also use Lyapunov-type one considers the issue of implementation. There are, however, analysis to compare the performance of two different realiza- several areas where interaction between the two fields may be tions. Within thisframework, one can pose a numberof of use. First of all, the techniquesused in digital signalprocess- other questions. Forexample, in the caseof floating point ing should be of usein considering the implementation of con- arithmetic, the quantization error depends on the size of the 1000 PROCEEDINGSVOL. IEEE, OF THE 66, NO. 9, SEPTEMBER 1978
signal. Canstate-space procedures foranalyzing “state- synthesis of speech, has received a great deal of attention in dependent noise” [ 161be ofvalue here? Questions such as the past few years [46], [ 52]-[ 541. We will use this problem these await future investigation. as the basis for our discussion of the identification question. Ourpresentation is necessarilybrief andintuitive, and the In this section we have seen some of the issues involved in reader is referred to these references for details. As discussed system design in the two fields. The issue of implementation in [ 521 a popular approach is to model a discretized speech is at the very heart of the problems considered by researchers signal {y(k)} as the output of a linear system, which, over short in digital signalprocessing, while researchers in controland enough intervals of time, can be represented by a time-invariant estimation have concentratedmore on the development of transfer function C(z). Here the input is taken as a periodic general design procedures for state-space models and methods pulse train (whose period is the pitch period) forvoiced sounds foranalyzing the characteristics of suchmodels. We have (such as vowels) and as white noise for unvoiced sounds (such seen thatthere are points of overlapand places in which as the sound “sh”). In addition, a common assumption is that techniques and concepts from one discipline may be of value G isan all-pole filter, whichleads to anautoregressive (AR) in theother. State-space techniques may be useful in the model analysis of multivariable structuresand in the analysis of sensitivity and quantization noise behavior of different struc- y(k)+aly(k-l)+...+apy(k-~)=u(k) . (13) tures. Such issues remain to be studied, but it is in the other This assumption has been justified in the literature under many direction that there is the most to be done. The issues involved conditions, although strongnasal sounds require zeros [ 5 21 in the digital implementation of systemsspecified by state- . The problem now is to determine the coefficientsa,, . . , up. spacedesign methods remainlargely unexplored.Numerous - Having these coefficients, one is in a position to solve a number problemsabound. Whatis theeffect of roundoff noise on of speech analysis and communication problems. For example, closed-loop controller performance, and how many bits must one can use the model (13) to estimate formant frequencies we use to achieve the desired regulation properties [ 2 1 ] , [ 221 , and bandwidths, where the formants are the resonances of the [241;[25], [ 28]? It iswell knownthat “optimal” con- vocal tract [ 551. In addition, one can use the model (13) for trollers and estimators require many arithmetic operations and efficient coding, transmission, and synthesis of speech [ 601. hencelead to lowsampling rates. Canwe improveoverall The basic idea here is the following: as the model (13) indi- performance by usinga simpler “suboptimal”system at a cates, the speechsignal y(k) contains highly redundant in- highersampling rate [ 131 ? Ifwe are controlling a complex formation, and a straightforward transmissionof the signal will system,“optimal” controllers require not only a great deal require highchannel capacity for accurate reconstruction of of computation,but also thecentralized processing of all speech. On the other hand, onecan interpret (1 3)as specifying information, and the cost of relaying information to a central a one-step predictor for y(k) in terms of preceding values of location may be prohibitive.Can we devicedecentralized y (assuming u(k) = 0). As discussed in [ 601, one often requires control architectures that take advantage both of, the struc- far fewer bits to code the prediction error u than the original ture of the dynamics of the system being controlled and the signal y. Thus, one arrives at an efficient transmission scheme capabilities of the available types of digital processors?Here (linear predictive coding-LPC): given y, estimate the ai, com- again, if we include the cost of informationtransfer, “sub- pute u, transmitthe ai and u. At the receiver, we then optimal” decentralized systems may outperform the “optimal” canuse (13)to reconstruct y. An alternativeinterpretation system (see [ 141, [ 231, [ 261 for some results and problems of this procedure is the following: given y, estimate G, pass concernedwith parallel and distributed processingand de- y throughthe inverse, all zero(moving average-MA) filter centralized control). l/G(z), transmit the coefficients in G and the output of the The study of problems such as these-i.e., the interaction of inversefilter. At the receiver, we then pass the received implementation and architecture issues and the design of con- signal through G to recover y (thus this procedure is causal, trol and estimation systems-is still in its infancy, and it ap and causally invertible). pears tooffer an extremely promisingavenue forresearch. The question remains as to how one estimates the ai. The We note that architectural issues have received a great deal of most widely used technique in the literatureis linearprediction. attention in the field of digital signal processing [ 101, [ 121, Using theinterpretation of (13), asspecifying aone-step andthis, together with the wealth of literatureon digital predictor for the signal y, we wish to choose the coefficients filter structures, indicates that thereis much to be gained from al , * * * ,ap to minimize the sum of squares of the prediction future interaction and collaboration. errors e(n)= y(n)- ?(PI), n E I. Here we assume that we are given y(O), . . ,y(N - l), while the set I can be chosen in dif- 11. PARAMETERIDENTIFICATION, LINEARPREDICTION, ferent manners, and wewill see in the following subsections LEAST SQUARES,AND KALMAN FILTERING that differentchoices can lead to differentresults and to A problem of greatimportance in manydisciplines is the different interpretations. determination of the parameters of a model given observations Beforebeginning these investigations, letus carry out the of the physical process being modeled. In control theory this minimizationrequired in linearprediction. Taking the first problem is often called the system identification problem, and derivative with respect to the ai of the sum of squared errors, it arises in many contexts. The reader is referred to thespecial and setting this equalto zero, we obtain the normal equations issue of the IEEETransactions on Automatic Control [41] and tothe surveypaper of hrom andEykhoff [421 for detailed discussions and numerous references. Parameter identification problems also arise in several digital signalprocessing applications. Several examples of such problems are given in the special issue of the PROCEEDINGS OF THE IEEE [64], and one of these, the analysis, coding, and WILLSKY:DIGITAL SIGNAL PROCESSING AND CONTROL AND ESTIMATIONTHEORY 1001
These equations aretypical of thetypes of equationsthat What does this mean as far as all-pole modeling via linear pre- arise inlinear, least squares problems, and their efficient diction goes? The answer to that is not much. In the all-pole solution has been the topic of many research efforts. modeling problem, we are equivalently only interestedin design- ing a FIR filter-i.e., a prediction filter that produces the best A. The Autoconelation Method, Kalrnan Filtering for estimate ofy(n)given the “data window”y(n- l), .. * , y(n - p). Stationary Process, and Fast Algorithms The coefficients of such a filter areprecisely a(l Ip), . . . , a(plp), Suppose we let I = all integers, where we define y(n) = 0 for and it does not matter (except from a computational point of n < 0, n 2 N. In this case, we find that view) that these coefficients were generated as part of a time- varying filter weighting pattern. N-l-li-jl On the other hand, the time-varying weighting pattern inter- cii = 1 y(n)y(n + Ii - gr(Ii - (16) pretation is extremelyimportant from a statistical point of n=o view,especially if one wishes to designrecursive predictors and the normal equations becomeTu = d, where a’ = (al, * * * ,up), that arecapable of incorporating all pastmeasurements and
(here the prime denotes transpose), d‘=(-r(l),-r(2); a, -r(p)), not just a data window in the case when y has a Markovian and T is a symmetric Toeplitz matrix [ 341, [ 521, [ 591, [68], representation [ 731 (Le., theijth element depends only on li - jl) with x(k + 1) = Ax(k) + w(k) y(k) = c’x(k) (20) Tjj = ci,.We also note [ 461 that if y is a stationary random process with autocorrelation where x is a random n-vector, A is a constant n X n matrix, c is aconstant n-vector, and w is a zeremean uncorrelated R(i) = E[y(nly(n(17) + i)l sequencewith covariance Q. Thecorrelation coefficients and if we want to find the one step prediction coefficients to of y can be computed by direct examination of (20). We note minimize E[e2(n)], we obtain anidentical set of (Toeplitz) that x and y will be stationary with equationswith r(i) replacedby R(i). This statistical point ~(i)=ci’r~c, i 2 0 (21) of view is extremely useful for obtaining certain insights into the approach. if A is stable andif n, the covariance of x, satisfies the Lyapunov The solution of such Toeplitz equations has been the subject equation of a great deal of attention in the mathematical, statistical, and AIIA’ - = -Q. (22) engineering literature [34],[35], [46], [48], [49]. An n efficient algorithm was proposed by Levinson [ 481, improved We now wish to designan optimalpredictor for recursively upon byDurbin [63] andstudied in the speechprocessing estimatingy(n). This is astandard estimation problem, and context by several authors, including Itakura and Saito [ 541. the solution is the Kalman filter [ 35 ] : The method essentially consists of solving forward and back- ward predictionproblems of increasing size in a recursive ?(n) = A?(n - 1) +AK(n - l)y(n - 1) 9(n) = c’x(n) manner.That is, thealgorithm computes the coefficients r(n- l)=y(n- 1)-9(n- 1) (23) a( 1 Ii), . . . , a(ili) for the best prediction of y(n) based on y(n - l), . * . , y(n - i) and the coefficientsb( 1 Ii), * . * , b(i1i) for where the time-varying gain satisfies the bestprediction of y(n - i - 1) based on y(n - i), * . * , y(n - 1). The algorithm iterates on i. As a part of thisalgo- P(nln - 1)c K( n ) = rithm, at eachstage onecomputes the prediction error (for c’P(nln - I)C’ bothforward and backward prediction), and thus one can determine when to stop based on the size of this quantity. Here P(nln - 1) is the covariance of theprediction error Also, at each stage we must compute a coefficient ki, which x(n)- ?(n), is known as the partial correlationcoefficient (see (461, [ 521, AP(nIn - l)cc’P(nln - 1)A‘ [ 541).Finally, while the Levinson algorithm is extremely P(n+ lIn)=AP(nln- I)A’+Q- efficient,it must be keptin mind thatthe computational c’P(nIn - 1)c savings reapedhere may be quite smallcompared tothe (25) calculation of the correlations according to( 16). Let us nowexamine what this algorithm means from a Let us make a few comments about these equations. Note statisticalpoint of view. Thealgorithm specifies estimators that the filter innovationsr(n) is precisely the prediction error of the form e(n), and its variance is c’P(nln - 1)c. Also, recall that in the all-pole framework, we could alternatively view the prediction i y^(i)= - Ca(jli)y(i - j) (18) fiiter asspecifying an inverse filter, which tookthe y’s as j=1 inputs and produced the uncorrelated sequence of prediction errors as the output. In the context of the Kalman filter, the i analogousfilter is the innovations representation (seerepre- 9(0)= - b(jli)y(j). (19) sentation IR-1 of [58]), in which we view the output of (23) j=1 asbeing y(n). Finally, note that one can computethe pre- Thus, we can think of the algorithm as providing us with the dictorcoefficients a(jli) as the weighting pattern of the time-valying coefficients of the weighting pattern of the opti- filter (23). mal one-step predictor andof the optimal initial time smoother. Examining (23)-(25), we see that the computation of the Note that these coefficients are, in general, time-varying (i.e,, recursivefilter coefficients requires the solution of the (dis- a(ili) #a(j)), since the mechanism of prediction is time- cretetime) Riccati equation (25). If x is ann-vector, then varying when one bases the prediction on only a fiite set of (using the fact that P is symmetric), (25) represents n(n + 1)/2 data. equations.For reasonably largevalues of n, this can be an 1002 PROCEEDINGSIEEE, OF THE VOL. 66, NO. 9, SEPTEMBER 1978 extreme computational load, especially given that all that is hasvarious y(i) asits elements. Suppose we have a^(N- 1) needed for the filter is the n X m gain matrix K (when y is and we now obtain the new data point y(N). We would like m-dimensional).Thus when m <
x(n + 1) = Ax(n) + w(n)y(n) = c’x(n) (3 1) where w is a zero-mean white-noise process with covariance Q. Refemng to (20)-(22), we see thatthis is equivalent to LRk- 1) R(N) R(N+ 1) - * R(2N-2) J finding a factorization: It is well-known [67] (see also Section I-A) that admits a R(i)= ch‘b, i 2 0 (32) R factorization (32) if and only if there is some integer n such b=P c APA‘-b=Pc P=-Q. (33) that rank HN n, for all N. (38) Examining (32) and (33), we see thatthe algorithm falls < naturally into two pieces: (i) find a triple (A, b,c) satisfying Ho’s original algorithm yielded a minimal realization (i.e., dim (32); (ii) fiid P and Q satisfying (33). One of the best known A in (32) is as small as possible) if a bound n was known in studies of this problem is that of Faurre [44], [ 561. As he advance. A far more critical question (from a practical point pointed out, the first step of the algorithm is simply the well- of view) is the partial realization question. Here we take into knowndeterministic realization problem when one is given account that we only have available a finite number of correla- the “weighting pattern” R(O),R( l), R(2), . . . This problem tions R(O),R( I), , R(N - l), and one would like to obtain hasbeen widely studied in theliterature f301, [3 11, [371, theminimal factorization that matches these. Onecan use [40], and we will make a few comments about this aspect of Ho’s algorithm for this, butit is not recursive-i.e., if we incor- the problem in a few moments. Before discussing the numerical porate R(N), we mustre-solve the wholeproblem. Fortu- aspects of the first step or the details of the second, let us see nately, Rissanen [ 391 and Dickinson et al. [ 371 have developed what the first part yields in the frequency domain [ 571. Let efficient, recursive procedures (the latter of which is based on the Berlekamp-Masseyalgorithm [38], which was developed +a Sy(z)= R(i)z-i. (34) for the scalar case). We note that these algorithms essentially i=-m solve the Pade approximationproblem, and we refer the reader to thereferences for details. Then, we see that the factorization (32) yields Thus,efficient algorithms exist for spectral factorization and one would expect good results if the process y truly has Sy(z)=c’(zZ- A)-’zb + c’(z-’I-A)-’Ab. (35) a Markovian representation and if one has the exact values of Defining a(z)= det (zl- A)and putting the two terms in (35) thecorrelations. Thispoints out conceptuala difference over a common denominator, we see that the first step in the between linear prediction and the above stochastic realization algorithm yields procedure. In linearprediction, no pretense is madeabout exactly matching a model. All that is wanted is a least squares fit, and thus one would expect this procedure to be relatively robust when one uses a finite record of real data to generate an estimate of the correlation function which is then used in That is, we have obtained a factorization of the denominator thelinear prediction procedure. On theother hand, it can of S,. If we can also factor the numerator we will have deter- easily beseen that an infinitesimalperturbation of HN in mined the desired transferfunction @(z)/a(z),which, when (37) can make it havefull rank. In this case, inthe partial driven bywhite noise,yields the spectrum Sy(z). It is clear realization procedures-which in essence are looking to match from (31) that it is the second part of the spectral factoriza- a model exactly-will yield a system of extremely high dimen- tion that is accomplished by the second step of the stochastic sion. Thus, it appears that thesealgorithms are inherently realizationalgorithm. Finally, note that the modelobtained sensitive to errors in estimates of the correlation coefficients. contains both poles and zeroes. In addition, if y has no Markovian representation, the linear There are several methods for performing the second step of prediction approach may yield a useful fit, while the partial the algorithm. Faurre [44] showed that (33) could be solved realization procedure may very well run astray as it tries to for values of P inside a given range,and he identified the fit the data “too closely”. smallest such covariance,P, , as that arising from aninnovations Does this mean that the aboveprocedure is of no usein representation of y-i.e., a steady-state Kalman filter. Thus to identifying parameters in a speech model? The answer to that completethe second step wecan either solvean algebraic is perhaps not. What is needed is a modification of the first Riccati equation or can use the “fast algorighms,” as described stepof the stochastic realization algorithm. As the version in subsection 11-A to compute the time-varying Kalman gain. described here stands, it is too sensitive, and, in fact, DeJong Lettingthe transients die out, we thenobtain the desired [68] has shown that these methods are numerically unstable steady-statefilter. Although this approach involvessolving in thatthe inexact minimal realization supplied by these for the entire gain time history, this procedure may be faster algorithms, as implemented on a finite wordlength computer, than direct solution of the algebraic Riccati equation. may not be a“numerical neighbor” of the sequence {R(i)} Let us now turn to the numerical aspects of the first stage- that is to be factored. By rephrasing the algorithm in terms of i.e., the computation of the factorization (32). The algorithms €-rank-the least rank of all systems within an “€-neighborhood” of Rissanen [39] and Ho [29] arebased on the examination of the given sequence-DeJong obtains a slower algorithm that 1004 PROCEEDINGSVOL. IEEE, OF THE 66, NO. 9, SEPTEMBER 1978 is similar to Rissanen’s but is numericallystable. This ap- 111. STABILITYANALYSIS proach is extremelyappealing for two reasons:(i) we can, In the field of digital signal processing, stability issues arise withinthis framework, seek minimal realizations in the when one considers the consequences of finiteword length €-neighborhood of a sequence {R(i)}that itself is not realizable in digital filters. On the one hand, a digital filter necessarily by a finite dimensional system; (ii) we can seek the “nearest” has finite range, and thus overflows can occur, while on the reduced-order realization of given dimension of a given system: other, one is inevitably faced with the problem of numerical In additionto the work of DeJong,a number of other quantization-roundoffor truncation. Since the fiiterhas methods havebeen proposed for“approximate” Padeap- finite range, the question of the state of the fiter growing proximations [62],[69], [ 701.One interesting possibility withoutbound is irrelevant.However, the nonlinearities is the all-pole approximations-i.e., we perform linear predic- in the filter, introduced by whatever form of finite arithmetic tion on the R(i). Thiswould require computing the correla- is used,can cause zero-input limit cycles and can also lead tion of the R(i)! (Notethat an all-pole assumptionhere to discrepanciesbetween the idealand actual response of would nor necessarily lead to an all-pole model in (3 1). the fiter to certain inputs. Following the discussions in [ 761, [821, the typical situation can be describedas follows One of our goals in thissection has been topoint out a number of similarities between the goals and techniques of the x(n + 1) =Ax(n) +Bu(n) y(n)= Cx(n) two disciplines. We havealso seen some of thedifferences, butothers have not beendiscussed. In particular,in this =N(x(n)) x(n) (39) section we have treatedidentification for identification’s where N is anonlinear, memoryless function that accounts sake. As pointed out in [42] in control system design. identi- for the effects of overflow and quantization. If one assumes ncation is often simply a means toward the goal of efficient that the associated linear system(Le., N = identity) is designed control.Thus, in manycontrol applications, the value of to meet certain specifications, one would like to know how identification is not measured by the accuracy of the param- the nonlinearity N affects overall performance. For example, eter estimates, but rather by the performance of the overall fiiterdesigners areinterested in determining bounds on the system. In addition, in controlthere are several types of magnitudes of limit cycles and in finding out how many bits identificationproblems, since one has theopportunity to one needs to keep the magnitudes of such oscillations within excite the systemthrough inputs. Different problems arise tolerable limits. if thesystem is operatingopen loop, in a time-invariant Onthe other side, a typical feedback control system is closed-loopmode, or in an adaptiveclosed loop mode. We described by referthe reader to [41], [43] formore on thissubject and forfurther references. In addition, in manyon-line control Y = Gl(e), e = u - G2(y) (40) problemsthe number of parameters to be identified is not verylarge-four or five. In fact, one of the key problems in where the input u, the output y,and the errore are functions practicaladaptive control is the carefulchoosing of which of time, and GI and Gz represent the dynamics of the forward few parameters to identify. andfeedback paths, respectively. In control theory one is On the digital filtering side, one is often interested in the interestedeither in the analysis orthe synthesis of such accuracy of theparameter estimates. This is of importance, systems. In the synthesis problem we are given an open loop for example, if one is attempting to design an all-polefilter system GI andare asked to designa feedback system G2 that matches a given impulse response in a least squares sense, such thatthe overallsystem has certain desirable stability or if’one is attempting to estimate formants from an all-pole properties.In the caseof stability analysis, onemay be speech model. On the other hand, for linear predictive coding, interested either in the driven or the undriven characteristics. the accuracy of the parameters may be of secondary interest, In thedriven case the problem involves determining if bounded while the primary concern is more efficient coding of speech inputs lead to bounded outputs and if small changes in u lead data. In this case, accuracy is of importance only in so far as to smallchanges in the y. In the undrivencase, we are in- itmakes the coding scheme more efficient. Also, in the terestedin seeing if the systemresponse decays, remains speech problem, we are usually dealing with many unknown bounded,or divergeswhen theonly perturbing influences parameters-between twelve and sixteen [ 521. are initial conditions. Withregard tothe speechproblem, we notethat linear It is clear that the problems of interest to researchersin prediction hasproven to be aparticularly appropriate tool both disciplineshave a good deal incommon, and, as we fora variety of reasons,ranging from the fact that the all- shall see, workers in each area have obtained results by draw- polemodel is often a realistic one to the property that the ing from very similar bags of mathematical tricks. However, linearprediction procedure tends tomatch the spectral en- thereare differences between the methods used andresults velope of the data [ 46). In this section we have described a obtainedin the two areas. In the analysis of digital filters number of relatedidentification concepts (see [ 1621for thework has been characterized by the study of systems more), some of whichmay be useful in solving problems in containingquite specific nonlinearities. In addition, much speech analysis, such asenhancement of noise-degraded speech. of the work has dealt with specific filter structures. In par- We have also pointed out a number of questions concerning ticular,second-order Titers havereceived agreat deal of some of these methods, such as the need for detailed numerical attention[751, [761, 1801, [821 sincemore complex filters analyses of the many “fast” algorithms, and the necessity of canbe built out of series-parallel interconnections of such further analysisand experimentation to assess whetherany sections. Also, the class of wave digital filters[78], [79] of these techniques can improve upon the performanceachiev- have been studied in some detail. Studies in these areas have able in speech processing using linear prediction. yielded extremely detailed descriptions of regions of stability WILLSKY:PROCESSINGDIGITALSIGNAL AND CONTROL AND ESTIMATION THEORY 1005
inparameter space and numerous upper and lower bounds Motivatedby these analogies, Fettweis[78] defines the on limit cycle magnitudes (see [76], [77], [86], [97]-[99]). notion of “instantaneous pseudopower,” which is a particular In control theory, on the other hand, the recent trend has quadraticform in the state of the WDF.By defining the been in the development of rather general theories, concepts, notion of “pseudopassivity” of such a fiter, Fettweis intro- andtechniques for stability analysis. A number of rather duces the notion of dissipativeness. With this framework, the powerfulmathematical techniques havebeen developed, pseudopowerbecomes a natural candidate for a Lyapunov but there has not been as much attention paid to obtaining function,and in [79], Fettweis and Meerkotter are able to tightbounds for specific problems. In addition, problems applystandard Lyapunov arguments to obtainconditions involving limit cycles have not received nearly as much atten- onthe arithmetic used thatguarantee the asymptotic sta- tion in recent years as issues such as bounded-input bounded- bility of pseudopassive WDF’s. Theintroduction of the outputstability and global asymptoticstability (although concept of dissipativeness in the study of stability is an often- there clearly is a relationship between these issues and limit usedidea (see the note of Desoer [ 87]),-and a number of cycles). important stability results have as their basis some notion of passivity. We also notethat the use of passivity concepts A. The Use of Lyapunov Theory andthe tools of Lyapunovtheory appear to be of some Thetechnique of constructingLyapunov functions to valuein the development of new digital filter,structures prove thestability ofdynamical systems has been used by that behave well in thepresence of quantization [801. researchersin both fields(see [95]for detailsand further Lyapunovconcepts have found numerous applications in discussions).Consider asystem withstate x(k) andwith controltheory. The construction of quadratic Lyapunov equilibriumpoint x = 0. A Lyapunovfunction V(x) for equationsfor linear systems is well understoodand is de- this system is a scalar function for which V(0)= 0 and which scribedin detail in [95]. The key result in this area is the is nonincreasingalong system trajectories (i.e., V(x(k)) is following. Consider the discrete-time system nonincreasing as a functionof time). If thisfunction has some additional properties, wecan x(k + 1) =Ax@). (4 1) prove stability or instability. Basically, we think of V as Thissystem is asymptoticallystable if and only if for any an“energy” function. One then obtains results depending positive definitematrix L, thesolution Q ofthe (discrete) uponhow energy behavesalong trajectories. Intuitively, if Lyapunov equation V is everywherepositive except at x = 0 and V(x(k)) de- creases monotonically,the system dissipates energy and is A’QA - Q = -L (42) stable. On the other hand, if V(xo)< 0 for some XO,then is alsopositive definite. In this case thefunction X’QXis the systemcannot be asymptotically stable, since thenon- aLyapunov function that proves the asymptotic stability increasingnature of V(x(k)) guarantees thatthe system of (41).Note that this result provides a variety of choices can’t approachthe zero energy state if startedat XO. One for Lyapunov functions (we can choose any L > 0 in (42). advantage of Lyapunov-typeresults is thatthe hypotheses Parker and Hess [86] obtain their bounds by choosing L =I for resultssuch as thosejust mentioned canbe checked (here (41) represents the ideal linear model). Tighter bounds withoutthe construction of explicitsolutions to difference might be possible with other choices of L, but, as they men- or differentialequations. However, themajor problem with tion, it is not at all clear how one would go about finding a thetheory is the difficultyin finding Lyapunov functions “better” choice (other than by trial and error). in general. In addition to their direct use in specific applications, one Withrespect to the limitcycle problem, Willson [75], of the most important uses of Lyapunovconcepts is as an [81]has utilizedLyapunov functions to determinecondi- intermediate step in the development of other more explicit tionsunder which second order digital filters will not have results. For example, the stability of optimal linear regulators overflowlimit cycles and will respond to “small” inputsin withquadratic criteria and of optimal linear estimators can a manner that is asymptotically close to the ideal response. beproven by constructing particularly natural quadratic Parker and Hess [ 861 and Johnson and Lack [98], [99] have Lyapunov functions [ 721, [ 1051. A further use of Lyapunov used Lyapunov functions to obtain bounds on the magnitude theory has been to provide a framework for the development of limit cycles. In each of these the Lyapunov function used of manymore explicit stability criteria. Examples of these was a quadratic form which in fact proved asymptotic stability are a number of the frequency-domain-stability criteria that for the ideallinear system. In Willson’s work [81], he was havebeen developed in the last 10 to 15 years(see [ 831 - able to showthat hisresults were in some sense tight by [85], [88], [89], [93], [94]). Theseresults are the subject constructingcounterexamples whenhis condition was vio- of the next subsection. lated.In [86], (981, [99] the bounds are not as good as others that havebeen found, and, as Parker and Hess state, this may be due to the difficulty of determining which qua- B. Frequency Domain Criteria, Passivity, and Lyapunov dratic Lyapunov function to use. As pointed out by Claasen Functions et al. [76],it appears to bedifficult to findappropriate We have already mentioned that the notion of passivity is Lyapunovfunctions for the discontinuous nonlinearities of importance in stability theory and have seen that Fettweis that characterize quantization. andMeerkotter havebeen able to utilizepassivity notions There is a class of digital fdters-wave digital filters (WDF) to study certain digital filters via Lyapunov techniques. The [ 781, [ 791 -for which Lyapunov techniques are particularly relationship between passivity, Lyapunov functions, and many useful. Such filters have been developed by Fettweis so that of the frequency domain criteria of stability theory is quite they possess many of the properties of classical analog filters. deep,and in this subsection wewish to illustratesome of 1006 PROCEEDINGSIEEE, OF THE VOL. 66, NO. 9, SEPTEMBER 1978 these ideas. We refer the reader to the work of J. C. Willems tains Popov'sstability condition [89]and the discrete-time [92], [96], [ 1031, in particular, for a detailed development. analog due to Tsypkin [85], [93] : suppose fis nondecreasing Thegeneral development in thesereferences is beyondthe and is strictly inside thesector [0, k].Then the feedback scopeof this paper, but wewill indicatesome of the basic system is finite gain input-outputstable if thereexists an ideas for a discrete-time system, denoted by the symbol G, a 2 0 such that with input u and output y. In this case, one can define input- I output stability as Re [(l +a(l - e-i"))C(eiw)] +- 20, QUE [0, 2771. k m uf<- 2 yf<- (43) (47) i= 1 1 i= Claasen et al. [82] have obtained direct analogs of (46) and Le., if the input has finite energy, so does the output. If we (47) for the absenceof limit cycles of periodN: can make the stronger statement 1 Re [C (eiZn'lN)I+: >0, l=O, l,***,N-(48)1 112 ( 5 yf)li2 < K( 5 ut) (44) i= 1 i= 1 or theexistence of ap2 0 such that we call K the input-output gain. A system is called passive N- 1 if (strictly passive if there is an E > 0 such that) Re1 + a (1 - eiznzp/N) [ p=1 N N N (45) (49) i=l Here f is inside the sector [O,._ k]. and is also nondecreasing inthe case of (49).The proofs given in[82] relyheavily Themotivation for the definition (45) stems from the in- onthe passivityrelations (48),(49) and an application of terpretation of the left-hand side of (45) the total energy as Parseval's theorem in order to contradict the existence of a input to the system.Thus a passive systemalways requires limit cycle of period N. This last step involves the assumed apositive amount of energy befed into it. This notion to periodicity in a crucial way, but the application of Parseval hasextremely strong ties to the usual notions of passivity and the use of the positive real relationship (48) is very rem- and dissipativeness forelectrical networks and is, infact, iniscentof stability arguments in feedback control theory a natural generalization of these concepts [ 1031, [ 1071. [92]. In the proof of (49) the monotonicity of f & used in Having thisframework, one can derive importantresults conjunctionwith a versionof the rearrangementinequality onthe stabilityand passivity of feedbackinterconnections [91], [ 921which has also been used to studystability of ofpassive systems(see [ 103]),much like the results of feedback control systems. Fettweis for his pseudo-passive blocks. As outlined by Willems As mentionedat the end of the precedingsubsection, in [ 1031,there are three basic stability principles. The first many frequency domain results canbe derived with Lyapunov- involves the interconnection of passive systems as mentioned typearguments. We have.also seen in this subsection that above, while the second is the small loop gain theorem (sta- many of these results can be derived via passivity arguments. bility arises if the gain around the loop is less than unity-a Clearly the two are related, and the crucial result that leads result used in the digital fiiter context in [ 1041 ). The third to this relationship is the Kalman-Yacubovich-Popov lemma result involves notions of passivity and of sector nonlinearities. [ 1001, [ 1011, [ 1031, which relafes the positive realness of A nonlinearity is inside (strictlyinside) the sector [a, b], if certaintransfer functions tothe existence of solutions to its graph is bounded by (strictly contained within) the lines particularmatrix equalities and inequalities. Kalman [ 1011 y = ax and y = bx. the effective gain of this nonlinearity Thus, utilizedthis result to obtaina Lyapunov-type proof of the is between a and b. As an example, the operation of round- Popov criterion, and Szego [ 1001 used a discrete-time version off is inside the sector [0, 21 (see [ 761, [82] for the sector to obtain a Lyapunov-theoretic proof of Tsypkin's criterion. characteristics of otherquantizers). To indicate how sector We also note that the positive real lemma plays a crucial role nonlinearityconditions can beused, consider (40) with GI in several other problem areas including the stochastic realiza- specifiedby a stablediscrete time transfer function C(z), tionand spectral factorization problem [44] and the study and Cz amemoryless nonlinearity, assumed to beinside f, of algebraic Riccati equation [ 1021. the sector [0, k]. this case, the general sector input-output In Finally, we notethat many of thesepassivity-Lyapunov stabilitytheorem reduces to showingthat (Cl (l/k)) is a + results haveinstability counterparts (e.g., see [74],[901). passive system, and, as developed in [92], [ 1031, this will be Such results may be useful in developing sufficient conditions the case if and only if C(z) (1 /k)is positive real. + for the existenceof limit cycles. 1 Re (G(eiw))+ - > 0, w E [0, 2n] k sectionthis In (46) we consideredhave some of the aspectsof stabilitytheory that point out the relationship among the which is Tsypkin'sstability criterion [ 1061.techniques, goals, andresults of researchers in both disciplines. A variant of this type of result involves the use of multipliers As we have seen, many of the results in the two disciplines [92] in which one modifies the feedbacksystem of (40)by involve the use of verysimilar mathematicaltools. On the inserting a second system in the forward path and its inverse other hand, the perspectives and goals of researchers in the in thefeedback path. Onecan thenapply the basic stabilitytwo fields are somewhat different. The development of a results to the modified G1 and C2. Inthis manner one ob- mutualunderstanding of these perspectives and goals can WILLSKY: DIGITAL SIGNAL PROCESSING AND CONTROL AND ESTIMATION THEORY 1007 only benefit researchers in both fields and is in fact absolutely crucial forthe successful studyof certain problems. For example, in theimplementation of digital controlsystems one must come to grips with problems introduced by quan- tization. Digital controller limit cyclesat frequencies near the resonances of theplant being controlled canlead to 000.....00 9 - -, - - - - - seriousproblems. Inaddition, the use of a digital filterin k 001l0000000 afeedback control loop creates new quantization analysis 0000000000 problems.Finite arithmetic limit cycles can occur only M in r I recursive (infiniteimpulse response) filters. However, if a nonrecursive (finite impulse response) fdter is used in a feed- Fig. 2. Support of a fust quadrant of “Northeast” (NE) function. Pos- sible nonzero locations are indicated by solid dots. backcontrol system, quantization errors it produces can lead to limit cycles of the closed-loopsystem [ 1041.How can one analyze this situation, and how does one take quan- tizationeffects into account in digital controlsystem de- sign? Questions such as these await future investigation. IV. MULTIPARAMETER SYSTEMS,DISTRIBUTED PROCESSES, AND RANDOM FIELDS A growing interesthas developed over the pastfew years into problemsinvolving signals andsystems thatdepend on (a) North (b) Earl (c) NE morethan one independent variable.In thissection we Fig. 3. Several possible directions of recursion for (52). considerseveral problem areas involving multiparameter signals andsystems in order to examine some of the key issues that arise. Foran up-to-date viewof some of the Note that from (52) and the figure, it is evident that we must research in this area, we refer the reader to the recent special store values of y(k, I) for (k,2) to the south and west of the issue of the PROCEEDINGS OF THE IEEE [ 1561. domain over which we wish to calculate y. If this domain is infinite in either direction, the required storage isalso infinite. A. Two Dimensional Systems and Filters In fact the required storage grows linearly as we increase the Inanalogy with the 1-D case, a2-D linear shift invariant domainin either direction (see [ 1081for details). Thus (LSI) system can be described by a convolution of the input storagerequirements in 2-D depend on far more than the x(m, n)and the unit impulse response h(m, n). Alternatively, order (M, N)of the filter. taking 2-D z-transforms, we obtain We also find that the storage requirements depend on the sequencingof the recursion.Several directions of recursion Y(z1, zz)=H(z~,Zz)X(Zl, ZZ). (50) areindicated in Fig. 3. Each recursion calls forits own se- Of special interest are the rational system functions, H =A/E, quence of data accessingand discarding. The N and E re- which arise from 2-D difference equations such as cursions appear to have particularly simple sequencing rules, but the data must be processed serially. On the other hand, the NE recursion has amore complex sequencing but leads to the possibility of parallel computation, since, for example, points 4, 5, and6 can be calculatedsimultaneously. The possible directions for recursion and potential use of parallel computation can be determined with the aid of a conceptual device-the precedence relation [ 1331, which partially orders Here Il ,I2 are finite setsof pairs of integers. points with the rule (m, if y(m, must be cal- Let us firstdiscuss theproblem ofrecursion. Given the n) < (I, k) n) culated before we can calculate y(2, equation (51), we want to use it to calculate the next output k). Let now return to the question of recursibility. Clearly given previous outputs andthe input. Unlike the1-D case, us the picture is symmetric-i.e., we can have NW, SE, and SW inwhich the index n has the interpretation of time, in the recursions,with b(k, 2) restricted to bea functionon the 2-D case, in general, it is not clear what “next” or “previous” correspondingquadrant. However,as shown byDudgeon mean. In fact,just given (51)it is not clear thatthere is [ 1081, this by no means exhausts the possibilities for recur- any definition of nextor previous that will allow us to sion. In addition tothe one quadrant functions, we can compute y(m, n)recursively. Dudgeon [ 1081, Pistor [ 1 191, obtain recursivedifference equations with Z)’s that andEkstrom and Woods [ 1351have studiedthis problem b(k, areone-sided [ 1081.In this case thesupport of b is asin in greatdetail. Let us considerone of themost important Fig. 4, and we cancalculate y(m, n) columnby column, special cases of (51) in which A = 1 and b has its support as recursing to the north and then shifting to the next column indicated in Fig. 2. We then have to the east. 1 MN Let us makeanother connection with 1-Dprocessing. y(m, n) = - - W, Oy(m - k, n - I) Suppose that one of the two indices, say m, has the interpre- b(o,o) k=O l=O tation as time. Then one might think ofy(m, n) and x(m, n) (k,1) + (0, 0) as (1-D)spatially distributed processes that evolve in time. +- 1 (52) Temporalcausality might then correspond tothe support b(0,O) x(m’ n)’ of 6 in Fig. 4 beingmodified deletingby pointsthethe on 1008 PROCEEDINGS OF THE IEEE, VOL. 66, NO. 9, SEPTEMBER 1978
leads to somecomplications. New difficultiesare also en- countered in the stud$ of the stability of recursive 2-D filters [ 1081, [ 1201. As in the 1-D case, the stability of the filter depends onthe direction of recursion,and there are many more possibilities in2-D. In addition, although there are analogs of results such as those in 1-D that test to see if all poles are inside the unit circle [ 1081, [ 1161,the required a. calculationsare far more complex. This increase in com- e. plexity also arises in the related problem of the stabilization e. of a given 2-D system function while keeping the magnitude .e e. of the frequencyresponse unchanged. In 1-D this is done easily by shifting those poles that lie outside the unit circle, but this cannot be done that easilyin 2-D, since we cannot Fig. 4. Support of a one-sided function. factor 2-D polynomials.’ Anotherstabilization approach in 1-D is spectralfactor- positive n axis,yielding a “strictly”one-sided function. In ization-i.e.,we write a given rational H(z) as theproduct this case, one could define the “state” of the system, and it of two pieces, HE(z) and Hw (z), where HE has all its poles is clear that this “state” will be finite dimensional only if the inside the unit circle (and hence is stable if used to process range of n is bounded, which is precisely when the require.d inputs in the eastern direction) and Hw has all its poles out- storage for the 2-D recursionis finite. side theunit circle (stable to the west).Thus, in2-D, one As mentioned earlier, the ability to solve a 2-D difference is temptedto seek factorizationsinto four stable quadrant equation recursivelyleads directly tothe definition of a fiters [ 1191 orinto two stable half-plane fiiters [ 1081, partial order on the part of the 2-D grid over which we wish [ 1231, [ 1351 much like the 1-D case. Such techniques have to solve theequation. Given thisprecedence relation, one been developed using 2-D cepstral analysis, and we refer the thenhas some freedom in deciding how to sequence the reader to the references. We do note that the lack of a fun- calculations.Specifically, ifwe think of asequence of cal- damentaltheorem of algebra does mean thatthe factors in culationsas determining a total orderon the part of the thesefactorizations will not in generalhave finite order 2-D grid of interest, all we require is thatthis total order denominators. be compatiblewith the precedence relation. Once wehave Afinal stabilization procedure is based on the guaranteed sucha total order, we can either view thisas transforming stabilityin 1-Dof least squares inverse. The least squares I-D filters into 2-D fiters or vice versa [ 1221.One widely inverse (LSI) is obtained using exactly the methodology one used order is the line-scan [ 1081, [ 1121, [ 1221: brings into playin performing linear prediction of speech. Given thedenominator B andits inversetransform b, one (i, j)< (I, if i< I or i=1 and j< k) k. seeksa finiteextent impulse response p thatapproximates Assuming we areinterested only in lines of finiteextent, the convolutionalinverse of b bychoosing the coefficients we can readily see one of the problems with this order and in p to minimize the sum of the squares of thedifference withorders in general. If we attemptto process theorder between b*p andthe unit impulse. In 1-D,one has the inputdata with a 1-D LSI system,the resulting 2-D pro- guaranteethat p is minimumphase (i.e., thatthe all pole cessing is not SI, essentially because our ordering has placed model 1/p is stable). In [ 1201,Shanks et al. conjectured the lastpoint on one line “next” to the firstpoint on the that this minimum phase property holds in 2-D. Under this next. assumption, they proposed the use of a double least squares We close our discussion of 2-Dorders and precedence inverse to stabilize and unstable denominator of a NE filter. relations by noting that these very same issues arise naturally Using this design procedure, numerous 2-D filters have been incertain feedback control problems. Ho andChu [ 1281 designed. Unfortunately,Genin and Kamp [ 1601have re- consideroptimal control problems in which onehas a set cently shown that this conjecture is false in 2-D if one con- of decisionmakers who base their decisions on certainob- strains oneself to quarter-plane Titers (although it is true in served data. Ho and Chu define a precedence relation among certain restricted cases [ 1571). On the other hand, Marzetta decisions: j < i if the decision of j affects the observation of [ 1501has shown that the desiredminimum phase property i. They assume that this is a partial order-i.e. that if j < i, does hold if the least squaresinverse problem is posed in we cannot have i< j (this is precisely the condition needed terms of half-plane fiiters. We will return to this point again for recursibility of 2-D fiters).Then, under a“partially later. nested information condition”-if j 4 i, then i’s observation As in theI-D case,a critical question in the design of includes knowledge of j’s observation-they solve an optimal 2-D IIR filters is the existence of limit cycles and the effect controlproblem. Witsenhausen [ 1291has also studiedthis of roundoff noise on the filter output. The results in [ 1171 partialorder and has pointed out that if one totally orders onthe existence of horizontal, vertical, andnoninteracting the set of decisionmakers in, away compatiblewith the diagonallimit cycles parallel 1-D results. Open questions precedencerelation, one can then define the state evolution involve the intriguingquestion of whetherone can extend of the system.Hence we see thatthere may be many pos- any of the other techniques discussed in Section 111. Do the sible sets of statescorresponding to differentcompatible passivity-Tsypskin-positive-real-frequency domainresults of total orders (just as storage requirements vary with the choice Claasen et al. [82] and others extend to the 2-D case? What of recursion). In the preceding discussion we have seen that the presence ‘This is often referred to as the “absence of thefundamental theorem of apartial order as opposed to the usual1-D totalorder of algebra’’ for multivariable polynomials (see, for example, [ 1081). WILLSKY:DIGITAL SIGNAL PROCESSING AND CONTROL AND ESTIMATION THEORY 1009 about the Lyapunov techniques of Willson [ 751 ? Of course The “if” part of thisresult involves theconstruction of a inthis case one wouldneed 2-D state spacemodels and a realization that is generalizationa of the1-D “standard 2-DLyapunov theory, whichin itself mightbe of interest controllableform.” Having suchrealizations, attention in providing a method to test for stability of 2-D LSI systems naturallyfocusses on minimality.This leads directly to the even with perfect arithmetic. notions of(local) controllability and observability, with The analysis of roundoff noise in 2-D filters can be carried finiterank conditions for these properties beingdeveloped out much as for 1-D filters, but another open question con- ina manner analogous to that of Roesser.The main mini- cerns the extension of the covariancenoise analysis method mality result of Marchesini and Fornasini is that minimality describedin Section I for1-D roundoff analysis. Again one implieslocal controllability and observability but that local wouldneed astate-space model in orderto consider this controllability and observabilitydo not imply minimality. question. Attasi [ 1301 has studied a special caseof (54),in which
B. Two-Dimensional State-Space Models A0 = -A1A2 = -A2A1. (55) In additionto 1-D state spacedescriptions for recursively In this case,the system transfer functionis separable (H(z1,z2) = ordered2-D systems [ 1331,some workhas been done on Hl(z,)H2(z2)),and, as shownin [1321, this is theonly the analysis of 2-Dstate-space models. Roesser [ 1341con- caseinwhich the global state is finitedimensional. As siders the NE model any FIR fiiter can be realized by (54), (55) any stable impulse response can be approximated arbitrarily closely by a system u(i + 1, j)=Alu(i, j) +A2h(i, j) +Blx(i, j) of thisform. This, of course, is neitherstartling nor nec- h(i, j + 1) =A3u(i,j) +A,h(i, j)+Bzx(i, j) essarily very useful,since the dimension of the resulting state-spacesystem maybe extremely large. Having this y(i,j) = Clu(i, j) C2h(i, j)+Dx(i, j) (53) + framework, Attasi defines dual notions of local controllability here x is theinput, y is the output, and u and h together andobservability and derives conditions somewhat simpler play the role of a“state” variable, “carrying” verticaland than in [ 13 11, [ 1341because of the assumedseparability. horizontalinformation, respectively. Giventhis model, Attasi alsoconsiders minimal realizations, obtains astate Roesserconsiders several issues, includingvariationa of space decomposition result and minimal realization algorithm constants formula to solve (53), a 2-D version of the Cayley- much like thosein I-D, and shows that minimality implies Hamilton Theorem, whichin turn is used to obtain an effi- controllabilityand observability. He also proves the con- cientmethod for computing the transition matrix, and the verse of this last result, but this is only true if one looks for notions of controllabilityand observability. In obtaining the minimal realization in the class of models satisfying (55). his algorithm for recursively computing the transition matrix We refer the reader to [ 1311 for anexample illustrating via the Cayley-Hamiltontheorem, Roesserused thenotion these points. of 2-Deigenvalues in acrucial manner, and in the usual Undoubtedlythe major contribution ofAttasi’s work is nonfactorizable case the calculation of zeroes of acharac- that he did somethingwith hismodels, and wewill discuss teristicpolynomial is extremelydifficult. This not only his filtering results in the next subsection. He also developed complicateshis transition matrix algorithm, but it makes a 2-D Lyapunov equation, which he used in a generalization stability tests more difficult, as we have already mentioned. of the “invarianceprinciple,” [ 1521.The exact implication Furthermore,the model (53) is limited to quadrant-causal of this result for 2-D stability theory and its potential utility systems.This is perfectlyreasonable for the study of quad- insuch areas aslimit cycle analysis remain as questions for rant-recursivefilters, but its value for the analysis of other furtherwork. Attasi alsoconsiders systems as in (54), (55) 2-Dsignals is unclear.For example, Roesser mentionsthe whichare driven bywhite noise. Again heobtains a 2-D possibilityof a 2-D filtering theory, where (53) plays the Lyapunovequation for thestate covariance, and this result role of a“spatial shaping fiiter.” As Ekstromand Woods may be of some value in performing roundoff noise analysis [ 1351point out, one cannot obtainarbitrary spectra from for 2-D filters. In addition, he developsa stochastic realiza- a NE shapingfilter. Hence, one may need two suchfilters, tiontheory that exactly parallels the 1-Dcase with one aswell as a methodfor modelling the spectra of the signal rather surprising exception,in that, unlike the 1-D case, field. Finally, we note that Roesser’s “state” (u(i,j), h(i, j)) in the 2-D case the stochastic realization is essentially unique. might betterbe termed a “local state” [ 1311. As we saw This is dueprimarily to theadditional constraints imposed earlier,in recursively solving 2-D equations, the required by the fact that weuse a single quadrant shaping filter. An- amount of storagein general depends onthe size of the other novel feature of Attasi’sdevelopment is the necessity arrays of interest, while the dimensions of u and h correspond for using nonsquarefactors-i.e., to performthe required to theorder of the system, as in (52). factorization Issues of this type have also been considered by Fornasini and Marchesini [ 131 1. Theyconsider local NE state-space S(z1, zz) =H(zl,z2)H’(zi1, z:’) (56) descriptions of the form where H is NE causal and of the form (54), (55), one must x(m + 1, n + 1) =AOx(m, n) +Alx(m + 1, n) consider rectangular factors.For example, if y is ascalar +Azx(m, n + 1) +Bu(m, n) process, then H in general must be 1 X m, and, in fact, the aforementioned uniqueness resultfies the value of m. y(m, n) = Cx(m, n). (54) Recently, Morf et al. [ 1381 have made several noteworthy contributionsto 2-D state-space theory. They consider the Theyshow that a NE IIRfiiter can be realized as in (54) properties ofpolynomial and rational matrices in two vari- if and only if the transform of the impulse response is rational. ables in order to generalize the scalar 2-D polynomial results 1010 PROCEEDINGSIEEE, OF THE VOL. 66, NO. 9, SEPTEMBER 1978 of Bose (1371and the matrix 1-Dpolynomial results of PSF. Examination of (57) yields that His an N X N matrix Rosenbrock[139] and Wolovich [ 1401. Theconcepts of of NX N blocks {Hii), where the (m, n) element of Hfj is local controllability and observability for the Roesser model h(i, m, j, n). If the imagingsystem is shift-invariant, it is are explored in [ 1381, and the authors point out that these readilyseen that H is blockToeplitz, and, in fact,each of conditions neither imply nor are implied by the minimality the blocks is itself a Toeplitz matrix. Note also that if h is of the realization(this is donewith several instructive ex- separable, then amples). Toobtain notions of controllabilityand observ- h(i, j, m, n)=h,(i, m)hz(j,n)and H=A1 @ Az abilitythat are equivalent to minimality, Morf etal. gen- eralize theapproach of Rosenbrock,and this leads to the wlere @ denotes the tensor or Kronecker products, and Ai notions of modal controllabilityand observability and a is an N X N matrix obtainable from hi. relatedconcept of minimality. In thissetting the existence It is evidentfrom the precedingdevelopment that proba- ofminimal realizations becomes adifficult problem, and bilistic and statistical methods must play some role in image one may not evenexist if we restrict ourselves to systems processing. In this context, f, g, u, and perhaps h are random withreal parameters. In relatedwork, Sontag [ 1361has fields. For now we consider such a random field s(i, j)to be also foundrealizations of lowerdimension than those pro- characterized by its meanT(i, j) and its covariance posedby Fornasini and Marchesini, andhe has shown that minimalrealizations need not be unique up to achange of basis. All of thesefacts indicate thatthe 2-Dstate-space (59) model is an extremely complex one and offers some extremely difficultmathematical and conceptual problems. It remains The field will be called (wide-sense) stationary if to be seen whether any of these state models and realization r(i, j, m,n) = r(i - m, j - n). (60) theories canprovide auseful framework for solving2-D analysis and synthesis problems. Note that if s is ordered lexicographically, then its covariance R is the NZ X NZ matrix obtained from r in the same manner C. Image Processing, Random Fields, and Space-Time that H is obtained from the PSF h. We also observe that R Processes is block Toeplitz with Toeplitz blocksif s is stationary. Digitalprocessing of images fordata compression, noise One important problem in image processing is the efficient removal, or enhancement is one of the major areas of appli- representation of images forstorage or transmission [ 1531, cations of 2-D digital signalprocessing techniques. Inaddi- [ 1541. One well-known method for obtaining a less redun- tion, imageprocessing has spurred a great deal of workin dant representation of an image is the Karhunen-Loeve trans- the analysis of spatiallydistributed stochastic variables- form [ 1531, whichinvolves the diagonalization of R. How- random fields. Let g(i, j) denotethe image radiant energy ever,in general, this transform involves exorbitantamounts as afunction of two discretespatial variables, where, for of computation.There are, however, severalspecial cases thetime being, wewill assume thatthe system is free of inwhich this transform can becalculated efficiently. One noise. The imageresults from an image formationprocess of these,motivated by similar analysis performed by Hunt thattransforms the original radiant energy f(i, j) intothe [ 1531and Andrews and Hunt [ 1261, is quiteinstructive. observedimage. A general model that is often used for the Supposethat s is stationary,and that any particular pixel image formation process is isuncorrelated with ones sope distance d away.Then the blockToeplitz covariance matrix is nonzeroonly near the N maindiagonal (and the samecan be said for each of the g(i, j)= h(i, j, k, l)f(k,I), i, j = 1, * ,N blocks). We now modify R and its blocks, to make R block k,I=1 circulantwith circulant blocks. A blockcirculant matrix (57) is blockToepiitz with eachrow a cyclic shift to the right where h is the point-spreadfunction (PSF), which models of the precedingone, where the last block on the right of the smoothing and blur that take place in the image forma- one row becomes the first block on the left in the next row. Thismerely means replacing some of the zeroes in R with tion process [ l l l], [ 1261, [ 1531. Note that one important case of (57) is the shift-invariantcase, in which h depends nonzeroentries. Intuitively, imaging the imageas a flat array, wehave connectedopposite edges, fmt to createa only on i - k and j - I. In this case (57) is a 2-D convolution. In addition to the image formation process, one must take cylinder,and then a torus. The reason formaking this ap- intoaccount the process of wage recordingand storing. proximation is thatthe matrix of eigenvectors of R,, the Severalnoise-corrupted nonlinear image models have been circulantmodification of R, canbe computedefficiently developed [ 1261, [ 1531for this; however, asdiscussed in using the fastFourier transform (see [ 1261, [ 1621 1, and thusthe Karhunen-Loeveexpansion can be performed [ 1531 often one may be able to justify the use of an additive noise model quickly. As discussed inSection 11, one of themost widelyused q(i, i)=di, i) + u(i, i) (58) coding or compression.schemes for 1-D time series, such as where u is an additive noise process. We now turn our atten- speech, is linearprediction, in which we designa one-step tion to the analysis of this model. predictor or inverse whitening filter for the time series. This Atvarious points in this development, it will bemore methodhas severalappealing features in 1-D: it is efficient convenient to view f,g, q, and u as vectors by performing a (if one uses the Levinson algorithm);it 'leads to recursive scan(lexicographic) ordering, in which case we write q = codingand decoding algorithms; and it yields excellent per- Hf + u, where H is an N2X NZ matrixformed from the formance. In 2-D the situation is not as clear. What direction WILLSKY: DIGITAL SIGNAL PROCESSING AND CONTROL AND ESTIMATION THEORY 101 1 do we predictin and what old data do weuse todo the recursive methods, and an appealing question is the extension predictions? At thistime, answers to thesequestions are of suchfilters to 2-D.Anyone familiarwith 1-D Kalman beginning to be formed. Genin and Kamp [ 1601 have shown filteringtheory realizes thatthe design of the filter relies that NE predictors need not be minimum phase, and Marzetta heavily on adynamic representation of the received signal. [ 1501 has provided an argument for why this is in fact the Hence, to developsuch techniquesin 2-D, we needa more case. Specifically, in 1-D, we are guaranteed that the optimal complex model of an image than that provided by the mean predictor for y(n) based on y(n - l), y(n - 2), * , y(n - r) andcovariance. The need for the use of suchmodels is an is necessarilyminimum phase;however, ifwe skip some obviousdrawback to thisapproach, but the potential gains points in the past, e.g., if we predict y(n) based on y(n - l), in computational efficiency represent a distinct advantage. y(n - 2),and y(n - 4),the optimal predictor may not be Oneapproach to recursiveprocessing of images involves minimumphase. Marzetta points out that NE predictors do the 1-Dprocessing of the scan-orderedimage (see Section skip points. For example, consider the predictor of y(m, n) IV-A).This work has been developed by Nahi, Si!vemm, based on y(m - 1, n), y(m, n - 11, and y(m - 1, n - 1). If andtheir colleagues IlO91,11121, [1141, [141]. Suppose we totallyorder the points in the plane ina fashion com- we have an image f(m, n) (assumed to bezero for conve- patible with the partial order for calculating points recursively nience)with stationary covariance r(k, I), and we observe to the NE, then (m - 1, n), (m,n - l), and (m - 1, n - 1) q =f+ u where the additive noise u is, for simplicity, assumed will never be thethree immediate predecessors of (m,n). to bezero mean and white, with variance R. We now take Thus, just as in 1-D, there is no reason to expect the optimal the scan ordering of the N X N grid on which q, f, and u are predictor to beminimum phase. Marzetta then points out defined. Let us use the same symbols to denote the resulting that if we don't skip points, i.e., if we usea full half-plane 1-D processes. We then have predictor, we do get the minimum phase properties, Levinson- typealgorithms involvingreflection coefficients, etc. Note E[f(k) f(Z)] = S(k, I), k, I = 1, * . * , N2 (61) that this predictor is primarily of conceptualinterest, since where S(k, I) can be calculated from knowledge of r(m, n). the predictor involves the incorporation of an entire, infinite Note that the scanned image f(k)is not stationary due to the extentcolumn before any points in the preceding column abrupt change that occurs when the scanner reaches the end maybe included. We refer the reader to [ 1501 for details of one line andbegins thenext. We wish to useKalman andfor practical, suboptimal methods which also have the filteringtechniques in order to suppress the noise. In order minimum phase property. We also refer the reader to [ 1581 to do this, we need a state-space model for f. Unfortunately, for another generalizationof the Levinson algorithm to 2-D. as pointed out in [ 1121, S(k, I) does not have the required We now turnour attention to the problem of restoring separability that is needed in order for such a realization to blurredand noise-corrupted images [lll], [126],[153]. exist.Hence, somesort of approximation is needed,and Anumber of nonrecursivemethods havebeen developed severalhave been developed. The simplestof these involves for the removal of blur and for noise rejection: inverse filter- findinga stationary approximation R(k) to(61), muchas ing;2-D minimum mean-square error (MMSE)Wiener fiil- Manry and Aggarwal foundshift-invariant approximations tering;etc. We refer the reader tothe survey [ 1531for tothe shift-varyingscanning filters they studied in [ 1221. more on these methods and for further references. We merely Having R(k), one can then usesome realization procedure point out here that techniques such as the Wiener filter have to finda Markov model that realizes orapproximates the some difficulties and limitations as image processing systems. given correlation function. To a great extent this is due to the fact that the MMSE cri- We can now obtain an image restoration scheme by direct terion is not particularly well-suited to the way in which the application of Kalmanfiltering. Several comments are in human visualsystem works [ 1271.In particular, the Wiener order. We fnst note that the filter has an artificial causality; filter is overly concerned with noise suppression. In addition, only the points below and to the left on the same line affect inorder to make the filter computationally feasible, one the estimate of a given pixel. This can be partially removed often assumes stationarity. This in turn leads to a filter that by the use of smoothing. With the model we have developed, is insensitive t.o abrupt changes, i.e., it tends to smooth edges this can be done efficiently with two Kalman filters, scanning andreduce contrast. On the other hand, in high contrast inopposite directions and starting at opposite ends of the regions, thehuman visualsystem will readilyaccept more image. Theresulting estimate still hasdifficulties because noise inorder to obtaingreater resolution. Several schemes of the effects at the ends of lines. 'In this case, one can re- havebeen proposed that areaimed at trading-off between movesome of these difficulties by transposing the image thepotentially high-resolution, poor noise performance of andperforming the same type of processingagain; we then the inversefilter andthe lower resolution, good noise per- have NE, NW, SE, and SW Kalman filters. formance of the Wiener filter. One of these is the constmined The recursive methods discussed so farhave assumed that least squares filter [ 1 1 1, 1 [ 1261. there is no blurring due to a nontrivial PSF. If there is such Several other observations can be madeconcerning the blurring,essentially we mustdevelop scan-ordereda 1-D processingsystems mentioned so far. As mentionedearlier dynamicalmodel for the effect of the blur and then incor- theyare nonrecursive and in principle require the block poratethis model into our Kalmanfilter. The simplestex- processing of the entire image or substantial sections of the ample of this, motion blur along the direction of the scan, image. , Hence thecomputational burden of theseschemes was considered by Aboutalib and Silverman [ 1 141 (see [ 161 ] canbe quite high. In 1-D, one finds that recursive methods for considerationof more generalblurs). Again thissystem areoften preferable to nonrecursiveones because of their offerscomputational advantagesover nonrecursive schemes, computational advantages. As discussedin [ 1261 the1-D butthe restoration system may be verysensitive to errors Kalman filteroffers great computational savings over non- in theknowledge of the PSF. 1012 PROCEEDINGSIEEE, OF THE VOL. 66, NO. 9, SEPTEMBER 1978
The previous technique did not directly use a 2-D recursive two 1-D Kalman filters starting at opposite ends of the line. model for the image. The first work along this line was that Furthermore, the optimal smoother can again be implemented of Habibi [ 1131who considered a2-D, recursive, auto- with two filters of the type devised by Attasi, one sweeping regressive shaping filter the columns in order of increasing m, and the other in order of decreasing m. This is reminiscent of thedecomposition x(k+1,I+1)=~~~(k+1,I)+~~x(k,I+1)-~~~2~(k,l)of zero phase filters into two half-plane filters[ 1 191,[ 1351. +do- P:)(l -Pi) w(k, 0 (62) The method of proof used by Attasi involves the taking of z-transforms along the n directionand the treatment of where w(k, I) is awhite, zero mean, unit varianceprocess. m asa time variable.Essentially we regard the 2-Dsystem Assuming measurements of the form 7 =x + u, Habibi then as ahigh-dimensional (infdte if thedomain of n is un- developed a suboptimal estimator to estimate x(R + 1, 1 + 1) bounded) 1-D system,where weuse spatiala transform based on { y (m, n) Im < k, n < I}. The suboptimality of “along” the 1-Dstate vector in order to simplify the cal- Habibi’s estimator arises essentiallybecause x is onlythe culations. The keystep in Attasi’sdevelopment is aderiva- localstate, and one needs to estimatethe global statefor tion of a set of Riccati equations, parametrized by the trans- optimalfiltering. The mostcomplete study of optimal 2-D form variable z, forthe powerspectral density S,(z) of Kalman filtering has been performed by Woods and Radewan e(m, n) considered as a function of n. One can then factor [ 1481. We assume that we have a one-sided causal dynamic these spectra to obtain the 1-D realizations of the e’s. model for the random field Methods which transform the data in one direction in order M +M to simplify or to study recursion in the other have also been x(m, n) = b(k, I)x(m - k, n - I) used in several other image processing schemes. For example, k=l I=-M amethod verysimilar to Attasi’swas used in [ 1591. In M addition,Jain andAngel [ 1181 haveconsidered fields de- t b(0, I)x(m,n - 2) + w(m, n). (63) scribed by a nearest neighbor, interpolative equation [ 1101 1-1 x(m, n) =al[x(rn, n + 1) +x(m, n - 1)l Suppose we want to estimate x(m, n) given all values of q = +a2[x(m + 1, n)+x(m - 1, n)l + w(m,n). (68) x + u in the past, where past is defined relative to the direction of recursionin (63). Woods and Radewan point outthat Following [ 1 181, let us consider the vector scan process, i.e. this canbe doneoptimally with an extremely high dimen- we process an entire line of observed data, y = x + u, at a time. sional Kalman filter to estimate the global state of the system, Defining the resulting 1-D vector processes x,, y, , w, , and which in this case has dimension on the order of MN (M= urn, we can write (68) as order of the filter,N = width of the image). xm+1 = Qxm - x,-1 +wm (69) Optimalline-by-line Kalman filtering for images has also beenconsidered by Attasi [ 1301using a stochastic version where Q is asymmetric, tridiagonal, Toeplitz matrix. Jain of themodel discussedin Section IV-B. Specifically the and Angel point out that the diagonalizationof Q, A = M’QM, image is assumed to be generated by a separable vector analog can be performed with the aid of the FFT without any ap- of the model used by Habibi [ 1131 proximation. Thus, if we define thetransformed quantities X, , V,, etc., (X, = M‘x,) we obtain a set of N decoupled x(i, j)=Flx(i - 1, j)+Fzx(i, j - 1) estimationproblems, indexed by j (which indexes the com- -F1F2x(i- 1,j- I)tw(i- 1,j- 1) ponents of the transformed vectors): - - q(i, j) = f(i, j)+ u(i, j) = Hx(i, j)+ u(i, j). (64) Xm+l,i = hiXm,i - x,-1, i + Em,j (70) - - - We wish to obtain the optimal estimate ?(m, n) of x(m, n) Ym,i -Xm,i +um,i. (71) given q(i, j) for i < m and all j. The optimal estimate in this Each of these problems can be solved using a Khan filter, caseconsists essentially of two 1-D operations.Suppose we and we obtain an extremely efficient implementation: trans- have 2(m - 1, n) for all n. We fit predictahead one line formthe observations, solve the low-dimensionaldecoupled to obtain estimationproblems (perhaps inparallel), and transform back. F(m, p) =F12(m- 1, n), for all n. (65) As we have seen, optimal 2-DKalman fitering algorithms Note that each of these estimates is calculated independently. requirelarge amounts of storageand computation. Thus, We now observe the new line of measurements q(m, n) for all thestudy of suboptimalestimators that require less com- n, and we create the errorprocess and the error measurement putation is of importance.One suboptimal fdter developed in [ 1481 is the reducedupdate Kalman filter.Examining e(m, n) = x(m, n) - F(m, n) (66) the optimal fiter of Woodsand Radewan, we see that the y(m, n) =q(m, n) - HZ(m, n) =He(m, n) + u(m, n). predict cycle is computationally straightforward. One simply uses the recursion (63) assuming no noise and using preceding (67) estimates.The measurement update part of theoptimal Thuswe havea 1-D estimationproblem: estimate e(m, n) filter, on the other hand, involves updating the estimates of for all n, given y (m, n) for all n. Attasi shows that one can all of thecomponents of thestate. Assuming N >>M, obtain a finite dimensional 1-Drealization for e(m, n) as a we expectthat a given pixel is mostcorrelated only with function of n. Hence, this estimation problem reduces to the a small percentage of the elements of the state vector. There- usual1-D smoothing problem. The solution consists of fore, it seemsreasonable only to update.the estimates of WILLSKY: DIGITAL SIGNAL PROCESSING AND CONTROL AND ESTIMATION THEORY 1013 those components of the state that are within a certain dis- as these to the detection of boundaries in images (see [ 1151, tance of thepoint beingprocessed, i.e., we constrainmany [ 1421 for some work along theselines). of the gain elements to bezero and essentially allow only Throughout this subsection we have seen several examples “near neighbor updates.” of 2-D signal processing problems in which good use is made We havenow surveyed a number of nonrecursiveand re- of the transformation of the signals obtained by considering cursive estimationmethods. The recursive techniquescome them to be 1-D vector time signals, in which the other inde- with many of the same criticisms that were made concerning pendent spatial variable is used to index components of the nonrecursivefilters. They require detailed models of the vectors.There are, of course,many problems in which the imagestatistics and image formation process, and they are processes to be studied truly are space-time processes [ 1621, essentiallybased on the MMSE criterion. Hence, they in and in many of these the use of 2-D concepts can often be general will sacrifice resolution in favor of noise suppression. of great value. One of the best examples of this type arises In addition,these recursive techniques necessarilyaffect in problems of seismic signal processing [ 1241, [ 1251, [ 1491 the imagebecause of the assumed modelstructure. Some in which we observe the time response of the earth usinga of the recursive techniques allow the inclusion of image spatialarray of sensors. Otherapplications plus several spe- blur,while in other cases theextensions to includeblur cific problemformulations are discussed in [ 1621.In addi- have yet to bedeveloped. Also, we haveseen thatin some tion, [ 1621 contains a brief discussion of results and formu- cases optimal Kalmanfiltering is extremelycomplex, and lationsthat utilize both 1-D and 2-D tools of stochastic suboptimal,but intuitively appealing, recursivefilter struc- calculus and martingale theory[ 1431, [ 1441. Such techniques turesmust be used.In other cases we haveobserved that arein their infancy and work continues to determinetheir the use of the structure of the assumed model canlead to utility and limitations. We note only that the problem of the extremelyefficient optimal estimation algorithms (with the lack of anatural total order in 2-Dcauses difficulties in aid of transformtechniques). In addition, although work extendingI-D stochastic calculus concepts to 2-D. This is inthis areahas been limited in extent [ 1 141, [ 14 11 the not surprising, given the severalcomplications that we have recursive techniquesare directly amenable to the analysis already discussed. of space-varyingand nonstationary models. Thus, in spite Given the several examples described earlier in this section, of the many qualifications, we find enough positive attributes it is ourcontention that there is potentiallymuch to be to warrant continued study of recursive techniques for image gainedby utilizing both the perspectiveof 2-D signals and restoration. systems as well as that of 1-D space-time systems in studying Oneimportant area forfuture work involves the reliance problemsof either type. As afinal example ofhow this on Q priori information. As mentionedearlier, one often conceptmight be used, consider the study oflarge inter- canassume knowledge of the PSF or can determineit by connectedsystems. In this case we letthe spatial variable observing knowntest scenes throughthe imagingsystem. indexsubsystem variableswhich may be vector quantities In other cases, we may not have such information and must themselves. A general linear model then is the recursive 2-D estimatethe PSF as well as the image. Thusone important model question concerns the robustness of these techniques in the face of modellingerrors. As mentionedin Section I, tech- x(k + 1, i) = Aijx(k, j)+ BijU(k, j)+ ~(k,j) niques do exist for the sensitivity analysis of 1-D state-space i i models and 1-D Kalman filters. Can we extend these methods (72) to the 2-D case, and how well do the 2-D algorithms perform? y(k, i) = C ~ijx(k,i) + v(k, j). (73) In addition,methods abound in 1-D for on-line parameter i identificationand adaptive estimation in the presence of unknown parameters. Can we apply these methods with any Much asin the analysis of 2-DKalman filters [ 1481,the success to the 2-D problem? off-lineanalysis of such systems (solution of Lyapunov or A second area of concern is the resolution-noise-suppression Riccati equations, for example), as well as the on-line imple- tradeoff. As mentionedearlier, the human visualsystem is mentation of centralizedoptimal controllers or estimators, willing to accept more noise in certain regions, such as edges, maybecome prohibitively complex. Indeed the analogy inorder to improveresolution. Thus, in relatively slowly extendsfarther, as the “nearestneighbor” constrained filter varying regions of the image, we would like to remove noise, of Woods-Radewaninvolves precisely the samephilosophy whilewhere thereare abrupt scene changes orother high- as is used inmany decentralized control and estimation frequency fluctuations of interest, we would prefer to forego problems [261, [ 1451. noise suppression in favor of resolution [ 1 1 1.1 In this context Let us note that there may be many other useful insights an important problem is the detection of edges or boundaries to be drawn from this type of analogy. For example, if the between different regionsin an image. We also note that in model(721, (73) falls intothe class consideredby Attasi, many applications the determination of the boundaries them- then the optimal centralized Kalman filter can be efficiently selves may be the key Issue [ 1421. In recent years a variety implemented usingAttasi’s line-by-line optimal filter, which of techniques havebeen developed fordetecting and recog- involves dynamics in the transmission of information among nizing various types of boundaries in 2-D data (as an example, subsystems(Attasi’s two filters“along” each “line”). As a see [ 1551 ). In1-D, a variety of recursive techniques have second example, in thecase in which(72) and (73)are spatially beendeveloped for theestimation and detection of abrupt invariant, Melzer and Kuo [ 1461 and Chu [ 1471 made good changes in signals [ 15 11. Thesetechniques havebeen suc- use of the structure by taking spatial transforms in studying cessfullyapplied in a wide variety of applications,and an centralizedand decentralized optimal controllers. Similar importantquestion then is theextension of methodssuch analysis is contained in [ 1621 for the case of a finite string 1014 PROCEEDINGS OF THE IEEE, VOL. 66, NO. 9, SEPTEMBER 1978 with circular symmetry. Much as in the case of block circu- Parks of Rice University, Prof. Alfred Fettweis of the Univer- lant approximations [ 1261, this allows us to use FFT tech- sity of Bochun, and Prof. Hans Schussler of the University of niques to reduce the complexity of the on and off-line cal- Erlangen. This paper was a great deal of work, but interaction culations for centralized controllers in a manner very similar with people such as these has made it a wonderful experience. to that of Jainand Angel [ 1181. Inaddition, the use of spatial windowing techniques [ 11 to obtain nearest neighbor REFERENCES decentralizedcontrol algorithms may allow us to develop I 11 A. V. Oppenheim and R. W. Schafer, Digital Signal Processing. useful designs for such circularly symmetric systems. Englewood Cliffs, NJ: Prentice-Hall, 1975. [ 21 C. T. Chen, Introduction to Linear System Theory. New York: Holt, Rinehart and Winston, 1970. V. CONCLUDINGREMARKS [ 31 R.Yarlagadda, “Network synthesis-A state-space approach,” EEE Trans.Circuit Theory, vol.CT-19, pp. 227-232, May In this paper, we have examined a number of broad research 1972. areas that have attracted workers in two disciplines: digital sig- [4] R. E.Kalman, “Mathematical description of linear dynamical systems,”SZAMJ. Conrr.,vol. 1, pp. 152-192, 1963. nal processing and control and estimation theory. The goal of [ 51 L. M. Silverman and H. E. Meadows, “Equivalence and synthesis this examination has been the acquisition of some perspective of time-variable linear systems,” in hoc. 4th Allerton Conf. Cir- on relationshipsamong thequestions asked, methods used, cuit and System Theory, pp. 776-784, 1966. 61J. C. Willems and S. K. Mitter,“Controllability, observability, and general philosophies adopted by researchers in these dis- pole allocation, and state reconstruction,” ZEEE Trans. Auto- ciplines. Upon undertaking this study it was my feeling that mat.Contr.,vol. AC-16,no.6,pp. 582-595, Dec. 1971. . suchperspective would be extremely valuable in promoting 7) A. S. Morse and W. M. Wonham, “Status of noninteracting con- trol,” IEEE Trans. Automat. Contr., vol.AC-16, no. 6, pp. 568- collaborationand interaction among researchers in thetwo 581, Dec. 1971. fields. Upon concluding this study, I think that my initial feel- 81 I. B. Rhodes, “A tutorial introduction to estimation and filter- ing has been thoroughly substantiated. Not only are there nu- ing,” ZEEE Trans. Automat. Contr., vol. AC-16, no. 6, pp. 688- 706, Dec. 1971. merous examples of questions in one discipline that can bene- 91 J. B. Cruz, Jr.,Ed., FeedbackSystems. NewYork: McGraw- fit from the pointof view of the other, butalso we have found Hili.- - --, 1972 - - . - . [ 101 R. E. Crochiere, “Digital network theory and its application to a number of new issues that naturally arose from combining theanalysis and design of digital filters,” Ph.D. dissertation, the two pointsof view. M.I.T., Cambridge, MA, Apr. 1974. Each of the disciplines has its own distinct character, and [ 111 L. R. Rabiner and B. Gold, Theory and Application of Digital Signal Processing. EnglewoodCliffs, NJ: Prentice-Hall Inc., clearlythese will andshould be maintained. On theother 1975. hand, each discipline can gain from understanding the other. [ 121 R. E. Crochiere and A. V. Oppenheim, “Analysis of linear digi- tal networks,” Proc. IEEE, vol. 63, no. 4, pp. 581-595, Apr. State-space methods have their limitations, such as in specify- 1975. ing useful digital algorithms and structures. On the other hand, I 131 G. K. Roberts, “Consideration of computer limitations in imple- state-spacemethods provide extremely powerful computer- menting on-line controls,” M.I.T. ESL Rept. ESL-R-665, Cam- bridge, MA, June 1976. aidedalgorithms for noise analysis, optimal design specifica- [ 141 NASA Langley Workshop on Realizing the Potentials of Micro- tion, etc. State-space ideas also allow one to consider multi- processorsfor Aircraft Avionics Systems, NASA Langley Re- variableand time-varying systems. Allof theseaspects of search Center, Hampton, VA, Jan. 22, 1976. I 151 S. R. Parker and P. E. Guard, “Correlated noise due to roundoff state-space theory may prove of value to peopleinvolved in in fued point digitalfilters,” ZEEE Trans.Circuits Syst., vol. digital signal processing. On the other side, researchers in digi- CAS-23, no. 4, pp. 204-211, Apr. 1976. 1161 D. L. Kleinman, “On the stability of linear stochastic systems,” tal filtering have answered many crucial questions related to ZEEE Trans. Automat. Contr., vol. AC-14, pp. 429-430, Aug. turning design specifications into implementable designs. The 1969. deepunderstanding that workersin digital signal processing [ 171 S. Y. Hwang, “Roundoff noise in state-space digital fitering: A generalanalysis,” ZEEE Trans. Acoust.,Speech, Signal Pro- haveconcerning the problems of digital implementation is cessing, vol. ASSP-24, pp. 256-262, June 1976. something that researchers in control and estimation woulddo [ 181 C. T. Mullis and R. A. Roberts, “Synthesis of minimum round- well to gain. Thus it seems clear that a mutual understanding off noise fued point digital filters,” IEEE Trans. Circuit Syst., vol. CAS-23, no. 9, pp. 551-562, Sept. 1976. will prove beneficial to all concerned. [ 191 A. Sripad and D. L. Snyder, “Steady-state, roundoff-error anal- Numerous questions have been raised and speculation on va- ysis indigital filters realized withfixed-point arithmetic,” hoc. in Allerton Conf. (Urbana), IL, Oct. 1976. rious possibilities hasbeen made throughoutthis paper. 1201 A. Sripad and D. L. Snyder, “A necessary and sufficient condi- Whether any of these issues has a useful answer is a question tionfor quantization errors to be uniform and white,” IEEE for the future. It is my feeling that many of them do, and it is Trans. Acoustics, Speech SignalProcessing,vol. ASSP-25, no. 5, PP. 442-448, Oct. 1977. my hope that others will think so as well. I211 R. E. Rinkand H. Y. Chong, “Performance of state regulator systems with floating-point computation,” submitted to ZEEE ACKNOWLEDGMENTS Trans. Automat. Contr. 22‘1 R. E. Rink,“Optimal utilization of fixed-capacity channels in I wish to expressmy deepest appreciation to Prof.Alan feedback control,”Automatica,vol. 9, pp. 251-255, 1973. Oppenheim of M.I.T., who got me started on all this, who pro- 231 N. R. Sandell, “Information flow in decentralized systems,” in Directions in Large Scale Systems, Y. C. Ho and S. K. Mitter, vided a great deal of the insight that forms the backbone of Eds. New York: Plenum Press, 1976. this paper, and who engaged with me in innumerable hours of 241 J. B. Knowles and R. Edwards, “Effect of a finite word-length delightful debate on just about everything said in this paper. computer in a sampled-data feedback system,” hoc. ZEE, vol. 112,VV. 1197-1207.1965. There also aremany othercolleagues who have playedimportant [25]J. B.--Slaughter,“Quantization errors indigital control sys- roles in helping me piece together the picture I’ve painted in tems,” IEEE Trans. Automat.Conrr., vol.AC-9, 1964, pp. this paper. Particular thanks must goto Prof. JamesMcClellan, 70-74. (261 N. R. Sandell, P. Varaiya, and M. Athans, “A survey of decen- Mr. David Chan, and Prof. Nils Sandell of M.I.T., Dr. Wolfgang tralized control methods for large scale systems,” ZEEE Trans. Mecklenbriiuker of PhiLips Research Laboratory, Prof. Thomas Automat. Contr., Apr. 1978. I271 M. Athans, Ed., “Special issue on linearquadratic-gaussian prob- Kailathand Prof. Martin Morfof StanfordUniversity, Prof. lem,” ZEEE Trans. Automat.Contr., vol.AC-16, No. 6, Dec. BradleyDickinson of PrincetonUniversity, Prof. Thomas 1971. W ILL SK Y: DIGITAL SIGNAL PROCESSING AND CONTROL AND ESTIMATIONWILLSKY:CONTROLPROCESSING1015SIGNALTHEORYDIGITALAND AND
[28] R. Curry, er al., “State estimation with coarsely quantized high- [SS] M. Geversand T. Kailath, “An innovations approach to least data-ratemeasurements,” IEEETrans. Aerosp. Elecron. Syst., squares estimation-Part VI, Discrete-time innovations represen- VOI. AES-11, pp. 613-620, 1975. tations and recursive estimation,” IEEE Trans. Automat. Contr., I291 B. L. Hoand R. E. Kalman,“Effective construction of linear vol. AC-18, pp. 588-600, 1973. state variable models from input-output functions,” Regelungs- 1591 M. Morf, T. Kailath, and B. Dickinson, “General speech models technik, vol. 14, pp. 545-548, 1966. andlinear estimation theory,” in SpeechRecognition. New [ 301 L. Silverman, “Realization of linear dynamical systems,” IEEE York: Academic Press, 1975. Trans. Automat. Conrr.,vol. AC-16, pp. 554-567, Dec. 1971. 1601 B. S. Atal and M. R. Schroeder, “Adaptive predictive coding of [ 311 L. B. Jackson, A. G. Lindgren, and Y. Kim, “Synthesis of state- speech signals,”BellSys. Tech. J., pp. 1973-1986, 1970. space digital filters with low roundoff noise and coefficient sen- [61] M. Morf, B. Dickinson, T. Kailath, and A. Vieira, “Efficient so- sitivity,” Proc. IEEE Int.Symp. Circuits Syst., Phoenix,AZ, lutionsof covariance equations for linear prediction,” IEEE 1977. Trans. Acoustics, Speech,Signal Processing, vol. ASSP-25, no. 5, [ 321T. Kailath, “Some new algorithms for recursive estimation in pp. 429-433, 1977. constant linear systems,” IEEE Trans. Inform. Theory, vol. IT- [62] R. Parthasarathy and H. Singh, “On suboptimal linear system re- 19, no. 6, pp. 750-760, Nov. 1973. duction,”Proc. IEEE, vol. 63, pp. 1610-161 1, Nov. 1975. [ 331 A. Lindquist, “A new algorithm for optimal filtering of discrete- [63] J. Durbin, “The fitting of time-series models,” Rev. Inst. Znter- time stationary processes,” SZAM J. Control, vol. 12, no. 4, pp. nut. Statist., vol. 28, pp. 233-244, 1960. 736-746, Nov. 1974. [641 A. V. Oppenheim, Ed., “Special issue on digital signal process [ 341 A. Lindquist, “On Fredholm integral equations, Toeplitz equa- ing,”Prm. IEEE,vol. 63, no. 4, Apr. 1975. tions and Kalman-Bucy filtering,” Applied Math. and Opt., vol. 1651 L. Ljung, T. Kailath, and B. Friedlander, “Scattering theory and 1, no. 4, pp. 355-373. linearleast squares estimation-Part I: Continuoustime prob- [ 351 T. Kailath, “A view of three decades of linear filtering theory,” lems,”Proc. IEEE,vol. 64, no. 1, pp. 131-139, Jan. 1976. IEEETrans. Inform.Theory, vol.IT-20, no. 2,pp. 146-181, 1661 B. Friedlander,T. Kailath, and L. Ljung, “Scattering Theory Mar. 1974. and linear least squares estimation-Part 11: Discrete-time prob- [ 361 J. Rissanen. “A fast algorithm for optimum linear predictors,” lems,” J. Franklin Inst., Jan. 1976. IEEETrans. Automat. Contr., vol. AC-18, no. 5, p. 555, Oct. [67] R. E. Kalman, P. L. Falb, and M. A. Arbib, Topics inMathemat- 1973. ical System Theory. New York: McGraw-Hill, 1969. [ 371 B. W. Dickinson, M. Morf, and T. Kailath, “A minimal realiza- [68] L. S. DeJong,“Numerical aspects of realization algorithms in tionalgorithm for matrix sequences,” IEEETrans. Automat. linear systems theory,” Ph.D. dissertation, Technical University Contr.. vol. AC-19, no. 1, pp. 31-38, Feb. 1974. of Eindhoven, The Netherlands, July 1975. [ 381 J. L. Massey, “Shift register synthesis and BCH decoding,” IEEE 169) R. N. McDonough and W. H. Huggins, “Best least-squares repre- Trans. Inform. Theory,vol. IT-15,pp. 122-127. Jan. 1969. sentationof signals by exponentials,” IEEETrans. Automat. [ 391 J. Rissanen, “Recursive identification of linear systems,” SUM Conrr., vol. AC-13, no. 4, pp. 408-412, Aug. 1968. J. Contr., vol. 9, no. 3, pp. 420-430, 1971. 1701 C. S. Burrus and T. W. Parks, “Time domain design of recursive 1401J. Rissanen and T. Kailath, “Partial realization of random sys- digital filters,” IEEE Trans. Audio Electrmcoust., vol. AU-18, tems,” Automatica, vol. 8, no. 4, pp. 389-396, July 1972. no. 2, pp. 137-141, June 1970. 1411 T. Kailath, D. Q. Mayne, and R. K. Mehra,Eds., “Special issue [71] M. Morf, L. Ljung, and T. Kailath, “Fast algorithms for recur- on system identification and time-series analysis,” IEEE Trans. sive identification,” ZEEE Trans. Automat.Contr., to appear; Automat. Contr.,vol. AC-19, no. 6, Dec. 1974. also in Proc. I976 IEEE Con$ on Decision and Control, Clear- (421 K. J. Astrom and P. Eykhoff, “System identification-Asurvey,” water Beach, FL. Dec. 1976. Automatica,vol. 7, pp. 123-162, 1971. 721 A. H. Jazwinski, StochasticProcesses and Filtering Theory. [43j T. Soderstrom,L. Ljung, and I. Gustavsson,“A comparative New York: Academic Press, 1970. study of recursive identification methods,” Report 7427, Lund 731 G. Szego, OrthogonalPolynomials. NewYork: Amer. Math. Instituteof Technology, Dept. of Automat. Contr., Lund, SOC., 1959. Sweden, Dec. 1974. 741 E. Noldus, “On theInstability of Nonlinear Systems,” IEEE (441 P. Faurre,“Representation of stochastic processes,” Ph.D. dis- Trans. Automat. Contr., vol. AC-18, no. 4, pp. 404-405, Aug. sertation, Stanford Univ., Stanford, CA, 1967. 1973. [45] J. D. Gibson, J. L. Melsa, and S. K. Jones, “Digital speech analy- 75 1 A. N. Willson, Jr., “Limit cycles due to adder overflow in digital sis using sequential estimation techniques,” IEEE Trans. Acous- filters,” IEEE Trans. on Circuit Theory, vol. CT-19, no. 4, pp. tics, Speech Signal Procesring, vol. ASSP-23, pp. 362-369, Aug. 342-346, July 1972. 1975. 761 T. A. C. M. Claasen, W. F. G. Mecklenbrauker, and J. B. H. Peek, 146 J. Makhoul, “Linear prediction: A tutorial review,”Proc. IEEE, “Effects of quantization and overflowin recursive digital filters,” vol. 63, no. 4, pp. 561-580, Apr. 1975. IEEE Tmns.Acoustics, Speech, and Signal Processing, vol. I47 F. Itakura and S. Saito, “A statistical method for estimation of ASSP-24. no. 6. DO. 511-529. Dec. 1976. speechspectral density and formant frequencies,” Electron. [ 77 1 I. W. Sandberg and J. F. Kaiser, “A bound on limitcycles in Commun. Japan,vol. 53-A, no. 1, pp. 36-43, 1970. fixed-pointimplementation of digital filters,” IEEE Trans. [481 N. Levinson, “The wiener RMS error in filter design and predic- Audio and Electroucoust., vol. AU-20, no. 2, pp. 110-112, June tion,”Appendix B in N. Weiner, Extrapolation,Interpolation 1972. and Smoothing of StationaryTime Series, M.I.T. Press,Cam- 178‘I A. Fettweis, “Pseudopassivity, sensitivity, and stability of wave bridge, MA, 1942. digitalfdters,” ZEEE Trans. Circuit Theory, vol.CT-19, pp. !49 R. A. Wiggins and E. A. Robinson, “Recursive solutions to the 668-673, NOV. 1972. multichannel filtering problem,” J. Geophys. Res., vol. 70, pp. I79 1 A. Fettweis and K. Meerkotter, “Suppression of parasitic oscilla- 1885-1891, 1965. tions in wave digital filters,” ZEEE Trans. on Circuits and Syst., 501 S. Chandrasekhar, RadiativeTransfer. NewYork: Oxford Uni- vol. CAS-22. no. 3, pp. 239-246, Mar. 1975. versity Press, 1950. 801 K. Meerkotter and W. Wegener, “A new second-order digital fil- 51 ] J. Casti and E. Tse, “Optimal linear filtering theory and radia- ter without parasitic oscillations,” Archiv fib Elektronikund tive transfer: Comparisons and interconnections,” J. Math. Anal. Ubertrag., vol. 29, pp. 312-314, July-Aug. 1975. and Appl., vol. 40, pp. 45-54, Oct. 1972. 811 A. N. Willson,Jr., “Some effects of quantization and adder 52) J. D. Markel and A. H. Gray, Jr., Linear Prediction of Speech. overflow on theforced response of digital filters,” Bell Sys. New York: Springer-Verlag, 1976. Tech. J., vol. 51, no. 4, pp. 863-887, Apr. 1972. 531 B. S. Atal, “Speech analysis and synthesis by linear prediction 821 T. Claasen, W. F. G. Mecklenbrauker, and J. B. H. Peek, “Fre- of the speech wave,” J. Acoust. SOC.Amer., vol. 47, p. 65,1970. quency domain criteria for the absence of zero-input limit cy- 541 F. Itakura and S. Saito, “Speech analysis-synthesis based on the cles in nonlinear discrete-time systems with applications to digi- partialautocorrelation coefficient,” presented at Acoust. SOC. tal fdters,” BEE Tramon Circuits Sysr., vol. CAS-22, no. 3, pp. of Japan Meeting, 1969. 232-239, Mar. 1975. 1-55] J. D. Markel, “Basic formant and F, parameter extraction from [ 831 Ya. 2. Tsypkin, “Frequency criteria for the absolute stability of adigital inverse filter formulation.” IEEETrans. Audio Elec- nonlinear sampled data systems,” Automat. Remote Contr.,vol. froacousr., VOI. AU-21, pp. 154-160, 1973. 25, PP. 261-267, Mar. 1964. 1561 P. Faurreand J. P. Marmorat,“Un Algorithme de Realization [84] -, “Absolute stability of a class of nonlinear automatic sam- Stochastique,” C.R. Acad. Sci. Paris, ser. A, vol. 268, pp. 978- pied data systems,’’ Automat. Remote Contr.,vol. 25, pp. 918- 981, 1969. 923, July 1964. 1571 B. W. Dickinson, T. Kailath,and M. Morf, “Canonidmatrix [SS] -, “A criterionfor absolute stability of automatic pulse- fractionand state-space descriptions for deterministic and systems with monotonic characteristics of the nonlinear elem- stochasticlinear systems,” IEEE Trans Automat. Contr., vol. ment,”Sov. Phys. Dokl., vol. 9, pp. 263-266, Oct. 1964. AC-19, no. 6, pp. 656-667, Dec. 1974. (86) S. R. Parker and S. F. Hess, “Limit cycle oscillations in digital 1016 PROCEEDINGS OF THE IEEE, VOL. 66, NO. 9, SEPTEMBER 1978
filters,” IEEE Trans. CircuitTheory, vol.CT-18, pp. 687- [ 1161 D.D. Siljak,“Stability criteria for two-variable polynomials,” 697, Nov. 1971. IEEE Trans.Circuits Syst., vol.CAS-22, no. 3, pp. 185-189, 1871 C. A.Desoer, “On the relation between pseudo-passivity and Mar. 1975. hyperstability,” IEEE Trans. on Circuits Syst., vol. CAS22. no. [117] G. A. Maria and M. M. Fahmy,“Limit cycle oscillations in a 11, Pp. 897-898, NOV. 1975. cascade of two-dimensional digital filters,”IEEE Trans. Circuits 1881 V. M. Popov, Hyperstability of Control Systems. NewYork: Syst., vol. CAS-22, no. 10, pp. 826-830, Oct. 1975. Springer-Verlag, 1973. I1181 A. K. Jain and E. Angel, “Image restoration, modelling, and re- [89] R. W. Brockett and J. L. Winems, “Frequency domain stability ducticm ofdimensionality,” IEEE Trans. Comput., vol.C-23, criteria: Parts I and 11,” IEEE Trans. on Automat. Contr., vol. no. 5, pp. 470-476, May 1974. AC-10, pp. 255-261 and 401-413,1965. I1191 P. Pistor, “Stability criterion for recursive filters,” IBM J. Res. [90] R. W. Brockettand H. B. Lee,“Frequency domain instability Devel.,vol. 18, no. 1, pp. 59-71, Jan. 1974. criteriafor time-varying and nonlinear systems,” Proc. IEEE, [ 1201 J. L. Shanks, S. Treitel, and J. H. Justice, “Stability and synthe- vOl. 55, pp. 604-619, 1967. sis oftwo-dimensional recursive filters,” IEEE Trans. Audio 1911 J. C. Willems and R. W. Brockett, “Some new rearrangement in- Electracoust., vol. AU-20, no. 2, pp. 115-128, June 1972. equalities having application in stability analysis,” IEEE Trans. I121 1 J. M. Costaand A. N. Venetsanopoulos, “Design ofcircularly Automat. COntr.. VOl. AC-13. no. 5. DD. 539-549. OCt. 1968. symmetrictwo-dimensional recursive filters,” IEEE Trans. J. C. Willems, heAwlysis’of Feedback Systems. Cambridge Acoustics., Speech, Signal Processing, vol. ASSP-22, no. 6, pp. MA, The M.I.T. Press, 1971. 432-443, Dec. 1974. R. W. Brockett, “The status of stability theory for deterministic 11221 M. T. Manry and J. K. Agganval, “Picture processing using one- systems,” IEEE Trans. Automat. Contr., vol. AC-11, pp. 596- dimensional implementations of discrete planar filters,” ZEEE 606, July 1966. Trans. Acoustics, Speech Signal Processing, vol. ASP-22, no. 3, [94] G. Zames,“On the input-output stability of nonlinear time- pp. 164-173, June 1974. varying feedback systems, PartsI and 11,” IEEE Trans. Automat. 1231 P. Whittle,“Stochastic processes in severaldimensions,” Bull. Contr., vol. AC-11, pp. 228-238 Apr. 1966, and pp. 465-477, Int. Stat. Inst., vol. 40, pp. 974-994, 1963. July 1966. 1241R. L. Sengbushand M. R.Foster, “Design andapplication of [ 951J. L. Willems, Sitability Theory of DynamicalSystems. New optimalvelocity filters in seismic exploration,” IEEE Trans. York: Wiley, 1970. Comput.,vol. C-21, no. 7,pp. 648-654, July 1972. 1961 J. C. Willems,“Dissipative dynamical systems Part I: general I1251 J. M. Davis, “Velocity analysis: An application of deterministic theory,”and “Part 11: linearsystems with quadratic supply estimation to reflection seismology,”IEEE Trans. Comput., vol. rates.” Archive for Rational Mechanics and Analvsis.-. vol. 45. C-21, no. 7, pp. 730-734, July 1972. no. 5; pp. 321-343, 1972. 261H. C. Andrewsand B. R. Hunt, Dgital Image Restoration. S. R. Parker and S. Yakowitz, “A general method for calculating Englewood Cliffs, NJ: Prentice-Hall, Inc., 1977. quantization error bounds due to roundoff in multivariabledigi- 271 T. G. Stockham,“Image processing in the context of avisual tal filters,” IEEE Trans. Circ. and Sys., vol. CAS-22, no. 6, pp. mode1,”Proc. IEEE, vol. 60, no. 7,pp. 828-842, July 1972. 570-572,June 1975. 281 Y. C. Ho and K. C. Chu, “Team decision theory and informa- G. W. Johnson, “Upper bound on dynamic quantization errorin tion structures in optimal control problems: Part1,”IEEE Trans. digital control systemsvia the direct method of liapunov,”IEEE Automat. Contr., vol. AC-17, No. 1, pp. 15-22, Feb. 1972. Trans. Automat. Contr., vol. AC-10, no. 4, pp. 439-448, Oct. 291 H. S. Witsenhausen, “Some remarks on the concept of state,” in 1965. Directions in Large Scale Systems, Y.-C. Ho and S. K. Mitter, G. N. T. Lack and G.W. Johnson, “Comments on ‘Upper bound Eds., New York: Plenum Press, 1976. on dynamic quantization error in digital control systems via the 301 S. Attasi,“Modelling and recursive estimation for double in- direct method of Liapunov,” IEEE Trans. Automat. Contr., vol. dexed sequences,” in System Identification: Advances and Case AC-11, pp. 331-334,Apr. 1966. Studies, R. K. Mehra and D. G. Lainiotis, Eds., New York: Aca- G. P. Szego, “On the absolute stability of sampled-data control demic Press. 1976. systems,”Proc. Nut. Acad. Sci., vol. 256, no. 49, 1963. E. F0rnasin.i and G. Marchesini, “State-space realization theory R. E. Kalman, “Lyapunov functions for the problemof Lur’e in of two-dimensional filters,” IEEE Trans. Automat. Contr., vol. automatic control,” Proc. Nut. Acad. Sci., vol. 49, pp. 201-205, AC-21,no. 4, pp. 484-492,Aug. 1976. 1963. M. Flies, “Series reconnaissables, rationnelles et algebriques,” J. C. Willems, “Least squares stationary optimal control and the Bull. Sci. Math.,vol. 94, pp. 231-239, 1970. algebraic Riccati equation,” IEEE Trans. Automat. Contr., vol. D. S. K. Chan, “A novel framework for the description of real- AC-16, pp. 621-634, Dec. 1971. ization structures for 1-D and 2-D digital filters,” in hoc. Elec- -, “Mechanisms for the stability and instability in feedback tronics andAerospace Systems Conf, Sept.1976, Arlington, systems,”Proc. IEEE, vol. 64, no. 1, pp. 24-35, Jan. 1976. VA; also Ph.D. dissertation, Dept. of Elec. Eng. and Comp. Sci., A. Fettweis and K. Meerkotter, “On parasitic oscillations in digi- M.I.T., Cambridge, MA, May 1978. tal filters under looped conditions,” IEEE Trans. Circuits and R. P. Rower, “Adiscrete state-space model for linear image Syst., vol. CAS-24, no. 9, pp. 475-481, Sept. 1977. processing,” IEEE Trans. Automat. Contr., vol. AC-20, no. 1, P. Doratoand A. H. Levis, “Optimallinear regulators: The pp. 1-10, Feb. 1975. discrete-time case,” IEEE Trans. Automat. Contr., vol. AC-16, M. P. Ekstrom and J. W. Woods, “Two-dimensional spectral fac- pp. 613-620, Dec. 1971. torization with applications in recursive digital filtering,” IEEE Ya. 2. Tsypkin,“Fundamentals of the theory of non-linear Trans. Acoustics, Speech,SignaIProcessing, vol. ASSP-24, no. 2, pulse control systems,” Proc. of the 2nd IFAC, Basel, Switzer- pp. 115-128, Apr. 1976. land,~~.172-180,1963. E.D. Sontag,“On linear systems and noncommutative rings,” B.D. 0. Anderson and R. W. Newcomb,“Linear passive net- Math. Syst. Theory, vol. 9, no. 4, pp. 327-344, 1976. works: Functional theory,” Proc. BEE, vol. 64, no. 1, pp. 72- N. K. Bose, “Problems and progress in multidimensional systems 88, Jan. 1976. theory,”Roc. IEEE, vol. 65, no. 6, pp. 824-840, June 1977. OS] R. M. Mersereau and D. E. Dudgeon, “Two-dimensional digital M. Morf, et aL, “New results in 2-D systems theory, Part I: 2-D filtering,”Proc. IEEE, vol. 63,no. 4,pp. 610-623, Apr. 1975. polynomial matrices, factorization, and coprimeness,” and “Part OS] S. R. Powell and L. M. Silverman, “Modelingoftwo-dimensional 11: 2-D state space models-Realization and the notions of con- covariancefunctions with application to imagerestoration,” trollability, observability, and minimality,”Proc. IEEE, vol. 65, IEEE Trans. Automat. Contr., vol. AC-19, no. 1, pp. 8-13, Feb. no. 6, pp. 861-812,945-961, June 1977. 1974. H. H. Rosenbrock, State-Space and Multivariable Theory. New 10) J. W. Woods,“Two-dimensional discrete Markovian fields,” York: Wiley, 1970. BEE Trans. Inform.Theory, vol. IT-18,no. 2, pp. 232-240, W. A.Wolovich, LinearMultivariable Systems NewYork: Mar. 1972. Springer-Verlag, 1974. 111 M. M. Sondhi, “Image restoration: The removal of spatially in- M. S. Murphy and L. M. Silverman, “Image model representa- variant degradations,” Proc. IEEE, vol. 60, no. 7, pp. 842-853, tion and line-by-line recursive restoration,” 1976 IEEE Conf on Julv, 1972.-. Decision and Control, Cleamater Beach, FL, Dec. 1976. [ 112) NI-E. Nahi, “Role of recursive estimation in statistical image en- 11421 N. E. Nahiand S. Lopez-Mora,“Estimation of object bound- hancement,”Proc. IEEE, vol. 60, no. 7,pp. 872-877,July 1972. aries in noisy images,’’ 1976 IEEE Conf. on Decision and Con- [ 113) A.Habibi, “Two-dimensional Bayesian estimate of images,” trol, Cleamater Beach, FL, Dec. 1976. Proc. IEEE,vol. 60,no. 7, pp. 878-883, July 1972. I1431 E. Wong and M. Zakai, “Martingales and stochastic integrals for I114) A. 0. Aboutalib and L. M. Silverman, “Restoration of motion processes with a multidimensional parameter,”Z.Wahrscheinlich- degradedimages,” IEEE Trans. CircuitsSysr., vol.CAS-22, keitstheorie, vol. 29, pp. 109-122, 1974. no, 3, pp. 278-286, Mar. 1975. A. S. Willsky and P. Y. Kam, “Recursive estimation of signals I1151 N. E. Nahiand A. Habibi,“Decision-directed recursive image subject to random propagationdelays,” submitted toAppl. Math. enhancement,” IEEE Trans. Circuits Syst., vol. CAS-22, no. 3, and Optimization, pp. 286-293, Mar. 1975. D.P. Looze, P. K. Houpt, N. R. Sandell,and M. Athans, “On W IL LSK Y : DIGITAL SIGNAL PROCESSING AND CONTROL AND ESTIMATIONWILLSKY:PROCESSINGCONTROLSIGNALDIGITALANDTHEORY AND 1017
decentralizedestimation and control with application to free- [ 1541 A. Habibi and G. S. Robinson, “A survey of digital picture cod- way ramp metering,” IEEE Trans. Auromat. Contr.,Apr. 1978. ing,”Computer, vol. 7, no. 5, pp. 22-34, May 1974. [ 146) S. M. Melzer and B. C. Kuo, “Optimal regulation of systems de- [ 1551 R. Bajcsy and M. Tavakoli,“Computer recognition of roads scribed by a countably infinite number of objects,” Auromatica, from satellite pictures,” IEEE Trans. Syst., Man, and Cybernet- vol. 7, pp. 359-366, 1971. ics, vol. SMC-6, no. 9, pp. 623-637, Sept. 1976. 1471 K. C. Chu,“Optimal decentralized regulation for a string of 1561 N. K. Bose,“Special issue on multidimensional systems,”Proc. coupledsystems,” IEEE Trans. Automat.Contr., vol. AC-19, EEE, vol. 65, no. 6, June 1977. pp. 243-246, June 1974. 1571 E. I. Jury, V. R. Kolavennu, and B.D. 0. Anderson, “Stabiliza- I 1481 1. W. Woodsand C. H.Radewan, “Kalman filtering in two- tion of certain two-dimensional recursive digital Wters,” hoc. dimensions,” IEEE Trans. Inform.Theory, vol. IT-23, no. 4, IEEE, vol. 65, no. 6, pp. 887-892, June 1977. pp. 473-482, July 1977. 1581 J. H. Justice, “A Levinson-type algorithm for two-dimensional I.1491 L. C. Woodand S. Treitel.“Seismic signal orocessina.” -. Proc. Wiener filtering using bivariate Szego polynomials,”Proc. IEEE, IEEE,vol. 63,110.4,pp. 649-661, Apr. i975: vol. 65. no. 6. DD. 882-886. June 1977. [ 1 SO] T. L. Marzetta, “A linear prediction approach to two-dimensional [ 1591 F. C. Schoute; ‘M. F. ter Horst, and J. C. Willems, “Hierarchic spectralfactorization and spectral estimation,” Ph.D. disserta- recursive image enhancement,” IEEE Trans. Circuits and Syst., tion,Department of Elec.Eng. and Comp. Sci., M.I.T., Cam- vol. CAS-24, no. 2, pp. 67-78, Feb. 1977. bridge, MA, Jan. 1978. [ 1601 Y. Genin and Y. Kamp, “Two-dimensional stability and ortho- [ 151 ] A. S. Willsky, “A survey of design methods for failure detection gonalpolynomials on the hypercircle,” Proc.IEEE, vol. 65, in dynamic systems,” Automatica, vol. 12, pp. 601-611, Nov. no. 6, pp. 873-881, June 1977. 1976. [ 161 1 A. 0. Aboutalib, M. S. Murphy, and L.M. Silverman, “Digital [ 152 I J. P. LaSalle, “An invariance principle in the theory of stability, restoration of images degraded by general motion blurs,” IEEE differentialequations and dynamical systems,” in Differential Trans. Automat. Control, vol. AC-22, no. 3, pp. 294-302, June Equations and Dynomical Systems, J. K. Hale and J. P. LaSalle, 1977. Eds., New York: Academic Press, pp. 277-286, 1967. [ 162) A. S. Willsky, Digital Signal Processing and Control and Estima- [ 1531 B. R. Hunt, “Digitalimage processing,” Proc.IEEE, vol. 63, tionTheory-Points of Tangency, Areasof Intersection, and no. 4, pp. 693-708, Apr. 1975. Parallel Directions. Cambridge, MA: M.I.T. Press, to appear.