A View of Three Decades of Linear Filtering Theory
Total Page：16
File Type：pdf, Size：1020Kb
146 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-20, NO. 2, MARCH 1974 A View of Three Decades of Linear Filtering Theory Invited Paper THOMAS KAILATH, FELLOW, IEEE Abstrucf-Developments in the theory of linear least-squares estima- fields; problems of scalar and matrix polynomial factoriza- tion in the last thirty years or so are outlined. Particular attention is tion with applications in network theory and stability paid to early mathematical. wurk in the field and to more modem develop- theory [135], [177] ; the solution of linear equations, ments showing some of the many connections between least-squares filtering and other fields. especially as they arise in constructing state-variable realizations from impulse-response or transfer-function I. IN~~DUCTION AND OUTLINE data, which in turn is related to the Berlekamp-Massey HE SERIES of survey papers of which this is a part algorithm for decoding BCH codes [289], [191]; and the was begun largely to commemorate the twenty-fifth inversion of multivariable linear systems [357], [361], anniversaryT of the publication of Shannon’s classic paper [362]. There are also more purely mathematical ramifica- on information theory. However, 1974 is also twenty-five tions in Hilbert-space theory, operator theory, and more years after the publication in the open literature of Wiener’s generally in functional analysis [245], [250], [261]. famous monograph, “Extrapolation, Interpolation and The section headings give a quick idea of the scope of Smoothing of Stationary Time Series, with Engineering the paper. Applications” [I], so that it is appropriate this year to I. Introduction. commemorate this event as well. [As noted elsewherein this II. A Key Linear Estimation Problem. issue, this month is also the tenth anniversary of Wiener’s III. Wiener Filters and Early Generalizations. death (March 18, 1964).] Not only was this work the direct IV. Kalman Filters. causefor the great activity of the last three decadesin signal V. Recursive Wiener Filters. estimation, but it was perhaps the greatestfactor in bringing VI. New Algorithms for Time-Invariant Systems. the statistical point of view clearly into communication VII. Some Early Mathematical Work. theory and also control theory. It may suffice to quote VIII. Canonical Representations of Continuous-Time Shannon’sown major acknowledgment: “Credit should also Processes. be given to Professor N. Wiener, whose elegant solution IX. Recent Results on Innovations Processes and of the problems of filtering and prediction of stationary Some Applications. ensembleshas considerably influenced the writer’s thinking X. Karhunen-Lo&e Expansions, Canonical Correla- in this field.” tions and State Models. The subject of estimation is a vast one, and most of our XI. Concluding Remarks. attention will be devoted to the particular problems of XII. Bibliography. linear least-squares estimation, or linear Jiltering as it has generally come to be called in the engineering literature. Needless to say, the choice of material and emphasis Even though least-squares estimation is clearly only a in this paper are mine; the field is a vast one and can be small part of the possible forms of estimation theory, in surveyed in various ways. My main aims are to provide the author’s opinion it is perhaps the most interesting and some perspective on presently used methods, to bring out most important part. Least-squarestheory not only pro- the significance and relevance of some relatively early, vides useful solutions to certain specific estimation prob- but often neglected, work in this field, and to illustrate lems, but it also has connections to and implications for a some of the connections between least-squarestheory and surprisingly large number of other problems, both statistical other fields. and deterministic. As some examples we mention signal In Section II we formulate the problem of determining detection [93], [291]; the calculation of mutual information the causal linear least-squaresestimate of a signal process in certain channels [303] ; the solution of integral equations corrupted by additive white noise. Although this is only [288], [292], two-point boundary value problems in many one of a large number of possible estimation problems, it is a key one in the sensethat its solution underlies that of many others. Manuscript received June 15, 1973; revised October 3, 1973. This work was supported in part by the Air Force Office of Scientific Sections III-VI describe some of the current approaches Research, Air Force Systems Command, under Contract AF 44-620- to solution of this key problem. Information theorists and 69-C-0101, and in part by the Joint Services Electronics Program communication engineers have been more familiar with under Contract N-00014-67-A-01 12-0044. The author is with Stanford University, Stanford, Calif. 94305. problems in which covariance information is given about KAILATH: LINEAR FILTERING THEORY 147 signal and noise, which are usually called Wiener filtering The bibliography is of necessity rather vast, although problems. Control engineersdeal more often with prob- it could easily have beeneven larger. On severaloccasions, lems wherethe signal and noise are describedby state-space the availability of convenientbibliographies has led me to models, usually called Kalman filtering problems. Because omit various references.This is undoubtedly an injustice of differing backgrounds,mutual knowledge of these two to the authors of many fine papers, but it seemsto be generalapproaches is often small, and one of our aims is unavoidable.I haveattempted to do more justice to papers to bring out the close and useful relation that must exist published in this journal, and in fact all suchlinear filtering betweenthese two approaches.O f courseno proofs can be papers in the period 1968-1972have been included in the given here, but the main results are stated and their signif- bibliography, even though no explicit referencemay have icanceand role explained.Appropriate referencesare given beenmade to them in the text. This has also beendone for for the proofs. certain other papers appearing in other journals that I The discussionin SectionsIII-VI is fairly self-contained, feel contain some ideas or approachesthat may appeal to but at various points allusions are made to earlier results our readers. The choice is of necessity rather subjective in the mathematical literature, especially of the forties. and any omissions should be regardedas a measureof my This work is explored in SectionsVII and VIII, partly for ignorancerather than a consciousslight. the record but really becauseit ’contains ideas that in my The bibliography is organized under five subheadings, opinion have still not yet beenadequately appreciated and though the division of papers between the five categories exploited. For example, the work of Krein (1944)and of is on occasion somewhat arbitrary, partly becausethe Levinson (1947)is only now beginning to be rediscovered fields are of coursenot completely exclusive.In retrospect, and extended. Limitations of space again prevent any some reassignmentswould really have been desirable, but detaileddiscussion, but I havetried to provide someguidance I have not had the courageor the time to attempt them. for a reader interested in further study. Moreover, even a I must repeat that the inevitable limitations of time, casual reader might find some fascinating nuggetsexposed space, and personal knowledge are undoubtedly reflected here, although I should stressthat the lode is really much in this survey.The only palliative I can offer is that perusal richer. of the various referenceswill enablethe readerto learn many In Section IX, I have described in somewhat more additional facts and results that could not be covered in detail one vein that I have personally found to be very the paper and to make his own judgment of any controv- illuminating and powerful: the role of canonical or in- ersial matters. novations representationsof random processes.Again I 1I.A KEY LINEAR ESTIMATION PROBLEM havegiven only referencesto many results and applications, but I could not resist being a little more specific about Some Early History oneaspect; name ly, the connectionsto spectralfactorization From the earliest times, peoplehave been concerned with and to the so-called positive real functions of network interpreting observations and making estimates and pre- and complex variable theory. The aim is to show at least dictions. Neugebauer[370] has noted that the Babylonians one explicit connection between stochastic and deter- useda rudimentary form of Fourier seriesfor suchpurposes. ministic problems. A fact I attempt to stress in Sections As with so much else, the beginnings of a “theory” of VIII and IX is the importance of deterministic system estimation in which attempts are made to minimize various structure in the theory of random processes.This is at the functions of the errors can apparently be attributed to moment an active field of research,but one that is being Galileo Galilei in 1632[174]. Then came a whole seriesof largely carried out in control theory. There is scope for illustrious investigators, including the young Roger Cotes many more communication theorists to enter this field. (of whom Newton said “had he lived, we might haveknown Conversely, in the last section I have briefly referred to something”), Euler, Lagrange, Laplace, Bernoulli, and some recent results in information theory that have useful others. implications for estimation. The time seemsto be ripe for a As is well known, the method of least squares was fruitful symbiosis. apparently first usedby Gaussin 1795[197], though it was Section X is a brief look at the role and significance of first published by Legendrein 1805 [198]. (It is less well seriesexpansions. These should be relatively more familiar known that Adrain in America, unaware of thesedevelop- to readersof this journal, and I havetherefore concentrated ments, independentlydeveloped the method in 1808[196]). on some special, but often overlooked, aspects of such Since then, there has been a vast literature on various expansions.The sectionconcludes in fact with an indication aspects of the least-squaresmethod.