Sparse Penalties in Dynamical System Estimation
Total Page:16
File Type:pdf, Size:1020Kb
1 Sparsity Penalties in Dynamical System Estimation Adam Charles, M. Salman Asif, Justin Romberg, Christopher Rozell School of Electrical and Computer Engineering Georgia Institute of Technology, Atlanta, Georgia 30332-0250 Email: facharles6,sasif,jrom,[email protected] Abstract—In this work we address the problem of state by the following equations: estimation in dynamical systems using recent developments x = f (x ) + ν in compressive sensing and sparse approximation. We n n n−1 n (1) formulate the traditional Kalman filter as a one-step update yn = Gnxn + n optimization procedure which leads us to a more unified N framework, useful for incorporating sparsity constraints. where xn 2 R represents the signal of interest, N N We introduce three combinations of two sparsity conditions fn(·)jR ! R represents the (assumed known) evo- (sparsity in the state and sparsity in the innovations) M lution of the signal from time n − 1 to n, yn 2 R and write recursive optimization programs to estimate the M is a set of linear measurements of xn, n 2 R state for each model. This paper is meant as an overview N of different methods for incorporating sparsity into the is the associated measurement noise, and νn 2 R dynamic model, a presentation of algorithms that unify the is our modeling error for fn(·) (commonly known as support and coefficient estimation, and a demonstration the innovations). In the case where Gn is invertible that these suboptimal schemes can actually show some (N = M and the matrix has full rank), state estimation performance improvements (either in estimation error or at each iteration reduces to a least squares problem. convergence time) over standard optimal methods that use an impoverished model. Therefore we will be particularly interested in the case of highly underdetermined measurements of the state vector Index Terms—Compressive Sensing, Dynamical Systems, (i.e., M N), which will require us to leverage of State Estimation knowledge of the state dynamics and signal structure to I. INTRODUCTION accurately estimate the state at each iteration. One special, well studied, case of the system (1) In data analysis, signal models play a crucial role in assumes Gaussian statistics on both the measurement our approach to acquiring and processing signals and noise and the modeling error (n ∼ N (0; Qn), νn ∼ images. This has been especially clear in recent work N (0; Rn)), and a linear state evolution function (i.e. where sparsity-based models have enabled dramatic im- fn(x) = Fnx). The solution in this case is known as provements in the solution to linear inverse problems, the Kalman filter [3] which estimates the optimal state especially in the context of compressive sensing (CS) for at time n given the current set of measurements and data acquisition from undersampled measurements [1], the previous state estimate and its covariance. While [2]. The excitement about these results is compounded the Kalman filter is least-squares optimal in this special by the simplicity: the algorithms often rely on solving an case, it cannot incorporate other innovations statistics or `1-regularized least squares problem that can be solved additional information about the structure of the evolving with reasonable efficiency. signal. While many applications involve static signal estima- One type of signal structure which has been of par- tion, many more (e.g., video) have additional statistics ticular interest recently is a low dimensional (sparse) in the temporal signal evolution that could be exploited. signal structure. Compressive sensing results show that For example, though each video frame may have a knowledge of such a structure can allow for low signal sparse decomposition in some basis, the regular motion recovery error in highly undersampled systems. While in the scene is likely to induce regular changes in the such structure has been leveraged to great success in coefficients from one frame to the next. In the classic static signal processing problems, the question of how to dynamical systems literature, a model for a changing successfully and efficiently utilize sparsity information in state vector and measurement process is often described dynamic signal estimation remains a topic of continued research. n can be solved for by a temporally local calculation The framework we present in this paper seeks to which yields the same solution as performing the calcu- capture the essence of Kalman filtering by minimizing lation intensive inverse, thereby immensely reducing the the estimation complexity at each iteration while still complexity in terms of the inverse problem that needs to accounting for both the approximately known dynamics be solved. Additionally, given the least-squares nature as well as the knowledge of sparsity in the system. This of the objective, the problem setup permits an analytic approach is most similar to that taken in [4] which uses solution. The efficiency given both the analytic and a similar explicit optimization function to account for local nature of the solution, coupled with the increasing sparse measurement noise. While not globally optimal calculation speed of matrix operations has opened the in the sense of traditional Kalman approaches (since door for optimal, fast, realizable tracking in a framework the proposed algorithms only use the previous state common enough to be widely applicable. information to estimate the current state), we show that The underlying assumptions the Kalman filter makes the proposed approaches can show gains in estimation on the system (linearity and Gaussianity), however, error or convergence time of the estimate over both can be quite restrictive in broader settings: Any de- the Kalman filter or independent CS recovery at each viation may cause the algorithm to yield erroneous time step (which are both operating with impoverished solutions. Many modifications have been devised to models due to ignoring sparsity or temporal regularity, address this shortcoming including the Extended and respectively). We present in this paper three different Unscented Kalman filters [5], [6] to address nonlinear ways to incorporate sparsity models into a dynamical state dynamics and many variations of robust Kalman system that result in estimation algorithms that are filters have been designed to address non-Gaussian noise intuitive modifications to the `1-regularized least squares models [7], [8]. Since most system and noise models problem commonly used in CS. Specifically, the three are difficult to account for analytically and efficiently, forms of sparsity in the dynamical system we propose most of these extensions attempt to modify the resulting are 1) the evolving signal xn is sparse, 2) the error in the Kalman filter algorithm directly in order to increase the signal prediction νn is sparse and 3) both the signal and robustness of the estimation. The one main shortcoming the prediction error are sparse. For each of these three of the Kalman filter and its derivatives is that none of the signal models we write a CS-like optimization program algorithms explicitly take into account any underlying to solve for the signal at each time step. signal structure (such as sparsity in the signal). II. BACKGROUND AND RELATED WORK B. Compressive Sensing A. Classical Linear State Estimation With the introduction of compressive sensing tech- niques, we are now equipped to efficiently deal with an The task of estimating the evolving state xn has become the topic of innumerable studies and algorithms, entirely new set of signals: signals which are sparse in the most notable of which is the aforementioned Kalman some basis. Many classical least-squares problems have filter [3]. The optimal least squares estimate sought benefited greatly from utilizing the inherent low dimen- by the Kalman filter essentially requires the inversion sionality of the signal. Most notably, CS algorithms can of a large matrix with submatrices consisting of the increase the recovery capability of a sparse vector x from many fewer linear measurements y = Gx than measurement matrices Gn, the dynamics matrices Fn, and the identity matrix. For n very small, a small number would otherwise be possible with least-squares solutions of measurements per iteration means that this matrix is (M N). The recovery, given a certain sparsity on underdetermined, and thus the inverse will generally be x and compliance with certain conditions on G (G erroneous. As n gets very large, this matrix becomes satisfies the RIP condition), is performed by regularizing more and more complete (the number of columns grows the classical least squares optimization program with an as nN and the number of rows grows as n(N + M)), `1 norm, which means that the error in the solution given by 2 x^ = arg min ky − Gxk2 + λkxk1 (2) jointly estimating all states via a large matrix inverse x fast approaches the noise floor. where λ is a parameter which trades off data fidelity The main result of the Kalman filter shows that the full with sparsity. This optimization program is known as the matrix inverse need not be calculated if, at each iteration, Basis Pursuit De-Noising (BPDN). While many fields, we are interested only in the estimate of the current state such as image processing, have benefited greatly from given previous measurements. Instead each state at time CS results, the application to dynamic signal estimation 2 has not been so straight-forward. In addition to the actual by the total optimization over the entire time-line problem of how the estimation should be updated at each " n iteration, the question of where in the system sparsity n X 2 fx^kgk=0 = arg min kyk − Gkxkk −1 fx gn Qk ;2 could be leveraged to increase estimation performance k k=0 k=0 is not well addressed. We seek in this work to explore n # X 2 where in the dynamic system it may be useful to consider + kxk − Fkxk−1k −1 ; (3) Rk ;2 sparse underlying structures as well as how this sparsity k=1 may be leveraged.