VISUAL TRACKING USING SEQUENTIAL IMPORTANCE SAMPLING WITH A STATE PARTITION TECHNIQUE Yan Zhai 1, Mark Yeary 1, Joseph P. Havlicek 1 Jean-Charles Noyer 2, Patrick Lanvin 2 1School of Electrical and Computer Engineering University of Oklahoma, Norman, OK USA 2Laboratoire d’ Analyse des Syst`emes du Littoral Universit¶e du Littoral C^ote d’Opale, Calais Cedex, France Corresponding Author’s E-mail: [email protected], [email protected] ABSTRACT tion filter (JPDAF) [1]. Hidden Markov models (HMM) Sequential importance sampling (SIS), also known as par- have also been widely used [2, 3]. ticle filtering, has drawn increasing attention recently due More recently, a sequential importance sampling (SIS) to its superior performance in nonlinear and non-Gaussian technique, viz. particle filtering (PF), has been recognized dynamic problems. In the SIS framework, estimation accu- as a significant target tracking algorithm because of its ac- racy depends strongly on the choice of proposal distribution. curacy, robustness and flexibility in non-linear and non- In this paper we propose a novel SIS algorithm called PF- Gaussian scenarios [4]. Various PF tracking approaches SP-PEKF that is based on a state partition technique and have been reported, including the condensation method [5], a parallel bank of extended Kalman filters designed to im- switching PF [6], unscented PF [7] and others. However, prove the accuracy of the proposal distribution. Our results performance of these techniques is strongly influenced by show that this new approach yields a significantly improved the choice and accuracy of the proposal distribution (PD). estimate of the state, enabling the new particle filter to ef- In this paper, we introduce a new PF approach where the fectively track human subjects in a video sequence where PD is generated using the state partition technique and mul- the standard condensation filter fails to maintain track lock. tiple weighted EKFs. This technique provides highly accu- Moreover, because of the improved proposal distribution, rate and robust state estimates in the presence of noise and the new filter can achieve a given level of performance us- strong clutter, which improves tracking performance while ing fewer particles than its conventional SIS counterparts. simultaneously reducing the need for a large number of par- ticles. As an illustrative example, we apply the new ap- proach to track human perambulation in a highly cluttered 1. INTRODUCTION environment. Robust, accurate visual object tracking is fundamental to a variety of computer vision applications including robotics, 2. SEQUENTIAL IMPORTANCE SAMPLING human tracking and biometric identification, intelligent transportation systems, smart rooms, and military target- The conventional PF technique [4] is based on a nonlinear ing systems. Usually, the objects of interest in a video se- system modeled in state space according to quence are represented by state space models that may in- x(t) =f(x(t ¡ 1)) + v(t ¡ 1) volve strong nonlinearities. In addition, the noise and back- y(t) =h(x(t)) + n(t) (1) ground clutter that are almost always present in real-world video sequences make the visual tracking problem particu- where x(t) and y(t) denote the state vector and observations larly challenging. Conventional methods for dealing with and where v(t) and n(t) are the process and observation nonlinear and non-Gaussian problems include the extended noises. The filtering objective is to simulate the posterior Kalman filter (EKF), Bayesian multiple-hypothesis filters, distribution p(x(0:t)jy(1:t)) via a large set of randomly gen- and approximate variants including the probability data as- erated samples called particles. Applying Bayes’ theorem, sociation filter (PDAF) and joint probability data associa- this distribution is given by p(y(1:t)jx(0:t))p(x(0:t)) This work was supported in part by the U.S. Army Research Labora- p(x(0:t)jy(1:t)) = R : tory and the U.S. Army Research Office under grant W911NF-04-1-0221. p(y(1:t)jx(0:t)p(x(0:t)) dx(0:t) 0-7803-9134-9/05/$20.00 ©2005 IEEE SIS approximates the posterior distribution according to where F(¢) denotes the Jacobian. From (5), we have that Ns X xn (t) + xr (t) ¼ f(xn (t ¡ 1)) + vn (t ¡ 1) p(x(0:t)jy(1:t)) ¼ !~(i)(t)±(x(0:t) ¡ x(i)(0:t)) ; (2) i i i i ¡ v (t ¡ 1) + F(x (t ¡ 1))x (t ¡ 1) + v(t ¡ 1); i=1 ni ni ri where Ns is the number of particles. The normalized im- which can be simplified to obtain portance weight !~(i)(t) is given by xri (t) ¼ F(xni (t¡1))xri (t¡1)+v(t¡1)¡vni (t¡1) : (6) !(i)(t) !~(i)(t) = P ; (3) Manipulating the output equation in a similar way yields Ns (j) j=1 ! (t) y(t) ¼ h(xni (t)) + H(xni (t))xri (t) + n(t); (7) where !(i)(t) denotes the importance weight associated with the ith particle at time t, given by where H(¢) is the Jacobian. Note that the dynamic model given by (6) and (7) approximates the nonlinear system (1) (i) (i) (i) but is linear in the residual part of the state xri (t). Thus, a (i) (i) p(y(t)jx (t)) p(x (t)jx (t ¡ 1)) ! (t) = ! (t ¡ 1) : bank of EKFs can be applied to update xr (t) and the overall q(x(i)(t)jx(i)(0:t ¡ 1); y(1:t)) i system state is estimated by the weighted sum (4) The particles x(i)(t) are drawn from the PD x(i)(t) » XN (i) q(x(t)jx (0:t ¡ 1) ; y (1:t)): Designing a proper PD q(¢) x^(tjt) = x^i(tjt)wi(t); (8) associated with the SIS is one of the greatest challenges in i=1 implementing a PF and can profoundly affect the filter per- where x^i(tjt) = xni (t) +x ^ri (tjt): The filter weights wi(t) formance. In the next section, we introduce a new, robust in (8) are given by approach to this important task. Li(tjt)wi(t ¡ 1) wi(t) = PN ; (9) 3. NEW PF ALGORITHM (PF-SP-PEKF) i=1 Li(tjt)wi(t ¡ 1) where It is well-known that using the state transition prior as the h 1 proposal (also known as condensation) is not robust in L (tjt) = jR (tjt ¡ 1)j¡0:5 exp ¡ k y(t) i yi 2 strongly cluttered environments, since this proposal does i 2 ¡1 not include the most recent observations [4, 7]. In this sec- ¡ y^i(tjt ¡ 1) k Ryi (tjt ¡ 1) : (10) tion, we suggest a new PD based on the state partition tech- nique and a parallel bank of EKFs (SP-PEKF) [8]. We Finally, the overall estimation error covariance Rtotal is briefly review the SP-PEKF method and then show how it XN can be incorporated within the PF framework. Rtotal(tjt) = fRi(tjt) + [^x(tjt) ¡ x^i(tjt)] SP-PEKF is a technique for calculating the statistics of i=1 T the states of a nonlinear system. The rationale is to generate £ [^x(tjt) ¡ x^i(tjt)] gwi(t) : (11) a set of samples xi, i 2 [1;N], associated with each state x according to the given initial distribution N(x(0);R(0)). Our unique contribution in this paper is developing a new PF algorithm (PF-SP-PEKF) which uses the SP-PEKF state These samples are partitioned as x(t) , xn(t) + xr(t), estimates for its PD q(¢). At each iteration, particles are where xn(t) and xr(t) denote the nominal and residual parts of the true state, respectively. After partitioning, the sam- drawn from a normal distribution with mean (8) and co- ples are propagated through a parallel filter bank and the es- variance (11). These particles are then propagated as in timated states are generated as a weighted sum of the filtered the conventional PF according to (2)-(4) with a resampling step. In order to take maximum advantage of the state es- samples. Initially, samples xi(0) are generated according to timates derived from the particle filter, the final estimates x^(0) = xn (0) +x ^r (0) and R(0) = Rn (0) + Rr (0), i i i i from the combined approach are recursively fed back into where xn (0) =x ^(0), x^r (0) = 0, Rn (0) = R(0), and i i i the SP-PEKF filter. This update stage serves as a “correc- Rri (0) = 0. Here, Rni and Rri represent the covariance of the nominal state and the residual state, respectively. tion” step for the SP-PEKF filter at each iteration. We typ- ically choose the number of filter bank channels N in (8)- The nominal state is updated according to xni (t) = (11) much smaller than the number of particles Ns; our ex- f(xni (t¡1))+vni (t¡1), where vni (t) is distributed iden- tically to v(t) but is a different realization. The system state perience is that taking 10 · N · 20 is generally suffi- cient to obtain significantly improved performance relative trajectory is then linearized about xn (t) according to i to, e.g., a comparable condensation filter. The overall PF- x(t) ¼ f(x (t ¡ 1)) ni SP-PEKF algorithm is illustrated in Fig. 1 and summarized + F(xni (t ¡ 1))xri (t ¡ 1) + v(t ¡ 1); (5) below. The PF-SP-PEKF algorithm: Calculate Particle Weights (3) (1) EKF1 ω (t) ² Sequential Importance Sampling (SIS) Step: (2) (t) EKF2 ω – Sample from the proposal distribution ac- Normalize (3) cording to (8) and (11): ω (t) Particle EKF Output 3 Weights & (i) (i) (i) x (t) » q(x (t)jx (0:t ¡ 1); y(1:t)) Observations Resample Generate Particles =N (^x(tjt);Rtotal(tjt)) ; (Ns) Update using weighted sum given in (8) − (11) – Evaluate and normalize the importance EKFN Evaluate estimated mean and variance ω (t) weights according to (3) and (4); ² Resampling Step: Generate a new set of particles Fig.