Robust Visual Motion Analysis: Piecewise-Smooth Optical Flow and Motion-Based Detection and Tracking Ming Ye A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy University of Washington 2002 Program Authorized to Offer Degree: Electrical Engineering University of Washington Abstract Robust Visual Motion Analysis: Piecewise-Smooth Optical Flow and Motion-Based Detection and Tracking by Ming Ye Co-Chairs of Supervisory Committee: Professor Robert M. Haralick Electrical Engineering Professor Linda G. Shapiro Computer Science and Engineering This thesis describes new approaches to optical flow estimation and motion-based detection and tracking. Statistical methods, particularly outlier rejection, error analysis and Bayesian inference, are extensively exploited in our study and are shown to be crucial to the robust analysis of visual motion. To recover optical flow, or 2D velocity fields, from image sequences, certain models of brightness conservation and flow smoothness must be assumed. Thus how to cope with model violations especially motion discontinuities becomes a very challenging issue. We first tackle this problem from a local approach, that is, finding the most representative flow vector for each small image region. We recast the popular gradient-based method as a two-stage regression problem and apply adaptive robust estimators to both stages. The estimators are adaptive in the sense that their complexity increases with the amount of outlier contamination. Due to the limited contextual information, the local approach has spatially varying uncertainty. We evaluate the uncertainty systematically through covari- ance propagation. Pointing out the limitations of local and gradient-based methods, we further propose a matching-based global optimization technique. The optimal estimate is formulated as maximizing the a posteriori probability of the optical flow given three image frames. Using a Markov random field flow model and robust statistics, the formulation reduces to mini- mizing a regularization type of global energy function, which we carefully design so as to accommodate outliers, occlusions and local adaptivity. Minimizing the resulting large-scale nonconvex function is nontrivial and is often the performance bottleneck of previous global techniques. To overcome this problem, we develop a three-step graduated solution method which inherits the advantages of various popular approaches and avoids their drawbacks. This technique is highly efficient and accurate. Its performance is demonstrated through experiments on both synthetic and real data and comparison with competing techniques. By making only weak assumptions of spatiotemporal continuity, the two proposed tech- niques are applicable to general scenarios, for example, to both rigid and nonrigid motion. They serve as a foundation for object-based motion analysis. Many of their conclusions are also extendable to other visual surface reconstruction problems such as image restoration and stereo matching. The last part of the thesis describes a motion-based detection and tracking system designed for an airborne visual surveillance application, in which challenges arise from the small target size (1 £ 2 ¡ 3 £ 3 pixels), low image quality, substantial camera wobbling and plenty of background clutters. The system is composed of a detector and a tracker. The former identifies suspicious objects by the statistical difference between their motion and the background motion; the latter employs a Kalman filter to track the dynamic behavior of objects in order to detect real targets and update their states. Both components operate in a Bayesian mode, and each benefits from the other’s accuracy. The system exhibits excellent performance in experiments. In an 1800-frame real video, it produces no false detections and tracks the true target since the second frame, with average position error below 1 pixel. This probabilistic approach reduces parameter tuning to a minimum. It also facilitates data fusion from different information channels. TABLE OF CONTENTS List of Figures iv List of Tables vi Chapter 1: Introduction 1 1.1 Optical Flow Estimation . 4 1.2 A Local Method with Error Analysis . 9 1.3 A Global Optimization Method . 10 1.4 Motion-Based Target Detection and Tracking . 12 1.5 Thesis Outline . 13 Chapter 2: Estimating Optical Flow: Approaches and Issues 15 2.1 Brightness Conservation . 16 2.2 Flow Field Coherence . 19 2.3 Typical Approaches . 20 2.4 Robust Methods . 23 2.5 Error Analysis . 28 2.6 Hierarchical Processing . 30 Chapter 3: Local Flow Estimation and Error Analysis 34 3.1 A Two-Stage-Robust Adaptive Technique . 34 3.1.1 Linear Regression and Robustness . 35 3.1.2 Two-Stage Regression Model . 40 3.1.3 Choosing Estimators . 42 3.1.4 Experiments and Analysis . 43 i 3.2 Adaptive High-Breakdown Robust Methods For Visual Reconstruction . 50 3.2.1 The Approach . 51 3.2.2 Experiments and Analysis . 53 3.2.3 Discussion . 58 3.3 Error Analysis on Robust Local Flow . 61 3.3.1 Covariance Propagation . 61 3.3.2 Experiments . 65 3.3.3 Discussion . 67 Chapter 4: Global Matching with Graduated Optimization 70 4.1 Formulation . 71 4.1.1 MAP Estimation . 72 4.1.2 MRF Prior Model . 72 4.1.3 Likelihood Model: Robust Three-Frame Matching . 74 4.1.4 Global Energy with Local Adaptivity . 75 4.2 Optimization . 76 4.2.1 Step I: Gradient-Based Local Regression . 77 4.2.2 Step II: Gradient-Based Global Optimization . 77 4.2.3 Step III: Global Matching . 78 4.2.4 Overall Algorithm . 79 4.3 Experiments . 81 4.3.1 Quantitative Measures . 82 4.3.2 TS: An Illustrative Example . 82 4.3.3 Barron’s Synthetic Data . 85 4.3.4 Real Data . 90 4.4 Conclusions and Discussion . 93 Chapter 5: Motion-Based Detection and Tracking 96 5.1 Bayesian State Estimation . 98 ii 5.2 Kalman Filter . 100 5.3 Tracking . 101 5.4 Motion-Based Detection . 103 5.5 Bayesian Detection . 111 5.6 The Algorithm . 113 5.7 Experiments . 114 5.8 Discussion . 116 Chapter 6: Conclusions 117 6.1 Summary and Contributions . 117 6.2 Open Questions and Future Work . 121 Bibliography 125 iii LIST OF FIGURES 1.1 Example Optical flow on flower garden sequence . 2 1.2 Motion estimation by template matching . 5 1.3 Motion analysis for airborne video surveillance . 12 2.1 Aperture problem . 29 2.2 Hierarchical processing . 32 3.1 Comparison of Geman-McClure norm and L2 norm . 38 3.2 Block diagram of the two-stage-robust adaptive algorithm . 44 3.3 Central frame of the synthetic sequence (5 frames, 32 £ 32) . 45 3.4 Correct flow field . 45 3.5 OFC cluster plots at three typical pixels . 46 3.6 TS sequence results . 47 3.7 Pepsi sequence results . 48 3.8 Pepsi: estimated flow fields . 49 3.9 Random sampling based algorithm for high-breakdown robust estimators . 51 3.10 Adaptive algorithm for high-breakdown robust estimators . 52 3.11 TS: trial set size map . 54 3.12 TS: correct and estimated flow fields . 55 3.13 TT, DT middle frame . 57 3.14 YOS middle frame . 57 3.15 OTTE sequence . 58 3.16 TAXI sequence results . 59 3.17 TAXI: intensity images of x-component . 60 iv 3.18 TS motion boundary . 66 3.19 TAXI motion boundary . 68 3.20 TAXI: motion boundary on images subsampled by 2 . 69 4.1 Comparison of Geman-McClure norm and L2 norm . 73 4.2 System diagram (operations at each pyramid level) . 80 4.3 TS sequence results . 83 4.4 Error cdf curves. 86 4.5 DTTT sequence results (motion boundaries highlighted in (a)). 88 4.6 Taxi results. 89 4.7 Flower garden results. 91 4.8 Traffic results. 92 4.9 Pepsi can results. 93 5.1 A typical detection-tracking system . 97 5.2 Proposed Bayesian system . 98 5.3 Example data sets . 105 5.4 f16502 target pixel candidates . 106 5.5 f18300 and f19000 target pixel candidates . 108 5.6 Target pixels for f16502, f18300 and f19000 . 110 5.7 Detection results w and w/o priors on f16503 . 112 5.8 Two sample frames . ..