Technischen Universität München Winter Semester 2012/2013
TRACKING and DETECTION in COMPUTER VISION Mean Shift Tracking
Slobodan Ilić Agenda
Mean Shift Theory • What is Mean Shift ? • Density Estimation Methods • Deriving the Mean Shift • Mean shift properties
Applications • Clustering • Object Tracking
Mean Shift Tracking Ilic Slobodan Intuitive Description
Objective : Find the densest region Distribution of identical billiard balls Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Objective : Find the densest region Distribution of identical billiard balls Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Center of mass
Objective : Find the densest region Distribution of identical billiard balls Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Center of mass
Mean Shift Objective : Find the densest region vector Distribution of identical billiard balls Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Center of mass
Mean Shift Objective : Find the densest region vector Distribution of identical billiard balls Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Center of mass
Mean Shift Objective : Find the densest region vector Distribution of identical billiard balls Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Center of mass
Mean Shift Objective : Find the densest region vector Distribution of identical billiard balls Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Center of mass
Mean Shift Objective : Find the densest region vector Distribution of identical billiard balls Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Center of mass
Mean Shift Objective : Find the densest region vector Distribution of identical billiard balls Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Center of mass
Mean Shift Objective : Find the densest region vector Distribution of identical billiard balls Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Center of mass
Mean Shift Objective : Find the densest region vector Distribution of identical billiard balls Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Center of mass
Mean Shift Objective : Find the densest region vector Distribution of identical billiard balls Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Center of mass
Mean Shift Objective : Find the densest region vector Distribution of identical billiard balls Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Center of mass
Mean Shift Objective : Find the densest region vector Distribution of identical billiard balls Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Center of mass
Mean Shift Tracking Ilic Slobodan Intuitive Description Region of interest
Center of mass
Mean Shift Tracking Ilic Slobodan What is Mean Shift ?
A tool for: Finding modes in a set of data samples representing an underlying probability density function (PDF) in RN
Mean Shift Tracking Ilic Slobodan What is Mean Shift ?
A tool for: Finding modes in a set of data samples representing an underlying probability density function (PDF) in RN
PDF in feature space
Mean Shift Tracking Ilic Slobodan What is Mean Shift ?
A tool for: Finding modes in a set of data samples representing an underlying probability density function (PDF) in RN
PDF in feature space • Color space
Mean Shift Tracking Ilic Slobodan What is Mean Shift ?
A tool for: Finding modes in a set of data samples representing an underlying probability density function (PDF) in RN
PDF in feature space • Color space • Scale space
Mean Shift Tracking Ilic Slobodan What is Mean Shift ?
A tool for: Finding modes in a set of data samples representing an underlying probability density function (PDF) in RN
PDF in feature space • Color space • Scale space • Actually any feature space you can conceive
Mean Shift Tracking Ilic Slobodan What is Mean Shift ?
A tool for: Finding modes in a set of data samples representing an underlying probability density function (PDF) in RN
PDF in feature space • Color space • Scale space • Actually any feature space you can conceive • …
Mean Shift Tracking Ilic Slobodan What is Mean Shift ?
A tool for: Finding modes in a set of data samples representing an underlying probability density function (PDF) in RN
Data
Mean Shift Tracking Ilic Slobodan What is Mean Shift ?
A tool for: Finding modes in a set of data samples representing an underlying probability density function (PDF) in RN
Non-parametric Density Estimation Discrete PDF Representation
Data
Mean Shift Tracking Ilic Slobodan What is Mean Shift ?
A tool for: Finding modes in a set of data samples representing an underlying probability density function (PDF) in RN
Non-parametric Density Estimation Discrete PDF Representation
Data
Non-parametric Density GRADIENT Estimation (Mean Shift) PDF Analysis Mean Shift Tracking Ilic Slobodan Non-Parametric Density Estimation
Assumption : The data points are sampled from an underlying PDF
Assumed Underlying PDF Real Data Samples
Mean Shift Tracking Ilic Slobodan Non-Parametric Density Estimation
Assumption : The data points are sampled from an underlying PDF
Assumed Underlying PDF Real Data Samples
Mean Shift Tracking Ilic Slobodan Non-Parametric Density Estimation
Assumption : The data points are sampled from an underlying PDF
Assumed Underlying PDF Real Data Samples
Mean Shift Tracking Ilic Slobodan Non-Parametric Density Estimation
Assumption : The data points are sampled from an underlying PDF
Assumed Underlying PDF Real Data Samples
Mean Shift Tracking Ilic Slobodan Non-Parametric Density Estimation
Assumption : The data points are sampled from an underlying PDF
Assumed Underlying PDF Real Data Samples
Mean Shift Tracking Ilic Slobodan Non-Parametric Density Estimation
Assumption : The data points are sampled from an underlying PDF
Data point density implies PDF value !
Assumed Underlying PDF Real Data Samples
Mean Shift Tracking Ilic Slobodan Non-Parametric Density Estimation
Assumed Underlying PDF Real Data Samples
Mean Shift Tracking Ilic Slobodan Non-Parametric Density Estimation
Assumed Underlying PDF Real Data Samples
Mean Shift Tracking Ilic Slobodan Non-Parametric Density Estimation
Assumed Underlying PDF Real Data Samples
Mean Shift Tracking Ilic Slobodan Non-Parametric Density Estimation
Assumed Underlying PDF Real Data Samples
Mean Shift Tracking Ilic Slobodan Non-Parametric? Density Estimation
Assumed Underlying PDF Real Data Samples
Mean Shift Tracking Ilic Slobodan Parametric Density
Assumption : The data points are sampled from an underlying PDF
Assumed Underlying PDF Real Data Samples
Mean Shift Tracking Ilic Slobodan Parametric Density
Assumption : The data points are sampled from an underlying PDF µ
Assumed Underlying PDF Real Data Samples
Mean Shift Tracking Ilic Slobodan Parametric Density
Assumption : The data points are sampled from an underlying PDF µ
Estimate
Assumed Underlying PDF Real Data Samples
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Parzen Windows - Function Forms
A function of some finite number of data points x1…xn
Data
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Parzen Windows - Function Forms
A function of some finite number of data points x1…xn
Data
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Parzen Windows - Function Forms
A function of some finite number of data points x1…xn
In practice one uses the forms: Data
Same function on each dimension
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Parzen Windows - Function Forms
A function of some finite number of data points x1…xn
In practice one uses the forms: Data
or
Same function on each dimension Function of vector length only
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Various Kernels A function of some finite number of
data points x1…xn
Data
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Various Kernels A function of some finite number of
data points x1…xn
Data
Examples:
• Epanechnikov Kernel
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Various Kernels A function of some finite number of
data points x1…xn
Data
Examples:
• Epanechnikov Kernel
• Uniform Kernel
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Various Kernels A function of some finite number of
data points x1…xn
Data
Examples:
• Epanechnikov Kernel
• Uniform Kernel
• Normal Kernel
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Gradient
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Gradient
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Gradient
Give up estimating the PDF ! Estimate ONLY the gradient
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Gradient
Give up estimating the PDF ! Estimate ONLY the gradient
Using the Kernel form:
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Gradient
Give up estimating the PDF ! Estimate ONLY the gradient
Using the Kernel form:
Size of window
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Gradient
Give up estimating the PDF ! Estimate ONLY the gradient
Using the Kernel form: We get :
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Gradient
Mean Shift Tracking Ilic Slobodan Kernel Density Estimation Gradient
Mean Shift Tracking Ilic Slobodan Computing The Mean Shift
Mean Shift Tracking Ilic Slobodan Computing The Mean Shift
Mean Shift Tracking Ilic Slobodan Computing The Mean Shift
Mean Shift Tracking Ilic Slobodan Computing The Mean Shift
Yet another Kernel density estimation !
Mean Shift Tracking Ilic Slobodan Computing The Mean Shift
Yet another Kernel density estimation !
Mean Shift Tracking Ilic Slobodan Computing The Mean Shift
Yet another Kernel density estimation !
Mean Shift Tracking Ilic Slobodan Computing The Mean Shift
Mean Shift Tracking Ilic Slobodan Computing The Mean Shift
Mean Shift Tracking Ilic Slobodan Computing The Mean Shift
Mean Shift Tracking Ilic Slobodan Computing The Mean Shift
Simple Mean Shift procedure: • Compute mean shift vector
Mean Shift Tracking Ilic Slobodan Computing The Mean Shift
Simple Mean Shift procedure: • Compute mean shift vector
•Translate the Kernel window by m(x) Mean Shift Tracking Ilic Slobodan Mean Shift Mode Detection
Mean Shift Tracking Ilic Slobodan Mean Shift Mode Detection
What happens if we reach a saddle point ?
Mean Shift Tracking Ilic Slobodan Mean Shift Mode Detection
What happens if we reach a saddle point ?
Perturb the mode position and check if we return back
Mean Shift Tracking Ilic Slobodan Mean Shift Mode Detection
What happens if we reach a saddle point ?
Perturb the mode position and check if we return back
Mean Shift Tracking Ilic Slobodan Mean Shift Mode Detection
What happens if we reach a saddle point ?
Perturb the mode position and check if we return back
Mean Shift Tracking Ilic Slobodan Mean Shift Mode Detection
What happens if we reach a saddle point ?
Perturb the mode position and check if we return back
Updated Mean Shift Procedure:
Mean Shift Tracking Ilic Slobodan Mean Shift Mode Detection
What happens if we reach a saddle point ?
Perturb the mode position and check if we return back
Updated Mean Shift Procedure: • Find all modes using the Simple Mean Shift Procedure
Mean Shift Tracking Ilic Slobodan Mean Shift Mode Detection
What happens if we reach a saddle point ?
Perturb the mode position and check if we return back
Updated Mean Shift Procedure: • Find all modes using the Simple Mean Shift Procedure • Prune modes by perturbing them (find saddle points and plateaus)
Mean Shift Tracking Ilic Slobodan Mean Shift Mode Detection
What happens if we reach a saddle point ?
Perturb the mode position and check if we return back
Updated Mean Shift Procedure: • Find all modes using the Simple Mean Shift Procedure • Prune modes by perturbing them (find saddle points and plateaus) • Prune nearby – take highest mode in the window Mean Shift Tracking Ilic Slobodan Mean Shift Properties
Mean Shift Tracking Ilic Slobodan Mean Shift Properties
• Automatic convergence speed – the mean shift vector size depends on the gradient itself.
• Near maxima, the steps are small and refined
Mean Shift Tracking Ilic Slobodan Mean Shift Properties
• Automatic convergence speed – the mean shift vector size depends on the gradient itself. Adaptive Gradient Ascent • Near maxima, the steps are small and refined
Mean Shift Tracking Ilic Slobodan Mean Shift Properties
• Automatic convergence speed – the mean shift vector size depends on the gradient itself. Adaptive Gradient Ascent • Near maxima, the steps are small and refined
• Convergence is guaranteed for infinitesimal steps only infinitely convergent, (therefore set a lower bound)
Mean Shift Tracking Ilic Slobodan Mean Shift Properties
• Automatic convergence speed – the mean shift vector size depends on the gradient itself. Adaptive Gradient Ascent • Near maxima, the steps are small and refined
• Convergence is guaranteed for infinitesimal steps only infinitely convergent, (therefore set a lower bound)
• For Uniform Kernel ( ), convergence is achieved in a finite number of steps
• Normal Kernel ( ) exhibits a smooth trajectory, but is slower than Uniform Kernel ( ).
Mean Shift Tracking Ilic Slobodan Real Modality Analysis
Mean Shift Tracking Ilic Slobodan Real Modality Analysis
Run the procedure in parallel Mean Shift Tracking Ilic Slobodan Real Modality Analysis
Run the procedure in parallel Mean Shift Tracking Ilic Slobodan Real Modality Analysis
The blue data points were traversed by the windows towards the mode Mean Shift Tracking Ilic Slobodan Real Modality Analysis An example
Window tracks signify the steepest ascent directions Mean Shift Tracking Ilic Slobodan Mean Shift Strengths & Weaknesses
Mean Shift Tracking Ilic Slobodan Mean Shift Strengths & Weaknesses
Strengths :
Mean Shift Tracking Ilic Slobodan Mean Shift Strengths & Weaknesses
Strengths :
• Application independent tool
Mean Shift Tracking Ilic Slobodan Mean Shift Strengths & Weaknesses
Strengths :
• Application independent tool • Suitable for real data analysis
Mean Shift Tracking Ilic Slobodan Mean Shift Strengths & Weaknesses
Strengths :
• Application independent tool • Suitable for real data analysis • Does not assume any prior shape (e.g. elliptical) on data clusters
Mean Shift Tracking Ilic Slobodan Mean Shift Strengths & Weaknesses
Strengths :
• Application independent tool • Suitable for real data analysis • Does not assume any prior shape (e.g. elliptical) on data clusters • Can handle arbitrary feature spaces
Mean Shift Tracking Ilic Slobodan Mean Shift Strengths & Weaknesses
Strengths :
• Application independent tool • Suitable for real data analysis • Does not assume any prior shape (e.g. elliptical) on data clusters • Can handle arbitrary feature spaces • Only ONE parameter to choose
Mean Shift Tracking Ilic Slobodan Mean Shift Strengths & Weaknesses
Strengths :
• Application independent tool • Suitable for real data analysis • Does not assume any prior shape (e.g. elliptical) on data clusters • Can handle arbitrary feature spaces • Only ONE parameter to choose • h (window size) has a physical meaning, unlike K-Means
Mean Shift Tracking Ilic Slobodan Mean Shift Strengths & Weaknesses
Strengths : Weaknesses : • Application independent tool • Suitable for real data analysis • Does not assume any prior shape (e.g. elliptical) on data clusters • Can handle arbitrary feature spaces • Only ONE parameter to choose • h (window size) has a physical meaning, unlike K-Means
Mean Shift Tracking Ilic Slobodan Mean Shift Strengths & Weaknesses
Strengths : Weaknesses :
• Application independent tool • The window size (bandwidth • Suitable for real data analysis selection) is not trivial • Does not assume any prior shape (e.g. elliptical) on data clusters • Can handle arbitrary feature spaces • Only ONE parameter to choose • h (window size) has a physical meaning, unlike K-Means
Mean Shift Tracking Ilic Slobodan Mean Shift Strengths & Weaknesses
Strengths : Weaknesses :
• Application independent tool • The window size (bandwidth • Suitable for real data analysis selection) is not trivial • Does not assume any prior shape • Inappropriate window size (e.g. elliptical) on data clusters can cause modes to be • Can handle arbitrary feature merged, or generate additional spaces “shallow” modes • Only ONE parameter to choose • h (window size) has a physical meaning, unlike K-Means
Mean Shift Tracking Ilic Slobodan Mean Shift Strengths & Weaknesses
Strengths : Weaknesses :
• Application independent tool • The window size (bandwidth • Suitable for real data analysis selection) is not trivial • Does not assume any prior shape • Inappropriate window size (e.g. elliptical) on data clusters can cause modes to be • Can handle arbitrary feature merged, or generate additional spaces “shallow” modes • Only ONE parameter to choose • Use adaptive window • h (window size) has a physical size meaning, unlike K-Means
Mean Shift Tracking Ilic Slobodan Mean Shift Applications
Mean Shift Tracking Ilic Slobodan Clustering
Cluster : All data points in the attraction basin of a mode
Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer
Mean Shift Tracking Ilic Slobodan Clustering
Cluster : All data points in the attraction basin of a mode
Attraction basin : the region for which all trajectories lead to the same mode
Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer
Mean Shift Tracking Ilic Slobodan Clustering
Cluster : All data points in the attraction basin of a mode
Attraction basin : the region for which all trajectories lead to the same mode
Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer
Mean Shift Tracking Ilic Slobodan Clustering
Cluster : All data points in the attraction basin of a mode
Attraction basin : the region for which all trajectories lead to the same mode
Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer
Mean Shift Tracking Ilic Slobodan Clustering
Cluster : All data points in the attraction basin of a mode
Attraction basin : the region for which all trajectories lead to the same mode
Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer
Mean Shift Tracking Ilic Slobodan Clustering Synthetic Examples
Simple Modal Structures
Mean Shift Tracking Ilic Slobodan Clustering Synthetic Examples
Simple Modal Structures
Complex Modal Structures Mean Shift Tracking Ilic Slobodan Clustering Real Example
Feature space: L*u*v representation
Mean Shift Tracking Ilic Slobodan Clustering Real Example
Feature space: Initial window L*u*v representation centers
Modes found Modes after Final clusters pruning
Mean Shift Tracking Ilic Slobodan Clustering Real Example
L*u*v space representation
Mean Shift Tracking Ilic Slobodan Clustering Real Example
2D (L*u) space Final clusters representation
Mean Shift Tracking Ilic Slobodan Clustering Real Example
2D (L*u) space Final clusters representation
Not all trajectories in the attraction basin reach the same mode
Mean Shift Tracking Ilic Slobodan Non-Rigid Object Tracking
… …
Mean Shift Tracking Ilic Slobodan Non-Rigid Object Tracking
Real-Time Object-Based Driver Video Surveillance Assistance Compression
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking General Framework: Target Representation
… Current … frame Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking General Framework: Target Representation
Choose a reference model in the current frame
… Current … frame Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking General Framework: Target Representation
Choose a reference Choose a model in the feature space current frame
… Current … frame Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking General Framework: Target Representation
Choose a Represent the reference Choose a model in the model in the feature space chosen feature current frame space
… Current … frame Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking General Framework: Target Representation
Model Candidate … Current … frame Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking General Framework: Target Representation Start from the position of the model in the current frame
Model Candidate … Current … frame Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking General Framework: Target Representation Start from the Search in the position of the model’s model in the neighborhood in current frame next frame
Model Candidate … Current … frame Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking General Framework: Target Representation Start from the Search in the Find best position of the model’s candidate by model in the neighborhood in maximizing a current frame next frame similarity func.
Model Candidate … Current … frame Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking General Framework: Target Representation Start from the Search in the Find best position of the model’s candidate by model in the neighborhood in maximizing a current frame next frame similarity func.
Repeat the same process in the next pair of frames
Model Candidate … Current … frame Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking General Framework: Target Representation
Choose a reference target model
Kernel Based Object Tracking, by Comaniniu, Ramesh, Meer Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking General Framework: Target Representation
Choose a Choose a reference feature space target model
Quantized Color Space
Kernel Based Object Tracking, by Comaniniu, Ramesh, Meer Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking General Framework: Target Representation
Choose a Represent the Choose a reference model by its feature space target model PDF in the feature space
0.30
Quantized 0.23
Color Space 0.15 Probability 0.08
0 1 2 3 . . . m color Kernel Based Object Tracking, by Comaniniu, Ramesh, Meer Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking PDF Representation
Target Model Target Candidate (centered at 0) (centered at y)
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking PDF Representation
Target Model Target Candidate (centered at 0) (centered at y)
0.30
0.23
0.15 Probability 0.08
0 1 2 3 . . . m color
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking PDF Representation
Target Model Target Candidate (centered at 0) (centered at y)
0.30 0.30
0.23 0.23
0.15 0.15 Probability Probability 0.08 0.08
0 0 1 2 3 . . . m 1 2 3 . . . m color color
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking PDF Representation
Target Model Target Candidate (centered at 0) (centered at y)
0.30 0.30
0.23 0.23
0.15 0.15 Probability Probability 0.08 0.08
0 0 1 2 3 . . . m 1 2 3 . . . m color color
Similarity Function: Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Finding the PDF of the target model
model Target pixel locations
0
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Finding the PDF of the target model
model Target pixel locations
0 A differentiable, isotropic, convex, monotonically decreasing kernel • Peripheral pixels are affected by occlusion and background interference
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Finding the PDF of the target model
model Target pixel locations
0 A differentiable, isotropic, convex, monotonically decreasing kernel • Peripheral pixels are affected by occlusion and background interference
The color bin index (1..m) of pixel x
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Finding the PDF of the target model
model Target pixel locations
0 A differentiable, isotropic, convex, monotonically decreasing kernel • Peripheral pixels are affected by occlusion and background interference
The color bin index (1..m) of pixel x
Probability of feature u in model
0.30
0.23
0.15 Probability 0.08
0 1 2 3 . . . m Mean Shift Tracking color Ilic Slobodan Mean-Shift Object Tracking Finding the PDF of the target model
model Target pixel locations
0 A differentiable, isotropic, convex, monotonically decreasing kernel • Peripheral pixels are affected by occlusion and background interference
The color bin index (1..m) of pixel x
Probability of feature u in model
0.30
0.23 Normalization 0.15 Pixel weight factor Probability 0.08
0 1 2 3 . . . m Mean Shift Tracking color Ilic Slobodan Mean-Shift Object Tracking Finding the PDF of the target model
model candidate Target pixel locations
0 y A differentiable, isotropic, convex, monotonically decreasing kernel • Peripheral pixels are affected by occlusion and background interference
The color bin index (1..m) of pixel x
Probability of feature u in model Probability of feature u in candidate
0.30 0.30
0.23 0.23 Normalization 0.15 Pixel weight 0.15 factor Probability Probability 0.08 0.08
0 0 1 2 3 . . . m 1 2 3 . . . m Mean Shift Tracking color color Ilic Slobodan Mean-Shift Object Tracking Finding the PDF of the target model
model candidate Target pixel locations
0 y A differentiable, isotropic, convex, monotonically decreasing kernel • Peripheral pixels are affected by occlusion and background interference
The color bin index (1..m) of pixel x
Probability of feature u in model Probability of feature u in candidate
0.30 0.30
0.23 0.23 Normalization Normalization 0.15 Pixel weight 0.15 Pixel weight factor factor Probability Probability 0.08 0.08
0 0 1 2 3 . . . m 1 2 3 . . . m Mean Shift Tracking color color Ilic Slobodan Mean-Shift Object Tracking Similarity Function Target model:
Target candidate:
Similarity function:
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Similarity Function Target model:
Target candidate:
Similarity function: The Bhattacharyya Coefficient 1
1
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Similarity Function Target model:
Target candidate:
Similarity function: The Bhattacharyya Coefficient 1
1
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Target Localization Algorithm
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Target Localization Algorithm Start from the position of the model in the current frame
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Target Localization Algorithm Start from the Search in the position of the model’s model in the neighborhood in current frame next frame
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Target Localization Algorithm Start from the Search in the Find best position of the model’s candidate by model in the neighborhood in maximizing a current frame next frame similarity func.
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Approximating the Similarity Function
Model location: Candidate location:
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Approximating the Similarity Function
Model location: Candidate location:
Linear approx.
(around y0)
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Approximating the Similarity Function
Model location: Candidate location:
Linear approx.
(around y0)
Independent of y
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Approximating the Similarity Function
Model location: Candidate location:
Linear approx.
(around y0)
Independent of y
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Approximating the Similarity Function
Model location: Candidate location:
Linear approx.
(around y0)
Independent of y
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Approximating the Similarity Function
Model location: Candidate location:
Linear approx.
(around y0)
Independent of y
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Approximating the Similarity Function
Model location: Candidate location:
Linear approx.
(around y0)
Independent of y
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Approximating the Similarity Function
Model location: Candidate location:
Linear approx.
(around y0)
Independent of y
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Approximating the Similarity Function
Model location: Candidate location:
Linear approx.
(around y0)
Independent of y Density estimate! (as a function of y)
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Maximizing the Similarity Function
The mode of = sought maximum
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Maximizing the Similarity Function
The mode of = sought maximum
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Maximizing the Similarity Function
The mode of = sought maximum
Important Assumption:
The target representation provides sufficient discrimination
One mode in the searched neighborhood
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Applying Mean-Shift
The mode of = sought maximum
Original Find mode of using Mean-Shift:
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Applying Mean-Shift
The mode of = sought maximum
Original Find mode of using Mean-Shift:
Extended Find mode of using Mean-Shift:
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Applying Mean-Shift
The mode of = sought maximum
Original Find mode of using Mean-Shift:
Extended Find mode of using Mean-Shift:
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking About Kernels and Profiles
Extended Find mode of using Mean-Shift:
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking About Kernels and Profiles A special class of radially symmetric kernels: The profile of kernel K
Extended Find mode of using Mean-Shift:
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking About Kernels and Profiles A special class of radially symmetric kernels: The profile of kernel K
Extended Find mode of using Mean-Shift:
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Choosing the Kernel A special class of radially symmetric kernels:
Epanechnikov kernel:
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Choosing the Kernel A special class of radially symmetric kernels:
Epanechnikov kernel: Uniform kernel:
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Choosing the Kernel A special class of radially symmetric kernels:
Epanechnikov kernel: Uniform kernel:
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Adaptive Scale Problem : The scale of The scale (h) the target of the kernel changes in must be time adapted
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Adaptive Scale Problem : The scale of The scale (h) the target of the kernel changes in must be time adapted
Solution : Run localization 3 times with different h
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Adaptive Scale Problem : The scale of The scale (h) the target of the kernel changes in must be time adapted
Solution : Run localization 3 times with different h
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Adaptive Scale Problem : The scale of The scale (h) the target of the kernel changes in must be time adapted
Solution : Run localization 3 times with different h
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Adaptive Scale Problem : The scale of The scale (h) the target of the kernel changes in must be time adapted
Solution : Run localization 3 times with different h
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Adaptive Scale Problem : The scale of The scale (h) the target of the kernel changes in must be time adapted
Solution : Run Choose h that localization 3 achieves times with maximum different h similarity
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Results
Feature space: 16×16×16 quantized RGB Target: manually selected on 1st frame Average mean-shift iterations: 4
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Results
Partial occlusion Distraction Motion blur
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Results
Partial occlusion Distraction Motion blur
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking Results
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking The Scale Selection Problem Kernel too big
Kernel too small
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking The Scale Selection Problem Kernel too big
h mustn’t get Poor too big or too localization Kernel too small small!
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking The Scale Selection Problem Kernel too big
h mustn’t get Poor too big or too localization Kernel too small small!
In uniformly Problem colored regions, : similarity is invariant to h
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking The Scale Selection Problem Kernel too big
h mustn’t get Poor too big or too localization Kernel too small small!
In uniformly Smaller h may Problem colored regions, achieve better similarity is : similarity invariant to h
Mean Shift Tracking Ilic Slobodan Mean-Shift Object Tracking The Scale Selection Problem Kernel too big
h mustn’t get Poor too big or too localization Kernel too small small!
In uniformly Smaller h may Nothing keeps h Problem colored regions, achieve better from shrinking similarity is : similarity too small! invariant to h
Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Motivation
Spatial Simultaneou localization s localization for several in space and scales scale Previous method This method Mean-shift Blob Tracking through Scale Space, by R. Collins Mean Shift Tracking Ilic Slobodan Lindeberg’s Theory Selecting the best scale for describing image features
y
x
σ
Scale-space Differential operator 50 strongest representation applied responses Mean Shift Tracking Ilic Slobodan Lindeberg’s Theory Selecting the best scale for describing image features
Mean Shift Tracking Ilic Slobodan Lindeberg’s Theory Selecting the best scale for describing image features
Scale-space representation Mean Shift Tracking Ilic Slobodan Lindeberg’s Theory Selecting the best scale for describing image features
Scale-space Laplacian of representation Gaussian (LOG) Mean Shift Tracking Ilic Slobodan Lindeberg’s Theory Selecting the best scale for describing image features 2D LOG filter with scale σ
Laplacian of Gaussian (LOG) Mean Shift Tracking Ilic Slobodan Lindeberg’s Theory Selecting the best scale for describing image features 2D LOG filter with scale σ
3D scale-space representation
y
x
σ
Mean Shift Tracking Ilic Slobodan Lindeberg’s Theory Selecting the best scale for describing image features 2D LOG filter with scale σ
3D scale-space representation
y
x
σ
Best features are at (x,σ) that maximize L Mean Shift Tracking Ilic Slobodan Lindeberg’s Theory Multi-Scale Feature Selection Process
Original Image
Mean Shift Tracking Ilic Slobodan Lindeberg’s Theory Multi-Scale Feature Selection Process
3D scale-space function Original Image
Convolve
Mean Shift Tracking Ilic Slobodan Lindeberg’s Theory Multi-Scale Feature Selection Process
250 strongest responses 3D scale-space (Large circle = large scale) function Original Image
Convolve Maximize
Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Approximating LOG using DOG
2D LOG filter 2D DOG filter with scale σ with scale σ
Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Approximating LOG using DOG
2D Gaussian 2D Gaussian 2D LOG filter 2D DOG filter with μ=0 and with μ=0 and with scale σ with scale σ scale σ scale 1.6σ
Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Approximating LOG using DOG
2D Gaussian 2D Gaussian 2D LOG filter 2D DOG filter with μ=0 and with μ=0 and with scale σ with scale σ scale σ scale 1.6σ
Why DOG? • Gaussian pyramids are created faster • Gaussian can be used as a mean-shift kernel
Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Approximating LOG using DOG
2D Gaussian 2D Gaussian 2D LOG filter 2D DOG filter with μ=0 and with μ=0 and with scale σ with scale σ scale σ scale 1.6σ
Why DOG? • Gaussian pyramids are created faster • Gaussian can be used as a mean-shift kernel DOG filters at 3D spatial multiple scales kernel
Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Approximating LOG using DOG
2D Gaussian 2D Gaussian 2D LOG filter 2D DOG filter with μ=0 and with μ=0 and with scale σ with scale σ scale σ scale 1.6σ
Why DOG? • Gaussian pyramids are created faster • Gaussian can be used as a mean-shift kernel DOG filters at 3D spatial multiple scales kernel
Scale-space filter bank
Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Using Lindeberg’s Theory
Recall:
Model: at
Candidate:
Color bin:
The likelihood Pixel weight: that each candidate pixel belongs to the target Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Using Lindeberg’s Theory
Weight image
Recall:
Model: at
Candidate:
Color bin:
The likelihood Pixel weight: that each candidate pixel belongs to the target Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Using Lindeberg’s Theory
3D spatial kernel Weight image (DOG) Recall: Centered at current location Model: at and scale
Candidate:
Color bin:
The likelihood Pixel weight: that each candidate pixel belongs to the target Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Using Lindeberg’s Theory
1D scale kernel (Epanechnikov) 3D spatial kernel Weight image (DOG) Recall: Centered at current location Model: at and scale
Candidate:
Color bin:
The likelihood Pixel weight: that each candidate pixel belongs to the target Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Using Lindeberg’s Theory
1D scale kernel 3D scale-space (Epanechnikov) representation 3D spatial kernel Weight image (DOG) Recall: Modes are blobs in Centered at current location the scale-space at Model: and scale neighborhood Candidate:
Color bin:
The likelihood Pixel weight: that each candidate pixel belongs to the target Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Using Lindeberg’s Theory
1D scale kernel 3D scale-space (Epanechnikov) representation 3D spatial kernel Weight image (DOG) Recall: Modes are blobs in Centered at current location the scale-space at Model: and scale neighborhood Candidate:
Color bin: Need a mean-shift
The likelihood procedure that finds Pixel weight: that each local modes in candidate pixel belongs to the E(x,σ) target Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Example
Image of 3 blobs
A slice through the 3D scale- space representation
Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Applying Mean-Shift
Use interleaved spatial/scale mean-shift Scale Spatial stage: stage: Fix σ and Fix x and look for the look for the best x best σ
Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Applying Mean-Shift
Use interleaved spatial/scale mean-shift Scale Spatial stage: stage: Fix σ and Fix x and look for the look for the best x best σ
Iterate stages until convergence of x and σ Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Applying Mean-Shift
Use interleaved spatial/scale mean-shift Scale Spatial stage: stage: Fix σ and Fix x and look for the look for the best x best σ
σ
σopt
Iterate stages σ until 0 convergence of x x0 xopt x and σ Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Applying Mean-Shift
Use interleaved spatial/scale mean-shift Scale Spatial stage: stage: Fix σ and Fix x and look for the look for the best x best σ
σ
σopt
Iterate stages σ until 0 convergence of x x0 xopt x and σ Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Applying Mean-Shift
Use interleaved spatial/scale mean-shift Scale Spatial stage: stage: Fix σ and Fix x and look for the look for the best x best σ
σ
σopt
Iterate stages σ until 0 convergence of x x0 xopt x and σ Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Results Fixed-scale
Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Results Fixed-scale
± 10% scale adaptation
Mean Shift Tracking Ilic Slobodan Tracking Through Scale Space Results Fixed-scale
± 10% scale adaptation
Tracking through scale space
Mean Shift Tracking Ilic Slobodan