Local Feature Learning Gustavo Carneiro Tutorial ICIP 2013 Local Feature Learning and Non-rigid Matching Why Local Features? Model Test 1 Test 2 GloBal Local MORE ROBUST MORE DISTINCTIVE 2-dim feature space G 2-dim feature space L G. Carneiro - University of Adelaide 2 Local Feature • Limited spaal support + RoBust to changes and parPal occlusion - Discriminang power (compensated By # descriptors) • Applicaons – Visual classificaon – Image matching G. Carneiro - University of Adelaide 3 Representaon • An image is represented By a set of N descriptors (or parts) G. Carneiro - University of Adelaide 4 Representaon • An image is represented By a set of N descriptors (or parts) f1=[ ,x1,θ1,σ1] scale appearance geometry dominant orientation position 5 Representaon • An image is represented By a set of N descriptors (or parts) f2 = [ a2 , g2 ] G. Carneiro - University of Adelaide 6 Representaon • An image is represented By a set of N descriptors (or parts) F={ f1 , f2 , f3 , ... , fN } G. Carneiro - University of Adelaide 7 Image Matching [Schaffalitzky & Zisserman, ECCV02] G. Carneiro - University of Adelaide 8 Visual Classificaon • Instance-Based recogniPon [Lowe,IJCV04] • Class recogniPon G. Carneiro - University of Adelaide 9 Objecve Funcons • Image matching – Maximize precision and recall of feature matching • Instance-Based recogniPon and class recogniPon – Minimize classificaon error G. Carneiro - University of Adelaide 10 Hand-designed Local Features 90’s to 2000’s • ‘Where’ step – Repeatable Harris “corners” [Harris88] DoG [Lowe99,04] Sum square diffs Diff of Gaussians RoBust to rotaon RoBust to scale • ‘What’ step – RoBust – Disncve SIFT [Lowe99,04] 128 dimensions RoBust to rigid transforms and Brightness 11 Quesons • Are these hand-designed features opPmal? • Image matching and instance-Based recogniPon (Does it maximize matching precision and recall?) – If disPncPve & roBust then max. precision and recall • Class recogniPon (Does it minimize classificaon error?) – If roBust & disPncPve then min. classificaon error G. Carneiro - University of Adelaide 12 PosiPve Evidence viewpoint scale [Mikolajczyk and Schmid, PAMI’05] • SIFT-like features showed superior performance – Dominates matching and classificaon applicaons G. Carneiro - University of Adelaide 13 Matching ProBlem • Building Rome in a Day [Agarwal et al. ICCV09] – Reconstruct 3D scenes from large collecPon of images • Matching Based on SIFT features • Image similarity also Based on SIFT features G. Carneiro - University of Adelaide 14 Class RecogniPon • Bag of Features [Sivic and Zisserman, ICCV03] SVM Classifier • Also use SIFT features G. Carneiro - University of Adelaide 15 However • Good features to track [Shi and Tomasi,CVPR94] – Feature selecPon Based on model similarity • Detectability, Uniqueness, and Reliability [OBah and Ikeuchi,PAMI97] 16 But these are not SIFT… • True, But D. Lowe noPced something similar about the discriminang power of SIFT • Not all descriptors have the same discriminang power. • Can a similar thing Be said about the roBustness properPes of the feature? G. Carneiro - University of Adelaide 17 Explicit Characterizaon of RoBustness and Discriminang Power [Carneiro and Jepson, CVPR05] • RoBustness: Pon(sf(fl,fo);fl) ˜ Pβ(sf(fl,fo);aon,bon) • Disncveness : Poff(sf(fl,fo);fl) ˜ Pβ(sf(fl,fo);aoff,boff) • Detectability: Pdet(xl) P P Feature vector #260 on off Pdet(xl)=87% Feature vector #540 Pdet(xl)=67% 18 Phase correlaon Train Classifier to Select and Characterize “Good” Features • P(OBj|Match,Img) =(1/Z)P(Match|OBj,I)P(OBj|Img) 19 SelecPng and Characterizing Good Features… • Does it lead to more effecPve matching? • Does it lead to more effecPve classificaon? • Why can’t we learn the features By maximizing the actual oBjecPve funcPon? – Instead of designing and charactering individual features G. Carneiro - University of Adelaide 20 In the Beginning… • Perceptron [RosenBla 57] fi (x) wi y = sign(! fi (x)wi + b) • Features to use? – Again, hand-designed… 21 Learning Input Features? • Perceptron can only deal with linear proBlems • MulP-layer perceptron (non-linear acPvaon funcPons)? – Can deal with more complex proBlems! – Can we finally learn the input features from the image? (1) wi (2) wi ` … ! (!viwi + b) 22 Back-propagaon [Rumelhart et al. 86] • Algorithm that allowed training of mulP-layer perceptron could not handle more than 1-2 hidden layers – Long Pme to converge (if it converges at all) • Back to hand-designing/selecPon/ characterizaon of features… G. Carneiro - University of Adelaide 23 TradiPonal Methods • Matching ProBlems Hand-designed Features Outlier Matching RejecPon Hand-designed Features • Visual Classificaon ProBlems Hand-designed Supervised Features Classifier G. Carneiro - University of Adelaide 24 Race is on for the “Best” Hand-designed Features • No transformaon – Gray values • Frequency domain – Discrete Fourier transform (DFT), Discrete Cosine Transform (DCT), Discrete Wavelet Transform (DWT) • Good ReconstrucPon and Uncorrelaon – Principal Components Analysis (PCA) • Good ReconstrucPon and Independence – Independent Component Analysis (ICA), sparse coding • Linear class separability – Linear Discriminant Analysis (LDA) • Gradient Orientaon Histogram – SIFT, HOG, GLOH, RIFT, etc. • Image DifferenPals – Local jets • Sampling and representaon variaons – RoBust and disPncPve G. Carneiro - University of Adelaide 25 Also for the “Best” Classifiers • Generave Classifiers – Mixture Model – Naïve Bayes • Discriminave Classifiers – LogisPc Regression – MulP-layer Perceptron – Nearest NeighBor – Support Vector Machine – Boosng – Random Forest G. Carneiro - University of Adelaide 26 Outlier RejecPon • Random Sample Consensus (RANSAC) [Fischler and Bolles, 81; Torr and Murray, IJCV’97] • MSAC (M-esPmator) , MLESAC (max likelihood), IMPSAC (importance sampling), etc. • Consequence of feature matching failures • More about this later on Dr. Chin’s session G. Carneiro - University of Adelaide 27 Feature Learning • Using the tradiPonal architecture, can we learn the features with the opPmizaon funcPon used for the classifier? • Matching – maximize feature roBustness and discriminang power • Classificaon – minimize classificaon error G. Carneiro - University of Adelaide 28 Feature Learning - Matching • AcPve Shape and Appearance Models [Cootes et al. 95,98] posion x Appearance g • ComBine shape and gray level in a single PCA space • Gradient descent to perform matching G. Carneiro - University of Adelaide 29 Feature Learning - Matching • FERNS [M. Özuysal et al. CVPR’07] • Semi-naïve Bayes classifier: G. Carneiro - University of Adelaide 30 D. Capel. 2009 Feature Learning - Matching • Explicitly learn a feature transform that is – RoBust and discriminang • Photo Tourism dataset [Snavely et al. SIGGRAPH’06] used by Winder et al. [Winder and Brown, CVPR’07,09 and PAMI’11] – More than 100,000 patches (3 scenes) – BackprojecPng 3D points to 2D images from scene reconstrucPons – Variaons in scene locaon, Brightness and parPal occlusion G. Carneiro - University of Adelaide 31 Feature Learning - Matching • Discriminave Learning of Local Features [Brown, Hua and Winder’PAMI11] Learning carried out To maximize AUC in ROC graph T-Block: Steerable filters, E-Block: Linear Distance metric learning Gradients, DoG, etc. N-Block: Normalizaon to account for photometric variaons S-Block: 32 (Linear) Distance Metric Learning [Chopra et al.CVPR05,GoldBerger et al.NIPS04, WeinBerger & Saul JMLR09] • Image patches: • Linear transform: • Distance in T space: G. Carneiro - University of Adelaide 33 Results [Winder and Brown’PAMI11] • Errors at 95% (% of error when 95% of TP are found) • In parenthesis: dimensionality G. Carneiro - University of Adelaide 34 Feature Learning - Matching • Learn feature transforms from non-linear distance metric learning [Carneiro,CVPR’10] – Uses original image input (no sampling stage, and focus on emBedding stage) – Use photo tourism dataset – Non-linear distance metric learning [Sugiyama JMLR07] G. Carneiro - University of Adelaide 35 ComBining Feature Spaces • Breiman’s idea about ensemBle classifiers [Breiman 01]: – comBine low-Bias, high-variance (unstable) classifiers to produce low-Bias, low-variance classifiers. • Distance G. Carneiro - University of Adelaide 36 IntuiPon Unkown target proBlem Small dist. Large dist. T G. Carneiro - University of Adelaide 37 Random training proBlem 1 IntuiPon Unkown target proBlem Small dist. Largedist. T G. Carneiro - University of Adelaide 38 Random training proBlem Experiments • Using cross validaon, – 50 training classes for training each feature space – 50 training feature spaces G. Carneiro - University of Adelaide 39 Experiments • Matching database of Mikolajczyk and Schmid 40 Feature Learning - Matching • Convexify Brown et al.’s opPmizaon funcPon [Simonyan et al. ECCV’12] • Use BoosPng to produce non-linear feature transform [Trzcinski et al. NIPS’12] • More to come J, But idea is the same – Given classes of local descriptors, find transformaon that keep features from the same class together, and separate features from disPnct classes G. Carneiro - University of Adelaide 41 Back to the Classificaon ProBlem • Feature selecPon to minimize classificaon error – RoBust Real-Pme OBject DetecPon [Viola and Jones, IJCV’01] • Feature extracPon to minimize Bayes error (BE) – Minimum BE facilitates training [Carneiro and Vasconcelos, CRV’05] • Feature Learning – Supervised ConvoluPonal Networks [Lecun 90s unPl today] G. Carneiro - University of Adelaide 42 Supervised Convolutional Network [Lecun, 90s until today]