FINDING SALIENT OBJECTS IN AN IMAGE Anthony Hang Fai Lau Depart ment of Elecerical Engineering SIcGill Cniversity .\ Thesis siibmitted to the Facttlty of Graduate Studies and Research in partial fulfilment of the requirements for the degree of b taster of Engineering Bibliothèque nationale du Canada Aquisitioe and Acquisitions et Bibliographe Services services bibliographiques The author has granted a non- L'auteur a accordé une Licence non exclusive licence allowing the exclusive permettant a la National Lilbrary of Canada to Bibliothèque nationale du Canada de reproduce, loan, distn'bute or sel reproduire, prêter, distriibuer ou copies of tbis thesis in microform, vendre des copies de cette thèse sous paper or electronic formats. la forme de microfichell5.lm, de reproduction sur papier ou sur format électronique. The author retains ownership of the L'auteur conserve la propriété du copyright in this thesis. Neither the droit d'auteur cpi protège cette thèse. thesis nor substantial extracts fiom it Ni la thése ni des extraits substantiels may be printed or othemise de celle-ci ne doivent être imprimés reproduced without the author's ou autrement reproduits sans son permission. autorisation. Abstract Slany cornpliter vision applications. such as object recognition. active vision. anci content briçed image retrieval (CBIR) could be made both more efficient and effective if thtl ohjwts of interest cotild be segrnenred from the background. This thesis dis- ciisses the developnient and implementation of a complete ~insiipervisedobject-biised atttwion systern for locating salient objects in an image. Tlie rriajor conipotierits of this system are the segmentation and the attention procrss. Consiti~rabl~research haç beeri done in these two areas. but unforturiately. diere is still not a single rnethod that can be appiied reliably under al1 situations. \Ci. have analysecl the attention model proposed by Osberger and have founci chac thir rnettiod fails to idcntify some important regions that are saiierit to humaris. Noditications to this mode1 are proposed to correct some of these problenis. For the segnientation process. one important aspect is the measurernent of the qiiality of a partictilar segnientation. since the attention process depends solely on the segmenta- tion output. in particular. three different cluster tdidity rneasures are consiclered: a simple thresholcl-based indes. a non-parameter indes. and the modified Hubert in- des. From the esperirnenta1 resuIts. the simple threshold-based index is shown to outperforrn the other indices on most test images. We believe that the success of the rhreshold-basetl index is largely reIated to the incorporation of hurnan preference ici the selection of the threshold parameter. Résumé Dc ntirnbreuses applications en vision artificielle telles que la vision active et I'indesage (l'images basé sur le contenu pourraient etre rendues plus efficaces si les objets d'intérêt poutxient segmentés du fond de l'image. Cette thèse discute du développement et de l'implémentation d'un systérne d'attention non-supervisé basé sur des objets pour localiser rlcs objets saillants clans une image. Les cornposantes niajeiires de re systènie sont la segmentation et le mécanimsme d'itttclntion. Bien que que ces rleiis sujets aient été l'objet de nombreuses recherches. il n'tlsist~toujours à ce jour pas de méthode fiable qui puisse 6tre appliqtiée dans coiitcs les situations. Xoiis avons analysé le modéle d'attention propoé par Osberger ttr. nous avons trouve qu'elle ne réussi pas a identifier quelques unes ries régions sail- lantes évicicwes pour des humains. Des modifications à ce modèle sont proposées pour c.0rrigr.r iwtairis de ces problémes. Cu des aspects importants pour la seg- iiieiitatioti est Iri niesure de la quiilité d'une m'ethode en particulier puisque le pro- resstb rt*i;ittentionrepose uniquement sur Ir résultat tlc la segnientation. Plus partir- iiliérement. trois différentes rnéthocles de niesure rle vdirlitl; sont ronsirlér6es;: tin indes tlPterniin6 par un setiillage simple. un index non-paramétrique et une version niocC ifiée rie Iïncles d'Hubert. D'après les résultats expérimenta~~x.1' indes déterminé par un seuillage siniple surpasse les autres méthodes pour la plupart des images testces. Nous croyons que le succès de l'index déterminé par un seuillage simple est largement lié a l'incorporation de préfiences humaines dans la sélection du seuiI utilisé. Acknowledgements First of dl. I would like to thank niy supervisor. Prof. '1I.D. Levine. for tiis enthii- siastic guidance and support. He is always arailable when needed and is willing tu tliscuss with his stutirnts any clifficulties encountered during the research. I must also thank GiIbert Soucy for transiating the abstract to French. al1 the people at CI11 for providing a goorl working atmosphere. and my farnily for their unfailing support arid encoiiragenient throiighoiit the period of this wrk. TABLE OF CONTENTS .. Abstract ....................................... 11 ... LIST OF FIGURES ................................ 1-111 LIST OF T.4BLES ................................. sii CH-4PTER 1. Introrluction ........................... 1 1. The Xeed for Object-based Attention ................... 1 2 . 5Iotivation .................................. -.) 3 . .4n Overview uf the Approach ....................... --I 4 . Organisation of the Thesis ......................... 4 5 . Contributions ................................ 4 CH-APTER 2 . Literature Review ........................ 6 . 1. Perceptual Grouping ............................ I 1.1. Signal Level ............................... 10 1.2. Primitive Level ............................. 12 1.3. Stn~cturalLever ............................. 12 1.4. Conslusions ............................... 23 2 . \i-isual Attention System in Humans .................... 13 2.1. Strtlcture of the Human Visual S_vstem ................ L4 -1.2. Psychophysical Aspects of the Human Visual Attention System . i5 -1.3. CoucIrrsions ............................... '210 3 . Visual -ittention Systems in Slachines ................... 20 3.1. Conchsions ............................... 3-- 3 CH-APTER 3 . Perceptual Sdiency Sleasure ................... 23 1. Perceptrial Saliency Factors ......................... 23 1.1. Osberger and Maeder 's mode! ..................... 2-4 12 Discussion ................................ 26 1.3. Xew and Slodified Importance Factors ................ 29 2 . Methods for Combining the Importance Factors ............. 33 2.1. Osberger and Uaeder's Method .................... 33 2.1. Itti and Ibch's SIethod ......................... 33 '2.3. Discussion ................................ 34 CH-WTER 4 . Feaciire Selection ......................... :33 1. Colour .................................... 36 1.1. Coloiir Spaces .............................. 35 1.2. Conclusions ............................... -13 2. Testur~.................................... 43 2.1. Relatecl CVork on Texture ........................ -14 2.2. Rrlared \.Vork ori I'risupervised Segmentation of Satura1 1ni:iges . 4G 2.3. TestureRepresentation ......................... 47 4 Gabor Filter Biink ........................... 4s 2.5. Generation of Texture Feature Set ................... 51 - - 3 . Feattire Integration ............................. ai CH.4PTER .? . Iniag~Scgnientation ....................... -59 1. Revictv of Image Segmentation Techniques ................ 59 1.1. Clustering-bwed JIethods ....................... 60 1 . Edge-bnsetl Sl~tho& .......................... 61 1..3 . Rtyjori-basetf SI~thods......................... 62 1.4. Hybriti SIethods ............................. 63 1.3. Conclusions ............................... 63 2 . ?;ou-parametric Density Estimation for Image Clustering ........ 63 '1.1. Clustering .\ lgorithrn .......................... 64 2 Cliister Càlidity indices and Stopping Criteria ............. 65 '1.3. Post-processing ............................. 70 CH-iPTER 6. Etduation and Test Results ................... Ti 1. Determining Parameter Values ....................... 71 1.1. Weights for Colour . Texture . and Position .............. 71 1.2. Parameters Csed in Image Clustering ................. 73-- 2 . Cluster Measures .............................. t t 2.1. -4ssumptions Lsed in Each SIethod .................. 78 2.2. Test Images and Implementation Issues ................ 78 2.3. Test Results and Discussion ...................... 79 3 . Saliençy Factors ............................... S2 3.1. Determining the CCéights of Different Sdiency Factors ........ 82 3.2. Discussion ................................ S1 4 . -4pplications ................................. 3-1 4.1. Face finding ............................... 8-4 4 .2. [mage compression . machine vision . and CBIR ............ 86 CHJLPTER 7 . Conritisions ............................ Si 1. Dirertion of Future Work .......................... 88 .4 PPESDIS .-\ . Thc Grnphical Cser Interface (GLI) .............. 30 APPESDIS B . The Image Database ...................... 92 APPESDIS Ç . The Test Set and Results .................... 96 REFERESCES ................................... 99 LIST OF FIGLXS LIST OF FIGURES Systern Block Diagrarn. The niethod consists of three computational processes shown in the left-hand column. The data transferred twtween each process is inciicated in the shaded strips. Examples cif the input. interniediate data. and final output are shown in the right-harul colurnri ......................... 3 -4 simple