Graph representation and mining applied in comic images retrieval Thanh Nam Le To cite this version: Thanh Nam Le. Graph representation and mining applied in comic images retrieval. Computer Vision and Pattern Recognition [cs.CV]. Université de La Rochelle, 2019. English. NNT : 2019LAROS008. tel-02475609 HAL Id: tel-02475609 https://tel.archives-ouvertes.fr/tel-02475609 Submitted on 12 Feb 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. LA ROCHELLE UNIVERSITÉ ÉCOLE DOCTORALE EUCLIDE Laboratoire Informatique, Image et Interaction (L3i) THÈSE présentée par : Thanh Nam LE soutenue le : 29 mars 2019 pour obtenir le grade de : Docteur de l’Université de La Rochelle Discipline : Informatique et Applications Graph Representation and Mining Applied in Comic Images Retrieval [Représentation par graphes et Fouille de graphes : Application à la recherche d’images de bandes dessinées par le contenu] COMPOSITION DU JURY : Sébastien ADAM Professeur, Université de Rouen Normandie (France), Examinateur Jean-Christophe BURIE Professeur, Université de La Rochelle (France), Encadrant de thèse Josep LLADÓS Professeur, Université Autonome de Barcelone (Espagne), Rapporteur Muhammad Muzzamil LUQMAN Ph.D, Ingénieur de Recherche, Université de La Rochelle (France), En- cadrant de thèse Jean-Marc OGIER Professeur, Université de La Rochelle (France), Directeur de thèse Nicole VINCENT Professeur, Université Paris Descartes (France), Rapporteur Director Prof. Jean-Marc Ogier Laboratoire Informatique, Image et Interaction (L3i) Universit´ede La Rochelle (France) Co-Directors Prof. Jean-Christophe Burie Laboratoire Informatique, Image et Interaction (L3i) Universit´ede La Rochelle (France) Dr. Muhammad Muzzamil Luqman Laboratoire Informatique, Image et Interaction (L3i) Universit´ede La Rochelle (France) Reporters Prof. Nicole Vincent Laboratoire d'Informatique Paris Descartes (LIPADE) Universit´eParis-Descarte (France) Prof. Josep Llad´os Centre de Visi´oper Computador (CVC) Universitat Aut`onomade Barcelona (Spain) Examiner Prof. S´ebastienAdam Laboratoire d'Informatique, du Traitement de l'Information et des Syst`emes(LITIS) Universit´ede Rouen Normandie (France) This document was typeset by the author using LATEX 2". The research described in this book was carried out at the Laboratoire Informatique, Image et Interaction (L3i) from the Universtit´ede La Rochelle, and at the Centre de Visi´oper Computador (CVC) from the Universitat Aut`onoma de Barcelona. Copyright © 2019 by Thanh-Nam LE. All rights reserved. No part of this publication may be repro- duced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the author. Acknowledgements It would not have been possible to achieve this thesis and write this manuscript without the help and support of the kind people that helped me through such a long and arduous journey. I would like to first express my gratitude to my three supervisors, for their com- pany over the past years of Ph.D program, as I worked my way from writing the very first initial proposal to a complete study. Thank you Professor Jean-Marc Ogier, who have given me the chance to partici- pate in this interesting research project related to graph and comics. Such combina- tion was really new to me at the beginning, when we first met in Hanoi. Later on, you were much occupied with the new demanding role of presidency, yet you always found time to help with improving the quality of my work, be it a paper, or a thesis manuscript, or a general advice. Thank you Professor Jean-Christophe Burie, who helped me throughout my re- search work, giving me the valuable suggestions, and guided me in completing this thesis, especially in the final phases. Thank you for your great help in extending the funding for my research. I appreciate your meticulousness in revising all the writing, and how you helped me with all the administrative paperwork. Thank you Muzzamil, for all detailed guidance and encouragements, for the count- less hours of exchange we had together, which I'll never forget. You were not only a tutor, but also a good friend indeed. I thank you for your help, not only concerning research and scientific aspects, but also on other matters of life during my Ph.D. Yes, it is not exaggerated: your continuous encouragement and patience are much appreciated, as they pushed me to go through the crucial and rough stages of my thesis. I'm also thankful to all the members of my PhD jury. In particular, I thank the reviewers, Professor Nicole Vincent and Professor Josep Llados for reviewing my thesis manuscript and providing their valuable remarks, and I thank Professor Sebastien Adam for spending his valuable time as my thesis examiner. During my time working on this thesis, I had the great fortune of being able to travel to other institutions and meet excellent research teams there. It helped prevented isolated lab time of doing research in an echo chamber. Going to other places gave me the chance to get to know new minds, new ideas, new people and new culture. Thank you again Josep for receiving me as a visiting PhD student and for the kind guidance during my three months at CVC, UAB Barcelona. You were always willing to help and give best suggestions. Working with you was a great pleasure. I i ii ACKNOWLEDGEMENTS would also like to thank Dr. Motoi Iwata and Professor Koichi Kise for receiving me twice at Osaka Prefecture University and for the great experience I had in Japan. Thank you to all my friends, colleagues, and all the staffs at L3i for your support and for the years that I passed among you. I prefer to not mention names, since the list would be very long, and I don't want to miss anyone. You all were very kind to me, and I always felt so comfortable working in the lab, thanks to the hospitality the people here, despite a minor language barrier. My parents and my wife deserve my deepest appreciation, for their love, encour- agement, understanding, patience and support. They help me successfully complete this important phase of my life. And thank you my little son, for always being my unlimited source of joy and hope in life. This work is dedicated to you, Phillipe Gia-Minh. Abstract Graphs are powerful mathematical modeling tools used in various fields of Computer Science, and in Pattern Recognition in particular. In information retrieval tasks from image databases where content representation is based on graphs, the evaluation of similarity is based both on the appearance of spatial entities and on their mutual relationships. In this thesis we present a novel scheme of Attributed Relational Ad- jacency Graphs representation and mining, which has been applied in content-based retrieval of comic images. We first address the problem of graph representation of image and its applications in Pattern Recognition, with a focus on content-based im- age retrieval (CBIR) applications. The images used in this thesis are comics images, which have their inherent difficulties in applying content-based retrieval, such as their abstractness, partial occlusion, scale change and shape deformation due to viewpoint changes. We propose a graph representation that yields stable graphs and allow to retain high-level and structural information of objects of interest in comic images. Next, we extend the indexing and matching problem to graph structures representing the comic image, and apply it to the problem of retrieval. The graphs in the graph database representing the whole comic volume are then mined for frequent patterns (or frequent substructures). This step is to overcome the non-repeatability problem caused by the unavoidable errors introduced into the graph structure during the graph construction stage, which ultimately create a semantic gap between the graph and the content of the comic image. Finally, we demonstrate the effectiveness of the system with a database of annotated comic images. Experiments of performance measures is addressed to evaluate the performance of this CBIR system. Keywords: graph matching, graph based modeling, graph mining, frequent pat- tern mining, pattern recognition, deformable object recognition, content-based image retrieval, comic image retrieval iii R´esum´e Les graphes sont de puissants outils de mod´elisation math´ematiqueutilis´esdans divers domaines de l'informatique, et en particulier en reconnaissance des formes. Dans une t^ache de recherche d'informations dans de grandes bases de donn´ees images o`ula repr´esentation du contenu est bas´eesur des graphes, l'´evaluation de la similarit´eest bas´ee`ala fois sur l'apparence des entit´esspatiales et leurs relations mutuelles. Dans cette th`ese,nous pr´esentons un nouveau sch´emade repr´esentation et d'exploration de graphes d'adjacence relationnels attribu´es.Ce sch´emaa ´et´eappliqu´epour rechercher des personnages dans les images de bandes dessin´ees. Nous abordons d'abord le probl`emede la repr´esentation sous forme d'un graphe de l'image et de ses applications en reconnaissance des formes, en mettant l'accent sur les applications de recherche d'images bas´eessur le contenu (CBIR). Les images utilis´eesdans cette th`ese sont des images de bandes dessin´ees,qui poss`edent des sp´ecificit´esqui sont des freins pour les m´ethodes de recherche d'information par le contenu utilis´eesdans la litt´erature. Le contenu, des bandes dessin´ees,tel que les objets et les personnages, est com- plexe.