Segmentation and indexation of complex objects in comic book images A dissertation submitted by Christophe Rigaud at Universitat Autonoma de Barcelona to full the degree of Doctor of Philosophy. Bellaterra, December 11, 2014 UNIVERSITÉ DE LA ROCHELLE ÉCOLE DOCTORALE S2IM Laboratoire Informatique, Image et Interaction (L3i) THÈSE présentée par : Christophe RIGAUD soutenue le : 11 décembre 2014 pour obtenir le grade de : Docteur de l’université de La Rochelle Discipline : informatique et applications Segmentation et indexation d’objets complexes dans les images de bandes déssinées JURY : Bart LAMIROY Professeur associé, Université de Lorraine (France), Examina- teur, Président du jury Simone MARINAI Professeur associé, Université de Forence (Italie), Rapporteur Apostolos ANTONACOPOULOS Professeur associé, Université de Salford (Grande-Bretagne), Rapporteur Koichi KISE Professeur, Université d’Osaka (Japon), Examinateur Jean-Philippe DOMENGER Professeur, Université de Bordeaux (France), Examinateur Jean-Christophe BURIE Professeur, Université de La Rochelle (France), Directeur de thèse Dimosthenis KARATZAS Professeur associé, Université Autonome de Barcelone, Enca- drant de thèse Jean-Marc OGIER Professeur, Université de La Rochelle, Encadrant de thèse Director Prof. Dr. Jean-Christophe Burie Laboratoire Informatique, Image et Interaction Universit´ede La Rochelle (France) Co-Directors Dr. Dimosthenis Karatzas Centre de Visi´oper Computador Universitat Aut`onomade Barcelona (Spain) Prof. Dr. Jean-Marc Ogier Laboratoire Informatique, Image et Interaction Universit´ede La Rochelle (France) Thesis Prof. Dr. Bart Lamiroy committee Laboratoire Lorrain de Recherche en Informatique et ses Applications Universit´ede Lorraine (France) Prof. Dr. Koichi Kise Department of Computer Science and Intelligent Systems Osaka Prefecture University (Japan) Prof. Dr. Jean-Philippe Domenger Laboratoire Bordelais de Recherche en Informatique Universit´ede Bordeaux (France) European Dr. Simone Marinai evaluators Dipartimento di Ingegneria dell'Informazione Universit`adegli Studi di Firenze (Italy) Dr. Apostolos Antonacopoulos The School of Computing, Science & Engineering University of Salford (United Kingdom) iv This document was typeset by the author using LATEX 2". The research described in this book was carried out at the Laboratoire Informatique, Image et Interaction, Universtit´ede La Rochelle and at the Centre de Visi´oper Computador, Universitat Aut`onomade Barcelona. Copyright © 2014 by Christophe Rigaud. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the author. ISBN: XXX Printed by Ediciones Gr´aficasRey, S.L. To my parents... Nothing in life is to be feared, it is only to be understood. Maria Sk lodowska-Curie (1867 - 1934) Live as if you were to die tomorrow. Learn as if you were to live forever Mahatma Gandhi (1869 - 1948) Acknowledgement First of all I would like to thanks Dr. Simone Marinai and Dr. Apostolos Antona- copoulos to have reviewed this European thesis and for sharing interesting comments and discussions. I am also grateful to Pr. Dr. Koichi Kise and Pr. Dr. Jean-Philippe Domenger for accepting to be part of the jury and to Dr. Bart Lamiroy to have presided the jury. Sincere thanks to my supervisors Pr. Dr. Jean-Christophe Burie, Dr. Dimosthe- nis Karatzas and Pr. Dr. Jean-Marc Ogier for their valuable expertise, friendship, dynamism, review of this thesis and join supervision of this work between the Centre de Visioper Computador (CVC) of the Universitat Autonomade Barcelona (UAB), and the Laboratoire Informatique, Image et Interaction (L3i) of the Universitede La Rochelle (ULR). Also, thank you to Remy Mullot for accepting me in his lab (director of the L3i until 2012) and both institutions for providing me all the facilities to carry out this work in a great environment and atmosphere. I would like to thanks all the people from the Intelligent Media Processing Group of Osaka Prefecture University (Japan) for giving me opportunity of doing research stays with a Manga's team: Pr. Dr. Koichi Kise, Dr. Motoi Iwata, Dr. Weihan Sun, the secretaries and all the master's students including Atsushi Ito and Masaki Tukada which have beneted of a research stay in France. I have learnt so many things about research, work, life style and food (thank you Hideto!). Thank you to the CVC members for having welcomed me on several occasions (European doctorate and then joint supervision), especially people from the docu- ment analysis group: Josep Llados,Dimosthenis Karatzas, MarcalRusi~nol,Oriol Ramos, Volkmar Frinken, Alicia Fornes, Jon Almazan, Lluis Gomez,LlusPere De las Heras, Anjan Dutta, Prass, David Fernandez,Nuria Cirera, Francisco Cruz and Claire Perez-Mangadofrom the secretary. Thank you to the eBDtheque project's team for all the fruitful meetings we had together, Karell Bertet, Jean-Christophe Burie, Arnaud Revel, Alain Bouju, George Louis, Jean-Marc Ogier, Clement Guerin,Antoine Mercier (annotation tool and dataset's website), Norbert Tsopze and Nam Le Thanh. Thanks to the L3i lab members that participated to the ground truth day and also to the trainees that contributed to this project. Sincere thanks to the authors and publishers that have kindly agreed to share part of their works with scientists which allowed us to evalu- ate, share and make reproducible this three year thesis work. In alphabetical order: Pascal Boisgibault, Cyb, Fred, Sergio Garcia, Olivier Jolivet, Lamisseb, GeraldLub- i ii ACKNOWLEDGEMENT bin, Winsor McCay, Midam, Marion Montaigne, Nicolas Roudier, Alain Saint Ogan, Treblaand Lewis Trondheim. Thanks also to their editors: Actes Sud, Ankama, Bac@BD, Clair de Lune, Dargaud, Delcourt, Doc En Stock, Dupuis, Hachette and Studio Cyborga. Finally, a special thank to the CIBDI1, Free Public Domain Golden Age Comics and the Department of Computer Science and Intelligent Systems of Os- aka Prefecture University who kindly provided material from their personal collection. Thank you to the people who have taught me how to teach, Vincent Courboulay, Anthony Bourmaud, Karel Bertet, Armelle Prigent, Arnaud Revel and Renaud Peteri. I would like to thanks the European Doctorate founds of the University of La Rochelle, European Regional Development Fund, the region Poitou-Charentes (France), the General Council of Charente Maritime (France), the town of La Rochelle (France), the Spanish research projects TIN2011-24631, RYC-2009-05031 and the Sakura project of Campus France for the nancial support of this work including the dierent research stays in Spain and Japan. Kind thanks to all the volunteer contributors of operating system, software, programming language and libraries related to the completion of this work (Fedora, NetBeans, Sublime Text, Gimp, Inkscape, Firefox, LATEX, Python, IPython Notebook, Numpy, SciPy, Matplotlib, OpenCV). I would like to thanks people that gave me the taste of academic research of image processing during the last year of engineering school and the research master nal year project. You helped me a lot in the application process of this unique and wonderful thesis project, thank you so much. Thank you to the members of the association of the Ph.D. students of La Rochelle (ADocs) for all the local events we organised together and the help for making and promoting a short lm about my research work for young audience. Grateful thanks to all the Ph.D. candidates, young doctors, engineers and admin- istrative aliated to the L3i with whom I spend most of the time (Romain, Clement, Guillaume, Cyril, Sebastien,Omar, Van, Giap, Sophea, Dounia, Maroua, Rouaa, Marcela, Bich, Imen, Hind, Phuong, Jane, Muzzamil, Benjamin, Joseph, Bruno, Olivier, Kathy, Caro, Sarah, Dom, Stephane& Arnold and those I have forgotten sorry...). Special thanks to the 121 bis oce team (Nat, Mika, To,Elo, Nam, Bao, the Roundabout, the Cow and the Space invaders ^^) for these three years of gentleness, happiness and teasing that we shared together in this unique open space where science and creativity grow on the walls, roof and in the air. And nally, I would like to innitely thanks my parents, brother and all my family for their endless love and support at a distance of ve hundreds and twenty six kilometres as the crow ies. Special thanks to my love for having followed me throughout this adventure with patience, dynamism, joy of living and also many contributions to this work (ground truthing, thesis review, etc). 1CiteInternationale de la Bande Dessineeet de l'Image Abstract Born in the 19th century, comics is a visual medium used to express ideas via images, often combined with text or visual information. It is considered as a sequential art, spread worldwide initially using newspapers, books and magazines. Nowadays, the development of the new technologies and the World Wide Web is giving birth to a new form of paperless comics that takes advantage of the virtual world freedom. However, traditional comics still represent an important cultural heritage in many countries. They have not yet received the same level of attention as music, cinema or literature about their adaptation to the digital format. Using information technologies with classic comics would facilitate the exploration of digital libraries, faster theirs translations, allow augmented reading, speech playback for the visually impaired etc. Heritage museums such