Department of Computer Science - University of Fribourg, Switzerland RECOGNITION OF ULTRA LOW RESOLUTION, ANTI-ALIASED TEXT WITH SMALL FONT SIZES THESIS Submitted to the the Faculty of Science, University of Fribourg (Switzerland) To obtain the degree of Doctor Scientarium Informaticarum Farshideh Einsele-Aazami from Switzerland and Iran Thesis N± 1610 Imprimerie St Paul, Fribourg 2008 Accepted by the Faculty of Science of the University of Fribourg following the proposal of: - Prof. Ulrich Ultes-Nitsche, University of Fribourg (Jury President) - Prof. Rolf Ingold, University of Fribourg (Thesis Director) - Prof. Horst Bunke, University of Bern, Switzerland (External Expert) - Prof. Roger Hersch, Federal Institute of Technology Lausanne (EPFL), Switzerland (Ex- ternal Expert) - Dr Jean Hennebert, University of Fribourg (Expert) Fribourg, July 3rd, 2008 Faculty Dean Thesis Director Prof. Titus Jenny Prof. Rolf Ingold Acknowledgements This thesis was conducted in Computer Science Department of the University of Fribourg and was founded by Swiss National Science Foundation (SNF), University of Fribourg and Hasler Stiftung. This thesis is the achievement of 5 years of research in the area of document analysis and image processing. Firts of all, I would like to express my sincere appreciation to Prof. Rolf Ingold, for accepting me in his research group DIVA, for his belief on me, for his valuable guidance and insight, encouragment and reliance throughout this research. I am also greatly indebted to Dr Jean Hennebert for supervising me especially with his valuable knowledge on the hidden Markov models and besides for his patient and guidance in various domains of research and scienti¯c writing. I also would like to thank the Jury President Prof. Ulrich Nitsche for accepting to preside my thesis defense and also the o±cial referees of this thesis: Prof. Hurst Bunke and Prof. Roger Hersch for the honor they gave me and for their excellent expertise and precious advices to improve this thesis and for the wonderful discussions we had during the thesis defense process. I am also thankful for the support and friendly atmosphere in the entire DIVA group that made me work with pleasure for my research. Especially I want to thank my colleague Andreas Humm ¯rstly for his useful programming and LaTex tips and secondly for being a good friend and not let me feel down in the crucial periods of this thesis. As well, I express my gratitude to the secretary team of Department of Informatics (DIUF) and especially to Mrs Eliane Schaller for her optimistic and kind way. Lastly but most importantly, I thank my wonderful husband Alex for his love and patience and moral support he gave me during every phase of this thesis. I am also very lucky to have such lovely kids as Cyrus and Semira that always heard about the progress of Mami's work with great interest. i ii ACKNOWLEDGEMENTS Abstract In this thesis, we address the problem of recognizing text at ultra low resolution (between 72-100 dpi) that is anti-aliased and has font sizes between 7-12 points. Such text is frequently found in web images and is an important carrier of information indexing and retrieval on the web. Anti-aliasing is applied to smooth out edges and diagonals, thus making text more legible on computer screens. However, the anti-aliased characters are mostly connected and can not be segmented with common segmentation methods used in classical document analysis. Moreover, anti-aliasing produces contextual noise at the left and right sides of adjacent characters causing drastic deterioration in recognizing such characters, even if they have been segmented. The classical OCR systems are not capable to recognize such text, as they are basically designed for bilevel text with a resolution of more than 150 dpi, typically obtained using flat scanners. Several research works report on algorithms for detection and recognition of text embedded in web images. However, these works are often directed towards the implementation of front-end image processing methods in order to enhance the image quality and pass it through classical OCRs. The main contribution of the work presented here is about a novel approach to segment and recognize anti-aliased text at ultra low resolution (ULR) with small font sizes. The dedicated recognizer, presented in this thesis, can be directly applied on the detected and extracted anti- aliased text with no needs for using of cost-intensive pre-processing methods to enhance image quality to be recognized with classical OCRs. Several studies on single characters and words are presented in this report. We used syn- thetical character and words for all our studies. The ¯rst study aimed at choosing an adequate feature extraction methodology for anti-aliased low-resolution characters. The features that were selected are based on the computation of central moments which are translation invariant and convey su±cient spatial information from character shapes at ultra low resolution. The second study aimed at understanding the mutual influence of the connected characters in a word. In both studies, the recognition of characters is performed by computing an estimation of the probability density functions of central moment features using Gaussian based estimators. We could reach good Character Recognition Rates (CRR) of 99.93% for isolated characters. These results have shown that central moments are adequately discriminative. The CRRs for contextual characters were less accurate (98.45%) due to the contextual noise between adjacent iii iv ABSTRACT characters. On the other hand, these experiments have shown that we need an intrinsic model to simultaneously segment and recognize connected characters in ULR anti-aliased words with small font sizes. Therefore, hidden Markov models (HMMs) using a sliding window feature extraction have been chosen to build dedicated word recognizers. In addition, the insights of the previous studies have been bene¯cial to the design of the word recognizers. Furthermore, each word is modeled using the character models which are constrained by minimum and mixmum width constraints. The contextual noise is modeled as an inter-character state inserted between character models. Further optimizations of the HMM implementation have also been implemented to further in- crease the Word Recognition Rate (WRR), such as using mixture of Gaussian models instead of mono Gaussian models to estimate the emission probabilities. We have implemented two dedicated recognizers. The ¯rst one is a dictionary-based recog- nizer which can recognize up to 60'000 words. The second one is an open vocabulary recognizer that can recognize any arbitrary word in any language written in Latin alphabet. We performed experiments in a mono-font context and achieved a WRR of up to 97.56% for the dictionary based recognizer. We achieved an overall WRR of 97.43% for the open vocabulary recognizer. Furthermore, we assessed the performance of the open vocabulary recognizer in a multi-font context more speci¯cally as a sans serif and a serif recognizer. We achieved an overall WRR of 96.94% for both cases. Zusammenfassung In dieser Dissertation wird das Problem der Texterkennung in Anti-Aliased Bildern mit sehr tiefer Aufl¨osungvon 72 bis 100 dpi und bei SchriftgrÄossenzwischen 7 bis 12 Punkten behandelt. Solche Texte kommen hÄau¯gin Bildern im Web vor. Da die darin enthaltenen textlichen In- formationen von grosser semantischer Bedeutung sein kÄonnen,verbessert eine ausreichend gute Erkennung die QualitÄatder Indexierung und erhÄoht auch die Gewinnung von Informationen bei automatischen inhaltsbasierten Suchsystemen. Anti-Aliasing ist eine Technik, die zur Verbesserung der Lesbarkeit bei einer limitierten Anzahl von Pixeln verwendet wird. Dabei werden die Pixel in Graustufen dargestellt und ermÄoglichen damit eine bessere Erkennbarkeit, speziell bei Kanten und Diagonalen. Anti- Aliasing fÄuhrtaber auch dazu, dass ein Teil der einzelnen Buchstaben auf jeweils beiden Seiten miteinander verschmilzt. Diese Verschmelzung fÄuhrtdazu, dass die einzelnen Buchstaben nicht mehr mit den bekannten Segmentierungsmethoden der klassischen Dokumentanalyse aufschlÄussel- bar sind. ZusÄatzlich erzeugt diese Technik ein so genanntes "kontextliches Rauschen", sowohl auf der linken wie auch auf der rechten Seite der einzelnen Buchstaben. Dieses Rauschen fÄuhrtzu einer deutlichen Verschlechterung der Schrifterkennungsrate selbst dann, wenn die Buchstaben eines jeden Wortes vorgÄangigsegmentiert worden sind. Die klassischen OCR-Systeme sind nicht in der Lage, solche Texte ausreichend zu erkennen, da sie grundsÄatzlich zum Erkennen von Scans aus gedrucktem Text mit einer minimalen Aufl¨osungvon 150 dpi entworfen worden sind. In der Literatur ¯ndet man so auch zahlreiche Artikel, die von der Extraktion und Erkennung von Tex- ten in Web-Bildern handeln. Diese Arbeiten konzentrieren sich aber fast ausschliesslich auf die Textextraktion und die ErhÄohung der Aufl¨osungdes extrahierten Textbildes mittels Anwendung diverser Preprocessing Techniken. Die somit erhaltenen Textbilder werden dann anschliessend mittels eines kommerziellen OCR Softwares erkannt. Die hier vorgestellte Arbeit verwendet einen neuartigen Ansatz, sowohl fÄurdie Segmentierung wie auch fÄurdie Erkennung von Anti-Aliased Texten in Bildern mit sehr niedriger Aufl¨osung auch bei kleinen Schriften. Die hier eingesetzten Methoden werden direkt auf den extrahierten Anti-Aliased Text angewendet und verwenden somit auch keine Preprocessing Methoden zur Verbesserung