The hOCR Microformat for OCR Workflow and Results
Thomas M. Breuel U. Kaiserslautern and DFKI Trippstadter Str. 122 67663 Kaiserslautern Germany [email protected]
Abstract with a wide variety of layouts. In addition, our system will have to integrate into existing workflows and text Large scale scanning and document conversion ef- databases. In choosing an output format for our sys- forts have led to a renewed interest in OCR systems tem, we evaluated many of the existing formats and en- and workflows. This paper describes a new format for countered numerous limitations. For example, existing representing both intermediate and final OCR results, formats usually have limited multi-lingual capabilities, developed in response to the needs of a newly developed lack support for many common typographic phenom- OCR system and ground truth data release. The format ena, lack a well-defined underlying page and typeset- embeds OCR information invisibly inside the HTML ting model, and often force the use of separate formats and CSS standards and therefore can represent a wide for final output and intermediate results. range of linguistic and typographic phenomena with al- Faced with these and other limitations of existing ready well-defined, widely understood markup and can formats, it was important to find an alternative solu- be processed using widely available and known tools. tion. To this end, we defined a number of requirements: The format is based on a new, multi-level abstraction of the ability to cope with common typographic and OCR results based on logical markup, common typeset- • ting models, and OCR engine-specific markup, making layout phenomena in the major languages and it suitable both for the support of existing workflows and writing systems the development of future model-based OCR engines. the ability to support all processing stages in an • OCR system with a single format 1 Introduction the ability to represent layout information in terms • of a generative typesetting model Over the history of OCR systems, a significant num- the ability to encapsulate existing OCR engine ber of formats have been proposed for representing the • output formats output of OCR systems. We can distinguish three major classes of OCR output formats: logical for- the ability to represent and associate information • mats, suitable for direct use of OCR results by end from different stages of OCR processing users (RTF, HTML, LaTeX, and Microsoft Word), Furthermore, ideally, we wanted to reuse as much of OCR engine-specific formats [5, 9], and benchmark- existing formats and tools as possible, and we wanted ing formats [11, 10] proposed for benchmarking vari- a format that permitted future extensibility. As we will ous aspects of OCR systems. Many OCR engines sup- see, our format fulfills all these requirements. port multiple output formats and represent informa- tion (like text) redundantly within those formats, and commercial and research engines each have their own, 2 A Microformat mutually incompatible formats. We are currently developing a multi-lingual OCR Our approach to addressing the various require- system that is targeting the major writing systems ments without creating a complex new format from and languages of the world, and is intended to deal scratch has been to adopt HTML/XHTML [2] as
Authorized licensed use limited to: Technische Universitat Kaiserslautern. Downloaded on September 10, 2009 at 08:04 from IEEE Xplore. Restrictions apply. the basis format, together with CSS (cascading style hOCR format: sheets) [4, 8] for representing typographic markup, and
2
Authorized licensed use limited to: Technische Universitat Kaiserslautern. Downloaded on September 10, 2009 at 08:04 from IEEE Xplore. Restrictions apply. ocr_document ocr_page ocr_linear ocr_carea ("content area") ocr_title ocr_line [SPAN] ocr_author ocr_float ocr_abstract (subclasses for images, formulas, etc.) ocr_part ocr_separator ocr_chapter [H1] ocr_noise ocr_section [H2] ocr_sub*section [H3,H4] Figure 3. Typesetting-related elements repre- ocr_display ocr_blockquote [BLOCKQUOTE] sent blocks and floats that are filled with the ocr_par [P] document content in reading order in stan- dard typesetting models.
Figure 2. Logical markup available in hOCR. This is markup for the logical hierarchy of a document, independent of where or how into blocks and floats ([1], [6], Microsoft Word, Adobe on the page it is rendered. This particu- Framemaker). Typesetting is performed by flowing the lar markup is usable both for individual doc- linear, logically marked up text into these boxes in uments (memos, articles, etc.), as well as reading order. The combination of logical markup and compound documents consisting of multi- typesetting markup permits us to use hOCR as an in- ple, possibly interleaved, texts (newspapers, termediate format for performing OCR as model-based magazines, collections). reverse typesetting, an approach advocated, for exam- ple by Kopec and Chou [7].
P, etc.). Those structuring elements are recognized 3.3 Engine-Specific Markup by many HTML processing applications and browsers. hOCR recommends a mapping of hOCR structuring At the lowest level, the hOCR format represents elements onto HTML elements that OCR systems can OCR engine-specific, physical layout, like text blocks, use if there is no other reason to prefer alternative map- images, and other page content. This kind of physical pings. Note that when a mapping is used, the hOCR markup is the most commonly found output of exist- elements can be encoded directly in the HTML tags, ing OCR systems. Common elements include “text as in
. blocks”, “figures”, “lines”, and “words”. One important element is ocr_linear, used for rep- However, unlike typesetting markup, which is gen- resenting output for document types like newspapers. erally well-defined in terms of typesetting models, the Newspapers generally consist of multiple stories, each kind of physical markup produced by OCR engines with a fairly regular, linear document structure (title, is implementation dependent. For example, a “text author, sections, paragraphs, etc.) and with text in block” in an engine may be defined in terms of the ex- reading order. However, the different articles of the istence of whitespace separators of minimal size, or the newspaper themselves do not have a unique reading alignment of individual characters. Likewise, “blocks” order; at the top level, the document is a collection of are often also style-dependent; for example, a docu- unordered articles, each of which is a self-contained lin- ment rendered in a style with vertical inter-paragraph ear document (possibly with cross-references to other spacing may be represented with a single block for each documents). This relationship is represented by the paragraph, while in the same document rendered in a ocr_document and ocr_linear elements. different style, an entire column of multiple paragraphs may be returned as a single block in the OCR system. 3.2 Typesetting Markup In contrast, in a typesetting model of page layout, these two styles would be represented in the same way as a In the previous section, we have discussed markup flowable content area, which would also correspond to for representing common linear document structure, the underlying page layout in the source document in found in many typesetting and word processing sys- any of the standard typesetting systems. tems. In order to generate paginated, printed output, Another source of implementation dependencies is typesetting systems and word processors use a nearly the definition of words. OCR systems sometimes define universal typesetting model that divides each page “words” in terms of spacing, at other times linguisti-
3
Authorized licensed use limited to: Technische Universitat Kaiserslautern. Downloaded on September 10, 2009 at 08:04 from IEEE Xplore. Restrictions apply. cally. In order to indicate that engine-specific markup is import libxml2,re,os,string not portable between OCR engines, its elements are doc = libxml2.parseFile(’doc.xhtml’) prefixed with ocrx_ instead of ocr_. Common phys- lines = doc.xpathEval("//*[@class=’ocr_line’]") ical markup tags are ocrx_block, ocrx_line, and for line in lines: ocrx_word. Authors are free to extend the set of ocrx_ textnodes = line.xpathEval(".//text()") elements as needed. for node in textnodes: Given that physical markup is OCR engine specific, print node.getContent() the question arises why one would want to represent it at all. The reason is that many existing workflows al- Figure 4. Example of processing hOCR ready rely on this kind of markup. For example, a spe- markup in Python. This outputs the text con- cific workflow may run the Abbyy or Textbridge OCR tained in the document line-by-line. Simple engine and then perform rule-based post-processing of processing of hOCR documents is enabled the physical layout information produced by those sys- by relying on standard DOM functions, avail- tems. By having a portable, OCR-engine independent able for HTML processing. representation of these features, we can carry forward engine-specific information and yet use portable tools to manipulate the content. Furthermore, although this kind of markup is engine-specific, there are many com- meta information (software version numbers, scan reso- monalities so that code that works for one engine can, lution, number of pages, processing host, etc.), but also after some testing and modification, be used with an- includes headers that indicate the capabilities of the other engine. OCR engine. In particular, the hOCR meta informa- tion must indicate which kind of markup the engine is 3.4 Character Recognition capable of generating, regardless of whether the actual document contains that markup. Without this meta We have already mentioned above that one advan- information, it is impossible to tell whether the lack of tage of the hOCR format over other OCR formats a specific markup in an OCR output file (e.g., para- is that it can reuse the expertise that has gone into graphs, language identification) is due to the absence the development of textual representations for HTML. of the corresponding markup in the source document, Generally speaking, all common style-, font-, script-, or due to the inability of the OCR engine to recognize language-, and typesetting-specific phenomena (hy- that form of markup. OCR system capabilities are in- phenation, spacing, ruby, kashida, etc.) are to be rep- dicated using HTML meta tags simply by listing the resented using their HTML, CSS, and Unicode repre- element names and attributes the OCR system is, in sentations. principle, capable of generating. Inclusion of this meta One area where existing HTML does not provide information is therefore an important part of an OCR an obvious representation is in the representation of system independent OCR format. segmentation and recognition alternatives and weights. Generally, segmentation alternatives are represented using the HTML INS and DEL tags, with associated 3.6 Other Content classes and additional information encoded in the el- ements. In terms of recognition confidences and geo- Let us note briefly that other common typeset- metric segmentation information, hOCR provides two ting objects already have common representations for sets of properties: engine-specific properties (com- their embedding into HTML: MathML for mathemat- monly, “confidence scores” and “bounding boxes”), ics, ChemML for chemical formulas, the IMG tag for and engine-independent properties (posterior probabil- bitmapped images, and SVG for vector graphics. ities and character cutting paths). In addition, there is a wealth of additional embedded formats already available for encoding information such 3.5 Meta Information as resume structures, addresses, identity, personal re- lationships, and bibliographic references. Furthermore, An important requirement for hOCR files is that the hOCR format itself may be extended, or separate, their headers contain information about what kind of additional OCR-related embedded formats may be de- processing was used in order to obtain the output. fined for handling special needs such as indexing, table This information consists not only of the usual OCR of contents, etc.
4
Authorized licensed use limited to: Technische Universitat Kaiserslautern. Downloaded on September 10, 2009 at 08:04 from IEEE Xplore. Restrictions apply. 4 Tool Support cessing and interchange format. A full specification of the hOCR format and sample documents are available The hOCR format has been developed over the last at xxxxxx. 12 months and we have gathered significant hands-on experience with it. It has been used by different groups References as a workflow format representing the output of com- mercial OCR systems, as the output format of a newly [1] Extensible stylesheet language (xsl) version 1.1. developed OCR system (to be released in open source http://www.w3.org/TR/xsl/, 2001. format), and as the format for the release of OCR ground truth for research in OCR. Both integration [2] HTML/XHTML Specifications. into existing HTML output routines of OCR engines, http://www.w3.org/MarkUp, 2006. as well as development of new hOCR generators from scratch has turned out to be easy. [3] The ocropus ocr system home page, 2007. The ability to use tools intended for standard HTML [4] Bert Bos, Hakon Wium Lie, Chris Lilley, and Ian with hOCR has turned out to be very useful. Out- Jacobs. Cascading Style Sheets, level 2: CSS2 put in hOCR format can be viewed and edited using Specification. standard HTML viewers and HTML editors. It can http://www.w3.org/TR/REC-CSS2/, 1998. also be processed using standard DOM processing tools (see Figure 4). Such processing generally preserves the [5] Daniel S. Connelley and Beth Paddock. Xdoc hOCR information and presents all levels of OCR out- data format: Technical specification. Xerox put (content, physical, typesetting, logical) within a Imaging Systems part no. 00-07571-00, 2000. single, consistent document. The specific properties of hOCR have also made [6] Donald Knuth. TeXbook. Addison-Wesley, 1984. it possible to create general-purpose tools that work [7] G. E. Kopec and P. A. Chou. Document image for many different OCR engines. For example, post- decoding using markov source models. IEEE processing of the output of commercial OCR systems PAMI, 16(6), 1994. (e.g., deriving logical layout information from engine specific output, replacing engine-specific fonts with [8] Eric A. Meyer and Bert Bos. Introduction to standard HTML fonts) can be implemented using tools CSS3. http://www.w3.org/TR/css3-roadmap/, that transform hOCR input into hOCR output and 2001. only require small amounts of adaptation to different OCR engines. We have also begun creating a standard [9] Nuance, Inc. SSDOC-SCHEMA2 (XML suite of evaluation tools that measure OCR accuracy Schema). and the performance of different levels of layout anal- http://www.scansoft.com/omnipage/xml/SSDOC- ysis directly on hOCR-encoded OCR output. SCHEMA2.xml, 2006. 5 Conclusions [10] RAF Technologies, Inc. (Redmond, WA). Illuminator user’s manual. The hOCR format introduces several distinct con- http://documents.cfar.umd.edu/ cepts in OCR output formats and workflows: the use resources/source/illuminator.html, 1995. of HTML-based embedded formats for the representa- tion of OCR outputs, the use of typesetting models in [11] B. Yanikoglu and L. Vincent. Pink panther: a the design of OCR formats, and the use of a single for- complete environment for ground-truthing and mat for representing results of all levels of OCR within benchmarking document page segmentation. a single output file. The format has proven itself for Pattern Recognition, 31:1191–1204, 1998. about a year in actual use now, and it will be the basis of the OCRopus [3] open source OCR system and an upcoming release of ground truth data from scanned books and magazines. We hope that a universal, ex- tensible standards-based OCR format, together with ground truth data, OCR software, and various tools will be persuasive enough to induce both commercial vendors and researchers to standardize on a single pro-
5
Authorized licensed use limited to: Technische Universitat Kaiserslautern. Downloaded on September 10, 2009 at 08:04 from IEEE Xplore. Restrictions apply.