DOI: 10.1111/cgf.12597 COMPUTER GRAPHICS forum Volume 00 (2015), number 0 pp. 1–10 Fast Rendering of Image Mosaics and ASCII Art Nenad Markusˇ1, Marco Fratarcangeli2, Igor S. Pandziˇ c´1 and Jorgen¨ Ahlberg3 1University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, Croatia {nenad.markus, igor.pandzic}@fer.hr 2Chalmers University of Technology, Dept. of Applied Information Technology, Goteborg,¨ Sweden [email protected] 3Linkoping¨ University, Dept. of Electrical Engineering, Computer Vision Laboratory, Linkoping,¨ Sweden [email protected] Abstract An image mosaic is an assembly of a large number of small images, usually called tiles, taken from a specific dictionary/codebook. When viewed as a whole, the appearance of a single large image emerges, i.e. each tile approximates a small block of pixels. ASCII art is a related (and older) graphic design technique for producing images from printable characters. Although automatic procedures for both of these visualization schemes have been studied in the past, some are computationally heavy and cannot offer real-time and interactive performance. We propose an algorithm able to reproduce the quality of existing non-photorealistic rendering techniques, in particular ASCII art and image mosaics, obtaining large performance speed-ups. The basic idea is to partition the input image into a rectangular grid and use a decision tree to assign a tile from a pre-determined codebook to each cell. Our implementation can process video streams from webcams in real time and it is suitable for modestly equipped devices. We evaluate our technique by generating the renderings of a variety of images and videos, with good results. The source code of our engine is publicly available. Keywords: ASCII art, image mosaics, decision trees, SSIM ACM CCS: Rendering [Computer Graphics]: Non-photorealistic rendering 1. Introduction We introduce a novel technique that significantly increases the speed of ASCII art and image mosaic rendering. This is achieved As pointed out by Tran [Tra99], a good image mosaic is striking by addressing the problem as a classification task which is then because it cleverly puts together otherwise ordinary and unrelated solved with an optimized binary decision tree. The tree samples pixel features of the individual tiles into a coherent larger framework. Be- intensities at relevant locations in the input image and compares sides having aesthetic value, image mosaics have been investigated them with a set of thresholds in order to assign an appropriate tile to in the context of copyrighted material protection and hiding secret each image region. The thresholds and pixel sampling locations are data [BG03]. ASCII art is a similar graphic design technique that set during the tree learning stage. This simple procedure produces aims at producing images from printable characters. It originated visually appealing results, as shown in Figure 1, while being at least in its modern form in times when printers had limited graphical an order of magnitude faster than other approaches reported in the capabilities and transferring images over computer networks was literature. The limitation of our method is in rendering with a small inappropriate due to bandwidth constraints. It has its value as an number of tiles (low resolution) and/or with a limited codebook. interesting reminder of graphical expression on early computer in- However, these limitations also apply to other approaches. A sample terfaces. Today, ASCII art is commonly used in media that cannot rendering produced with our method can be seen in Figure 2. display images or mainly uses text. These include e-mails, bulletin boards, discussion forums, Internet chats and short message services The runtime part is visually summarized through the example in (SMS). Figure 3. c 2015 The Authors Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd. Published by John Wiley & Sons Ltd. 1 2 N. Markusˇ et al. / Fast Rendering of Image Mosaics and ASCII Art Figure 1: The Lenna image (left) rendered in ASCII with the SSIM-based mapping [WBSS04] (middle) and our decision tree–based approach (right). Our approach produces the same visual results while being two orders of magnitude faster. simplify the process [MJN11] or convert the image to ASCII art in a completely automatic way [aal, OR08, TTIN13, XZW10]. The usual way to automatically generate ASCII art is to subdivide the image into a rectangular grid and replace each cell with a font glyph in such a way that the average grey level stays approximately the same. A representative system that uses this technique is AAlib [aal], written in the 1990s. It relies on pre-computed lookup tables and is able to produce visually appealing results in real time. Another interesting approach has been described by O’Grady and Rickard [OR08]. They treat the conversion of binary images to ASCII art as an optimization problem and use an algorithm based on non-negative matrix factorization techniques [LS99] to solve it. The approach has limited applications due to the considerable processing time needed for a single image. Inspired by the techniques for optimizing binary images for printing [PQW*08], Takeuchi et al. [TTIN13] propose a method which approximates the characteristics of the human visual system with a Gaussian filter and uses this metric to drive the optimization procedure which involves placing individual font glyphs. The authors manage to achieve real-time frame rates on Figure 2: A flower rendered in ASCII tiles with our method. images of moderate size by implementing the algorithm on a GPU. Recently, research has focused on structure-based art [XZW10] which represents an object by rendering its rough outline in ASCII 2. Related Work characters. Their system achieves impressive results that rival even the artwork created by artists. However, it cannot process images Generating ASCII artwork by hand is a labourious task that requires in real time on modern machines and, thus, has a different area of careful placement of individual glyphs. There exist tools which application than the approach we describe in this paper. D00 D10 D11 L "5" "a" "L" "*" Figure 3: ASCII art rendering example. The input greyscale image is partitioned into a rectangular grid (left) and each cell is passed through a decision tree (middle) in order to replace it with an ASCII glyph (right). The decisions in internal tree nodes, Dij , are obtained by comparing the intensity of a pixel at a pre-determined location with a threshold. c 2015 The Authors Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd. N. Markusˇ et al. / Fast Rendering of Image Mosaics and ASCII Art 3 As with ASCII art, image mosaics are usually generated by par- [WBSS04]. Each region of the image is assigned a font glyph that titioning the input image in a grid and assigning a tile from a is most similar to it, as measured by the SSIM index. To obtain codebook to each cell. (There are also techniques for generating the SSIM index between two arrays of pixels (a grid cell and a image mosaics with non-rectangular, rotated or slightly deformed font glyph in our case), we first compute luminance, contrast and tiles [KP02, OK08], which is a different problem from the one structure similarities: we address here.) Silvers [Sil97] described the first computer pro- ⎧ ⎨ 1, if μ and μ are 0 gramme for automatizing this process. The similarity between a 1 2 L = 2μ1μ2 tile and a grid cell is usually calculated by taking L , L or some 12 ⎩ , otherwise, 1 2 μ2 + μ2 string distance between them [Sil97, Tra99], or by using techniques 1 2 from content-based image retrieval [FR98, ZNZ03]. Klein et al. ⎧ [KGFC02] extended the idea to rendering video streams with an ⎨ 1, if σ1 and σ2 are 0 assembly of small video tiles. An overview of classical algorithms C = 2σ1σ2 12 ⎩ , otherwise, can be found in [BBFG07]. Liaw, Tsai, and Tsai [LTT08] improve 2 + 2 σ1 σ2 on basic techniques by incorporating multi-resolution analysis to include the information from surrounding grid cells. The approach ⎧ ⎨ produces gentle colour changes, smooth shapes and more refined 1, if σ1 is 0 or σ2 is 0 S = σ 2 textures. Kang et al. [KSRY13] speed up the basic method based 12 ⎩ 12 , otherwise, on per-pixel distance calculation [Sil97, Tra99] by first downsam- σ1σ2 pling the tiles/cells to n × n thumbnails, where n is a small number where μ and μ are the pixel mean values of the arrays, σ 2 and σ 2 (3 or 4). The computation time needed to find the best matching 1 2 1 2 are their variances and σ 2 is their correlation. The SSIM index is tile is significantly reduced. In addition, the mosaic generation pro- 12 computed as cess becomes more robust to small image perturbations and noise. = l c s In our preliminary experiments, we observed that the approach of SSIM12 L12C12S12, (1) Kang et al. [KSRY13] provides the best trade-off between image mosaic quality and speed, thus, we use it as a baseline for compari- where exponents l, c and s are parameters used to adjust the relative son. The aforementioned approaches require the search of the whole importance of each component. The downside of this approach is its codebook for the best matching tile, and this demands large compu- computational load: it requires a loop over all font glyphs for each 2 tational resources (the search complexity is linear in the size of the grid cell to compute σ12. codebook). This issue has been addressed by Choi et al. [CJKK13] In the case of colour images, we take into account the spatial and Kang et al. [KSRY11]: they report real-time performance by distribution of colour to increase the sharpness of generated image optimizing the tile set and taking advantage of GPU acceleration.