EasyFont: A Style Learning-Based System to Easily Build Your Large-Scale Handwriting Fonts ZHOUHUI LIAN, BO ZHAO, XUDONG CHEN, and JIANGUO XIAO, Institute of Computer Science and Technology, Peking University, China Generating personal handwriting fonts with large amounts of characters 1 INTRODUCTION is a boring and time-consuming task. For example, the official standard Computer fonts are widely used in our daily lives. Nowadays, texts GB18030-2000 for commercial font products consists of 27,533 Chinese displayed in almost all books, posters, computers, mobile devices, characters. Consistently and correctly writing out such huge amounts of characters is usually an impossible mission for ordinary people. To solve and the like, are rendered using various fonts mainly created by this problem, we propose a system, EasyFont, to automatically synthesize professional companies. Although the number of font products has personal handwriting for all (e.g., Chinese) characters in the font library increased rapidly in the last two decades, existing resources still by learning style from a small number (as few as 1%) of carefully-selected cannot satisfy the demand of every individual since more and more samples written by an ordinary person. Major technical contributions of people want to render texts in their own handwriting styles, which our system are twofold. First, we design an effective stroke extraction algo- are unique and full of personal information. Communicating with rithm that constructs best-suited reference data from a trained font skele- others by posting texts in personal handwriting styles instead of ton manifold and then establishes correspondence between target and ref- using uniform printing font styles could make those experiences erence characters via a non-rigid point set registration approach. Second, more conformable and interesting. we develop a set of novel techniques to learn and recover users’ overall However, building a handwriting font library with a large num- handwriting styles and detailed handwriting behaviors. Experiments in- cluding Turing tests with 97 participants demonstrate that the proposed ber of different characters is not easy. As we know, it is not a prob- system generates high-quality synthesis results, which are indistinguish- lem for writing systems (e.g., English) that only contain a small able from original handwritings. Using our system, for the first time, the alphabet. For example, to create your own English handwriting practical handwriting font library in a user’s personal style with arbitrar- font, only 26 letters and corresponding capitals need to be writ- ily large numbers of Chinese characters can be generated automatically. ten. The whole font generation process can be accomplished in a It can also be observed from our experiments that recently-popularized few minutes by using some existing tools (e.g., FontCreator (2017) deep learning based end-to-end methods are not able to properly handle and FontLab (2017)). Yet, the task becomes tougher when the num- this task, which implies the necessity of expert knowledge and handcrafted ber of characters included in the font library increases. Let us take rules for many applications. Chinese fonts as an example, the official character set GB18030- CCS Concepts: • Computing methodologies → Neural networks; 2000 consists of 27,533 Chinese characters. What’s more, shapes Shape modeling; and structures of many Chinese characters are very complicated. Figure 1(a) shows an example of the Chinese character pronounced 6 Additional Key Words and Phrases: Handwriting, Chinese, style learning, fonts as “biang,” which has 57 strokes. As we know, to be a qualified font library, not only the glyph of each character should represent the ACM Reference format: correct meaning, but also the style of all glyphs must be consis- Zhouhui Lian, Bo Zhao, Xudong Chen, and Jianguo Xiao. 2018. EasyFont: A tent. According to a report made by FounderType (Founder 2017), Style Learning-Based System to Easily Build Your Large-Scale Handwriting a leading Chinese font producing company, it takes more than 12 Fonts. ACM Trans. Graph. 38, 1, Article 6 (December 2018), 18 pages. months for a group with three to five experienced font designers to https://doi.org/10.1145/3213767 generate a GB18030-2000 Chinese font library. Therefore, building a complete Chinese font library in his/her own personal handwrit- ing style is usually an impossible task for an ordinary person. One possible way to accelerate the efficiency of producing large- This work was supported by the National Natural Science Foundation of China (Grant No.: 61472015, 61672056 and 61672043), National Language Committee of China scale font libraries is to exploit the redundancy of components (i.e., (Grant No.: ZDI135-9), National Key Research and Development Program of China radicals, stroke sets, and strokes) for characters in a given charac- (2017YFB1002601) and Key Laboratory of Science, Technology and Standard in Press ter set. In other words, typically, components of characters in a Industry (Key Laboratory of Intelligent Press Media Technology). Authors’ addresses: Z. Lian, B. Zhao, X. Chen, and J. Xiao, No. 128 Zhongguancun selected subset are adequate to cover all the characters’ compo- North Street, Haidian District, Beijing 100080, China; emails: {lianzhouhui, bozhao, nents. Following this intuitive idea, several methods have been re- chenxudong, xiaojianguo}@pku.edu.cn. ported (Lin et al. 2014; Zhou et al. 2011) to generate a given user’s Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed handwriting font library from a number of characters written by for profit or commercial advantage and that copies bear this notice and the fullcita- the user. However, there exist the following two intrinsic draw- tion on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- backs that hinder the application of this kind of method in prac- publish, to post on servers or to redistribute to lists, requires prior specific permission tical use: (1) It is not guaranteed that all components to be reused and/or a fee. Request permissions from [email protected]. can be correctly extracted, which prevents those methods from be- © 2018 Association for Computing Machinery. 0730-0301/2018/12-ART6 $15.00 ing fully automatic; (2) Not only is the percentage of characters https://doi.org/10.1145/3213767 that need to be written too large (more than 20%), but also the ACM Transactions on Graphics, Vol. 38, No. 1, Article 6. Publication date: December 2018. 6:2 • Z. Lian et al. needs to write a small amount (as few as 1%) of carefully-selected Chinese characters on blank papers, take pictures of them, and upload those photos to our system. After receiving these text im- ages, a GB18030-2000 font library in the user’s personal hand- Fig. 1. Examples of some writing systems that contain large numbers of writing style can be automatically generated by the system in different characters. about two hours. More specifically, we first extract the writing tra- jectory of each stroke for every individual character image seg- quality of the auto-generated font library is not satisfactory for mented from input text pictures based on a non-rigid point set real applications. registration approach and several heuristic rules. Then, artificial The other possible solution attracting many researchers is neural networks (ANNs) are utilized to learn and reconstruct the the utilization of deep learning techniques, which have become user’s overall handwriting style, which can be decomposed into extremely popular in the last few years. Methods adopting deep stroke shape style and stroke layout style. Meanwhile, handwrit- neural networks not only obtain state-of-the-art performance in ing details including stroke connectivity and shapes of contours many classical Computer Vision tasks (He et al. 2016; Krizhevsky are also properly described and recovered. Finally, a complete per- et al. 2012;Longetal.2015; Russakovsky et al. 2015; Simonyan and sonal font library can be generated by vectorizing both images Zisserman 2014) including image classification, object detection, of human-written samples and machine-generated handwritings sematic segmentation, and so on, but also become more and for all other characters. Experiments including Turing tests with more competitive in solving generative problems (Gatys et al. 97 participants verify that the proposed system is able to accu- 2017; Isola et al. 2017) thanks to the introduction of Generative rately learn personal handwriting style, automatically synthesize Adversarial Networks (GAN) (Goodfellow et al. 2014) and its indistinguishable handwritings, and quickly generate high-quality variants (Arjovsky et al. 2017). Deep learning based synthesis large-scale font libraries for ordinary people. Our experiments also approaches are good at transferring color/texture styles for demonstrate that domain knowledge is still critical for many ma- images, but less capable of constructing new shapes with complex chine learning tasks (e.g., handwriting synthesis and font design), structures. Scripts, such as Chinese characters, are shapes with in which only a small number of training samples are available. high-level information and complicated structures in which even Recently-popularized deep learning based end-to-end approaches tiny variations of location and geometry for their elements may cannot handle those kinds of problems well without hand-crafted greatly change their meanings and/or styles. As we can see from features and rules. To the best of our knowledge, our work (Lian our experimental results, these kinds of end-to-end methods work et al. 2016) is the first to be able to automatically generate aprac- poorly for the task of synthesizing Chinese characters in hand- tical handwriting font library in a user’s personal style with ar- writing styles. This is mainly due to the lack of understanding bitrarily large numbers of Chinese characters.