Handwritten Boolean Algebra Derivation Recognition and Error Identification

2020 24th International Conference Information Visualisation (IV)

Handwritten Boolean Algebra Derivation Recognition and Error Identiﬁcation

Wisut Silaratana Natasha Dejdumrong Department of Computer Engineering Department of Computer Engineering King Mongkut’s University of Technology Thonburi King Mongkut’s University of Technology Thonburi Bangkok, Thailand Bangkok, Thailand [email protected] [email protected]

Abstract—Boolean algebra is an important study for the anal- needs to concern as well. Some of mistakes are very tiny and ysis and simplification of logic circuit. There are many methods time consuming to find. The red circle from Fig. 2 shown that can simplify Boolean algebra expression quickly. However, error position that makes derivation go in wrong ways. To the simplification Boolean algebra expression by applying Boolean algebra laws was commonly taught in the class. The teacher or best of our knowledge, there are not any software specifically student must spend a lot of time on checking the correctness of made for Boolean algebra derivation checking from image of this method line by line with bare eyes. This paper proposed the handwriting. software to helps them by automatically find error locations from This paper proposed a software for Boolean algebra deriva- image of handwritten Boolean expression derivation. The image tion checking from handwritten image which is the inte- was segmented into the lines before characters would be extracted from each line. After that, the characters were recognise and grating of mathematical expression recognition (MER) and rewriting into the text form. The expression of each line was the computer algebra system (CAS). The input image was simplified by the system, then it will be compared to the next changed from handwritten Boolean algebra expression image line. Finally, the system returns the result in format of input to computer understandable Boolean algebra expression by image overlayed with a result from finding error process. MER before these expressions will be fed to CAS. The Index Terms—Boolean algebra, Derivation, Handwritten recognition expression was reduced to simplest form by CAS and then compare to simplest form in the next line. The comparison will perform continuously until the last pair of expression then I. INTRODUCTION shows the result to user. This software can help people check Boolean algebra is a part of mathematics that is used to the correctness of Boolean algebra derivation faster and more analyze logic problems. Nowadays, we use Boolean algebra accurate. to analyze and design logic circuits to be less complicated while still working the same way. In general, logic circuits are often written as equations of variables, in which each variable represents only two conditions, 0 or 1. The operators such as AND, OR, NOT are often written in other operational symbols, for examples, the OR operator is represented by +, AND operator is often written as a multiplication of variables and NOT is denoted as a bar sign. There are 4 methods used to simplify Boolean algebra expression consisting of Karnaugh Map(K-map) [4], Using Boolean algebra laws [3], Quine-McCluskey algorithm [4] and Espresso algorithm [5]. This paper focus on checking the correctness of using Boolean algebra laws to simplify Boolean algebra expression, which is only one method that must be Fig. 1. Boolean algebra derivation’s pathways done by humans. The solution of this method is usually done in handwritten form because it is easier in writing than typing. Checking the correctness of this method rapidly is very challenging because finding errors line by line by human II. RELATED WORKS is the only way to verify this method. In the addition, difference derivation’s pathway may lead to the same or the MER is a method used to change mathematical expression other simplest form as shown in Fig. 1. Speed is not only image into computer intelligible form. It consists of 2 parts: one problem of checking the correctness of Boolean algebra symbol recognition and structural analysis. There are many of derivation by humans, the accuracy is the important issue that MER study in the literature as presented in [2].

978-1-7281-9134-8/20/$31.00 ©2020 IEEE 654 DOI 10.1109/IV51561.2020.00115 user capture image from the camera on the mathematical expression. III. METHODOLOGY We developed this software by using Mathematica. Fig. 3 shows the overall of the software consisting of 5 task steps as following: • Image preparation • Line segmentation • Character extraction and recognition • Reconstruction as text forms • Error locating and Showing results This software was designed to receive input as a image of handwriting Boolean algebra derivation. Then the program shows results as output image overlay with text in the location where the error in the problem solving process occurs.

Fig. 2. A Tiny mistake in Boolean algebra derivation

A. Symbol recognition Fig. 3. Overall of the software Mathematical expression (ME) is often found in multiple lines of text format. These lines need to be located and A. Image preparation separated before applying recognition. The methods such as The purpose of this step is to convert the input image from histogram projection [8] or Hough transform [9] are used to RGB color image to invert binary color image for using in the locate the lines. When the lines were separated, the next step next step. The input image was rasterized into gray-scaled and is to extract the symbols from these lines. Recursive X-Y cut rounded into integer by rounding function before subtracting [6] and projection profile cutting [7] are the example of the all pixel values by 1 to get invert binary color image with methods that used for symbol segmentation. The difference white foreground and black background image as shown in between these methods is the primitive objects used for pro- Fig. 4. jection. Recursive X-Y cut uses bounding box while projection profile cutting uses pixels. The Convolutional Neural Networks (CNN) is wildly used for character or symbol classification because this technique performs well on many problems such as handwriting recognition and image classification [1]. [10] [11] [12] are the examples of MER studying that used CNN for symbols recognition. However, other famous techniques such as Hidden Markov Model (HMM) [13] and Support Vector Machine (SVM) [14] [15] can be also used for symbol recognition. Fig. 4. A process in image preparation step

B. Structural analysis B. Line segmentation Structural analysis is a method for rearranging symbols from We divide the input image into multiple lines to reduce the symbol recognition steps to have correct syntax. There are complication of structure analysis. The image was separated spatial relations between symbol in mathematical expression. by using horizontal histogram projection as shown in Fig. 5. The associated attributes such as symbol’s location, bounding The space that occurs from projection will be considered as box, size and identity are used to identify spatial relation space between the lines. between symbols. C. Characters extraction and recognition C. Existing application software In this step, we used the connected components func- 1. Wolfram alpha is a web based application that can operate tion that provided in Mathematica to extract the characters with many mathematics ﬁelds. This application gets input from from the images of each line before sending to classify by the user, then return answer and solution as a result. model. Our model is convolutional neural network (CNN) 2. Photomath is a mobile application that gives answer and that trained on handwritten alphabet images from EMNIST shows how to solve the problem step by step instantly when dataset and handwritten mathematical symbol images from

655 Fig. 6. List of components associated attributes data

form that equivalent to the simplest form of previous line will Fig. 5. Histogram from horizontal projection be chosen as expression of that line. To prevent computers from the confusion of the meaning of symbols, we change these symbols to their original form CROHME dataset. The EMNIST dataset contains 62 classes by changing from + to and add && to the back of the and CROHME dataset contains 82 classes. In this work, we use character that contains other characters or symbols ( next to 8 classes from EMNIST dataset as follows: A,B,C,D,W,X,Y it. For the NOT operator that is commonly written in a straight and Z and 4 classes from CROHME dataset consisting of (,),- line horizontally above the letter or what we call a bar symbol, and +. The associated attributes of each component such as will be changed to a symbol ! and put in front of that letter. symbol’s image, size, boundingbox, width, length and result In the event that the bar covers more than one character, the of recognition are delivered to the next step when all of system will enter both! And ( before the first character and components was classified. Fig. 6 is an example of data that ) after the last character under the bar. Fig. 8. The symbol delivered to the next step. under bar sign will be considered from symbol bounding box compare to bar sign bounding box. The system will assign any D. Reconstruction as text forms symbol to group of symbols under bar sign if bounding box There are two objectives in this stage. The first one is to of that symbol was dominated by bar sign bounding box at put the recognized symbol into the correct Boolean algebra least 50 percent. expression. The second step is to change Boolean algebra expression into the format that can manipulate by CAS. For the first objective, similar characters such as ( and C are often wrong classified. Fig. 7 is illustration of method to identify the correct expression. The system will replace the characters that are often misclassified with other similar symbols. The expression that passes the rules will be put into candidate expression list. The rules that we use are as follows: 1. No (+ pattern occurs in the expression. 2. There must be at least 1 character between ( and ). Fig. 7. Method to identify the correct expression 3. There is no ( or ) symbol that is no pair left in the expression. E. Error locating and Showing results If there are more than one candidate expression. All of Boolean expression of each lines will be simplified by CAS, candidate will be simplified by CAS. The candidate’s simplest then compare to the simplest form of next line. If the current

656 Fig. 8. Example of text form reconstruction from result of text recognition expression was transformed to the next expression correctly, the simplest form of these two lines will be treated equivalent. Fig. 9 is an illustration of this software result. The word ”correct” and ”incorrect” will overlay at the position next to the expression.

Fig. 10. The Data before noise elimination

Fig. 9. Result of the software

IV. EXPERIMENTAL RESULTS Our software was designed to work well under the condition of the input image as follows. There is only one line per expression, first line is proposition expression, no continuous characters written and the writing paper should be blank white paper as shown in Fig. 4a In image preparation step, There are very small noises generated from image binarization. Most of them contain only 1 to 10 pixels as shown in Fig. 10. These noises also labeled by the connected component function which cause unnecessary recognition. We eliminate these noises by replacing them with background color. Fig. 11 is illustration of noise elimination’s result. From counting and comparing the number of components in the image before and after applied this method, we found that the total components in Fig. 10 were decreased from 77 components to 11 components in Fig. 11 Accuracy of classification is very important for this software. Wrong classification from the symbol that some features are similar such as A, +, - can be still happened sometimes as shown in Fig. 6. The component number 1 and 9 are bar sign but the result of recognition are + and A respectively. We think about how to solve this problem by increasing data to train the model. Fig. 11. The Data after noise elimination

657 V. CONCLUSION This software is alternative tool for checking the correctness of Boolean algebra simpliﬁcation step. Because checking derivation line by line by human is time consuming and easy to cause errors. The objective is to reduce the mistake and time consuming from the checking process. MER and CAS are applied to develop this software on Mathematica. There are many things can be improved for examples, recognition accuracy, speed of this software, method to handle with noise in the input image. In the future, this software should be able to specify the type of error, give suggestions to the user and manipulate with multiple inputs.

REFERENCES [1] Graham, B., 2014, Spatially-Sparse Convolutional Neural Networks. [Online] Available: https://arxiv.org/abs/1409.6070 [2018 Nov 3]. [2] Chan, K., & Yeung, D., 2000, “Mathematical Expression Recognition: a Survey.” International Journal on Document Analysis and Recognition, Vol.3, pp. 3-15. [3] Electronics Tutorials, 2014 Aug 25, Laws of Boolean Algebra [Online], Available: https://www.electronics-tutorials.ws/boolean/bool 6.html [2018 Nov 3]. [4] Katz, Randy H, Contemporary Logic Design, 2nd ed., Pearson Education International, New Jersey. [5] DCU, 2001, Expresso Minimization Algorithm [Online], Available: www.physics.dcu.ie/ bl/digi/unitd17.pdf [2018 Nov 3]. [6] J. Ha, R. M. Haralick, I. T. Phillips., 1995, ”Understanding mathematical expressions from document images”, 3rd International Conference on Document Analysis and Recognition, 1995 August 14-16, Montreal, Canada, pp. 956–959 [7] M. Okamoto, B. Miao., 1991, ”Recognition of mathematical expressions by using the layout structures of symbols”, 1st International Conference on Document Analysis and Recognition, 1991 September 30-October 2, Saint-Malo, France, pp. 242–250 [8] R. P. d. Santos, G. S. Clemente, T. I. Ren & G. D. C. Cavalcanti, 2009, ”Text Line Segmentation Based on Morphology and Histogram Projection”, 10th International Conference on Document Analysis and Recognition, 2009 July 26-29, Barcelona, Spain. [9] Kumar, M.R., Sahyadri, J., Pradeep, R., Babu, P.S., & Kumar, P., 2012, “A Simple Text-line Segmentation Method for Handwritten Documents”, IJCA Proceedings on National Conference on Advanced Computing and Communications, 2012 August, pp.46-61, [10] Zhang, J., Du, J., Zhang, S., Liu, D., Hu, Y., Hu, J., Wei, S., Dai, L., 2017, “Watch, Attend and Parse: An End-Toend Neural Network Based Approach to Handwritten Mathematical Expression Recognition”, Pattern Recognition, Volume 71, pp. 196-206. 196–206. [11] Wu, J., Yin, F., Zhang, Y., Zhang, X., & Liu, C., 2020, ”Handwritten Mathematical Expression Recognition via Paired Adversarial Learning.”, International Journal of Computer Vision. pp.1-16 [12] Drsouza, L., & Mascarenhas, M., 2018, ”Ofﬂine Handwritten Mathemat- ical Expression Recognition using Convolutional Neural Network”, 2018 International Conference on Information, Communication, Engineering and Technology, 2018 August 29-31, Narhe, Pune, India. [13] Su, B., Ding, X., Peng, L., and Liu, C., 2013, “A Novel Baseline independent Feature Set for Arabic Handwriting Recognition”, 12th International Conference on Document Analysis and Recognition, 2013 Aug 25-28, Washington, DC, USA, pp. 1250–1254. [14] Rizky, M., Nurtanio, I., & Areni, I. S., 2017, ”Mbojo Character Recog- nition Using Shearlet Transform and Support Vector Machine”, 2018 International Seminar on Intelligent Technology and Its Applications, 2018 August 30-31, Bali, Indonesia. [15] Simistira, F ., Papavassiliou, V., Katsouros, V., and Carayannis, G., 2014, “Recognition of Spatial Relations in Mathematical Formulas”, 14th International Conference on Frontiers in Handwriting Recognition, Heraklion, 2014 Sept 1-4, pp. 164-168

658