S S symmetry
Article An Assistive Technology for Braille Users to Support Mathematical Learning: A Semantic Retrieval System
Zahra Asebriy 1,*, Said Raghay 1 and Omar Bencharef 2 1 Faculty of Sciences and Techniques, Cadi Ayyad University, Marrakesh 40000, Morocco; [email protected] 2 Higher School of Technology, Cadi Ayyad University, Essaouira 44000, Morocco; [email protected] * Correspondence: [email protected]
Received: 14 September 2018; Accepted: 19 October 2018; Published: 26 October 2018
Abstract: Mathematical learning from digital libraries and the web is a challenging problem for people with visual impairments and blindness. In this paper, we focus on developing the mathematical learning skills of braille users with a new assistive technology developed to retrieve semantically mathematical information from the web. This kind of research is still in the study phase. This paper presents an overview of assistive technologies for braille users followed by a description of the proposed system, which works in four phases. In the first phase, we translate a query math formula in braille into MathML code, and then we extract the structural and semantic meaning from the MathML expressions using multilevel presentation. In the classification phase, we choose a multilevel similarity measure based on K-Nearest Neighbors to evaluate the relevance between expressions. Finally, the query result is converted to braille math expressions. Experiments based on our database show that the new system provides more efficient results in responding to user queries.
Keywords: braille code; mathematical-expression retrieval; MathML; assistive technologies
1. Introduction Assistive technologies help people with visual impairments to improve their lives. They remove many barriers to use computer technologies for studying, reading, writing digital documents, interacting with each other, and retrieving information on the web and/or in digital libraries. People with visual impairments use speech synthesizers to access the displayed information on the computer screen or braille display (e.g., MAVIS, LaBraDoor, LAMBDA) to write electronic documents, send emails, and take notes, or hearing by screen readers (e.g., Safari+ VoiceOver, Chrome, MathPlayer, MathTalk, JAWS). However, these assistive technologies have some limitations to access all kinds of information like images, tables, graphs, and mathematical notations [1,2]. People with visual impairments face serious difficulties in dealing with and studying mathematics and sciences. They are limited in accessing printed and digital formats of mathematical documents, as math expressions are highly symbolic and nonlinear. Having access to the content of digital mathematical documents (Word, PDF, and web pages) should be in braille or by voice to be easily used by people with sight disabilities in their studies and communications. Another constraint for people with visual impairments is that mathematical braille code is not universal, as literary braille and each country has its own code. To illustrate, Nemeth [3] and GS Braille are used in North America, India, Malaysia, and other countries. RNIB (Royal National Institute for the Blind) [4] is used in the United Kingdom, Marburg in Germany, and CMU (In English: Unified Mathematical Code) in Spain, while Middle Eastern countries use Nemeth Braille. Braille code is not a language and it has limits in the conversion of certain information like mathematical notations. As opposed to text, mathematical expressions are not directly convertible into braille code because mathematical notations have two dimensional writings, while braille code is one-dimensional.
Symmetry 2018, 10, 547; doi:10.3390/sym10110547 www.mdpi.com/journal/symmetry Symmetry 2018, 10, 547 2 of 16
Similarly, markup language mathematics such as LaTeX [5] and MathML [6] are used to standardize and normalize mathematical expressions in order to facilitate the conversion into braille. An additional constraint is the complexity of math expressions and the definition of similarity in math [7]. For example: a + b is identical to x + y, and 3x2 + x + 1 and 4x3 + 5 are both polynomial expressions. In this paper, the focus is on how to help people with visual impairments and blindness to develop their mathematical learning skills by searching and retrieving mathematical documents from the web or on the digital libraries. In this context, a new assistive technology was developed and adapted for braille users to retrieve math expressions. In the second section, we present an overview of existing software solutions for accessibility, studying mathematics and retrieving math expressions. In Section3, we present mathematical braille codes and converters. In Section4, we present and discuss our proposed system. The experimental results are in Section5. Finally, we present the conclusion and future work.
2. State of the Art We review the state-of-the-art process in three stages: we started by performing a keyword search from 2000 to 2017 for phrases like “braille code”; “mathematical expressions retrieval”; “MathML”; “Assistive Technologies”, and many related words. To ensure that we did not miss any important research articles, we conducted the same research on Thomson Reuters (Clarivate), Scopus, and Google Scholar. It was clear that there is considerable lack of this line of research. Second, we reviewed the abstract of 107 articles to keep only papers related to our study. Third, we developed our bibliographic study over three points: accessibility, accessibility to math content, and retrieval of math content.
2.1. Accessibility to Mathematical Content by People with Visual Impairments and Blindness
2.1.1. Accessibility People with visual impairments and blindness, like sighted people, can use computers, tablets, smartphones, and other electronic devices, but differently. They use assistive technologies to help them to easily and independently have access to these devices. Therefore, it is crucial to demonstrate some essential assistance for blind people, encompassing software or hardware systems:
• Screen readers are software applications that allow people with visual impairments to listen to or read the content of a text. It converts the text displayed on the computer screen into speech synthesizer or braille, and the former requires just a simple audio output. However, braille transcription requires additional hardware called refreshable braille displays. In the literature, there are many software solutions, like JAWS (Job Access With Speech), NVDA (Nonvisual Desktop Access), VoiceOver, and Windows Eyes [8]. • A refreshable braille display is an electromechanical system used by people with blindness to read with their fingers like braille hardcopy to decipher the information displayed on the screen. This device can display up to 80 characters on the screen into ephemeral pins, and it changes continuously when the user moves the cursor via control keys. This hardware can also send commands to the computer through the function keys and the routing cursor. With this feature, the blind can perform research, takes notes, and communicate with other people. This device can be used simultaneously or along with a speech-synthesizer system. • Scanner and Optical Character Recognition allows visually impaired people to scan and read printed books. By connecting this hardware with the computer, the printed document can be scanned and converted into an electronic file that is displayed as text on the computer screen. This text can be read and/or listened to by the blind using refreshable braille displays and screen reader software, respectively. The disadvantage of this material lies in the limitation of digitizing all types of information like images, mathematical notations, and tables. Symmetry 2018, 10, 547 3 of 16
Braille e-books, note takers, and voice-recognition software are other assistive technologies that help people with visual impairments to independently interact with the computer and sighted people. Thus, they have some limitations in processing and accessing the content of all types of information such as images, tables, mathematical notations; in addition, the price of these assistive devices is high.
2.1.2. Assistive Technologies for Braille Users to Study and Access Mathematical Content People with visual impairments have many barriers while studying and accessing digital Mathematical documents. They need assistive technology to aid with their disabilities. In this section, it is of paramount significance to present some software solutions intended for blind people to independently use and manipulate digital content, especially that related to mathematics. Braille Math Extension to RoboBraille (BMER) [9] is an automated and flexible system for transcribing schoolbooks created for sighted people into braille books. This system consolidates several braille transcriptions and four math conversion libraries; Sensus SB4 [10] and LibLouis [11] are used for literary braille text conversions, while fMath [12] and UMCL [13] handle the conversion from MathML to a variety of math braille codes. P. B. Stanley et al. [14] presented an assistive software application for people with visual impairments to use mathematics. All mathematics converted into MathML is translated into Nemeth Braille Code (NBC) [3], and the latter can be directly transferred to a refreshable braille device, downloaded into a braille device, and embossed by a third-party application [15,16]. J. A. Gardner [17] described the new LEAN Math application. He gives a brief overview of the features of this system. LEAN Math is an editor for MathType equations in Microsoft (MS) Word. People with visual impairments can use this application to create, edit. and manipulate math equations in braille and/or audio. N. Soiffer [18] proposed the latest revision of the MathPlayer4 system [19], which is an Internet Explorer a plug-in that displays MathML and makes it accessible to assistive technology like screen readers and magnifiers. This version includes features to allow the navigation of mathematical expressions. The results of this system indicated that the comprehension rates were similar compared to the preferred modality used by blind students. A. Salamonczyk and J. B. Pawlowska [20] presented a novel approach to the problem of semantic translation of mathematical expressions to Polish text. To overcome this problem, they used MathML to help and allow blind Polish people to read online documents containing mathematics. People with blindness can access and study digital mathematics using the aforementioned software solutions. However, they cannot research or retrieve, as independently as sighted people, mathematical notations from digital libraries or the web. These solutions cannot retrieve this kind of information.
2.2. Retrieval of Mathematical-Expression Systems Nowadays, several software solutions have been developed to retrieve mathematical documents from the web or digital libraries. Unfortunately, there is no system for people with visual impairments and blindness. In this section, we present existing systems for math=retrieval information and software applications developed for sighted people. Math Information Retrieval (MIR) [21] is based on combining math and text keywords in a query and merging the results of relaxed subqueries. The latter are generated from the original query and they consist of k keywords and f formulae using the Leave Rightmost Out (LRO) method. The partial result lists of all the subqueries are merged into the final list presented to the user. This research used different querying strategies and result merging. The NTCIR-11 [22] Math Task 2 database was employed to evaluate the strategies in this system. The development of the Czech Digital Mathematics Library motivated Masaryk University to start developing a Math Information Retrieval system called Math Indexer and Searcher (MIaS). This system is a math-aware full text-based search engine. It supports a combination of text and math Symmetry 2018, 10, 547 4 of 16 searching. In this subject, Reference [23] presents similarity search for mathematics with different query languages, such as Presentation MathML, Content MathML, Presentation and Content MathML, and Tex. the authors used the NTCIR math task database to evaluate this system. WikiMirs is a mathematical information-retrieval system for Wikipedia proposed in the literature [24]. It is based upon both textual and structural similarities. This system includes four main modules, specifically, a preprocessor, tokenizer, indexer, and ranker. Experimental results that depend on Wikipedia dataset version 01-10-2012 show that WikiMirs retrieves more accurate results and ranks formulae better in Wikipedia compared with two other search engines, Wikipedia Search and Egomath [25]. MathWebSearch (MWS) presented in the literature [26] is a content-based full-text search engine. It provides low-latency answers to full-text queries that contain formulae and text. MWS combines exact and powerful formula unification with the full-text search capabilities of ElasticSearch for a simultaneous full-text search for mathematical documents. NTCIR Math Task Dataset was used to evaluate the MWS. A Mathematical Retrieval System proposed in reference [27] used a novel indexing and matching model, and it is based on both textual and spatial similarities. This system includes six components, namely, user interface for WebPages or PDF documents, a preprocessor that converts formulae markups from different sources into Presentation MathML, a tree construction that constructs semioperator trees from MathML, a tokenizer that extracts all terms, an indexer that uses Term Frequency-Inverse Document Frequency (TF-IDF) and level to each term in the inverted index files, and a ranker that finds the matched terms in the index files. The same dataset collected from publicly available WebPages and PDF documents was used to compare the results of this system with MIaS and WikiMirs. This system provides more efficient mathematical-formulae index and retrieval. The math search presented in Reference [28] proposed a lattice-based approach based on Formal Concept Analysis (FCA). The latter has been used for information retrieval [29–31] as it is a powerful data-analysis technique. This system includes several phases. The first encompasses the conversion of a math query into uniform format as Presentation MathML, then the extraction of features from MathML, after the construction of a mathematical concept lattice with the extracted features, and finally the relevant expressions that are retrieved and ranked when the query and the document share the same attributes. The dataset collected in this paper is 489 math expressions with different functions. S. Yang et al. [32] presented a math information-retrieval system using a novel data structure called FDS (Formal Description Structure) index. They realized exact matching between a math query and the candidate formula data rather than the document. The query-mode algorithms used in this system are: global query mode, local query mode, and operational query mode. For evaluating the efficiency of this proposed system, they constructed a dataset with 2557 mathematical expressions collected from several mathematics documents, and they compared the results of this system with two search engines, WikiMirs and EgoMath. T. Schellenberg et al. [33] presented a math information-retrieval system based on layout substitution trees indexing from a LaTeX dataset. They used a TF-IDF technique to rank or retrieve search results. To evaluate the performance and efficiency of this system, the authors used the same document database and search queries in the Zanibbi and Yuan systems. The results ranked by both systems are comparable. In this paper, like sighted people, we have proposed our assistive technology for people with blindness to help them study, access, and retrieve mathematical expressions from the web and/or digital libraries. SymmetrySymmetry 2018 2018, ,10, 10, ,x, x xFOR FORFOR PEER PEERPEER REVIEW REVIEWREVIEW 55 of of 17 17 SymmetrySymmetry 2018 2018, ,,10 10, ,,x xx FOR FORFOR PEER PEERPEER REVIEW REVIEWREVIEW 55 of of 17 17 3.3. Mathematical Mathematical Braille Braille SymmetrySymmetry3. Mathematical 2018 2018, ,10 10, ,x x FOR FORBraille PEER PEER REVIEW REVIEW 55 of of 17 17 3.3. Mathematical Mathematical Braille Braille SymmetrySymmetrySymmetry 2018 20182018, 10,, 1010, x,, x xFOR FORFOR PEER PEERPEER REVIEW REVIEWREVIEW 5 55of ofof 17 1717 3.1.3.3.3.1. Mathematical Mathematical Mathematical Mathematical Braille BrailleBraille Codes Codes Symmetry3.1.Symmetry3.1. Mathematical Mathematical 2018 20182018, 10,, 1010, x,, xxFOR Braille FORFORBraille PEER PEERPEER Codes Codes REVIEW REVIEWREVIEW 5 55of of 17 1717 3.3.3. Mathematical MathematicalMathematicalBrailleBraille isis aa tactileBrailletactile BrailleBraille writing writing systemsystem usingusing raisedraised dotsdots forfor thethe visuallyvisually impairedimpaired andand thosethose withwith SymmetrySymmetry 2018 2018, 10, 10, x, xFOR FOR PEER PEER REVIEW REVIEW 5 5of of 17 17 strong3.1.Symmetry3.1. Mathematical Mathematical visual 2018, 10disabilities., x Braille FORBraille PEER Codes Codes The REVIEW system is named after its inventor, the Frenchman Louis Braille (18095 of –17 3.3.strong Mathematical MathematicalBrailleBraille visual isis disabilities. aa tactileBraille tactileBraille writing writing The system systemsystem is using namedusing raisedraised after dotsitsdots inventor, forfor thethe the visuallyvisually Frenchman impairedimpaired Louis andand Braille thosethose (1809 withwith– 1852)3.1.SymmetrySymmetry3.1.Symmetry Mathematical Mathematical who 2018 2018 2018 lost, ,10 ,10 10 , his,x ,x xFOR BrailleFOR FORBraillesight PEER PEER PEER Codesafter Codes REVIEW REVIEW REVIEW an accident. 55 5of of of 17 17 17 3.strong1852)3.1.strong3. Mathematical Mathematical BrailleMathematicalBraille whovisual visual lost isis disabilities. disabilities. a a his tactileBrailletactile Braille sight writing writingCodesafter The The an system system system systemaccident. is is usingnamed usingnamed raised raisedafter after its dots itsdots inventor, inventor, forfor thethe the thevisuallyvisually Frenchman Frenchman impairedimpaired Louis Louis andand Braille Braille thosethose (1809 (1809 withwith–– 3.Symmetry Mathematical 2018, 10, x FORBraille PEER REVIEW 5 of 17 3.1.Symmetry1852)strong1852)3.1.Symmetry Mathematical Mathematical Mathematicalwho who visual2018 2018 lost ,lost 10, 10 disabilities., hisx , his xFORBraille FOR Braille sight b sightbraille PEERraille PEER Codes after Codesafter REVIEW is TheREVIEWis diverse diversean ansystem accident. accident. and and is namednot not universal. universal. after its There Thereinventor, are are several severalthe Frenchman b brailleraille b bodes, Louisodes, for forBraille instance: instance: (18095 5of of 17 17 – strong3.3.3.1.3. Mathematical MathematicalMathematical BrailleMathematicalBrailleBraille visual is isisdisabilities. a a atactile Braille tactiletactileBrailleBraille writing writingCodeswriting The system system systemsystem is using named usingusing raised raisedraisedafter itsdots dotsdots inventor, for forfor the thethe the visually visuallyvisually Frenchman impaired impairedimpaired Louis and andand Braille those thosethose (1809 with withwith– Symmetry1852)3.1.3.1.1852)Symmetry3.1. Mathematical MathematicalMathematical Mathematicalwho who 2018 2018 lost ,lost 10, 10, hisx , his xFOR BrailleFOR Braille Braillesight bsight b raillePEERraille PEER Codes after CodesafterCodes REVIEW is REVIEWis diverse diverse an an accident. accident. and and not not universal. universal. There There are are several several b brailleraille b bodes,odes, for for instance: instance:5 5of of 17 17 3.•strong•strong3. strong Mathematical MathematicalBrailleMathematicalBrailleMathematical visual visualvisual is is disabilities. adisabilities. disabilities.a tactile Brailletactile Braille b brailleraille writing writing The notation, TheThenotation, system systemsystemsystem system used used is is is using named usingnamednamedin in France. France. raised raisedafter afterafter its dots itsitsdots inventor, inventor,inventor, for for the the the visually thethevisually Frenchman FrenchmanFrenchman impaired impaired Louis LouisLouis and and Braille BrailleBraille those those (1809 (1809(1809with with– –– •1852)Symmetry3.1.3.1.•1852)strong •1852)3.1. Mathematical MathematicalUnifiedMathematical MathematicalUnifiedwhoMathematical whowho visual2018 lost ,lostlost 10Mathematical Mathematical , disabilities.his 547 hishis Braille BrailleBraillesight b sight sightbbraillerailleraille Codes after Codes Codesafterafter is Theisnotation,Code Codediverse diversean an ansystem accident. accident.(UMC),accident. (UMC), and usedand is notnamed notpracticed in practiced universal.France. universal. after in in itsboth bothThere Thereinventor, Latin Latin are are Americaseveral Americaseveralthe Frenchman b bandraille railleand Spain. Spain.b bodes, Louisodes, for forBraille instance: instance: (18095 of 16 – strong3.strong3.• Mathematical MathematicalBrailleMathematicalBrailleUnifiedMathematicalBraille visual visual is is isdisabilities. Mathematicaldisabilities. a aa tactile Brailletactile tactileBraille b brailleraille writing writingwriting The notation,The notation, Code system system system systemsystem (UMC), used used is is named using usingnamed usingin inpracticed France. France. raised raisedafterraised after inits dotsits dots dots bothinventor, inventor, for for forLatin the thethe the Americathe visually visuallyvisually Frenchman Frenchman and impaired impairedimpaired Spain. Louis Louis and andand Braille Braille those thosethose (1809 (1809 with withwith–– ••1852) 3.1.• TheMathematicalUnifiedMathematicalTheMathematicalUnifiedMathematical who Nemeth Nemeth lost Mathematical Mathematical his Code,Braille Code,b sight bbraillerailleraille developedCodes after developedis isisCode diverseCode diversediverse an accident.(UMC), (UMC), byand byandand the thenot notpracticednot practiced mathematician mathematicianuniversal. universal.universal. in in both bothThere ThereThere Abraham Latin AbrahamLatin are areare Americaseveral Americaseveralseveral Nemeth, Nemeth, b bbrailleand railleandraille Spain.primar Spain.primarb bbodes,odes,odes, il ililfory foryfor used usedinstance: instance:instance: in in the the 1852)3.1.•strongstrong1852)•3.1. strong Mathematical MathematicalBraille BrailleMathematicalwhoUnifiedBraille whovisual visualvisual lost lost is is is disabilities. Mathematical disabilities. adisabilities.his aa his tactile Brailletactile tactileBraillesight bsight brailleraille writingCodes afterwriting writingCodesafter The notation, The Thenotation, Code an systeman system system systemaccident. system systemaccident. (UMC), used used is isis usingnamed using namedusingnamed in in practiced France. France. raised raised raisedafter afterafter in its dotsits itsdotsdots both inventor, inventor,inventor, for for forLatin the thethe the Americathevisuallythe visuallyvisually Frenchman FrenchmanFrenchman andimpaired impairedimpaired Spain. Louis LouisLouis and andand Braille BrailleBraille those thosethose (1809 (1809 (1809 with withwith––– ••• MathematicalUnitedTheUnifiedUnitedTheMathematical Nemeth Nemeth States States Mathematical Code, andb Code,and brailleraille the the developed developedis Middleis MiddlediverseCode diverse (UMC), East. East.andby byand the thenot not practiced mathematician universal.mathematician universal. in bothThere There Abraham LatinAbraham are are several Americaseveral Nemeth,Nemeth, b braille andraille primar Spain.bprimar bodes,odes, il foril yfory used usedinstance: instance: in in the the 3.1.•strong1852)3.strong1852)3.1.• •1852)strong Mathematical Mathematical BrailleMathematicalUnified MathematicalBrailleMathematicalwho whowho visual visualvisual lost lostlostis Mathematical isdisabilities. disabilities. adisabilities. his ahishis tactile Braille tactile Braille sight bsight sight bbraillerailleraille writing Codes afterwriting Codesafter after Thenotation, The Theisnotation,notation,Code diversean ansystem an system systemsystem accident. systemaccident.(UMC),accident. used usedusedand is isis using named namedusing namedin notpracticed in in France. France. France. universal.raised raised after afterafter in its dotsits itsbothdots inventor, There inventor,inventor, for Latinfor theare the America the theseveralvisually thevisually Frenchman FrenchmanFrenchman band railleimpaired impaired Spain. bLouis Louisodes,Louis and and Braille forBrailleBraille those thoseinstance: (1809 (1809 (1809with with– – – •• TheMathematicalUnitedTheUnitedMathematical MarburgNemeth MarburgNemeth States States Code, Code, andCode, bCode,and brailleraille the the developed used developedusedis Middleis Middlediverse diverse in in Germany Germany East. East.by and byand the the not not and mathematicianand mathematicianuniversal. universal. Austria. Austria. There There Abraham Abraham are are several several Nemeth, Nemeth, b brailleraille primar primarb bodes,odes,il ilfory yfor used usedinstance: instance: in in the the •strong1852)1852)• strong 1852)• MathematicalUnified MathematicalUnifiedMathematicalUnifiedwho whovisualwho visual lost lostlost Mathematical disabilities. Mathematical Mathematical disabilities.his hishis sight b sight sightbbraillerailleraille after after after Thenotation, The isnotation,Code Code Code diversean ansystem ansystem accident. accident.(UMC),accident. (UMC),(UMC), used used andis is named named in notpracticed in practicedpracticed France. France. universal. after after in initsin itsboth both both inventor,There inventor, Latin LatinLatin are Americathe AmericaseveralAmericathe Frenchman Frenchman band railleandand Spain. Spain.Spain. bLouis odes,Louis Braille forBraille instance: (1809 (1809– – •3.1.• • MathematicalBrailleMathematicalMathematicalBrailleThe Marburg is is a a tactile Brailletactile BrailleCode,bBrailleraille writing Codeswriting used Code,notation,Code, insystem system createdGermanycreated used using using byinby andFrance. thetheraised raised Austria. authorityauthority dots dots for for ofof thethe visually UnitedUnitedvisually Kingdom Kingdomimpaired impaired and and andand used those usedthose in inwith with thethe •• • TheUnitedMathematicalUnitedThe NemethMarburg MarburgNemeth States States Code, andCode, bCode,andCode,raille the the developed used developedusedis Middle Middle diverse in in Germany Germany East. East. byand by the thenot andmathematician andmathematicianuniversal. Austria. Austria. There Abraham Abraham are several Nemeth, Nemeth, braille primar bprimarodes,il ilforyy used usedinstance: in in the the •strong1852)•• strong1852) • UnifiedMathematical MathematicalUnifiedThewhoMathematical visualwho visual Nemeth lost lost Mathematical disabilities. Mathematical disabilities.his his sight Code,b sight bbraillerailleraille after afterThe developed notation, The isnotation,Codeisnotation, Code diverse andiverse system ansystem accident. (UMC),accident. (UMC), used usedused andbyandis is named the namedin practicednot notin in practiced France. mathematicianFrance. France. universal.universal. after after in inits itsboth both inventor, ThereThereinventor, Latin LatinAbraham areare Americathe America severaltheseveral Frenchman FrenchmanNemeth, bandb railleandraille Spain. Spain.primar bLouisb odes,Louisodes, ilBraille forBraillefory used instance:instance: (1809 (1809in the– – ••• UnitedTheMathematicalTheUnitedMathematical Marburg Marburg Kingdom Kingdom Code, Code, BrailleBraille and and used used Ireland. Code, Ireland.Code, in in Germany createdGermanycreated by byand and thethe Austria. Austria. authorityauthority ofof thethe UnitedUnited Kingdom Kingdom andand usedused inin thethe ••• 1852) • TheMathematicalUnitedUnifiedBrailleTheUnitedMathematicalUnifiedUnited who Nemeth Nemeth islostStates MathematicalStatesStates aMathematical tactilehis Code, and Code,b sight andandbrailleraille writing the thethe developed after developedis Middle is Code Middle MiddlediverseCode diverse systeman accident.(UMC), (UMC),East. East.byEast.and by usingand the the not practicednot practicedmathematician raised mathematicianuniversal. universal. dots in in both forThereboth There the AbrahamLatin AbrahamLatin are visually are Americaseveral Americaseveral Nemeth, impairedNemeth, b brailleand railleand Spain.primar andSpain.primarb bodes,odes, those il ililfory fory used with usedinstance: instance: strongin in the the 1852)•••1852) • MathematicalUnited MathematicalUnifiedwhoUnitedMathematical who lost lost Kingdom KingdomMathematical his his sight b sight bbraillerailleraille and andafter after notation, notation,Ireland.notation, Ireland.Code an an accident. accident. (UMC), used usedused in inin practiced France. France.France. in both Latin America and Spain. •visual• • TheMathematicalMathematicalTheUnitedThe disabilities. Marburg following MarburgMarburgfollowing Kingdom Code, The Braille Code,Code,Braille tabletable and system used used used(Table Code, (TableCode,Ireland. in isinin Germanycreated named1)GermanyGermanycreated1) showsshows after byby and and theand thethe its Austria. Austria.differencesAustria. authoritydifferences inventor,authority ofthe of between between thethe Frenchman UnitedUnited thethe Kingdom symbolsKingdomsymbols Louis Braille of andofand mm athematical (1809–1852)usedusedathematical inin thethe •• • UnitedMathematicalUnifiedTheTheUnitedMathematicalThe Nemeth NemethNemethfollowing States MathematicalStates Code,and Code, bCode, andb raillerailletable the the developed developedisdeveloped Middle is (TableCode Middlediverse diverse (UMC), 1)East. East.and by byandbyshows the the notthe notpracticed mathematician mathematicianuniversal.mathematician theuniversal. differences in both There There LatinAbraham AbrahamAbraham arebetween are Americaseveral several Nemeth, Nemeth,Nemeth,the b braille andraillesymbols Spain.primar primarbprimar bodes,odes, of il il foril ymforyy used athematicalusedinstance:used instance: in inin the thethe •• MathematicalUnitedUnifiedUnitedMathematicalUnified Kingdom Kingdom MathematicalMathematical b brailleraille and and notation, Ireland. notation, Ireland. CodeCode (UMC),(UMC), used used in in practicedpracticed France. France. inin bothboth LatinLatin AmericaAmerica andand Spain.Spain. •bwho•b •raille raille TheMathematicalUnited lostTheMathematicalTheMathematical c codes: odes: Marburg his followingMarburgfollowing States sight the the French afterFrenchCode, and Braille Code, BrailleBrailletabletable the an notation,used notation, used accident.(TableCode, Middle (TableCode,Code, in in Germanycreated 1)Germany createdUMCcreated 1)East.UMC showsshows , ,,and byand bybyand andthe thethe Nemet thetheNemet Austria. Austria.differencesauthority differencesauthorityauthorityh.h. of ofof between betweenthe thethe United UnitedUnited thethe Kingdom symbolsKingdomKingdomsymbols ofand ofandand mm used athematical athematicalusedused in inin the thethe ••• UnifiedTheTheUnitedUnifiedTheUnited Nemeth MarburgNemethNemeth MathematicalStates StatesMathematical Code, Code, Code, andand the the developed developeduseddeveloped Code Middle MiddleCode in (UMC), Germany(UMC), East.East. by byby the the the practiced practiced mathematician andmathematicianmathematician Austria. in in both both LatinAbraham AbrahamLatinAbraham America America Nemeth, Nemeth,Nemeth, and and Spain. primar Spain.primarprimar ilililyyy used usedused in inin the thethe • • MathematicalUnitedMathematicalMathematicalUnitedUnited Kingdom KingdomKingdom b braille brailleraille and andand notation, is Ireland.notation, Ireland.Ireland. diverse used andused notin in France. universal.France. There are several braille bodes, for instance: ••b• b •raille raille MathematicalTheUnitedTheMathematicalThe c codes: odes:Marburg following MarburgMarburgfollowing States the the French FrenchCode,and Braille Code, Code, Brailletabletable the notation,used notation,used used (TableCode, Middle(TableCode, in inin Germanycreated 1)Germany Germanycreated 1)UMC East.UMC showsshows , ,,andby andby and and theandthe theNemetthe theNemet Austria. Austria.differencesAustria.authority differencesauthorityauthorityh.h. of of between betweenthe thethe United United thethe Kingdom symbolsKingdomsymbols ofand ofand mm usedathematical athematicalused in in the the • • UnifiedTheUnitedUnifiedTheUnited Nemeth Nemeth Mathematical States StatesMathematical Code, Code, andand the thedeveloped developed Code Middle MiddleCode (UMC), (UMC), East.East.by by the the practiced practiced mathematician mathematician in in both both LatinAbraham LatinAbraham America America Nemeth, Nemeth, and and Spain. primar Spain.primar ililyy used used in in the the •brailleUnitedTableTableUnited codes: 1 1.Kingdom ..Kingdom D Dtheifferencesifferencesifferences French and and between betweenbetweennotation, Ireland. Ireland. m m athematicalUMC athematical, and bNemetbrailleraille ch.codes:odes: FrenchFrench notation,notation, UnifiedUnified MathematicalMathematical b••b •raille raille TheMathematicalTheMathematicalUnitedTheUnitedTheMathematical c codes: odes:Marburg following MarburgMarburgfollowingfollowing Kingdom Statesthe the French French Code, Code,Braille andCode,Braille table Brailletabletable and the notation, usednotation, used used(Table Code, (TableCode,(TableIreland.MiddleCode, in inin Germany created Germany1) GermanycreatedUMCcreated1) 1)UMC East. shows showsshows, , and byand by by and and theand the theNemetthe theNemet Austria. Austria.differences Austria. authority differencesdifferencesauthorityauthorityh.h. of ofof between betweenbetween the thethe United UnitedUnited the thethe Kingdom symbols KingdomKingdomsymbolssymbols of and ofofandand m mm athematical used usedathematicalathematicalused in inin the thethe • • TheCodeUnitedMathematicalCodeTheTable Nemeth (Nemeth UMC( UMC 1States. D)ifferences )and andCode, andCode, braille Nemeth. Nemeth. the developed betweendeveloped notation, Middle m East. usedathematicalby by the the in mathematician France.mathematician braille codes: Abraham AbrahamFrench notation, Nemeth, Nemeth, Unified primar primar Mathematicalililyy used used in in the the UnitedTableUnitedTableUnited 1 1.Kingdom .KingdomKingdom DDifferencesifferences and andand betweenbetween Ireland. Ireland.Ireland. mm athematical athematical bbrailleraille ccodes:odes: FrenchFrench notation,notation, UnifiedUnified MathematicalMathematical •b•b• braille raille railleTheUnifiedMathematicalMathematicalTheUnitedTheMathematical c codes:c odes:odes:Marburg following Marburgfollowing MathematicalStatesthe thethe French FrenchFrenchCode, BrailleandCode, Braille tableBrailletable the notation, used notation, notation,used(Table Code, (TableCodeCode,MiddleCode, in in Germany (UMC), created1) GermanycreatedUMC1)created UMC UMCEast. showsshows, ,and,practiced by andandby by and theand the Nemetthe theNemetNemet Austria. differencesAustria. authoritydifferences authorityauthority inh.h.h. both Latin of of betweenof between the thethe America United UnitedUnited thethe andKingdom symbolsKingdomKingdomsymbols Spain. of and ofandand mm athematical used usedathematicalused in inin the thethe UnitedTableCodeTableCodeUnited ( UMC (1 UMC1.States .States DDifferencesifferences) )and and and and Nemeth. Nemeth. the the betweenbetween Middle Middle mm East.athematical athematicalEast. bbrailleraille ccodes:odes: FrenchFrench notation,notation, UnifiedUnified MathematicalMathematical b••braille railleMathematicalTheUnitedUnitedTheMathematicalTheUnited c odes:c odes: followingMarburgfollowing Kingdom KingdomKingdomthe the French French Braille Code,Braille tabletable and andand notation, notation,used (TableCode,Ireland. Ireland.(TableCode,Ireland. in created 1)GermanyUMCcreated 1) UMC showsshows, and, by andby and the the Nemetthe theNemet Austria. differencesauthority differencesauthorityh.h. of of between betweenthe the United United thethe Kingdom symbolsKingdomsymbols ofand ofand mm usedathematical usedathematical in in the the •b railleTheCodeTheCode c odes:Marburg Nemeth (Marburg followingUMC(UMC the) )and and Code,FrenchCode, Code, Nemeth. tableNemeth. developedused notation, used(Table in in Germany Germany1) UMC byshows the, and and mathematician andthe Nemet Austria. Austria.differencesh. Abraham between Nemeth, the symbols primarily of m usedathematical in the TableTableUnitedTable 1 1.1 ..KingdomD DDifferencesifferencesifferences and between betweenbetween Ireland.SymbolsSymbols m mmathematical athematicalathematical FrenchFrench b brailleb raille raille UMCc UMCodes:ccodes:odes: French FrenchFrenchNemethNemeth notation, notation,notation, Unified UnifiedUnified Mathematical MathematicalMathematical •bb •braille raillerailleMathematicalUnitedTheTheMathematicalUnited c codes:c odes:odes: followingfollowing Kingdom States Kingdomthe thethe French French French andBraille Brailletabletable and theand notation, notation, notation, (TableCode,Ireland. Middle(Table Code,Ireland. created 1) created1)UMC East. UMCUMC showsshows, ,,and by andandby thethe Nemet the NemetNemet differencesauthority differencesauthorityh.h.h. of of between betweenthe the United United thethe Kingdom symbolsKingdomsymbols ofand ofand mm usedathematical athematicalused in in the the CodeCodeThe ( UMCfollowing(UMC) )and and Nemeth. tableNemeth. (Table 1) shows the differences between the symbols of mathematical TableTableCode 1 (UMC.1 ..D Differencesifferencesifferences) and Nemeth. between betweenbetweenSymbolsSymbols m mathematicalathematical FrenchFrench b braille railleraille cUMC UMCodes:ccodes: French FrenchNemethNemeth notation, notation, Unified Unified Mathematical Mathematical b•bbrailleraillerailleUnitedTheUnitedThe c codes:c odes:odes: Marburg followingfollowing Kingdom Kingdom the thethe French FrenchFrench Code, tabletable and and notation, used notation,notation, (TableIreland. (TableIreland. in Germany 1) UMC1) UMC UMC showsshows, ,,and andand and the theNemet NemetNemet Austria. differencesdifferencesh.h.h. betweenbetween thethe symbolssymbols ofof mmathematicalathematical CodeTableTableCodeTable ( UMC ( 1 UMC1.1 .. D DDifferences)ifferences ifferences)and) and Nemeth. Nemeth. between betweenbetweenSymbols Symbols m mmathematicalathematicalathematical FrenchFrench b bbraille raille raille UMC c UMCcodes:codes:odes: French FrenchFrenchNemethNemeth notation, notation,notation, Unified UnifiedUnified Mathematical MathematicalMathematical b•braillerailleMathematicalThe c odes:codes: following the the French French Braille table notation, notation, Code,(Table created UMC1) UMC shows, ,and byand theNemet Nemet authoritydifferencesh.h. of between the United the Kingdomsymbols andof m usedathematical in the TheCodeTheCode following (following UMC(UMC) )and and table Nemeth.table Nemeth. (Table (Table 1 1 1) 1) shows shows the the differences differences between between the the symbols symbols of of m mathematicalathematical UnitedTableTableCodeTable (1 UMC1.1 Kingdom .. D DDifferencesifferencesifferences) and Nemeth. and between betweenbetween Ireland.SymbolsSymbols Symbols m mmathematicalathematicalathematical FrenchFrenchFrench b b braille railleraille UMC c UMCcUMCodes:codes:odes: French FrenchFrench NemethNemethNemeth notation, notation,notation, Unified UnifiedUnified Mathematical MathematicalMathematical bbrailleraille c odes:codes: the the French French notation, notation,11 UMC UMC, ,and and Nemet Nemeth.h. TableCodeCodeTableCode ( (UMC1 (UMCUMC.1 .D Differences)ifferences) )and andand Nemeth. Nemeth.Nemeth. between betweenSymbols Symbols m mathematicalathematical FrenchFrench b braille raille UMCc UMCodes:codes: French French NemethNemeth notation, notation, Unified Unified Mathematical Mathematical The following table (Table1)11 shows the differences between the symbols of mathematical braille TableCodeTableCode ( UMC1 (UMC.1 .D Differences)ifferences )and and Nemeth. Nemeth. between betweenSymbols SymbolsSymbols m mathematicalathematical FrenchFrenchFrench b braille raille UMCc UMCodes:UMCcodes: French French NemethNemethNemeth notation, notation, Unified Unified Mathematical Mathematical codes: the French notation, UMC,2121 1and Nemeth. CodeCode ( UMC(UMC) )and and Nemeth. Nemeth.Symbols SymbolsSymbols FrenchFrenchFrench UMCUMCUMC NemethNemethNemeth 22 11 Table 1. Differences betweenSymbolsSymbols mathematical FrenchFrench braille codes:UMCUMC French notation,NemethNemeth Unified Mathematical Code 21121 (UMC) and Nemeth. SymbolsSymbols FrenchFrench UMCUMC NemethNemeth 33 2112 12
Symbols23 32 French UMC Nemeth 121 3 3232 1121 44 3223 23
443 2323 2
434 232 3
54354 4 3 33 5 4354 53
55 43443 4
654465 45
66 5545 4
65565 44 7 665576 56
77 656 5
77676 565 8 76687 67
888 SymmetrySymmetrySymmetry 2018 20182018,, , 10 1010,, , x xx FOR FORFOR PEER PEERPEER REVIEW REVIEWREVIEW77 86 666 of ofof 17 1717 6 88 SymmetrySymmetrySymmetry 2018 20182018, ,10, 1010, ,x, x xFOR FORFOR PEER PEERPEER REVIEW REVIEWREVIEW7 767 66 6of ofof 17 1717 66 9989 8 7787 0 8900 98 787 090900 888 77 9 88989 + +9++ 9 898 ++ 9+ 898 9
999 -- - 99 -- - 99
***
** *
// /
// /
^^^
^^^
√√√ √√√ ===
===
(( (
(( (
)) )
)) )
[[ [
[[ [
]] ]
]] ]
{{ {
{{ {
}} }
}} }
TheTheThe standardstandardstandard braillebraillebraille withwithwith sixsixsix pointspointspoints allowsallowsallows onlyonlyonly fewfewfew combinationscombinationscombinations ininin orderorderorder tototo codecodecode thethethe characters.characters.characters.TheTheThe standardstandardstandard ToToTo mitigatemitigatemitigate braillebraillebraille thisthisthis withwithwith problemproblemproblem sixsixsix pointspoints points ininin computing,computing,computing, allowsallowsallows onlyonlyonly theytheythey fewfewfew addedaddedadded combinationscombinationscombinations twotwotwo pointspointspoints ininin at at atorderorderorder thethethe end toendendtoto code codecode tototo have havehave thethethe characters.eighteightcharacters.characters.eight--point-pointpoint Tobraille. braille.Tobraille.To mitigatemitigatemitigate This ThisThis addition addition thisadditionthisthis problemproblemproblem allows allowsallows in in inmaking makingmaking computing,computing,computing, 256 256256 combi combicombi theytheytheynationsnationsnations addedaddedadded (28), (28),(28), twotwotwo which which which pointspointspoints exact exactexact atatat ly thelylythethe correspond correspond correspond endendend tototo havehavehavesss to toto eightthetheeighttheeight ASCII ASCIIASCII--point-pointpoint table table tablebraille. braille.braille. used usedused This ThisThis in inin computing.addition computing. computing.additionaddition allows allowsallows In InIn this thisthis making makingmaking paper, paper,paper, 256 256256 we wewe combi combi combiuse useuse standard standardstandardnationsnationsnations (28), braille(28), (28),braillebraille which whichwhich with withwith exact exact exactsix sixsix points. points.lypoints.lyly correspond correspondcorrespond ss sto toto thethethe ASCII ASCIIASCII table tabletable used usedused in inin computing. computing.computing. In InIn this thisthis paper, paper,paper, we wewe use useuse standard standardstandard braille braillebraille with withwith six sixsix points. points.points. 3.2.3.2.3.2. Mathematical MathematicalMathematical Braille BrailleBraille Converters ConvertersConverters 3.2.3.2.3.2. Mathematical MathematicalMathematical Braille BrailleBraille Converters ConvertersConverters ThereThereThere are areare numerous numerousnumerous braille braillebraille converters convertersconverters in inin the thethe literature literatureliterature,, , like: like:like: ThereThereThere are areare numerous numerousnumerous braille braillebraille converters convertersconverters in inin the thethe literature literatureliterature, ,,like: like:like:
SymmetrySymmetry 2018 2018, ,,10 10, ,,x xx FOR FORFOR PEER PEERPEER REVIEW REVIEWREVIEW 66 of of 17 17 SymmetrySymmetry 2018 2018, ,10 10, ,x x FOR FOR PEER PEER REVIEW REVIEW 66 of of 17 17 SymmetrySymmetrySymmetry 2018 20182018, ,,10 1010, ,,x xx FOR FORFOR PEER PEERPEER REVIEW REVIEWREVIEW 666 of ofof 17 1717 Symmetry 2018, 10, x FOR PEER REVIEW00 6 of 17 SymmetrySymmetry 2018 2018, ,10 10, ,x x FOR FOR PEER PEER REVIEW REVIEW 66 of of 17 17 00 SymmetrySymmetry 2018 2018, ,10 10, ,x x FOR FOR PEER PEER REVIEW REVIEW 66 of of 17 17 Symmetry 2018, 10, x FOR PEER REVIEW000 6 of 17 SymmetrySymmetry 2018 2018, 10, 10, x, xFOR FOR PEER PEER REVIEW REVIEW 6 6of of 17 17 SymmetrySymmetry 2018 2018, 10, 10, x, x FOR FOR PEER PEER REVIEW REVIEW+0 +0 6 6of of 17 17 Symmetry 2018, 10, x FOR PEER REVIEW 6 of 17 Symmetry 2018, 10, x FOR PEER REVIEW0++0 6 of 17 Symmetry 2018, 10, x FOR PEER REVIEW 6 of 17 +0+0+ 0 SymmetrySymmetry 2018 2018, 10, 10, x, xFOR FOR PEER PEER REVIEW REVIEW 6 6of of 17 17 0-++-0 0 SymmetrySymmetry 2018 2018, ,10 10, ,x x FOR FOR PEER PEER REVIEW REVIEW 66 of of 17 17 +0-+- 0 SymmetrySymmetrySymmetry 2018 20182018, ,10, 1010, ,x, xx FOR FORFOR PEER PEERPEER REVIEW REVIEWREVIEW 666 of ofof 17 1717 0-++-0 -+ SymmetrySymmetrySymmetry 2018 2018 2018, ,10 ,10 10, ,x ,x xFOR FOR FOR PEER PEER PEER REVIEW REVIEW REVIEW 66 of6 of of 17 17 17 +0*-*0+ - +
-0*+ 0*-+ 0
0+*-0* +-* 0-
Symmetry 2018, 10, 547 +-/*/+ *- - 6 of 16
*+/-+* / + -
+-/*+/ */-+ * Table 1. Cont.
*/ ^*-/^ /- * Symbols French UMC Nemeth ^/*-^ /- *-
--^*/^ -^ /* -/
^/*^ * / /
*^/^* */ * √√^ */^ */*^ √√ /=^/=^ /^ √√√ =^/=^/ / √√ ˆ^/== /=^ / √√ ^(==(^ √√√√ =^(=^( ^ √√√ ^(=^=( (^ = √√ ==)()= ( = √√ (= (= ))( √√ (=)()= () (
√(√ √)( =[)[= ) ( √√√ ))=[( =[) = (
=([)=[ )[( =)
[])([] [() )
[]() ][ ( )(
]()][ (] []) ([
{[)]{ ])[ [
{]{)[ {] ) [)
)[{] ){ ]{[ )]
}}][{} {[] ]
{}[] }{ [ ][
[]}{ [} {}] [{
The standard braille with six{ }{ points allows only few combinations in order to code the characters. TheThe standardstandard braillebraille withwith ]} } six]{six pointspoints allowsallows onlyonly fewfew combinationscombinations inin orderorder toto codecode thethe To mitigate this problem in computing, they added twopoints at the end to have eight-point braille. characters.characters.TheThe standardstandard ToTo mitigatemitigate braillebraille thisthis withwith problemproblem }{ }]six {] six points points inin computing,computing, allowsallows onlyonly theythey fewfew addedadded combinationscombinations twotwo pointspoints inin at atorderorder thethe end endtoto code codetoto have have thethe ] Thiseightcharacters. addition-Thepoint standard braille.To allows mitigate This makingbraille addition this with 256problem combinationsallows } six points in making computing, allows (28), 256 which combi only they exactly nationsfew added combinations corresponds (28), two which points toexactin theat order lythe ASCII correspond end to table codeto usedhaves theto eightcharacters.TheThe-point standardstandard braille.To mitigate Thisbraillebraille additionthis withwith problem ]{} ] six{ allows]} six pointspoints in making computing, allowsallows 256 onlycombionly they fewnationsfew added combinationscombinations (28), two which points inexactin atorderorder lylythe correspond toendto codecode to have thesthe to in computing. In this paper, we use standard braille with six points. thecharacters.eightthecharacters.eightcharacters. ASCII ASCIIThe-The-pointpoint standardtable standard table braille.To Tobraille.To mitigateused mitigateusedmitigate This Thisbraille braillein in computing. additioncomputing. additionthisthisthis withwith problemproblemproblem }{ allows {}six}allows six In In points points this this in ininmaking making computing,paper, computing,computing,paper, allowsallows 256 256we we combi onlycombiuse onlyuse theytheythey standard standard nationsfewnationsfew addedaddedadded combinationscombinations (28), braille (28),braille twotwotwo which which points pointswith pointswith exact inexactsixin six at atatorder orderpoints. points. lythelythethe correspond correspond end toendendto code code tototo have havehave sthesthe to to eightcharacters.theeightcharacters.theeight ASCII ASCIIThe-The-point-pointpoint standard table standard table braille. To braille.Tobraille. mitigateused mitigateused This ThisbrailleThis braillein in computing. additioncomputing. additionadditionthisthis withwith problemproblem } allowssix allowsallowssix In In points points this thisin inmaking makingmaking computing,paper, computing,paper, allowsallows 256 256256 we we combi only combicombiuse only use theythey standard standardfewnationsnationsfewnations addedadded combinationscombinations (28), braille(28), (28),braille twotwo which which which points pointswith with exact in exactexactinsix six at atorder orderpoints. points.lythelylythe correspond correspondcorrespond endtoendto code code toto have have sthess theto toto 3.2. Mathematical Braille Converters{{ }{ 3.2.3.2. Mathematical Mathematical Braille Braille Converters Converters characters.theeighttheeightcharacters.the ASCII ASCIIASCIIThe-The-pointThepoint standardtable standard tablestandardtable Tobraille. braille.To mitigateused mitigateusedused This This braille braillein braille inin computing. addition computing. computing.additionthisthis withwith with problemproblem } allows } sixallowssix six In InIn points pointsthis points thisinthis inmaking making computing,paper, computing,paper,paper, allowsallowsallows 256 256 we wewe combi onlycombiuse only useonlyuse theythey standard standard standardnations fewnationsfew few addedadded combinationscombinationscombinations (28), braille (28), braillebraille twotwo which which points withpoints withwith exact inexactsixin insixsix at atorder orderpoints. orderpoints. points.lythelythe correspond correspond end toendto to codecode tocodeto have have s thes thetothe to { {{ eight3.2.the3.2.eightcharacters.the ASCII Mathematical ASCIIMathematicalThere-point-point table aretable braille. braille.To numerous used mitigateused Braille Braille This This in in computing.Convertersaddition computing.Converters addition braillethis problem convertersallows allows In In this this makingin making paper, computing, inpaper, the 256 256 literature,we we combi combiuse use they standard standardnationsnations like: added (28), braille (28),braille two which which pointswith with exact exactsix six at points. ly points.lythe correspond correspond end to havess to to characters.thecharacters. ASCIITheThereThereTheThe standard standard standardaretable areTo To numerous numerousmitigate usedmitigate braillebraillebraille in computing. thisbraille thisbraille withwithwith problem problem converters} converters six} sixsix In pointspointspoints thisin in computing, paper,in computing,in allows allowsallowsthe the literature weliteratureliterature only onlyonlyuse theythey standard few,fewfew ,,like: addedlike:like:added combinationscombinationscombinations brailletwo two points pointswith inin in six at orderatorder orderpoints.the the end toendtoto code codecode toto have have thethethe 3.2.3.2.3.2. Mathematical MathematicalMathematical Braille BrailleBraille Converters ConvertersConverters thecharacters.eighttheeightcharacters.characters.eight ASCII ASCIIThereThe-There-Thepoint-pointpoint standardtable standardtableare To braille.are braille.ToTobraille. numerous mitigate numerousused mitigateusedmitigate This ThisbrailleThis inbraille in computing. addition computing. additionthis additionthisbraille thisbraille with with problem problemproblem converters allows converterssix allowsallowssix In In points thispoints inthis inmaking in making making computing, paper, computing,computing,paper,in in allows allowsthe the 256 256 we256 literature weliterature combi only combiuse combionly use they theythey standard standard nationsfewnations, fewnations , addedlike: like:addedadded combinationscombinations (28), braille (28),braille(28), two twotwo which whichwhich points withpoints pointswith exact in exactsix inexactsix at atat orderpoints. orderpoints. lythe lythelythe correspond correspondcorrespond end toendend to code codeto toto have havehave sthes stheto toto • 3.2. Labradoor:Mathematical (LaTeX-to-braille-door)Braille Converters}} } It converts a LaTeX document including mathematical eight3.2.thecharacters.3.2.theeightcharacters.eightthe ASCII Mathematical ASCIIMathematical-ASCIIpoint--pointpoint table tablebraille.table To braille.braille.To mitigateused usedmitigate usedBraille Braille This ThisThis in inin Convertersadditioncomputing. computing.Convertersaddition computing.additionthisthis problemproblem allows allows allows In InIn this this inthismakingin makingmaking computing,paper, computing, paper,paper, 256 256 256 we wewe combi combicombi use useuse theythey standard standardnationsstandardnationsnations addedadded (28), braille(28), (28), braille brailletwotwo which which which points with points withwith exact exactexactsix sixsix atat points. ly points. points.thelylythe correspond correspondcorrespond endend toto havehavess sto toto TheformulaeThereThereThereThe standard standardare areare numerous as numerousnumerous well braillebraille as braille literary braillebraille withwith }converters }converterssixconverters} textsix pointspoints in braille in inin allows allowsthe thethe notation. literature literatureliterature onlyonly few,fewThis ,,like: like:like: combinationscombinations project is one inin of orderorder the first toto code attemptscode thethe 3.2. 3.2. Mathematical Mathematical Braille Braille Converters Converters thecharacters.eightthecharacters.eightthe ASCII ASCIIASCIITheThere-ThereThepoint-point standardtable standard tableare table Toarebraille. Tobraille. numerous mitigatenumerousused mitigateusedused This brailleThis braillein inin computing. addition computing. thiscomputing.addition braille this braille withwith problemproblem converters convertersallowssix allowssix In InIn points pointsthis in thisthis inmaking making computing, paper, computing,paper, inpaper,in allows allowsthe the 256 256 weliterature weliteraturewe combi only usecombi only useuse theythey standard standardstandardfewnationsfew, nations ,addedlike: like:added combinationscombinations (28),braille (28),braille brailletwotwo which which points withpoints withwith inexact sixinexact sixsix at atorder points. order points. thelypoints.lythe correspond correspond end toendto codecode toto have have sthesthe to to 3.2. atMathematical automating Braille the conversion Converters of standard-form mathematical text to print the works in braille, eight3.2.theeightthe3.2. ASCIIMathematical ASCII-Mathematicalpoint-point table tablebraille. braille. used usedBraille BrailleThis This in in additioncomputing.Converters computing.additionConverters allows allows In In this thismaking making paper, paper, 256 256 we we combi combiuse use standard nationsstandardnations (28), braille(28), braille which which with with exact exactsix six points. lypoints.ly correspond correspond s sto to characters. characters.ThereandTheThereTheThe it standard standard standardare is ToareTo used numerous mitigatenumerousmitigate to braillebraillebraille produce this braillethis braille withwith with problemproblem mathematical converters converterssixsixsix points pointspoints inin computing, computing,in in textsallows allowsallowsthe the literature forliterature onlyonlyonly secondary-education theythey few,few few ,like: addedlike:added combinationscombinationscombinations twotwo pointspoints and ininin atuniversity at orderorder order thethe end toendtoto students.codecode codetoto have have thethethe 3.2. 3.2.3.2. Mathematical MathematicalMathematical Braille BrailleBraille Converters ConvertersConverters theeightcharacters.eightcharacters.thecharacters. ASCII TheASCIILabradorThere-TheTherepoint-TheTherepoint standard tablestandard standard tableare braille.are Toarebraille.ToTo uses numerous numeroususedmitigatenumerous mitigatemitigateused This onlybraille This brailleinbraille in computing.additionMarburg computing. addition this this braillethis braillebraille with withwith problemproblemproblem converters allowssix Brailleconvertersconvertersallowssixsix In In points pointsthis points thisin makingin notationinmaking computing,paper, computing,computing, paper,in inallows inallows allowsthe thethe 256 and256 we literaturewe literatureliterature combionly combiuse thereonly only use theytheythey standard standardfew nations isfewnations,few , , like:addedno like:addedaddedlike: combinations conversioncombinationscombinations (28), braille (28),braille twotwotwo which which pointswithpointspoints with with inexact inexactsix insix other at orderat at orderpoints.order ly points. thelythethe correspond nationalcorrespond endto endendto to code code code tototo codes. havehave havesthe sthe theto to 3.2. 3.2. Mathematical Mathematical Braille Braille Converters Converters characters.•theeightcharacters. theeighteightcharacters. ASCII ASCIIThereLiblouis-ThereThere-point-pointpoint table aretable To braille.are areTobraille.braille.To numerous mitigate numeroususednumerous [mitigateusedmitigate11 ]:This ThisThis in in computing. addition computing. thisaddition additionthisbraille Itthis braillebraille problem isproblemproblem converters allows converters convertersallowsallows capable In In this inthis inmakingin makingmaking computing, paper, computing,computing,paper,in inin ofthe thethe 256 256 256weliterature wetranscribingliteratureliterature combi combiusecombi use they theythey standard standardnations, nationsnations , like:added, addedlike:like:added numerous (28), braille (28),(28), brailletwo twotwo which whichwhich points pointswith pointswith exact exactsixexact documentsix at atat points. points.thely lythelythe correspond correspondcorrespond end endend to formatstoto have havehavesss to toto 3.2. 3.2. Mathematical Mathematical Braille Braille Converters Converters eightthethetheeight ASCII ASCIIASCII-TherepointThere-point table tablearebraille.table are braille. numerous numerousused usedused This This in inin additioncomputing. computing. computing.addition braille braille allowsconverters converters allows In InIn this this thismaking making paper, paper,paper,in in the the 256 256 we literature weweliterature combi combiuse useuse standard standardnationsstandard,nations ,like: like: (28), braille (28),braillebraille which which with withwith exact sixexact sixsix points. lypoints.points.ly correspond correspond s sto to eight (such-point as braille. DTBook This and addition XML) inallows braille making and 256 supports combinations numerous (28), which languages. exactly correspond Liblouis alsos to the3.2.3.2. ASCIIMathematical Mathematical table used Braille Braille in computing.Converters Converters In this paper, we use standard braille with six points. the the ASCIITheresupportsASCIIThere tableare tableare mathematicalnumerous numerous usedused inin computing.computing. braille braille braille converters converters code InIn thisthis like paper,inpaper, in Nemeth the the literature weliteraturewe and useuse standard Marburg.standard, ,like: like: braillebraille It is developed withwith sixsix points.points. as a library to be 3.2.3.2.3.2. Mathematical usedMathematicalMathematical in other Braille BrailleBraille programs. Converters ConvertersConverters ThereThere are are numerous numerous braille braille converters converters in in the the literature literature, ,like: like: 3.2.•3.2.3.2. Mathematical MathematicalBraMaNetMathematical [Braille 34 BrailleBraille]: SoftwareConverters ConvertersConverters that converts Presentation MathML only into French Braille. ThereThereThere are areare numerous numerousnumerous braille braillebraille converters convertersconverters in inin the thethe literature literatureliterature, ,,like: like:like: ThereTheThereThere transcribed are areare numerous numerousnumerous file isbraille braillebraille a standard converters convertersconverters textual in inin the thefilethe literature literature soliterature that it, ,like:, can like:like: easily be used by any other braille
transcription software, such as BraMaNetest based on XSLT technology. Diverse parameters are
authorized, including the possibility of modifying the braille output table in order to adapt itself Symmetry 2018, 10, 547 7 of 16
to any material. It also comes with a script called MetaBraMaNet that automatically converts from a Microsoft Word document containing mathematical formulae written in MathType to braille. • UMCL [13]: (Universal Maths Conversion Library) It is a software project for the conversion of mathematical formulae between various formats: MathML specific notation and braille. • Infty [35]: This is an integrated system aiming to make printed scientific documents, including mathematical formulae, accessible. The system consists of three applications’ components: an OCR system named “InftyReader”, a publisher named “InftyEditor”, and conversion tools in various formats. The vocal interface ChattyInfty [36] allows partially sighted users to access and publish the expressions with a speech output. • Math2Braille: open-source software that allows the conversion of MathML files into braille. The process of converting MathML into braille relies on protocols and procedures that were developed in a previous project on access to music.
4. Retrieval Mathematical Equations System for People with Visual Impairments and Blindness Symmetry 2018, 10, x FOR PEER REVIEW 7 of 16 Due to the complexity of our system, we have taken a bottom–up approach to better represent ourthat, work. builds We the start tree by substructure drawing up of the the overall request schema and the of extraction our system of and characteristics we dissect itsin the components. form of the featuresFigure vectors.1 illustrates Finally, the there general is the architecture classification of our of similar proposed vect system.ors to the The request system in allows the database. people withThe visualresults impairments of the classification and blindness are converted to semantically into braille search and the mathematicalpresented to the equations user. We from explain the web in ordetail digital the libraries. various phases The development of the system of thisin the architecture following issections. inspired from classification systems [37].
Figure 1. Proposed system. Figure 1. Proposed system. The system takes place in two modes, specifically, offline mode or indexing mode described by the discontinued4.1. Transcription line, of and Braille online Expression mode described into MathML by arrows. Code In offline mode, the mathematical equations of the database are converted into MathML [6] code. Afterward, there comes the semantic analysis of The difficulties of understanding a braille code lead us to use the Accessible Equation Editor the equations in presentation MathML and the construction of the tree substructure for every formula, (AEE) [38] extension to edit the equations in Nemeth Braille Code. With this extension, we can and then the extraction of the characteristics through this tree substructure in the form of features transcribe equations into braille, which is used as a request equation of our system. The query vectors. Finally, these vectors are stored in the database. conversion interface is shown in Figure 2. In online mode, the user launches a request equation in braille code. The request is firstly pretreated in MathML using a presentation markup that analyzes it semantically and then, based on
Figure 2. Braille conversion interface.
The preprocessing of braille queries into uniform mathematical representation plays an important role in the field of mathematical research. There are several markup languages like LaTeX, and presentation and content MathML. MathML is recommended by W3C and it is the most
Symmetry 2018, 10, x FOR PEER REVIEW 7 of 16
that, builds the tree substructure of the request and the extraction of characteristics in the form of the features vectors. Finally, there is the classification of similar vectors to the request in the database. The results of the classification are converted into braille and presented to the user. We explain in detail the various phases of the system in the following sections.
Symmetry 2018, 10, 547 8 of 16
that, builds the tree substructure of the request and the extraction of characteristics in the form of the features vectors. Finally, there is the classification of similar vectors to the request in the database. The results of the classification are converted into braille and presented to the user. We explain in detail the various phases of the system inFigure the following1. Proposed sections. system.
4.1.4.1. Transcription Transcription of of Braille Braille Expression Expression into into MathML MathML Code Code TheThe difficulties difficulties of of understanding understanding a a braille braille code code lead lead us us to to use use the the Accessible Accessible Equation Equation Editor Editor (AEE)(AEE) [ 38[38]] extension extension to to edit edit the the equations equations in in Nemeth Nemeth Braille Braille Code. Code. WithWith this this extension, extension, we we can can transcribetranscribe equations equations into into braille, braille, which which is is used used as as a a request request equation equation of of our our system. system. The The query query conversionconversion interface interface is is shown shown in in Figure Figure2. 2.
FigureFigure 2. 2.Braille Braille conversion conversion interface. interface.
The preprocessing of braille queries into uniform mathematical representation plays an important SymmetryThe 2018 preprocessing, 10, x FOR PEER REVIEWof braille queries into uniform mathematical representation plays8 of 16an roleimportant in the role field in of the mathematical field of mathematical research. Thereresearch. are There several are markup several languagesmarkup languages like LaTeX, like commonandLaTeX, presentation andformat presentation andfor contentweb andpresentation. MathML. content MathML. MathML Currently, isMath recommended itML is isused recommended by by W3Cnumerous and by itW3C systems is the and most itto is common retrievethe most mathematicalformat for web expressions presentation. on Currently,the web [27,28] it is used and by by numerous other applications, systems to such retrieve as a mathematical MathML to Nemethexpressions translator on the web[14]. [27,28] and by other applications, such as a MathML to Nemeth translator [14]. In our system, we use presentation MathML to code the semantic and notational structure of mathematical equationsequations (Figure (Figure3). This3). This formula formula representation representation is marked is marked and limited and bylimited two markups, by two .
Figure 3. MathML code of mathematical expression 𝑙𝑜𝑔 (𝑠𝑖𝑛x +1 ). Figure 3. MathML code of mathematical expression log(sin ( 3 )).
4.2. Semantic-Tree Construction The structures of the mathematical formulae are naturally hierarchical. They are read and understood in a hierarchical way. The structure of the upper level is more important than those of the lower levels. The upper level generally determines the category and skeleton of the mathematical formula, while the knots of the lower levels usually keep variables or constants that do not bring sense to the content of the formula structure. The structure representation tree can be directly extracted from the MathML code (Figure 4). This representation loses numerous useful semantic contents in searching for mathematical formulae.
Figure 4. Structure representation tree in MathML of equation 𝑥(𝑦 + ).
This representation can have two types of relations: • One-dimensional: Symbols are horizontally connected, e.g., expressions connected by operators ‘+’, ‘–‘, ‘*’, etc. Symbols of this type of relation are connected by the same horizontal label,
Symmetry 2018, 10, x FOR PEER REVIEW 8 of 16
common format for web presentation. Currently, it is used by numerous systems to retrieve mathematical expressions on the web [27,28] and by other applications, such as a MathML to Nemeth translator [14]. In our system, we use presentation MathML to code the semantic and notational structure of mathematical equations (Figure 3). This formula representation is marked and limited by two markups, .
Symmetry 2018, 10, 547 9 of 16 Figure 3. MathML code of mathematical expression 𝑙𝑜𝑔 (𝑠𝑖𝑛 ).
4.2.4.2. Semantic-Tree ConstructionConstruction TheThe structuresstructures ofof thethe mathematicalmathematical formulaeformulae areare naturallynaturally hierarchical.hierarchical. They are readread andand understoodunderstood inin aa hierarchicalhierarchical way.way. TheThe structurestructure ofof thethe upperupper levellevel isis moremore importantimportant thanthan thosethose ofof thethe lowerlower levels.levels. TheThe upperupper levellevel generallygenerally determinesdetermines thethe categorycategory andand skeletonskeleton ofof thethe mathematicalmathematical formula,formula, whilewhile the the knots knots of of the the lower lower levels levels usually usually keep keep variables variables or constants or constants that dothat not do bring not sensebring tosense the contentto the content of the formulaof the formula structure. structure. TheThe structure representation representation tree tree can can be be directly directly extracted extracted from from the theMathML MathML code code (Figure (Figure 4). This4). Thisrepresentation representation loses loses numerous numerous useful useful semantic semantic contents contents in searching in searching for mathematical for mathematical formulae. formulae.
Figure 4. Structure representation tree in MathML of equation 𝑥(𝑦2 + ). Figure 4. Structure representation tree in MathML of equation x y + x .
ThisThis representationrepresentation cancan havehave twotwo typestypes ofof relations:relations: •• One-dimensional:One-dimensional: SymbolsSymbols areare horizontallyhorizontally connected,connected, e.g.,e.g., expressionsexpressions connectedconnected byby operatorsoperators ‘+’,‘+’, ‘–‘,‘–‘, ‘*’, ‘*’, etc. etc. Symbols Symbols of thisof th typeis type of relation of relation are connected are connected by the sameby the horizontal same horizontal label,
Figure 5. Semantic tree in MathML of equation 𝑥(𝑦 + 2 ). Figure 5. Semantic tree in MathML of equation x y + x .
Our algorithm is based on the definition of different levels for each mathematical formula. The first level is defined by the search of all the main operators (+, –, *…) identified in MathML
Figure 6. Multilevel tree construction of equation 𝑐𝑜𝑠(𝑥) ∗𝑠𝑖𝑛(𝑙𝑛(𝑥)).
Symmetry 2018, 10, x FOR PEER REVIEW 9 of 16
Structures of the one-dimensional relations cannot be used in mathematical research. and for that reason, semantic interpretation is necessary in the conversion of the structure representation tree in one-dimensional relation into a matching semantic presentation. This conversion uses methods and algorithms that interpret the operator and level priority (Figure 5).
Symmetry 2018, 10, 547 10 of 16 Figure 5. Semantic tree in MathML of equation 𝑥(𝑦 + ). Our algorithm is based on the definition of different levels for each mathematical formula. The first levelOur is defined algorithm by theis based search on of the all definition the main of operators different (+, levels –, * ...for each) identified mathematical in MathML
FigureFigure 6. 6. MultilevelMultilevel tree tree construction construction of of equation equation cos𝑐𝑜𝑠((x𝑥))∗∗𝑠𝑖𝑛(𝑙𝑛sin(ln((x𝑥))))..
4.3. Feature Extraction
The extraction phase aims to extract the features vector of every mathematical formula of the database. For that, we used the conventional vector model with the multilevel extraction method: Extraction from the multilevel tree shown in Figure6 for every mathematical formula in the database. The conventional vector model is based on the vector concept. This model introduces relevance weights associated to features terms by using different matching functions. It assumes that each equation can be presented by a vector whose nonzero coordinates correspond to the equation’s feature terms. The search for similarity between two equations is executed by calculating the similarity between vectors associated with each of them. The extraction method’s base concepts become as follows:
E: Database set of the equations E1, E2, E3, . . . , En. T: Set of features terms of a database’s equations. The terms are the feature-vector variables: operators (‘+’, ‘–’, ‘*’ . . . ) and functions (cos, sin, sqrt, log, . . . ). V(T): Space vector of dimension |T| = n ∈ N is defined as the number of terms contained in universe T. Vil: Features vector of equation Ei in an l level of the multilevel tree. Wikl: Value of the relevance weight of feature term Tk in equation Ei in the l level of the multilevel tree. In our case, if Tk does not belong to the descriptor terms of equation Eil, Wikl takes a value of 0; otherwise, Wikl takes a value of 1. Symmetry 2018, 10, 547 11 of 16
The application of this method for every level of the multilevel tree allows to have for each equation Ei a set of features vector Vi (Vi1, Vi2, ... , Vin). The first level’s vector Vi1 is obtained by calculating the occurrence number of each term Tk of vector space V(T) in equation Ei while ignoring the content of the terms “exp” and stocking them in features vector Vil. The “exp” terms are treated by repeating the same method applied at the first level. We repeat the same procedure for all remaining levels of equation Ei. The variable and constant terms contain the number of occurrences of the different variables and constants that exist in each level in equation Ei.
4.4. Classification When the user launches a request, the system proceeds by converting it into MathML code and, subsequently, the multilevel tree is constructed. After that comes the extraction of the features using the multilevel method and then storing the extracted features in the form of vector VQ, where VQ(VQ1, VQ2, ... , VQn). The search for similarities between query and database equations is performed by the distance calculation. Therefore, we proposed in this paper to use the K-Nearest-Neighbor (KNN) method. The KNN algorithm is a widely used classification technique [39–41]. It is one of the simplest algorithms as it stocks all available cases and classifies the new ones by similarity measures. Even with such simplicity, KNN can output highly competitive results. We have used this algorithm to look for similarities between the mathematical equation of the request and that of the database. This similarity is based on the calculation of the distance between features vector VQ where VQ(VQ1, VQ2, ... , VQn) stemming from the extraction method. The distance formulae can be expressed as follows:
• Euclidian distance: r 2 = T − d VQl, Vyl ∑i=1 VQli Vyli (1)
where: VQl and Vyl are the features vectors of the request equation and an equation Ey, respectively, for level l. T: is the number of terms in vector space V(T). • Minkowski distance: Euclidean distance is a special case for p = 2 of Minkowski distance.
T 1 d V V = ( V − V ) p Ql, yl ∑i=1 Qli yli (2)
The calculation of the similarity using the multilevel method is based on multilevel structural and semantic similarity between mathematical equations. The degree of similarity is obtained according to all levels of the multilevel tree (Figure6). After having defined the levels of the request and the features vector VQ(VQ1, VQ2, ... , VQn), as a first step, we search all mathematical expressions having similar features vectors of the first level Vi1 to VQ1 by the calculation of the distance between them. Then, we proceed to the next level just for the similar equations retrieved in the first level. This procedure is repeated in the second level. We continue with the same technique for all levels of the query until we obtain the vectors closest to the request. Reverse indexing allows us to retrieve all mathematical equations corresponding to similar founded feature vectors.
4.5. Braille Transcription In this step, we used Math2Braille in order to convert similar query results into braille. As we do not have a refreshable braille display to examine the output, the latter is displayed on the screen. Figure7 illustrates the system output. Symmetry 2018, 10, x FOR PEER REVIEW 11 of 16
Euclidean distance is a special case for p = 2 of Minkowski distance. (2) 𝑑 𝑉 ,𝑉 =( 𝑉 −𝑉 ) The calculation of the similarity using the multilevel method is based on multilevel structural and semantic similarity between mathematical equations. The degree of similarity is obtained according to all levels of the multilevel tree (Figure 6). After having defined the levels of the request and the features vector VQ(VQ1, VQ2, …, VQn), as a first step, we search all mathematical expressions having similar features vectors of the first level Vi1 to VQ1 by the calculation of the distance between them. Then, we proceed to the next level just for the similar equations retrieved in the first level. This procedure is repeated in the second level. We continue with the same technique for all levels of the query until we obtain the vectors closest to the request. Reverse indexing allows us to retrieve all mathematical equations corresponding to similar founded feature vectors.
4.5. Braille Transcription In this step, we used Math2Braille in order to convert similar query results into braille. As we Symmetrydo not have2018, a10 refreshable, 547 braille display to examine the output, the latter is displayed on the screen.12 of 16 Figure 7 illustrates the system output.
Figure 7. Visualization the system output in braille.
5. Experimental Results
5.1. Database Analyzing the performance of a retrieval mathemat mathematical-expressionical-expression system system is is not not an an easy task, because there are few standard reference datasets, contrary to other more common tasks of retrieval information. Normally, Normally, each retrieval mathematical-information system builds its own database for evaluation. The The comparison comparison between between our our system system and and others others is is also also difficult difficult since they are either unavailable or inaccessible. We createdcreated aa datadata system system composed composed of of mathematical mathematical expressions expressions to evaluateto evaluate the the efficiency efficiency of the of thesuggested suggested system. system. All ofAll the of data the aredata constructed are constructed using MathType,using MathType, an interactive an interactive tool for thetool creation for the creationof mathematical of mathematical material foundmaterial in Microsoftfound in WordMicrosoft and PowerPointWord and PowerPoint under the MathType under the Rubin MathType tab to Rubinfacilitate tab editing, to facilitate inserting, editing, and inserting, creating mathematicaland creating mathematical equations. The equations. game elements The game of the elements dataset can be easily converted into MathML or LaTeX (Figure8). Symmetryof the dataset 2018, 10 can, x FOR be PEEReasily REVIEW converted into MathML or LaTeX (Figure 8). 12 of 16
FigureFigure 8. 8. ConversionConversion of of mathematical mathematical eq equationuation into MathML or LaTeX.
In thisthis set,set, we we created created 6925 6925 mathematical mathematical expressions expressions using using symbols symbols from from five languages:five languages: Latin, Latin,Arabic, Arabic, Tifinagh, Tifinagh, Hebrew, Hebrew, and Japanese.and Japanese. For For each each language, language, we we wrote wrote 1385 1385 differentdifferent typestypes of mathematical expressions, such as polynomial, algebraic, statistical, trigonometric, and logarithmic expressions. In this paper, we used a Latin setset to evaluate the performance ofof ourour system.system.
5.2. System Results In this section, we present the results obtained from the suggested system using the KNN algorithm with Minkowski distance. The query results of the proposed system are shown in Table 2 below. For each request, we present the first six results in braille. To make it easier to read braille, we converted the queries and results into readable format.
Table 2. Obtained results.
Queries Queries in Braille Results ⠇⠝⠦⠭⠴⠖⠭ ⠇⠝⠦⠁⠖⠃⠴ ⠇⠝⠦⠽⠴ 𝑙𝑛(𝑥) +𝑥 𝑙𝑛(𝑎+𝑏) 𝑙𝑛 (𝑦) 𝑙𝑛 (𝑥) ⠇⠝⠦⠭⠴ ⠡⠤⠇⠝ ⠇⠝⠦⠭⠴⠖⠻ ⠇⠝⠦⠡⠤⠭⠴ ⠦⠜⠁⠖⠣⠴ 𝑙𝑛(𝑥) +7 𝑙𝑛(1−𝑥) 1−𝑙𝑛 √𝑎 +2 ⠎⠊⠝⠦⠭⠴ ⠭⠎⠊⠝⠦⠭⠴ ⠎⠊⠝⠦⠱⠖⠭⠴ 𝑠𝑖𝑛 (𝑥) 𝑥𝑠𝑖𝑛 (𝑥) 𝑠𝑖𝑛 (5+ 𝑥) ⠎⠊⠝⠦⠣⠴⠖⠎⠊⠝ ⠯⠎⠊⠝ 𝑠𝑖𝑛 (𝑥) ⠎⠊⠝⠦⠭⠴ ⠎⠊⠝⠦⠱⠖⠭⠈⠣⠴ ⠦⠡⠖⠃⠴ ⠦⠭⠖⠣⠴⠙⠭ 𝑠𝑖𝑛(2) +𝑠𝑖𝑛 (1+𝑏) 𝑠𝑖𝑛 (𝑥+2)𝑑𝑥 𝑠𝑖𝑛 (5+ 𝑥 ) ⠡⠖⠭⠽ ⠵⠀⠭⠖⠱ ⠡⠌⠭⠖⠽ 1 1+𝑥𝑦 𝑧𝑥 +5 +𝑦 𝑥 1+𝑥 ⠡⠖⠭ ⠡⠌⠰⠭⠖⠡⠆ ⠑⠦⠣⠴⠖⠡ ⠑⠦⠁⠖⠁⠌⠣⠴ 1 𝑎 𝑒(2) + 1 𝑒(𝑎 + ) 𝑥+1 2 ⠰⠡⠖⠭⠆⠌⠰ ⠡⠌⠭⠖⠽ ⠡⠌⠰⠭⠖⠡⠆ ⠞⠁⠝⠦⠭⠴⠆ 1 1 1+𝑥 1 +𝑦 ⠡⠌⠭ 𝑥 𝑥+1 𝑡𝑎𝑛 (𝑥) 𝑥 ⠰⠣⠀⠭⠆⠌⠰⠡⠤⠭⠆ ⠰⠭⠖⠡⠆⠌⠰⠭⠤⠡⠆ ⠰⠇⠝⠦⠭⠴⠖⠡⠆⠌⠭ 2𝑥 𝑥+1 𝑙𝑛 (𝑥) +1
1−𝑥 𝑥−1 𝑥 ⠇⠝⠰⠀⠆⠦⠎⠊⠝⠰ ⠣⠇⠝⠦⠭⠴⠇⠝⠦⠽⠴ ⠇⠝⠦⠁⠴⠤⠇⠝ 𝑙𝑛 (𝑠𝑖𝑛(1+𝑥)) ⠡⠤⠇⠝⠦⠜⠁⠖⠣⠴ ⠆⠦⠡⠖⠭⠴⠴ ⠶⠡ ⠦⠁⠖⠃⠴
Symmetry 2018, 10, 547 13 of 16
5.2. System Results In this section, we present the results obtained from the suggested system using the KNN algorithm with Minkowski distance. The query results of the proposed system are shown in Table2 below. For each request, we present the firstSymmetry six results 2018, 10, in x FOR braille. PEER ToREVIEW make it easier to read braille, we converted the queries and results13 of 17 into readable format. Table 2. Obtained results. Table 2. Obtained results.
Queries Queries in Braille Results ⠇⠝⠦⠭⠴⠖⠭ ⠇⠝⠦⠁⠖⠃⠴ ⠇⠝⠦⠽⠴ ( ) + ( + ) ( ) ( ) ⠇⠝⠦⠭⠴ ⠡⠤⠇⠝ ⠇⠝⠦⠭⠴⠖⠻ ⠇⠝⠦⠡⠤⠭⠴ ⠦⠜⠁⠖⠣⠴ ( ) +7 (1− ) 1− √ +2 ⠎⠊⠝⠦⠭⠴ ⠭⠎⠊⠝⠦⠭⠴ ⠎⠊⠝⠦⠱⠖⠭⠴ ( ) ( ) (5+ ) ⠎⠊⠝⠦⠣⠴⠖⠎⠊⠝ ⠯⠎⠊⠝ ( ) ⠎⠊⠝⠦⠭⠴ ⠎⠊⠝⠦⠱⠖⠭⠈⠣⠴ ⠦⠡⠖⠃⠴ ⠦⠭⠖⠣⠴⠙⠭ (2) + (1+ ) ( +2) (5+ ) ⠡⠖⠭⠽ ⠵⠀⠭⠖⠱ ⠡⠌⠭⠖⠽ 1 1+ + 5 + 1+ ⠡⠖⠭ ⠡⠌⠰⠭⠖⠡⠆ ⠑⠦⠣⠴⠖⠡ ⠑⠦⠁⠖⠁⠌⠣⠴ 1 (2) + 1 ( + ) +1 2 ⠰⠡⠖⠭⠆⠌⠰ ⠡⠌⠭⠖⠽ ⠡⠌⠰⠭⠖⠡⠆ ⠞⠁⠝⠦⠭⠴⠆ 1 1 1+ 1 + ⠡⠌⠭ +1 ( ) ⠰⠣⠀⠭⠆⠌⠰⠡⠤⠭⠆ ⠰⠭⠖⠡⠆⠌⠰⠭⠤⠡⠆ ⠰⠇⠝⠦⠭⠴⠖⠡⠆⠌⠭ 2 +1 ( ) + 1
SymmetrySymmetry 20182018,,, 1010,,, xxx FORFORFOR PEERPEERPEER REVIEWREVIEWREVIEW 1− −1 1313 ofof 1616 ⠣⠇⠝⠦⠭⠴⠇⠝⠦⠽⠴ ⠇⠝⠦⠁⠴⠤⠇⠝ ⠡⠤⠇⠝⠦⠜⠁⠖⠣⠴ 2𝑙𝑛2𝑙𝑛((𝑥𝑥))⠶⠡𝑙𝑛𝑙𝑛(( 𝑦𝑦)) =1=1 1−𝑙𝑛1−𝑙𝑛√√ ( ( 𝑎𝑎+2)+2) 𝑙𝑛𝑙𝑛⠦⠁⠖⠃⠴((𝑎𝑎))−𝑙𝑛−𝑙𝑛 (𝑎+𝑏)(𝑎+𝑏) ⠇⠝⠰⠀⠆⠦⠎⠊⠝⠰ 2 ( ) ( ) =1 1− (√ +2) ( ) − ( + ) ( (1+ )) ⠇⠝⠦⠭⠴⠖⠇⠝⠇⠝⠦⠭⠴⠖⠇⠝ ⠆⠦⠡⠖⠭⠴⠴ ⠭⠀⠎⠊⠝⠦⠭⠴⠭⠀⠎⠊⠝⠦⠭⠴ ⠇⠝⠦⠡⠤⠭⠴⠇⠝⠦⠡⠤⠭⠴ ⠇⠝⠦⠭⠴⠖⠇⠝ ⠭⠀⠎⠊⠝⠦⠭⠴ ⠇⠝⠦⠡⠤⠭⠴ ⠦⠭⠖⠽⠴⠦⠭⠖⠽⠴ ⠦⠭⠖⠽⠴ 𝑥𝑠𝑖𝑛(𝑥) 𝑙𝑛 (1 − 𝑥) 𝑙𝑛((𝑥))+𝑙𝑛 (𝑥+𝑦) 𝑥𝑠𝑖𝑛(𝑥) ( ) (1𝑙𝑛 − )(1 −𝑥) ( 𝑙𝑛) + ( + )𝑥 +𝑙𝑛 (𝑥+𝑦) ⠰⠡⠖⠎⠊⠝⠦⠭⠴ ⠰⠡⠤⠉⠕⠎⠦⠭⠴ ⠰⠡⠖⠎⠊⠝⠦⠭⠴⠰⠡⠖⠎⠊⠝⠦⠭⠴ ⠰⠡⠤⠉⠕⠎⠦⠭⠴⠰⠡⠤⠉⠕⠎⠦⠭⠴ ⠑⠦⠣⠴⠤⠡⠑⠦⠣⠴⠤⠡⠑⠦⠣⠴⠤⠡ ⠆⠌⠰⠡⠤⠭⠆⠆⠌⠰⠡⠤⠭⠆⠆⠌⠰⠡⠤⠭⠆ ⠆⠌⠰⠡⠤⠭⠆⠆⠌⠰⠡⠤⠭⠆⠆⠌⠰⠡⠤⠭⠆ 1+𝑠𝑖𝑛 (𝑥) 1−𝑐𝑜𝑠 (𝑥) ⠑⠈⠰⠦⠉⠕⠎⠰⠀⠆⠦⠽⠴ 1+𝑠𝑖𝑛1+ ( ) (𝑥) 𝑒(2) − 1 1 − ( )1−𝑐𝑜𝑠 (𝑥) ( ( (( ))( ( (( ) )))) ( )) ⠑⠈⠰⠦⠉⠕⠎⠰⠀⠆⠦⠽⠴⠑⠈⠰⠦⠉⠕⠎⠰⠀⠆⠦⠽⠴ (2)𝑒(2) − 1 − 1 𝑒𝑒 1−𝑥1−𝑥1− 1− 1−𝑥1−𝑥 ⠖⠎⠊⠝⠰⠀⠆⠦⠽⠴⠴⠖⠎⠊⠝⠰⠀⠆⠦⠽⠴⠴⠖⠎⠊⠝⠰⠀⠆⠦⠽⠴⠴ ⠑⠦⠣⠴⠖⠡⠑⠦⠣⠴⠖⠡⠑⠦⠣⠴⠖⠡ ⠯⠎⠊⠝⠦⠭⠖⠣⠴⠙⠭⠯⠎⠊⠝⠦⠭⠖⠣⠴⠙⠭⠯⠎⠊⠝⠦⠭⠖⠣⠴⠙⠭ ⠎⠊⠝⠦⠑⠦⠭⠖⠡⠴⠴⠎⠊⠝⠦⠑⠦⠭⠖⠡⠴⠴⠎⠊⠝⠦⠑⠦⠭⠖⠡⠴⠴ 𝑒(2)𝑒(2) (2) + +1 + 1 1 ( +2) 𝑠𝑖𝑛 𝑠𝑖𝑛 (𝑥+2)𝑑𝑥(𝑥+2)𝑑𝑥 ( ( 𝑠𝑖𝑛𝑠𝑖𝑛 + 1))(𝑒(𝑥(𝑒(𝑥 +1))+1))
5.3. Discussions 5.3.5.3. Discussions Discussions Braille users face many difficulties to study and access to the content of scientific documents Braille users face many difficulties to study and access to the content of scientific documents BrailleBraillecontaining usersusers mathematical faceface manymany notations. difficultiesdifficulties With toto studystudythis new andand algorithm accessaccess toto proposed, thethe contentcontent the ofofpeople scientificscientific with documentsdocumentsvisual containingcontainingimpairments mathematicalmathematical mathematical can independently notations.notations. notations. retrieve WithWith With mathematic thisthis thisnewnew newalalgorithmalgorithm expressions algorithm proposed,proposed, and develop proposed, thth etheire peoplepeople themathematical people withwith visualvisual with impairmentsvisualimpairmentslearning. impairments cancan independentlyindependently can independently retrieveretrieve mathematicmathematic retrieve mathematicalalal expressionsexpressions expressions andand developdevelop and theirtheir develop mathematicalmathematical their learning.mathematicallearning. In Table learning. 2, we found that: InIn Table TableThe2 2,obtained2,, we wewe found foundfound results that: that:that: are several notations, for example, for the test request ⠇⠝⠦⠭⠴ ( ( )) we TheThefound obtained obtained ⠇⠝⠦⠽⠴ resultsresults ( ( )) are andare severalseveral⠇⠝⠦⠁⠖⠃⠴ notations, notations, ( ( + for for)). example, example,These results for for showthe the test test that request request our system ⠇⠝⠦⠭⠴⠇⠝⠦⠭⠴ can retrieve (((𝑙𝑛ln𝑙𝑛((𝑥x𝑥)) )) we we similar math equations with several notations. The suggested system takes into account the foundfound ⠇⠝⠦⠽⠴⠇⠝⠦⠽⠴ (((𝑙𝑛𝑙𝑛ln((𝑦𝑦y)))) andand ⠇⠝⠦⠁⠖⠃⠴⠇⠝⠦⠁⠖⠃⠴ (((𝑙𝑛𝑙𝑛ln(((𝑎+𝑏𝑎+𝑏a + b))).).). These These results resultsresults show showshow that thatthat our our system system can can retrieve retrieve structural and semantic aspect to retrieve math expressions. This characteristic helps learners with similarsimilar mathmath equationsequations withwith severalseveral notations.notations. TheTheThe suggestedsuggestedsuggested systemsystemsystem takestakestakes intointointo accountaccountaccount thethethe similarvisual math impairments equations withto search several and notations. retrieve documents The suggested containing system mathematical takes into accountformulas thewith structural the structural and semantic aspect to retrieve math expressions. This characteristic helps learners with andstructural semanticsame semanticand aspectsemantic meaning to retrieveaspect with todifferent math retrieve expressions.notations. math expressions. This characteristic This characteristic helps learners helps learners with visual with visual impairments to search and retrieve documents containing mathematical formulas with the visual impairmentsIn our system, to wesearch looked and to retrieveretrieve mathematical documents expressions containing with mathematical several types formulas of similarities, with the samesame such semanticsemantic as identical meaningmeaning similarity, withwith differentdifferent subexpression notations.notations. similari ty, or categorical similarity. As an example, the InIn ourour system,system, wewe lookedlooked toto retrieveretrieve mathematicalmathematical expressionsexpressionsexpressions withwithwith severalseveralseveral typestypestypes ofofof similarities,similarities,similarities, such as identical similarity, subexpression similarity, or categorical similarity. As an example, the such as identical similarity, subexpression similarity, or categorical similarity. As an example, the testtest requestrequest ⠡⠖⠭⠡⠖⠭ (((1+𝑥1+𝑥),), wewe foundfound amongamong thethe retrievedretrieved resultsresults ⠡⠌⠰⠭⠖⠡⠆⠡⠌⠰⠭⠖⠡⠆ ((( ),), wherewhere itit takestakes thethe testtest requestrequest asas aa subexpression.subexpression. TheThe automaticautomatic conversionconversion ofof braillebraille mathematicalmathematical equationsequations intointo MathMLMathML isis notnot easy.easy. ItIt hashas severalseveral difficulties,difficulties, especiallyespecially forfor complexcomplex expressionsexpressions containingcontaining severalseveral functionfunction categories.categories. ThisThis deficiencydeficiency justifiesjustifies thethe obtainedobtained resultsresults fromfrom thisthis typetype ofof expression,expression, asas isis thethe casecase forfor thethe resultsresults ofof thethe twotwo testtest queries:queries: •• ⠇⠝⠰⠆⠦⠎⠊⠝⠰⠆⠦⠡⠖⠭⠴⠴⠇⠝⠰⠆⠦⠎⠊⠝⠰⠆⠦⠡⠖⠭⠴⠴ ininin readablereadablereadable formatformatformat 𝑙𝑛𝑙𝑛 (𝑠𝑖𝑛(𝑠𝑖𝑛((1+𝑥1+𝑥)))) ( ( ) ( )) •• ⠑⠈⠰⠦⠉⠕⠎⠰⠆⠦⠽⠴⠖⠎⠊⠝⠰⠆⠦⠽⠴⠴⠑⠈⠰⠦⠉⠕⠎⠰⠆⠦⠽⠴⠖⠎⠊⠝⠰⠆⠦⠽⠴⠴ ininin readablereadablereadable formatformatformat 𝑒𝑒( ( ) ( )) TheThe converters,converters, braillebraille intointo MathMLMathML oror MathMLMathML intointo braille,braille, areare inaccessibleinaccessible oror commercial.commercial. AsAs aa result,result, therethere areare notnot enoughenough assistiveassistive technologiestechnologies forfor peoplepeople withwith visualvisual impairmentsimpairments toto studystudy oror accessaccess mathematicalmathematical sciencescience content.content. ToTo realizerealize thisthis system,system, wewe hadhad manymany difficultiesdifficulties inin findingfinding converters,converters, eithereither braillebraille toto MathMLMathML oror vicevice veversa.rsa. ThatThat isis whywhy wewe usedused twotwo differentdifferent converterconverter types,types, andand adaptedadapted themthem toto achieveachieve ourour goals.goals. OnOn thethethe oneoneone hand,hand,hand, wewewe usedusedused thethethe AccessibleAccessibleAccessible EquationEquationEquation EditorEditor extensionextension asas anan inputinput toto editedit andand convertconvert NemethNemeth BrailleBraille equationsequations intointo presentationpresentation MathML.MathML. OnOn thethe otherother hand,hand, wewe usedused MathtoBrailleMathtoBraille asas converterconverter inin thethe output.output. TheThe useuse ofof thesethese twotwo differentdifferent convertersconverters hashas aa negativenegative influenceinfluence onon thethe relevancerelevance ofof thethe obtainedobtained results.results. DespiteDespite thisthis problem,problem, ourour systemsystem hashas givengiven relevantrelevant andand importantimportant resultsresults inin respondingresponding toto useruser queriesqueries asas shownshown inin TableTable 2.2.
6.6. ConclusionsConclusions InIn thethe presentpresent paper,paper, wewe proposedproposed anan assistiveassistive technologytechnology forfor peoplepeople withwith visualvisual impairmentsimpairments toto retrieveretrieve mathematicalmathematical expressionsexpressions inin digitaldigital librarieslibraries oror thethe web.web. ThisThis systemsystem allowsallows thesethese peoplepeople toto study,study, searchsearch for,for, andand havehave accessaccess toto mathematmathematicalical andand scientificscientific documentdocument contentcontent inin thethe web.web. TheThe resultsresults demonstrateddemonstrated thethe efficiencyefficiency ofof ourour sysystem.stem. TheyThey areare encouragingencouraging toto developdevelop otherother assistiveassistive technologiestechnologies thatthat cancan provideprovide flexflexibleible andand personalizedpersonalized learninglearning andand educationeducation
SymmetrySymmetry 2018, 201810, x, FOR10, x PEERFOR PEER REVIEW REVIEW 13 of 1613 of 16 SymmetrySymmetry 2018, 201810, x, FOR10, x PEERFOR PEER REVIEW REVIEW 13 of 1316 of 16
2𝑙𝑛(𝑥2𝑙𝑛) 𝑙𝑛((𝑥𝑦))𝑙𝑛=1(𝑦) =1 1−𝑙𝑛1−𝑙𝑛 ( 𝑎 +2) ( 𝑎 +2) 𝑙𝑛(𝑎)𝑙𝑛−𝑙𝑛(𝑎) −𝑙𝑛 (𝑎+𝑏) (𝑎+𝑏) 2𝑙𝑛(𝑥) 𝑙𝑛(𝑦) =1 1−𝑙𝑛√ √ ( 𝑎 +2) 𝑙𝑛(𝑎) −𝑙𝑛 (𝑎+𝑏) ⠇⠝⠦⠭⠴⠖⠇⠝⠇⠝⠦⠭⠴⠖⠇⠝ ⠭⠀⠎⠊⠝⠦⠭⠴⠭⠀⠎⠊⠝⠦⠭⠴ ⠇⠝⠦⠡⠤⠭⠴⠇⠝⠦⠡⠤⠭⠴ ⠇⠝⠦⠭⠴⠖⠇⠝ ⠭⠀⠎⠊⠝⠦⠭⠴ ⠇⠝⠦⠡⠤⠭⠴ ⠦⠭⠖⠽⠴ ⠦⠭⠖⠽⠴⠦⠭⠖⠽⠴ 𝑥𝑠𝑖𝑛(𝑥)𝑥𝑠𝑖𝑛(𝑥) 𝑙𝑛 (1𝑙𝑛 − 𝑥) (1 − 𝑥) 𝑙𝑛(𝑥)𝑙𝑛+𝑙𝑛(𝑥) +𝑙𝑛 (𝑥+𝑦) (𝑥+𝑦) 𝑥𝑠𝑖𝑛(𝑥)𝑥𝑠𝑖𝑛(𝑥) 𝑙𝑛 𝑙𝑛(1 − 𝑥)(1 − 𝑥) 𝑙𝑛 𝑥 𝑙𝑛+𝑙𝑛(𝑥) +𝑙𝑛 (𝑥+𝑦) (𝑥+𝑦) ⠰⠡⠖⠎⠊⠝⠦⠭⠴⠰⠡⠖⠎⠊⠝⠦⠭⠴ ⠰⠡⠤⠉⠕⠎⠦⠭⠴⠰⠡⠤⠉⠕⠎⠦⠭⠴ ⠰⠡⠖⠎⠊⠝⠦⠭⠴ ⠑⠦⠣⠴⠤⠡⠑⠦⠣⠴⠤⠡ ⠰⠡⠤⠉⠕⠎⠦⠭⠴ ⠆⠌⠰⠡⠤⠭⠆ ⠑⠦⠣⠴⠤⠡ ⠆⠌⠰⠡⠤⠭⠆ ⠆⠌⠰⠡⠤⠭⠆⠆⠌⠰⠡⠤⠭⠆ ⠆⠌⠰⠡⠤⠭⠆⠆⠌⠰⠡⠤⠭⠆ 1+𝑠𝑖𝑛1+𝑠𝑖𝑛 (𝑥) (𝑥) 1−𝑐𝑜𝑠1−𝑐𝑜𝑠 (𝑥) (𝑥) ( ( ( ) ( () )) ( ))⠑⠈⠰⠦⠉⠕⠎⠰⠀⠆⠦⠽⠴⠑⠈⠰⠦⠉⠕⠎⠰⠀⠆⠦⠽⠴ 𝑒(2)𝑒(2) − 1 − 1 𝑒 𝑒( ( ) ( )) ⠑⠈⠰⠦⠉⠕⠎⠰⠀⠆⠦⠽⠴ 1−𝑥1−𝑥 𝑒(2) − 1 1−𝑥1−𝑥 𝑒 ⠖⠎⠊⠝⠰⠀⠆⠦⠽⠴⠴⠖⠎⠊⠝⠰⠀⠆⠦⠽⠴⠴ 1−𝑥 1−𝑥 ⠖⠎⠊⠝⠰⠀⠆⠦⠽⠴⠴ ⠑⠦⠣⠴⠖⠡ ⠯⠎⠊⠝⠦⠭⠖⠣⠴⠙⠭ ⠎⠊⠝⠦⠑⠦⠭⠖⠡⠴⠴ ⠑⠦⠣⠴⠖⠡⠑⠦⠣⠴⠖⠡ ⠯⠎⠊⠝⠦⠭⠖⠣⠴⠙⠭⠯⠎⠊⠝⠦⠭⠖⠣⠴⠙⠭ ⠎⠊⠝⠦⠑⠦⠭⠖⠡⠴⠴ ⠎⠊⠝⠦⠑⠦⠭⠖⠡⠴⠴ 𝑒(2)𝑒(2) + 1 + 1 𝑠𝑖𝑛 𝑠𝑖𝑛 (𝑥+2)𝑑𝑥 (𝑥+2)𝑑𝑥 𝑠𝑖𝑛𝑠𝑖𝑛 (𝑒(𝑥 (𝑒(𝑥 +1)) +1)) 𝑒(2)𝑒(2) + 1 + 1 𝑠𝑖𝑛 (𝑥+2)𝑑𝑥 𝑠𝑖𝑛𝑠𝑖𝑛 (𝑒(𝑥 (𝑒(𝑥 +1)) +1))
5.3. Discussions5.3. Discussions BrailleBraille users users face facemany many difficulties difficulties to study to study and andaccess access to the to contentthe content of scientific of scientific documents documents containingcontaining mathematical mathematical notations. notations. With With this thisnew new algorithm algorithm proposed, proposed, the thpeoplee people with with visual visual impairmentsimpairmentsimpairments can independentlycancan independentlyindependently retrieve retrieveretrieve mathematic mathematicmathematical expressionsal expressions and anddevelop develop their their mathematical mathematical learning.learning.learning. In TableIn Table 2, we 2, found we found that: that: ( )𝑙𝑛(𝑥) The Theobtained obtained results results are severalare several notations, notations, for example, for example, for the for testthe requesttest request ⠇⠝⠦⠭⠴ ⠇⠝⠦⠭⠴ (𝑙𝑛( 𝑥 ((𝑙𝑛)) (we𝑥)) we ( ) ( ) foundfound ⠇⠝⠦⠽⠴ ⠇⠝⠦⠽⠴ (𝑙𝑛( 𝑦(𝑙𝑛)) (and𝑦)) and⠇⠝⠦⠁⠖⠃⠴ ⠇⠝⠦⠁⠖⠃⠴ (𝑙𝑛( 𝑎+𝑏(𝑙𝑛(𝑎+𝑏)). These)). These results results show show that thatour systemour system can retrievecan retrieve Symmetryfound 2018, 10, 547 ( ) and ( ). These results show that our system can retrieve14 of 16 similarsimilar math math equations equations with with several several notations. notations. The TheThesuggested suggestedsuggested system systemsystem takes takestakes into intointoaccount accountaccount the thethe structuralstructural and andsemantic semantic aspect aspect to retrieve to retrieve math math expressions. expressions. This This characteristic characteristic helps helps learners learners with with visualimpairmentsvisual impairments impairments to to search search to search and and retrieve andretrieve retrieve documents documents documents containing containing containing mathematical mathematical mathematical formulas formulas formulas with with thewith the same the samesemanticsame semantic semantic meaning meaning meaning with with different with different different notations. notations. notations. In ourInIn system, ourour system,system, we looked wewe looked to retrieve to retrieve mathematical mathematical expressions expressions expressions with with with several several several types types types of similarities, of of similarities,similarities, suchsuchsuch as identical asas identicalidentical similarity, similarity, similarity, subexpression subexpression subexpression similari similarity, similarity, orty, or categorical or categorical categorical similarity. similarity. similarity. As As an anAs example, example,an example, the testthe test requesttest request ⠡⠖⠭ ⠡⠖⠭ (1+𝑥 (1+𝑥), we), found we found among among the retrieved the retrieved results results ⠡⠌⠰⠭⠖⠡⠆ ⠡⠌⠰⠭⠖⠡⠆ ( 1 ),( where ), where it takes it takes the the requesttest request ⠡⠖⠭(1 + (1+𝑥x), we), foundwe found among among the retrievedthe retrieved results results ⠡⠌⠰⠭⠖⠡⠆( x + ),( where), where it takes it takes the testthe 1 test requesttest request as a as subexpression. a subexpression. The TheTheautomatic automaticautomatic conversion conversion conversion of ofbraille of braille braille mathematical mathematical mathematical equations equations equations into MathMLintoMathML MathML is is not not is easy. easy.not It easy. hasIt has several It has severaldifficulties,several difficulties, difficulties, especially especially especially for complex for complexfor complex expressions expressions expressions containing containing containing several several function several function categories. function categories. categories. This deficiency This This deficiencyjustifiesdeficiency justifies the justifies obtained the obtainedthe results obtained results from results thisfrom typefrom this ofthistype expression, type of expression, of expression, as is theas is caseas the is forcasethe the casefor results thefor resultsthe of results the of two of the twotestthe twotest queries: queries:test queries: • • 𝑙𝑛( (𝑠𝑖𝑛(1+𝑥) )) •⠇⠝⠰⠆⠦⠎⠊⠝⠰⠆⠦⠡⠖⠭⠴⠴ ⠇⠝⠰⠆⠦⠎⠊⠝⠰⠆⠦⠡⠖⠭⠴⠴ in readable inin readablereadable format format format 𝑙𝑛 ln(𝑠𝑖𝑛𝑙𝑛(sin1+𝑥 (𝑠𝑖𝑛((11+𝑥)+) x)))) ( (( ) ( )( )) ( )) • •⠑⠈⠰⠦⠉⠕⠎⠰⠆⠦⠽⠴⠖⠎⠊⠝⠰⠆⠦⠽⠴⠴ ⠑⠈⠰⠦⠉⠕⠎⠰⠆⠦⠽⠴⠖⠎⠊⠝⠰⠆⠦⠽⠴⠴ in readable in readable format format 𝑒 𝑒( ( ) ( )) • ⠑⠈⠰⠦⠉⠕⠎⠰⠆⠦⠽⠴⠖⠎⠊⠝⠰⠆⠦⠽⠴⠴ in readable format e𝑒(cos (y)+sin (y)) The Theconverters, converters, braille braille into intoMathML MathML or MathML or MathML into intobraille, braille, are inaccessible are inaccessible or commercial. or commercial. As As The converters, braille braille into into MathML MathML or or MathML MathML into into braille, braille, are are inaccessible inaccessible or commercial. or commercial. As a result,a result, there there are notare enoughnot enough assistive assistive technologies technologies for people for people with with visual visual impairments impairments to study to study or or Asa result, a result, there there are arenot not enough enough assistive assistive technologies technologies for forpeople people with with visual visual impairments impairments to study to study or accessaccess mathematical mathematical science science content. content. To realizeTo realize this thissystem, system, we hadwe hadmany many difficulties difficulties in finding in finding oraccess access mathematical mathematical science science content. content. To To realize realize this this system, system, we we had had many many difficulties difficulties in findingfinding converters,converters, either either braille braille to MathML to MathML or vice or viceversa. ve rsa.That That is why is why we usedwe used two twodifferent different converter converter converters, eithereither braillebraille to to MathML MathML or or vice vice versa. versa. That That is why is why we usedwe used two differenttwo different converter converter types, types,types, and andadapted adapted them them to achieve to achieve our goals.our goals. On theOn onethe onehand, hand, we usedwe used the Accessiblethe Accessible Equation Equation andtypes, adapted and adapted them to them achieve to achieve our goals. our Ongoals. the On one the hand, one wehand, used we the used Accessible the Accessible Equation Equation Editor EditorEditor extension extension as anas inputan input to editto editand andconvert convert Nemeth Nemeth Braille Braille equations equations into intopresentation presentation extensionEditor extension as an input as toan edit input and to convert edit and Nemeth convert Braille Nemeth equations Braille into presentationequations into MathML. presentation On the MathML.MathML. On theOn otherthe other hand, hand, we usedwe used MathtoBraille MathtoBraille as converter as converter in the in output.the output. The Theuse ofuse these of these otherMathML. hand, On we the used other MathtoBraille hand, we asused converter MathtoBraille in the output. as converter The use in of thesethe output. two different The use converters of these two twodifferent different converters converters has ahas negative a negative influence influence on the on relevancethe relevance of the of obtainedthe obtained results. results. Despite Despite hastwo adifferent negative converters influence on has the a relevancenegative ofinfluence the obtained on the results. relevance Despite of the this obtained problem, results. our system Despite has this problem,this problem, our systemour system has givenhas given relevant relevant and andimportant important results results in responding in responding to user to userqueries queries as as giventhis problem, relevant our and system important has resultsgiven relevant in responding and important to user queries results as in shown responding in Table to2 user. queries as shownshown in Table in Table 2. 2. 6. Conclusions 6. Conclusions6. Conclusions In the present paper, we proposed an assistive technology for people with visual impairments In theIn presentthe present paper, paper, we proposed we proposed an assistive an assistive technology technology for people for people with with visual visual impairments impairments toIn retrieve theIn presentthe mathematicalpresent paper, paper, we expressions proposedwe proposed an in assistive digitalan assistive libraries technology technology or the for web. people for Thispeople with system with visual allowsvisual impairments impairments these people to retrieveto retrieve mathematical mathematical expressions expressions in digital in digital libraries libraries or the or web.the web. This Thissystem system allows allows these these people people to retrieveto study,retrieve mathematical search mathematical for, andexpressions expressions have access in digital in to digital mathematical libraries libraries or the andor web.the scientific web. This This system document system allows allows content these these inpeople the people web. to study,to study, search search for, andfor, andhave have access access to mathemat to mathematical andical andscientific scientific document document content content in the in web.the web. Theto study, results search demonstrated for, and thehave efficiency access to of mathemat our system.ical They and are scientific encouraging document to develop content other in the assistive web. The Theresults results demonstrated demonstrated the efficiencythe efficiency of our of oursystem. system. They They are encouragingare encouraging to develop to develop other other technologiesThe results demonstrated that can provide the flexibleefficiency and of personalized our system. learning They are and encouraging education experiencesto develop inother the assistiveassistive technologies technologies that thatcan canprovide provide flex ibleflex ibleand andpersonalized personalized learning learning and andeducation education mathematicalassistive technologies sciences by that addressing can provide the unique flex andible diverse and needspersonalized of learners learning with visual and impairments. education In future work, we aim to develop a complete e-learning platform for mathematical sciences, dedicated to visual impairments and blindness. This platform would contain voice assistance to automatically read a web page by simple hover on, and the start of a voice search. We also aim to develop mathematical braille converters that support all braille codes.
Author Contributions: All authors contributed equally to this work. Funding: This research received no external funding. Acknowledgments: The authors would like to thank the anonymous referees for their fructuous remarks which improve the quality of our paper. Conflicts of Interest: The authors declare no conflict of interest.
References
1. Ma´ckowski,M.S.; Brzoza, P.F.; Spinczyk, D.R. Tutoring math platform accessible for visually impaired people. Int. J. Comput. Biol. Med. 2018, 95, 298–306. [CrossRef][PubMed] 2. Beck-Winchatz, B.; Riccobono, M.A. Advancing participation of blind students in Science, Technology, Engineering, and Math. Int. J. Adv. Space Res. 2008, 42, 1855–1858. [CrossRef] 3. Nemeth, A. The Nemeth Braille Code for Mathematics and Science Notation 1972 Revision; American Printing House for the Blind: Louisville, KY, USA, 1972. 4. RNIB. Braille Mathematics Notation 1987; Royal National Institute for the Blind: Peterborough, UK, 1987. 5. Waud, D. What Is Next? Available online: http://www.tex.uk/cgi-bin/texfaq2html (accessed on 5 May 2018). Symmetry 2018, 10, 547 15 of 16
6. Ausbrooks, R.; Buswell, S.; Dalmas, S.; Devitt, S.; Diaz, A.; Hunter, R.; Smith, B.; Soiffer, N.; Sutor, R.; Watt, S. Mathematical Markup Language. (MathML) version 2.0. 2000. Available online: https://www.w3.org/ Math/ (accessed on 23 October 2018). 7. Asebriy, Z.; Raghay, S.; Bencharef, O.; Kaloun, S. A semantic approach for mathematical expression retrieval. IJACSA 2016, 7, 190–194. [CrossRef] 8. Wongkia, W.; Naruedomkul, K.; Cercone, N. i-Math: Automatic math reader for Thai blind and visually impaired students. Int. J. Comput. Math. Appl. 2012, 64, 2128–2140. [CrossRef] 9. Vlad, P.C.; Tanja, S.; Lars, B.C. Braille Math Extension to RoboBraille: A Universal Software Solution for Converting Math into Braille. In Proceedings of the ICCHP 2016, Linz, Austria, 13–15 July 2016; pp. 15–18. 10. Christensen, L.B.; Klaus, J. Multilingual two-way braille translation. In Proceedings of the ICCHP ‘96 Proceedings of the 5th International Conference on Computers Helping People with Special Needs, Linz, Austria, 17–19 July 1996. 11. LibLouis. Available online: http://www.liblouis.org (accessed on 1 October 2018). 12. fMath. Available online: http://www.fmath.info (accessed on 28 September 2018). 13. UMCL. Available online: https://sourceforge.net/projects/umcl (accessed on 23 October 2018). 14. Stanley, P.B.; Karshmer, A.I. Translating MathML into Nemeth Braille Code. In Proceedings of the 10th International Conference, ICCP 2006, Linz, Austria, 11–13 July 2006; pp. 1175–1182. 15. Duxbury Systems. Available online: http://www.duxburysystems.com/ (accessed on 27 September 2018). 16. Amaya XML Editor. Available online: https://www.w3.org/Amaya/ (accessed on 20 October 2018). 17. Gardner, J.A. The Lean Math Accessible MathML Editor. In Proceedings of the ICCHP 2014, Paris, France, 9–11 July 2014; Part I, LNCS 8547; pp. 580–587. 18. Soiffer, N. A Study of Speech Versus Braille and Large Print of Mathematical Expressions. In Proceedings of the ICCHP 2016, Linz, Austria, 13–15 July 2016; Part I, LNCS 9758; pp. 59–66. 19. Soiffer, N. Browser-independent Accessibilité Math. In Proceedings of the W4A 15, Florence, Italy, 18–20 May 2015. 20. Salamonczyk, A.; Pawlowska, J.B. Translating of MathML Formulas to Polish Text, Example Applications in Teaching the Blind. In Proceedings of the 2nd International Conference on Cybernetics (CYBCONF), Gdynia, Poland, 24–26 June 2015. 21. Liska, M.; Sojka, P.; Ruzicka, M. Combining text and Formula Queries in Math Information Retrieval. In Proceedings of the 24th ACM International Conference Son Information and Knowledge Management (CIKM 2015), New York, NY, USA, 19–23 October 2015. 22. Aizawa, A.; Kohlhase, M.; Ounis, I.; Schubotz, M. NTCIR-11 Math-2 Task Overview. In Proceedings of the 11th NTCIR Conference on Evaluation of Information Access Technologies, Tokyo, Japan, 9–12 December 2014; pp. 88–98. 23. Liska, M.; Sojka, P.; Ruzicka, M. Similarity Search for Mathematics: Masaryk University team at the NTCIR-10 Math Task. In Proceedings of the 10th NTCIR Conference, Tokyo, Japan, 18–21 June 2013. 24. Hu, X.; Gao, L.; Lin, X.; Baker, J.B. WikiMirs: A Mathematical Information Retrieval System for Wikipedia. In Proceedings of the 13th ACM/IEEE-Cs Joint Conference on Digital Libraries ACM, Indianapolis, IN, USA, 22–26 July 2013; pp. 11–20. 25. Egomath. Available online: http://egomath.projekty.ms.mff.cuni.cz/ (accessed on 18 June 2018). 26. Hambasan, R.; Kohlhase, M.; Prodescu, C. MathWebSearch at NTCIR-11. In Proceedings of the 11th NTCIR Conference, Tokyo, Japan, 9–12 December 2014. 27. Lin, X.; Gao, L.; Hu, X.; Tang, Z.; Xiao, Y.; Liu, X. A mathematical retrieval system for formulae in layout presentations. In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR 14), Gold Coast, QLD, Australia, 6–11 July 2014; pp. 697–706. 28. Nguyen, T.T.; Cheung, S.H.; Chang, K. A lattice-based approach for mathematical search using Formal Concept Analysis. Int. J. Expert Syst. Appl. 2012, 39, 5820–5828. [CrossRef] 29. Carpineto, C.; Romano, G. Order-theoretical ranking. Int. J. Am. So. Inf. Sci. 2000, 51, 587–601. [CrossRef] 30. Cheung, K.S.K.; Vogel, D. Complexity reduction in lattice-based information retieval. Int. J. Inf. Retr. 2005, 8, 285–299. [CrossRef] 31. Hao, S.; Shi, C.; Niu, Z.; Cao, L. Concept coupling learning for improving concept lattice-based document retrieval. Int. J. Eng. Appl. Artif. Intell. 2018, 69, 65–75. [CrossRef] Symmetry 2018, 10, 547 16 of 16
32. Yang, S.Q.; Tian, X.D.; Yu, B.T. A maintenance algorithm of FDS based mathematical expression index. In Proceedings of the International Conference on machine learning and Cybernetics, Lanzhou, China, 13–16 July 2014. 33. Schellenberg, T.; Yuan, B.; Zanibbi, R. Layout-Based Substitution Tree Indexing and Retrieval for Mathematical Expressions. In Proceedings of the Document Recognition and Retrieval XIX, Burlingame, CA, USA, 22–26 January 2012. 34. BraMaNet. Available online: http://handy.univ-lyon1.fr/MH/bramanet/bramanet.php (accessed on 24 June 2018). 35. Suzuki, M.; Kanahori, T.; Ohtake, N.; Yamaguchi, K. An Integrated OCR Software for Mathematical Documents and Its Output with Accessibility. In Proceedings of the International Conference on Computers for Handicapped Persons, Paris, France, 7–9 July 2004; Springer: Berlin/Heidelberg, Germany, 2004; pp. 648–655. 36. Komada, T.; Yamaguchi, K.; Kawane, F.; Suzuki, M. New Environment for Visually Disabled Students to Access Scientific Information by Combining Speech Interface and Tactile Graphics. In Proceedings of the International Conference on Computers for Handicapped Persons, Linz, Austria, 11–13 July 2006; Springer: Berlin/Heidelberg, Germany, 2006; pp. 1183–1190. 37. Duda, R.O.; Hart, P.E.; Stork, D.G. Pattern Classification, 2nd ed.; John Wiley & Sons: Hoboken, NJ, USA, 2000; p. 680. 38. Accessible Equation Editor. Available online: http://accessibility.pearson.com/aee/ (accessed on 7 July 2018). 39. Maillo, J.; Ramírez, S.; Herrera, I.T.F. kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data. Int. J. Knowl.-Based Syst. 2017, 117, 3–15. [CrossRef] 40. Pedronnette, D.C.G.; Gonçalves, F.M.F.; Guilherme, I.R. Unsupervised manifold learning through reciprocal kNN graph and Connected Components for image retrieval tasks. Int. J. Pattern Recogn. 2018, 75, 161–174. [CrossRef] 41. Zhan, Y.; Liu, J.; Gou, J.; Wang, M. A video semantic detection method based on locality-sensitive discriminant sparse representation and weighted KNN. Int. J. Visual Commun. Image Represent. 2016, 41, 65–73. [CrossRef]
© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).