End-To-End Solution for Accessible Chemical Diagrams
Total Page：16
File Type：pdf, Size：1020Kb
End-to-end Solution for Accessible Chemical Diagrams Volker Sorge, Mark Lee Sandy Wilkinson School of Computer Science School of Education University of Birmingham, UK University of Birmingham, UK {V.Sorge|M.G.Lee}@cs.bham.ac.uk [email protected] ABSTRACT relatively simple diagram. Since the majority of images and Chemical diagrams are an important means of conveying in- diagrams on the web use raster-based image formats (e.g. formation in chemistry and biosciences to students, starting gif, png, and jpeg), magnification tools also struggle with as early as secondary school. But even in electronic teaching dealing with diagrams since magnification does not propor- material, diagrams are commonly given as bitmap graphics tionally increase resolution leading to a loss of image quality, leaving them inaccessible for visually impaired learners. We which, in practice leaves most diagrams completely inacces- present an end-to-end solution to making these diagrams sible to visually impaired users. Web accessible, by employing image analysis solutions to There have been a number of approaches to make scien- recognise and semantically analyse diagrams, and by regen- tific diagrams accessible. In particular, for chemical dia- erating them in a format that makes them amenable to assis- grams a number of tools have been built to support visually tive technology. We provide software tools that allow read- impaired users in editing, reading and exploring molecules ers to interactively engage with diagrams by exploring them (see next section for details). The main problem with these step-wise and on different layers, enabling aural rendering approaches is that they require both authors and readers to of diagrams and their individual components together with use specialist software to create and read diagrams, possi- highlighting and magnification to assist readers with low vi- bly also restricting them to particular platforms only, which sion or learning difficulties. Our technology builds on open reduces their effectiveness in practice. standards, supporting a number of computing platforms, We solve this problem for the particular case of chemi- browsers, and screen readers, and is extensible to diagrams cal diagrams by creating a workflow that bridges the gap in other STEM subjects. from images to accessible diagrams in a way that neither relies on authors to produce images in some special format nor requires readers to familiarise themselves with a new be- Keywords spoke tool. Instead, we apply an image analysis system to Accessible Diagrams, Image Transformation, Chemistry bitmap images to extract chemical meaning from diagrams. We then transform the images into Scalable Vector Graph- 1. INTRODUCTION ics (SVG) and use cheminformatics tools to create an un- derlying semantic representation in a Shadow DOM, which Visually impaired learners represent a sizeable minority enables the automatic generation of meaningful descriptions of users of scientific material. For example, it is estimated for components of the diagram. A JavaScript tool allows that there are 25,000 visually impaired children and young the user to \step through" the image via speech with syn- adults in England and Wales who require specialist educa- chronised highlighting and/or magnification in an ordinary tion support. More than 60% of this group are educated in web-browser using the semantic markup together with WAI- mainstream schools and are often without specialist techni- ARIA elements. Our experiments and initial user tests have cal equipment, such as high definition magnification tools indeed shown that our approach can cater to a variety of and braille embossers. different assistive technology platforms and users. The majority of users rely on software-based assistive tech- We therefore see the main contribution of our work in pro- nology such as screen readers and magnifiers. While to a cer- viding the first end-to-end solution to creating accessible di- tain degree, text can be handled using existing screen readers agrams, starting with the analysis of bitmap images and gen- and related tools, diagrams are often completely inaccessible erating explorable diagrams that seamlessly integrate with since even if alternative text is provided, it in no way com- assistive technology solutions already familiar to a reader. pares with the richness of information provided by even a This paper is structured as follows: We discuss related Permission to make digital or hard copies of all or part of this work for work in the next section before giving the necessary back- personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies ground on chemical diagrams and different forms of their bear this notice and the full citation on the first page. To copy otherwise, to representation in Sec. 3. We then present an overview of republish, to post on servers or to redistribute to lists, requires prior specific our process that turns diagrams from bitmap format into permission and/or a fee. Request permissions from [email protected] fully accessible diagrams in scalable vector graphics (SVG) W4A 2015, May 18 - 20, 2015, Florence, Italy in Sec. 4. Sections 5{8 then describe the single steps in Copyright is held by the owner/author(s). Publication rights licensed to more detail, in particular, the image analysis, the genera- ACM ACM 978-1-4503-3342-9/15/05 ...$15.00 tion of annotated SVG diagrams, their semantic enrichment http://dx.doi.org/10.1145/2745555.2746667. and the support for interactive engagement with diagrams we transform these results into an XML structure that con- via a browser front end. We illustrated the latter with an ex- nects this information to the SVG diagram. This allows us tensive example in Sec. 9 before concluding by summarising to use standard web technology, like JavaScript and WAI- results of our experiments and user studies in Sec. 10. ARIA, to implement reader interaction with the diagram, which is platform, browser and screen reader independent, 2. RELATED WORK thus offering users a barrier-free reading experience. There have been a number of approaches to make scientific diagrams accessible, using a variety of means such as sonifi- 3. COMMUNICATING MOLECULES cation, touch exploration and haptic feedback. [3] presents In this section we present some background on how chem- general guidelines for the presentation of tables and line ical diagrams are commonly communicated in teaching and graphs using sonification and speech, while [21] presents a scientific literature. There is naturally a wide gap between multimodal tool that allows blind users to create and explore how molecules are taught to students in secondary school line graphs and charts via the Web using haptic feedback and how they are communicated to experienced chemist. and sonification. This idea is extended to three-dimensional From the point of view of creating accessible diagrams au- representations of graphs in [5, 4], which presents an author- tomatically, it is of course not known in advance who the ing and teaching tool for the creation accessible graphs and target audience will be and how much detail a reader might their exploration via highlighting and speech using touch need, to comprehend a particular diagram. Therefore, it is screens. While these approaches are suitable for relatively necessary to make diagrams accessible on multiple levels of simple structures with homogeneous layouts, they can not be granularity as well as cater for readers of different expertise easily adapted for semantically richer graphs such as chemi- by producing several alternative audio descriptions. cal diagrams. Consequently a number of tools and environ- We introduce the different types of diagrammatic repre- ments have been built to support visually impaired users in sentations for molecules we are working with as well as the editing, reading and exploring molecules. necessary concepts that we use to achieve our goal of ade- [19] describe various projects involving visually impaired quately describe molecules for different audiences. chemistry students entering chemical information as ASCII- based SMILES (simplified molecular input line entry specifi- 3.1 Forms of Diagrams cation) markup, which is then translated via Open Babel for There are a number of different ways in which molecules use in commercial screen readers, embossers and 3D print- can be represented diagrammatically, varying mainly in the ers. The use of SMILES allows visually impaired students to explicitness in which the chemistry is presented. The par- avoid the need of standard graphical user interfaces for draw- ticular choice of diagrams depends often on the target au- ing chemical molecules but does require prior knowledge of dience of a document. While advanced scientific textbooks basic chemistry and of the SMILES markup language. and publications usually go for the most abstract or most [1] presents the Kekul´esystem that reads molecular struc- space-saving representation, introductory texts in secondary tures from Chemical Markup Language files, which are used schools use the most explicit representations possible. How- to generate graph-based structures which can be explored ever, in general, secondary School teaching material presents by a combination of keyboard input and speech synthesiser all different types of possible representations. Consequently, output. A key aspect of the system is that it infers and it is important that a system like ours can work with the makes explicit chemical structure which is only implicit in whole variety of diagrams. the original CML markup. This allows the user to fully ex- Figure 1 shows the three main categories of diagrams, each plore any molecule at different levels, for instance by moving representing the molecule commonly known as Aspirin. Dia- between adjacent atoms or higher level features. grams given in displayed formula are the most detailed ones, [12, 7] present NavMol, a bespoke system that allows VI where all atoms are given explicitly by their chemical sym- users to edit and navigate molecular structures.