Haptic Interfaces for Individuals with Visual Impairments

Issue 07 – 2018 Journal –Peer Reviewed

BENJAMIN VERCELLONE Missouri Department of Social Services Haptic Interfaces [email protected] JOHN SHELESTAK for Individuals with Department of Biological Sciences, Kent State University [email protected] Visual Impairments YASER DHAHER Department of Mathematics, Kent State University [email protected] & ROBERT CLEMENTS Department of Biological Sciences, Kent State University ABSTRACT [email protected] Vision enables a person to simultaneously perceive all parts of an object in its totality, relationships to other objects for scene navigation, identifcation and planning. These types of perceptual processes are especially important for com- prehending scenes in gaming environments and virtual representations. While individuals with visual impairments may have reduced abilities to visualize objects using conventional sight, an increasing body of evidence indicates that compensatory neural mechanisms exist whereby tactile stimulation and additional senses can activate visual pathways to generate accurate representations. The current thrust to develop and deploy virtual and augmented reality systems capable of transporting users to diferent worlds has been associated with a push to include additional senses for even greater immersion. Clearly, the inclusion of these additional modalities will provide a method for individuals with visual impairments to enjoy and interact with the virtual worlds and content in general. Importantly, adoption and development of these methods will have a dramatic impact on inclusion in gaming and learning environments, and especially in relation to science, technology, engineering, and mathematical (STEM) education. While the development, standardization and implementation of these systems is still in the very early stages, gaming devices and open source tools are continuing to emerge that will accelerate the adoption and integration of the novel modes of computer interaction and provide new ways for individuals (with visual impairments) to experience digital content. Here we present the rationale and benefts for using this type of multimodal interaction for individuals with visual impairments as well as the current state of the art in haptic interfaces for gaming, education and the relationships to enhanced end user immersion and learning outcomes. As these technologies and devices are adopted and evolve, they are poised to have a dramatic impact on entertainment, education and quality of life of individuals with visual impairments.

KEYWORDS: Haptic media, Special education, Special needs, Video games, Visual impairments

Benjamin Vercellone, John Shelestak, Yaser Dhaher, & Robert Clements https://www.gamejournal.it/07_vercellone-shelestak-shaher-clements 75 INTRODUCTION Successfully navigating and interacting with video game environments is natu- rally tied to the ability of the player to visually discern objects on the display and understand spatial relationships between these objects. As such, the realm of gaming has been heavily focused on increasing rendering quality to provide greater acuity and immersion for the user. This has undoubtedly pushed both the hardware and sofware forward, however, little of this development efort has been focused on how to convey in game elements without the use of vision, specifcally for gamers who are visually impaired. Touch based methods have been classically used by the blind as a substitution for vision, with the most obvious example being the braille method of tactile writing whereby a system of raised bumps organized into six block cells, typically on embossed paper, are used to represent characters and punctuation. Invented by Louis Braille (Duxbury Systems, 2017) in the early 19th century this method has stood the test of time afrming the utility of touch, or haptic, sensations as an alternate method to convey visual concepts to the blind. In the realm of computer interaction for gaming and visualization of complex imagery, haptics have been integrated for many years via “rumble pads” and force-feedback devices providing users the ability to crudely physically sense digital events (Orozco, 2012). While these systems ofer a level of multi-modal interaction for the sighted user, they have had limited use for seeing impaired individuals with most games for the visually impaired based around the use of sound for cue identifcation. However, tactile games for individuals with visual impairments based of popular WII console games (VI-bowling and VI-tennis) have been successfully developed using vibration to indicate ball timing, controller position and object placement (Tony, 2010). The crudeness of these cues and the difculty in iso- lating them spatially limits the vibration only approach. It does afrm the prin- cipal and ability of using tactile feedback to accurately represent spatial concepts within the gaming realm. This would be especially true for haptic interfaces referenced to real world space with rapid refresh-rates that could provide much of the information that a sighted gamer would get visually. While individuals with visual impairments may have reduced abilities to visualize objects using conventional sight, an increasing body of evidence indicates that compensatory mechanisms exist whereby tactile stimulation and additional senses can activate the visual system to generate accurate spatial representations. Indeed, reading braille is known to activate the visual cortex (Burton, 2006) of the brain in the blind (a region that typically codes visual information) and refects the recoding of tactile sensations into “visual” signals. Studies also indicate that blind participants are typically able to better discriminate 3D shapes via tactile and haptic interaction (Norman, 2011) than sighted persons. Research evidence indicates the blind have diferent patterns of activation in the brain during haptic tasks (Roder, 2007) as well as during vibrotactile stimulation (Burton, 2004). Moreover, large-scale changes manifested in both structural and functional diferences in brain connectivity indicate that the changes are not limited to the visual areas of the brain but refect large changes in associated areas and “visual” perception as a whole. The idea of perceptual learning and substitution of sensory signals (where visual input is substituted for touch) can aford the ability of “sight” to subjects who are physically incapable of do- ing so. The remarkable capacity of the human brain to reorganize itself results in long lasting changes. This type of reorganization of visual pathways afrms the utility of a haptic approach to represent virtual environments to individuals with visual impairments. This will not only provide a means for additional brain reorganization but also that inherent diferences in their brain structure suggests that perceptual learning will be more benefcial in this population with a dramatic impact on in game orientation and perception. With the advent of new devices facilitating the generation of complex spatial and temporal representations of real world and virtual scenes, as well as the incorporation of tactile devices that allow physical interaction with computer generated imagery has emerged a novel platform for augmenting seeing impaired interfaces. Coupling this with evidence that individuals with visual impairments are able to acquire spatial knowledge using ancillary neural pathways via harnessing additional senses indicates that this is a promising and potentially game changing avenue to enhance user engagement and immersion. Here we detail the current state of haptic technologies as it relates to using these methods for encoding spatial information to individuals with visual impairments for both gaming and education. We begin with a discussion of the available hardware, and then we propose how these technologies could be used to convey in game information and learning as well as identify barriers and potential solutions. Finally, we address future developments and next steps for dealing with these innovations in the long term.

NEW TECHNOLOGIES AT STAKE

HAPTIC MEDIA Simple haptic feedback has been a part of gaming now for many years, but newer technologies are allowing for a larger range of haptic sensations across diferent digital experiences. Newer controllers, such as the Steam controller have improved on this feedback, allowing for a greater range of in-game information through fner control of the haptic output including haptic information on speed, boundaries, textures, and in-game actions (Steam Store, 2015). Microsof is developing various haptic controllers to achieve various sensations such as texture, shear, variable stifness, as well as touch and grasp, to enhance the level and variety of stimulation provided by a VR experience (Strasnick, 2018; Whitmire, 2018). Haptic styluses ofer a diferent handheld experience via force feedback instead of vibrotactile feedback and can be used to interact with objects, or as a physical interface for more advanced simulations (Stein- Issue 07 – 2018 Haptic Interfaces for Individuals with Visual Impairments

berg, 2007). Further, advances in wearable haptics; gloves, vests and full body suits have also expanded the potential experience of video games. Gloves allow for diferent actuators to be woven into the fabric or attached as an exoskeleton to achieve both vibro-tactile and force feedback (Virtual Motion Labs, 2018; VRGluv, 2017). The force feedback from the exoskeleton can defne the edges of hard surfaces while vibrotactile actuators give information surface texture. In addition to exoskeleton-based force feedback, some gloves feature pneumatic actuators that can provide pressure directly to the skin, creating a realistic feeling of skin displacement. Beyond gloves, the same technology is being applied to wearable vests and full body suits to create the most complete and immersive feel possible. Vests allow for targeted sensation to the body depending on where the stimulation is received in a game or VR environment (Kor-Fx, 2014; Hardlight VR, 2017; Woojer, 2018). Electrical nerve and muscle stimulation is also used for haptic feedback and also provides temperature control for changing virtual environments (Teslasuit, 2018). One practical problem with VR gaming is that moving around a space while wearing a headset can be difcult and dangerous. Haptic or omni-directional treadmills help fx this problem by anchoring the user to an area while still allowing for unfettered movement in any direction. More commonplace treadmills are also integrating haptic force feedback to give the sensation of moving through a real place while running on a treadmill (Nor- dicktrack, 2018). Indeed, a great deal of focus has also been spent on developing haptic touchscreens and incorporating this hardware into experiences for blind users. Newer haptic feedback seeks to alter the interaction between screens and fngers to create a more diverse sense of touch such as friction or texture simulation (Tanvas, 2018). Other developments include the manipulation of ultra- sound to achieve tactile stimulation (Hap2U, 2018) with potential to create 3D surfaces in midair. Clearly as these new haptic technologies evolve and standardize the impact on immersive gaming and blind user interaction will be broad.

VISUALLY-IMPAIRED SPECIFIC DEVICES A number of tactile hardware implementations specifcally designed for individuals with visual impairments also exist. The Graphiti is tactile device consisting of an array equidistant pins with variable height and can be connected to a smartphone to control the height of the pins to represent text or camera images (Graphiti, 2018). The variable pin height could conceivably be used to represent limited information regarding the Z-axis, color, or whatever else someone may reasonably desire, such as an in game map. The system also ofers a way to zoom in and out which is important from a tactile perspective, since human fngers cannot perceive and interpret information as minute and dense as the human eyes. The BrainPort V100 (Wicab, 2018) provides real world raw geometric information based on wearable camera input via a dongle inserted in the mouth that provides electro-tactile stimulation to the tongue. Bubble like

Benjamin Vercellone, John Shelestak, Yaser Dhaher, & Robert Clements https://www.gamejournal.it/07_vercellone-shelestak-shaher-clements 78 Issue 07 – 2018 Haptic Interfaces for Individuals with Visual Impairments

patterns convey spatial information on the tongue and the user learns about shape, direction, relative distance, and size. BLITAB have successfully created a tactile braille tablet that utilizes “tixels” that dynamically rise above the surface of the display and can convert text to braille (Blitlab, 2018). The hardware also provides audio cues for the user but is limited in its ability to convey graphics, and objects in 3D space. BlindPAD ofers a similar technology consisting of an array of electromagnetic “taxels” with 8mM pitch and a rapid refresh rate (Blindpad, 2018). The interactive technology is able to display graphics, maps and symbols as a tactile representation. This system has been successfully used in the education realm as well as can enhance users with visual impairments sense of space and their knowledge of unknown places. These latter points would be especially helpful if used for gaming to enhance blind users’ ability to navigate and investigate virtual environments and worlds. While these systems would clearly be benefcial for reading braille and other 2D applications, it would appear spatial haptics currently in development for VR applications would be capable of creating 3D touch sensations closer to real world experiences. The enhancement in the variety of tactile stimulation available only deepens the experiences and the value these technologies can provide. It is clear that the development of the specifc devices for individuals who are blind have been based around familiar technologies typically used to convey text; raised bumps on a fat surface. Nevertheless, creation of a common interface for these devices to video game environments or primitives (potentially by direct access to the graphics depth bufer) would create a new platform for both education and entertainment that would defnitely empower its users.

AUGMENTING VISUALLY-IMPAIRED GAMING This explosive development in hardware opens new avenues to convey spatial and game specifc information in real time to gamers who have visual impairments. We have identifed areas to specifcally exploit these technologies to support in-game navigation, information extraction and user interaction. While some methods are specifc to a certain device, the same principles would work at some scale using any haptic feedback from rumble pack in a controller to a fully-fedged VR vest.

1. Navigation: Fundamental to almost all gaming experiences is the requirement to correctly navigate and understand virtual environments. Tactile maps, proximity cues and new in game tools could enhance blind gameplay.

a. Tactile maps used by the blind have been shown to augment navigation when used during and prior to route traversal. However, the specifc touch cues must be carefully considered when represent- ing diferent types of information (roads, water, grass). Undoubt-

Benjamin Vercellone, John Shelestak, Yaser Dhaher, & Robert Clements https://www.gamejournal.it/07_vercellone-shelestak-shaher-clements 79 Issue 07 – 2018 Haptic Interfaces for Individuals with Visual Impairments

edly, a tactile approach to facilitate navigation of gaming environments would also ofer similar benefts to the real-world case. It has been shown that the haptic approach when coupled with additional modalities such as audio can provide a method for individuals/gamers with visual impairments to generate cognitive maps of virtual environments using multimodal cues (Lahav, 2012), this is one of the major challenges for navigating gaming environments. Gaming environments (top down map) represented on a connected cell phone or other tactile tablet device (listed above) and interacted with via touch (vibration, tactile pixels) would provide an inexpensive solution. Here, haptic cues would orient the user using published frame- works for providing haptic access to 2D maps (Kostopoulos, 2007) to enhance knowledge and traversal of gaming environments.

b. Proximity cues. As well as knowledge of entire gaming maps, information about discrete and proximal in game cues (walls, surfaces, barriers) is essential to spatial awareness, navigation and gameplay. Haptic outputs (within a vest for example)) at discrete regions over the body activated by cues extracted from the in-game display would rapidly provide feedback about in game surfaces. Using this strategy, a wall detected on the lef side of the in-game player would activate a haptic sensor on the lef side of the player. Haptic output modulation (such as vibration frequency and amplitude) would code for object distance and textures using proximity sensors as has been shown successful in physical applications (Keys, 2015).

c. New tools. Here we propose the development of new methods for users to probe spatial environments. Specifically creating methods for users to digitally explore regions of the virtual environment and providing targeted haptic feedback. Specific tools such as a haptic sonar, floor probe and virtual compass whereby the users can target specific areas of the screen and be provided feedback regarding structural elements and orientation. The proposed haptic sonar would probe a customizable portion of the environment and activate an output (a rumble pack for example) if an object is within the frustum of the probe. Floor probes would more specifically sample areas directly in front of the user (floor) within the game and indicate hazards (water, textures, holes) by varying haptic outputs to facilitate locomotion. Orientation is key to navigating any environment, and the creation of a haptic compass to identify gaze direction in the form of a necklace (or array of actuators around the body) where feedback rotates around the ring and indicates north. These three strategies have been shown to help blind navigation in

Benjamin Vercellone, John Shelestak, Yaser Dhaher, & Robert Clements https://www.gamejournal.it/07_vercellone-shelestak-shaher-clements 80 Issue 07 – 2018 Haptic Interfaces for Individuals with Visual Impairments

real-life applications (Quest, 2018; Choiniere, 2017; Visell, 2009) suggesting they may beneficial for navigating virtual environments.

2. In-Game Information: Layered upon game scene spatial information are characters, entities, game packs, enemies and other relevant dynamic game information. In order to provide a separate stream of information a distinct and generalized in game warning and information system is proposed. The information system should convey location, type of entity, and utility on a scale (good, indiferent, bad) to the user. Notifcations would serve to provide spatial output to the user in the form of actuators at geographically separate points (around the body to indicate position for a vest or vibrator array for a glove). The nature of actuator output would convey the type of in-game entity at the specifc location. For example, a health pack would initiate a series of short low intermittent vibration pulses but an enemy would cause fully maximal and continuous vibration at the actuator. Utilizing strength of haptic output to represent danger (good, bad, indiferent) and frequency to represent type of entity is a potentially viable solution.

3. User Interaction: A major factor contributing to immersive gaming is the ability to interact with digital scenes and in-game characters. Diferent to navigation and in game cue identifcation, interaction typically includes feedback from the entity, such as dialog, a response (enemy fres) or change in the entity (health pack disappears). At least two types of interaction are possible, the frst is scene object interaction (e.g. pick up a key or a health pack) and the second is dynamic entity interaction (engage an enemy, interface with a non-playing, or friendly, character). With the former, interaction is relatively passive and layering this information on top of the navigational cues above makes sense whereby a separate tone (or haptic output) indicates the passive entities response. However, when engaging enemies, or other characters, active interaction requires additional feedback and dynamic responses. Here, a “lock in” mode, whereby all haptic resources are dedicated to interacting with the dynamic entity would provide directed attention to the entity. Under this system, the in-game noti- fcation system would identify an entities utility to the user via haptic cues (very good or very bad) and the user can initiate interaction, or “lock in” all haptic resources, to track the object and map it to spatial actuators and convey entity specifc information to the user (hostil- ity, temporal changes, weapon activity). Once the user “un-couples” attention (or the character disappears from the area), the full array of scene probing once again is reestablished. This method is designed to

Benjamin Vercellone, John Shelestak, Yaser Dhaher, & Robert Clements https://www.gamejournal.it/07_vercellone-shelestak-shaher-clements 81 Issue 07 – 2018 Haptic Interfaces for Individuals with Visual Impairments

“silence” inconsequential haptic information during times of critical engagement to increase clarity and resolution of the engagement.

POSSIBLE LIMITATIONS Many of the approaches above have been shown to beneft individuals with visual impairments interactions with real-world scenes and as such provide a potentially benefcial framework for virtual scenarios. However, caution must be taken when trying to implement such a layered and complex haptic system. The ability to multiplex diferent signals (using amplitude/frequency modulation as well as additional modalities) and silence less relevant information (via “locking in” attention and feedback) is key to the successful application of a system with a relatively limited set of physical outputs compared to inputs. Care should be taken to reduce overload to the user, potentially accomplished by using additional modalities to convey specifc notifcations or events (a bell to indicate a health pack). Haptic resolution (dynamic response and spatial con- straints) is also a major factor that should be considered when localizing feedback systems on the body. It is known that our ability to discriminate between two points varies over position on the skin (Purves, 2001) as such placement of haptic actuators is of great importance. These numbers vary from a 5mm threshold on the thumb to 45 mm two-point discrimination threshold along the shoulder, suggesting that haptic placement as well as device capabilities should be considered when localizing haptic sources on the body.

POTENTIAL LEARNING APPLICATIONS Importantly, adoption and development of these methods will also have a dramatic impact on inclusion in gaming and learning environments, especially in relation to STEM education. Understanding spatial properties including shape, size, distance and orientation is foundational and essential to developing spatial thinking skills and understanding many topics including algebra, trigonom- etry, calculus, chemistry, physics, biology and higher mathematics. Typically, students use sight to internalize geometric and spatial properties, however, the students with visually impairments need to have a diferent mode of experienc- ing this. While the development, standardization and implementation of these systems is still in the very early stages, gaming devices and open source tools are continuing to emerge that will accelerate the adoption and integration of the novel modes of computer interaction and provide new ways for students to experience digital content. The use of tactile graphics which employ the sense of touch rather than vision to convey spatial properties have been used to de- liver mathematics instruction (Brawand, 2016), biology (Reynaga-Pena, 2015), chemistry (Copolo, 1995) and physics (Holt, 2018). 3D printing technology is also a valid method for delivering science content (Grice, 2015; Kolitsky, 2014) to blind students but clearly sufers from an inability to modify representations that are typically static, with limited use in the gaming sphere. Tools are be-

Benjamin Vercellone, John Shelestak, Yaser Dhaher, & Robert Clements https://www.gamejournal.it/07_vercellone-shelestak-shaher-clements 82 Issue 07 – 2018 Haptic Interfaces for Individuals with Visual Impairments

coming to emerge that standardize the procedure of generating tactile graphical representations (Pather, 2014); however, more are clearly required to broaden adoption of these practices. While many of these early studies incorporated printed static models or embossed paper, with the explosion of hardware and sofware computer haptics are increasingly being used to convey spatial information using force feedback in real world space but further development of both platforms and content are integral to expanding. With support for vibration output on many tablet and cellphone devices it is one of the simplest ways to incorporate tactile response is via activating this output (Awada, 2013; Dia- gram Center, 2017) and successfully conveys mathematics concepts (Cayton- Hodges, 2012), graphics and additional STEM content (Hakkinen, 2013) but not without limitations (Klatzky, 2014). Most STEM based studies have relied on surface haptic approaches while less have incorporated spatial haptic feedback systems (Nikolakis, 2004; Lahav, 2012; Evett, 2009) but it is known to facilitate knowledge acquisition of 3-dimensional objects (Jones, 2005). By incorporating 3-dimensional space co-registered to the real world into the haptic workfow clearly provides a more fundamentally “real” experience. Further, it results in an explosive increase in the potential amount of information that can be conveyed to the user or gamer. It is clear that the use of additional senses can help provide cognitive representations of environments and spatial structures and as VR gaming technologies evolve it is clear they need to be incorporated into applications to augment seeing impaired learning in STEM.

FUTURE DIRECTIONS Our lab and others are currently investigating new directions in haptic interaction for navigating physical and virtual environments as well as identifying geometric primitives using multimodal cues and triggers. Recent studies in our lab have been focused on evaluating tactile (touch) feedback for the instruction of students who are blind or visually impaired. Pilot studies using the GeoMagic device have afrmed the utility of the method where blind users are able to autonomously recognize designed representations and specifc shapes in only a few seconds. This suggests a clear beneft for teaching geometry and basic mathematical principles, something we are currently exploring. Coupling this with the fact that our infrastructure can both represent dynamic objects that change over time and new models can be simply built or downloaded. Our excitement for the method is hampered by the fact that the technology does not simulate the physical sense of touch in a natural or indeed accessible way. We acknowledge that devices do exist that can simulate touch via spatial force feedback but cost and clearly prohibits the use and mass rollout of such devices. This is coupled with the fact that sofware is extremely limited and requires extensive programming knowledge to be extended. As such, using of the shelf technology we have also developed a hand tracking system with vibrational feedback for touching virtual objects. Using vibration motors attached to each

Benjamin Vercellone, John Shelestak, Yaser Dhaher, & Robert Clements https://www.gamejournal.it/07_vercellone-shelestak-shaher-clements 83 Issue 07 – 2018 Haptic Interfaces for Individuals with Visual Impairments

fngertip the user is able to probe a virtual object in space and sense 3D objects (Dhaher, 2017). We intend to evaluate the technology and develop lessons for the students with visual impairments by deploying these instruments to learn foundational spatial concepts. In addition, we are actively developing interfaces for probing external and virtual environments with tactile feedback for navigation and virtual tele-presence applications. Using scene depth imaging and physical triggers located at strategic points on the body, users can continuously physically sense details regarding proximal objects and walls in the (virtual) environment. For example, vibration on the right side of the body indicates presence of an object in the right part of the environment. The use of more specialized “virtual” probes in development for scanning depth bufers and providing physical and multimodal feedback. Importantly, this thrust is toward the development of a common interface for navigating (gaming) environments using (virtual) environment probes with accompanied audio and touch based feedback. The end of goal of which is to create a method for probing running applications on the graphics processing unit (GPU) to map inputs and outputs to multi-modal cues (sound, vibration). As is true for the integration of any technology portability, compatibility and easy end user access to functionality is critical to adoption. The current state of haptic development is in its early stages and as such relatively disparate with most projects based on creating individual applications rather than platforms for haptic integration into existing sofware. The development of end- user friendly middleware, or other programming interfaces, to provide a means to generalize the access of both non-specifc sofware and output signals to any type of (haptic) device is critical. A number of platforms for haptic interaction and programming have emerged relatively recently but are still beyond the reach of most end users since they require programming sofware and detailed knowledge. Openhaptics Professional (3DSystems, 2018) is a commercially available programming interface that is designed to facilitate creation of a wide range of sofware incorporating haptic feedback and interaction. CHAI3D (Chai3D, 2018) is a similar open source project that is designed to be compatible with a large amount of existing hardware devices and incorporates the ability to extend support for custom or new hardware. The interface exists as a C++ simulation environment that supports a number of core functions for volume rendering, visualizing CAD fles and beyond. Both interfaces sufer from the signifcant requirement of programming knowledge as well as each project is essentially its own specifc application. A relatively mature open source library, the Virtual Reality Peripheral Network (Taylor, 2001), supported by the National Institutes of Health allows access to VR tracking devices as well as a limited number of hardware haptic implementations. VRPN aims to be a device-independent interface to virtual reality peripherals and provides a way for applications to communicate via the VRPN’s client-server architecture. The system provides an excellent way to interface hardware tracking, controllers

Benjamin Vercellone, John Shelestak, Yaser Dhaher, & Robert Clements https://www.gamejournal.it/07_vercellone-shelestak-shaher-clements 84 Issue 07 – 2018 Haptic Interfaces for Individuals with Visual Impairments

and human interface devices. However, VRPN still requires specifc modules for each hardware device as well as client module for the running sofware application. The Open Source Virtual Reality (OSVR, 2018) sofware development kit (SDK) allows developers to create applications with access to all supported VR headsets and controllers by including libraries during the creation of applications. This limits the scope of OSVR as well as denying end users with no programming knowledge the ability to efectively use the sofware. A number of solutions have emerged that have solved some of these issues and allow running sofware applications to have no knowledge of the specifc targeted hardware while still providing interaction. The open source project OpenVR (OpenVR, 2018) is a programming interface and runtime binary created by Valve (Valve, 2018) that permits many types of VR hardware (including haptic devices) to communicate with running applications. Importantly, the system is compatible with SteamVR (SteamVR, 2018) and can be installed from their portal but is still limited to compatible devices and games/applications available within the SteamVR portal. An older and apparently unmain- tained interface does allow sofware agnostic access to arbitrary hardware. The Glovepie system uses an input/output mapping system to allow end users the ability to map arbitrary controller inputs to control running applications (GlovePie, 2010). It should be noted that none of the above libraries explicitly support devices created for the blind, an issue that clearly needs to be rem- edied to move seeing impaired gaming forward. Some progress has been made towards allowing any running game or application to have access to any piece of haptic hardware, but these implementations are not broad enough in scope (minimal application/hardware support) and potentially difcult to implement. In addition, the abstraction layers are designed for interaction using the inherent interface and methods built into the controlling sofware (for example, a device can only be used to control typical in game activities such running or jumping) rather than the ability to generally probe the games 3D environment and structure. To accomplish this it will likely be required to extract information from running applications at a lower level, for example via the graphic processing units’ (GPU) depth bufer. This could provide low level access for developed environmental “probes” or tools that are implemented as a simple installable binary providing (haptic) devices access to the display lists of the running application. These digital probes could then be simply translated into pertinent outputs for (blind) users that activates and supports both the hardware specifcally designed for the seeing impaired as well as the next generation of VR haptic devices.

CONCLUSIONS Here we present the rationale and benefts for using multimodal interaction for gamers with visual impairments as well as the current state of the art in haptic interfaces for gaming, education and the relationships to enhanced end user im-

Benjamin Vercellone, John Shelestak, Yaser Dhaher, & Robert Clements https://www.gamejournal.it/07_vercellone-shelestak-shaher-clements 85 Issue 07 – 2018 Haptic Interfaces for Individuals with Visual Impairments

mersion and learning outcomes. As these technologies and devices are adopted and evolve they are poised to have a dramatic impact on the entertainment, education and quality of life of individuals with visual impairments. However, accessibility, sofware implementations and a common interface point would signifcantly help broaden the impact. As far as the latter, an underlying interface, or middleware library, built upon a standardized graphic libraries (such as OpenGL) that provides simple cross platform haptic access to depth bufers, geometric primitives and the virtual environment would simplify, enhance and expedite the incorporation of new methods of tactile computer interaction. More importantly, it would function with any running game without modif- cation. It is clear that the human eye is able to acquire much more information temporally than the somatosensory system and touch based representations of color may be arbitrary and difcult to convey but haptic representations will clearly ofer a much more natural interface for video games for individuals with visual impairments. The key will be harnessing next generation devices via a simple extraction layer to provide meaningful and compatible haptic outputs. The 2016 Disability Status report (Erickson, 2016) indicates that 2.4% (7,675,600 people) of non-institutionalized people in the US population reported a visual disability. In addition, it is estimated that there are 36 million blind persons worldwide with a further 217 million with moderate/severe impairment (Bourne, 2017) indicating that the development and integration of new haptic interfaces (both hardware and sofware) is warranted to support the entertainment and education of this population of potential gamers.

REFERENCES

3Dsystems (2018). The OpenHaptics® Developer Edition. impairment: a systematic review and meta-analysis. Lancet Retrieved from: https://www.3dsystems.com/haptics-devices/ Global Health, 5(9), pp. 888-997. openhaptics Brawand, A.C., & Johnson, N.M. (2016). Efective Methods Awada, A., Issa, Y., B. Tekli, J., & Chbeir, R. (2013). for Delivering Mathematics Instruction to Students with Visual Evaluation of touch screen vibration accessibility for blind Impairments. Journal of Blindness Innovation and Research, 6(1). users. In Proceedings of the 15th International ACM SIGACCESS Burton, H., McLaren, D.G., & Sinclair, R. J. (2006). Reading Conference on Computers and Accessibility, p. 48. Embossed Capital Letters: An fMRI Study in Blind and Blindpad (2018). Blindpad (Tactile Tablet). Retrieved Sighted Individuals. Human Brain Mapping, 27(4), pp. 325-339. from https://www.blindpad.eu/ Burton, H., Sinclair, R.J., & McLaren, D. G. (2004). Cortical Blitab (2018). WORLD’S FIRST TACTILE TABLET. Activity to Vibrotactile Stimulation: An fMRI Study in Blind Retrieved from https://blitab.com/ and Sighted Individuals. Human Brain Mapping, 23(4), pp. Bourne, R. R. A., Flaxman, S. R., Braithwaite, T., Cicinelli, 210-228. M.V, Das, A., & Jonas, J. B. (2017). Vision Loss Expert Cayton-Hodges, G. Marquez, L. van Rijn, P. Keehner, M. Group. Magnitude, temporal trends, and projections of the Laitusis, C. Zapata-Rivera, D. Bauer, M., & Hakkinen, global prevalence of blindness and distance and near vision M.T. (2012). Technology Enhanced Assessments in Mathematics

Benjamin Vercellone, John Shelestak, Yaser Dhaher, & Robert Clements https://www.gamejournal.it/07_vercellone-shelestak-shaher-clements 86 Issue 07 – 2018 Haptic Interfaces for Individuals with Visual Impairments

and Beyond: Strengths, Challenges, and Future Directions. Paper Modalities, Haptics-e, The Electronic Journal of Haptics Research, presented at the Invitational Research Symposium on 3(6). Technology Enhanced Assessments, Washington, DC. Keyes, A., D’Souza, M., & Postula, A. (2015). Navigation for Chai3D (2018). CHAI3D is a powerful cross-platform C++ the blind using a wireless sensor haptic glove. Paper presented at the simulation framework. Retrieved from http://www.chai3d.org/ 4th Mediterranean Conference on Embedded Computing concept/about (MECO), Budva, Montenegro. Choinière, J. P., & Gosselin, C. (2017). Development Klatzky, R. L., Giudice, N. A., Bennett, C. R., & Loomis, J. and Experimental Validation of a Haptic Compass Based M. (2014). Touch-Screen Technology for the Dynamic Display on Asymmetric Torque Stimuli. IEEE Transactions on of 2D Spatial Information Without Vision: Promise and Haptics, 10(1), pp. 29-39. Progress. Multisensory Research, 27, pp. 359-378. Copolo, C.E., & Hounshell, P. B. (1995). Using three- Kolitsky, M. A. (2014). 3d printed tactile learning objects: dimensional models to teach molecular structures in high Proof of concept. Journal of Blindness Innovation and Research, 4(1). school chemistry. Journal of Science Education and Technology, 4(4), Kor-FX (2014). Immersive Gaming Vest. Retrieved from http:// pp. 295-305. korfx.com/ Dhaher, Y., & Clements, R. (2017). A Virtual Haptic Platform Kostopoulos, K., Moustakas, K., Tzovaras, D. Nikolakis, to Assist Seeing Impaired Learning: Proof of Concept. Journal G. Thillou, C., & Gosselin, B. (2007). Haptic Access to of Blindness Innovation and Research, 7(2). Conventional 2D Maps for the Visually Impaired. Springer Diagram Center. (2017). Adding Haptic Feedback to HTML. International Journal on Multimodal User Interfaces, 1(2), pp. 13-19. Retrieved from http://diagramcenter.org/integrating-haptic- Lahav, O., Schloerb, D.W., Kumar, S., & Srinivasan, M.A. feedback.html (2012). A Virtual Environment for People Who Are Blind – A Duxbury Systems. (2017). Louis Braille and the Braille System. Usability Study. Journal of assistive technologies, 6(1), pp. 38-52. Retrieved from https://www.duxburysystems.com/braille.asp Nikolakis, G., Tzovaras, D., Moustakidis, S., & Strintzis, Erickson, W., Lee, C., & von Schrader, S. (2016). Disability M.G. (2004). CyberGrasp and PHANTOM Integration: Enhanced Status Report: United States. Ithaca, NY: Cornell University Haptic Access for Visually Impaired Users. Paper presented at 9th Yang-Tan Institute on Employment and Disability (YTI). Conference Speech and Computer Saint-Petersburg, Russia. Evett, L., Battersby, S., Ridley, A., & Brown, D. J. (2009). Nordictrack (2018). FreeStride Trainer Series. Retrieved An interface to virtual environments for people who are blind from https://www.nordictrack.com/ellipticals using Wii technology — mental models and navigation. Journal Norman, J.F., & Ashley, N. (2011). Blindness enhances tactile of Assistive Technologies, 3(2), pp. 30-39. acuity and haptic 3-D shape discrimination. Attention, perception GlovePie. (2010). Control Games with Gestures, Speech, and Other & psychophysics, 73(7), pp. 23-31. Input Devices! Retrieved from https://sites.google.com/site/ OpenVR (2018). OpenVR is an API and runtime that allows access carlkenner/glovepie to VR hardware from multiple vendors without requiring that applications Graphiti. (2018). Introducing Graphiti — A Revolution in Accessing have specifc knowledge of the hardware they are targeting. Retrieved Digital Tactile Graphics and More! from https://github.com/ValveSofware/openvr Retrieved from https://www.aph.org/graphiti/ Orozco, M., Silva, J., El Saddik, A., & Petriu, E. (2012). Grice, N., Christian, C., Nota, A., & Greenfeld, P. (2015). The Role of Haptics in Games. In A. El Saddik (ed.), Haptics 3D Printing Technology: A Unique Way of Making Rendering and Applications (pp. 217-234). London, United Hubble Space Telescope Images Accessible to Non-Visual Kingdom: IntechOpen. Learners. Journal of Blindness Innovation and Research, 5(1). OSVR (2018). Open Source Virtual Reality. Retrieved Hakkinen, M., Rice, J., Liimatainen, J., & Supalo, C. (2013). from http://www.osvr.org/ Tablet-based Haptic Feedback for STEM Content. Paper Pather, A. B. (2014). The innovative use of vector-based tactile presented at the International Technology & Disabilities Conference, graphics design sofware to automate the production of raised- San Diego, CA. line tactile graphics in accordance with BANA’s newly adopted Hap2U (2018). Piezo actuators generate ultrasonic vibrations on guidelines and standards for tactile graphics, 2010. Journal of a glass screen and modify the friction of your fnger. Retrieved Blindness Innovation and Research, 4(1). from http://www.hap2u.net/technology/ Purves, D., Augustine, G.J., & Fitzpatrick, D. Hardlightm VR (2017). Hardlight Suit. Retrieved from http:// (2001). Neuroscience. Diferences in Mechanosensory Discrimination www.hardlightvr.com/ Across the Body Surface. Sunderland, MA: Sinauer Associates. Holt,, M., Gillen, D., Cook, C., Miller, C.H., Nandlall, Quest (2018). Tacit is a sonar-enabled glove that helps the blind S.D., Setter, K., Supalo,, C., Thorman,, P., & Kane, S.A. to detect when objects are nearby. Retrieved from https://ww2. (2018). Making Physics Courses Accessible for Blind Students: kqed.org/quest/2011/08/23/glove-with-sonar-helps-the-blind- strategies for course administration, class meetings and course navigate/ materials. Physics Education, arXiv:1710.08977. Reynaga-Peña, C. G. (2015). A Microscopic World at the Jones, M. G., Bokinsky, A., Tretter, T., & Negishi, A. Touch: Learning Biology with Novel 2.5D and 3D Tactile (2005). A Comparison of Learning with Haptic and Visual Models. Journal of Blindness Innovation and Research, 5(1).

Benjamin Vercellone, John Shelestak, Yaser Dhaher, & Robert Clements https://www.gamejournal.it/07_vercellone-shelestak-shaher-clements 87 Issue 07 – 2018 Haptic Interfaces for Individuals with Visual Impairments

Röder, B., Rösler, F., & Hennighausen, E. (2007). Diferent Tony, M., Foley, J., Folmer, E. (2010).Vi-bowling: a tactile cortical activation patterns in blind and sighted humans during spatial exergame for individuals with visual impairments. encoding and transformation of haptic images. Psychophysiology, In Proceedings of the 12th international ACM SIGACCESS 34(3), pp. 292-307. conference on Computers and accessibility (pp. 179-186). New York, Steam Store. (2015, Nov 10). Steam Controller. Retrieved NY, ACM. from https://store.steampowered.com. Valve (2018). We make games, Steam, and hardware. Join us. SteamVR. (2018). SteamVR Home: New Maps, Asset Packs, and Retrieved from https://www.valvesofware.com/en/ More. Retrieved from https://steamcommunity.com/steamvr Virtual Motion Labs (2018). Wireless Finger and Hand Motion Steinberg, A.D., Bashook P.G., Drummond, J., Ashraf S., & Capture. Retrieved from http://www.virtualmotionlabs.com/ Zefran M. (2007). Assessment of faculty perception of content Visell, Y., Law, A., Cooperstock, & J. R. (2009). Touch validity of Periosim©, a haptic-3D virtual reality dental training Is Everywhere: Floor Surfaces as Ambient Haptic simulator. Journal of Dental Education, 71(12), pp. 1574-1582. Interfaces. IEEE Transactions on haptics, 2(3), pp. 148-159. Strasnick, E., Holtz, C., Ofek, E., Sinclair, M., & Benko, H. VRGluv (2017). Feeling Is Believing. Retrieved from https:// (2018). Demonstration of Haptic Links: Bimanual Haptics vrgluv.com/ for Virtual Reality Using Variable Stifness Actuation. Whitmire, E., Benko, H., Holz, C., Ofek, E., & Sinclair, In Proceedings of CHI 2018. ACM New York, NY, USA. M. (2018). Demonstration of Haptic Revolver: Touch, Tanvas. (2018). Rediscover Touch. Retrieved from https:// Shear, Texture, and Shape Rendering on a VR Controller. tanvas.co/ In Proceedings of CHI 2018. ACM New York, NY, USA. Taylor, R. M., Hudson, T. C., Seeger, A., Weber, H., Juliano, Wicab (2018). BRAINPORT TECHNOLOGIES – helping J., & Helser, A. T. (2001). VRPN: a device-independent, people with disabilities live a BETTER LIFE!, Retrieved network-transparent VR peripheral system. In VRST ’01 from https://www.wicab.com/ Proceedings of the ACM symposium on Virtual reality sofware and Woojer (2018). Ready. Play. Feel. Retrieved from https://www. technology (pp. 55-61). New York, NY: ACM. woojer.com/vest/ Teslasuit (2018). Ultimate tech in Smart Clothing. Retrieved from https://teslasuit.io/

Benjamin Vercellone, John Shelestak, Yaser Dhaher, & Robert Clements https://www.gamejournal.it/07_vercellone-shelestak-shaher-clements 88 Issue 07 – 2018 Haptic Interfaces for Individuals with Visual Impairments

Benjamin Vercellone, John Shelestak, Yaser Dhaher, & Robert Clements https://www.gamejournal.it/07_vercellone-shelestak-shaher-clements 89 www.gamejournal.it