Recent Researches in Electrical and Computer Engineering

About Visual

DRAGOS MORARU, COSTIN-ANTON BOIANGIU Computer Science Department “Politehnica” University of Bucharest Splaiul Independentei 313, Sector 6, Bucharest, 060042 ROMANIA [email protected], [email protected]

Abstract: Starting with the 80’s, technology capable of aiding damaged human senses started to emerge, the first commonly used device being the cochlear implant. However there are other types of devices, more or less sophisticated that are not well known. This paper investigates existing technologies able to aid persons with a visual sense deficit, by converting the stimuli and sending them for processing to a different region of the brain or through a different path.

Key-Words: substitution of senses, image to sound, multi-sensory, medicine and technology

1 Introduction were greatly expanded and the processing power of The conversion of a light signal into sound can be computing systems has increased greatly, the accomplished in many ways. The conversion problem of achieving a connection between the method and the result representation vary greatly, electronics and the human brain remains an issue. depending on the purpose and the resources [41] available. Currently, we have a working knowledge on how The main application for this conversion, which the sensory organs work and how they transform the will be treated in this article, is to achieve signals received from the environment into nerve substitution of senses: develop an artificial system impulses. The way the brain processes this to substitute the visual sense of a blind person with information remains an active area of research. the auditory sense. From the structural point of Based on neurological and medical studies view, such a system consists of three components: a performed, there are many correlations between sensor or a sensor network that collects visual information obtained from the senses and the information from the environment (e.g. a video generated responses of different areas of the camera), an electronic system that processes this cerebral cortex. Based on this information, detailed information and converts it to audible signal, and a maps showing the locations of the cerebral cortex device (speaker, headset) that converts this signal that are specialized in processing information from into sound to be heard by the person. Alternatively, the senses and the role of each area in processing a visual information received could be converted into particular type of specific information were created. tactile stimuli, or auditory and tactile stimuli For a person to be able to have artificial sight, simultaneously, which would allow the there are two different approaches: either recover of a greater volume of information. that sense itself through an artificial visual prostheses [4] or use another unaffected sense by which information will be transmitted to the 2 Current Status affected sense, thus achieving a substitution of the Concern for the creation of artificial human senses senses [2]. Furthermore, the two methods are able to replace damaged natural senses appeared described, with emphasis on substitution of the with the development of technology, the senses which uses a conversion process from picture development of electronic systems able to receive to sound. and produce light and sound signals and especially the development of digital computing systems, capable of performing advanced processing of the 2.1 Making visual prostheses A visual prosthesis is a device or set of devices received information [1][2]. While devices capable implanted in the body that allow a blind person to of receiving and recording light and sound signals obtain a of the surrounding world

ISBN: 978-1-61804-315-3 115 Recent Researches in Electrical and Computer Engineering

(usually at a much lower quality as compared to a the visual area. In this case, at the pulse application, normal person). the patient can see a series of colored lights. This Depending on which part of the patient's visual type of stimulation is ineffective because they system has been affected and the technology cannot accurately stimulate points of the field of available, we can make a connection between the view, so in practice this method cannot provide a prosthesis and the (in patients who have functional sight. the affected, but healthy optic nerve) or Many research projects have focused on the directly with the brain (when the patient has the development of implantable visual prosthesis on the optic nerve damaged or in cases where this retina [8][10][11]. The idea that underpins their connection is more likely to succeed). In most cases, achievement is that in many cases, patients lose information is taken from the outside through a partial or total sense (after eye disease) [12] [13] video camera or another image capturing device. [14] due to the destruction of the retinal The first attempt of a visual prosthesis photoreceptors layer. The photosensitive layer of implantation occurred in 1982 when João Lobo cells is responsible for receiving light and Antunes, Portuguese surgeon implanted an converting it into electrical impulse. In many experimental electronic device into the eye of a patients, this is the only cell layer affected, while the blind patient. rest remain healthy and/or partly functional. Thus, a Two years later, a team of researchers led by Dr. retinal implant is intended to stimulate these healthy William H. Dobelle developed a cerebral layers of cells (ganglion cell layer). [14] [16] For implantable visual prosthesis [5][7]. The device example, a team of researchers from the University consists of a grid of 68 electrodes implanted in the MIT have developed a retinal prosthesis implant brain of the patient, directly stimulating the visual consisting of a network of electrodes that stimulate cortex. The information in the environment is the ganglion cell layer. Subsequently, the team has gathered by an assembly containing an external improved their design, by usage of wireless video camera and an ultrasonic distance measuring transmission of the information received from the system. The information received is processed by a external camera containing the internal implant. computer system and sent to the network of A similar device for commercial application was electrodes. Following stimulation of the visual developed by an US company, Second Sight. [17] cortex, the patient perceives an array of black and The implant restores a patient’s poor vision that white dots. [6] Although the image resolution is allows him to visualize the surrounding space and quite low, it is sufficient to restore some sense of its limits, to recognize objects and read large letters. sight to the patients and allows the perception of an [18] [19] This system has enjoyed a commercial object outline or reading of large letters. The visual success, being the first FDA approved device for the acuity is comparable to the visual acuity of the United States Agency for use as a visual prosthesis. peripheral area of the retina of a person with normal Other research teams have developed retina sight or of a person suffering from a high degree of implantable prosthesis capable of receiving light. myopia that is uncorrected by optical means. Basically, they created a prosthesis that reproduces Most subsequent research to achieve visual the normal functioning of the retina (an artificial prostheses implanted in the brain relies on the retina). This prosthesis is implanted on the retina principle conducted by Dr. Dobelle [34] [35] [36], and it contains a network of sensors that convert with various improvements of transmitting light stimuli into electrical impulses (similar to a information from the exterior to the implant, such as normal retina). a wireless signal and electricity transmission system Such a prosthesis is, for example, the one [29] [30]. It also aims to achieve a durable implant developed by a German team at the University of and increased image quality. Researchers from the Tübingen. The device is implanted on the retina, but "Laboratory of Neural Prosthesis", Illinois Institute unlike other prostheses, it contains a network of of Technology developed a device similar to that of photodiodes that convert light into electrical impulse Dr. Dobelle, but with intra-cortical electrodes. This which is received by the photoreceptors, like from a type of implant, at least theoretically, has the healthy retina. Photo-natural receptors are more advantage of a better resolution. sensitive than the implanted photodiodes. They are The visual system may be externally stimulated not sensitive enough to be stimulated at normal electrically or magnetically without the need of an levels of exposure to the visible light in the eye. To implant on the retina or brain. For example, the solve this problem, researchers have developed an brain can be stimulated by applying external external power to amplify current. The device has magnetic pulses to the head area corresponding to

ISBN: 978-1-61804-315-3 116 Recent Researches in Electrical and Computer Engineering

enjoyed some success, and patients can recognize In terms of information, an image can be simple objects and read large letters. represented by a 2D array in which each element is According to some patients, this device enjoyed the value of a pixel. Each frame can be converted pleasing feedback. With the sight obtained, they can directly into a tactile stimuli by directly mapping be guided around to identify lines and object each pixel in the image on a touch sensitive area of outlines, to orient in a room depending on the light the body. This is possible because the body has a of an object. Other researchers have developed large number of tactile sensors in the skin, and for a implantable photovoltaic retinal prosthesis [27] [28], picture with low resolution, a 1:1 mapping between but in this case the implantation procedure proved a pixel in the image and a point on the skin can be difficult, because of the difficult supply system. achieved. For images with higher resolution this is not possible because the "resolution" sense of touch is not as great as the visual sense. Also the brain's 2.2 Substitution of the visual sense ability to interpret signals from different sensors is Implantable visual prosthesis described in the different. In addition, the human sense of touch is previous chapter have the advantage that they give not uniform. There are areas of high sensitivity, the patient a real view (real visual perception). Thus such as fingers or tongue, and there are areas where a patient who was previously completely blind or the sense of touch is much diminished. If you want a with a low vision will get benefit from a functional higher tactile resolution a skin area with high view (even if the functionality is quite limited) and sensitivity must be selected (or a skin area of a can still use his visual cortex even if the retina is larger size). inoperative. [20] [21] Unlike tactile, in the case of the auditory sense, Unlike achieving an implant, substituting the the frame mapping will not be a 1:1 ratio at the visual sense with another sense does not imply same time. From a mechanical point of view, the surgery and a blind person could recover some audio material is vibration of air, one-dimensional features of the visual sense without assuming any signal, as opposed to the image, where each frame is risk. Also, substitution of senses could theoretically a matrix of x * y pixels where x and y represent the be used when the patient has an affected visual resolution of the image. But the human ear does not cortex and an implant would not work or if the receive sounds in this way. Inside the sounds are patient is suffering from other diseases that would separated by frequency so that auditory information make an operation very risky. is received and processed by the brain largely The substitution of the senses aims only to depending on their frequency. This can be recover the functionality with no direct stimulation considered a two-dimensional signal, considering of the brain so that it cannot produce a real visual the size range of frequencies and time axis. experience. However, similar perception can be The following describes several methods of produced, which can be interpreted subjectively like image-sound conversion. content with visual information.

2.2.2 vOICE Technology 2.2.1 The scientific basis for sense substitution vOICE technology is a system that performs Initially, researchers have assumed that once substitution of the visual sense with the auditory developed, the brain remains unchanged throughout (sound-image conversion) [52] [55]. The user has life. After several decades of research, it was found attached a video camera that takes images of the that actually the brain can suffer both anatomical environment. These are processed and turned into (structural) and physiological (functional) changes sounds that are heard by the user in a helmet. [50] and that both neural pathways and synapses may Conversion is based on the notions discussed change. This phenomenon has been termed neuro- above. The sound signal is considered two plasticity. [38] [39] [40] dimensional. The x-axis is the sound frequency, and The brain can remodel and adapt situations and the y-axis is time. In general, a frame is considered environmental conditions to which it is subjected with a duration of 1. Each image frame is scanned and when trained in using other senses in processing from left to right, and at every moment a sound with visual information, the brain can adapt structurally corresponding frequencies is generated. This is a and functionally to process the visual information light-sound association. The brighter the pixel, the received [41] [45] [46]. To test this theory, many louder it will be heard. [54] Later versions tested for researchers have tried substituting visual sense with a dark-noise compliance. [51] At the end of a other senses (usually auditory or tactile sense). conversion, the process resumes for the next image.

ISBN: 978-1-61804-315-3 117 Recent Researches in Electrical and Computer Engineering

In this project was studied the ability of neuro- the disadvantage of a slow frame rate change. [55] plasticity of the brain, reflected in visual processing This is normal because the visual-auditory system adapting to the information obtained from substitution systems must sweep the signal on an hearing [50] [53]. The experiments were performed axis individually, while visual-tactile systems can on subjects with normal visual sense, but process a full frame (2D array of points) in one blindfolded so as to use only the information time. received by hearing. Improved system performance can be achieved Subjects were aimed at the location of simple by image preprocessing. The system could detect objects, with the purpose of identifying simple relevant details and could differentiate the rest of the shapes and navigating an area with obstacles. image (increasing the difference in contrast) so that Previous studies indicated that, the success rate objects become clearly identifiable and can be more obtained from subjects receiving notice of the easily isolated from the rest of the image. systems operation at the beginning of the experiment does not vary significantly from the success of those who do not know any details on 2.2.3 Prosthesis for vision substitution using how the system performs conversion. [55] The hearing (PSVA) success rate in locating objects was much better than The equipment is similar to those used in vOICE: a that of identifying objects. Even after repeated video camera captures images, they are processed training for object recognition, the success rate has and heard in the user’s headset. The association not increased significantly, but the recognition time performs a pixel-frequency vertical axis, similar to was decreased. the voice system, but this time it achieves an inverse Other studies have followed people for several correlation between the visual and auditory model. months, in which they practiced with the visual The vision about the retina is mapped over an system replacement. By using brain imaging inverted linear model of the cochlea. The authors techniques, images of the subject’s brain were argue that a system based on this type of mapping is obtained. The images showed an activation of brain feasible and there are practical advantages in visual- areas normally used in pattern recognition and auditory substitution. A prototype that allows spatial processing, which confirms the validity of optimizing of the operations specific to this neuro-plasticity theory of the brain in the case of substitution process has been developed. [68] visual-auditory substitution. It is interesting that after some practice with the device, subjects 2.2.4 The EyeMusic system whether blind or with normal vision, say that they It works much like the vOICE system, with the came to perceive sound signals as images, and the difference that the ordinate axis height is entire process of identification and recognition as represented by musical notes of a musical sight. [51] instrument. [56] As in the vOICE system, a bright Despite the positive results achieved, the system pixel will generate a sound with higher amplitude. cannot provide the same performance in terms of [57] The novelty it brings is that it allows the visual acuity with other methods such as addition of light intensity and color to be identified implantable prostheses or other methods of visual [59] [61]. For each color that it is represented, the substitution. corresponding musical notes are generated by a Reading and identifying letters is difficult, if not different tool. Thus, the user can, for example, impossible. identify a red apple in a bowl of green apples. Although theoretically, the image resolution can Despite improvements, EyeMusic suffers from be increased (going to be comparison with a visual- the same weaknesses as the vOICE. Identifying tactile substitution) by the inclusion of several objects (compared to their location) is quite frequencies, or by scanning the image from the left difficult, even for a trained user. and right slower, which is not necessarily in the Despite receiving more complex sensory benefit of patient. While a low resolution proves to information, the brain cannot understand the input be insufficient to provide a functional view, a high properly and easily use it to read outlines and shapes resolution creates difficulties in integrating of objects (although as in the case of voice, patients information to charge an overview of the frame. It can learn to recognize and process information as must be a compromise between the final resolution well as the visual). In addition, as with other types and speed perception of a frame. Compared to other of visual and auditory substitution, the identification types of replacement systems, vOICE system, and in and processing of information is very slow general all visual-auditory substitution systems have

ISBN: 978-1-61804-315-3 118 Recent Researches in Electrical and Computer Engineering

compared to other methods, such as visual results are not unique to The Vibe system, similar prosthetic implants. results were obtained with other systems such as vOICE replacement or with visual-tactile substitution systems. 2.2.5 The Vibe The brain has the ability to perceive stimuli received from the senses as separate objects designed in 2.2.6 Visual-tactile substitution systems space. [64] When, for example, a human being These systems use tactile receptors in the skin to looks at an object, the object reflects light rays that replace damaged visual sense [69] [70]. Because a reach the eye, the retina, where it stimulates the visual matrix of pixels can be mapped directly to a photosensitive cells. Although the actual stimulus sensitive area of the skin, these systems allow for a occurs in the vicinity of the body, inside the retina, higher resolution and a more precise mapping of the image is not a stimulus from inside, but an each pixel. If in the case of visual-auditory object in space at a distance from the observer, substitution, the exact location of the pixel was which is separated from the stimulus on the retina. difficult to perceive and interpret by the user, in Similarly, when we hear a noise, vibration visual-tactile substitution the user can locate more perception is not inside the ear, but a remote sound precisely the spatial area stimulated. For this reason, produced at a distance, that comes from a certain such systems have the potential to effectively direction. This natural ability of the brain is not allowing the patient handwriting recognition, object limited to process information from the senses and it outline recognition, shapes and their precise location can be extended to other areas. in space [72] [71]. The quality of incoming stimuli In visual-auditory substitution of the senses, it is is much too small to allow a person to drive a car intended that an auditory stimulus artificially using such a device, or to move in space in a generated and heard in the headset to be perceived complex environment with visual details. [72] In as the existence of information and/or items in such cases, the perception can be improved using space. Such a technique is used in the software "The methods similar to those used visual aids systems, Vibe" [63]. Basically, it is intended that through the such as simplifying the image (reducing the generated sound, to achieve spatial perception of the resolution to the point that performance would start environment. [65] The practical applications of such to fall) to retain only relevant details that matter and a system would improve the movement, orientation can be effectively identified. in space, movement and avoiding obstacles in the It is recalled that the skin sensory organ has environment rather than identifying a object or variable sensitivity so that a determining factor for performing other complex tasks. The algorithm the success of a visual-tactile substitution system is divides the image received from a camera into to choose an area with sufficient sensitivity and an localized areas of pixels called receptive fields. appropriate size for the perceived stimulation. The Each receptive field is considered a virtual sound tactile resolution in a given area depends not only source in the visual space and will sound on the number of receptors in that area, but it is also accordingly. For each of these fields, the algorithm limited in how they connect with the brain and by calculates an average brightness to be used when the brain's ability to process and identify generating the sinusoidal sound. The frequency of independent stimuli received from different areas. sound is calculated based on the pixel coordinates of Also, the physical and electrical stimulation of the the center of gravity of each receptive field. [63] item and its size have a decisive role in the After numerous studies, observations on how the development of a system that provides a high remote system can produce the perception of stimuli quality perception. In addition to the actual were made. [66] At first, subjects reported hearing resolution of the area of skin used, the resolution of something with no special significance, which are the system is limited by the number of stimulation not associated with remote stimuli. Later, however, elements. [75] after training with the device, subjects reported that Visual-tactile substitution systems can be divided they have acquired the ability of perceiving specific into two categories: general systems for use in a sounds differently from other random sounds and number of situations and specialized systems that started associating them with distant information. are intended for a specific purpose (e.g. for reading Basically, some sounds were no longer perceived as text). local sounds, but as stimuli located in a three Visual-tactile substitution systems preceded dimensional space, caused by objects located in visual-auditory substitution systems. The first space, each with a corresponding sound. Such visual-tactile substitution system known in the

ISBN: 978-1-61804-315-3 119 Recent Researches in Electrical and Computer Engineering

scientific literature was developed by Bach-y-Rita device). System resolution is 24 rows and 6 columns and included a video camera for capturing images (the required resolution was determined by the use and a tactile stimulation system based on vibration of the device). The user moves his fingers from left (the video was converted into tactile vibrations) to the right as if reading a written text. It is located on the back of the user. [71] [72] Using this important that the vertical resolution, which gives system it was possible to achieve the identification the number of rows to be higher. Although the of simple objects spatial orientation by using device also works with a single column, there was a contrast difference. The research was continued significant improvement in performance with using different skin areas. [73] [74] Ultimately, increased number of columns (up to 6 because the Bach-y-Rita concluded that the most effective way fingertip fits around 6 columns). The system was to use tactile stimulation was using the tongue marketed by the company Telesensory Systems Inc., because it has high spatial resolution and the device Silicon Valley, California and has been steadily will be greatly reduced in dimensions compared to improved from the original version, but eventually older versions that stimulate other parts of the body. was abandoned. [80] Although initial results were promising, following further development of the project could easily find that the present system severely limits the 3 Conclusions possibility of playing more complex views. They Although image-sound substitution systems take a large amount of information of the visual developed so far can substitute a simple visual field, which is more than the brain can process and perception, providing the brain with information interpret compared to a much smaller amount of that can be perceived and interpreted in a manner information from other areas. similar to natural vision (due to the phenomenon of Virtually every sense a human has, operates in a neuro-plasticity), their applicability has not been certain way and the brain is adapted to process demonstrated in real and practical cases. In contrast, information in a proper sense. The mere fact that a studies indicate a weak brainpower in identifying certain sense has a higher spatial or informational details, shapes, outlines, based on the auditory and resolution does not mean that it allows the tactile stimuli. perception of nonspecific information better than By its nature, the brain has a greater capacity for another sense with lower resolution, but specialized processing visual information than other types of in processing information of that certain type. For information. Therefore, it is much harder to develop example, although visual sense is able to process a a system to provide a detailed and conscious much larger amount of information than hearing, for perception of the environment on other than visual a deaf person, it is almost impossible even with sensations, regardless of how they are acquired and intense, training to be able to recognize words in a processed and the number of neurons stimulated. graphic representation (visual) sound (diagram From this point of view, implantable visual spectrum), because the brain is not easily adapted to prosthetics through direct connection to the visual recognize such information of the visual field. [49] cortex generates authentic sensations directly, so As with visual-auditory substitution systems, it is that, in theory, they have a significant advantage not clear how much the phenomenon of neuro- over any visual substitution method. [49] plasticity can be exploited in order to use It remains to be seen which are the limits of specialized brain areas in visual processing. neuro-plasticity in allowing the usage of other In general, visual-tactile substitution systems senses for visual processing. Based on studies with specific use allow obtaining better performance conducted so far, possibility of using a substitution in situations for which they were designed [79] [81]. system to achieve a real visual perception for An example is Optacon [78], invented by Professor producing real visual experience, remains an open John Linvill of Stanford University, for optical question. character reading on paper or on a monitor by a blind person. Light signals are converted into a binary touch signal (each pixel is white or black). References: The touch produces vibrations that are perceived by [1] Bach-y-Rita, Paul. "Brain plasticity as a basis the patient, by placing his fingers on the touch for recovery of function in humans." surface. Unlike other visual-tactile devices for Neuropsychologia 28.6 (1990): 547-554. general replacement which stimulate a fixed surface [2] Bach-y-Rita, Paul, and Stephen W. Kercel. on the skin, in the case of Optacon the surface and "Sensory substitution and the human–machine its position varies (the user moves his fingers on the

ISBN: 978-1-61804-315-3 120 Recent Researches in Electrical and Computer Engineering

interface." Trends in cognitive sciences 7.12 [16] Dorn, Jessy D., et al. "The detection of motion (2003): 541-546. by blind subjects with the epiretinal 60- [3] Humayun, Mark S., Eugene de Juan Jr, and electrode (Argus II) retinal prosthesis." JAMA Robert J. Greenberg. "Visual prosthesis and ophthalmology 131.2 (2013): 183-189. method of using same." U.S. Patent No. [17] Humayun, Mark S., et al. "Interim results from 5,935,155. 10 Aug. 1999. the international trial of Second Sight's visual [4] Rizzo, Joseph F., and John Wyatt. "REVIEW: prosthesis." Ophthalmology 119.4 (2012): 779- Prospects for a Visual Prosthesis." The 788. Neuroscientist 3.4 (1997): 251-262. [18] Ahuja, A. K., et al. "Blind subjects implanted [5] Dobelle, William H., M. G. Mladejovsky, and with the Argus II retinal prosthesis are able to J. P. Girvin. "Artificial vision for the blind: improve performance in a spatial-motor task." Electrical stimulation of visual cortex offers British Journal of Ophthalmology 95.4 (2011): hope for a functional prosthesis." Science 539-543. 183.4123 (1974): 440-444. [19] Barry, Michael P., and Gislin Dagnelie. "Use of [6] Dobelle, William H., et al. "Artificial vision for the Argus II retinal prosthesis to improve visual the blind by electrical stimulation of the visual guidance of fine hand movements." cortex." Neurosurgery 5.4 (1979): 521-527. Investigative ophthalmology & visual science [7] Dobelle, W. H., and M. G. Mladejovsky. 53.9 (2012): 5095-5101. "Phosphenes produced by electrical stimulation [20] Weiland, James D., Alice K. Cho, and Mark S. of human occipital cortex, and their application Humayun. "Retinal prostheses: current clinical to the development of a prosthesis for the results and future needs." Ophthalmology blind." The Journal of physiology 243.2 (1974): 118.11 (2011): 2227-2237. 553-576. [21] Winter, Jessica O., Stuart F. Cogan, and Joseph [8] Bernard, Thierry M., Bertrand Y. Zavidovique, F. Rizzo. "Retinal prostheses: current and Francis J. Devos. "A programmable challenges and future outlook." Journal of artificial retina." Solid-State Circuits, IEEE Biomaterials Science, Polymer Edition 18.8 Journal of 28.7 (1993): 789-798. (2007): 1031-1055. [9] Greenberg, Robert J. "Visual prostheses: A [22] Koyanagi, Mitsumasa, et al. "Neuromorphic review." Neuromodulation: Technology at the vision chip fabricated using three-dimensional Neural Interface 3.3 (2000): 161-165. integration technology." Solid-State Circuits [10] Kyuma, Kazuo, et al. "Artificial ---fast, Conference, 2001. Digest of Technical Papers. versatile image processors." Nature 372 (1994): ISSCC. 2001 IEEE International. IEEE, 2001. 197-198. [23] Zrenner, Eberhart. "Visual Prosthesis, [11] Chow, Alan Y., and Vincent Chow. Subretinal Devices." (2014). "Independent photoelectric artificial retina [24] Zrenner, Eberhart, et al. "Subretinal electronic device and method of using same." U.S. Patent chips allow blind patients to read letters and No. 5,556,423. 17 Sep. 1996. combine them to words." Proceedings of the [12] Chow, A. Y., et al. "Safety, Feasibility and Royal Society B: Biological Sciences (2010): Efficacy of Subretinal Artificial Silicon rspb20101747. RetinaTM Prosthesis for the Treatment of [25] Kimura, Mutsumi, et al. "Artificial retina using Patients with Retinitis Pigmentosa." thin-film photodiodes and thin-film Investigative Ophtalmology and Visual Science transistors." Japanese journal of applied 43.12 (2002): 2849. physics 45.5S (2006): 4419. [13] Chow, Alan Y., et al. "The artificial silicon [26] Edward, Tassicker Graham. "Retinal retina microchip for the treatment of visionloss stimulator." U.S. Patent No. 2,760,483. 28 from retinitis pigmentosa." Archives of Aug. 1956. ophthalmology 122.4 (2004): 460-469. [27] Loudin, James, et al. "Photovoltaic retinal [14] Chow, A. Y., et al. "Subretinal Artificial prosthesis." SPIE BiOS. International Society Silicon Retina Microchip for the Treatment of for Optics and Photonics, 2011. Retinitis Pigmentosa: 3 1/ 2 Year Update." [28] Wang, Lele, et al. "Photovoltaic retinal Investigative Ophtalmology and Visual Science prosthesis: implant fabrication and 45.5 (2004): 3398. performance." Journal of [15] Doorish, John F. "Artificial retina." U.S. Patent 9.4 (2012): 046014. No. 5,865,839. 2 Feb. 1999. [29] Klauke, Susanne, et al. "Stimulation with a wireless intraocular epiretinal implant elicits

ISBN: 978-1-61804-315-3 121 Recent Researches in Electrical and Computer Engineering

visual percepts in blind humans." Investigative [42] Merabet, Lotfi B., and Alvaro Pascual-Leone. ophthalmology & visual science 52.1 (2011): "Neural reorganization following sensory loss: 449-455. the opportunity of change." Nature Reviews [30] Walter, Peter, et al. "Cortical activation via an 11.1 (2010): 44-52. implanted wireless retinal prosthesis." [43] Merabet, Lotfi B., et al. "What blindness can Investigative ophthalmology & visual science tell us about seeing again: merging 46.5 (2005): 1780-1785. and neuroprostheses." Nature [31] Schwahn, Hartmut N., et al. "Studies on the Reviews Neuroscience 6.1 (2005): 71-77. feasibility of a subretinal visual prosthesis: data [44] Théoret, Hugo, Lotfi Merabet, and Alvaro from Yucatan micropig and rabbit." Graefe's Pascual-Leone. "Behavioral and neuroplastic archive for clinical and experimental changes in the blind: evidence for functionally ophthalmology 239.12 (2001): 961-967. relevant cross-modal interactions." Journal of [32] Suaning, Gregg J., et al. "The bionic eye Physiology-Paris 98.1 (2004): 221-233. (electronic visual prosthesis): a review." [45] Rauschecker, Josef P. "Compensatory plasticity Australian and New Zealand journal of and sensory substitution in the cerebral cortex." ophthalmology 26.3 (1998): 195-202. Trends in 18.1 (1995): 36-43. [33] Cha, Kichul, Kenneth W. Horch, and Richard [46] Draganski, Bogdan, et al. "Neuroplasticity: A. Normann. "Mobility performance with a changes in grey matter induced by training." pixelized vision system." Vision Research 32.7 Nature 427.6972 (2004): 311-312. (1992): 1367-1372. [47] Renier, Laurent, et al. "Cross-modal activation [34] Schmidt, E. M., et al. "Feasibility of a visual of visual cortex during depth perception using prosthesis for the blind based on intracortical auditory substitution of vision." Neuroimage micro stimulation of the visual cortex." Brain 26.2 (2005): 573-580. 119.2 (1996): 507-522. [48] Kaczmarek, Kurt A., et al. "Electrotactile and [35] Bak, M., et al. "Visual sensations produced by vibrotactile displays for sensory substitution intracortical microstimulation of the human systems." Biomedical Engineering, IEEE occipital cortex." Medical and Biological Transactions on 38.1 (1991): 1-16. Engineering and Computing 28.3 (1990): 257- [49] Lenay, Charles, et al. "Sensory substitution: 259. limits and perspectives." Touching for knowing [36] Troyk, Philip R., D. Detlefsen, and Glenn A. (2003): 275-292. DeMichele. "A 16-channel stimulator ASIC for [50] Auvray, Malika, Sylvain Hanneton, and J. use in an intracortical visual prosthesis." Kevin O Regan. "Learning to perceive with a Proceedings of the 10th Annual Conference of visuo-auditory substitution system: the International FES Society. 2005. Localisation and object recognition withThe [37] Eckhorn, Reinhard, et al. "Visual resolution vOICe'." Perception-London- 36.3 (2007): 416. with retinal implants estimated from recordings [51] http:// www.seeingwithsound.com, accessed in cat visual cortex." Vision research 46.17 27.05.2015 (2006): 2675-2690. [52] Ward, Jamie, and Peter Meijer. "Visual [38] Kays, Jill L., Robin A. Hurley, and Katherine experiences in the blind induced by an auditory H. Taber. "The dynamic brain: neuroplasticity sensory substitution device." Consciousness and mental health." The Journal of and 19.1 (2010): 492-500. neuropsychiatry and clinical neurosciences [53] Amedi, Amir, et al. "Shape conveyed by visual- 24.2 (2012): 118-124. to-auditory sensory substitution activates the [39] Cramer, Steven C., and Jeff D. Riley. lateral occipital complex." Nature neuroscience "Neuroplasticity and brain repair after stroke." 10.6 (2007): 687-689. Current opinion in neurology 21.1 (2008): 76- [54] Meijer, Peter BL. "An experimental system for 82. auditory image representations." Biomedical [40] Cramer, Steven C., et al. "Harnessing Engineering, IEEE Transactions on 39.2 neuroplasticity for clinical applications." Brain (1992): 112-121. 134.6 (2011): 1591-1609. [55] Brown½, David, Tom Macpherson, and Jamie [41] Bach-y-Rita, Paul, Mitchell E. Tyler, and Kurt Wardô. "Seeing with sound? Exploring A. Kaczmarek. "Seeing with the brain." different characteristics of a visual-to-auditory International journal of human-computer sensory substitution device." Perception 40 interaction 15.2 (2003): 285-295. (2011): 1120-1135.

ISBN: 978-1-61804-315-3 122 Recent Researches in Electrical and Computer Engineering

[56] Abboud, Sami, et al. "EyeMusic: Introducing a [68] Capelle, Christian, et al. "A real-time “visual” colorful experience for the blind using experimental prototype for enhancement of auditory sensory substitution." Restorative vision rehabilitation using auditory neurology and neuroscience 32.2 (2014): 247- substitution." Biomedical Engineering, IEEE 257. Transactions on 45.10 (1998): 1279-1293. [57] http:// brain.huji.ac.il/ site/ em.html, accessed [69] Bach-y-Rita, Paul. "Tactile sensory substitution 27.05.2015 studies." Annals-New York Academy Of [58] Bologna, Guido, Juan Diego Gomez, and Sciences 1013 (2004): 83-91. Thierry Pun. "Vision Substitution Experiments [70] White, Benjamin W., et al. "Seeing with the with See ColOr." Natural and Artificial Models skin." Perception & Psychophysics 7.1 (1970): in Computation and Biology. Springer Berlin 23-27. Heidelberg, 2013. 83-93. [71] Bach-y-Rita, Paul. "Tactile vision substitution: [59] Levy-Tzedek, Shelly, Maayan Halimi, and past and future." International Journal of Amir Amedi. "Seeing with Your Ears: A Neuroscience 19.1-4 (1983): 29-36. Wondrous Journey Across the Senses." [72] Bach-y-Rita, Paul, and Barry Hughes. "Tactile Understanding Neuroscience 1 (2013): 27. vision substitution: some instrumentation and [60] Levy-Tzedek, S., et al. "Fast, accurate reaching perceptual considerations." Electronic spatial movements with a visual-to-auditory sensory sensing for the blind. Springer Netherlands, substitution device." Restorative neurology and 1985. 171-186. neuroscience 30.4 (2012): 313-323. [73] Jones, Lynette A., Brett Lockyer, and Erin [61] Hamilton-Fletcher, Giles, and Jamie Ward. Piateski. "Tactile display and vibrotactile "Representing colour through hearing and pattern recognition on the torso." Advanced touch in sensory substitution devices." Robotics 20.12 (2006): 1359-1374. Multisensory research 26.6 (2013): 503-532. [74] Sampaio, Eliana, Stephane Maris, and Paul [62] Maidenbaum, Shachar, et al. "Depth-To-Audio Bach-y-Rita . "Brain plasticity:‘visual’acuity of Sensory Substitution for Increasing the blind persons via the tongue." Brain research Accessibility of Virtual Environments." 908.2 (2001): 204-207. Universal Access in Human-Computer [75] Wall, Steven A., and Stephen Brewster. Interaction. Design and Development Methods "Sensory substitution using tactile pin arrays: for Universal Access. Springer International Human factors, technology and applications." Publishing, 2014. 398-406. Signal Processing 86.12 (2006): 3674-3695. [63] Auvray, Malika, et al. "There is something out [76] Bachyrit. p. "Symposium-Prosthetic Aids For there: distal attribution in sensory substitution, Blind-Visual Information Through Skin-Tactile twenty years later." Journal of integrative Vision Substitution System." Transactions neuroscience 4.04 (2005): 505-521. American Academy Of Ophthalmology And [64] Siegle, Joshua H., and William H. Warren. Otolaryngology 78.5 (1974): o729-o740. "Distal attribution and distance perception in [77] Sadato, Norihiro, et al. "Tactile discrimination sensory substitution." Perception 39.2 (2010): activates the visual cortex of the recently blind 208. naive to Braille: a functional magnetic [65] Hanneton, Sylvain, Malika Auvray, and resonance imaging study in humans." Barthélemy Durette. "The Vibe: a versatile Neuroscience letters 359.1 (2004): 49-52. vision-to-audition sensory substitution device." [78] Linvill, John G. "Research and Development of Applied Bionics and Biomechanics 7.4 (2010): Tactile Facsimile Reading Aid for the Blind 269-276. (The Optacon). Final Report." (1973). [66] Epstein, William, et al. "Is there anything out [79] Hauger, J. Scott. "Reading machines for the there? A study of distal attribution in response blind: a study of federally supported to vibrotactile stimulation." Perception 15.3 technology development and innovation." (1986): 275-284. (1995). [67] Loomis, Jack M., Roberta L. Klatzky, and [80] Kendrick, Deborah. "From Optacon to Nicholas A. Giudice. "Sensory substitution of Oblivion: The Telesensory Story." (2005). vision: importance of perceptual and cognitive [81] Candela, Anthony R. "Legends and Pioneers of processing." Assistive technology for blindness Blindness Assistive Technology, Part 2." and low vision. CRC Press, Boca Raton (2012): (2006). 161-193. [82] Runyan, Noel H., and Deane B. Blazie. "The continuing quest for the'Holy Braille'of tactile

ISBN: 978-1-61804-315-3 123 Recent Researches in Electrical and Computer Engineering

displays." SPIE NanoScience+ Engineering. 2011. International Society for Optics and Photonics,

ISBN: 978-1-61804-315-3 124