Recent Researches in Electrical and Computer Engineering About Visual Sensory Substitution DRAGOS MORARU, COSTIN-ANTON BOIANGIU Computer Science Department “Politehnica” University of Bucharest Splaiul Independentei 313, Sector 6, Bucharest, 060042 ROMANIA [email protected], [email protected] Abstract: Starting with the 80’s, technology capable of aiding damaged human senses started to emerge, the first commonly used device being the cochlear implant. However there are other types of devices, more or less sophisticated that are not well known. This paper investigates existing technologies able to aid persons with a visual sense deficit, by converting the stimuli and sending them for processing to a different region of the brain or through a different path. Key-Words: substitution of senses, image to sound, multi-sensory, medicine and technology 1 Introduction were greatly expanded and the processing power of The conversion of a light signal into sound can be computing systems has increased greatly, the accomplished in many ways. The conversion problem of achieving a connection between the method and the result representation vary greatly, electronics and the human brain remains an issue. depending on the purpose and the resources [41] available. Currently, we have a working knowledge on how The main application for this conversion, which the sensory organs work and how they transform the will be treated in this article, is to achieve signals received from the environment into nerve substitution of senses: develop an artificial system impulses. The way the brain processes this to substitute the visual sense of a blind person with information remains an active area of research. the auditory sense. From the structural point of Based on neurological and medical studies view, such a system consists of three components: a performed, there are many correlations between sensor or a sensor network that collects visual information obtained from the senses and the information from the environment (e.g. a video generated responses of different areas of the camera), an electronic system that processes this cerebral cortex. Based on this information, detailed information and converts it to audible signal, and a maps showing the locations of the cerebral cortex device (speaker, headset) that converts this signal that are specialized in processing information from into sound to be heard by the person. Alternatively, the senses and the role of each area in processing a visual information received could be converted into particular type of specific information were created. tactile stimuli, or auditory and tactile stimuli For a person to be able to have artificial sight, simultaneously, which would allow the perception there are two different approaches: either recover of a greater volume of information. that sense itself through an artificial visual prostheses [4] or use another unaffected sense by which information will be transmitted to the 2 Current Status affected sense, thus achieving a substitution of the Concern for the creation of artificial human senses senses [2]. Furthermore, the two methods are able to replace damaged natural senses appeared described, with emphasis on substitution of the with the development of technology, the senses which uses a conversion process from picture development of electronic systems able to receive to sound. and produce light and sound signals and especially the development of digital computing systems, capable of performing advanced processing of the 2.1 Making visual prostheses A visual prosthesis is a device or set of devices received information [1][2]. While devices capable implanted in the body that allow a blind person to of receiving and recording light and sound signals obtain a visual perception of the surrounding world ISBN: 978-1-61804-315-3 115 Recent Researches in Electrical and Computer Engineering (usually at a much lower quality as compared to a the visual area. In this case, at the pulse application, normal person). the patient can see a series of colored lights. This Depending on which part of the patient's visual type of stimulation is ineffective because they system has been affected and the technology cannot accurately stimulate points of the field of available, we can make a connection between the view, so in practice this method cannot provide a prosthesis and the optic nerve (in patients who have functional sight. the retina affected, but healthy optic nerve) or Many research projects have focused on the directly with the brain (when the patient has the development of implantable visual prosthesis on the optic nerve damaged or in cases where this retina [8][10][11]. The idea that underpins their connection is more likely to succeed). In most cases, achievement is that in many cases, patients lose information is taken from the outside through a partial or total sense (after eye disease) [12] [13] video camera or another image capturing device. [14] due to the destruction of the retinal The first attempt of a visual prosthesis photoreceptors layer. The photosensitive layer of implantation occurred in 1982 when João Lobo cells is responsible for receiving light and Antunes, Portuguese surgeon implanted an converting it into electrical impulse. In many experimental electronic device into the eye of a patients, this is the only cell layer affected, while the blind patient. rest remain healthy and/or partly functional. Thus, a Two years later, a team of researchers led by Dr. retinal implant is intended to stimulate these healthy William H. Dobelle developed a cerebral layers of cells (ganglion cell layer). [14] [16] For implantable visual prosthesis [5][7]. The device example, a team of researchers from the University consists of a grid of 68 electrodes implanted in the MIT have developed a retinal prosthesis implant brain of the patient, directly stimulating the visual consisting of a network of electrodes that stimulate cortex. The information in the environment is the ganglion cell layer. Subsequently, the team has gathered by an assembly containing an external improved their design, by usage of wireless video camera and an ultrasonic distance measuring transmission of the information received from the system. The information received is processed by a external camera containing the internal implant. computer system and sent to the network of A similar device for commercial application was electrodes. Following stimulation of the visual developed by an US company, Second Sight. [17] cortex, the patient perceives an array of black and The implant restores a patient’s poor vision that white dots. [6] Although the image resolution is allows him to visualize the surrounding space and quite low, it is sufficient to restore some sense of its limits, to recognize objects and read large letters. sight to the patients and allows the perception of an [18] [19] This system has enjoyed a commercial object outline or reading of large letters. The visual success, being the first FDA approved device for the acuity is comparable to the visual acuity of the United States Agency for use as a visual prosthesis. peripheral area of the retina of a person with normal Other research teams have developed retina sight or of a person suffering from a high degree of implantable prosthesis capable of receiving light. myopia that is uncorrected by optical means. Basically, they created a prosthesis that reproduces Most subsequent research to achieve visual the normal functioning of the retina (an artificial prostheses implanted in the brain relies on the retina). This prosthesis is implanted on the retina principle conducted by Dr. Dobelle [34] [35] [36], and it contains a network of sensors that convert with various improvements of transmitting light stimuli into electrical impulses (similar to a information from the exterior to the implant, such as normal retina). a wireless signal and electricity transmission system Such a prosthesis is, for example, the one [29] [30]. It also aims to achieve a durable implant developed by a German team at the University of and increased image quality. Researchers from the Tübingen. The device is implanted on the retina, but "Laboratory of Neural Prosthesis", Illinois Institute unlike other prostheses, it contains a network of of Technology developed a device similar to that of photodiodes that convert light into electrical impulse Dr. Dobelle, but with intra-cortical electrodes. This which is received by the photoreceptors, like from a type of implant, at least theoretically, has the healthy retina. Photo-natural receptors are more advantage of a better resolution. sensitive than the implanted photodiodes. They are The visual system may be externally stimulated not sensitive enough to be stimulated at normal electrically or magnetically without the need of an levels of exposure to the visible light in the eye. To implant on the retina or brain. For example, the solve this problem, researchers have developed an brain can be stimulated by applying external external power to amplify current. The device has magnetic pulses to the head area corresponding to ISBN: 978-1-61804-315-3 116 Recent Researches in Electrical and Computer Engineering enjoyed some success, and patients can recognize In terms of information, an image can be simple objects and read large letters. represented by a 2D array in which each element is According to some patients, this device enjoyed the value of a pixel. Each frame can be converted pleasing feedback. With the sight obtained, they can directly into a tactile stimuli by directly mapping be guided around to identify lines and object each pixel in the image on a touch sensitive area of outlines, to orient in a room depending on the light the body. This is possible because the body has a of an object. Other researchers have developed large number of tactile sensors in the skin, and for a implantable photovoltaic retinal prosthesis [27] [28], picture with low resolution, a 1:1 mapping between but in this case the implantation procedure proved a pixel in the image and a point on the skin can be difficult, because of the difficult supply system.