Improving Perception from Electronic Visual Prostheses Justin Robert Boyle BEng (Mech) Hons1 Image and Video Research Laboratory School of Engineering Systems Queensland University of Technology Submitted as a requirement for the degree of Doctor of Philosophy February 2005 Keywords image processing, visual prostheses, bionic eye, artificial human vision, visual perception, subjective testing, visual information ii Abstract This thesis explores methods for enhancing digital image-like sensations which might be similar to those experienced by blind users of electronic visual prostheses. Visual prostheses, otherwise referred to as artificial vision systems or bionic eyes, may operate at ultra low image quality and information levels as opposed to more common electronic displays such as televisions, for which our expectations of image quality are much higher. The scope of the research is limited to enhancement by digital image processing: that is, by manipulating the content of images presented to the user. The work was undertaken to improve the effectiveness of visual prostheses in representing the visible world. Presently visual prosthesis development is limited to animal models in Australia and prototype human trials overseas. Consequently this thesis deals with simulated vision experiments using normally sighted viewers. The experiments involve an original application of existing image processing techniques to the field of low quality vision anticipated from visual prostheses. Resulting from this work are firstly recommendations for effective image processing methods for enhancing viewer perception when using visual prosthesis prototypes. Although limited to low quality images, recognition of some objects can still be achieved, and it is useful for a viewer to be presented with several variations of the image representing different processing methods. Scene understanding can be improved by incorporating Region-of-Interest techniques that identify salient areas within images and allow a user to zoom into that area of the image. Also there is some benefit in tailoring the image processing depending on the type of scene. Secondly the research involved the construction of a metric for basic information required for the interpretation of a visual scene at low image quality. The amount of information content within an image was quantified using inherent attributes of the image and shown to be positively correlated with the ability of the image to be recognised at low quality. iii Table of Contents Abstract .................................................................................................................. iii List of Figures........................................................................................................... viii List of Tables................................................................................................................ x Statement of Original Authorship ............................................................................ xi Acknowledgements.................................................................................................... xii Publications............................................................................................................... xiii Chapter 1 Introduction ........................................................................................... 14 1.1 Overview ....................................................................................................14 1.2 Aim.............................................................................................................15 1.3 Scope ..........................................................................................................15 1.4 Thesis Structure..........................................................................................16 1.5 Contributions..............................................................................................17 Chapter 2 Image Quality and Visual Perception.................................................. 19 2.1 Introduction ................................................................................................19 2.2 Visual Perception Physiology ....................................................................20 2.3 A Visual Hierarchy Model .........................................................................22 2.3.1 Early Vision Effects ...........................................................................23 2.3.2 Cognitive Effects................................................................................31 2.4 Region-of-Interest ......................................................................................35 2.5 Visual Information .....................................................................................39 2.6 Chapter Summary.......................................................................................41 Chapter 3 Visual Prosthesis Application............................................................... 43 3.1 Overview ....................................................................................................43 3.2 General Introduction to the Application.....................................................43 3.3 Current Visual Prosthesis Research ...........................................................44 3.3.1 Retinal Systems..................................................................................45 3.3.2 Optic Nerve Systems..........................................................................47 3.3.3 Visual Cortex Systems .......................................................................48 3.4 Image Processing specifically related to Bionic Eye Projects ...................49 3.4.1 Vision Chip Developments ................................................................49 3.4.2 CCD-based Systems...........................................................................51 3.4.3 Receptive Field Modeling ..................................................................53 3.4.4 Multiple Resolution Work..................................................................54 3.5 Digital Imaging Applicable to Visual Prostheses ......................................55 3.5.1 Digital Imaging and Human Vision ...................................................55 3.5.2 Image Characteristics and Visual Understanding ..............................58 3.6 Thesis Research Questions and Approach .................................................67 3.6.1 Image Processing Requirements ........................................................68 3.6.2 Testing Method ..................................................................................69 3.7 Chapter Summary.......................................................................................71 iv Chapter 4 Recognition Performance ......................................................................72 4.1 Overview.................................................................................................... 72 4.2 Subjective Tests to Determine Useful Processing Methods ...................... 73 4.2.1 Methodology ...................................................................................... 73 4.2.2 Images Chosen ................................................................................... 74 4.2.3 Results................................................................................................ 75 4.2.4 Test Conclusions ................................................................................ 85 4.3 Subjective Tests to Determine Influence of Image Type........................... 87 4.3.1 Methodology ...................................................................................... 87 4.3.2 Images Chosen ................................................................................... 88 4.3.3 Results................................................................................................ 90 4.3.4 Test Conclusions ................................................................................ 96 4.4 Chapter Conclusions .................................................................................. 97 Chapter 5 Quantifying Information Content ........................................................98 5.1 Introduction................................................................................................ 98 5.2 Perceived Information Content in Images................................................ 100 5.2.1 Images Used..................................................................................... 100 5.2.2 Multidimensional Visual Information Model: ................................. 101 5.2.3 Test Method ..................................................................................... 103 5.2.4 Test Participants and Instructions .................................................... 104 5.2.5 Test Results ...................................................................................... 105 5.2.6 Strong Visual Information Rankings ............................................... 111 5.2.7 Test Conclusions .............................................................................. 113 5.3 Information Content Model Fitting.......................................................... 113 5.3.1 Possible Image Attributes for a Visual Information Metric............. 113 5.3.2 Metric Development for a Specific Image Quality Class ................ 118 5.3.3 Information Content Metric for all Image Quality Classes.............. 127 5.4 Correlations