IMAGE PROCESSING ON A MOBILE PLATFORM A thesis submitted to the University of Manchester for the degree of Master of Science in the Faculty of Engineering and Physical Sciences 2009 By Samantha Patricia Bail School of Computer Science Contents Abstract 5 Declaration 6 Copyright 7 Acknowledgements 8 1 Introduction 9 1.1 Description of the Project . 9 1.2 Motivation . 10 1.3 Main Objectives . 11 1.4 Scope . 11 1.5 Dissertation Overview . 13 2 Project Background and Literature Review 15 2.1 Overview . 15 2.2 Mobile Platforms . 15 2.3 Mobile Phones as Assistive Devices . 18 2.4 Image Processing and Object Detection . 18 2.5 Related Work . 19 2.6 Analysis of Methods for Object Detection . 23 2.7 Factor Graph Belief Propagation . 24 2.8 Chapter Summary . 30 3 Application Design 31 3.1 Overview . 31 3.2 Requirements Analysis . 31 3.3 Software Architecture . 33 2 3.4 Image Processing Methods and Algorithms . 36 3.5 Training Images . 44 3.6 Issues Affecting the System Performance . 45 3.7 Chapter Summary . 46 4 System Implementation 47 4.1 Overview . 47 4.2 Implementation Tools . 47 4.3 Image Capturing . 48 4.4 Phase One: Feature Extraction . 49 4.5 Phase Two: Object Recognition . 53 4.6 Result Output . 55 4.7 Optimisation for Symbian S60 devices . 55 4.8 Chapter Summary . 56 5 Testing 57 5.1 Overview . 57 5.2 Description of the Testing Procedures . 57 5.3 System Performance Evaluation . 60 5.4 Chapter Summary . 61 6 System Evaluation 62 6.1 Overview . 62 6.2 Analysis of the Research Methodology . 62 6.3 Review of the Project Plan . 64 6.4 Improvements . 65 6.5 Chapter Summary . 66 7 Conclusion and Future Work 67 7.1 Project Summary . 67 7.2 Future Work . 69 Bibliography 72 A Listings 75 3 List of Figures 1.1 Two exit signs according to BS 5499-4 . 13 2.1 Worldwide smartphone sales to end users 2008 . 17 2.2 Example of a factor graph . 25 3.1 Class diagram showing the organisation of the application classes 35 3.2 State diagram for the emergency exit sign recognition software . 36 3.3 Sobel kernels used for horizontal and vertical derivatives . 37 3.4 Four examples of emergency exit signs captured with a phone camera 45 4.1 Individual steps of edge detection . 51 4.2 Two examples of binary sign templates . 54 4 Abstract Emergency exit signs are an indispensable part of any safety precautions for public buildings. In case of an emergency, they indicate safe escape routes and emergency doors, using an internationally recognizable sign: A green and white sign with icons showing a running person, a door, an arrow pointing into the direction of the escape route and the word Exit (or other words describing an emergency exit), in different combinations. These signs can be easily detected and interpreted by sighted people, but are unsuitable for visually impaired persons who cannot rely on visual indicators. This project deals with the issues of recognizing emergency exit signs with a mobile device. It describes the development of a piece of software that runs on a Symbian OS smartphone and can be used to detect emergency exit signs using the phone’s camera. In case of a detection, the device indicates this through an acoustic signal and, if an arrow is present on the sign, the software specifies the direction through text output. In order to achieve fast processing times, the study also deals with the low computing power of smartphones. The chosen approach is based on belief prop- agation on factor graphs, a method drawn from statistics, which is used in com- bination with other image processing tasks such as template matching. While the success of an efficient implementation depends strongly on the observance of necessary optimisations in both the choice of algorithms and coding practice, the general feasibility of image processing on the chosen mobile platform is demon- strated by this project. 5 Declaration No portion of the work referred to in this thesis has been submitted in support of an application for another degree or qualification of this or any other university or other institute of learning. 6 Copyright i. Copyright in text of this dissertation rests with the Author. Copies (by any process) either in full, or of extracts, may be made only in accordance with instructions given by the Author. Details may be obtained from the appropriate Graduate Office. This page must form part of any such copies made. Further copies (by any process) of copies made in accordance with such instructions may not be made without the permission (in writing) of the Author. ii. The ownership of any intellectual property rights which may be described in this thesis is vested in the University of Manchester, subject to any prior agreement to the contrary, and may not be made available for use by third parties without the written permission of the University, which will prescribe the terms and conditions of any such agreement. iii. Further information on the conditions under which disclosures and exploita- tion may take place is available from the Head of the School of Computer Science. 7 Acknowledgements I would like to thank my supervisor Dr Tim Morris for his support and helpful guidance throughout all stages of the project, as well as Dr David Rydeheard who would always provide me with good advice whenever I came across any difficulties on the course. Many thanks to Marcus Groeber for his advice regarding Symbian, and to Volodymyr Ivanchenko for providing the Crosswatch application for testing purposes. Thanks to my family and especially my mother and grandfather who sup- ported me during my never ending studies (I’m offto the next round). My thanks goes out to Simon for his incredible patience, as well as his family for all their help. Thanks to all my friends in the UK and in Germany, especially to Dr B., and to my housemates for their motivational talks. I would also like to mention all the students who spent so many days (and nights) in the MSc lab and provided me with advice and chats. Danke. 8 Chapter 1 Introduction 1.1 Description of the Project Visual signs provide a means of orientation for sighted people within unfamiliar locations such as offices, hospitals and other public buildings. Particularly in emergency situations, emergency exit signs point the way to important escape routes, thus making them a legal requirement for buildings of a certain size. However, for people with visual impairments, these vital resources cannot be utilized as a guidance aid. Using a mobile tool to detect these emergency signs and output the necessary information in acoustic form can make them accessible to people who cannot rely on their eyesight to recognize visual objects. This can be helpful in unknown or complex buildings, when the escape routes cannot be memorized and there is no other person immediately available that could provide guidance to find the right escape route. This project will carry out research into the feasibility of such a guidance system, analyse different methods to achieve the task and describe a way of im- plementing the system on a mobile platform. Upon completion of the work, we will have gained insights into an efficient implementation of computationally de- manding procedures such as computer vision algorithms on mobile devices with low processing power. In addition, the software will be a demonstration of how modern technology such as the smartphone platform with its wide scope of pos- sible applications can be used to assist blind and visually impaired people. 9 1.2. MOTIVATION 10 1.2 Motivation There are over two million people in the UK living with significant sight loss, out of which over 300.000 are officially registered as blind or partially sighted [RNI]. Numerous tools and techniques are available to blind people to help them complete everyday tasks more safely and with greater independence. Such assis- tance can come in the form of guide dogs and white canes (for navigating around unfamiliar obstacles in public spaces) but also in lesser known forms, an example of which are digital water level sensors that sound an alarm when a vessel is full. The use of modern information technology has become increasingly popular in the past few years, with companies providing mobile talking book players, braille output devices for mobile phones and text-to-speech software for computers. To sighted people, many everyday tasks such as locating exit signs in public places are hardly thought about; it is something that is done almost subcon- sciously. However, for a blind or partially sighted person, not being able to identify the quickest and safest way out of a building can have serious, poten- tially dangerous consequences. It is this particular problem that will form the core of this study. While mainly based in the discipline of computer vision, this project has two important aspects: First, adapting modern technology in order to provide as- sistance to visually impaired people, without the need to produce specifically designed devices for them, which is connected to the notion of accessibility. Sec- ondly, the implementation of a computationally demanding task such as image processing on a platform with restricted computing power. This fact makes it necessary to move away from some of the traditionally used methods that prove to computationally demanding, and explore novel approaches, simplified versions of algorithms and approximations that can be used to achieve a lightweight im- plementation. We acknowledge that the idea of using computer vision for visually impaired people is not ground breaking,however, it still is rarely seen on mobile platforms.