Bachelor Thesis Controlling a Mindstorms Robot using Image Processing on Android Devices Shasindran Poonudurai Martikel-Nr: 700461 Studiengang Mechatronik Fakültät Technik Hochschule Reutlingen Betreuer: Prof.-Dr. Stefan Mack Prof.-Dr. Jens Weiland Contents List of Figures ........................................................................................................................................ 4 Acknowledgements .............................................................................................................................. 6 Chapter 1: .............................................................................................................................................. 7 Introduction ............................................................................................................................................ 7 1.1 Lego Mindstorms NXT ............................................................................................................... 9 1.1.1 NXT Intelligent Brick ........................................................................................................... 9 1.1.2 The Motors ......................................................................................................................... 10 1.1.3 The Sensors ...................................................................................................................... 11 1.2 Image Processing for Mobile Robots .................................................................................... 13 1.3 Android and Smartphone ........................................................................................................ 14 1.3.1 Mobile Operating System (OS) ....................................................................................... 15 1.3.2 Smartphone Hardware ..................................................................................................... 15 1.3.3 Development Environment for Apps .............................................................................. 16 1.3.4 Creating an Android Application Project ....................................................................... 17 1.3.5 Android Activity Lifecycle ................................................................................................. 19 1.4 OpenCV ..................................................................................................................................... 23 1.4.1 Computer Vision ............................................................................................................... 23 1.4.2 What is OpenCV? ............................................................................................................. 26 1.4.3 OpenCV Structure ............................................................................................................ 27 1.5 System Design ......................................................................................................................... 28 1.6 Example Applications outside NXT-Projects. ...................................................................... 30 Chapter 2: ............................................................................................................................................ 31 2.0 Lego NXT System ........................................................................................................................ 31 2.1 Communication via LCP commands ..................................................................................... 31 2.1.1 General protocol architecture .......................................................................................... 32 2.2 Bluetooth Connection with Lego NXT ................................................................................... 33 Chapter 3: ............................................................................................................................................ 39 3.0 OpenCV for Android .................................................................................................................... 39 3.1 OpenCV Java API .................................................................................................................... 40 3.2 OpenCV Native ......................................................................................................................... 41 3.3 OpenCV Tutorial 0: Android Camera .................................................................................... 42 3.5 Object Detection via Color Segmentation ............................................................................ 57 3.5.1 Color and Color Models ................................................................................................... 57 2 3.5.2 Image Acquisition and Image Enhancement ................................................................ 59 3.5.3 Pixel Classification ............................................................................................................ 60 3.5.4 Image Labeling .................................................................................................................. 60 3.5.5 Feature Extraction ............................................................................................................ 61 Chapter 4: ............................................................................................................................................ 63 4.0 Controlling NXT via Android Smartphone ................................................................................ 63 4.1 Implementation of NXT Sensors ............................................................................................ 65 4.2 Application ................................................................................................................................. 67 4.2.1 Controlling via Collision Avoidance ................................................................................ 69 4.2.2 Controlling Via Image Processing .................................................................................. 71 4.2.3 Test and Optimization ...................................................................................................... 73 Chapter 5: ............................................................................................................................................ 75 5.0 Outlook .......................................................................................................................................... 75 Summary .............................................................................................................................................. 76 References .......................................................................................................................................... 77 3 List of Figures Figures 1. 1: Components of LEGO Mindstroms NXT autonomous Robot ................................. 8 Figures 1. 2: Components of Lego Mindstorms NXT 2.0 [1] .......................................................... 9 Figures 1. 3: NXT Intelligent Brick and Components .................................................................... 10 Figures 1. 4: Structure of NXT Servomotor [1] ............................................................................... 10 Figures 1. 5: The Android App Project wizard in Eclipse ............................................................. 18 Figures 1. 6: The Activity Lifecycle of an Android application [8] ............................................... 20 Figures 1. 7: An illustration of the Activity lifecycle, expressed as a step pyramid. [9] ............ 21 Figures 1. 8: Comparison between a human‘s eye and a digital camera. [12] ......................... 24 Figures 1. 9: To a computer, the eagle’s eye is just a grid of numbers. .................................... 25 Figures 1. 10: The basic structure of OpenCV [11] ....................................................................... 27 Figures 1. 11:. System Design ......................................................................................................... 29 Figures 2.1: Communication block diagram.[15] ........................................................................... 21 Figures 2.2 : General protocol architecture[15] .............................................................................. 32 Figures 2.3: Bluetooth protocol package[15] .................................................................................. 35 Figures 3. 1: The illustration of software stacks for Android application that use OpenCV for Android[19] .......................................................................................................................................... 39 Figures 3. 2: Android application using the Java API[20] ............................................................. 40 Figures 3. 3 Android application using the Android NDK[20] ....................................................... 41 Figures 3. 4: UML (Unified Modeling Language) Class Diagram of Tutorial 0 .......................... 44 Figures 3. 5: Graph frame rate against Image Size ...................................................................... 53 Figures 3. 6: Graph frame rate against Image Processing Methods. ......................................... 54 Figures 3. 7: Graph frame rate against illuminance ...................................................................... 55 Figures 3. 8: The basic steps of Image processing to detect an object via