Lightweight Algorithms for Depth Sensor Equipped Embedded Devices Henry Zhong A thesis in fulfillment of the requirements for the degree of Doctor of Philosophy School of Computer Science and Engineering Faculty of Engineering The University of New South Wales May 2017 THE UNIVERSITY OF NEW SOUTH WALES Thesis/Dissertation Sheet Surname or Family name: Zhong First name: Henry Other name/s: Abreviation for degree as given in the University calendar: PhD School: School of Computer Science and Engineering Faculty: Faculty of Engineering Title: Lightweight Algorithms for Depth Sensor Equipped Embedded Devices Abstract 350 words maximum Depth sensors have appeared in a variety of embedded devices. It can be found in tablets, smartphones and web cameras. At the time of writing, research into depth sensor equipped embedded devices is still in its infancy. This has led to some key questions: What kinds of applications can take advantage of depth sensor equipped embedded devices and the question of efficiently implementing algorithms on resource-constrained embedded devices? The purpose of this thesis is address these questions. We do so by presenting 3 prototype systems and accompanying lightweight algorithms. The prototypes demonstrate example application for the use of depth sensors in pervasive computing. The novel algorithms make use of depth data to solve common problems, while being lightweight enough on resource consumption to operate on embedded devices. Our algorithms are lightweight because we use simpler features compared to existing algorithms. We do this while achieving better results by several metrics compared to the current state of the art. It is hoped the presented work enlighten the reader on the possible applications for depth sensor equipped embedded devices. The 3 prototypes target 3 major areas of research. First is QuickFind: which contains an algorithm to perform fast segmentation and object detection using a depth sensor, it is applied to a prototype augmented reality assembly aid. The second is WashInDepth: which contains a fast hand gesture recognition algorithm for monitoring correct handwashing. The third is the VeinDeep: for fast vein pattern recognition using a depth sensor, we use it to secure depth sensor equipped embedded devices. It is the first instance where depth sensors have been used for vein pattern recognition. Declaration relating to disposition of project thesis/dissertation I hereby grant to the University of New South Wales or its agents the right to archive and to make available my thesis or dissertation in whole or in part in the University libraries in all forms of media, now or here after known, subject to the provisions of the Copyright Act 1968. I retain all property rights, such as patent rights. I also retain the right to use in future works (such as articles or books) all or part of this thesis or dissertation. I also authorise University Microfilms to use the 350 word abstract of my thesis in Dissertation Abstracts International (this is applicable to doctoral theses only). Signature Witness Date The University recognises that there may be exceptional circumstances requiring restrictions on copying or conditions on use. Requests for restriction for a period of up to 2 years must be made in writing. Requests for a longer period of restriction may be considered in exceptional circumstances and require the approval of the Dean of Graduate Research. FOR OFFICE USE ONLY Date of completion of requirements for Award ii Originality Statement `I hereby declare that this submission is my own work and to the best of my knowl- edge it contains no materials previously published or written by another person, or substantial proportions of material which have been accepted for the award of any other degree or diploma at UNSW or any other educational institution, except where due acknowledgement is made in the thesis. Any contribution made to the research by others, with whom I have worked at UNSW or elsewhere, is explicitly acknowledged in the thesis. I also declare that the intellectual content of this thesis is the product of my own work, except to the extent that assistance from others in the project's design and conception or in style, presentation and linguistic expression is acknowledged.' Henry Zhong January 31, 2017 Copyright Statement `I hereby grant the University of New South Wales or its agents the right to archive and to make available my thesis or dissertation in whole or part in the University libraries in all forms of media, now or here after known, subject to the provisions of the Copyright Act 1968. I retain all proprietary rights, such as patent rights. I also retain the right to use in future works (such as articles or books) all or part of this thesis or dissertation. I also authorise University Microfilms to use the 350 word abstract of my thesis in Dissertation Abstract International (this is applicable to doctoral theses only). I have either used no substantial portions of copyright material in my thesis or I have obtained permission to use copyright material; where permission has not been granted I have applied/will apply for a partial restriction of the digital copy of my thesis or dissertation.' Henry Zhong January 31, 2017 Authenticity Statement `I certify that the Library deposit digital copy is a direct equivalent of the final officially approved version of my thesis. No emendation of content has occurred and if there are any minor variations in formatting, they are the result of the conversion to digital format.' Henry Zhong January 31, 2017 Abstract Depth sensors have appeared in a variety of embedded devices. This includes tablets, smartphones and web cameras. This has provided a new mode of sensing, where it is possible to record an image and the distance to everything in the image. Some pervasive computing applications have taken advantage of depth sensors, such as crowd sourced 3D indoor mapping. However, research into this area is still in its infancy, some questions remain before widespread adoption. These questions are: What kinds of applications can take advantage of depth sensor equipped embed- ded devices and the question of efficiently implementing algorithms on resource- constrained embedded devices? The purpose of this thesis is address these questions. We do so by presenting 3 prototype systems and accompanying lightweight algorithms. Each algorithm uses depth sensors to overcome problems in visual pattern matching and are lightweight enough to run on embedded platforms. We do this while achieving better results by several metrics compared to the current state of the art. These metrics include pattern matching accuracy, asymptotic complexity, run time and memory use. The first contribution of this thesis is QuickFind, for fast segmentation and object detection algorithm, it is applied to a prototype augmented reality assembly aid. We test it against two related algorithms and implement our prototype on a Raspberry Pi. The two related algorithms are: Histogram of Oriented Gradients (HOG), a pop- ular object detection algorithm. Histogram of Oriented Normal Vectors (HONV), a state of the art algorithm specifically designed for use with depth sensors. Our test data is the RGB-D Scenes v1 dataset consisting of 6 object classes, in 1434 scenes of domestic and office environments. On our test platform QuickFind achieved the best results with 1/18 run time, 1/18 power use, 1/3 memory use compared to HOG and 1/279 run time, 1/279 power use, 1/15 memory use compared to HONV. Quick- Find has a lower asymptotic upper bound and almost double the average precision compared to HOG and HONV. The second contribution of this thesis is WashInDepth, for fast hand gestures recog- nition, it is applied to a prototype to monitor correct hand washing. We test it i against HOG, HONV and implement our prototype on a Compute Stick. WashIn- Depth is an extension of QuickFind. Segmentation is replaced with a background removal step. QuickFind features are used to perform hand gesture recognition, based on video recorded from a depth sensor. We test with 15 participants with 3 videos each for a total of 45 videos. WashInDepth achieved the best results with average of 94% accuracy and a run time of 11 ms. HOG achieved 86% average accuracy and 19 ms average run time. HONV achieved 88% average accuracy and 22 ms average run time. All 3 algorithms had average memory usage within 4 KiB of each other. The third contribution of this thesis is VeinDeep. VeinDeep performs identifica- tion using vein pattern recognition. We repurpose depth sensors for this task. As far as we are aware, it is the first instance where depth sensors have been used for this purpose. The prototype application for VeinDeep is designed for securing smartphones with integrated depth sensor. As such devices are not widely available at the time of writing, the system is simulated on a Compute Stick with attached depth sensor. We test VeinDeep against two related algorithms The two related algorithms are: Hausdorff distance, an older but popular algorithm for vein pattern recognition. Kernel distance, an algorithm more recently applied to vein pattern recognition. We test with 20 participants, 6 images per hand for a total of 240 im- ages. On our embedded platform VeinDeep achieved the best results with 1/6 run time, 2/3 memory use compared to Hausdorff distance. 1/3 run time, 1/2 memory use compared to Kernel distance. VeinDeep had precision of 0.98, recall of 0.83. At the same recall level Hausdorff distance had precision of 0.5, Kernel distance had precision of 0.9. VeinDeep also had lower average complexity compared to Hausdorff and Kernel distance. Although the prototypes in this thesis focus on three specific problems. The algo- rithms accompanying the prototypes are general purpose.