Raspberry Pi: a Smart Video Monitoring Platform David Emanuel Ribeiro Gaspar Thesis to obtain the Master of Science Degree in Embedded Systems and Computer Engineering Supervisor: Prof. Nuno Filipe Valentim Roma Examination Committee: Chairperson: Prof. Miguel Nuno Dias Alves Pupo Correia Supervisor: Prof. Nuno Filipe Valentim Roma Member of the Committee: Prof. Renato Jorge Caleira Nunes November 2014 2 Acknowledgments My parents. My family. My friends. To Dr. Nuno Roma for all the patience, help and support. i Abstract Recent computing trends have led to the release of very low-cost (yet highly capable) single- board computing platforms. The usage of such devices - in this case, the Raspberry Pi - con- nected to an inexpensive Universal Serial Bus (USB) webcam allows the implementation of a very low-cost video processing station. The computational capabilities offered by the Raspberry Pi, combined with the good image quality that is made available by most off-the-shelf webcams, allow the implementation of smart monitoring platforms for a broad range of applications. By combining such a hardware platform with a software architecture consisting of inter-changeable modules, to interface the webcam and to process the gathered data, it is possible to implement a vast set of systems for autonomous video analysis. Under this scenario, the proposed sys- tem is able to perform movement detection and heat-map calculation over a period of time, by analysing a video feed provided by a physically fixed camera. Several possible alternatives for the implementation of each module are presented and discussed, together with a presentation and analysis of the system performance and its real-world applicability. Keywords Smart Video Monitoring, Video Surveillance, Motion Detection, Single-Board Computer, Low- Cost Embedded Platform iii Resumo As tendenciasˆ actuais no mundo da computac¸ao˜ temˆ levado a` disponibilizac¸ao˜ de platafor- mas de computacc¸ao˜ de board unica´ que, apesar do seu reduzido custo, apresentam boas ca- pacidades de computac¸ao.˜ O uso destes dispositivos - neste caso concreto, a Raspberry Pi - ligado a uma camaraˆ USB permite a implementac¸ao˜ de uma estac¸ao˜ de tratamento v´ıdeo de baixo custo. A Raspberry Pi, combinada com a boa qualidade de imagem da camara,ˆ per- mite a implementac¸ao˜ de plataformas inteligentes de monitorizac¸ao˜ para o mais variado leque de usos. Combinando esta plataforma hardware com uma arquitectura software assente em modulos´ inter-conectaveis´ para ligar a` web-cam e tratar os dados obtidos, e´ poss´ıvel implemen- tar um vasto leque de sistemas autonomos´ de tratamento de video. O sistema proposto e´ capaz de realizar detecc¸ao˜ simples de movimentos e calculo´ de mapas termicos´ atraves´ da analise´ do v´ıdeo proveniente de uma camaraˆ fixa. Varias´ alternativas para a implementac¸ao˜ de cada modulo´ sao˜ apresentadas, juntamente com uma analise´ a` performance do sistema e a sua viabilidade no mundo real. Palavras Chave Monitorizac¸ao˜ Inteligente de Video, Video-Vigilancia,ˆ Detecc¸ao˜ de Movimento, Computador de Placa Unica,´ Plataforma Embebida de Baixo-Custo, v Contents 1 Introduction 1 1.1 Motivation . .2 1.2 Objectives . .4 1.3 Requisites . .6 1.4 Document Structure . .6 2 Existing Solutions 9 2.1 Security Video-Surveillance . 10 2.2 Sports Data Gathering . 12 2.3 Wilderness Cameras . 13 2.4 Human Monitoring Systems . 14 2.5 Community Approaches . 15 2.6 Discussion . 15 3 Related Technology 19 3.1 Hardware Platforms & Peripherals . 20 3.1.1 Processing Platform . 20 3.1.2 Camera . 22 3.2 Detection Algorithm . 25 3.3 Software Libraries . 28 3.3.1 MATLAB Computer Vision System Toolbox . 29 3.3.2 OpenCV . 29 3.3.3 Motion . 30 3.4 Discussion . 30 3.4.1 Hardware . 30 3.4.2 Detection algorithm . 33 3.4.3 Software Libraries . 33 4 Proposed Architecture 35 4.1 Hardware Layer . 36 vii Contents 4.1.1 Processing Board & Resources . 36 4.1.2 Peripherals . 37 4.2 Software Layer . 37 4.2.1 Capture Module . 37 4.2.1.A Capture from USB Web-cam . 38 4.2.1.B Capture from Local File . 39 4.2.2 Processing Module . 40 4.2.2.A Video Store . 40 4.2.2.B Simple Movement Detection . 41 4.2.2.C Activity Mapping . 42 4.2.2.D Activity Mapping with Dynamic Reference Updating . 44 4.2.3 Communication Protocol . 45 4.2.3.A Initialization . 45 4.2.3.B Main Cycle . 46 4.2.3.C Finishing . 46 5 Implementation 47 5.1 General Structure . 48 5.2 Capture Module Implementation . 52 5.2.1 Capture from USB Device . 52 5.2.2 Capture from Sequence File . 52 5.3 Processing Module Implementation . 52 5.3.1 Video Storage . 52 5.3.2 Motion Detection . 53 5.3.3 Activity Mapping . 53 5.3.4 Activity Mapping with Dynamic Reference Updating . 55 5.4 Communication Protocol . 56 5.4.1 Execution Parameters Structure . 57 5.4.2 Initial Negotiation . 57 5.4.3 Main Cycle . 58 5.4.4 Finishing Execution . 59 6 Experimental Results 61 6.1 Performance . 62 6.1.1 Movement Detection . 62 6.1.2 Activity Mapping . 63 6.1.3 Activity Mapping with Dynamic Reference Updating . 64 6.2 Real-World Behaviour . 65 viii Contents 6.2.1 Movement Detection . 65 6.2.2 Activity Mapping . 66 6.2.2.A Sequence 1 . 66 6.2.2.B Sequence 2 . 67 6.2.3 Activity Mapping with Dynamic Reference Updating . 67 6.2.3.A Sequence 1 . 67 6.2.3.B Sequence 2 . 68 7 Future Work 71 8 Conclusions 73 A Appendix A 79 B Appendix B 81 C Appendix C 83 D Appendix D 85 E Appendix E 87 ix Contents x List of Figures 1.1 Raspberry Pi board. .3 1.2 Example of heat activity map. .5 2.1 Swann DVR8-3425 - an example of a video surveillance system with a set of cam- eras and a central unit. 10 2.2 Logitech Alert example configuration. 11 2.3 Heat map generated by a player in a football match, showing the areas of the pitch he spent the most time in. 12 2.4 An example of a motion-triggered wilderness camera. 13 2.5 Human tracking by RetailNext. Note the humans enclosed in purple boxes and, in green, the path they took. 14 3.1 Eee PC 4G, the first netbook released by ASUS. 21 3.2 Intel Galileo, an example of an x86 single-board computer. 21 3.3 BeagleBone Black Revision C board. 22 3.4 The Raspberry Pi camera module. 23 3.5 Logitech C270 web-cam. 23 3.6 Frame Differencing execution flowchart. 26 3.7 A difference frame with two visible cars. Extracted from http://www.mathworks. com/discovery/object-detection.html........................ 26 3.8 A photograph taken with a high ISO setting. 27 4.1 Hardware schematic of the system. 36 4.2 Software schematic of the system. 37 4.3 USB Capture Module execution flowchart. 39 4.4 Capture From File Module execution flowchart. 39 4.5 Video Store Processing Module execution flowchart. 40 4.6 Simple Movement Detection Processing Module execution flowchart. 41 4.7 Colour scheme showing warm and cold colours. 42 4.8 Activity Mapping Processing Module execution flowchart. 43 xi List of Figures 4.9 Activity Mapping with Dynamic Reference Updating Processing Module execution flowchart. 44 4.10 Capture to Processing Module Communication.