A PROOF OF CONCEPT FOR PEOPLE TRACKING APPLICATION USING SMART-SPACES, ONLING SERVICES AND COMPUTER VISION Frank Wickström Master of Science Thesis Supervisor: Prof. Johan Lilius Advisor: Dr. Sébastien Lafond Department of Information Technologies Åbo Akademi University March 2013 ABSTRACT People share more information about themselves and other than ever before and most of the data stored by services are not used to the extent that they could be. This has led to an increased popularity in data mining and knowledge extraction, where existing data is being used to create intelligent systems and artificial intelligence. Smart-spaces, where data is used to create an environment where devices and humans interact are also on the rise as the internet of things becomes more popular and devices are becoming aware of their environments. In this thesis we present an implementation of how to track people given existing sources of data. We cover how we combine data sources, and make them aware of each other by connecting them to a single database. The created software is able to track people based on information from web services as well as IP cameras, and is managed via a web interface. We have come to the conclusion that the information already available today could be used to create new and smart applications if combined and processed the right way. Keywords: people tracking, smart-spaces, data mining, smart-m3 i CONTENTS Abstract i Contents ii List of Figures iv 1 Introduction 1 1.1 AMEBA project . .1 1.2 Problem statement . .1 1.3 Thesis structure . .2 2 People tracking 4 2.1 Tracking techniques . .4 2.1.1 Human recognition . .5 2.1.2 Electronic tracking . .6 2.1.3 Location prediction and data mining . .7 2.1.4 Web services . .8 2.2 Ethics, intrusiveness and ignorance . .8 3 Choosing and combining technologies 10 3.1 Application platform . 10 3.1.1 Web framework . 11 3.1.2 Web application and background tasks . 11 3.1.3 Web server . 12 3.2 Online APIs . 13 3.3 Databases and media storage . 14 3.3.1 Smart-M3 . 14 3.3.2 SQL and NoSQL combination . 15 3.3.3 Image and video storage . 16 3.4 Face detection and recognition . 16 4 Implementation 18 4.1 Platform . 18 4.2 Krang Demonstrator . 18 ii 4.2.1 Functionality . 19 4.3 Implementation overview . 19 4.3.1 Directory hierarchy . 20 4.4 Code used from external sources . 21 4.4.1 Python . 21 4.4.2 HTML and JavaScript . 23 4.4.3 Other software . 23 4.5 Back-end design of Krang . 24 4.5.1 Database model . 25 4.5.2 Ontology model . 29 4.5.3 Face detection and recognition . 36 4.5.4 Foursquare . 39 4.5.5 Google Calendar . 40 4.5.6 Åbo Akademi Department of Information Technology . 41 4.6 Front-end design of Krang . 44 4.6.1 Profile . 44 4.6.2 Settings . 46 4.7 Use Cases . 48 4.7.1 Complementing location information . 48 4.7.2 Event notification system . 50 4.7.3 Information display with user data . 52 4.8 Web SIB Explorer and Smart-M3 Tutorial . 53 5 Discussion 55 5.1 Conclusions . 55 5.2 Future work . 56 Bibliography 57 Swedish summary 62 Appendices 63 A Configurations 64 A.1 uWSGI . 64 A.2 Nginx . 64 B Web scraping 66 B.1 research.it.abo.fi login . 66 iii LIST OF FIGURES 2.1 RFID tag used at 25C3 CCC conference [1]. .7 4.1 Back-end servers and services with corresponding Django apps. 20 4.2 Complete database model, without database fields. 25 4.3 Core app database model. 26 4.4 Gcal app database model. 27 4.5 Foursquare app database model. 27 4.6 Freqo app database model. 28 4.7 Krangs ontology classes and their relationships. 31 4.8 Person class, with object and data properties. 31 4.9 Room class, with object and data properties. 32 4.10 Floor class, with object and data properties. 32 4.11 Building class, with object and data properties. 32 4.12 Phone class, with object and data properties. 33 4.13 Camera class, with object and data properties. 33 4.14 VideoData class, with object and data properties. 33 4.15 GoogleCalendarEvent class, with object and data properties. 34 4.16 FoursquareCheckin class, with object and data properties. 34 4.17 UserLocation class, with object and data properties. 34 4.18 CameraLocation class, with object and data properties. 35 4.19 FoursquareAccount class, with object and data properties. 35 4.20 GoogleCalendarAccount class, with object and data properties. 35 4.21 Original Local Binary Pattern description [2]. 37 4.22 Circular Local Binary Pattern description [3]. 38 4.23 Flow diagram for OAuth 2.0 [4]. 39 4.24 Flow diagram for Foursquare Real-Time API. 40 4.25 Door displays in the ICT building. 42 4.26 User profile page in Krang with functional elements marked. 45 4.27 User settings page in Krang. 47 4.28 Flowchart of complementing location information use case. 48 4.29 Flowchart of event notification Celery task. 50 4.30 Flowchart of a Celery task to change a status display using multiple data sources. 52 iv 4.31 Web SIB Explorer Class Tree view. 54 v 1 INTRODUCTION Smart-spaces and environments are becoming more popular as we see a growth in the use of mobile devices and sensors. And with a new interest in the Internet of things, we might soon see everyday objects such as chairs and door talking to each other. The more data are retrieved by sensors the more data storage and processing needed to get the most out of the information. Data processing however is mostly done by the owner of the data, and combining data and data mining is becoming the natural next step to create more intelligent software. In this thesis we introduce a way of collecting information from several data sources that feature location data. By combining information from these sources we create an application which can track people’s current location with more precision than by using a single source of location data. 1.1 AMEBA project This thesis was done within the AMEBA (Agent based Management of EmBedded dAta reservers) project. The AMEBA project aim to show the advantages of three- dimensional data storage by using distributed agent-based data management. This thesis aims to create a demonstrator for showing some of the features that can be ac- complished by using such data management. 1.2 Problem statement As storage grows bigger and cheaper, companies are starting to store more and more information than what was possible before. Most of the additional information is no longer created by humans, but by machines. Information such as logs, meta-data and sensory information are being added to content created by human, and is putting a higher demand on storage than ever before. However one of the problems with the 1 large amounts of data is that it is never processed in any way. Every time users upload an image taken by their mobile phone to a photo sharing application it may contain a lot more information than just the image. If the GPS was enabled, location informa- tion might be embedded in the image, and other meta-data such as ISO and exposure information might be included as well. Facebook also adds information about a users location to status update, if the user has allowed it. All of this extra information is only used if the service storing deemed it fit for processing. Otherwise they are stored, and most likely not used until they ultimately are deleted. We propose a system letting a third party process the meta information, in our case location information. By combining location data from several sources, more intelli- gent people tracking is possible which could include predictions of peoples next move. The data can also come from sources such as video surveillance systems, in which several terabytes can be stored every day. The data from such systems might stay un- touched as surveillance system operators might never need access to it. We propose an application processing the video stream and extracting addition information, such as human recognition and movement prediction. By running face recognition algorithms on the video stream, people tracking could be done automatically. Knowing the direc- tion the camera is pointed in and tracking at what direction a person is walking, one could follow a person around without human interaction. The databases storing data in larger applications are very non interactive as pro- grams often have to query information, before they can get data to process. If applica- tions are to respond quickly to new information added, it will have to constantly poll for the information, which increases the load on the database server. The solution we purpose to lowering the response time is to use the Smart-M3 triple-store database, which supports subscriptions and can push information to an application when the ap- propriate data has been added to the database. Something which puts less strain on the database, and makes response times smaller. 1.3 Thesis structure Chapter 2 includes current techniques used for tracking people, as well as the intru- siveness of tracking people with such methods. We distinguished four main methods which are used for tracking. Human recognition, which includes using computers for recognizing people. Electronic tracking, which uses electronic devices such as GPS for 2 tracking. Location prediction, which can be used when enough data exists and lastly web services which are the newcomer when it comes to tracking, and offers several location based services for tracking people.