A PROOFOFCONCEPTFORPEOPLE TRACKINGAPPLICATIONUSING SMART-SPACES, ONLING SERVICES ANDCOMPUTERVISION

Frank Wickström

Master of Science Thesis Supervisor: Prof. Johan Lilius Advisor: Dr. Sébastien Lafond Department of Information Technologies Åbo Akademi University March 2013 ABSTRACT

People share more information about themselves and other than ever before and most of the data stored by services are not used to the extent that they could be. This has led to an increased popularity in data mining and knowledge extraction, where existing data is being used to create intelligent systems and artificial intelligence. Smart-spaces, where data is used to create an environment where devices and humans interact are also on the rise as the internet of things becomes more popular and devices are becoming aware of their environments. In this thesis we present an implementation of how to track people given existing sources of data. We cover how we combine data sources, and make them aware of each other by connecting them to a single database. The created software is able to track people based on information from web services as well as IP cameras, and is managed via a web interface. We have come to the conclusion that the information already available today could be used to create new and smart applications if combined and processed the right way.

Keywords: people tracking, smart-spaces, data mining, smart-m3

i CONTENTS

Abstract i

Contents ii

List of Figures iv

1 Introduction 1 1.1 AMEBA project ...... 1 1.2 Problem statement ...... 1 1.3 Thesis structure ...... 2

2 People tracking 4 2.1 Tracking techniques ...... 4 2.1.1 Human recognition ...... 5 2.1.2 Electronic tracking ...... 6 2.1.3 Location prediction and data mining ...... 7 2.1.4 Web services ...... 8 2.2 Ethics, intrusiveness and ignorance ...... 8

3 Choosing and combining technologies 10 3.1 Application platform ...... 10 3.1.1 Web framework ...... 11 3.1.2 Web application and background tasks ...... 11 3.1.3 Web server ...... 12 3.2 Online APIs ...... 13 3.3 Databases and media storage ...... 14 3.3.1 Smart-M3 ...... 14 3.3.2 SQL and NoSQL combination ...... 15 3.3.3 Image and video storage ...... 16 3.4 Face detection and recognition ...... 16

4 Implementation 18 4.1 Platform ...... 18 4.2 Krang Demonstrator ...... 18

ii 4.2.1 Functionality ...... 19 4.3 Implementation overview ...... 19 4.3.1 Directory hierarchy ...... 20 4.4 Code used from external sources ...... 21 4.4.1 Python ...... 21 4.4.2 HTML and JavaScript ...... 23 4.4.3 Other software ...... 23 4.5 Back-end design of Krang ...... 24 4.5.1 Database model ...... 25 4.5.2 Ontology model ...... 29 4.5.3 Face detection and recognition ...... 36 4.5.4 Foursquare ...... 39 4.5.5 Google Calendar ...... 40 4.5.6 Åbo Akademi Department of Information Technology . . . . 41 4.6 Front-end design of Krang ...... 44 4.6.1 Profile ...... 44 4.6.2 Settings ...... 46 4.7 Use Cases ...... 48 4.7.1 Complementing location information ...... 48 4.7.2 Event notification system ...... 50 4.7.3 Information display with user data ...... 52 4.8 Web SIB Explorer and Smart-M3 Tutorial ...... 53

5 Discussion 55 5.1 Conclusions ...... 55 5.2 Future work ...... 56

Bibliography 57

Swedish summary 62

Appendices 63

A Configurations 64 A.1 uWSGI ...... 64 A.2 Nginx ...... 64

B Web scraping 66 B.1 research.it.abo.fi login ...... 66

iii LISTOF FIGURES

2.1 RFID tag used at 25C3 CCC conference [1]...... 7

4.1 Back-end servers and services with corresponding Django apps. . . . 20 4.2 Complete database model, without database fields...... 25 4.3 Core app database model...... 26 4.4 Gcal app database model...... 27 4.5 Foursquare app database model...... 27 4.6 Freqo app database model...... 28 4.7 Krangs ontology classes and their relationships...... 31 4.8 Person class, with object and data properties...... 31 4.9 Room class, with object and data properties...... 32 4.10 Floor class, with object and data properties...... 32 4.11 Building class, with object and data properties...... 32 4.12 Phone class, with object and data properties...... 33 4.13 Camera class, with object and data properties...... 33 4.14 VideoData class, with object and data properties...... 33 4.15 GoogleCalendarEvent class, with object and data properties...... 34 4.16 FoursquareCheckin class, with object and data properties...... 34 4.17 UserLocation class, with object and data properties...... 34 4.18 CameraLocation class, with object and data properties...... 35 4.19 FoursquareAccount class, with object and data properties...... 35 4.20 GoogleCalendarAccount class, with object and data properties. . . . . 35 4.21 Original Local Binary Pattern description [2]...... 37 4.22 Circular Local Binary Pattern description [3]...... 38 4.23 Flow diagram for OAuth 2.0 [4]...... 39 4.24 Flow diagram for Foursquare Real-Time API...... 40 4.25 Door displays in the ICT building...... 42 4.26 User profile page in Krang with functional elements marked...... 45 4.27 User settings page in Krang...... 47 4.28 Flowchart of complementing location information use case...... 48 4.29 Flowchart of event notification Celery task...... 50 4.30 Flowchart of a Celery task to change a status display using multiple data sources...... 52

iv 4.31 Web SIB Explorer Class Tree view...... 54

v 1 INTRODUCTION

Smart-spaces and environments are becoming more popular as we see a growth in the use of mobile devices and sensors. And with a new interest in the Internet of things, we might soon see everyday objects such as chairs and door talking to each other. The more data are retrieved by sensors the more data storage and processing needed to get the most out of the information. Data processing however is mostly done by the owner of the data, and combining data and data mining is becoming the natural next step to create more intelligent software. In this thesis we introduce a way of collecting information from several data sources that feature location data. By combining information from these sources we create an application which can track people’s current location with more precision than by using a single source of location data.

1.1 AMEBA project

This thesis was done within the AMEBA (Agent based Management of EmBedded dAta reservers) project. The AMEBA project aim to show the advantages of three- dimensional data storage by using distributed agent-based data management. This thesis aims to create a demonstrator for showing some of the features that can be ac- complished by using such data management.

1.2 Problem statement

As storage grows bigger and cheaper, companies are starting to store more and more information than what was possible before. Most of the additional information is no longer created by humans, but by machines. Information such as logs, meta-data and sensory information are being added to content created by human, and is putting a higher demand on storage than ever before. However one of the problems with the

1 large amounts of data is that it is never processed in any way. Every time users upload an image taken by their mobile phone to a photo sharing application it may contain a lot more information than just the image. If the GPS was enabled, location informa- tion might be embedded in the image, and other meta-data such as ISO and exposure information might be included as well. Facebook also adds information about a users location to status update, if the user has allowed it. All of this extra information is only used if the service storing deemed it fit for processing. Otherwise they are stored, and most likely not used until they ultimately are deleted. We propose a system letting a third party process the meta information, in our case location information. By combining location data from several sources, more intelli- gent people tracking is possible which could include predictions of peoples next move. The data can also come from sources such as video surveillance systems, in which several terabytes can be stored every day. The data from such systems might stay un- touched as surveillance system operators might never need access to it. We propose an application processing the video stream and extracting addition information, such as human recognition and movement prediction. By running face recognition algorithms on the video stream, people tracking could be done automatically. Knowing the direc- tion the camera is pointed in and tracking at what direction a person is walking, one could follow a person around without human interaction. The databases storing data in larger applications are very non interactive as pro- grams often have to query information, before they can get data to process. If applica- tions are to respond quickly to new information added, it will have to constantly poll for the information, which increases the load on the database server. The solution we purpose to lowering the response time is to use the Smart-M3 triple-store database, which supports subscriptions and can push information to an application when the ap- propriate data has been added to the database. Something which puts less strain on the database, and makes response times smaller.

1.3 Thesis structure

Chapter 2 includes current techniques used for tracking people, as well as the intru- siveness of tracking people with such methods. We distinguished four main methods which are used for tracking. Human recognition, which includes using computers for recognizing people. Electronic tracking, which uses electronic devices such as GPS for

2 tracking. Location prediction, which can be used when enough data exists and lastly web services which are the newcomer when it comes to tracking, and offers several location based services for tracking people. Chapter 3 describes how we chose each technology in the demonstrator, and how a combination of them can be made as well as how they complement each other. We first describe the web platform used in our application and how we can connect online services and their APIs to the application. We also show how we can store data from online services as well as other data sources. Lastly we describe how we chose the face recognition algorithm used in our implementation. Chapter 4 contains the implementation of the application. We describe the structure of the project, third party softwares that was used, functionality of the application, and how all of the components were implemented. Furthermore we describe how the front- end was design as well as give three use cases of our application. Chapter 5 concludes the thesis with a discussion regarding the demonstrator and its usage. We also discuss future work that could be done to create an even more feature-rich application.

3 2 PEOPLETRACKING

People tracking is becoming more commonly used by governments, in health care and by other employers to track workers, perpetrators, patiens or suspicious behavior. This thesis does not intended to cover full-fledged security systems, but it is however valuable to consider the many ways of how such systems track people. While it is usually useful to have a human monitoring security systems, in this chapter we will mainly look into automated solutions for tracking. We will go through the advantages and disadvantages of some of the most common ways of tracking people, including human recognition, electronic tracking, location prediction and web usage. Some of these methods have been used for a long time, while others are emerging as new ways of tracking people’s position. People usually value their privacy fairly high, and many mind having their picture taken and other people following them. For this reason we will also look into the ethics and intrusiveness of tracking people. We also shortly cover ignorance people have towards tracking systems, and why many do not mind people tracking them when the method of tracking is less intrusive.

2.1 Tracking techniques

We often refer to people tracking as following people on security cameras and by cross-referencing the cameras location to the human in the video feed you get the per- sons location. This is however only one of several possible ways of tracking. Thanks to key-card systems, mobile location services and Internet usage we can today get a fairly accurate position of a person without being in sight. Airports for example are a frequent user of tracking techniques. Airports usually have systems in place to track passengers from when they first enter the terminal to when they enter the airplane. Today’s bigger airports will most likely implement technology such as face recogni- tion [5], RFID tracking [6] as well as their normal video surveillance system which

4 has people constantly monitoring others whereabouts. If tickets are booked online, the booking system could get your current location at the time of booking, and have a relatively precise location of the passenger. The following section will give an overview of some of the techniques used for tracking people. Most of the techniques described below are already in use today and some are also used in the demonstrator introduced later on in this thesis.

2.1.1 Human recognition

Several techniques for human recognition have been used for numerous years, includ- ing biometric solutions such as iris and retina scanners [7], finger and hand-print scan- ners [8]. As computers have become faster, recognizing people in videos has become a large area of research [9][10][11]. Lately even DNA is used as a fast way to recognize people [12][13]. These ways of recognizing people are used for authentication as well as security systems where tracking people can be of interest. For instance, finger print scanners has been an optional accessory for computer for some years, mainly to work alongside passwords for authenticating when people log into their operating systems or applications [14]. In some cases however, this has not been a feasible solution, and has rendered the systems they are used on even more vulnerable than before the new security measures were installed. The problems with fingerprint scanners are usually not the hardware itself, but the underlying software using the hardware [15]. Recognizing people in videos and pictures is a hard task, which has led to the development of an extensive amount of algorithms produced and tested for this purpose [16]. Most of the solutions to date use a combination of already existing techniques in image manipulation and combine them into new solution, as can be seen with Principal Component Analysis (PCA) techniques, which uses eigenvectors to find and recognize faces [11]. While the main focus of this project is to implement a facial recognition algorithm into the program, several other ways of recognizing people in videos exist. For instance the posture of a person can be used to determine if a human is in a video [17], a technique which could be combined with algorithms recognizing movement to create a system to track people [18]. Some techniques for recognizing people use the context of which people are in. For instance if there is a photo in a newspaper, a computer can based upon the caption determine who is in the picture. In this case, a combination of face detection and language models is used to recognize the people in the image [19].

5 A persons interest may also be of value. With the launch of the Microsoft Kinect [20], researchers could acquire on a relatively cheap depth camera. With depth infor- mation at hand, several new solutions for activity and face recognition could be made. Solutions such as 3D face reconstruction became cheaply available [21], and new face recognition algorithms that benefit from depth information was developed [22]. It also became easier to distinguish between gestures, which in turn made it easier to de- termine the activity of a person [23]. While activity recognition algorithms already existed before the Kinect [24], research benefited greatly from it. With the help of a 3D image, robots could also be made to easier distinguish between people, and made more autonomous by being able to see image depth [25].

2.1.2 Electronic tracking

The use of external devices can also be used for tracking people. An estimation of the location can be acquired by tracking their use of key-cards or passports. There are also other techniques that can be used. Bluetooth tracking can be used if a user has left their device in discoverable mode and Wi-Fi MAC addresses can be tracked when a user connect to a new access point [26]. As Google generated their street-view service, they collected the MAC addresses of most Wi-Fi access points surrounding their camera cars [27]. By doing another drive by, and by finding the same MAC address they could know if you have moved or not. There is no information pointing to Google actually doing this, but as more devices comes integrated with wireless technologies, tracking people will become easier if no countermeasures are put in place. Combining technologies to get a better accuracy of a person’s location can also be used as Mori et al. present by combining pressure sensors in the floor, with RFID tracking to get a good estimate of a person’s location [28]. They did so by sensing the presence of a human with the presence sensor, and identifying the person by triangulat- ing the position with multiple RFID transceivers. There are also open source solutions for RFID tracking, such as the one developed by the OpenBeacon project [1]. Their RFID tags and readers have been used to track the participants of conferences such as 25C3 CCC and BruCON 2011 [1]. GPS tracking systems are also an extensively used method of tracking, as it gives a good outdoor position location estimate, and is constantly getting better at indoor

6 Figure 2.1: RFID tag used at 25C3 CCC conference [1]. locationing. GPS tracking systems is also a relatively cheap method of tracking as no external sensors has to be put in place by its operators. This also makes it suitable for long range tracking, such as of people traveling by car, Alzheimer’s patients or people with dementia wandering away from caring facilities [29] [30].

2.1.3 Location prediction and data mining

With enough data about a persons whereabouts, one could eventually predict their next move with a good precision. As shown by Ingrid Burbey at Virginia Polytechnic Institute and State University, where a person’s location could be estimated with a prediction rate of 77%-91% with an allowed tolerance of plus/minus twenty minutes [31]. In this case, WiFi access points (APs) were used to track college freshmen for 14 weeks, which gave the initial test data. Burbey then used Markov models on the data to predict the user’s next location. Similar predictions could also be done using cellular networks, which could enable mobile operators to predict where you will be going next [32]. The winners of Nokia’s Mobile Data Challenge 2012 also showed that by com- bining a user’s location with the location of his phone book contacts, a good location estimate could be achieved [33]. Their algorithms could estimate a person’s location 24 hours into the future with an average margin of only twenty meters. Solutions such as this could be used by the authorities to estimate when a crime is about to be committed. Commercial applications could also benefit from location prediction when

7 showing advertisements or giving suggestions to points of interest that might interest the consumer. When using location prediction the future location of a person is not always of importance, but rather knowing points of interest connected to the person. Researchers from Georgia Institute of Technology shows that by tracking a person’s movement with a GPS, and feeding the information into a Markov Model, they could estimate user’s main points of interest [34].

2.1.4 Web services

Services also exists which lets its users willingly share their locations with other peo- ple. Services such as Foursquare and Google Latitude are used mainly to share a user’s location, while other services such as Facebook and Google+ implement a location sharing features only as a part of their services. Foursquare allows its user to check into places while competing for points with your friends. The more places the user checks in, the more points that person gets. It also helps keep track of where a user friend is, and help people to find each other. Google Latitude works by sharing its user’s current location with friends. The user can choose how exact the position is, by selecting if the location should be provided as an exact GPS coordinate, or on city level precision. Facebook and Google+ share the location of the user when a new status update is created. This is an opt-in feature which both services provide. This gives the users status update more context. While these services do not aim to track people per se, when combined, the amount of information being shared could be used to track a user’s whereabouts, and might also give a prediction of the user’s next location.

2.2 Ethics, intrusiveness and ignorance

While tracking people is a great asset to a security system, or even to some companies that want to streamline the work more efficiently, ethics and security might be an is- sue. People might not appreciate the spreading or private information, which people tracking gives arise to. Norwich Union, now part of Aviva, was a British insurance company that issued GPS tracking devices on cars registered by drivers between the age of 18 and 21, charging premiums depending on the time they used their vehicles. Disney has offered

8 GPS tracking for parents to put on their children, in order to know their location at ever time. Some truck driving companies in the USA also issued GPS tracking to reduce the amount of overtime, by tracking the movement of the trucks [30]. Systems like these might have moral or ethical issues, as the people being subjects to the tracking might be able to choose whether or not tracking is enabled. It might be that the person is underage, or that the work place simply demands that GPS tracking systems are installed in work related vehicles. Whether or not it is morally right to track people is a very subjective topic. Some people might not mind being tracked, while others beliefs or ethics might make them of a completely different opinion. On of the simple solutions of figuring the feasibility of a tracking system is to collect the opinions of people being exposed to the technology [35]. The advantages and disadvantages should also be weighed against each other, as people usually want to stay private, and not have other people tracking them. During the time of this project there was a camera set up in the corridor of the lab for 6 weeks, mainly to test facial recognition algorithms. The camera was located close to the ceiling and was barely visible. An e-mail was sent to the workers of the lab, stating that a camera had been put in place, and if anyone feels uncomfortable with it being installed they should let us know. While complains were given, one could easily note that people were looking straight into the camera when they walked by. This behavior did eventually stop after some time, which begs the question why? The main answer from people working in the lab was that they forgot that the camera was even there. While the psychology behind this is not going to be covered by this thesis, it is still interesting to note how people tend to ignore they are being monitored after a while.

9 3 CHOOSINGANDCOMBINING TECHNOLOGIES

In this chapter we present the technologies used in our implementation as well as com- parisons to similar solutions. The main parts of the system is the application plat- form, application services and the data storage. These represent respectively the sys- tem which the program will run on, the services the application uses as well as where the data is stored during the usage of the program. As stated in Section 1.2 the aim of this project was to combine several of exist- ing technologies into one single system. We chose the technologies in part by using software created under the DIEM research project and in part by technologies chosen at the planning stage of the thesis. The part included from the DIEM was the Smart- M3, a smart-space solution, which included a triple store database. Storage solutions for image and video files were also compared, as well as creating native against web development approaches. As the system should include several ways to track a person, facial recognition algorithms and web application programming interfaces (APIs) were implemented to create a diversity of technologies.

3.1 Application platform

When choosing the application platform for a project of this nature we had two choices, we could create a native application created specifically for one platform, or a web ap- plication. While native applications run on the user’s operating system which offers speed and works offline, web application offer device versatility and robustness when deploying. Both have their benefits, but as we had more experience with web devel- opment, a web platform was chosen. We will discuss our choices of software and integration in the following sections.

10 3.1.1 Web framework

The framework chosen for our application was Django [36], an open-source Python web framework. The main reason for choosing Django was our precious knowledge of the software, which made development faster and the learning curve smaller. Django is one of the largest web frameworks for Python, and has been in development since 2005. It follows the mode-view-controller architecture, which separates the handling of the data with users interacting with it. One of Django’s main advantages is its ability to reuse code, due to its use of apps. Apps are parts of a program, created for a specific task. Django also uses an object relational mapper between data models written as Python classes and a relational database. It uses a web template system, Jinja, for handling information between the database and the user. Django also has a URL dispatcher, programmable by regular expressions. As Django uses apps to separate different tasks, it makes the task of further ex- tending and existing application easier. Apps have their own models, views, URL dispatcher, tests and can be included in an existing program by copying the app folder into the project. Linking a new app with the rest of the application is made by modify- ing the main URL dispatcher to include the new apps dispatcher. Django also comes integrated with an admin interface, where one can control the database content. Admin interfaces usually have to be created by the application devel- opers, but as Django comes bundled with one, it speeds up the development process. The admin interface can also be modifies by developers, by adding features and chang- ing styling.

3.1.2 Web application and background tasks

Web frameworks such as Django do not support native background processing, and other applications have to be linked with Django to enable such functionality. While Python modules such as subprocess may be used to run processes in the background, a more robust solution was to use a task queue such as Celery [37]. Celery is an asyn- chronous distributed task queue written in Python. It uses a broker to store new tasks and handle results; the recommended broker by the Celery developers is RabbitMQ [38]. For running background tasks in Django with Celery, the Python package django- celery has to be present, and configurations has to be made to the Django applications

11 settings file.When using the RabbitMQ broker, only a minimal configurations has to be made to Django, which includes small modifications to the applications settings.py files. When creating tasks, a task file which includes functions written to be run in the background has to be created. These functions can then be called by another ap- plications, or run as a scheduled task. Django, Celery and RabbitMQ can also run on separate hardware if necessary. This distributes the work load, which makes the appli- cation more robust. As this is a small application we chose to keep all servers running on the same machine.

3.1.3 Web server

Django comes bundled with a development server. The Django Software Foundation however does not recommending the usage of this server for deploying applications to the public. There is however several choices when choosing a server to run Django application on and some performs better at serving particular sets of data. This led us to use a combination of servers to host our application. As Django is written in Python the first choice was to select a Web Server Gateway Interface (WSGI), which will serve as an interface between the web server and our application, sometimes called an application server. We also had to choose a web server to make the application available on the network. The web server of choice when running web based Python applications is Nginx. It is a web server and reverse proxy written in C. According to the Internet service company Netcraft, Nginx is one of the fastest growing web servers on the Internet, with a current market share of 12.07% [39]. The main advantage from other web servers such as Apache is speed. Several benchmarks show that Nginx is the fastest web server to date [40][41][42], Nginx has also proven to be fast at serving static content, while letting other processes handle dynamic processing. It also has good integration with Gunicorn and uWSGI which was the reason we chose Nginx as our web server From previous experiences we knew that the two main choices for WSGI servers were uWSGI[43] and Gunicorn[44]. Both are pre-forking WSGI server, which means that they spawn worker nodes at start-up, cutting the overhead of forking when running a new task. The server was selected based upon favored features such as ease of use, speed and stability. Gunicorn is written in Python and is created to rely on a proxy server to handle the first steps of HTTP requests. This means that a web server would receive HTTP

12 requests, and if the requests are dynamic it would send them to Gunicorn, which pro- cesses them and return them to the web server once finished. Gunicorn can be used as a web server directly, but as response times would be an issue, and it is not devel- oped to handle static files, such as images and cascade style sheets (CSS), its develop- ers do not recommend that setup [45]. There are several ways to integrate Gunicorn with Django, Django applications can be run directly with Gunicorn using the guni- corn_django command and add parameters, or use a configurations file and load it via the gunicorn_paster command. uWSGI is written in C, and similarly to Gunicorn it relies on a proxy server to handle the first step of client requests. Its main method of configuration is using an ini file. The main advantage from Gunicorn is that uWSGI is better integrated with the web server Nginx. This makes the connection between Nginx and uWSGI faster than the one between Gunicorn and Nginx. This is also the main reason we chose to use Nginx and uWSGI for deploying our application. For combining uWSGI and Nginx with our Django application we first created a configuration file specifying project specific paths, and uWSGI settings. We configured uWSGI to use a unix socket instead of a TCP socket as this is usually the fastest configuration when running uWSGI and Nginx on the same server. A PID file was also specified for easy access to the process ID as can be seen in Appendix A.1. We configured Nginx to connect to the socket created by uWSGI and also configured it to server all static files, while uWSGI handled the dynamic processing. Nginx was also configured to run over SSL to make all connection encrypted as can be seen in Appendix A.2.

3.2 Online APIs

One of the main purposes of the application was to be able to track people’s where- abouts, which made online services supporting user location of the particular interest. We found that Facebook, Foursquare and Google Calendar were good sources of such location information. As our application was meant as a proof of concept applica- tion and integrating Facebook would take too long we decided to use Foursquare and Google Calendar as our main sources of information. Foursquare uses oAuth for authentication and API requests are handles over a se- cure SSL connections. They also offer a push API, which sends information to a

13 developer set URI, every time an authorized user with ones application checks in. This lightens the load of the application, as it does not have to poll for new check in man- ually. We chose to use the push API as it could be integrated with the application easily. The push API works by sending JSON data to a specified address, which the de- veloper then needs to process. One way to handle the data is to store it in a database, and use upon demand. Our solution stores the check-in information in the Smart-M3 database every time a user checks in a location. This way, applications that want to integrate with our Smart-M3 SIB, is able to get a notification when a new check-in has been made, as well as when the user’s location is changed.

3.3 Databases and media storage

Data storage is one of the main parts of web applications and they are constantly hit by users accessing data when new web pages are loaded. As one of the project require- ments was to use the Smart-M3 software, we began exploring its possibilities, and how to integrate it in to the application. A comparison between multimedia storage solution was also made, comparing databases to file systems.

3.3.1 Smart-M3

Smart-M3 is an open-source software that aims to increase the intelligence of entities and devices by connecting them together via the sharing of data. Smart-M3 strives to be multi-vendor, multi-device and multi-part compatible, therefore the M3 at the end of the name. Smart-M3 uses Semantic Information Brokers (SIBs) to distribute the data between devices. A SIB is placed on top of a Triplestore database, and acts as a handler in between the actual database and the Knowledge Processors (KPs), as can be seen in Figure 2.1. Several SIBs can be connected to each other, to form a smart space. The SIB is not tied to any specific triplestore, but as of writing there are only two publicly available implementations in the Smart-M3 Sourceforge repository [?]. One id developed by Nokia and the other by The Advanced Research Center on Electronic Systems at the University of Bologna. What distinguishes Smart-M3 aside other triple and graph storage solutions is its way to handle the flow of data in the network. Instead of constantly querying for new data on the triplestore, devices can set up subscriptions to certain queries and get a

14 push like behavior from the system when the subscribed data is modified. The worker nodes in a Smart-M3 smart space are called Knowledge Procesors. The KPs handle all transaction with the SIB such as insert, update and delete actions. The KPs also handle subscriptions to the SIB. It should be noted that SIBs and the KPs can run on the same machine, but usually the SIB runs on a computer faster and with more memory then the rest of the KP nodes in the network. KPs are also made to be multi-platform, and several implementations exists including versions for C [7] , C# [8], Python [9], Java [10], JavaScript [11] and PHP [11].

3.3.2 SQL and NoSQL combination

For database storage we chose to combine a NoSQL database [46], Smart-M3, with a PostgeSQL database[47]. The choise for Smart-M3 came from the project specifica- tion, and PostgreSQL has proven to work well as a database server for web projects in the past. The reason to combine these databases instead of using just one was be- cause Smart-M3 does not support user authentication, which makes it a bad choice for storing passwords and other sensitive information. The PostgreSQL also serves as a redundant database while moving over part of the database handling to the Smart-M3. We combined these two databases by overriding Djangos save method when saving models. By doing so we could easily integrate Smart-M3 while still holding on to the PostgreSQL database for redundancy. A benefit of using Smart-M3 is also that other services can connect to the SIB database and create applications based on the information flow in the SIB. As Smart- M3 offers the possibility of subscribing to information, other applications can work with the data of another application, without knowing how that information came to be in the first place. Data can also be fed back into the SIB, to complement or changing the information of another program. This way applications have only one source of information, and does not have to communicate directly with each other. This reduces application coupling, and as several SIBs can be connected to each other there will also be redundancy, this making applications more robust. All applications connected to the Smart-M3 smart-space should however be able to handle changes from another application, without failing. This making the development process a bit more tedious.

15 3.3.3 Image and video storage

For media storage one of the simples and most effective solutions is to store videos and image on the file system, while keeping track of their file paths and meta-data in a database. While using only a database offers some good features as ease of syn- chronization between databases, easy access and less fragmentation. File systems are usually faster, takes load of the database, requires less code to access its files and gives better control over media files. Storing files on the file system also simplifies the pro- cess of exporting files to other programs for transcoding or other post processing work. When choosing a good file system to store and accessing large amount of files we compared EXT4 and XFS. EXT4 is the standard file system use in many mod- ern day Linux distributions such as Ubuntu and Fedora and XFS is promoted as a high-performance, highly scalable file system for large data storage. Empirical tests however show that the difference between the two file systems is minimal, and that there are no great benefits of using XFS over EXT4 [48]. The extra time of setting up an XFS file system is therefore not beneficial, and we thereby chose to use EXT4.

3.4 Face detection and recognition

Face detection and recognition has existed for a long time, and several algorithms have been developed since Woodrow W. Bledsoe first showed that face recognition could be done with computers in the early 1960s [49][50]. As we had set out to use the OpenCV library for image processing, we found that it offered one ready-made solution for face detection. We found that the most research regarding face recognition had been done using Eigenfaces, Fisherfaces as well as Local Binary Patterns (LBP). For face detection, OpenCV uses Haar Cascade Classifiers, which work by using a cascade file, containing Haar features of faces which are compared to the features of an image to determine the existence of a face [51]. A Haar feature is calculated by first placing simple rectangular sections, containing two or more adjacent rectangular regions which are either white or black over a detection window. The pixel intensity inside each region is then summed up. The regions summed up intensities are then used to categorize section of the window. If these sections exist in an image, we have a match in the cascade file. This technique requires the developer to train and create cascade files in advance, which includes the Haar features of the object to be detected, in our case the face.

16 For face recognition we compared research done using Eigenfaces, Fisherfaces and LBP. Eigenfaces uses principal component analysis (PCA) to extract facial features from a set of images of a person [52]. Fisherfaces on the other hand uses linear dis- criminant analysis (LDA) to extract facial features [53]. Both methods need training data of people to be able to later recognize them. The training data usually consists of several pictures of the subject, in various lightning conditions. These two methods work fairly well, but both algorithms have problems when lighting conditions changes from the initial training data. Local binary patterns however are less error prone when it comes to recognizing faces in new lighting conditions [9]. It also needs the least amount of training images of the three, to get a fairly good prediction rate. This was the main reasons for choosing to use LBP as our face recognition algorithm in this project. In subsection 4.5.3 in chapter 4 we will go into greater details about how LBP works, and how it was implemented into the project.

17 4 IMPLEMENTATION

This chapter covers the Krang demonstrator created for the AMEBA project. It presents the implementation of both the front- and back-end implementation. The implementa- tion is structured in part by Djangos app structure and part by best practice implemen- tations from previous knowledge. The front-end was developed with usability in mind, as the demonstrator was to be used as a showcase for the AMEBA project. Use case scenarios will be described in section 4.7 at the end of this chapter.

4.1 Platform

The back-end implementation was created for the x86 and x64 instruction set plat- forms, running a Linux based operating system. The implementation has however been tested to work on a Raspberry Pi which uses an ARM1176 processor using the ARMv6 instruction set. The front-end implementation was created for non-mobile web browsers utilizing the Gecko or WebKit layout engine such as Mozilla Firefox, Google Chrome and Apple Safari browsers.

4.2 Krang Demonstrator

The Krang demonstrator is a distributed web-based people tracking application. It was created for the AMEBA project, as a proof of concept for demonstrating smart-spaces and intelligent software. A smart-space can be a specific location on environment which uses electronics, such as computers and sensors, to give more information about that particular space. It can also include data from existing services, which are inter- preted in a new way, to get a more intelligent environment. The main purpose of the demonstrator was to use already existing software and algorithms to create a program which could benefit from having several sources of

18 information. We focused mainly on people tracking and face recognition. By combin- ing services such as Google services, Foursquare and face recognition, we created a program which could track a person’s location and use the information to notify other users of what people are currently doing. The demonstrator integrates the Smart-M3 triples store which enables other devices to access the information created by our soft- ware. This makes the software application more modular, and creating new features for the program can be done by only managing the Smart-M3 data. Therefore it does not have to be integrated into the demonstrator itself. Another purpose of the demonstrator is to distribute the processing as much as possible, to make it mappable on several processing elements, or devices. For this reason the servers and middleware used in Krang was partly chosen by how well they are able to scale.

4.2.1 Functionality

The Krang demonstrator is able to tell the location of a person, based on the data supplied from multiple sources. The source being either user input, data received in relation to user actions or data retrieved automatically through methods such as face recognition. The location is then used to determine if a user is currently available in his room, close by or at a different location. After determining the users availability other devices such as lights or status updates can be set accordingly. A calendar late-to-event feature is also implemented, which is able to tell if a person is going to be late to a meeting. By checking for a user’s current location and current calendar events, we can tell if a person is going to be late to a meeting, and is then able to notify the other participants by e-mail. This particular scenario is discussed in detail in the event notification system use case in section 4.7.2.

4.3 Implementation overview

The software is created using the Django web framework, and uses its app structure for separating different tasks. As a rational database, we used SQLite during development and PostqreSQL during production, and the Smart-M3 SIB was used for storing triples. The database models and their dependencies can be seen in Figure 4.2 and the ontology classes for the triple store can be seen in Figure 4.7.

19 Figure 4.1: Back-end servers and services with corresponding Django apps.

The implementations back-end is written mainly in Python, with the use of third party modules, some of which make calls to C or C++ modules. The back-end is handled by Nginx, uWSGI, Django, Celery, RabbitMQ, OpenCV as well as APIs for Google Distance Matrix, Google Calendar and Foursquare. The front-end was created using the front-end framework Twitter Boostrap and the javascript library jQuery as well as custom written jinja2 HTML templates, JavaScript and CSS files. The different back-end apps and their dependencies can be seen in Figure 4.1.

4.3.1 Directory hierarchy

The application is structured into several directories, some with subdirectories. This makes the application structure more clear and development easier. There is no clear structure guideline from the Django Software Foundation how applications should be structured other than placing apps in separate folders. The following structure was used for the implementation: krang/ This folder is the main project folder of the Django application.

/core/ Core app of the application. Includes the main functions for handling what is shown to users. /templatetags Template tags related to the core view. /foursquare/ Foursquare app, holds functions related to handling Foursquare requests and settings. /freqo/ Face recognition app. /gcal/ Google calendar app, holds functions for handling Google Calendar re- quests and settings.

20 /templatetags Template tags related to the google calendar view. /libs/ Libraries developed specifically for this application, as well as third party libraries used, such as face recognition algorithms. /facedet Face detection library. /facerec Face recognition library. /helper Helper functions for the face recognition library. /videofacerec Library for linking face detection and recognition. /helper Helper functions for videofacerec library. /logs/ Holds main logs for the application. /media/ Static media files, developed for the application as well as third party libraries and frameworks. /css Cascade style sheets related to the project. /vendor Third party CSS files. /img Images related to the project. Subfolders might be dynamically gen- erated by the application. /js JavaScript files related to the project. /vendor Third party JavaScript files. /static/ Static files for the Django admin interface. /templates/ Application templates /core Templates for the core view. /templatetags Core app templates spcifically for the templatetags.

4.4 Code used from external sources

The following section contains code from external sources that was used in our appli- cation. It includes several Python packages, front-end web frameworks and libraries for computer vision and background processing in Django.

4.4.1 Python

Several Python packages were used for developing or application. It should be noted that some of the packages listed below have their dependencies not listed. There were

21 also packages, helping the development of the application that were used, but not listed here. Their packages can be found in the requirements.py file in the applications root folder.

Mechanize Mechanize is a Python port of a popular Perl module, for stateful pro- grammatic web browsing. It uses urllib2 for fetching web pages, and can also be used as an FTP client. It includes features such as cookie handling, HTTP refresh modifi- cation, robot.txt handling, proxies and HTTP authentication. Its main use is to write automated web crawlers or screen scrapers. Licence: BSD

BeautifulSoup BeautifulSoup is a tool to help with screen scraping web pages in Python. It turns HTML or similar code as XML into a Python object, which offers functions for searching, navigating and modifying the code. In combination with pack- ages such as Mechanize, Beautiful Soup offers an easy way of screen scraping. Licence: BSD

Celery Celery is an asynchronous distributed task queue; its main focus is real-time operations but can also be used for running scheduled tasks. It uses a message broker for message handling, which needs to be installed separately. Licence: BSD

Django Django is a web framework for Python that implements a model-view-controller architecture. It includes an object relational mapper that handles the communication Python classes to database models. It offers support for web forms, authentication, caching, testing interfaces and several other features. It also includes an admin inter- face for handling database entries. Django integrates a development server, but should be run on top of a web server such as Nginx using WSGI when in production. Licence: BSD

Google APIs Client Library for Python Googles API Client Library offers access to most of Googles APIs using Python calls. In addition it also offers an OAuth2 module, which can be used with any OAuth2 server. In this project the library was used for accessing the Google Calendar API. Licence: Apache License 2.0

22 OAuth2Client OAuth2Client is part of Google’s APIs Client Library for Python. OAuth2Client is an OAuth2 client which enables Python calls to be made directly to an OAuth server, such as Googles, Facebooks or Foursquares OAuth API. It hides all HTTP calls so that all calls can be made in a more Pythonic way. It also offers ready- made made model classes for Django, which can be used for storing user credentials. Licence: Apache License 2.0

Flask Flask is a lightweight web framework for Python. It uses Jinja template en- gine, and offers a build in web server. Flask relies on extensions for features such as data base handling, web forms and authentication. Licence:

4.4.2 HTML and JavaScript

Twitter Boostrap Twitter Boostrap is a front-end framework for web development, developed by Twitter. It offers HTML and CSS templates as well as several Javascript extensions. Licence: Apache License v2.0 jQuery jQuery is a client side JavaScript library. It is the most used JavaScript li- brary today [54]. It includes wide range of features such as Document Object Model (DOM) traversal and manipulation, AJAX as well as being multi-browser compatible. Licence: MIT

CodeMirror CodeMirror is a in browser code editor written in JavaScript. It sup- ports syntax highlighting for a wide range of programming languages such as C, Python, SQL, SPARQL and CoffeeScript. It is customizable via CSS and supports a JavaScript API. Licence: MIT

4.4.3 Other software

RabbitMQ RabbitMQ is a message broker that can be used in combination with Celery. It stores all jobs and their results for Celery tasks, and can run a separate server

23 than Celery if need be. Licence: Mozilla Public License (MPL)

OpenCV OpenCV (Open Source Computer Vision) is a computer vision library mainly focusing on real-time processing. It is developed by Intel, but can be run on several architectures such as x86, x64 and ARM. It is written in C++ but has wrappers for languages such as Python, Java and MATLAB. OpenCV include several computer vision algorithms for applications such as face and gesture recognition, robotics and motions tracking. Licence: BSD

SimpleCV SimpleCV is a computer vision library build upon OpenCV. It offers shortcuts and combines features of OpenCV for faster development. Some examples being Kinect, JPEG stream and macro support. SimpleCV is written in Python. Licence: BSD

Smart-M3 Smart-M3 provides a semantic web information sharing structure by com- bining devices together with a central database. It is multi-platform and includes inter- faces for several programming languages including Python, Java, JavaScript, C, C++ and C#. Licence: BSD

4.5 Back-end design of Krang

In this section we discuss how the chosen technologies were implemented and how they connected to the Django web-framework. Firstly we go through how the database and the ontology are structured. Django uses an app-structure, where each app includes a specific feature of a Django application. Every app includes its own database struc- ture, and we describe the app and the database model at the same time, as the models define the app. Secondly we define the ontology used by the Smart-M3 SIB. We also describe how the ontology and the app models are connected together. Thirdly we define how face recognition using a local binary pattern algorithm was implemented. Forthly the social media service Foursquare is then described, and how their real-time API can be used to get push notifications when a user checks-in. We then describe

24 Google Calendar, and how we integrates calendar events in the application. Lastly we describe how we got information from the Åbo Akademi Department of Informa- tion Technology web pages by web scraping and how the information is used in the application.

4.5.1 Database model

Figure 4.2: Complete database model, without database fields.

Django uses an object relational mapper that works between Python classes and re- lational database models. The entire database model can therefore be written in pure Python, and Django will then translate it into SQL calls. The developer does however still need to define the type of database fields to use, and properties such as the length of a character field. One can also define custom field which makes it easier to add ad- ditional information in comparison to manipulating the database directly. Django also handles the connection to the database, and supports several databases, and database types can be change without the need to change the database model. Every app in a Django application has its own models file, where the models spe- cific for that app specified. When Django synchronizes with the actual database, all models files will be read, and the appropriate fields and columns will be created in the database. Figure 4.2 shows the structure of the entire database without the field values. The following sections will cover each of the apps database models separately.

25 Core

Figure 4.3: Core app database model.

The core app consists of three classes. Firstly an extension of Django’s User model, which adds fields for storing additional information about a user. As the core applica- tion includes a screen scraper for getting information from research.it.abo.fi, additional fields was created to store the information returned by the scraper. This information is stored in the UserProfile model, which include fields for information such as phone numbers, information display text, room number and current location. The UserProfile model uses a one-to-one field, to extend Django’s User model. This is the recom- mended way to extend the User model by the Django Software Foundation. Secondly a social profile was also added, for storing information about the user’s social media connections. Some fields were created as character fields, to store the users username, while others, such as personal blogs or the users Google+ page were added as URL as we cannot predict these links. The user profile model is linked with the user profile via the foreign key social in the user profile. Some of the debug fields are still left in Figure 4.3. The fields temp_pass and cookies are used for debugging certain features, as the program is not yet ready to be used by end users. It should also be noted that the user model has been left untouched, and is developed by the Django developers and not written as part of this project.

26 Google Calendar

Figure 4.4: Gcal app database model.

The Google calendar app uses Googles OAuth2Client package, which includes two specific Django database fields made for storing OAuth credentials. The fields Flow- Field and CredentialsField are used for storing information such as the users access token and user id. As users can have several calendars in the Google Calendar account, a settings model was created for storing the specific calendar the user wants to use. The Settings model, as well as the FlowModel and CredentialsModel are linked to the User model via an id field.

Foursquare

Figure 4.5: Foursquare app database model.

The foursquare app uses three models for storing data. The FSUser model holds infor- mation about the user’s identity, including the user’s access token, which is used for authenticating the user Foursquare account with our application. The fs_id field holds the users unique Foursquare ID, which is used for identifying the user. The Foursuqare user is also linked with his or her user profile via the user field.

27 A settings model was created for holding Foursquare specific configurations. The field only_ict holds information if a Foursquare check-in should only be stored if it is related to the ICT building at Joukahaisengatan 3-5 in Turku. The field no_update specifies if the users Foursquare check-in should be registered at all. The settings module is linked to a user’s foursquare model, via the settings field in FSUser model. A user’s Foursquare history is stored in the FSHistory model. It has fields for storing the check ins location, city, country, coordinates and timestamp. It is linked with the user via the user field which is a foreign key to the User model. It should be noted that Foursquare uses OAuth for authentication as Google Cal- endar. The choice of not using the third party package from Google for authenti- cation with Foursquare was made to better learn how OAuth authentication works. The oauth2client package could be used with the Foursquare model by including the django_orm modules FlowField and CredetialsField functions.

Face recognition

Figure 4.6: Freqo app database model.

The face recognition app includes two models for storing user specific data, three mod- els for cameras and their settings and one model for storing a history of when users have been recognized by the cameras. Some of the fields pictured in Figure 4.6 are not in active use in the program, and are there for future features. These fields are explained in the paragraphs below. The Fuser model stores data about a face recognition user. It has two foreign keys, one to the User model and another to the FSettings model, which holds user specific configurations. The face_id field holds an ID for the specific user. At this point in

28 development, the application will store the user’s university login name. The images field holds the number of available images of the users face. The FSettings model keeps track of the users configurations. It features three alter- native of which coffee_tracking is currently not in use by the application. The history fields stores information about if the user want the application to keep a history of when the user has been recognized by a camera. The tracking field holds information about where or not the user want to be tracked by cameras. This is to give the user the ability temporarily disables tracking by face recognition. The Camera model stores data about camera locations and gives a description about the cameras. Is has one foreign key, settings, to the CameraSettings model which holds camera specific configurations. The fields location and location_name differ in that lo- cation stores an exact location, such as an address or coordinates, while location_name stores an verbose name of the location. The name filed store the name of the camera and the field description give a short description of the camera. The CameraSettings model stores information about IP and path of the video stream in ip and video_path while the source filed stores information about the kind of media that can be fetched from the camera. The default choices being video or an image. The resolution field holds information about the resolution of the video stream or the image and fetch_amount tells how many videos or images should be taken if motion is detected. Every camera can store the media in separate FTP server, and the ftp, ftp_user and ftp_pass fields hold authentication information about the camera specific FTP server. The ftp_wait fields hold information about the timeout value for the FTP connection. The history model links users and cameras together by storing information about when a user has been recognized and by which camera he has been seen. A time-stamp keeps track of when the user was recognized. The coffee field is not in current use in the application, and was meant to track the amount of times a users goes for coffee during a work day.

4.5.2 Ontology model

When working with a graph database, such as the Smart-M3 SIB, an ontology model gives a structure to the data added to the database. It shows the connection between different classes, and specifies types for properties. While an ontology is not necessary for a graph database, it makes developers job easier as a clear structure of the database

29 is given from the start. The ontology language used in our application is the Web Ontology Language (OWL). We tried to create an ontology that resembled the rational database as close as possible. Some parts, such as authentication information was left out as the SIB does not support user authentication, making it publicly available for everyone who has network access to it. As the application was meant to run as a distributed system, we could not assume that the SIB and the other parts were to be run on the same computer, or network. These issues rendered the SIB insecure to store sensitive information. Ontologies consist of classes, properties, instances and relations, but can also in- clude rules, axioms and restrictions. A class is a type of an object, and all instances have a class. A class can be a subclass of another, thereby inheriting all of the parent’s characteristics. As we are using OWL all classes are subclasses of the root class Thing, which can be seen in Figure 4.7. Properties give classes certain characteristics, and can be compared to fields in a relational database. There are two kinds of property types, object properties and data properties. Object properties link classes together and data properties holds data values for a class. Naming of properties are often done by includ- ing the prefixes has or is followed by an attribute. The attribute for object properties are often named by including the class name after the prefix. We will now describe the classes and their properties used in our application.

30 Figure 4.7: Krangs ontology classes and their relationships.

The ontology is first split up in to a physical and an abstract class, to give a clear separation between tangible things and data objects. These classes do not have any properties, and serve mainly as a separator.

Figure 4.8: Person class, with object and data properties.

Subclasses for PhysicalThing include the classes Person, Environment and Device. The class Person corresponds to the User class in the Django model. A person can be linked to several phones and a user specific Google Calendar, Foursquare, room and location class, as pictured in Figure 4.8. It has one data property, hasName, to store

31 the person name. The environment class includes three subclasses and no properties and serves as a class separator. The environment here specifies the physical environment of a location, it includes the subclasses Room, Floor and Building .

Figure 4.9: Room class, with object and data properties.

The Room class includes one object property, linking a room to a specific floor, and one data property storing the room name or number as pictured in Figure 4.9. This class is used to store a person’s room, and to specify the exact location of a camera.

Figure 4.10: Floor class, with object and data properties.

The Floor class includes one object property linking a floor to a building, and one data property storing the number or name of a floor, as pictured in Figure 4.10. This class is used mainly to give more information about a location, for instance if a room number is known, the floor number is also known.

Figure 4.11: Building class, with object and data properties.

The Building class includes two object properties for linking floors, and rooms to a specific building as pictured in Figure 4.11. This class is used as an extension to the room and floor class, to get more information about a location. For handling devices, the Device class was created as a subclass of PhysicalThing. It has two sub sections, Phone and Camera. It has no properties, and serves as a separator, and makes querying for all devices easier.

32 Figure 4.12: Phone class, with object and data properties.

The Phone class includes one object property linking a person and a phone, and one data property to store the phone number as pictured in Figure 4.12. This class is used for adding phone devices to a person.

Figure 4.13: Camera class, with object and data properties.

The Camera class includes two objects properties to link a location with a camera and to associate video data with a camera. It also includes two data properties storing the cameras IP address and the path to the video source as pictured in Figure 4.13. This class is used to store information about IP cameras to be use with the face recognition app. The Data class separates data types from each other. It has one sub class, Video- Data, and more sub classes can be added if more data types occure.

Figure 4.14: VideoData class, with object and data properties.

The VideoData class, pictured in Figure 4.14, includes two object properties, link- ing a camera to a specific video, and to associate people with a video. It also includes two data properties, to a specify the videos storage path, and to flag the video. The class is used to store data specific for a video file stored by a camera, and to store the people found by the face recognition app. The Event class separates event types from each other. It has two sub classes, GoogleCalendarEvent and FoursquareCheckin.

33 Figure 4.15: GoogleCalendarEvent class, with object and data properties.

The GoogleCalendarEvent class, pictured in Figure 4.15, includes one object prop- erty to link a Google calendar account to an event. It also includes four data properties to store the location, start- and end time and the title of an event. It is used to store events from the Google Calendar.

Figure 4.16: FoursquareCheckin class, with object and data properties.

The FoursquareCheckin class, pictured in Figure 4.16, includes one property to link a Foursquare account to a check-in. It also includes four data properties to store the address, city, venue and time stamp of a check-in. It is used to store a person’s Foursquare checkin. The LocationData class separates location types from each other. It includes two sub classes, UserLocation and CameraLocation.

Figure 4.17: UserLocation class, with object and data properties.

The UserLocation class, pictured in Figure 4.17, includes one object property link- ing a person to a location. It also includes four data properties to store the address,

34 city and venue of a location as well as a timestamp of when the location was asso- ciated with a person. It is used to store a person’s locations, it can be a copy of the information of another location such as a foursquare check in.

Figure 4.18: CameraLocation class, with object and data properties.

The CameraLocation class, pictured in Figure 4.18, includes two object properties for linking a floor and a building to a specific camera. It also includes one data property to store the camera venue. It is used to store camera locations for cameras in the application. The Account class separates account types from each other. It includes two sub classes, FoursquareAccount and GoogleCalendarAccount.

Figure 4.19: FoursquareAccount class, with object and data properties.

The FoursquareAccount class, pictured in Figure 4.19, includes two object proper- ties linking a Foursquare checkin to a Foursquare account and for linking a person to an account. It is used to store Foursquare account information, and it does however not include any authentication information, as the Smart-M3 SIB database does not supply sufficient security to store such information.

Figure 4.20: GoogleCalendarAccount class, with object and data properties.

The GoogleCalendarAccount class, pictured in Figure 4.20, includes two object properties for linking an event to a Google Calendar calendar and for linking a person to an account. It also includes one data property storing the identification string of a

35 specific calendar. It is used to store Google Calendar account information and as with the FoursquareAccount class, this class does not store authentication information, as the Smart-M3 SIB data base is not secure enough.

4.5.3 Face detection and recognition

To integrate face detection and recognition, the OpenCV library was used. It provides several common computer vision algorithms, and has a good community support. As the amount of algorithms created for the purpose of detecting and recognizing faces are extensive we opted to use a ready-made implementation developed by Philipp Wagner [2]. His solution uses haar cascades for detection and local binary patterns for recogni- tion. The solution by Wagner also includes two applications alongside his library files with his algorithm. The applications include a full example on how to use his library, as well as a face extractor which automatically stores images of detected faces in a video stream. For collecting data and creating a database of faces, we installed an IP-camera in our lab corridor which sent ten second video clips every time movement was detected. The video clips where then manually checked if people of interest where in them, and then run through the face extractor application. All images were then categorized according to the person in the image, and put in distinct folders with the name of the person’s username. This way the face recognition application would be able to draw the conclusion between user and face. The extractor uses OpenCV’s CascadedDetector class which uses haar cascades for detecting certain objects. The haar cascades are XML files with data extracted from a training set. The training is done feeding several hundreds of images into a classifier which then creates the cascade files. Most of the images include the object that is to be recognized, called positive examples, and some of the images with the object not present, creating negative examples. These cascade files can be time consuming to create, and in our case, collecting images of faces can take a long time, and processing these images, when the cascades are created, could take several days. For this reason OpenCV has included multiple cascades for detecting human features in their library. These cascades can then be used in applications that require detection of features such as the face, mouth, eyes or the upper-body of a human. The face recognition application uses the images collected from the face extractor

36 Figure 4.21: Original Local Binary Pattern description [2]. to create a predictive model to recognize people. The recognition application uses Lo- cal Binary Patterns (LBP) to recognize objects, in our case faces. The original LBP algorithm went through every pixel in an grayscale image, and looked at its neigh- boring pixels. The intensity of the pixel is then compared to its neighbors and if the neighbor has a higher intensity it gets assigned the number 1, else 0. This will create a pattern of only once and zeroes, as shown in Figure 4.21 [2]. This technique however was shown not to be efficient at encoding details differing in scale. This led to the development of circular LBP, which goes through all the pixels as the original imple- mentation, but instead of selecting neighboring pixels, a circle of points are created around the pixel. The amount of points and the radius from the center pixel can be selected freely, however the more points selected the higher computational power is needed. If a point corresponds to a pixel, its intensity is compared to the center pixels, and as in the original algorithm, if the intensity is higher than the center pixels it is assigned a 1, else a 0 as pictured in Figure 4.22. If a point does not correspond to a pixel, say it covers two, the correct pixel to compare with is chosen based upon bilinear interpolation [2]. When the pattern is created the object is classified by comparing the histogram of the image with histograms created from the images in the training set. If the histograms match, or are a close match, and object or in our case a person has been recognized [3].

37 Figure 4.22: Circular Local Binary Pattern description [3].

We used IP cameras for our main video source in Krang, and all testings were a done with a NorthHQ NQ-9006 Wi-Fi enabled camera. The camera can stream both MPEG-4 and MJPEG at the same time at a resolution of 640x480 at 30 frames per second. To be compatible with as many cameras as possible we only used the MJPEG stream of the camera, as most IP-cameras support this format. For capturing data from the video stream we created a small helper library, ipcamera.py, which includes two classes, one to connect an IP camera, and another to handle the saving of a video clip from a stream. These classes in turn uses OpenCV as well the SimpleCV framework, which is built upon OpenCV. SimpleCV includes ready-made functions to connect to JPEG streams, such as the one from an IP camera supporting MJPEG. It also supports authentication if the video stream is password protected, a feature usually active on most IP cameras. We store data from the video stream as video clips, this gives us more control over the processing of the video. It also solves the problem when more than one video stream needs to be processed, and only one computer is processing the video. We can then queue up several video clips and process them in order without spawning more than one video processing task, giving us a first-come-first serve processing structure.

38 Another benefit is that we can go back and forth in the video if needed, without the need to create a large buffer as it would be required if the stream was directly processed.

4.5.4 Foursquare

As described in section 3.2 and subsection 2.1.4, Foursquare is a location services where users share their locations with their friends. It also offers a push API called the Foursquare Real-Time API, which pushes information about user’s check-ins to a server specified by the developer. Foursquare uses OAuth for application authentica- tion, and also forces the communication to be encrypted by not accepting any other push addresses other than secure HTTP ones. As we had already configured Nginx to run over SSL, no special precautions was needed to create our Django app.

Figure 4.23: Flow diagram for OAuth 2.0 [4].

For authentication we wrote an OAuth authenticator which includes the FSUser database model and one callback function. To authenticate with our application the user will first need to click a link including our applications client ID, it will redirect to the Foursquare web page four user authentication, and allow our application access to the users check-ins. Foursquare will then send the user back to an application specific URL, including an user code. Our application will then respond to the call with an application specific client secret and the user code, if the code and the secret are correct, Foursquare will respond with a user access token as pictured in Figure 4.23. The access token is then stored in the database and can then be used to make calls to the

39 Foursquare server, and get information about the user. As the Smart-M3 SIB does not support authentication, and storing user sensitive data would create a security issue, the access token is not stored in the Smart-M3. The access token itself was not used in our application, but has to be fetched in order to give access to Foursquares Real-Time API.

Figure 4.24: Flow diagram for Foursquare Real-Time API.

As pictured in Figure 4.24, the API works by sending out POST request with at- tached JSON data which includes information about check-in data every time an au- thenticated user checks-in. The URL Foursquare sends the request to is user specified when creating a developer account at Foursquare. One is also supplied with a push secret, which will be included in all push requests when creating an account. The push secret is used to validate that the data sent to the server is coming from Foursquare and not a third party source. To store users check-ins the FSHistory model is used in the relational database and the class FoursquareCheckin in the SIB. Storing of a check-in data happens at the same time as the Foursquare server pushes the data to the server. The function handling the push request first checks the push secret and then checks if the user has set any of the settings parameters described in the FSSettings. After the checks the data is stored in the rational database as well as in the SIB. If the user has agreed to update his status according to his Foursquare updates, the users current location is changed to the one of the check-in. In the rational database as well as in the SIB this happens by setting the users current venue to the venue of the check-in.

4.5.5 Google Calendar

Google Calendar is mostly used to store event information and meeting management. If the user has entered location information about an event we could also use that for locating a user based on the current time and the event data. Google offers a Python API for managing user events, and also offers a package for handling authentication

40 in Django. As with Foursquare, Google Calendar uses OAuth for authentication, and applications need an access token for every user to be able to access their data. We used the Google provided OAuth2Client package which has ready-made models to store user credentials in Django as can be seen in Figure 4.4 in section 4.5.1. As with Foursqurare, when a user want to add their Google Calendar account to our application they first need to authenticate with Google and give us permissions to their user data. This is done in the same way as with Foursquare, but only using the Google provided OAuth2Client. The process of authenticating follows the same structure as in the Foursquare app seen in Figure 4.23. The user is sent to Google’s authentication servers, which asks the user for credentials, where after a call is made to our application with a code. The callback with a code is then sent, in addition to our application specific client ID and client secret to Google’s servers which responds with the user’s access token. There are however two main differences from the Foursquare app: A) we can pass the entire request from Googles callback to a function in OAuth2Client package which handles the extraction of JSON data posted by Google and returns the user credentials. These credentials are then stored in a custom CredentialField field also supplied by the OAuth2Client. B) Google grants access to multiple APIs through a single OAuth server, which mean the scope that we want access to needs to be specified. One also needs to specify if the application requires offline access, which allows the application to retrieve information even if the user is not currently using the application. The scopes used in our application where calendar, userinfo.profile, userinfo.email as well as offline access. These scopes give our application access to the user’s calendars, as well as their user profile and email. As users may have more than one calendar in their account, after authentication with Google we let the user decide which calendar to use and stores it in a character field in the database. This calendar can then be used to access upcoming events and their locations, as well as to be shown on their profile page.

4.5.6 Åbo Akademi Department of Information Technology

To make the application more personal, information from the web page of Department of Information Technologies at Åbo Akademi was integrated into Krang. The page includes information such as photos, e-mails, phone, and room numbers for most of the workers at the Department of Information Technologies. A user can also log into the web page for additional information such as door display info.

41 Figure 4.25: Door displays in the ICT building.

The web page was to be used for its information, as well as a mean of authentication for the Krang application. We will now go through the two separately and explain how information from the page was extracted and used. Firstly we were interested in using the page as a means of authentication to the application. By doing so, we eliminated the need to create a registration page for our application, and users could use their existing credentials to log in to Krang. The algorithm used for the authentication is shown in listing 4.1. In the application, this algorithm is run at the first time when the user tries to log in. If the log in is a success the password and username is stored in the local database, else an error message is shown to the user.

1. Fetch the log in page 2. Find the log in form 2. Enter log in credentials in to the form. 3. Submit the form 4. Check if a

tag with the class"portalmessage error" exists in the returned page 5. If the element exists return False as the log in failed, else return False

Listing 4.1: Department of Information Technology at Åbo Akademi login algorithm

In practice the algorithm is performed by a web scraper written in Python using

42 the third party packages Mechanize and BeautifulSoup. Web page fetching is done by Mechanize and HTML parsing is handled by BeautifulSoup. This method is only run when a user logs in the first time. An example implementation can be seen in appendix B.1. Secondly we wanted to extracted information from the web page. This was done in two steps, first by accessing the database storing the information shown on the web page, and then by creating a crawler to download the person profile pictures. The information for a worker stored in the database includes name, phone number, room number and email as well as information about the door display status. The door displays are custom build displays, fixed next to the doors of most rooms in the Åbo Akademi University part of the ICT building in Turku, Finland. These displays include three LEDs, one red, one yellow and one green, as well as a small display, and a button, as shown in Figure 4.25. A user can control it either by a remote control in their rooms or via a web interface. One can also reconfigure the remote control via the web interface, as well as change the text on the screen or add small images. If granted access to the database where the door display data is stored, one can change the displays information from there by changing display specific values which holds the current light status and the text displayed on the screen. All but the profile picture was gathered by a Python script using the SQLAlchemy Python package for accessing the database. A wrapper for the database was also cre- ated, including functions for changing values in the database, and checking the door display status. For downloading the profile picture the standard Python package urllib2 was used to fetch the image of a user. We found that the image name was the same as user’s username, making it trivial to figure out the URL to the image. It is then stored in the applications media storage on the file system. With the information gathered from the web page, and the connection to the database, we get the knowledge of where a person work desk is located, and what the person looks like. This information can later be used in applications such as estimating where the person will be next and for face recognition. The door display can get information about a person’s whereabouts as he or she might write it on the display, and chang- ing the lights on the door signals if the person is currently located in his room or not, making it easier to track that person. With being able to change the lights and text of the door display we can use them

43 to notify an users of a person’s current status. For instance if the person is away from the office, the lights can be changed to red and the text is change to "not available".

4.6 Front-end design of Krang

The front-end of Krang was developed using the Twitter Bootstrap as the main front- end framework. Their starter template was used as a base HTML template, and jQuery was used as the main java script library. The front-end was created with usability and minimalism in mind, and was created to be used as an extension of the research.it.abo.fi webpage, to show more information about the people working at the Department of In- formation Technologies at Åbo Akademi University. The core view of the application is shown in Figure 4.26. If not stated otherwise, all calls being made from the front- end to the back-end are made using Ajax. This means that the requests are done in the background and pages does not have to reload when a request is made. All requests are either done by jQuerys ajax, get or post methods. The core design of the page includes a navigation bar at the top, including the name of the application, as well as quick links to currently available pages of the users. To the far right of the navigation bar is the user management, if the user is not logged in it shows log in prompt, else it includes links to the users profile and settings as well as a log out link. The rest of the page contains the content of the current page viewed. A footer including copyright information is also present on every page viewed. The landing page, when first accessing the application, was design to get the user authenticated with the system as fast as possible. It is also the page that non-authenticated users are redirected to if they try to access a page that requires authentication. It in- cludes a welcome message, and a large log-in button. The button, when pushed, opens the log in prompt in the top right corner.

4.6.1 Profile

The profile page consists of three main parts, a quick view of the user, a user calendar and a history section as shown in Figure 4.26. The quick view includes an image of the user, which is fetched from the research.it.abo.fi page, and some information about the user. The information includes the name, door lights, location, status, phone number, room number and e-mail, as well as social media locations in which the user is active. The door lights are created to resemble the lights outside each person’s door, including

44 three small icons corresponding to each light on the door display. If the light outside the door is on, that is also shown on the webpage. If a light is not on, it is shown as grey on the users profile page. If the user is logged in these icon are interactive, and one can click on them to change their state. The location and status texts are also interactive if the viewer of the profile is also the owner of it. The fields are created as span elements, with the HTML5 property contenteditable set to true. This gives them the property of being editable if a user clicks on the text. Whenever a change is made in the quick view, a save button appears below the information. This button is only shown if a change is made, and if clicked hides away after some seconds.

Figure 4.26: User profile page in Krang with functional elements marked.

The calendar is created by a template tag, which fetches information from the Google Calendar app. This means that the calendar is defined in its own template, and then imported by the profile page. It lists the dates of the current month, without any weekday or month information. It was made to look as simplistic as possible, to follow the applications minimalistic design. The calendar dates color is black by default and light blue if an event is scheduled for a certain date. The events corre- spond to the events located in the Google Calendar that is chosen on the settings page. When clicking on a date with an event, that day’s events are listed below the calendar, including the name of the event as well as its start and end time.

45 The history section is located below the quick view and the calendar, and includes all the events and tasks that the user has been part of. This includes calendar events, foursquare check-ins or of the user has been recognized by a camera. If the user is logged in, and hovers over an entry in the history a cross icon appears which lets the user delete its history if need be. For creating a more complete look, some buttons and text areas were inserted in the profile page. These includes a text area containing information about what the user is doing at this moment, giving a look at what the system could be able to do by adding more information sources and also a section for sending messages to another user and a button to follow a user’s updates. These are only esthetic features, and have no function, the front-end design is however ready if these functionalities were to be added to the back-end at a later time.

4.6.2 Settings

The settings page includes two sections, one to add social media information, and one to link other services to Krang as shown in Figure 4.27. The social media section in- cludes several well-known social media services such as Facebook, Linked-In, Twitter and Google+. The user can insert their username or an URL to their profiles, which will then show up on their profile page. If a username is inserted, the page tries to recreate the link to the user profile on that particular service. If this fails, the user can manually enter the URL. To save the information a save button is located beneath the section.

46 Figure 4.27: User settings page in Krang.

The account linking section consists of three buttons, one for each service currently implemented, these being Foursquare, Google Calendar and face recognition. While face recognition is part of Krang itself, it is created as an opt-in feature due to moral and ethical reasons. When clicking either the Foursquare or Google Calendar button the user will be sent to the services authentication page, where they will be prompted to insert their username and password. If the users credentials are correct they will be sent back to the settings page, and the button will change color from blue to green signaling that the accounts has now been linked. Clicking the face recognition button will instantly turn the feature on, and the user will be tracked. After enabling any of the services an unlink, and settings button will also appear to the right of the link button. The unlink button will simply unlink the service with Krang, and remove any in- formation connected to it. The settings button will show a settings modal window for each individual service. For Foursquare the users have the choice of Krang not updat- ing their status when a check-in is made, or to only update the status if the location is related to their work, in this case the ICT building in Turku. For Google Calendar the user can select which calendar to use, something which is essential for the system to work. Face recognition settings include the ability to temporarily disable tracking, or to only update the status but not add it to the history.

47 4.7 Use Cases

This section covers use-cases of the Krang demonstrator, and shows how it can be used to track people. By combining technologies, we can create uses that otherwise would be hard to implement. All cases include a flow chart to represent the flow of each use case, as well as the service or application used in each step. The flow charts also include a legend describing the different icons used in the chart.

4.7.1 Complementing location information

Figure 4.28: Flowchart of complementing location information use case.

As our system integrates several ways of tracking people, the information from a per- son’s new location might include information from the current location. If this is the case the two locations can complement each other to create more information about a location as shown in Figure 4.28. This way the system knows more about a location than it did with only one source of information. If the application gets new information about a user’s whereabouts it will first com- pare the new location to the old one. If the location address or venue is the same, the application checks if the new location includes additional information to the current

48 one. If this is the case the additional information is added to the user’s current user location, if not the new location is discarded and the old one is kept. If however the new location does not match either the address or the venue of the old one, then the user’s location is updated to the new location. This kind of behavior can be useful if users use several different ways to broadcast their location, for example Foursquare, GPS and Facebook. If a camera has recognized the person, the location of the camera can also be added to the user current location, as additional data. This way the application can get an as accurate location as possible of a user, including not only the venue or address but also the exact floor, room and coordinates. Applications benefiting from this kind of information can be a patient tracking system, when patient with dementia or Alzheimer’s could be tracked in a hos- pital. Other more general uses can be exact package tracking, where the receiver would know exactly where their package or for tracking workers, to know where employees are currently located.

49 4.7.2 Event notification system

Figure 4.29: Flowchart of event notification Celery task.

Django comes with a build in SMTP client which, if configured, can send out mails to a single or multiple addresses. Krang uses the mail system for sending out mails if a user was going to be late to a meeting. This was done by creating a celery task, as pictured in Figure 4.29, that at a regular schedule checks if there is an upcomming event for all users authenticated with Krang. If an event was scheduled for the next 15 mintutes, Krang checks if the user has a current location. If that is the case, a comparison of locations is made, and if they are not the same, a calculation of the distance and drive time from the users current location to the one of the events is made. This is done using the Google Distance Matrix (GDM), which takes two addresses returning the driving distance, as well as the driving time for that route. A calculation if the users could make it in time if he was currently driving is then made, by taking the delta of the current time and the driving time added the to time when the users last location was

50 set, as shown in algorithm 1. If the time delta is negative, the users is going to be late to the meeting. Krang will then check if there are other participants who has responded to the event. If this is the case, these peoples email addresses are gathered in a list and a mail will be sent out to them contaning information about who is going to be late, as well as by how much. This kind of systems can be useful for companies or organization to message meet- ing or event participants about people who are going to be late. If deployed at a large scale it could also be used for workers who have set appointments with clients, to no- tify if they are going to be late. An example use case for this would be a hostipal where doctors could be notified if patients are about to be late, and let the doctor take another client in between.

Algorithm 1 (Late time calculation)

∆T = ct − (dt + ut)

51 4.7.3 Information display with user data

Figure 4.30: Flowchart of a Celery task to change a status display using multiple data sources.

Information displays or lights, such as the one described in subsection 4.5.6, which lets people display their current status or whereabout can be installed outside offices or meeting rooms to let others know if the person is busy or if the room is currently occupied. These status displays often have to be manually changed to the right status, and are often left unused, as users forget or simple does not bother changing the status. Our system can with the help of people tracking and face recognition set the status of such displays without user interaction. Figure 4.30 shows a flowchart how such as system works. In the chart the application first check for events in a meeting room, given a interval.

52 By creating an Celery task that runs at a regular schedule. The task would then check user calendars, and looking for events with the same location as the room. If such events are found then check if the current time is between the start and end time of the event. If this is the case the information display changes to occupied by setting a certain led or showing text on a display. The application could also be configured to check for an upcoming event and change the status in before it occurs. If however there are no events schedules for a room, an alternative route could be taken, the application could check for people inside the room instead. This can be done by looking for faces inside the room. This can be done by first checking for cameras inside the meeting room, and then starting to stream data from the camera back to our application which processes the video. If a face, or several faces are found in the room, then it might be in use. To be sure, a re-check timer can be set to check the status of the room some minutes later. If there are still faces found in the room, it is probably in use, and we can se the information display to occupied. This means that the application could work even if the people inside the room are not users of the application, and we don’t have access to their calendars. Systems like this could be used in offices buildings where big companies, or where several companies share the same spaces. The application could also be linked to an existing booking server, which would show if a room is currently in use, or if a booking can be made immediately. Other uses could also be in aiding larger schools where lecturers need to book rooms in advance, by showing rooms currently in use and history about how rooms have been used in the past.

4.8 Web SIB Explorer and Smart-M3 Tutorial

Developers using Smart-M3 for the first time might find the learning curve a bit too steep as there is almost no documentation to read, and only a few open-source projects use the Smart-M3. For this reason we created a small tutorial introducing new users to Smart-M3 in parallel with creating Krang. It goes through the process of creating a small home automation application using the Python kernel processing interface (KPI). The tutorial was releases as a TUCS technical report, and can be accessed at WHERE IS THE TUTORIAL ?!?!

53 Figure 4.31: Web SIB Explorer Class Tree view.

We also noticed that when creating and working with ontologies for the Smart-M3, debugging became a time consuming task. One would have to write code for testing queries, and create parsers for the results, in order to debug. To solve this problem the Web SIB Explorer was created, which enabled us to get a quick overview of the SIB and to test out queries and subscriptions with ease. It was created as a web application, made to run locally, using the small Python web framework Flask. The application includes features such as:

1. Listing all classes and properties.

2. A class tree view for navigating ontology classes and their objects, as well as delete objects as can shown in Figure 4.31.

3. A querying feature using the CodeMirror code editor for syntax highlighting and bracket matching functionality.

4. Subscription handling using a SQLite database for storing current subscriptions and data about them.

5. A small benchmark feature to give a quick overview of how fast the SIB was running on different platforms.

The application can be accessed at the applications github repository at http://eslab.github.com/Web- SIB-Explorer/.

54 5 DISCUSSION

5.1 Conclusions

By using multiple sources of data we created a web service which was able to track people, given a user’s consent. The program integrates technologies such as Foursquare, Google Calendar and face recognition for getting the user location. All technologies integrated into Krang existed before the creation of our solution. We created Krang as a glue between the services and devices, to make them aware of each other. In total we used over ten existing technologies for creating one unified smart system. For this we had to use storage solutions that integrate well with software using several data sources. The program uses a dual database setup, using a PostgreSQL database for redundancy as well as for storing sensitive data such as user credentials and a Smart-M3 triple store to store all non-sensitive information. By using Smart-M3s subscribe feature, other devices could also act upon changes in the database, which makes our software more modular. For storing image and video data we opted to use the file system of the operating systems to store the actual data and store Meta-data about for files in the databases. This gives more control over the data, and if required enables easy exportation of media files. We demonstrated how other devices and systems can use the data collected by our software for their benefit. Status displays outside offices could be change according to the user’s current location, and other users could be notified if someone is going to be late to an event or meeting. This required integrating several other technologies such as Google Distance Matrix, and SMTP. It also showed how existing systems could benefit from ours. Firstly we have shown that by combining technologies and by making them aware of each other we could create a more intelligent system. Secondly, new solutions to create data for smart-spaces might be redundant, as usable source of information might already exist, but used in a different way. Companies such as Facebook have stored

55 user data and their activities for several years, and are now starting to show the data’s true possibilities. Facebook Graph Search is a feature in which users can use natural language to search for information about other people on Facebook. The feature is using the vast amount of information gathered from its users; data that they had access to several years ago, but have only now been using. With the increasing interest in data mining, the internet of things as well as combining data from multiple sources, computers might soon know more about our lives than we do.

5.2 Future work

While adding more data sources is always an improvement to a tracking system, a more important feature would be the ability to create rules. Features presented in the thesis are all hard coded examples, and there is no syntax for creating rules. Creating rule syntax, and an interpreter for this, would greatly improve the usability of the application. To let users create their own rules would also add to the usability, as well as user-friendliness of the application. A suggested syntax for the rules would be to have an if-this-then-that structure, where the user would first select a trigger, this, and then add an action, that, if the trigger gets a hit. Other improvements include probabilistic models, in which the system could pre- dict future movement of its users. This would open up possibilities such as reschedul- ing of events if the system notices an abnormality in the predicted route a user has taken, or to create a more effective daily work routine for people on the run. This could also be applied in other areas such as tracking packages or as an early warning system for planes and boats.

56 BIBLIOGRAPHY

[1] Bitmanufaktur GmbH. Openbeacon active rfid project. URL: http://www. openbeacon.org/OpenBeacon_Active_RFID_Project.

[2] Philipp Wagner. local binary patterns. URL: http://www.bytefish.de/blog/local_ binary_patterns.

[3] Ville Arola, Niko Kurvinen, and Eero Siniluhta. Local binary pattern. URL: https: //noppa.lut.fi/noppa/opintojakso/ct50a6100/lisatty/local_binary_pattern.pdf.

[4] Social Lipstick. Under the covers of oauth 2.0 at facebook, 5 2010. URL: http: //www.sociallipstick.com/?p=239.

[5] Peter Murray. Heathrow airport to adopt face scanners to screen passengers. URL: http://singularityhub.com/2011/08/06/ heathrow-airport-to-adopt-face-scanners-to-screen-passengers/.

[6] Gerrit Hornung. The european regulation on biometric passports: Legislative pro- cedures, political interactions, legal framework and technical safeguards. URL: http://www.law.ed.ac.uk/ahrc/script-ed/vol4-3/hornung.asp.

[7] A.C. Weaver. Biometric authentication. Computer, 39(2):96 – 97, feb. 2006. doi:10.1109/MC.2006.47.

[8] S. Liu and M. Silverman. A practical guide to biometric security technology. IT Professional, 3(1):27 –32, jan/feb 2001. doi:10.1109/6294.899930.

[9] T. Ahonen, A. Hadid, and M. Pietikainen. Face description with local binary patterns: Application to face recognition. Pattern Analysis and Machine Intelli- gence, IEEE Transactions on, 28(12):2037 –2041, dec. 2006. doi:10.1109/ TPAMI.2006.244.

[10] L. Wiskott, J.-M. Fellous, N. Kuiger, and C. von der Malsburg. Face recog- nition by elastic bunch graph matching. Pattern Analysis and Machine Intelli- gence, IEEE Transactions on, 19(7):775 –779, jul 1997. doi:10.1109/34. 598235.

57 [11] M.A. Turk and A.P. Pentland. Face recognition using eigenfaces. In Computer Vi- sion and Pattern Recognition, 1991. Proceedings CVPR ’91., IEEE Computer So- ciety Conference on, pages 586 –591, jun 1991. doi:10.1109/CVPR.1991. 139758.

[12] Andrew J. Hopwood, Cedric Hurth, Jianing Yang, Zhi Cai, Nina Moran, John G. Lee-Edghill, Alan Nordquist, Ralf Lenigk, Matthew D. Estes, John P. Haley, Colin R. McAlister, Xiaojia Chen, Carla Brooks, Stan Smith, Keith Elliott, Pieris Koumi, Frederic Zenhausern, and Gillian Tully. Integrated microfluidic system for rapid forensic dna analysis: Sample collection to dna profile. Analytical Chemistry, 82(16):6991–6999, 2010. URL: http://pubs.acs.org/doi/abs/10.1021/ ac101355r, doi:10.1021/ac101355r.

[13] Fan Jing and Jiao Ruili. New fast algorithm on dna sequence alignment. In Bioinformatics and Biomedical Engineering , 2009. ICBBE 2009. 3rd Interna- tional Conference on, pages 1 –4, june 2009. doi:10.1109/ICBBE.2009. 5163102.

[14] L.P. Hewlett-Packard Development Company. Hp notebook pcs - using finger- print reader to login to applications and web sites. URL: http://h20000.www2. hp.com/bizsupport/TechSupport/Document.jsp?objectID=c00958213.

[15] Ms. Smith. Laptop fingerprint reader destroys ’entire security model of windows accounts’. URL: http://www.networkworld.com/community/blog/ laptop-fingerprint-reader-destroys-entire-security-model-windows-accounts.

[16] Mislav Grgic and Kresimir Delac. Face recognition homepage: Algorithms. URL: http://www.face-rec.org/algorithms/.

[17] D. Ramanan, D.A. Forsyth, and A. Zisserman. Strike a pose: tracking people by finding stylized poses. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, volume 1, pages 271 – 278 vol. 1, june 2005. doi:10.1109/CVPR.2005.335.

[18] D.M Gavrila. The visual analysis of human movement: A survey. Computer Vision and Image Understanding, 73(1):82 – 98, 1999. doi:10.1006/cviu. 1998.0716.

[19] T.L. Berg, A.C. Berg, J. Edwards, M. Maire, R. White, Yee-Whye Teh, E. Learned-Miller, and D.A. Forsyth. Names and faces in the news. In Com- puter Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, volume 2, pages II–848 – II–854 Vol.2, june-2 july 2004. doi:10.1109/CVPR.2004.1315253.

[20] Microsoft Corporation. Kinect for windows. URL: http://www.microsoft.com/ en-us/kinectforwindows/.

58 [21] Michael Zollhöfer, Michael Martinek, Günther Greiner, Marc Stamminger, and Jochen Susmuth. Automatic reconstruction of personalized avatars from 3d face scans. Computer Animation and Virtual Worlds, 22(2-3):195–202, 2011. URL: http://dx.doi.org/10.1002/cav.405, doi:10.1002/cav.405. [22] T. Leyvand, C. Meekhof, Yi-Chen Wei, Jian Sun, and Baining Guo. Kinect identity: Technology and experience. Computer, 44(4):94 –96, april 2011. doi:10.1109/MC.2011.114. [23] K.K. Biswas and S.K. Basu. Gesture recognition using microsoft kinect. In Automation, Robotics and Applications (ICARA), 2011 5th International Con- ference on, pages 100 –103, dec. 2011. doi:10.1109/ICARA.2011. 6144864. [24] Bangpeng Yao and Li Fei-fei. Grouplet: A structured image representation for recognizing human and object interactions. In Computer Vision and Pat- tern Recognition (CVPR), 2010 IEEE Conference on, pages 9 –16, june 2010. doi:10.1109/CVPR.2010.5540234. [25] M. Van den Bergh, D. Carton, R. de Nijs, N. Mitsou, C. Landsiedel, K. Kuehn- lenz, D. Wollherr, L. van Gool, and M. Buss. Real-time 3d hand gesture interac- tion with a robot for understanding directions from humans. In RO-MAN, 2011 IEEE, pages 357 –362, 31 2011-aug. 3 2011. doi:10.1109/ROMAN.2011. 6005195. [26] Simon Hay and Robert Harle. Bluetooth tracking without discoverabil- ity. In Tanzeem Choudhury, Aaron Quigley, Thomas Strang, and Koji Suginuma, editors, Location and Context Awareness, volume 5561 of Lec- ture Notes in Computer Science, pages 120–137. Springer Berlin Heidelberg, 2009. URL: http://dx.doi.org/10.1007/978-3-642-01721-6_8, doi:10.1007/ 978-3-642-01721-6_8. [27] Ian Paul. Google street view’s wi-fi snooping engineer is outed. URL: http://www.pcworld.com/article/254774/google_street_views_wi_fi_snooping_ engineer_is_outed.html. [28] T. Mori, Y. Suemasu, H. Noguchi, and T. Sato. Multiple people tracking by integrating distributed floor pressure sensors and rfid system. In Systems, Man and Cybernetics, 2004 IEEE International Conference on, volume 6, pages 5271 – 5278 vol.6, oct. 2004. doi:10.1109/ICSMC.2004.1401031. [29] Frank Miskelly. Electronic tracking of patients with dementia and wan- dering using mobile phone technology. Age and Ageing, 34(5):497–499, September 2005. URL: http://ageing.oxfordjournals.org/content/34/5/497.short, arXiv:http://ageing.oxfordjournals.org/content/34/5/ 497.full.pdf+html, doi:10.1093/ageing/afi145.

59 [30] K. Michael, A. McNamee, and M. Michael. The emerging ethics of humancentric gps tracking and monitoring. In Mobile Business, 2006. ICMB ’06. International Conference on, page 34, june 2006. doi:10.1109/ICMB.2006.43.

[31] Ingrid Burbey. Predicting future locations and arrival times of individuals.

[32] SARA MALM and LUCY OSBORNE. Mobile phone companies can predict future movements of users by building a profile of their lifestyle. URL: http://www.dailymail.co.uk/sciencetech/article-2190531/ Mobile-phone-companies-predict-future-movements-users-building-profile-lifestyle. html.

[33] Will Oremus. What happens when our cellphones can predict our every move? URL: http://www.slate.com/blogs/future_tense/2012/08/08/cellphone_tracking_ what_happens_when_our_smartphones_can_predict_our_every_move_.html.

[34] D. Ashbrook and T. Starner. Learning significant locations and predicting user movement with gps. In Wearable Computers, 2002. (ISWC 2002). Proceedings. Sixth International Symposium on, pages 101 – 108, 2002. doi:10.1109/ ISWC.2002.1167224.

[35] M. Bassick, T. McNamara, and D. Sullivan. Employee surveillance: An ethical consideration. The Ethical Imperative in the Context of Evolving Technologies, page 77.

[36] Django Software Foundation. The web framework for perfectionists with dead- lines. URL: https://www.djangoproject.com/.

[37] Ask solem and contributors. Celery: Distributed task queue. URL: http://www. celeryproject.org/.

[38] Inc. VMware. Rabbitmq - messaging that just works. URL: http://www.rabbitmq. com/.

[39] NETCRAFT LTD. January 2013 web server survey. URL: http://news.netcraft. com/archives/2013/01/07/january-2013-web-server-survey-2.html.

[40] Mondotech. Apache 2.4 vs nginx benchmark showdown, February. URL: http: //mondotech.blogspot.fi/2012/02/apache-24-vs-nginx-benchmark-showdown. html.

[41] S.Dmitrij. Nginx vs cherokee vs apache vs lighttpd, April 2012. URL: http: //www.whisperdale.net/11-nginx-vs-cherokee-vs-apache-vs-lighttpd.html.

[42] Joshua Zhu. Apache 2.4 faster than nginx?, February 2012. URL: http://blog. zhuzhaoyuan.com/2012/02/apache-24-faster-than-nginx/.

60 [43] Unbit. uwsgi. URL: http://projects.unbit.it/uwsgi/.

[44] Benoit Chesneau. Gunicorn - python wsgi http server. URL: http://gunicorn.org/.

[45] Paul J. Davis. Why do i need nginx and something like guni- corn?, November 2011. URL: http://serverfault.com/questions/331256/ why-do-i-need-nginx-and-something-like-gunicorn.

[46] nosql database.org. Nosql databases. URL: http://nosql-database.org/.

[47] The PostgreSQL Global Development Group. Postgresql. URL: http://www. postgresql.org/.

[48] Michael Larabel. Linux 3.7 file-system benchmarks: Ext4, btrfs, xfs, November 2012. URL: http://www.phoronix.com/scan.php?page=article&item=linux_37_ fsthree&num=1.

[49] Woodrow W. Bledsoe. The model method in facial recognition.

[50] Woodrow W. Bledsoe. A man-machine facial recognition system-some prelimi- nary results.

[51] Phillip Ian Wilson and John Fernandez. Facial feature detection using haar classifiers. J. Comput. Sci. Coll., 21(4):127–133, April 2006. URL: http: //dl.acm.org/citation.cfm?id=1127389.1127416.

[52] M.A. Turk and A.P. Pentland. Face recognition using eigenfaces. In Computer Vi- sion and Pattern Recognition, 1991. Proceedings CVPR ’91., IEEE Computer So- ciety Conference on, pages 586 –591, jun 1991. doi:10.1109/CVPR.1991. 139758.

[53] P.N. Belhumeur, J.P. Hespanha, and D.J. Kriegman. Eigenfaces vs. fisherfaces: recognition using class specific linear projection. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 19(7):711 –720, jul 1997. doi:10.1109/ 34.598228.

[54] W3Techs. Usage of javascript libraries for websites. URL: http://w3techs.com/ technologies/overview/javascript_library/all.

61 SAMMANFATTNING

SUPERBRUBRIK

This chapter is left empty until the structure and text of the English version is close to complete.

Introduktion

Implementering

Sammanfattning

62 Appendices

63 ACONFIGURATIONS

A.1 uWSGI

1 [uwsgi] 2 socket = /tmp/.sock 3 chdir = /PATH/TO/DJANGO/PROJECT 4 pythonpath = /PATH/TO/VIRTUAL_ENV 5 env = DJANGO_SETTINGS_MODULE=.settings 6 module = django.core.handlers.wsgi:WSGIHandler() 7 aster = true 8 processes = 4 9 daemonize = /PATH/TO/DEAMONIZE/LOG.log 10 pidfile = /PATH/TO/PIDFILE.pid

A.2 Nginx

1 server { 2 listen 443; 3 server_name localhost; 4 ssl on; 5 ssl_certificate /PATH/TO/server.crt; 6 ssl_certificate_key /PATH/TO/server.key; 7 8 9 location / { 10 include uwsgi_params; 11 uwsgi_pass unix:///tmp/krang.sock; 12 } 13

64 14 location /media { 15 root /DJANGO/PROJECT/MEDIA_DIR; 16 } 17 18 location /static { 19 root /DJANGO/PROJECT/STATIC_DIR; 20 } 21 }

65 BWEBSCRAPING

B.1 research.it.abo.fi login

1 def itLogin(self, username, password): 2 """ Logs in to research.it.abo.fi page 3 Args: 4 username(str): Users username 5 password(str): Users password 6 Returns: 7 True if the user was successfully logged in else False. 8 """ 9 10 logger.info("Logging in with:"+username) 11 12 13 self.getPage(self.URL_LOGIN) 14 self.br.select_form(nr=1) #2nd form= login 15 self.br['__ac_name'] = username # Set username 16 self.br['__ac_password'] = password # Set password 17 self.br.submit() # Submit form 18 19 soup = BeautifulSoup(self.br.response()) 20 logger.info("Got soup") 21 22 try : 23 # Element not be present if we have logged in 24 error = soup.find('dl',{'class':'portalMessage error'})

66 25 26 if not error: 27 self.logins['it'] = True 28 logger.info("All ok, returning True") 29 return True 30 else: 31 logger.info("Got an error logging in, returning False") 32 return False 33 34 except AttributeError: 35 logger.info("Attribute error!") 36 return False

67