EXPLOITING SENSOR DATA SEMANTICS FOR SMARTPHONE-BASED LIFELOGGING: TOWARDS THE DEVELOPMENT OF DIGITAL PROSTHETIC MEMORY ON SMARTPHONES
By
SHAUKAT ALI
Supervised By
PROF. DR. SHAH KHUSRO
DEPARTMENT OF COMPUTER SCIENCE UNIVERSITY OF PESHAWAR, PESHAWAR, PAKISTAN
(SESSION 2009-2010) ii
EXPLOITING SENSOR DATA SEMANTICS FOR SMARTPHONE-BASED LIFELOGGING: TOWARDS THE DEVELOPMENT OF DIGITAL PROSTHETIC MEMORY ON SMARTPHONES
By
SHAUKAT ALI
This Thesis is submitted to the Department of Computer Science University of Peshawar in Partial Fulfillment of the Dissertation Requirements for the Degree of Ph.D. in Computer Science
DEPARTMENT OF COMPUTER SCIENCE UNIVERSITY OF PESHAWAR, PESHAWAR, PAKISTAN
(SESSION 2009-2010) i
Certificate of Approval
This is to certify that the research work presented in this thesis, entitled "Exploiting Sensor Data
Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic
Memory on Smartphones" was conducted by Mr. Shaukat Ali under the supervision of Prof.
Dr. Shah Khusro.
No part of this thesis has been submitted anywhere else for any other degree. This thesis is submitted to the Department of Computer Science, University of Peshawar in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the field of Computer Science.
Student Name: Mr. Shaukat Ali Signature: ______
Examination Committee
a) External Examiner 1: Dr. Jamil Ahmad Signature: ______Chairman/Assistant Professor Department of Computational Sciences Research Center for Modeling & Simulation NUST, Pakistan. b) External Examiner 2: Dr. Engr. Sami ur Rahman Signature: ______Assistant Professor Department of Computer Science & IT University of Malakand, Pakistan. c) Internal Examiner: Prof. Dr. Azhar Rauf Signature: ______Professor Department of Computer Science University of Peshawar, Pakistan.
Supervisor Name: Prof. Dr. Shah Khusro Signature: ______
Name of Dean/HOD: Prof. Dr. Shah Khusro Signature: ______
ii
Forward Certificate
The research entitled “Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging:
Towards the Development of Digital Prosthetic Memory on Smartphones” is conducted under my supervision and the thesis is submitted to University of Peshawar in partial fulfillment for the degree of Ph.D. in Computer Science.
______Dr. Shah Khusro Professor Department of Computer Science University of Peshawar
iii
Author's Declaration
I, Mr. Shaukat Ali, hereby state that my PhD thesis titled "Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on
Smartphones" is my own work and has not been submitted to previously by me for taking any degree from this University of Peshawar or anywhere else in the country/world.
At any time if my statement is found to be incorrect even after my graduation the university has right to withdraw my PhD degree.
______Mr. Shaukat Ali Date: 12/01/2018
iv
Rights of Thesis
Copyright © 2018 by Shaukat Ali
All rights reserved. No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, or other electronic or mechanical methods, without the prior written permission of the researcher.
______Mr. Shaukat Ali [email protected]
v
Dedication
Dedicated to My loving parents & family
vi
Acknowledgements
First, I would like to present my sincere thanks to Almighty Allah for giving me the strength and courage to complete the tedious and hard course of Ph.D. and coming up with this thesis. I would like to dedicate this thesis to my loving parents and family members who have been a constant source of inspiration for me. I am especially thankful to my wife for her unselfish support and encouragement. Certainly, I could not have survived during my Ph.D. without their support, motivation, and endless love.
I acknowledge the inspirational instructions and guidance of Professor Dr. Shah Khusro who is my Ph.D. supervisor. His valuable guidance always remained my strength, which not only gave me the deep knowledge and support of the subject but also straighten my research directions and research methodology. His intelligence and scientific vision empowered me of leaving no stone unturned during my Ph.D. His profound knowledge and innovative guidance always impressed me and saved me from disaster whenever I faced any. I am feeling no doubt in my mind to acknowledge that without his constant support and guidance I could not have accomplished my
Ph. D. process. I am thankful to my lab mates Mr. Irfan Ullah, Mr. Akif Khan, and Mr. Nasir
Khan for accompanying me in my every day research work. Beyond all word, it was great and full of excitement to work with them. Lastly, I cannot forget my beautiful children. I hope that when they grow up, they will understand that why I could not give them my attention and time during some of important years of their lives. This Ph.D. research work is supported by Higher
Education Commission (HEC) of Pakistan under Indigenous Fellowship 5000 Program. I am also thankful to HEC for his support.
MR. SHAUKAT ALI vii
Abstract
The paradigm of lifelogging promises the development of automatic systems for recording users' life events information digitally and develops an electronic prosthetic memory for providing complementary assistance to human biological memory. Several lifelogging systems are invented by assimilating computing and sensory technologies to capture, annotate, and retrieve lifelong information. The wearable technology has gained market traction; however, has several limitations including hard-to-work environment, number of sensors, uni-faceted, etc. Therefore, they are unable to use contextual semantics for organizing lifelog information like in human episodic memory. In addition, the large-scale adaptation of lifelogging is possible, if lifelogging functionality is integrated in devices that are already owned and maintained by users.
To bridge the gap, this thesis examines smartphone technology for developing a common understanding of using smartphone as a de-facto lifelogging device. The various contextual semantics from smartphone sensors data and their potential applications in lifelogging are identified. The semantic model (i.e., ontology) is developed and tested for using the contextual semantics to organize, annotate, and relate lifelog information in similar to human episodic memory, and provide enough contextual cues to recall lifelog information like associative recall in human memory. The semantic framework is proposed to unify the research efforts by incorporating smartphone's sensors and processing capabilities, and semantic model to develop a semantically enriched digital prosthetic memory on smartphone. The proof-of-concept application called Semantic Lifelogging (SLOG) is developed to show the practicality of the proposed framework. The empirical evaluation has shown effectiveness of the methodology. In addition, future research directions are highlighted to help researchers in finding research topics. viii
Table of Contents
Certificate of Approval ...... i
Forward Certificate ...... ii
Author's Declaration ...... iii
Rights of Thesis ...... iv
Dedication ...... v
Acknowledgements ...... vi
Abstract ...... vii
Table of Contents ...... viii
List of Figures ...... xiii
List of Tables ...... xv
List of Abbreviations ...... xvii
Chapter 1 : Introduction ...... 1
1.1 Introducing Lifelogging ...... 3
1.2 Lifelogging Types ...... 5
1.3 Terminologies...... 6
1.3.1 Lifelogging and Personal Information Management (PIM) ...... 6
1.3.2 Lifelogging and Context-Awareness ...... 8
1.3.3 Lifelogging and Lifelog ...... 9
1.4 Lifelogging and Human Memory ...... 9
1.4.1 Episodic and Semantic Memory ...... 10
1.4.2 Lifelogging for Memory Augmentation ...... 11
1.5 Impacts of Lifelogging ...... 13 ix
1.6 Research Motivation ...... 15
1.7 Statement of Thesis ...... 17
1.8 Research Objectives ...... 18
1.9 Research Significance ...... 18
1.10 Thesis Organization...... 19
1.11 Summary ...... 20
Chapter 2 : Towards Smartphone-Based Lifelogging ...... 21
2.1 History and Background of Lifelogging ...... 22
2.1.1 Desktop Metaphors and PIM ...... 23
2.1.2 Lifelogging via Wearable Computing ...... 24
2.1.3 Limitations of Conventional Lifelogging ...... 25
2.2 Overview of Smartphone-Based Lifelogging ...... 27
2.2.1 Generalized Architecture for Smartphone-Based Lifelogging Systems ...... 29
2.2.2 Taxonomies of Smartphone-Based Lifelogging Systems ...... 34
2.2.3 Smartphone-Based Lifelogging Systems...... 40
2.2.4 Limitations of Smartphone-Based Lifelogging Systems ...... 46
2.3 Use-Cases for Smartphone-Based Lifelogging ...... 54
2.3.1 Memory Augmentation and Assistance ...... 55
2.3.2 User Modeling ...... 55
2.3.3 Desktop Customization...... 56
2.3.4 Health Monitoring ...... 56
2.3.5 Environmental Impacts ...... 58
2.4 Summary ...... 58 x
Chapter 3 : Materializing Smartphone as a Lifelogging Device...... 61
3.1 Smartphone Sensors ...... 63
3.1.1 Smartphone Sensors for Lifelogging ...... 66
3.1.2 Smartphone Sensors Data Generation Strength...... 69
3.1.3 Smartphone Sensors Battery Power Consumption ...... 73
3.1.4 Smartphone Sensing: A New Application Paradigm ...... 79
3.2 Adaptability and Importance of Smartphone Technology ...... 84
3.2.1 Improved Technology ...... 86
3.2.3 Enhanced Efficiency ...... 95
3.2.4 Functionalities and Ultra Utilities...... 95
3.2.5 Easy Entertainment Accessibility ...... 96
3.3. Smartphone and Context-Awareness ...... 96
3.3.1 Passive Visual and Audio Contexts ...... 97
3.3.2 Location Context ...... 98
3.3.3 Physical Activities Context ...... 100
3.3.4 Personal Biometric and Environmental Context ...... 101
3.3.5 Communication Activities Context ...... 103
3.4 Common Daily Life Activities on Smartphone...... 103
3.5 Smartphone vs. Dedicated Lifelogging Devices ...... 106
3.6 Summary ...... 110
Chapter 4 : Semantic Modeling of Smartphone Sensors and Lifelog Data ...... 111
4.1 Semantic Web Technologies ...... 114
4.1.1 Ontologies ...... 115 xi
4.1.2 Ontology Description Languages ...... 118
4.2 SmartOntoSensor Ontology ...... 120
4.2.1 Goals and Design Rationales ...... 122
4.2.2 Materials and Methodology ...... 124
4.2.3 Evaluation and Discussion...... 149
4.2.4 SmartOntoSensor and Lifelogging ...... 162
4.3 Summary ...... 163
Chapter 5 : Proposed Framework and Implementation ...... 166
5.1 Framework Architecture ...... 167
5.1.1 Data Layer ...... 168
5.1.2 Semantic Layer ...... 174
5.1.3 Application Layer ...... 184
5.2 Proposed Framework Features ...... 185
5.2.1 Openness and Extendable ...... 185
5.2.2 Increased Sensors Coverage ...... 186
5.2.3 Semantic Lifelog Information Organization ...... 187
5.2.4 Improved Annotation ...... 188
5.2.5. Easy and Fine-grained Retrieval ...... 189
5.2.6 Improved User Modeling and Personalization ...... 190
5.2.7 Improved Surveillance and Sousveillance ...... 190
5.2.8 Multiplicity of Use-Cases ...... 191
5.3 Semantic Data Model ...... 191
5.4 Proof-of-Concept Prototyping ...... 194 xii
5.4.1 SLOG Implementation ...... 195
5.4.2 Location Lifelogging ...... 203
5.4.3 SMS Lifelogging ...... 204
5.4.4 Evaluation ...... 207
5.5 Summary ...... 214
Chapter 6 : Conclusion and Future Work ...... 216
6.1 Limitations and Future Work ...... 221
Publications ...... 226
References ...... 228
xiii
List of Figures
Figure 1.1: (a) Selective capture lifelogging; (b) Total capture lifelogging...... 7
Figure 2.1: General architecture of smartphone-based lifelogging systems...... 30
Figure 2.2: Taxonomy using smartphone deployment and role in the lifelogging process...... 36
Figure 2.3: Taxonomy of smartphone-based lifelogging systems based on their architectures. .. 38
Figure 2.4: Taxonomy of smartphone-based lifelogging systems based on their scope...... 39
Figure 2.5: Taxonomy of smartphone-based lifelogging systems based on storage modeling. ... 40
Figure 3.1: Layered architecture for SAVE...... 70
Figure 3.2: Screen shots of SAVE...... 71
Figure 3.3: Three-layer architecture of EnergyMonitorApp...... 75
Figure 3.4: Main user interface screenshots of EnergyMonitorApp...... 77
Figure 3.5: General architecture of people-centric smartphone sensing applications...... 83
Figure 3.6: Application domains of smartphone sensing applications...... 85
Figure 3.7: Smartphones sales by different vendors in the first quarter of 2017 and 2016...... 87
Figure 3.8: Number of apps available in the leading app stores by March 2017...... 92
Figure 3.9: Average number of apps used and time spent per month [148]...... 105
Figure 3.10: Users' common daily life activities on smartphones and execution time...... 106
Figure 3.11: Wearability comparison of SenseCam (left) and smartphone (right) [13]...... 107
Figure 4.1: 3SOC ontology design pattern for SOS...... 124
Figure 4.2: Abstract level structure of SOS...... 130
Figure 4.3: SOS Framework...... 132
Figure 4.4: Snippet of SOS concepts hierarchy...... 144
Figure 4.5: Snippet of SOS 'Sensor' class hierarchy...... 145 xiv
Figure 4.6: Objective analysis of SOS using multi-criteria approach...... 157
Figure 4.7: SPARQL query for retrieving locations having noise intensity greater than 65dB. 158
Figure 4.8: SPARQL query for detecting a context and service using low level sensory data. . 159
Figure 4.9: SOS asserted and inferred class hierarchies after using reasoner...... 161
Figure 4.10: Changing modes by ModeChanger using contextual information from SOS...... 164
Figure 5.1: Layered view of the proposed architecture...... 169
Figure 5.2: Sensors classification in the proposed semantic framework for lifelogging...... 170
Figure 5.3: Location context information of the received SMS object in JSON format...... 176
Figure 5.4: Metadata information extracted from a received SMS object in JSON format...... 177
Figure 5.5: Clustered information about receiving SMS object and location context...... 178
Figure 5.6: An excerpt of mapping rules for an incoming JSON message...... 179
Figure 5.7: An excerpt of semantic rules for an incoming JSON message...... 180
Figure 5.8: Excerpt of the semantic model created for a SMS event...... 182
Figure 5.9: GUI screenshots...... 197
Figure 5.10: Screenshots of email ID and password setting...... 198
Figure 5.11: Checking of SLOG status before starting an event...... 199
Figure 5.12: SLOG folder 'SLOGDATA' and sub-folders...... 201
Figure 5.13: SPARQL query to retrieve location information using time information...... 205
Figure 5.14: Screenshots of starting MyLocations service and displaying locations on map. ... 206
Figure 5.15: SPARQL query to retrieve SMS information using time information...... 207
Figure 5.16: Screenshots of starting MySMSs service and displaying SMS information...... 208
xv
List of Tables
Table 2.1: Types of memory errors...... 21
Table 2.2: An analysis of the available smartphone-based lifelogging systems...... 42
Table 2.3: Satellite view comparison of available smartphone-based lifelogging systems...... 47
Table 3.1: Sensory technology in some of the modern smartphones...... 65
Table 3.2: Potential applications of smartphone sensors for lifelogging...... 68
Table 3.3: Data volume generation strength of smartphone sensors...... 73
Table 3.4: Scenarios and their compositions...... 78
Table 3.5: Smartphone sensors power consumption in percentage in different activities...... 78
Table 3.6: Swift comparison of smartphones processing technologies...... 89
Table 3.7: Swift comparison of modern smartphones storage technologies and capacities...... 90
Table 3.8: Swift comparison of modern smartphones' size, weight, battery and user interface. .. 94
Table 3.9: Overview of using smartphone sensors for passive visual and audio contexts...... 99
Table 3.10: Overview of using smartphone sensors for localization by lifelogging apps...... 100
Table 3.11: Abstract level comparison of smartphone-based activity recognition researches. .. 102
Table 3.12: Comparison of Narrative Clip 2 and Samsung Galaxy Note 7 smartphone...... 109
Table 4.1: An excerpt of the SOS lexicon...... 126
Table 4.2: An excerpt of the SOS competency questions...... 127
Table 4.3: Relationship between scenarios and iterations...... 128
Table 4.4: An excerpt of SOS object properties...... 148
Table 4.5: An excerpt of SOS datatype properties...... 149
Table 4.6: Top-level terminological requirements fulfillment (using availability of concepts). 151
Table 4.7: Statistics of SOS and other sensors and sensor networks ontologies using OntoQA.153 xvi
Table 4.8: Hypothetical exemplary contexts and their corresponding modes values...... 162
Table 5.1: Qualitative Comparison of SLOG with Smartphone-Based Lifelogging Systems. .. 210
Table 5.2: Participants responses to the questions in the questionnaire...... 212
Table 5.3: Division of 5 level Likert-Scale response categories into nominal categories...... 213
Table 5.4: Results of the Chi-Square test using SPSS 16.0...... 214
xvii
List of Abbreviations
ASSIST Advanced Solider System and Information Technology
CARPE Continuous Archival and Retrieval of Personal Experiences
CESN Coastal Environment Sensor Network
CSIRO Common Wealth Science and Industrial Research Organization
DARPA Defense Advanced Projects Research Agency
ECG ElectroCardioGram
FFT Fast Fourier Transformation
FOAF Friend of a Friend
GPRS General Packet Radio Service
GPS Geographical Positioning System
GSM Global System for Mobile Communication
HCI Human Computer Interaction iOS iPhone Operating System
IR Information Retrieval
JSON JavaScript Object Notation
LOD Linked Open Data
Memex Memory Extender
Networked Environment for Personalized, Ontology-based NEPOMUK Management of Unified Knowledge
NFC Near Field Communication xviii
OCR Optical Character Recognition
ORSD Ontology Requirement Specifications Document
OWL Web Ontology Language
OWL-DL Web Ontology Language - Description Logic
PDA Personal Digital Assistance
PIM Personal Information Management
POEM Practical Ontology Engineering Model
RDF Resource Description Framework
RDFS Resource Description Framework Schema
RFID Radio Frequency Identification
SIS Stuff I've Seen
SLOG Semantic Lifelogging
SMS Short Messaging Service
SOAP Simple Object Access Protocol
SOS SmartOntoSensor
SPARQL Simple Protocol and RDF Query Language
SPSS Statistical Package for the Social Sciences
SSN Semantic Sensor Network
TV Television
UI User Interface
1
Chapter 1 : Introduction
The changing lifestyles and interactions result in new life events that crowd people's lives with information that they want to remember. People's functioning as humans largely depends on their abilities of using their biological memories for storing, retaining, and recalling information relevant to their current contexts [1, 2]. However, human memory is fallible and fades, making it difficult to recall details of past-experiences and cure the problem of forgetting things to do [1,
3]. People whether suffering with significant memory problems (e.g., amnesic patients) or not, both experiences difficulties in learning and recalling correct information from their memories due to several possible reasons including age, information size, no proper rehearsing, etc. To cope with the memory issues, people have used different external memory aids (e.g., writing diaries, photo albums, visiting old places, family calendars, reflections, etc.), over the time.
Certainly, these practices are proven helpful in capturing and retrieving information about daily life events using relevant cues and associative browsing [4, 5]. However, manual recording of daily life events is a tedious job and can result into missing the recording of important events. In addition, these recording experiences allow users to record details of only a small portion of their lives based on their priorities. Therefore, they cannot portray one's complete life experiences.
The advancements and proliferation of technologies have enabled people to record almost every moment of their daily life experiences visually, spatially, or verbally in a digital archive [6]. The lifelogging paradigm urges to investigate and develop digital memory prostheses (memory augmentation tools) by using computer technology. These prostheses are aimed to provide complimentary digital assistance to human organic memory in effective remembering of past
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
2 everyday experiences [7]. Prosthetic memory designers have mainly assimilated computer and sensory technologies to devise novel lifelogging devices for annotation, creation of flat collection of lifelog information, and retrieval of captured lifelog information. The lifelogging devices are memory augmentation tools that are intended to proactively capture people's content and contextual information and bring past experiences information to their attention for assisting them in the current situations [2]. In addition, they encourage people to use details of their past- experiences for developing their personal narratives for self-reflection, sharing past-experiences with friends, and performing new tasks or current tasks more efficiently and faster [2].
This research thesis attempts to take it a step forward and addresses the questions. What is the status of smartphone platform in the lifelogging paradigm? How the smartphone technological developments fulfill the sensory, storage, processing, usability, etc., requirements of lifelogging?
What contextual semantics can be extracted from smartphone sensors data how they can relate lifelog information? How smartphone platform has superiority over commercial, dedicated, and custom-built lifelogging devices? How Semantic Web technologies can be used for organizing, annotating, and relating lifelog information in a semantic lifelog model, in alike to human organic (i.e., episodic) memory? How semantic modeling can be useful in retrieving lifelog information to support different use-cases? How a framework using smartphone and Semantic
Web technologies can be defined to develop smartphone-based semantic lifelogging applications for augmenting human autobiographical memory? What are the limitations and future research directions in the smartphone-based semantic lifelogging paradigm? This chapter is aimed to establish introductory background of lifelogging in general and smartphone-based lifelogging in specific and outlines research objectives, motivation, and significances of this thesis.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
3
1.1 Introducing Lifelogging
The idea of automatic generation of personal digital archives starts from 1940's when Vannevar
Bush expressed his vision of an imaginary mechanized device called "Memory Extender
(Memex)" and highlighted the need to develop technology for enabling individuals to record their life information as digital memories which could be accessed with ease and speed [8].
Lifelogging is a buzzword having no universal and agreed definition [9]. However, a widely agreed definition of lifelogging is "a form of pervasive computing, consisting of a unified digital record of the totality of an individual's experiences, captured multi-modally through digital sensors and stored permanently as a personal multimedia archive" [10]. Technically, lifelogging is a self-surveillance technology that produces a comprehensive record of a person's lifetime experiences information in a variety of data types such as texts, multimedia (i.e., pictures, videos, audios), biological data (i.e., blood pressure, heart rate), news, weather information, etc. It is argued that using lifelogging for recalling memory can significantly help individuals in improving their life styles, and performances and productivity. For example, a lifelog could be helpful for the people in time management, retrieving information related to a specific event, analyzing and improving behaviors, enabling devices to behave smartly.
The key aspect of lifelogging is the recording of totality of individual life experiences, which was not possible earlier due to the unavailability or limitations of the capturing and associated technologies [9]. However, the recent advancements in lightweight computing devices have realized new methods for effective autobiography generation. The capturing, storage, processing, and communication technologies have been significantly improved to fulfill the data-intensive nature of lifelogging. The availability of lightweight, inexpensive, and ubiquitous sensors
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
4 enables for automatic and passive capturing of rich multi-model information about people, contexts, and environments. However, the type and number of sensors used is largely subjected on the kind of lifelogging application and type of information to be captured. For example, focused lifelogging would use one or two sensors but extreme lifelogging would employ a number of sensors. The storage capacity has been significantly improved and increased in size. It is suggested that the storage capacities will keep increasing at the speed with the increasing tendency of lifelogging by the people [11]. The availability of high-speed wireless networks has enhanced portable devices to connect and communicate with remote high computing and storage media for extensive data processing and storage.
Wearable lifelogging exploits these technological advancements and encourages researchers for the design and development of custom-built holistic lifelogging devices and applications such as
SenseCam [3]. However, the wearable lifelogging systems are limited in their features e.g., limited number of sensors and storage capacity, omnipresent access to the lifelog information, etc. In addition, it overloads users with extra devices, which creates hurdles and troubles in performing daily life activities. Therefore, feasible solution would be using of digital devices that constitutes all the sensory, processing and storage capabilities and are in common practice of the people in their day-to-day activities. One such pervasive device is smartphone, which is present in the pocket of almost every individual around the world, today [12].
Smartphones are our constant companions and know us very well beyond our imagination. A smartphone can know all about our activities including calling, text messaging, web surfing, music listening, watching videos, Television (TV) programs, social networking, conversations and meetings, e-shopping, gaming, visiting locations, and many more. State-of-the-art
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
5 smartphones has sophisticated computing, networking, and sensors capabilities with large data storage capacity. These features collectively turn a smartphone into people-centric sensor to capture and process multidimensional sensors and lifelog data. Today's smartphone integrates resources and functionalities that can provide high quality lifelogging experiences as compared to wearable lifelogging devices (e.g., SenseCam) [13]. Resultantly, the ubiquitous nature and technological advancements in smartphone makes it not hard to believe as de-facto lifelogging device [14, 15], which can make the automatic generation of autobiography more realistic.
However, for smartphone to fulfill the design requirement of a lifelogging device (i.e., seamless incorporation in daily life, resources efficiency, security, long-term lifelog data preservation, information retrieval from a lifelog dataset, etc.), several challenges and issues needs attentions and solutions [15]. These include number and capabilities of sensors, meeting storage requirements, managing battery power, superiority over dedicated lifelogging devices, etc.
1.2 Lifelogging Types
There are number of activities (e.g., quantified-self analytics, lifeblogs, and human black box), each of which is producing some kind of lifelog archive and are commonly referred to lifelogging [9]. However, lifelogging can be broadly categorized into situation-specific capture and total capture [16]. Situation-specific capture (also called focused lifelogging) is narrower in scope and is a domain-focused activity, determining recording of life experiences information about an aspect of a person's life with a prior clear understanding of the goals and applications of the effort, as shown in Figure 1.1 (a). For example, quantified-self analytics is a type of situation- specific capture where people capture data for a specific purpose such as monitoring behavioral changes, sleeping patterns, fitness, etc. Focused lifelogging is more common and has gained
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
6 market place because of retrieving immediate results from the captured data. Moreover, several cheap personalized health and wellness products are available in the market that log information about a person's physical activities performance and caloric energy expenditure such as Fit-Bit
OneTM, Nike FuelBandTM, LarkTM, etc. Total capture (also called extreme lifelogging) realizes the vision of capturing totality of life experiences and is the indiscriminate logging of information about all aspects of a person's life without having any prior insight understanding and end use-cases. It emphasizes on using all of the feasible information sources for continuous capturing of as many types of life data as possible to portray complete picture of a person's life experiences, as shown in Figure 1.1 (b). The data collected could be documents, images, videos, sounds, locations, ambient temperatures, etc., which are rich enough for clearly describing semantics of lifelog information. There are researchers who advocates heavily on the potential applications of total capture lifelogging such as MyLifeBits [17]. However, total capture is a data-intensive activity and is not as common as situation-specific capture due to infrastructural limitations (e.g., unavailability of required sensors ubiquitously), technological limitations (e.g., battery power), privacy concerns, and information management issues.
1.3 Terminologies
This section presents a detailed description and differentiation among some of the terminologies that have been commonly used both in the relevant literature and in this thesis.
1.3.1 Lifelogging and Personal Information Management (PIM)
Acquiring and keeping of valuable information is the prime characteristic of human behavior.
Besides human memory, over the time, people have used different approaches and methods for organizing, storing, and retrieval of their personal information. However, the emergence of
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
7 computer technology has instigated researchers to investigate relatively more secure and advanced techniques, which are commonly referred as Personal Information Management (PIM).
Figure 1.1: (a) Selective capture lifelogging; (b) Total capture lifelogging1.
PIM is different from general information management [18] and can be defined as an organized set of mechanisms and procedures combined in an information system for assisting individuals in managing and retrieving information in their digital personal information spaces to facilitate their daily tasks and fulfilling their leisure needs [19, 20]. Lifelogging systems are inherently PIM systems having the same objectives, but the application of ubiquitous and pervasive computing technologies makes lifelogging different from the PIM systems [16]. A PIM system is typically desktop activity based lifelogging and assist users in reminiscence and retrieval of the logged past-desktop activities. Recording users' activities on desktop can be beneficial for optimizing
1https://www.pinterest.com/amydspinterest/quantified-self-self-tracking-digital-health/
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
8 information access, learning users' behaviors, assisting users in browsing and searching their personal information for enhanced reminiscence [21]. However, PIM activity involves effortful recording and retrieving of selective digital objects for a specific purpose [16]. Literally, the vision of lifelogging extends beyond storing information about desktop activities only.
Lifelogging is effortless and includes automatic and comprehensive recording of information about people's personal interactions (i.e., with documents or colleagues), activities away from the computer (i.e., outside office in the real-world), related contexts, etc., in a format to support a number of use-cases including easy and effective recalling of past-experiences [16].
1.3.2 Lifelogging and Context-Awareness
Devices capable of understanding users' contexts fit into the vast framework of context- awareness. Context-awareness is the capability of a system for automatically adopting itself according to users' contexts by providing appropriate information and services without requiring any explicit interaction [22]. Technically, lifelogging systems can be viewed as subset of context-aware systems [14, 15]. However, it is different from context-aware systems in different aspects [15]. For example, lifelogging emphasizes on storing information for long-term access and context-aware systems retain no such information for long-term access. Context-aware systems require no background running/processing for all the time, whereas, lifelogging systems need to run passively and continuously to ensure complete capture. Context-aware systems may not suffer from privacy and resources overload problems, whereas, lifelogging system seriously suffer with them. However, both systems have a common point of understanding that is the importance of recognizing and using of contextual information. In addition, they emphasize on using of sensors technologies to capture and exploit contextual information.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
9
1.3.3 Lifelogging and Lifelog
Lifelogging is the procedure of using sensors and computing technologies for ambient, passive, and continuous capturing, storing, and retrieving of life experiences information of a person called lifelogger. The lifelogging process is materialized by using wearable devices with embedded sensors or sensors deployed in environment to sense a person's daily life activities.
Lifelog is a dataset representing the data generated as a result of lifelogging process. A lifelog can range from simple to complex. Simple lifelog may be composed of a specific type of lifelog information such as a collection of photo. Complex lifelog could be composed of a diverse set of lifelog information that could be either isolated or interlinked with each other semantically such as a collection of photos, sensors data, contexts data, etc. Typically, the size of a lifelog depends on type of lifelogging. For example, selective capture lifelogging can produce a light lifelog and total capture lifelogging can produce a huge lifelog. However, data in a lifelog could be worthless if not properly refined and semantically organized to represent a person's life events.
1.4 Lifelogging and Human Memory
Memory is the brain process related to the absorption and retention of information [23]. Broadly, human memory determines the ability to acquire, process, and archive an individual's life experiences and actions for later use. However, human memory becomes weak due to age factor and memory related diseases. The cues (e.g., sound, smell, taste, touch, etc.) are suggested as powerful tools for triggering human memory for reliving past-experiences and giving the feelings when the memory was born. Research related to human memory has primarily focused on its structuring and processes [7]. A comprehensive overview on human memory structure and processes is not possible in this thesis; however, it can be found in [7]. A sound understanding of
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
10 memory structure and common daily life memory problems can provide platform for developing effective memory aid tools [2]. In this section, we first describe the episodic and semantic memories, and second the application of lifelogging technology for augmenting human memory.
1.4.1 Episodic and Semantic Memory
Cohen and Conway have suggested episodic memory as the memory of specific events and experiences, which have certain temporal context [9, 24]. Episodic memory increases people's power to memorize and recall past personal experiences, emotions, locations, and other contextual facts in time and place [9]. Episodic memory enables people to mentally time travel
(i.e., transporting into the past-experiences and into the future) and recall specific episodes from past events such as what happened, where, and when [25]. This type of memory is autobiographical and personal, and contains the information stored in the long-term memory. It has personal feelings because events having strong emotional impacts enable to remember and recall details of the events in accurate terms. Semantic memory refers to memory of knowledge and facts about the real world, which is acquired over time but is independent of time, space, and other contexts [7, 26]. Information stored in semantic memory is of general nature about the world instead of specific about individuals' experiences. For example, remembering name of the capital of Pakistan. It may include knowledge about vocabulary and verbal ability, and knowledge picked up from environment (e.g., stories, myths, etc.). Semantic memory is sharable due to being independent of people's personal experiences and emotions. However, semantic memory has source in episodic memory due to learning new facts and knowledge from our own personal memory [26]. Both kind of memories interact with each other but episodic memory decreases overtime, whereas, semantic memory increases [27].
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
11
1.4.2 Lifelogging for Memory Augmentation
Human memory has infinite capacity of remembrance. However, forgetting is also a human memory characteristic. It is believed that memory prostheses can be very useful for augmenting human episodic memory in the near future [28]. Lifelogging can assist human episodic memory by providing permanent digital memory prosthesis to increase their remembrance power of past events, people's names, conversations, location of items, places visited, etc. [6, 16].
Computer technology is considered useful for supporting human biological memory functions
(i.e., capturing and retrieval) and a wide variety of computer-assisted technologies are developed and used over the time. The most common of these technologies are address books, schedules, and emails [7]. Electronic mail helped people in remembering their past events by sending emails to their email accounts with belief that regular checking of email box and seeing messages can enhance human memory performance [29]. Several websites are available for enabling individuals to keep record of their memories on web log entries. For example, blogger website enables users to create diaries using their email messages and allows them to post their time-stamped pictures, audios, and videos to their blogs. LifeSnapz is another free website that enables users to record, organize and share their life events and other mementos of significant importance with their trusted groups such as friends, colleagues, communities, etc. Using
LifeSnapz, user can associate pictures, videos, and text to describe events and can explore events using timelines, maps, and lists. In addition, several online social networking portals (e.g.,
Flicker, Facebook, Twitter, YouTube, etc.) enables people to create their profiles, and share their personal information (e.g., pictures, videos, and text messages) with textual description to facilitate the retrieval process. The EuroPARC project has investigated designing of memory
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
12 prostheses for memory augmentation and advocated that tracking users' activities data can generate powerful cues to enhance ability of remembering information about past events [30].
Many researchers in the memory prostheses projects believed that "the physical contexts can be a powerful cues for recall" because of the people's tendencies of dividing their past events into episodes [31]. Memory prostheses using computer technology can provide greater sense by connecting data with contexts. Contexts can provide effective cues for enabling individuals to trigger their episodic memories and retrieve past experiences information about an aspect of an event, which might also retrieve several other aspects of the same event [7]. For example, retrieving location of a party can also trigger memories about its participants.
Lifelogging uses both computer technology and contexts to result into digital memory prosthesis
(e-memory) of a user. The digital memory prosthesis is a dataset that contains diverse set of a user's life experiences information including contextual information to augment his/her biological memory [21]. Thus, the vision of lifelogging is supporting episodic memory by producing its surrogates. Therefore, lifelogging requires organizing and structuring lifelog data in the same way as brain stores information. It is possible, if lifelogging captures and segments a typical day into a series of events of variable durations (e.g., watching TV, cooking, travelling, playing, listening music, working on laptop, etc.) and relate them using contextual information.
The availability of digital objects (e.g., sensors, storage, etc.) can rationalize the vision of lifelogging. Gordon Bell has estimated that storage of 1 terabytes (TBs) is enough to store emails, research articles, books, pictures, conversations, audio tracks, video clips, etc., of a person collected in 60 years [32]. In addition, 30TBs of storage is enough to store high-quality video recordings of 70 years [33]. Researchers [34, 35] have reported that tendencies to record
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
13 our lives digitally, user-friendly information retrieval methods, availability of cheap data capturing and storage objects, and globally access to information sources enable the development of ultimate and immortal memory prostheses in the digital age.
Apart from memory reminiscence, lifelogging has been articulated as science and arts, entertainment, self-reflection, and journal [6, 36]. Lifelogging can help users in self-monitoring by reviewing their past behaviors for not repeating wrong behaviors and share experiences to stop any further mistakes. Beside behavior monitoring, lifelogging helps in health monitoring for understanding oneself. For example, collecting a person's heart beat rates while in interaction with others, one can infer knowledge about whom and where makes the person nervous.
1.5 Impacts of Lifelogging
The emergence of new technologies confirms the rationale of lifelogging - memory of life - to provide better understanding of an individual's life experiences. However, to get into more mainstream and being beneficial for society, lifelogging must exhibit significant usages in different domains and answer questions: what would be the value of lifelogging?, why would people like to capture totality of their lives experiences in multi-model formats?, etc. It has been observed that lifelogging can potentially improve people's lives in different aspects. People can preserve family history by inheriting lifelogs from their ancestors. In addition, inheriting lifelogs can also be helpful for the people in easing pain of loss of their loved ones such as parents, husband/wife, and children [37]. Lifelogging would enable people to track their lives and bring positive changes in their behaviors. In addition to tracking physical activities and counting number of steps, cheap lifelogging consumer level products are available that allows people to log and monitor patterns and quality of their sleeps. Sleep sensing devices are either self-
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
14 contained devices that can be worn on the wrist or can be contact-less (e.g., Renew SleepClock from Gear4) or smartphone apps. Furthermore, several apps can record users' activities for different applications of personal analytics such as smoking cessation, monitoring stress level, monitoring blood pressure and heart rate, diet monitoring for calories expenditure and weight loss, and diabetes monitoring by tracking sugar intake [9, 38]. Location logging enables individuals to log their movements for a variety of purposes including social purposes, fitness purposes, etc. The dedicated capturing devices (e.g., OMG Autographer and Narrative Clip 2) can be helpful for the people with memory problems such as Alzheimer's, dementia, etc., for re- living their past-experiences on triggers such as picture, aroma, voice, physical entity, etc. [9]. A society having high tendency of lifelogging can have least security risks and can reduce crime rates by providing enough information and opportunities to catch and deter criminals in the society [37]. Lifelogging can be helpful to diminish the physical clutter of an individual information, as the availability of large storage hard drives could retain immense amount of various types of information such as audio and video recordings, photographs, documents, movies, web pages browsed [7]. Lifelogging could help to keep personal information from being harmed by atmospheric conditions and natural calamities. Lifelogging can provide a composite tool by relating, organizing, and annotating life information to make information searching and retrieval easy and effective. Therefore, lifelogging could be helpful in recognizing abilities of individuals and can conceivably prompt to the raise of journalists, entertainers, and communicators. These together could imply that lifelogging could increase in productivity, creativity, physical wellness, and general prosperity. It is believed that lifelogging can enhance quality of life by providing users an opportunity to reflect on themselves [7]. For example, it can improve people's understanding, confidence, and responsibility in personal relationships.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
15
Furthermore, excessive capturing and advertising of lifelog information can be used in market research and paves the way for novel qualitative analysis to derive new knowledge by analyzing the data captured in the subjects' lifelogs [39].
1.6 Research Motivation
Human life is dynamic where new events and activities are emerging, making it difficult for desktop-based lifelogging to create complete digital memory prosthesis by storing and retrieving everything in real-time. The increasing availability of digital devices to capture and record information about people's everyday events and interactions has leveraged the vision of lifelogging beyond desktop computers. The developments of dedicated lifelogging devices have enabled people to capture, record, and share every events and activities of their lives digitally with or without requiring their computers [16, 32]. However, overloading a user's body with extra devices can produce hurdles in performing everyday activities and increases overall cost.
The advancements in inexpensive sensors technology, processing and data storage technologies, mobile computing, etc., have made lifelogging more feasible. Therefore, lifelogging functionality should be integrated in fully loaded devices that are in common use of the people.
To handle the issues, smartphone can be used as a version of Bush's Memex. Smartphone is our lightweight constant companion and has the capabilities to capture, store and retrieve information about our everyday events and activities. Both hardware and software developments have made smartphone as an ideal platform for future computing, providing the same users' experiences with certain additions and improvements as powerful computers of a few years back
[40]. However, using smartphone for lifelogging would suffer with information overload problem by capturing and storing huge collection of lifelog information generated from various
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
16 information sources such as sensors, Short Messaging Service (SMS), email, camera, phone calls, etc., with no proper understanding and relationships with each other. Therefore, it would make increasingly difficult to organize and retrieve the required lifelog information instantly and completely. To fulfill the vision of lifelogging, lifelog information on smartphone needs to be organized in similar to human episodic memory to increase its semantics and enable fine-grained retrieval and associative trail of information. Therefore, this whole practice demands for investigation of novel method for proper using of sensors and lifelog information on smartphone.
Sensors produce a vast amount of data in a unit of time (i.e., depending on type of sensors), which is in the raw form (e.g., latitude and longitude value of a location) and needs extraction of contextual semantic out of it. The contextual semantics will provide useful memory cues to semantically annotate and relate lifelog information of the same event to facilitate retrieval/visualization on smartphone. The successful aggregation, extracting of meaningful events and sub-events, and retaining interoperability among heterogeneous information sources can be realized using Semantic Web [41]. We believe that leveraging Semantic Web technologies to lifelogging paradigm can be helpful in this regard. Semantic Web technologies provide features/constructs to semantically annotate and relate lifelog information using contextual semantics and develop a semantic lifelog model to represent lifelog information with the same semantics as they exist in the real world and encoded in human episodic memory. This semantic formalism will not only increase interpretation of lifelog information but would also facilitate retrieval of required lifelog information and integration of new kind of lifelog information and relationships, which might emerge with the passage of time. In short, using smartphone to capture lifelog and contextual information and Semantic Web technologies to
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
17 provide semantic glue to the lifelog information, potentially can be the starting point of realizing true lifelogging systems.
1.7 Statement of Thesis
Most of the lifelogging researches have concentrated either on the development of novel lifelogging devices or on the identifications of application areas of lifelogging [9]. They, however, represent considerable challenges. From the information sciences perspective, the huge archive of personal and sensors data produced by lifelogging will raise the challenge of organizing and relating lifelog information semantically to make it more useful for the applications. This research thesis addresses the vision of lifelogging by leveraging smartphone's sensors and processing capabilities and Semantic Web technologies for semantically modeling lifelog information. The excessive usage of all possible smartphone sensors data will not only make the vision of capturing the totality of life experiences closer to reality but will also provide enough information to extract contextual and environmental semantics. As discussed earlier,
Semantic Web technologies will provide the semantic glue by developing a semantic model of lifelog information and semantically annotate and link them using the contextual semantics extracted from the sensors data, in the same way as they appear in the real world. The technique would assist people to gather, store, and semantically enrich and organize lifelog information. In addition, the semantic annotations would enable users to query the semantic lifelog model to retrieve information about past events that can be used as memory cues to simulate and trigger their biological memories to access detailed information about specific past events. In addition, a holistic smartphone-based semantic lifelogging framework is needed to be defined that integrate smartphone and Semantic Web technologies in combination with other software components into
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
18 a single suit. The semantic framework would streamline the research efforts and enable developers to develop smartphone-based lifelogging applications for several use-cases.
1.8 Research Objectives
This research thesis focuses on the following main objectives:
• To investigate the capabilities of a smartphone as lifelogging platform for capturing,
semantically storing, processing and retrieving lifelong personal information in a manner
close to human episodic memory.
• To exploit sensor data at different contextual levels for semantic annotation and
association of lifelog objects resulting in an enriched graph.
• To design, develop, and evaluate a comprehensive smartphone sensors ontology in Web
Ontology Language.
• To propose a smartphone-based semantic lifelogging architecture and demonstrate its
practicality by developing a proof-of-concept application.
1.9 Research Significance
As discussed earlier, this research is a step towards the total capturing of human experiences by exploiting sensors data semantic to capture lifelogging and contextual information, and semantically modeling lifelog information with meaningful annotations and associations for a variety of use-cases. Apart from its inherited advantages (i.e., entertainment, self-reflection, exchanging legacies, etc.), this research thesis enhances lifelogging experiences in several aspects. First, all of the operations of data collection, analysis, storage, and retrieval are made available on a single smartphone, which will relieve users from the purchase of and being
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
19 overburden with extra devices and services. Second, using Semantic Web technologies structures digital memory prosthesis in similar to human episodic memory that would increase understandability of lifelog information to applications and audiences, and empowers stakeholders to develop applications/services for solving real-world problems and gain competitive advantages. Third, extracting and using excessive contextual information will provide enough memory cue to reduce cognitive load in recalling past-experiences. Fourth, users can control the lifelogging process by defining events and policies for types of lifelog and contextual information to be captured, annotated, and shared, while ensuring security and privacy of lifelogging. Finally, using smartphone will increase the social acceptance and lifelogging trends by instigating users to track their lives at any level of requirements.
1.10 Thesis Organization
Overall hypothesis and objectives of this research work are discussed in the six chapters. Chapter
2 has presented a brief background description and research contributions in the field of smartphone-based lifelogging. It is aimed to provide a comprehensive understanding of the realm smartphone-based lifelogging technology, importance of sensors technology in the lifelogging domain, elaboration of topics to be addressed in this thesis, and foundation for the next chapter.
Chapter 3 has presented a detailed analysis of smartphone technology with the prospect of evaluating it as a lifelogging platform. It provides an affirm understandability and justification of superiority of smartphone technology over the others and its potential weightage as a de-facto lifelogging device. Chapter 4 has presented in detail the design and development of semantic model (i.e., ontology) for smartphone sensors and lifelog objects. It provides a semantic model to exploit sensors data semantics for adding semantics to the lifelog information and developing a
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
20 digital prosthetic memory in similar to human organic memory. Chapter 5 has presented a semantic framework for smartphone-based semantic lifelogging systems and demonstrated its practicality by developing and evaluating a proof-of-concept prototype application. It is aimed to increase lifelogging trend instead of being divided into separate islands. Chapter 6 concludes this thesis by highlighting research contributions and outlining a number of research directions/avenues (i.e., future work) for helping researchers in finding research topics.
1.11 Summary
This chapter provides a comprehensive overview of the different aspects of the concept lifelogging, lifelogging types, and related concepts used in the domain. It is elaborated that what is the importance of lifelogging in aiding human organic memory?; how lifelogging technology can enhance/improve users' lives by rightfully archiving and recalling daily life experiences in their everyday activities?; how smartphone technology can provide a promising platform for future lifelogging?; what is the objectives and significance of the research described in this thesis? The scope and importance of the approach used in this thesis is described and it is openly stated that how its contributions can be helpful in increasing the lifelogging paradigm. In Chapter
2, we give a literature review of the lifelogging research. We present briefly the history of lifelogging and the different approaches used by the researchers over the time (ranging from desktop metaphors to smartphone) to promote lifelogging culture along with their advantages and shortcomings. We also present a comprehensive overview and classifications of the research efforts in the field of smartphone-based lifelogging, potential use-cases of the smartphone-based lifelogging for gaining competitive advantages by the different stack holders, and limitations in the current approaches to provide background and topics for this thesis.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
21
Chapter 2 : Towards Smartphone-Based
Lifelogging
Preserving and remembering knowledge is a fundamental human property. However, human memory is sinful and results in remembering of certain things while in complete forgetting of others [4]. Forgetting is due to if an object in the long-term memory is not retrieved for a long period of time, it fades away [21]. Schacter has identified and classified the most-common memory problems into a framework, named the "Seven Deadly Sins of Memory", as shown in
Table 2.1, in which the former three are related with forgetting while the later four are related with distortion [42].
Table 2.1: Types of memory errors.
Type Memory Error Description
Transience Memory loss or fading over time.
Memory Absent-mindedness Losing of synchronization between attention and memory. Forgetting
Blocking Unsuccessful attempt to retrieve information from memory.
Misattribution Assigning a right memory to a wrong source.
Implanting memory by recalling events, which might not Suggestibility Memory happened in reality. Distortion Bias Effects of current knowledge or belief on remembering the past.
Pathological inability to forget past disturbing events which we Persistence do not want to remember.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
22
People have been using several lifelogging methods and technologies (e.g., strings on fingers, diaries, sticky notes, mnemonics, etc.) to store their life experiences for mitigating the problem of forgetting and other purposes such as remembering events, sharing with closed ones (i.e., friends, family, colleagues, etc.), or using as evidence (i.e., to prove integrity of an event).
However, the emergence of new types of events and interactions have complicated the people's lives and making it difficult to manually record and remember every important aspect of their lives. For example, a diary could suffer from many gaps and missing pieces due to rapid occurrence of changes or turmoil resulting in forgetting to add important events [43]. The advancements in capturing and computing technologies with their exponential decreases in costs have encouraged researchers to design and develop several automated systems to realize the vision and applications of lifelogging. The lifelogging technology aims at utilizing computing technology for personal use exclusively and transforms computer into digital memory prosthesis to assist human memory in remembrance. This chapter underpins the wider contributions into the field of lifelogging technology generally and smartphone-based lifelogging technology specifically, and tries to establish our understanding of their content compositions, contributions, and inspirations for this thesis.
2.1 History and Background of Lifelogging
Vannever Bush pioneered the idea of lifelogging in 1945 by presenting the idea of an imaginary mechanized device called Memex [8]. Memex is reported as an "enlarged intimate supplement to one's memory" whose technical interpretation could be a mechanized device for organizing information of lifetime similarly to human brain, which hints of the first lifelogging system [9].
Bush also provided inspiration for the current wearable camera-based lifelogging solutions and
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
23 emphasized on photography for capturing life records by introducing the concept of "camera hound", where a walnut-sized camera is mounted on an individual's forehead to capture pictures of anything of interest, which could be subsequently stored and indexed by the Memex [9].
Memex unfolded two important features, which are now considered essential for any modern lifelogging system: annotations and links.
The availability of technology made it possible for researchers to start and work on several projects regarding bringing Bush's innovative idea into reality. Using technology, lifelogging research can be categorized into desktop metaphor and PIM, and wearable computing. These two categories and their limitations are briefly discussed in the following subsections. For a more complete discussion on history of lifelogging and conventional lifelogging, the readers are advised to consult [7] and [21].
2.1.1 Desktop Metaphors and PIM
Desktop computer has been the prime focus of researchers to produce lifelog consisting detailed record of users' activities on desktop. Researchers have hypothesized the recording of desktop activities can bring several benefits including information access optimization, learning and projecting users' behaviors, time management, helping and facilitating users' processes of searching and browsing their personal information, assisting users in their reminiscence, etc.
Researchers, organizations, and academicians have suggested different approaches and methodologies for effective lifelogging of users' activities on desktops. Semantic Desktops (e.g.,
NEPOMUK [44], Gnowsis [45], SemanticLIFE [19], etc.) have leveraged Semantic Web technologies for organizing, managing, and searching data on users' desktop computers and computers in a network. Desktop search engines (e.g., Beagle++ [46], Stuff I've Seen (SIS) [47],
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
24 etc.) enable users to quickly search and retrieve their required personal information from their desktop computers and computers in an enterprise network, using a single query in the same way as web search engines are used for locating information on the Web [48]. A performance evaluation of desktop search engines can be found in our previous research work [49]. Desktop- based lifelogging applications (e.g., MyLifeBits [17, 32], LifeStream [50], DARPA's LifeLog
[51], Advance Soldier System and Information Technology (ASSIST) [52, 53], etc.) enable the acquisition, management, and organization of users' desktop information and activities, and provide effective search mechanisms to retrieve users' lifetime data swiftly.
2.1.2 Lifelogging via Wearable Computing
Wearable computing also known as body-worn computing or bearable computing refers to set of practices of designing and developing miniature electronic devices that could be worn constantly above or under the cloths to seamlessly extend both mind and body, and perform both general purpose and specific purpose new and previously unexpected functions [54]. The idea of wearable computing concentrates on the Continuous Archival and Retrieval of Personal
Experiences (CARPE) approach of lifelogging. In early 1990's, several manifestations were released based on the idea of recording all of a person's lifetime experiences in visual and audible formats by using cameras and electronic whiteboards [55, 56]. Steve Mann - father of wearable computing - coined the idea of wearable computing and focused on developing increasingly smaller wearable devices since 1980's with innovative and increased sensing, capturing, and displaying features for manipulating lifelog information [57, 58]. He urged using wearable life- capturing technologies for "sousveillance", which refers to using digitally captured life experiences for self-surveillance. About sensing, Mann had focused on visually capturing of the
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
25 world and contributed his long-time efforts in exploring the use of camera in wearable lifelogging. During experiments, he modified the physical appearance of his device by replacing the large helmet that he developed in the 1980's with the one smaller (similar to sunglasses) that he developed in 1990's from early personal imaging [59] to the "EyeTap" project [60]. All of the
Mann's efforts could be viewed as precursor of today's Google Glasses. Mann postulated that recognizing activities would provide information for automatic annotation of lifelog objects and would provide effective memory cues. In the "cyborglogger" lifelogging project, Mann and his colleagues developed a mobile phone application allowing users to capture lifelog objects (i.e., videos and pictures) using mobile phone camera, and store and share them using their social networking sites or personal homepages in real-time [61]. Besides Mann's efforts, several other researchers also contributed in the field by designing and developing several useful wearable devices such as [3, 62-69]. In addition, the detailed related discussion can be found in [7]. They all have argued that employing computing technologies to record videos and audios of daily life experiences can be effective memory aids in retrieving past-experiences. They have also emphasized on the importance of using sensors technology to capture both contents and contextual information (i.e., both objective and subjective) in as much detail as possible. The contextual data can be used as metadata to annotate and index lifelog information, which will facilitate retrieval and browsing of specific lifelog segment as per need.
2.1.3 Limitations of Conventional Lifelogging
Lifelogging, despite of having prolong history, is still in its early stages of development. Detailed investigations can provide beneficial information for the development of this field. A comprehensive review of challenges and opportunities in the lifelogging technology can be
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
26 found in [70]. In the following, a compact overview of the limitations/shortcomings in the conventional lifelogging solutions is presented to provide topics for this thesis.
The major concern is the inconvenient use of wearable devices. Most of the wearable devices are custom designed and built, which overburdens users, makes mobility difficult, and creates hurdles in performing daily life activities. In addition, they are expensive due to buying extra devices and may face societal critics as well. These lifelogging solutions are designed to work in a restricted environment (e.g., desktop computer, office, etc.); however, the occurrences of most of the daily life events are independent of space and time. Therefore, they cannot fulfill vision of capturing the totality of life experiences. These lifelogging solutions are constrained in capturing lifelog information (e.g., using of limited number and inappropriate sensors, hard mobility, no omnipresence, etc.) and might not be able to capture relevant information from a reality due to complexities of the physical world [10]. Therefore, these lifelogging solutions might capture only a portion of information instead of providing a complete picture of reality [71]. These lifelogging solutions have mainly focused on data captured and have refrained the associated problems of storage, management, organization, retrieval, etc. This requires additional manual efforts to maintain and operate lifelogging devices, and transfer/exchange the captured lifelog information to some backend storage (i.e., PC or cloud). The advancements in storage technology have lessened the storage problem and TBs memory is now commonly available.
However, the other problems are of significant nature, which are evident from the Gordon Bell's interview with Thompson in 2006 [7]. In the interview, he articulated that the increased size of
MyLifeBits has created information management problem. It is difficult to search a specific item of information and even Gordon Bell often finds himself lost in the forest. Moreover, searching
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
27 and retrieving a specific lifelog item (e.g., video clip or phone call) using conventional
Information Retrieval (IR) techniques is a huge challenge and they are not appropriate for lifelogging technology [9]. Memories are dissimilar to the data captured such as a photo taken at a specific point of time is different from the memory constructed whenever it is prompted. An important indicator in this regard can be the structuring and organization of lifelog information in similar to human episodic memory [9]. These lifelogging solutions build lifelog consisting of
"facts" about the aspects of people's lives with no emotional interpretation. These lifelogging solutions are also endangered with privacy (e.g., control over lifelog information in capturing and sharing [70], lifelog information as property of individual [37], etc.) and ethical (e.g., societal acceptance and resistance [66]) issues. Collectively, these issues and limitations can endanger the development of lifelogging technology. However, addressing these issues and limitations can realize the potentials of lifelogging, which will gradually reduce the resistances and criticisms, and the society will show more willingness towards adaptation of the technology.
As Harper et al. [72] had suggested that in the near future, electronic memory will have the same importance in people's lives as mobile phones and Internet have today.
2.2 Overview of Smartphone-Based Lifelogging
Smartphone is a technologically advanced bread of mobile phone that combines the features of mobile phone (i.e., voice calls, and SMSs) and Personal Digital Assistance (PDAs) (i.e., office management, web browsing, email, etc.) [40]. As discussed in Chapter 1, smartphones - highly common ubiquitous computing devices - could be a vision of Memex by offering new opportunities to unobtrusively record nearly all aspects of a person's life to construct and preserve a long-term digital prosthetic memory [1]. Smartphones, merely used as communication
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
28 devices, have now become our portable and affordable computational companions due to becoming cheaper, lighter, more accurate and powerful, less power consuming, and with high- resolution display. Smartphone can be a step towards "totality of life experiences" by ubiquitous and passive capturing and storing of excessive useful information about contents and contexts of users' daily life activities and events (i.e., audio/video calls, text messaging, web surfing, music listening, videos watching, social networking, e-shopping, gaming, locations visiting, etc.) in a verbatim and unbiased way. Realizing the potentials of smartphone, several researchers have presented smartphone-based lifelogging systems during the recent past years. Adopting smartphone for lifelogging is because of its advantageous characteristics over dedicated wearable devices. Thus, provides an alternative platform for monitoring lifestyles and activities [13].
Smartphone-based lifelogging systems are like context-aware systems that samples users' contexts for data collection and aggregation. A context-aware smartphone can potentially use the collected data to make inferences about schedules, suggest activities, and provide reminders without any manual interactions. For example, a potential application can be changing of a smartphone modes as per situation such as automatically turning off ringer when entering a movie theatre and turning it on after the film. Nokia Lifeblog project [73] was the first among the smartphone-based lifelogging systems. Nokia Lifeblog provided inspiration for many of the subsequent research efforts that resulted into the emergence of a new breed of smartphone-based lifelogging systems with increasing sensing and logging functionalities such as Pensevie [4],
MyExperience [74], Experience Explorer [12], etc. Despite of being advantageous, smartphone- based lifelogging faces specific obstacles and limitations, which have so far largely impeded its large-scale deployment. In this section, we discuss the contributions into the field of smartphone-
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
29 based lifelogging technology with the aims to establish our understanding of their contents compositions, contributions, limitations, and inspirations for this thesis.
2.2.1 Generalized Architecture for Smartphone-Based Lifelogging Systems
Technically, smartphone can potentially increase people's ability to obtain large amount of information about their selves and make observations about their environments. However, the ability to automatically capture, analyze and archiving large amount of data, and identify and retrieve events of interest, is currently limited and in its nascent stages. Using smartphone for end-to-end lifelogging processes is a complex phenomenon and involves many challenges [21].
The lack of standard guidelines for smartphone-based lifelogging freed researchers to propose solutions using their own experiences and methodologies that is creating separate islands and is the wastage of potentials, resources, and technology. Therefore, we have proposed a generalized architecture for smartphone-based lifelogging systems to improve systems' interoperability, data capturing and analysis, components reuse, communications and sharing, and minimize their heterogeneity. In addition, it will provide standard guidelines and will help us to examine the available smartphone-based lifelogging solutions. The architecture is more materialized in designing and developing of the proposed semantic framework in the Chapter 5. The architecture divides the lifelogging process into four modules, as shown in Figure 2.1: data collection engine
(DCE), software middleware (SM), semantic extraction and organization (SE&O), and retrieval and sharing (R&S). a. Data Collection Engine
Data collection engine (DCE) collects lifelog information from a user's Personal Information (PI) space on smartphone. A user's PI space is comprised of information resources representing
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
30 lifelog objects (e.g., pictures, documents, contacts, emails, calls, etc.) that are created by smartphone applications, and contextual and environmental data captured by sensors. To control the lifelogging process, users can define information sources from their PI spaces to be used in lifelogging. However, information resources from a user's PI space must be made accessible to a smartphone-based lifelogging system for instrumentation, automation, and querying. Inherently, smartphone-based lifelogging systems rely on sensors to realize the Memex vision of surrogate memory. However, as discussed in Chapter 1, using of type and number of sensors depend on type of lifelogging.
Figure 2.1: General architecture of smartphone-based lifelogging systems.
To create a rich record of life experiences, DCE captures data from sensors (i.e., smartphone sensors, wearable sensors, and environmental sensors) and applications (i.e., keeping information
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
31 about us and our environments), and forwards the captured information for further analysis and storage. The capturing process would be either reactive on event-based or proactive on polling- based, depending on requirements of an application. In reactive capturing, information would be captured with occurrence of an event (e.g., arriving of a SMS). In proactive capturing, information would be captured form managed information resources either continuously or periodically after regular time intervals. Comparatively, event-based capturing will be efficient because of less resources consumptions. b. Software Middleware
The lifelog data, especially the heterogeneous sensors data, provided by DCE would be in the raw format with no semantic descriptions and cannot be used directly. Therefore, needs further processing/refinements for conversion into useful lifelog information. Software middleware
(SM) would provide components to pre-process raw sensors data numerously such as aligning data both temporally and spatially, cleansing data from noise, fusion of data into uniform objects, computing and utilizing the trust and reliability of the incoming data, transforming data from one format into another, etc., without affecting the overall quality of the captured data. The functionality of SM depends on application. However, the raw sensors data (e.g., voice data from microphone) received from DCE would be segmented using time to extract features which could be used for classification, identifying daily activities, and other contextual information. The captures would be divided into fine-grained contexts and sub-contexts groups using context information. For example, first division into contexts groups using time information and next division into sub-contexts groups using location information. In addition, related information can also be retrieved from relevant applications such as a user's calendar. Similarly, metadata
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
32 associated with lifelog objects by the parent applications can also provide additional contextual information, which should be extracted for enhanced annotations. The lifelog, contextual, and metadata information would be fused and organized in a holistic unit for further processing and analysis. This area of lifelogging has not gained much of the research attention to date.
However, research experiences from the other fields (e.g., smartphone-based activities recognition) can be potentially used for learning about data quality, trust, and reputation. c. Semantic Extraction and Organization
Semantic extraction involves data analytics by extracting semantic knowledge from contents and contexts information received from DCE. SE&A would provide semantic glue to organize and relate lifelog information with the same semantics, as they exist in the real world and encoded in the human episodic memory. To cover the space between raw sensors data and human understandability needs employment of effective semantic extractions techniques and semantic modeling and reasoning techniques for identification and generation of semantic constructs out of lifelog information. For example, using of speech recognition for extraction of a person's name, email address, cell number, etc., from an audio conversation or using of Optical Character
Recognition (OCR) engine for extraction of objects and background information from an image.
Real-world objects are not only connected physically but also semantically and semantic interconnections largely depend on human interactions with the real world [75]. Semantic analysis involves several of the structuring, organizing, and summarization processes for mapping lifelog information into more discrete and meaningful data unit called event.
Organizing lifelog information into discrete events could be used by the mining patterns process for determining their uniqueness and regularity within a lifelogger's lifestyle. Event is not an
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
33 optimal lifelogging data unit but it has received most of the researcher's attentions to-date such as
[76]. Enriching lifelog data semantically at events or sub-events levels involves annotation using contextual and metadata information to describe lifelog information in a more meaningful way.
Annotations can also be used to define semantic relationships/associations between events and sub-events and related lifelog objects in alike to the real world. This annotating process will not only increase data representations and understandability, but will also enhance retrieval of specific lifelog information. Semantic organization and annotation also involves fine-grained enriching of lifelog information with data from Semantic Web knowledge stores (e.g., Linked
Open Data (LOD)). The lifelog information and annotations can be indexed (i.e., using Apache
Lucene2) for improving retrieval performances (i.e., improving recall and precision rates). d. Retrieval and Sharing
Once a lifelog is created, an appropriate retrieval model should be defined for the swift retrieval of specific lifelog information instantly and completely. To augment human memory, a smartphone-based lifelog should provide primitives of querying implicitly and explicitly for searching, summarization, and recommendation using time, location, calendar event, proximity, etc. Collecting detailed traces of users would enable development of a retrieval model to either provide information to users to fulfill their information needs or using a real-time context-aware recommendation engine. The potential use-cases of smartphone-based lifelogging are not yet clearly defined. However, smartphone-based lifelogging should potentially support several use- cases (i.e., both identified and unidentified) and a retrieval model typically depends on a use- case. We believe that potential use-cases can be inspired from the 5Rs of memory access
2 https://lucene.apache.org/core/
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
34 proposed by [16]: recollecting, reminiscing, retrieving information, reflecting, and remembering intentions. In addition, access methodology and Human Computer Interaction (HCI) factor should be considered while defining a use-case. Smartphone-based lifelogging has personal and social aspects. To fulfill the social aspect, smartphone-based lifelog can be shared with friends, colleagues, family members, etc., for numerous purposes. Lifelog information can be shared using several metaphors including cloud due to storage and global access limitations, social networks for exchanging life experiences and legacies, web blogs for creation of personal digital diary, and desktop for long-time preservation of personal experiences. However, privacy and security should be considered while defining retrieval and sharing policies.
2.2.2 Taxonomies of Smartphone-Based Lifelogging Systems
This section uses information from the previous section and presents categorization taxonomies of smartphone-based lifelogging systems using various attributes including basic architecture, role of smartphone, scope of lifelogging, sensors placement and sensing mechanisms, etc. The taxonomies are aimed at providing a complete reflection of the properties of the existing as well as possible smartphone-based lifelogging systems in the future. The taxonomies are represented graphically, where existing solutions (paid attention by the researcher) are represented with rectangles while rounded rectangles represent possible solutions, which have not been reported previously by the researchers and could be potentially new areas of future research.
Using role and deployment of smartphone and sensor placement, the smartphone-based lifelogging can be categorized roughly into in-situ systems and wearable systems (as shown in
Figure 2.2). Using of wearable technologies has been the prime target of researchers since years by custom-build sensing devices that are portable and carried by the weavers. However, a few
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
35 researchers have demonstrated the use of smartphone as alternative to the available wearable technologies (e.g., SenseCam [3]) such as [13]. This is usually done by harnessing a weaver with a smartphone. A smartphone either alone can be used as a wearable device utilizing all of its potentials (i.e., sensing, storage, processing, retrieval, etc.) or can be used in conjunction with other computing devices that could be wearable sensing devices (e.g., Google Glasses) or body- mounted computers/laptops for resources intensive processing, retrieval, etc. Researchers have reported using smartphone alone as wearable by mounting it on upper part of body (i.e., worn in a helmet on the head [1] or in a lanyard round the neck [13]); whereas, other potential placement areas (i.e., placement on waist and lower part of body) needs to be investigated and explored for potential applications and results. Researchers have also promoted using smartphone in combination with other systems, where smartphone could have different roles of sensing, storage, analysis, sharing, and retrieval. However, these systems are not practical because of overburdening users' bodies and increasing in cost. In-situ lifelogging means lifelogging in an instrumented environment (called smart environment), where lifelogging is highly dependent on sensors that are installed in a local infrastructure. However, the in-situ smartphone-based lifelogging systems can either rely fully on smartphone built-in sensors or sensors already deployed in an environment or any combination of them, with the varying roles of a smartphone related to sensing, storage, analysis, sharing, and retrieval. This criterion would eliminate users from wearing any type of wearable devices or sensors that would advantageously enable users to move freely and perform their daily life activities and actions. However, operations of in-situ smartphone-based lifelogging systems would be strictly restricted and dependent on the instrumented environment. In-situ smartphone-based lifelogging can aid in tracking people's lives in detail but no research attention has been paid to it to-date. Comparatively, all of the
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
36 smartphone-based lifelogging solutions are developed using wearable technology by requiring users to keep their smartphones close to their bodies.
Figure 2.2: Taxonomy using smartphone deployment and role in the lifelogging process.
Architecturally, smartphone-based lifelogging systems can be broadly divided into distributed and integrated categories (as shown in Figure 2.3). Rectangles and rounded rectangles hold the same meanings as in Figure 2.2. In distributed approach, the functionalities are distributed across smartphone and a remote server or PC, where smartphone is mainly used as a front-end device for capturing lifelog and contexts data or for low-level data processing and storage in addition, and remote server or PC is used as back-end for resources intensive processing, analysis, indexing, storage, and retrieval of lifelog information. The data captured by a smartphone is transferred to remote server using Internet or cellular networks technologies or by physically connecting smartphone to PC. The distributed approach has been the preferred choice by the
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
37 researchers due to resources constrained nature of smartphone (i.e., low processing power, battery power, storage, etc.) but can suffer with several problems that can degrade significances of smartphone-based lifelogging. These problems includes: (i) transmission delay can take place which can effect real-time monitoring system functionality; (ii) data uploading issues in case of no access to networks (i.e., Internet and cellular); (iii) excessive data transmissions can result in battery power depletion and increase in transmission cost; and (iv) privacy and security issues.
The integrated approach attempts to exploit the potentials and functionalities of state-of-the-art smartphones in performing all of the lifelogging operations without requiring any external supplements. It solves the problems of distributed approach by combining all features in a single holistic device. None of the available smartphone-based lifelogging solutions supports integrated approach in its complete essence. However, a few (i.e., UbiqLog [15], and Digital Diary [77]) are a bit closer to it. It is not hard to believe that recent technological advancements in smartphone can attract researchers' interests in developing solutions using integrated approach.
Based on the scope, the smartphone-based lifelogging systems can also be classified into two broad categories: total capture and selective capture (as shown in Figure 2.4), in similar to as discussed in Chapter 1. Rectangles and rounded rectangles hold the same meanings as in Figure
2.2. The selective capture logs information about specific aspects of a user's life in a few data types (e.g., audios, videos, images, message, notes, etc.) with a common understanding of predefined use-cases. The selective capture is common and gained market traction due to mining of immediate value from the focused data. Total capture is the creation of a unified digital record of multi-modally captured data of totality of life experiences in a variety of data types. The total capture has a broad spectrum and can support several use-cases. It is, however, complex and
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
38 requires sophistication in gathering, storing, processing and organizing into semantically meaningful form, and retrieval information for supporting various use-cases. None of the available smartphone-based lifelogging solutions supports total capture. However, the recent technological improvements and wide availability of apps for performing daily life activities on smartphone can attract researcher interests for total capturing of life experiences.
Figure 2.3: Taxonomy of smartphone-based lifelogging systems based on their architectures.
Based on the storage of lifelogging information, smartphone-based lifelogging research can be categorized into database and ontology (as shown in Figure 2.5). Rectangles and rounded rectangles hold the same meanings as in Figure 2.2. The database has been the primary choice of researchers for storing lifelogging information since years. They have successfully demonstrated storage and retrieval of information from database, which is partially on smartphone and mainly on remote backend storage servers. However, a database has fixed schema and cannot cope with the problem of accommodating new events and information that could emerge with the changing lifestyles of people over the time. In addition, daily life events information are related with each other in multiple semantic ways, which cannot be projected exactly in the present database technologies. Ontology is a Semantic Web technology that can potentially develop a
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
39 semantically enriched model of lifelog information. The semantic model would be more flexible and scalable as compared to database by adding/modification of new/existing lifelog information and relationships, reasoning and inferencing of new information based on existing information, and covering a wide variety of relationships and data sources. As discussed earlier, ontology can potentially organize and relate lifelog information with the same semantics as they appear in the real world and modeled in human episodic memory. A few of researchers have presented using of ontology formalism for lifelog information management in desktop environment such as [75,
78]. However, none of the smartphone-based lifelogging researches has paid attention to using ontology for modeling lifelog information on smartphone and is a potential research area.
Figure 2.4: Taxonomy of smartphone-based lifelogging systems based on their scope.
Apart from the above classification taxonomies, the available smartphone-based lifelogging research can be classified in several other ways as well such as sharing, retrieval, etc. As discussed earlier, lifelog information has two aspects that are private and public. Private aspect underpins that data remains in the user ownership and should not be shared with others, whereas, public aspect is related with sharing of lifelog information using consents of the users. Most of the researchers have presented methodologies of sharing lifelog information from smartphone
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
40 with their web interfaces, whereas, a few have postulated sharing on social media such as
Flicker. Furthermore, effective smartphone-based lifelogging practices should provide methods for the implicit swift real-time retrieval of lifelog information to augment human memory. Most of the researchers have postulated retrieval using contextual information and timeline display either directly on desktop computers or with the assistance of the Web, which is explicit and not in real-time. The advancements in smartphone can attract attentions of the researcher to provide retrieval methods to do things like combining, correlating, cross-referencing, data mining from heterogeneous sources, and presenting lifelog information in an appropriate and passive manner.
Figure 2.5: Taxonomy of smartphone-based lifelogging systems based on storage modeling.
2.2.3 Smartphone-Based Lifelogging Systems
Recent technological explorations in smartphone technology have appealed researchers, academia, and organization for using smartphone as an alternative lifelogging tool to custom- build devices. Realizing the fact, researchers have come up with several ideas of either exploiting the entire set of capabilities (i.e., sensing, processing, storing, and networking) of smartphone or using smartphone in conjunction with wearable devices or remote computing infrastructures. The
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
41 first generation of lifelogging apps (e.g., Saga3, Moves4, Rove5, etc.) for smartphone is already developed by the different organizations/researchers for passive capturing and archiving of specific types of life experiences and contexts data. This section presents a detailed discussion of the lifelogging systems developed specifically for smartphone. However, the discussion is limited to the systems that are available in the research publications and cited in the literature. A brief analysis of the available smartphone-based lifelogging solutions is shown in Table 2.2.
Nokia Lifeblog [73] was developed by Nokia as a commercial application for PCs and Nokia N- series smartphone and was expected to help users in collecting, finding, organizing, sharing, and archiving personal contents in an effortless way. Nokia called it "multimedia diary" that automatically records personal multimedia contents (i.e., images, videos, text and multimedia messages, sound clips, etc.) and enhances them with metadata information. Smartphone is used as life data recorder, data enhancer, portable data viewer, and data sharer tool, whereas, PC is used for data archiving, enhanced data viewer, and search tool. Nokia Lifeblog automatically collects metadata and contexts information, which are stored in database along with lifelog contents information. The collected items are organized into a timeline using the metadata and contextual information to facilitate their browsing and searching. Personal digital contents can be shared from timeline to online images album (e.g., Flicker) and blogging service (e.g., Typepad). iRemember [42] is audio-based memory tool for PDAs to continuously record users' everyday audio conversations from built-in as well as external microphones. The recorded sounds are transmitted to a large capacity PC wirelessly, where all audio recordings as well as associated
3http://www.getsaga.com/ 4http://moves-app.com/ 5http://roveapp.com/
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
42
data are archived and converted into text using Automatic Speech Recognition (ASR) tool. The
recordings are augmented with the related metadata (i.e., location, calendar event, email, local
weather, and news services) to improve their quality of memory recall. Using PC as a retrieval
tool, users can conduct keyword searches to find information in the collection and results are
displayed as a ranked list and on a timeline.
Table 2.2: An analysis of the available smartphone-based lifelogging systems.
System Sensors Contexts Contents Annotations Storage Sharing Retrieval Pictures, videos, Mobile GPS, Time, Location, Nokia SMS, Yes Phone and Camera, Object Name, Automatic SQLite Lifeblog [73] MMS, (Typepad) PC - metadata Phone Number notes, Timeline blogs iRemember Not PC - Microphone Time Audios Automatic No [42] Available Timeline Camera, Location, Time, Pictures, Semi- Lucene PC - Web Pensieve [4] Yes Microphone metadata Audios Automatic Indexer UI GPS, WiFi, Yes Location, Time, Experience GSM, Semi- Database - (Flicker PC - Neighborhood, Pictures Explorer [12] Bluetooth, Automatic MySQL Photo Timeline Keywords Camera Service) Acceleromet er, GPS, Pictures, Mobile Camera, Semi- Not WWW- Location, Time Audios, No Lifelogger [1] Microphone, Automatic Available Timeline Activities WiFi, Rotation GPS, Location, MemoryBook Bluetooth, Pictures, Not WWW - Time, Automatic RDF [79] Camera, text Available Timeline Neighborhood metadata SMSs, Not Semi- Not Smartphone- UbiqLog [15] Location, Time Pictures, Yes Available Automatic Available Timeline Calls SoundBlogs Microphone, Not Location, Time Audios Automatic Yes Smartphone [80] GPS Available SenseSeer Not Not Not Yes Smartphone, Not Available Cloud [14] Available Available Available (Cloud) WWW GPS, Digital Diary Location, Pictures, Yes Smartphone- Camera, Automatic SQLite [77] Neighborhood Audios (cloud) Timeline Infrared
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
43
Pensieve [4] augments human episodic memory and allows users for manually capturing experiences by taking pictures and recording audios using smartphone's camera and microphone respectively. The recorded data is annotated with date and time automatically and location manually. Upon connecting smartphone to a PC, the metadata is extracted from the captures for dividing them into contexts and more fine-grained sub-contexts groups. The captures are processed using Optical Character Recognition (OCR) and speech recognition engines to extract text and additional information. The captures are annotated with the collected and extracted information, related applications are updated accordingly (e.g., event entry in calendar), and indexed in Lucene. Users can query the index for searching and browsing the relevant captures.
Pensieve includes sharing features by enabling users to share their life information with others.
Experience Explorer [12] uses smartphone sensors to collect contextual information and user- generated contents using the device's camera. The contextual information includes outdoor location, proximity information, indoor location, cellular network cell ID, and date-time. The context information is periodically uploaded to context database on a network. User-generated contents are hosted form smartphone on external 3rd party media sharing service (e.g., Flicker photo service) for sharing with social connections. User interface is provided on PCs for requesting and visualizing the content-context relation form the explorer service. Explorer service uses contents and contexts information for answering clients' retrieval requests.
Mobile Lifelogger [1] provides digital memory assistance by forming activity language from sensors data using statistical natural language processing techniques. The proposed framework constitutes of mobile client, application server, and user interface parts. The mobile client application is developed for Symbian-based Nokia N95 smartphone, which is mounted on user's
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
44 helmet. The client records various sensors data (i.e., accelerometer, GPS, camera, microphone,
WiFi signal strength, and rotation) along with timestamps. The application server consists of components for pre-processing, modeling sensors data into activity language, storing data in a database, indexing of the activity corpus, hierarchical activity segmentation, and finally retrieving and labeling of the similar current and past activities of a user. Web Interface is a web application enabling users to browse and search their lifelog information.
MemoryBook [79] automatically generates lifelogging narratives from lifelog data. and enriches narratives by extracting information from the online Semantic Web knowledge stores for showing additional information about people, places, and things. A narrative generated is composed of information relevant with an event such as locations, images, etc. Imouto is a lifelogging system developed for smartphones and PDAs to collect sensory data. Viewer application segments the raw data into events and extracts information from online data stores
(i.e., DBPedia6, GeoNames, and Yet Another Great Ontology 2 (YAGO2)) in the Linked Open
Data (LOD) to produce rich visualization of events in a user's day. The web-based interface allows users to query Resource Description Framework (RDF) data using time range for generating narratives and display output as a web page comprising of narratives along with related images, maps, weather information, and hyperlinks.
SoundBlogs [80] enables to continuously record audios of long and short durations. Long duration audios can be feed into PC for feature extraction, segmentation into chunks of smaller clips, annotating clips with relevant semantic and acoustic tags (i.e., location, movement patterns, date and time, and calendar appointments), and archive sound clips along with related
6 http://wiki.dbpedia.org/
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
45 tags. From PC, the archived clips can be searched using keywords search and resultant sound clips are displayed as icon on a map using the location information. Selecting icon would prompt for listening of audio clip, provide a brief description of audio clip, and sharing of the audio clip.
Searching for a specific audio clip will not only display the audios that match the keywords but will also display the other audios recorded on the same day along with corresponding locations.
The archived audio recordings can be instantly blogged and shared with friends.
UbiqLog [15] is a generic, lightweight, configurable, and extendable lifelogging framework for smartphones. The framework emphasizes on sensors as core component of the lifelogging process and offers an open architecture, which is flexible in terms of enabling users to configure sensors (i.e., enabling/disabling of sensors and changing sensors settings), and adding/removing of sensors according to their needs. A specifically designed data model for lifelogging is proposed to further aid the flexibility feature of the framework. The architecture is generic and relives from the difficulties associated with custom-built systems. Using the proposed framework, an Android based prototype is developed with data capturing, visualization, and searching features. The implementation is evaluated regarding resources utilization, longituanal analysis, and usability requirements. The framework, however, stresses on the data collection part of lifelogging and ignores the semantic analysis and organization of the gathered data, and users' digital reflections.
SenseSeer [14] framework uses smartphone for data collection and stores data on a cloud server.
The framework extends UbiqLog [15] with back-end side with customizable analytic services for sensing, understanding semantics of life activities, and easy deployment of analytic tools with advanced interfaces. The framework is divided into three engines: collection engine, capture
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
46 engine, and semantic engine. Collection engine is similar to UbiqLog [15] and collects multi- sensors data for uploading to the cloud. Capture engine is a web service in the cloud for storing received data from smartphone into correct format and organization. Semantic engine is another web service, which employs a number of reasoning, machine learning, and statistical algorithms for data analytics and semantic extraction. However, semantic extraction is emphasized but no mechanisms or technologies are defined to evaluate viability of the framework.
Digital Diary [77] performs both the functions of capturing and retrieving lifelog information.
The system captures lifelog information in the form of images and audio clips. The captured lifelog information are stored temporarily locally and is uploaded to a cloud storage. To help in retrieving, the captured lifelog information is associated with three context elements: location, nearby person, and nearby objects. Evaluation showed that context elements especially nearby person are very helpful in recalling past-experiences.
2.2.4 Limitations of Smartphone-Based Lifelogging Systems
Certainly, smartphone-based lifelogging solutions can be helpful for people in increasing their power of recalling and sharing of past-experiences. However, we believe that smartphone-based lifelogging is still in its infancy because of the unavailability of significant amount of research work. Various aspects of the available smartphone-based lifelogging systems are discussed in the previous sections. However, their evaluation is difficult in terms of performance and resources consumption due to unavailability of detailed information about their custom-build architectures, developed applications, widely accepted tools, experimental details, and evaluation methods. In addition, they are not unified because of exploiting limited features of smartphone for providing solutions for a specific task or domain by collecting specific types of relevant lifelog and
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
47 contextual information. However, using knowledge from the previous sections, a satellite view comparison of the smartphone-based lifelogging systems from the different aspects is shown in
Table 2.3. In addition, limitations that are specifically related with smartphone-based lifelogging systems are highlighted in this section. The limitations are extensive and are provided with the intention to increase understanding of the researchers for improving and increasing the trend of smartphone-based lifelogging. However, the related limitations are addressed in thesis to fulfill its objectives.
Table 2.3: Satellite view comparison of available smartphone-based lifelogging systems.
Smartphone-Based Lifelogging Systems Parameter [73] [42] [4] [12] [1] [79] [80] [15] [14] [77] Distributed ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Integrated ✓ Architecture Open ✓ Features Scalable ✓ Flexible ✓ None Limited ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Sensors Usage All Internal ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ External ✓ ✓ Capturing Selective ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Scope Total Database ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Storage File ✓ ✓ ✓ ✓ Technology Ontology None ✓ ✓ ✓ ✓ ✓ Indexing Custom ✓ ✓ ✓ ✓ ✓ Method Standard None Retrieval Custom ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Model Standard None Use-Cases Specific ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Support Multitude None ✓ ✓ ✓ ✓ ✓ ✓ ✓ Evaluation Limited ✓ ✓ Method Excessive Semantic None ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Organization Limited ✓ Support Excessive
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
48 a. Distributed Architecture
Most of the systems are implemented with distributed architecture where smartphone is merely used for lifelog data collection and backend servers (i.e., Web or PC) are used for processing and storage. Therefore, requiring the users to have explicit connections and devices, which will not only increases the overall cost but also segregates lifelog information. Modern smartphone is capable of executing all of the lifelogging steps ranging from data collection to retrieval and exploration in a single place. Advantageously, this can ensure users' control on the lifelogging process and lifelog information, ubiquitous and omnipresent access to lifelog information, and data archiving without relying on additional technologies. b. Sensors Selection and Usage
The existing solutions have used either a single sensor (e.g., microphone [80]) or combination of a few sensors (e.g., GPS and camera [73]) for capturing and annotating of the lifelog information according to their own interests. However, none of them has evaluated individual sensors from the perspectives of their contributions to the overall performance of a lifelogging system and resources consumptions costs. Mostly, the current solutions primarily focus on visual lifelogging and acoustic in a few. In this respect, current solutions are of limiting lifelogging spectrum by addressing only a part of the problem of accurately capturing of life experiences. Technically, using broad range of sensors can provide rich contextual information about different aspects of an event, which could enhance event interpretation. This will also increase performance of smartphone-based lifelogging by providing more semantics to the lifelog information for advanced querying and exploration of lifelog information. However, this should not be at the cost of resources consumptions. Instead of using a static combination of sensors, adaptive and
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
49 dynamic selection of sensors and their sampling rates can produce more energy-efficient solutions. UbiqLog [15] has advocated for sensors selections but that requires explicit users' interactions instead of being dynamic and adaptive. In addition, effective schemes should be developed for fusing data from various sensors to obtain improved information for accurate estimations of contexts. c. Lifestyles Dynamics
New activities, events, and relationships emerge with the changing lifestyles and time. The available solutions are task-specific or application-specific, capturing contents and contexts information of predefined activities and events. Therefore, cannot be extended to handle newly emerging activities and events. In other words, they are static and focus on selective capture instead of total capture. In this respect, users have to use separate systems for capturing information about different aspects of their lives; thus, creating separate islands. This will create information fragmentation problem and would require users to switch between applications for complete augmentation of their memories, which is not practical and is a tedious job. Daily life activities and actions have different formats and relationships, requiring different capturing and archiving mechanisms. Therefore, dynamic and adaptive solutions are needed to be investigated that should be smart enough to not only fulfill users current lifelog information capturing and archival needs but also be able to handle future prospects into a single holistic structure. d. Open Architecture
Most of the available solutions have no open information about their architectures. Lack of architectural details about their internal components and functionalities, integration with other systems, modification and enhancements, users' customization controls, etc., troubles their
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
50 understanding and further improvements. UbiqLog [15] claims a flexible open architecture.
However, no internal details of the integral components and data model are presented. Therefore, a holistic widely acceptable open architecture for smartphone-based lifelogging systems is needed with complete description of individual components and their interactions. e. Database-Based Approaches
The smartphone-based lifelogging solutions have emphasized on storing and indexing contents, contextual, and metadata information using database technologies. As discussed earlier, database has a fixed schema and a slight change in the requirements would require restructuring of database schema. In addition, lifelog information are contextually related in several semantic ways, which cannot be represented exactly in the current database technologies. Therefore, using databases can leave semantic gap between the lifelog information stored and their occurrences in the real world. In addition, using databases cannot rationalize the sharing aspect of lifelogging because API access would be required for importing, exporting, and accessing of lifelog information. Therefore, more scalable and flexible data structure is needed to be investigated that should store any type of lifelog information and their relationships. Semantic Web technologies can solve this problem by formulating lifelog information using ontologies and storing in RDF triple format for advanced query, exploration, and connections with remote lifelog information. f. Information Indexing and Retrieval
A lifelog would contain excessive amount of a user's lifetime experiences information. Almost a year would be required to review/analyze the lifelog information collected and stored in a year.
Mostly, the smartphone-based lifelogging systems clusters, indexes, and visualizes lifelog information using location and time contextual information. However, the major challenge is the
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
51 useful indexing of a large lifelog for enabling users for effective and easy retrieval of lifelog segments of their interests. IR methods have been successfully used for solving semantic memory problems [9]. Therefore, leveraging of ideas and experiences from the IR domain in combination with semantic indexing and querying technologies can be helpful in fine-grained search and retrieval of specific information from a large lifelog. g. Personalization and Adjustment
The smartphone-based lifelogging solutions are lacking with personalization that is not allowing users to customize lifelogging process as per their needs and behaviors. Statically trained systems would not be adaptable/usable to new users due to differences in their needs and behaviors such as some people walk quickly and some people speak quickly as compared to others, etc. Some solutions are emphasizing on fixed positioning and orientation of smartphone for the lifelogging process (e.g., wearing smartphone in a lanyard around a user neck [13]) which sometimes become impractical due to societal and technical issues. Moreover, people would not like the lifelogging systems to track them in certain private situations. The provided systems mainly requires lifeloggers to manually adjust devices and organize lifelog information, which is less practical and less attractive due to producing extra burden on the users and limiting their freedom of performing daily activities. A practical solution is online training of a system, which would be intelligent enough to adapt to new users' preferences with minimum interactions. h. Use-Cases and Real-Time Feedback
The smartphone-based lifelogging solutions have mainly focused on capturing, archiving, and retrieving of lifelog information about a specific aspect or topic of a person's life. Due to being target-specific, the lifelog contains no rich life experiences to support a variety of use-cases
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
52 including reminding of menu ordered in the last visit to a restaurant, a person name who met in the last conference, etc. They are just restricted to provide an interface for searching and retrieving lifelog information in time line or displaying on map. Technically, smartphone-based lifelogging can support a variety of use-cases because of having modalities for detailed digital tracing of everything we do, processing to extract semantics and meanings, and organizing in a well-defined and explicit structure. However, the appropriate capturing may enable developers to investigate ingenious applications and tools to address the vision of lifelogging completely. The current solutions are also lacking with providing real-time assistive feedback for increasing human cognition, which has been the main source of popularity of quantified-self-analytics applications. In this respect, experiences from the previous researches can be utilized where smartphone is not used as primary sensing and classification devices but used as feedback device or interface [81]. To instigate users for extreme lifelogging, the smartphone-based lifelogging systems should provide real-time assistive feedback to users to persuade them in achieving their desired goals by using smartphone analogies such as text messaging, audio, light, graphical animations, etc. i. Lack of Evaluation
The smartphone-based lifelogging solutions are also observed of not having proper evaluations to justify their implementation, performance, and usability on smartphones. They have simply concentrated on capturing and retrieval of lifelog information using keywords or time information using smartphone interface or web interface. No proper methods and metrics are proposed for qualitative or quantitative evaluations of smartphone-based lifelogging systems. In addition, evaluation details are also missing such as sampling rate and capture rate of lifelog and
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
53 contextual information, position or orientation independency, etc. Proper evaluation methods and metrics should be investigated for smartphone-based lifelogging systems. In this respect, valuable experiences from other domains (e.g., recall and precision from IR) can be utilized. j. Semantic Organization
The smartphone-based lifelogging initiatives produce a flat collection of non-related daily life events with textual annotation and metadata maximally. It is like the information retrieval methods used prior to the Web, where retrieval models were established on presuming flat collections of text documents. However, this might work for many of the retrieval scenarios but not for lifelogging. Even a hierarchical data structure cannot represent a lifelog consisting of annotated events. We argue that organizing, searching, and retrieving a flat collection of lifelog information is a boring, frustrating, time consuming, and of less usage for users.
As discussed earlier, lifelog information are related with each other in different relations where the semantic interpretation of relationships depends on the humans' interactions with the real world. Technically, a smartphone-based lifelog would be a densely linked archive of daily life experiences where linking lifelog information in multitude of ways provides a graph. An important baseline rule for a useful smartphone-based lifelog would be organizing lifelog information in a semantic model/structure similar to human episodic memory, which is a significant challenge. Such semantic modeling would not only relate relevant lifelog information semantically using the contextual cues but would also help in developing appropriate retrieval model that would support a number of use-cases. We believe that semantically relating of captured lifelog information into events and sub-events would develop a linked graph that would facilitate organization, searching, and retrieval process, since personal lifelog information are
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
54 stored and related in episodes in the human memory. Valuable research experiences from the
Semantic Web domain (e.g., ontology) can be employed to organize smartphone-based lifelogging information in a subtler model that will closely match the real world.
2.3 Use-Cases for Smartphone-Based Lifelogging
E-memories developed using smartphone-based lifelogging could have revolutionary impacts on the people's lives, which were not possible in the past. It could change the meaning of being human by changing the way we work and learn, unleash our creativity, improve our health, and maintaining our relationships with our loved ones either dead or alive. Smartphone-based lifelogging could have varied and broad use-cases, which would become more clear as the technology becomes more popular and motivate people by providing some benefits. Whichever, might be the use-cases, this new technology should be developed and mapped into our lives instead of changing our lives for the technology.
The potential use-cases of smartphone-based lifelogging would depend on users' interests where some would be interested in recording information about themselves for their own benefits, some would like to record information for sharing with others, some would be interested in building a repository of life experiences in similar to maintaining a diary. Nearly all of the studies have emphasized on using smartphone-based lifelogging as memory aid due to not clear availability of its use-cases. Unfolding more use-cases will instigate the need of the development of novel capture technologies to tackle new data sources and will also indicate information needs of real world users for retrieval. Using 5R's of memory access and guidance from the book "Total
Recall" [34], use-cases of smartphone-based lifelogging can be identified. In this section, we are listing some of the potential use-case, mainly extracted from the available literature.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
55
2.3.1 Memory Augmentation and Assistance
Memory augmentation is the use of technologies to increase one's ability to retain information and retrieval of stored information from memory accurately and reliably. Continuing the legacy, a significant application of smartphone-based lifelogging is helping patients having episodic memory impairments such as Alzheimer's and other dementias. Information about life events are stored in human episodic memory and annotated with time and location information to facilitate event retrieval from episodic memory [31]. The available smartphone-based lifelogging systems have proven the creation a lifelog archive consisting of information about events, date-time, location, etc., which can be used as memory aid for augmenting episodic memory of users.
Moreover, the contextual information stored in lifelog can also assist people in their memory recalls. For example, reviewing of an image lifelog captured via smartphone camera and annotated with date-time and location can enhance short-term cognitive functioning.
2.3.2 User Modeling
User modeling is a sub-branch of HCI, which constitutes of processes and methodologies for building and modifying understandings about users. An individual's past actions provide enough information to predict his future decisions such as knowing a person's life could provide predictions about when and what he/she will buy [82].Tracking users' past activities has provided baseline for several of the user modeling approaches such as Doppelgänger user modeling system [83]. In information systems domain, past users' activities establish the foundation for user modeling is customization and personalization of systems according to their needs. A system is required to "say the right thing at the right time in the right way" [84]. Clarkson, in his experiments, has always carried a video recorder to extract users' life patterns [85]. Smartphone,
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
56 by default, has the modalities (e.g., audio/video recordings) to capture life activities and patterns for potential user modeling. In addition, tracking users by collecting detailed information about their activities on smartphone can also be valuable for online advertisement. Furthermore, continuing the legacy as reported in [86, 87], we believe that smartphone-based lifelogging can be beneficial for other systems such as recommendation systems. Smartphone-based lifelogging can also be used for self-reflection by enabling users to review how they have lived their lives.
2.3.3 Desktop Customization
Smartphone is growing to be a complete computing platform providing the same functionalities and potentials as desktop computers. Tracking activities on smartphone desktop can build contextual cues, which can be used to customize User Interface (UI) and information appearance on smartphone desktop and other desktop metaphors. Desktop customization can be of several benefits including search optimization, browsing of specific information objects, and optimize information access. Smartphone desktop customization using lifelogging is not addressed by any of the relevant researches. However, several projects in the desktop domain have used lifelogs for desktop customization e.g., LifeStream [50], SemanticLIFE [75], and iMemex [88].
2.3.4 Health Monitoring
Continuous recording and long-term preservation of an individual's biological data is also lifelogging [21]. Biometric sensors can give accurate data about physiological changes (e.g., pulse rate, blood pressure, skin conductivity, etc.) that happen amid different circumstances and activities [89]. Recording key biological signs benefit users in increasing self-understanding about their health status, enriching their health record, and helping in figure out how to affect their biological parameters consciously (biofeedback exercises). Biofeedback exercises are
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
57 proven helpful in improving users' health [90, 91]. Sensing and recording individuals' biological information has history long before the advent of digital age. Self-monitoring was priori actively logged (due to absence of passively logging technologies) normally using written diaries. Adolph
Meyer had invented a device called "life chart" for organizing medical information as dynamic biography demonstrating changes of habitant, birth, and deaths [21]. "Every Sign of Life" is among the earliest attempts to monitor a user's health by continuously recording his vital biological signs [92]. Some of the researchers have demonstrated that automatically created photographs can effectively increase short-term memory of Alzheimer patients as compared to other methods such as maintaining meticulous diaries. A number of companies have produced consumer level products for logging users' biological information and have gained traction in the marketplace including FitBit OneTM, Nike FuelbandTM, SenseCam, OMG Autographer, LarkTM, etc. These devices enable users to define daily or weekly targets to be achieved, tack users' activities, and provide reminders about health goals. Realizing the potentials of smartphone, researchers and academia have developed different applications that are demonstrating using of smartphones for health monitoring and laid foundation of new research area called "Mobile
Health Monitoring" [38]. DietSense project has used smartphone camera to create image log by automatically capturing of wearer's day [93]. The image log acts as a wearer's mealtime and provides feedback to help wearer in analyzing the diet intake and improving diet choice. There are apps to record users' activities like smoking cessation, diet monitoring for weight loss, tracking sugar intake for diabetes monitoring, etc., either manually or semi-automatically.
Similarly, there are apps on smartphone that uses inbuilt sensors to log users' movements for different purposes such as social, fitness, etc. Furthermore, a number of smartphone apps are developed that are using smartphone sensors to determine sleep quality of a user [9]. A detailed
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
58 survey of apps that uses smartphone sensing capability for capturing and logging users' health information is given in our previous work [38].
2.3.5 Environmental Impacts
Environmental pollution can cause severe effects on humans, animals, plants, and trees. One of the major challenges of Pakistan is the reduction of greenhouse gas emissions and moving towards a low carbon economy. Monitoring and recording users' indoor and outdoor energy consumption in smart environments can assist users in analyzing their energy patterns and identifying bottlenecks of their energy utilization systems. Several projects have employed sensor networks for collecting environmental parameters; however, large-scale deployment of sensors suffers from several problems including price, energy, bandwidth, and computational speed. Recent technological advancements in smartphone to connect with environmental sensors to record environmental parameters can be effectively exploited to determine environmental impacts on individuals, communities, etc. Distinguishable characteristics of smartphone (e.g., co- location with users, omnipresence in environment, etc.) make it an ideal tool to record information about individuals' transportation modes and their energy expenditures, and provide feedback [94, 95]. A number of smartphone apps (e.g., MobGeoSen [96], NoiseTube [97],
Nericell [98], etc.) are presented to record information about different types of environmental pollution and transportation modes. A survey about these applications can be found in our previous work [38].
2.4 Summary
Several lifelogging manifestations are produced over the time, which clearly indicates the active participation of research community in the exploration and development of lifelogging
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
59 techniques and devices. Wearable technology was used to provide better coverage for capturing lifelog information in completeness due to fact that occurrence of most of the daily life events is independent of space and time. Although, the wearable lifelogging solutions were of benefits to the users, however, they faced with several challenges and shortcomings. For example, fabricating users' bodies with extra devices and sensors can create problems numerously.
Similarly, the different native formats used to store lifelog information get obsolete with the passage of time. Thus, cannot be opened long after the systems that have created and stored them are gone. To overcome the problem associated with wearable technology, pervasive technology is considered as an alternative.
Smartphone technology is most ubiquitous and pervasive technology in the world today. In this chapter, we have explored and highlighted a comprehensive overview of the research efforts that have used smartphone within the context of lifelogging. A generalized architecture and several taxonomies are discussed with the aim to provide deep understanding and topics for this chapter.
The generalized architecture provides generic steps essential for smartphone-based lifelogging systems and taxonomies provides classification methods for smartphone-based lifelogging systems, highlighting the areas addressed by the researchers and other potential research areas.
An overview of the on-hand smartphone-based lifelogging researches is outlined, which has highlighted their main features, functions, and objectives. Sensors technology is used to provide contextual information (e.g., location, date and time, activity, etc.) to annotate the lifelog information and facilitate the retrieval process. In the comparative analysis, the available solutions are compared using a number of parameters to understand their limitations and provide justification for this research thesis. Potential use-cases of smartphone-based lifelogging are still
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
60 unknown; however, a number of potential use-cases are outlined to confirm viability of the novel paradigm.
It has been revealed that researchers in academia and organizations have merely used smartphone as a testing tool in their experiments. They have not provided any scientific justification or deep analysis of smartphone platform to highlight its potentials and develop a common consensus on using it as a de-facto lifelogging device. In the Chapter 3, we will elaborate and evaluate smartphone from different aspects to develop a common understanding of it as a genuinely lifelogging device. We will discuss that how the recent developments in smartphone technology have made it equivalent in power (i.e., both functionalities and resources wise) to the specially designed lifelogging devices such as SenseCam [13], etc.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
61
Chapter 3 : Materializing Smartphone as a
Lifelogging Device
By definition, personalized lifelogging systems would ease people by automatic and continuous recording their life events information [99]. The idea of lifelogging is old wine in a new bottle.
However, the recent popularity of lifelogging is due to several reasons including [100]: (i) exponential increase and wide availability of storage on cloud and personal computers; (ii) availability of varied advanced, cheap, and powerful sensors for capturing information about a person subjective and objective contexts; (iii) increase in the people's interest to capture and share information about themselves; and (iv) wide availability of novel advanced and powerful capturing technologies like Google Glasses. As discussed in Chapter 2, most of the wearable lifelogging devices were custom-built and relied heavily on external devices especially sensing and computing. Several specialized commercial lifelogging wearable devices (e.g., SenseCam,
OMG Autographer, Narrative Clip, etc.) are also fluted by the different organization/companies in the past few years. However, they are also limited in their features and have several other limitations. It is proposed that large-scale adaptation of lifelogging would be possible if lifelogging functionality is integrated into devices that are already owned and maintained by the users [13]. In this regard, an apparent candidate is the smartphone technology.
Smartphone is the wisest and convenient technological development in the history of humankind.
As discussed in Chapter 2, the technological improvement (i.e., both in hardware and software) have turned smartphone into computing device and enabled application developers to develop
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
62 applications for providing the same users' experiences as PCs with certain additions and improvements [101]. Technologically, smartphone already constitutes the resources and features that are used in the wearable lifelogging devices for life recording [100]. For example, smartphone applications typically define users' daily life activities/behaviors and sensors technology turn smartphone more into life recorder. The integration of computing and sensors capabilities in a single jacket can potentially turn smartphone into a de-facto lifelogging device.
Using smartphone as lifelogging device can have several practical advantages as compared to custom-build and commercially available lifelogging devices. These includes: (i) reduction in size and weight have enabled smartphone to approach the "wearability" of commercial lifelogging devices with certain improvements; (ii) smartphone has economy of scale and need no extra investments because of being already owned by the users; (iii) due to always been accompanied by the users eliminates the worries of recharging and maintenance of smartphone;
(iv) smartphone can provide life recording coverage to areas where traditional lifelogging devices are prohibited or hard to carry e.g., clubs, restaurants, etc.; (v) smartphone can provide coverage where it is needed the most and provide a close intact to the measuring phenomenon for getting accurate life events data; (vi) smartphone for lifelogging will have high societal acceptance due to its ubiquitous nature; (vii) smartphone can potentially provide "totality of life experiences" by capturing lifelog information in different data types and modalities instead of focusing on a specific aspect; (viii) human user assistances to smartphone can improve its lifelogging applicability; (ix) popularity of smartphone can fuel the trend and high-speed adaptation of the lifelogging technology. Therefore, as discussed in Chapter 2, researchers have articulated about using smartphone for lifelogging as a testing tool. However, none of them has provided insights evaluations and assertions of smartphone capabilities from the perspectives of
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
63 lifelogging system requirements. However, in a limited spectrum, efficiency of smartphone as lifelogging device has been demonstrated by carrying smartphone in a lanyard around neck to mimic SenseCam functionality [13]. This chapter fulfils the first and second objectives of this thesis. First, we aim to examine smartphone platform from several aspects to establish a common understanding of using smartphone as a potential replacement of custom-build and commercial lifelogging devices. Second, we identify the different contextual levels, which can be extracted from smartphone sensors data that can provide potential for semantically associating lifelog information into an enriched graph structure.
3.1 Smartphone Sensors
A sensor is an electronic transducer and a type of converter, which takes a physical quantity as input, and translates/converts and generates a functionality-related output in the form of electrical or optical signals to be read by an observer (i.e., human) or by a device (i.e., electronic instrument of some type) [101]. An example of sensor is mercury-in-glass thermometer that transforms the measured temperature into expansion and contraction of fluid in a calibrated glass tube, which is displayed and interpreted as temperature value by a human observer. There are numerous daily life appliances that constitute physical sensors and users use them while being unaware of their existence such as automobiles, electrical machines, medical, aerospace, robotics, manufacturing machines, etc.
Today's smartphones come with a broad range of built-in sensors, turning smartphones into more flexible and broadly available sensing methods [101]. In other words, the integration of specialized sensors has transformed smartphones from conventional communication devices into life-centric sensors [38]. Up to now, smartphone sensing not only contains advanced and
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
64 powerful sensors that can be found in real-world systems/devices but also several others, which were unimaginable before. Table 3.1 depicts sensors in modern smartphones in the market today.
A comprehensive review of smartphone sensors is available in our previous research work [101].
In the context of smartphone, sensors can be divided into five broad categories: physical, logical, virtual, informational, and application. Physical sensors are hardware components that are physically embedded in a smartphone as integral parts such as GPS, accelerometer, proximity, gyroscope, microphone, camera, etc. Logical sensors are software-based sensors that are formed by fusing data from more than one physical sensors such as e-compass is formed by combining data from 3D magnetometer and accelerometer. Virtual sensors are software services that runs in the background inconspicuously and capture information about user's activities on their smartphones. Informational sensors are applications such as contacts, calendar, etc., that use the processing and software capabilities of smartphones to record information about users' activities and interactions and make them accessible to other applications. Application sensors are smartphone apps that produces/creates lifelog objects and also attaches valuable metadata information with the lifelog objects. The software capability makes smartphone a step higher than dedicated sensor-oriented devices by allowing users to define new sensors according to their need by using data from the other sensors. The sensors categories are expended in the proposed semantic framework in Chapter 5. In addition to embedded sensors, sensing power of a smartphone can be increased by connecting with several specialized external sensors in environment through wireless networking technologies (e.g., Bluetooth).
In the following subsections, we discuss smartphone sensors from different aspects: potential applications of sensors for lifelogging, data generation strength of sensors, battery power
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
65 consumption rates of sensors, and utilization of sensors technology by researchers for the exploration of people-centric smartphone sensing applications.
Table 3.1: Sensory technology in some of the modern smartphones.
Net. Sensors** Smartphone Model Year Tech* A B C 1 2 3 4 5 6 7 8 9 10 11 12 13
Google Nexus 6 2014 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Samsung Galaxy Note 4 2014 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ HTC One (M8) for 2014 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Windows/Android BlackBerry Passport 2014 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Nokia Lumia 1520 2014 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Samsung Galaxy S6 2015 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Apple iPhone 6s 2015 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
LG Nexus 5X 2015 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Lenovo K3 Note 2015 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Samsung Galaxy Note 5 2015 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Huawei Nexus 6P 2015 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Google Nexus 6P 2015 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
HTC One A9 2015 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Microsoft Lumia 950 XL 2015 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
LG G5 2016 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
One Plus 3 2016 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
HTC 10 2016 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Motorola Moto Z 2016 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Apple iPhone 7 2016 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Samsung Galaxy Note 7 2016 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Samsung Galaxy S7 2016 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ * Networking Technologies: A-4G, B-Bluetooth, C-WiFi ** Sensors: 1-Microphone, 2-Camera, 3-GPS, 4-Accelerometer, 5-Proximity, 6-Compass, 7-Barometer, 8- Gyroscope, 9-FingerPrint, 10-SpO2, 11- Glonass, 12- Pedometer, 13-Hear-Rate
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
66
3.1.1 Smartphone Sensors for Lifelogging
Due to data-intensive nature, lifelogging requires employment of a broad range of sensors to capture and deliver lifelogging information to a receiver (i.e., hardware or software) for processing and storage into a lifelog. The dedicated lifelogging devices (i.e., both custom- designed and commercial) have also realized the fact and have included sensors in their design.
However, these sensors are limited in number to fit into all of the situations and do not provide any control over sensors during the sensing process. Using smartphone sensors for lifelogging can overcome the problems and have several additional advantages including: (i) combining sensing, processing, and storage features into a single holistic device transform smartphone into a smart sensor and ensure accurate estimations of measuring phenomenon; (ii) no need of mounting sensors on a user's body eliminates the hurdles in performing daily life activities and increases societal acceptance; (iii) support from smartphone battery power eliminates the problem of power management; (iv) support from enough battery power empowers to capture lifelog and contextual information for at least a day; (v) robust and unobtrusive to work in hard environmental conditions such as moisture and humidity; (vi) tolerant to drifts in calibrations;
(vii) economy of scale and cheap because of already embedded in smartphone; (viii) support from on-board enough storage to hold data for at least a day; (ix) advance and powerful, and provide accurate information as compared to sensors in commercial lifelogging devices; and (x) provide multimodal data to improve applications' functionalities [100]. A smartphone-based lifelogging system can use several sensors including motion sensor (i.e., accelerometer and gyroscope), positional sensor (i.e., GPS), acoustic sensor (i.e., microphone), optical sensor (i.e., camera), biomedical sensor (i.e., heart rate), etc., to continuously and unobtrusively capture multidimensional data for generating a rich lifelog [22, 100]. The resultant lifelog can constitute
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
67 a large amount of effective data about us. For example, who we are calling/messaging and how frequently, what and why activities we are doing, where and when we go, what and why information we consume, what images we are capturing with camera, what music we have been listening, who we interact with, what locations we have visited, etc. [12, 14]. Table 3.2 summarized the potential applications of smartphone sensors for lifelogging. However, using of many sensors can have side effects as well. For example, excessive usage of sensors can quickly drain out battery power, sensors can generate voluminous data in a short span of time, which requires high storage and processing power. These problems are more prominent is commercial lifelogging devices which provides no control on the sensing process. However, the programmable nature of smartphone enables smartphone-based lifelogging applications to customize the usage of sensors as per their needs. The smartphone can provide platform for developing automatic lifelogging solutions to capture lifelog data in required details by intelligently turning ON/OFF sensors and introducing intervals in the sensing process. In addition, the smartphone platform also enables developers to control sensors by defining the number and type of sensors to be used based on type of smartphone-based lifelogging.
As discussed in Chapter 2, the conventional lifelogging technologies primarily focus on visual lifelogging, which is not enough for decision-making. Therefore, it fall responsibility on the semantic processing tools, and human cognitive model of the real world (i.e., using five senses) and understanding to fill the semantic gap between the captured information and real-life event. On the other hand, smartphone sensing (i.e., physical, logical, and informational) capabilities can address the capture challenges by capturing sufficient lifelog information to fill the semantic gap and relieve users form the cognitive overload. Conclusively, the extensive .
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
68
Table 3.2: Potential applications of smartphone sensors for lifelogging.
No Categories Sensors Applications in Smartphone-Based Lifelogging Capturing occurrence information of any object in a range without being in Position Proximity physical contact in different systems such as logging of navigation patterns and 1 Sensors Sensor obstacle detection for people, especially visually impaired people, in a closed proximity, etc. Accelerometer Capturing motion information using acceleration forces both static and dynamic, and orientation information using angular momentums (i.e., roll, pitch, and yaw) Acceleration Sensor 2 for logging a powerful array of information about multitude of physical Sensors Gyroscope activities (e.g., walking, running, driving etc.), traffic accidents, games, falls, Sensor etc. Capturing temperature changes (i.e., heat, and cold) information generated by an Thermal Temperature object or in the external environment for logging information about body heat of 3 Sensors Sensor users in performing physical activities such as jogging, etc., environmental heat relevant to an event such as image or visit of a place. Capturing still photos or videos of users or objects in surrounding environment Optical 4 Camera Sensor for logging visual information about an event such as farewell party of senior Sensors student in a university. Ambient Light Mimics human eye to work under ambient light conditions by Sensor measuring/capturing light intensity of surrounding environment for logging 5 Light Sensors Back- information about an event such as tour to a historical place or help other Illuminated sensors such as camera sensor to adjust light while capturing a photograph. Sensor Establishing communication between smartphones by either touching or Near Field bringing them very close to each other for sharing lifelog data (e.g., photos, Communication videos, audios, etc.) locally, and logging information about indoor users' (NFC) Sensor localizations or electronic payments. Localization Capturing information about nearby WiFi access-point and GSM cell tower and WiFi Sensor finger prints (e.g., signal strength, cell tower ID, SSIDs, etc.) for sharing lifelog 6 Communicati data remotely, battery effective logging of users' localization information both on Sensors GSM Sensor indoor and outdoor, tracking movements of users, etc. Capturing information about the geo-locations from GPS satellites for logging GPS Sensor users' indoor and outdoor localization. Capturing identity information of Bluetooth-enabled devices/ objects in a user's Bluetooth vicinity/proximity, logging information about indoor user localization, and Sensor sharing of lifelog data locally. Capturing information about the strength and direction of North of a smartphone Direction Digital Compass to give right direction with respect to the north-south pole of earth by using 7 Sensors Sensor Earth's magnetic field for logging information about directions of events such as directions of users' physical activities (e.g., running, driving, etc.). Capturing atmospheric pressure information above the sea level for logging Altitude Barometer 8 information about short-term weather changes and altitude about events such as Sensors Sensor tracking of a hike in an unfamiliar territory. Heart Rate Capturing users' physiological information for logging information about heart Medical 9 Sensor rate, breathing, blood pressure, blood oxygen level, skin temperature and Sensors Biosensor condition, pulse rate, etc. Capturing voice information either produced by objects in a proximity or by Acoustic Microphone 10 users for logging voice information about events such as voice conversations, Sensors Sensor environmental noise pollution, traffic accident, etc. Capturing timing information from smartphone internal clock for logging 11 Time Sensors Clock Sensor information about events such as time of image capture.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
69
sensing capabilities of smartphone provide rich sensing experiences for lifelogging as compared to custom-build and commercially available devices.
3.1.2 Smartphone Sensors Data Generation Strength
The quality of a lifelogging system largely depends on the data captured by sensors. However, the accuracy of collected data largely depends on the data volume generation strength of a sensor. Technically, sensors with high resolution - higher sensitivity to produce greater amount of data - produces data with higher accuracy/precision level for intelligent decision making. An ideal sensor has output linear proportionality with input i.e., ∆I α ∆O, where ∆I is the frequency of variations in the input quantity and ∆O is the frequency of variations in the output. The sensors data analytics tries to detect anomalies by performing time-series or event-driven analytics instead of analyzing all data being captured [100]. The sensors data analytics enables filtering of sensors data locally and reducing the amount of data for storing or transporting due to either efficiency or security reasons. However, effective sensors data analytics required precise estimations of the sensors data generation capabilities [100]. One of the biggest challenges in smartphone-based lifelogging is the collection, storing, and interpretation of tremendous amount of data generated by smartphone sensors [100]. To tackle the sensors usage problems (e.g., power consumption and storage), insight knowledge of sensors' data volume generation strength is essential for lifelogging. The insight investigation will not only help for determining and quantifying data at a particular point in space and time but will also help in putting data in context over time and examining that how it correlates with related data.
To gain insights into data volume generation strength of smartphone sensors for lifelogging, we have designed and implemented an Android app namely Sensors dAta Volume Estimator
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
70
(SAVE) in our previous research work [100]. SAVE has a three-layer architecture consisting of user interface layer, processing layer, and physical layer, as shown in Figure 3.1. Each layer is composed of several sub-components that exploit the capabilities of the layer below. Screen shots of SAVE are shown in Figure 3.2.
Figure 3.1: Layered architecture for SAVE.
The user interface layer is the space where interaction between users and SAVE takes place. It is composed of three modules namely volume control module, volume visualizer module, and volume analyzer module. The volume control module is responsible for adjusting a sensor's reading frequency and time interval between consecutive readings to minimize and maximize a sensor's data volume generation speed. The volume visualizer module is showing a sensor's readings as well as maximum and minimum frequencies, maximum and minimum interval
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
71 between readings, and data volume per reading. The volume analyzer module is responsible for displaying each sensor accumulative data volume with per second time stamp.
Figure 3.2: Screen shots of SAVE.
The processing layer consists of three modules namely sensor module, calibrator module and recorder module. The sensor module is responsible for data processing and accesses the physical layer for retrieving data from all of the sensors. The calibrator module adjusts the reading rate of a sensor as per instructions from the user interface layer. The recorder module collects sensors data from sensor module and stores it in a local database along with reading frequency and time interval information from the calibrator module.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
72
The physical layer encompasses smartphone's built-in sensors and storage capabilities, and uses
Android's libraries (i.e., SensorManager) to turn ON/OFF sensors, adjust the reading frequency, and store the data volume generated by the sensors. SQLite is used by the recorder module for creating database to store sensors data and associated data for analysis purposes. From SQLite, the database can be imported to PCs for conducting more detailed and powerful analysis using applications such as NavicateLite, MS Excel, etc.
For accurate estimations of data volume generation strength of smartphone sensors, a comprehensive test program was designed to collect sensors data while performing daily life activities in different real-world scenarios including walking, driving, playing games, working in office, etc. [100]. The participants were instructed to perform all of their daily life activities uniformly and randomly for a period of two hours/day for a week to collect maximum data out of each sensor. After performing all of the tests, the collected sensory data collected was analyzed and results were compiled, as shown in Table 3.3. It is found that data generation strength of smartphone sensors depends on their sampling frequency rates and inherent working mechanisms. Comparatively, sensors with continuous data capturing nature (i.e., accelerometer, orientation, magnetic field, etc.) were found of generating more data as compared to event-based sensors. Furthermore, sensors with high reading frequencies were tend to generate huge amount of data in a short span of time. For example, accelerometer sensor was found of adjusting reading frequency range and resulted into generation of a huge volume of data as compared to others.
From the statistical analysis, conclusively, it is observed that smartphone sensors can produce exponentially increasing data volume that is enough for smartphone-based lifelogging systems.
Generating more and more sensory data is advantageous for accurate measurement and
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
73 presentation of tiny changes in the measuring phenomenon and determining the state and context of a user for lifelogging. The storage of such voluminous data either from a single sensor or from multiple sensors would raise the storage problem because the internal storage provided by state- of-the-art smartphones is not enough even to store the data generated by a single sensor for a year (e.g., accelerometer) [100]. However, storage is no more a problem. The external storage
(i.e., Secure Digital Card (SDCard)) technology has shown tremendous improvements and terabyte SDCard storage is marketed by the SanDisk7, which will be commonly available soon.
Table 3.3: Data volume generation strength of smartphone sensors.
Interval Samples/Second Volume/ Total Volume Sensor Name Sample Maximu Minimum Maximum Minimum Maximum in Bytes Minimum m 373.248 Accelerometer 0.01 Sec 0.6 Sec 1 102 12 38.07 GB MB 373.248 Orientation 0.12 Sec 0.6 Sec 1 8 12 2.98 GB MB 373.248 Magnetic Field 0.12 Sec 0.6 Sec 1 8 12 2.98 GB MB Not Not Proximity 0 5 2 0 1.86 GB Determined Determined Not Not 746.49 GPS 0 1 24 0 Determined Determined MB Not Not 373.24 Ambient Light 0 1 12 0 Determined Determined MB
3.1.3 Smartphone Sensors Battery Power Consumption
In smartphones, the battery size is restricted due to size and weight constraints of the devices
[102]. A modern smartphone featuring with a conventional cellular radio antenna, collection of sensors and services, touch screen, and many others requires greater energy source because each one is taking toll on the limited battery resource [103]. Empirically, applications using sensor can be the root cause of energy consumption by failing in determining the effective use of sensors
7https://www.theverge.com/circuitbreaker/2016/9/20/12986234/biggest-sd-card-1-terabyte-sandisk
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
74 and their data [104]. Therefore, sensors are to be used cost-effectively otherwise would result in quick complete battery drain out [104]. However, sensors vary in their battery energy consumption rates where some are very greedy as compared to others. For example, a switched on GPS receiver can completely drain out Nokia N95 8GB battery in 7.1 hours and 11.6 hours in indoor and outdoor localization respectively, whereas, accelerometer can take 45.9 hours to completely drain out the same battery [105].
The limited capacity of smartphone battery can foster big hurdles and severely restrict the effectiveness of smartphone-based lifelogging applications and services, which require huge energy due to using sensors. Theoretically, for smartphone sensors based systems, researchers have investigated energy consumption optimization at different levels (e.g., hardware and software) and defined energy management strategies either by immediately shutting down of unnecessary sensors or by carefully aligning of sensors duty cycles [106]. However, suggesting an effective strategy requires prior insight knowledge of different smartphone sensors energy consumption rates. This precise knowledge would enable smartphone-based lifelogging applications developers to employ sensors based on "where" and "how" philosophy [102] to produce qualitative lifelogging applications without jeopardizing the underlying platform. In other words, the effective use of smartphone sensors needs precise information about their energy consumption rates because the advantage of generating large volume of data should not be at the cost of greater and quick battery power lose. To effectively monitor, record, and analyze the energy consumption rates of the different smartphone sensors, an automated Android app namely EnergyMonitorApp is developed in our previous research work [107]. It consists of three layers: user interface layer, processing layer, and system layer (as shown in Figure 3.3).
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
75
Figure 3.3: Three-layer architecture of EnergyMonitorApp.
The user interface layer is composed of parent activity, from where other activities (e.g., graphs) can be invoked. The parent activity layout is composed of numerous controls (e.g., radio buttons, checkboxes, buttons, progress bars, etc.) for providing rich set of features and displaying information in percentage such as turning ON/OFF sensors, active sensors, screen light, battery remaining power, etc. The buttons on the parent activity enables users for controlling lower layers' components such as turning ON/OFF SensorApp. The parent activity works as an inspector, continuously pooling lower layer components for required energy consumption statistics and current battery status.
The processing layer is composed of three sub-components: utility service, power monitor service, and analyzer service. The utility service receives configuration commands from the
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
76 interface layer and invokes Android’s built-in modules in the system layer to fulfill the required tasks. Power monitor service starts automatically and periodically queries Android’s services in the system layer to retrieve, filter, split, and stores composite information including battery status, active sensors, energy consumption, etc. The analyzer service analyzes the recorded data, ranks sensors by their energy consumption rates, and pulls recoded information into PC for performing advance powerful analysis using statistical tools.
The system layer encompasses Android’s built-in services and libraries that are used by the application. The built-in services and libraries used in this layer include BatteryStates,
SensorManager, and SQLite. The BatteryStates service provides methods to retrieve different information including battery status, remaining power in percentage, health, temperature, etc.
SensorManager service provides methods to control sensors. SQLite is used for creating database to store energy consumption information that is to be used for analysis and conclusion purposes.
EnergyMonitorApp is implemented and tested on Android smartphone running with Ice Cream
Sandwich 4.0.3 or higher, as shown in Figure 3.4. It is tested to find the energy consumption rates of each smartphone sensor explicitly and in combination with other sensors at the same time in different real world scenarios. Each scenario is consisted of user state (i.e., motion or stationary), smartphone state (i.e., motion or stationary), sensors state (i.e., ON or OFF), environment state (i.e., building/indoor or open ground/outdoor), and user activity (.i.e., walking, upstairs, down stairs, standing, or sitting). These compositions are inspired from the real-world situations, which are commonly experienced by the users in their daily lives. Table 3.4 depicts the four scenarios along with their compositions. The energy consumption information of a sensor in real-world scenarios is estimated using the formulas shown in equation (1) and
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
77
equation (2). Equation (1) represents the energy consumed by a sensor Psensor is equivalent to the energy consumption information specified by the sensor manufacturer and the time unit Ttimeunit.
Equation (2) represents the total power consumed by a sensor in a scenario while the sensor is active. In equation (2), Psensor(j) represents the total energy consumed by a sensor Psensor in a scenario j and is equivalent to the summation of power consumed by the sensor Psensor in scenario time from 1 to n.
푃푠푒푛푠표푟 = 푃푝표푤푒푟 ∗ 푇푡푖푚푒푢푛푖푡 (1)
푛
푃푠푒푛푠표푟 (푗) = ∑ 푃푠푒푛푠표푟 (푡) (2) 푡=1
Figure 3.4: Main user interface screenshots of EnergyMonitorApp.
After performing the tests continuously for a week, the information about energy consumptions rate of each of the smartphone sensors is collected and analyzed. The overall energy consumption rates information of the sensors in the tests scenarios is shown in Table 3.5. It is found that GPS sensor is more power hungry and accelerometer is least power hungry in all of the scenarios. This is due to GPS frequent communications with satellites to find geo-location of
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
78 a smartphone. However, sensors showed variations in their energy consumption rates in indoor and outdoor, and stationary and motions situations. The sensors which are expected to have the same energy consumption rates in all of the possible situations due to their operating procedures
Table 3.4: Scenarios and their compositions.
Activities Scenario User Smartphone Sensor Environment Up Down Stan Walking Sitting Stairs Stairs ding Indoor Motion Motion ON Building ✓ ✓ ✓ ✓ motion Indoor Stationary Stationary ON Building ✓ ✓ Stationary Outdoor Motion Motion ON Open Ground ✓ ✓ ✓ ✓ Motion Outdoor Stationary Stationary ON Open Ground ✓ ✓ Stationary
Table 3.5: Smartphone sensors power consumption in percentage in different activities scenarios.
Activities Scenarios Sensors Indoor Motion Indoor Stationary Outdoor Outdoor (%) (%) Motion (%) Stationary (%)
Accelerometer 6.35 5.95 8 6.6
Proximity 6.7 5.9 5.9 6.9
Orientation 13.23 9.7 5.9 6.9
Light 6.9 6.0 6.0 6.0
Magnetic Field 13.0 11.4 12.5 12.9
GPS 46.0 45.22 53 51.0
Others 7.82 15.83 9.7 9.7
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
79
(e.g., accelerometer) resulted into different energy consumption rates. Conclusively, smartphone sensors are found power consuming components. Therefore, in smartphone-based lifelogging, efficient sensor power consumption management methods (e.g., using WiFi/GSM for user localization instead of GPS) should be used to not jeopardize smartphone normal functionalities.
3.1.4 Smartphone Sensing: A New Application Paradigm
The increasing incorporation of sensors in smartphone has fostered the proliferation of sensors- based application and has given rise to a novel area of research called smartphone sensing [108].
The smartphone ambient sensing power can be used as a primary tool for providing contextual information to a new class of smartphone cooperative services [106]. The sensors-based applications can capture a broad range of information about users' and their contexts and environments, which can be used locally for intelligent decision making (e.g., changing smartphone profile according to user context) and disseminated over a network to take other real- world advantages [38]. For example, using of GPS to provide location-aware services for facilitating a hiker's navigations in a rural area, using of accelerometer to aid functionalities to games and photography, using a person’s social network to find his current location, etc.
Smartphone can potentially provide effective platform of people-centric sensing applications
[109]. Over the past few years, researchers and developers have leveraged the smartphone sensing capability and introduced a number of people-centric smartphone sensing applications to solve real world problems of people in different domains, which would otherwise be impossible.
In addition, sensors applications can also ease quick data gathering in an urgent situation such as during a disaster-relief operation (i.e., earthquake, flood, etc.) personnel (e.g., sociologist, engineers, doctors, etc.) can use their smartphones to sense, monitor, and visualize real-world
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
80 phenomena for realizing public-health threats, environmental hazards, etc. The growing interest smartphone sensing applications is due to a number of reasons including [110]: (i) availability of cheap embedded sensors in smartphone have made the creation of disruptive sensing applications possible; (ii) smartphone is open and programmable which eliminates the barriers of entry for third-party programs and programmers; (iii) smartphone can provide coverage of hard to deploy geographical areas and capture valuable information about different aspects of people's contexts and environments in real-time; (iv) each smartphone vendor has an app store allowing application developers to deliver their applications to a large number of user across the globe, and (v) developers can upload services to back-end servers on the cloud to enjoy high-valued resources for computation of large scale sensors data and advanced features.
Participatory and opportunistic sensing are the two extremes in the design space of smartphone sensing applications, each with their respective pros and cons [110]. Participatory sensing emphasizes on the explicit active involvement of users in the sensing and decision-making processes to determine what and how to satisfy an application request and share data while abiding privacy constraints [111]. Opportunistic sensing makes data collection fully automatic and relieves users from the burden of taking explicit decisions for sensing, data storage, and satisfying application request [111]. Smartphone-based lifelogging demands for automatic and passive capturing of lifelog information; however, user involvement (i.e., participatory sensing) is essential in certain situations during the lifelogging process.
Up to now, we have a lengthy list of smartphone sensing applications developed by researchers, academia, and organizations, which have not only turned smartphones into people-centric sensing devices but also revolutionized the field of applications development. A comprehensive
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
81 survey of smartphone applications that uses smartphone sensing for capturing information about different aspects of people and their contexts to solve real world problems in different domains is presented in our previous research work [38]. These applications cannot be counted for lifelogging due to not fulfilling the long-time preservation of lifelog information aspect of lifelogging. However, they are context-aware and inherently related to lifelogging by clearly demonstrating the potential usages of smartphone sensors for capturing information about people and contexts, and processing of the captured information for useful decision-makings, which is the central idea of lifelogging. The survey [38] has shown using of smartphone sensors to capture contexts and contents information of a person's life. In the following subsections, a generalized architecture for smartphone sensing applications is presented to unify the research efforts, and real world domains are outlined where smartphone sensing capability is used. a. Generalized Architecture
Like smartphone-based lifelogging systems, smartphone sensing applications are architecturally different from each other. Applications are developed by developers in their own methodologies, which are creating separate islands. Therefore, we have proposed a general architecture for smartphone sensing applications in our previous research work [38], which has helped us in evaluating the smartphone sensing application in [38] and helped us in our proposed framework in the Chapter 5. This architecture can serve as a baseline for people-centric smartphone sensing applications and improve the applications' interoperability, components reuse, communications among applications, and minimize their heterogeneity. The architecture (as shown in Figure 3.5) divides people-centric smartphone sensing applications into five primary modules: information domain, sensing module, smartphone module, web server module, and visualization.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
82
The information domain represents the information resources from which a smartphone sensing application would extract its required data. Main sources of information include users' environments, contexts, activities, social relationships, and their physiological conditions.
However, an application’s information domain typically depends on the nature of its functionalities. For example, a user’s blood pressure monitoring system would extract its required information from users' physiological conditions, whereas, weather condition monitoring system would extract its required information from users' surrounding environments.
The sensing module represents the sensing components used by a smartphone sensing application. Sensors of varied nature (i.e., external or internal) can be used by an application to deduce required pieces of information from its information domain. However, the sensor usage is typically associated with an application's functionality. For example, a real-time
ElectroGradioGram (ECG) processing application would use external wireless ECG sensor and an old age fall monitoring application would use smartphone's accelerometer, microphone, and camera sensors. The sensing module might start capturing information proactively or reactively depending on the commands received from the sensing application in the smartphone module.
The smartphone module represents a smartphone platform to execute logic of a smartphone sensing application. A sensing application would either process the captured sensors data locally or forward it to a remote web server for enhanced processing and storage using Internet protocols
(i.e., Global System for Mobile Communication (GSM), Wireless Fidelity (WiFi), and General
Packet Radio System (GPRS)). Locally, a sensing application could perform the tasks of receiving sensory data from sensing module, performing different types of processing and inferencing to deduce new knowledge and accurate results from the sensors data, storing sensors
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
83 data and results, and presenting sensors data and derived conclusion in a meaningful form to the users. However, a sensing application would execute processing tasks under certain limitations to not jeopardize or create unnecessary load on smartphone.
Figure 3.5: General architecture of people-centric smartphone sensing applications.
The remote web server module represents an application running on a remote web server utilizing the sensors data and optionally conclusions forwarded by the smartphone module. Due to the underlying hardware and processing limitations of smartphone, a sensing application would delegate execution of complex operation/algorithms and storage to a remote web server. A remote application may perform the functions such as processing and inferencing on sensors
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
84 data, storing sensory data and derived conclusions, and reporting sensors data and conclusion to end users using web-based XML protocols (e.g., Simple Object Access Protocol (SOAP), etc.)).
The visualization module represents the methodology to provide the processed sensors data and derived conclusions to users. A visualization component can be either an integral part of a sensing application to render information directly on smartphone or third party applications (i.e., web browser) to render information on computers using web-based interfaces. b. Smartphone Sensing Application Domains
The highly penetration and availability of smartphones in human lives have opened up new disciplines of applications development for facilitating human lives in different spheres. As discussed earlier, the advancements in smartphone sensors technology have encouraged researchers and academia to develop a wide range of applications, which would otherwise be impossible. Analyzing the on-hand researches and available smartphone sensing applications, they can be classified hierarchically into groups and sub-groups using the domain areas, they address. Figure 3.6 depicts the various application areas of smartphone sensing applications. A detailed discussion of the smartphone sensing applications in different real-world domains is out of the scope of this thesis and can be found in our previous research work [38].
3.2 Adaptability and Importance of Smartphone Technology
To live a smooth life in the cotemporary world, people must face difficult challenges that are making technology as the deciding factor of people's standards. Research scholars from different fields and interests have developed a common consensus on smartphone technology as essential evolution in information technology [112]. The smartphone technology has made people's lives easy and comfortable, and has produced noticeable impacts on their behaviors and interactions,
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
85 standards of living and approaches to do things, marketing and business activities, education, and smartphone industry itself [113]. The rapid technological advancements in smartphone technology are due to meeting the people's increasing demands of advancements to make their lifestyles easy and modernized by extending our capabilities to solve real world or business problems in a timely manner. In addition to its main role of being sophisticated communication device to establish communication links across seas and borders, smartphone has access to a wide range of applications to play several other roles as well such as a tool for social presence and better exposure.
Figure 3.6: Application domains of smartphone sensing applications.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
86
Despite of being advantageous, success or failure of new information technology largely depends on users' responses to accept or reject it. Smartphone technology adoptability, popularity, and importance are increasing day by day due to showing promising features. Almost a decade ago, there was no idea of smartphone and its usage. However, today people cannot imagine living without smartphone. The amount of smartphone owing American adults has been increased from
35% in 2011 to 64% today. According to Gartner Inc., smartphone sales represented 78% of the total mobile phone sales and 380 millions smartphones are sold to the end-users globally in the first quarter of 2017 with an increase of 9.1% as compared to the first quarter of 2016 (as shown in Figure 3.7) [114]. According to survey conducted by Bank of America [115], 91% of the participants responded with smartphone as very important device, 60% of the participants responded with smartphone as even more important than morning coffee. In addition, 96% of the participants between the age of 18-24 counted smartphone as very important, and 93% of the participants of the same group considered smartphone as more important than deodorant and toothbrush. This wide popularity of smartphone technology is due to fulfilling people's needs of speed, quality, and effectiveness and combining these features into a single small solution that is enough to be carried in a pocket. In the following subsections, some of the reasons are presented that advocates for the importance and adoptability of smartphone technology in users' lives.
3.2.1 Improved Technology
The advancements in science and technology have empowered semiconductor technology to manufacture low-cost, high-power, and multi-functional mechanical devices called chips.
Smartphone technology takes the Moore's Law one-step forward by fabricating more and more functionalities in a single chip to compensate budget [106]. Today's smartphone is powered with
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
87 powerful hardware, advanced operating systems, and fast mobile broadband. They enable smartphone to execute sophisticated even scientific applications (e.g., Fast Fourier
Transformation (FFT) calculations), store and process data in a large volume, and communicate data with remote stations instantly [101]. Moreover, the reduction in weight has increased portability of smartphone. Collectively, they are making smartphone as sophisticated computing platform and miniature laptop, providing the same experiences and opportunities. In the following, we are presenting a swift overview of the advancements in smartphone technology.
Figure 3.7: Smartphones sales by different vendors in the first quarter of 2017 and 2016. a. Processor Power
It was believed that smartphone processor speed will not go beyond 1GHz due to power limitations [116]; however, QuelComm released dual-core snapdragon processor by the end of
2010 [101]. The smartphone processors, available in the market today are powerful enough to be used in low-end laptops or notebooks. For example, Qualcomm's 1GHz Snapdragon processor is
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
88 powerful enough to display a 12-inch screen at the resolution of up-to 1440 x 900 pixels. The quad-core technology has been recently introduced in the smartphones. It is believed that more and more improvements in the smartphone processor technology will be witnessed with the passage of time. Similarly, smartphone is integrated with Graphical Processing Unit (GPU) to execute graphical calculations and transformations, and reduces the burden of Central Processing
Unit (CPU) and enhance performance of the devices. Table 3.6 presents a swift comparison of some of the modern smartphones' processing technologies. b. Large Storage
As discussed earlier that according to Kryder's Law the storage capacity will keep increasing with the increasing demand of lifelogging data [11]. Today’s smartphones mostly have internal data-storage capacities ranging from 32 GB to 256 GB, which is much larger as compared to storages of few years back. A smartphone with 128GB storage (e.g., Samsung Galaxy Note 5) provides enough space to store images for more than 3 years if taken with a frequency of 1.65 million images per year [100].
The continued advancements and miniaturization in storage technologies have enabled the development of slim, lightweight, and high volume removable storage metaphors (e.g., microSD, microSDCX, and small form-factor disks) for the smartphones. As it was predicted in 2006
[116], the removable storage capacity has reached up-to 2TB in a single card, today. In addition to large internal and removable storages, today’s smartphone also support RAM up-to 6GB and providing room for the execution of complex applications. Increasing RAM capacity also contributes in enhancing speed of smartphone. Table 3.7 presents a swift overview of the modern smartphones storage technologies and capacities.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
89
Table 3.6: Swift comparison of smartphones processing technologies.
Graphical Model Manufacturer Central Processing Unit (CPU) Processing Unit (GPU) BlackBerry
Snapdragon 820 Quad-core 2x 2.15 Kryo & Quad-core 2 x1.6 GHz Kryo Adreno 530
DTEK60
OnePlus 3T Snapdragon 821 Quad Core 2x 2.35 GHz & 2x 1.6 GHz Adreno 530
Snapdragon Adreno 530 Dual-core 2.15 GHz Kryo & Dual-core 1.6 GHz Kryo Samsung 820 (Qualcomm (Qualcomm (Qualcomm Snapdragon) Galaxy S7 Snapdragon) Snapdragon) Quad-core 2.6 GHz Exynos M1 & Quad-core 2.3 GHz
Edge Exynos 8 Octa Mali-T880 Cortex A53 (Exynos) 8890 (Exynos) MP12 (Exynos) Apple iPhone Apple A10 Quad core 2.3 GHz ARMv8-A ----
7 Plus Fusion
LG G5 Snapdragon 820 Dual-core 2.15 GHz Kryo & Dual-core 1.6 GHz Kryo Adreno 530
HiSilicon Kirin
Huawei P9 Quad-core 2.5 GHz Cortex-A72 Mali-T880 MP4
955
HTC 10 Snapdragon 820 Dual-core 2.15 GHz Kryo & Dual-core 1.6 GHz Kryo Adreno 530
Snapdragon Adreno 530 Dual-core 2.15 GHz Kryo & Dual-core 1.6 GHz Kryo 820 (Qualcomm (Qualcomm Samsung (Qualcomm Snapdragon) Snapdragon) Snapdragon)
Galaxy S7 Quad-core 2.6 GHz Exynos M1 & Quad-core 2.3 GHz Exynos 8 Octa Mali-T880 Cortex A53 (Exynos) 8890 (Exynos) MP12 (Exynos) Sony Xperia X
Snapdragon 820 Dual-core 2.15 GHz Kryo & Dual-core 1.6 GHz Kryo Adreno 530
Performance
c. Operating Systems and Apps
Smartphone operating system uses the features of PC operating systems in combination with additional features of cellular, touch screen, WiFi, voice recorder, speech recognition, GPS mobile navigation, and PDAs [101]. Today's smartphone operating systems includes a wide range of features, libraries, and APIs to provide maximum flexibility, easiness, and freedom for
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
90 developing applications of different functionalities such as access to the internal sensors, making voice calls, Internet/Web connectivity, drawing maps, and playing multimedia [101, 117]. Like
PC operating systems, smartphone operating systems can be either open-source or proprietary such as Android is open-source platform that is backed by an industry consortium known as
Open Headset Alliance (OHA) and iPhone Operating System (iOS) is proprietary that can be used by the Apple devices only (i.e., iPhone, and iPad).
Table 3.7: Swift comparison of modern smartphones storage technologies and capacities.
Internal Storage Maximum Removable
RAM in GBs Storage Model 32 64 128 256 GB TB Type GB LPDDR3 LPDDR4
BlackBerry DTEK60 ✓ 2 microSDXC 4 ✓
OnePlus 3T ✓ ✓ NA* NA* NA* 6 ✓
Samsung Galaxy S7 Edge ✓ ✓ ✓ 2 microSDXC 4 ✓
Apple iPhone 7 Plus ✓ ✓ ✓ NA* NA* NA* 3 ✓
LG G5 ✓ 2 microSDXC 4 ✓
Huawei P9 ✓ 128 microSD 3 NA* NA*
HTC 10 ✓ ✓ 2 microSD 4 ✓
Samsung Galaxy S7 ✓ ✓ 2 microSDXC 4 ✓
Sony Xperia X ✓ ✓ 2 microSDXC 3 ✓
Performance
*NA: Not Available
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
91
To date, there is a very big list of smartphone operating systems including Android, iOS,
Windows 10 Mobile, Tizen, BlackBerry, Symbian, WebOS, Sailfish OS, etc., that are varying in their features and have different market shares. According to Gartner Inc. Android dominates the smartphone operating systems market in the 2nd quarter of 2016 and has market share of 86% with increase of 4% over the same period of time in 2015 [118].
In additions, operating system determines the features and functions available on a smartphone, which can be exploited by the third-party application developers to develop applications programmatically. Several SDKs and toolkits are available for each of the operating systems
(e.g., Android Studio for Android) that open new avenues of deployment of smartphone applications. Smartphone applications shortened "apps" are application software designed to run on smartphones. Initially, apps were aimed for general productivity and information retrieval such as email, calendar, contacts, and stock market and weather information [40]. However, rapid advancements, increase in public demands, identification of new application areas, and the availability of sophisticated development tools drove development attentions into other categories for solving real world problems. An important indicator is the availability of app store for efficient applications distributions. App stores make smartphone an appealing platform for applications' development [109] and enable developers to access a wide consumer market globally to generate revenue by charging users with nominal prices (e.g., average charges in the
Apple App Store is $1.02 per app). According to Statista, by March 2017 the top app stores have millions of downloadable apps where Android Google Play has the top position with 8.2 million apps (as shown in Figure 3.8) [119]. This availability of apps confirms the centrality of smartphones in the people's lives.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
92
Figure 3.8: Number of apps available in the leading app stores by March 2017. d. Other Technological Improvements
Apart from the mentioned facts, smartphone technology has shown improvements in several other aspects as well. In this section, we present a compact overview of these improvements.
Today's smartphone provides easier and advanced graphical user interfaces that are fully multimedia loaded, easy to navigate, easy to understand, touch enabled virtual keyboard, and user-friendly. The multi-touch technology is available in the smartphone to track more than one touch at the same time [117]. For example, a user can zoom-in a picture by touching the screen with two fingers and spreading them apart. The layer architecture (e.g., Cocoa-Touch for iPhone) together with powerful graphical framework makes smartphone user interface as the best presentation layer.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
93
The display size is increased, which support high resolution to enrich users' viewing experiences.
For example, Samsung Galaxy S7 Edge and BlackBerry DTEK60 have the same 5.5 inches touchscreen with 2560x1440 pixels screen resolutions display. The increase in size is due to the integration of more and more advanced components per square inch and the development of enhanced displays to visualize more information sophisticatedly. With increase in size, smartphone technology has shown decrease in weight as compared to the smartphones a few years ago. The decrease in weight improves smartphone's portability that is people can easily carry their smartphones from place to place. Furthermore, the battery power has been improved to meet the excessive power needs of the smartphone applications and underlying platforms. The increased battery power enables developers to develop sophisticated applications that are exploiting full potential of smartphone to solve the real world problems. However, as discussed earlier, research efforts are needed to increase smartphone battery power. Table 3.8 presents a swift comparison of modern smartphones' sizes, weights, battery power, and user interfaces.
3.2.2 Enhanced Connectivity Services
A prominent feature of smartphone is the integration of different connectivity modalities (e.g.,
GSM, and WiFi) to instantly connect and access the Internet. Internet access extends smartphone functionalities by providing different metaphors of connectivity with friends, colleagues, relatives, etc., beside phone calls and text messaging. In a survey, 85% of respondents have concluded smartphone as the central part of their everyday lives and marked it as the major source to remain up to date with their loved ones and social events [40, 120]. In addition to using social networking sites, several smartphone specific connectivity services are also developed that uses Internet instead of cellular network. For example, users uses Viber and WhatsApp
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
94 connectivity services for making international voice and video calls and instant messaging, and save money by reducing their phone bills. Another connectivity service is sending and receiving of emails on the move. Users can access their email accounts at any place and time using their smartphones.
Table 3.8: Swift comparison of modern smartphones' size, weight, battery and user interface.
Battery Weight User Model Size (mm) (g) Interface mAh Li-Ion Li-Po *RA **NRA
BlackBerry 153.9 x 75.4 x 6.99 165 3000 ✓ ✓ 5.5"
DTEK60
OnePlus 3T 152.7 x 74.7 x 7.35 15 3400 ✓ ✓ 5.5"
Samsung Galaxy 150.9 x 72.6 x 7.7 157 3600 ✓ ✓ 5.5"
S7 Edge
Apple iPhone 7 158.2 x 77.9 x 7.3 188 2900 ✓ ✓ 5.5"
Plus
LG G5 149.4 x 73.9 x 7.7 159 2800 ✓ ✓ 5.3"
Huawei P9 145 x 70.9 x 7 144 3000 ✓ ✓ 5.2"
HTC 10 145.9 x 71.9 x 9 160 3000 ✓ ✓ 5.2"
Samsung Galaxy 142.4 x 69.6 x 7.9 152 3000 ✓ ✓ 5.1"
S7
Sony Xperia X 143.7 x 70.4 x 8.7 164.4 2700 ✓ ✓ 5"
Performance
Apple iPhone 7 138.3 x 67.1 x 7.1 138 1960 ✓ ✓ 4.7"
*RA: Removable **NRA: Non-removable
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
95
3.2.3 Enhanced Efficiency
An important feature of smartphone is its efficiency. The technological advancements have enabled smartphones to execute tasks at a speed that is almost unparalleled. In fact, smartphone are faster and effective than PC in some occasions. The omnipresent and resources full nature of smartphone and availability of applications enables to solve real-world problems, which could not be solved by the PC, otherwise. For example, synchronizing email address on smartphone enables a user to perform business work and communications with the business partners while on the move. From the business point of view, smartphone enables users to access and share information with colleagues using services such as One Drive and Google Docs. Using smartphone, users can access their home computers whilst on the move for numerous purposes such as accessing documents on home computer for sending in an email to people.
3.2.4 Functionalities and Ultra Utilities
There is a common perception that smartphone will dwarf other digital computerized gadgets
(e.g., laptops, notebooks, and PCs) very soon. Today's smartphone encompass all of the functionalities and facilities that people require in their daily lives including e-mail, gaming panel, high-resolution camera, office suite, TV, and a wide variety of applications that a person can imagine. The availability of applications helps users in executing their daily life activities of any type. For example, configuring daily schedule, saving documents, watching videos, listening music, web browsing, emailing, video conferencing, finding nearest coffee shop, finding nearest best place for parking, performing online banking transaction, setting alarm, GPS tracking, mapping out routes for journeys and run schedule, and many more beyond users' imaginations.
Smartphone can be used as a fitness monitor to track heart rate, check weight, caloric intake, etc.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
96
3.2.5 Easy Entertainment Accessibility
Omnipresent access to latest entertainment is another prominent aspect of smartphone.
Smartphone have revolutionized entertainment by providing enormous easy and cheaper opportunities of entertainment. Instead of moving around with laptops, users can amuse themselves with their smartphones in long journeys. Just with a couple of clicks, users can directly access latest music, videos, and TV shows form the Internet. Smartphone can be potentially used to read favorite books and magazines online. In addition, smartphone also provides endless possibilities of gaming. Online mobile gaming has gained high market traction in the past few years. The enriched graphical resources of smartphone can display amazing graphics of the latest games that leaves users infused with enthusiasm.
3.3. Smartphone and Context-Awareness
The term "user context" refers to information that describes the situation of a person such as location, activity, environment, and preferences [22]. As discussed in Chapter 1, context- awareness represents the ability of a system for automatically adopting itself according to a user's context by providing appropriate information and services without requiring his/her active interactions [22]. Memory prostheses believes in importance of context over content and articulate that wise capturing of variable contextual information can provide powerful cues about users' activities and actions. These cues can be potentially used for augmenting human memory by indexing lifelog information automatically to ease the recall of past-experiences information from personal digital memories/libraries [30, 121]. A detailed overview of context importance in lifelogging is presented in [26]. Technically, lifelogging systems are context-aware pervasive systems with certain extensions [14, 15] that emphasize on passive and continuous capturing and
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
97 storing of contextual information from users' environments for assisting users in recalling their past-experiences information.
To sense and capture contextual information, sensors are used as the core component of context- aware systems. First generation of lifelogging systems has already been witnessed the use sensory technology to create a simple lifelog by capturing a specific user's contextual data (e.g., activity).The built-in sensors and connectivity with external sensors capabilities have enabled the development of smartphone personal sensing apps (as discussed earlier) to continuously and passively capture information about user's contexts. For example, recording locations, accelerations and movements of objects, nearby objects in a proximity, aspects of environment
(e.g., humidity, pressure, and temperature), etc., of a complete day with producing justifiable impacts on the battery power [9]. A comprehensive survey of using sensing capability of smartphone for context capturing and context-awareness is presented in our previous research work [22]. In the following subsections, we provide a swift discussion of the researchers that have used smartphone sensors for capturing different types of contextual information.
3.3.1 Passive Visual and Audio Contexts
Researchers have considered visual lifelogging as a main source of lifelogging. They have recognized that the use of images and videos can be very useful for people to remember the contexts of memories [122]. Using smartphone camera and microphone sensors for visual and audio lifelogging respectively are as effective as using of specialized or proprietary lifelogging devices [13, 42, 73]. Images captured using smartphone camera are found as valuable as images captured using SenseCam [13] and a real-time visual lifelogging solution using smartphone is developed by [123]. Similarly, audio lifelogging involves using of smartphone microphone
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
98 sensor to record speeches, music, conversations, and environmental sounds [80, 124].
Researchers have promoted that audio logging can be helpful in recognizing location, activity type, and people in a proximity using the voice matching or keywords and phrases spoken in a conversation [9]. A swift overview of smartphone applications using smartphone sensors for passive visual and audio contexts capturing is shown in Table 3.9.
3.3.2 Location Context
Location is a primitive context that can be used by computing agents to determine a user's current task and could be potentially used to develop a predictive model of his future movements. Active Badge system has highlighted the importance of location information for the design of memory aid systems by providing powerful cues for prosthetic memory [125].
Smartphone include a rich set of sensors (i.e., GPS, WiFi, GSM, etc.) for determining location of a user accurately. A swift overview of using smartphone sensors for localization lifelogging is shown in Table 3.10.
Global Positioning System (GPS) is the preferred choice of researchers due to its accuracy and infrastructure free nature [126]. Lifelogging researchers have found that using of GPS for estimating users' location contexts is not only useful in memory reconstruction but also in correct and flexible responses to different types of lifelog information retrieval queries [6, 63, 127].
Another potential application of GPS is building detection, which could be interpreted by the absence and presence of GPS signals. To overcome the problems associated with GPS (i.e., satellite signals weakness, unavailability inside buildings and vehicles, greater battery power consumption, etc.), WiFi-based and GSM-based localizations have been introduced in [127,
128]. Bluetooth sensor is suggested as an indoor localization indicator by detecting nearby
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
99 devices/objects and people in a vicinity, and can be used to gather social presence information
[128, 129]. Information from either GSM cell tower or GPS can be used in combination with
Bluetooth information for accurate localizations both in indoor and outdoor [79, 128]. In addition to localization, information obtained from Bluetooth scanning can be used as contexts of the people in vicinity [26]. In an experiment in MIT, event similarities and deep social patterns in users' activities are diagnosed by analyzing Bluetooth presence, duration and familiarity [36]. A combination of all of the localization techniques (i.e., GPS, WiFi, GSM, and Bluetooth) is also suggested for fine-grained localization even in case of absence of GPS [12].
Table 3.9: Overview of using smartphone sensors for passive visual and audio contexts
capturing.
Application/Publication Sensors Context
Experience Explorer[12] Camera Pictures
Nokia Lifelog[73] Camera Pictures
MemoryBook[79] Camera Pictures
Pensive [4] Camera, Microphone Pictures, Audios
iRemeber[13, 42] Microphone Audios
Mobile Lifelogger[1] Camera, Microphone Pictures, Audios
UbiqLog[15] Camera, Microphone Pictures, Audios
SenseSeer[14] Camera, Microphone Pictures, Audios
Digital Diary[77] Microphone, Camera Pictures, Audios
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
100
Table 3.10: Overview of using smartphone sensors for localization by lifelogging applications.
Sensors Publication Platform Localization Scope
GPS & WiFi & GSM [130] Smartphone Nokia N95 Indoor & Outdoor
Bluetooth [129] Smartphone Indoor
Bluetooth & GPS [126] Smartphone Indoor & Outdoor
WiFi & GSM [131] Smartphone Indoor & Outdoor
GSM & Bluetooth [128] Smartphone Nokia 6600 Indoor & Outdoor
GSM [127] Smartphone Nokia s60 Indoor & Outdoor
GPS & WiFi [1] Smartphone Nokia N95 Indoor & Outdoor
GPS & Bluetooth [79] Smartphone Indoor & Outdoor
GPS & WiFi & Bluetooth & GSM [12] Smartphone Indoor & Outdoor
3.3.3 Physical Activities Context
Recognizing human physical activities or special motions is one of the most important concerns in lifelogging systems that can be essentially used for automatic annotation of lifelog information
[5]. The integration of sensors in smartphone has provided a novel platform for the activities detection and a variety of methods have been proposed over the time with effective results. Out of the plethora of research, accelerometer has been proven as the most information rich and most accurate sensor for automatic activity recognition and has been used in most of the smartphone- based activity recognition and classification researches due to its inexpensive and effective characteristics. Most of the researchers have used accelerometer individually for simple activities recognitions. However, in some researches, accelerometer is used in conjunction with
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
101 other smartphone sensors (i.e., microphone, gyroscope, magnetometer, GPS, and pressure sensors) to recognize complex activities effectively. Data from multiple sensors is fused either at raw level or at higher level to recognize activities depending on an application's objectives. A number of classifier algorithms are implemented by the researchers in smartphone-based activity recognition systems. Some researchers have used a single classifier, whereas, others have used multiple classifiers to create multi-layer or hierarchical classification scheme. A comprehensive review of smartphone-based activity recognition research is presented in [132]. An abstract level comparison of the smartphone-based activities recognition research is shown in Table 3.11.
3.3.4 Personal Biometric and Environmental Context
Contextual information can also provide key indicators about user body, behavior, and environment during his/her daily life activities and actions. However, their nature varies from domain to domain. For example, automatic health monitoring systems would be interested in capturing physiological subjective contexts for pre-symptomatic testing; whereas, automatic air pollution monitoring system would be interested in capturing objective contexts (e.g., level of
CO in the air) for determining ambient air quality. In addition to smartphone sensors, several domain specific sensors are available such as sensors for heart rate, blood pressure, ECG, NOx, etc., that can capture respective information from users' subjective or objective contexts and provide to smartphone for processing, analysis, and decision-making. They help users in collecting information about their physiological changes that occur during various times of a day and during various daily life activities for detecting any abnormality and symptoms such as
Mobicare Cardio [147], and UbiFit [81]. Similarly, smartphone capability of connectivity with external sensors in smart homes can also capture and infer information about users' presence in
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
102 homes using infrared sensors and pressure sensors in chairs and beds, and indoor activities by monitoring electricity, gas, and water consumptions in homes.
Table 3.11: Abstract level comparison of smartphone-based activity recognition researches.
Publication Year Sensors Activities Classifier Platform
DET & [133] 2010 A, GPS WL, RUN,STI, BI, VH Nokia N95 DHMM WL, ST,SIT, RUN, WU, WD, BI, [134] 2012 A DET HTC G11 DR, JM, LA
[135] 2012 A WL, JOG,STI SVM iPhone
Samsung Galaxy [136] 2012 A WL, ST,SIT, RUN, BI, DR KNN Mini WL, RUN, VA, LA, WD, IR, BT, [137] 2012 A, Mic SVM Android Phone HD, FTT, BRD, unknown
[138] 2013 A WL, RUN,WU, WD, STI, BI, DR DET Samsung Galaxy Y
Samsung Galaxy [139] 2013 A WL, ST,SIT, RUN, BI, DR, PH DET Mini A, G, M, [140] 2013 WL, ST,SIT, RUN NB Google Nexus S GS, LA, OS WL, ST,SIT, RUN, WU, WD, Samsung Galaxy [141] 2013 A SVM LA S2 SVM & [142] 2013 A WL, JOG,WU, WD, BI HTC Nexus KMC
[143] 2013 A WL, ST,RUN, WU, WD, JM PNN LG Nexus 4
WL, JOG,WU, WD, STI, BI, DET & [144] 2013 A, M Samsung Nexus S ELU, ELD PNN Android [145] 2013 A, G, M WL, RUN, WU, WD, STI SVM Smartphone WL, RUN,WU, WD, BI, DR, [146] 2014 A, PS, Mic SVM LG Nexus 4 VH, JM, ELU, ELD, VA, WTV Sensors: A-Accelerometer; M-Magnetometer; Mic-Microphone; G-Gyroscope; PS-Pressure Sensor; GS- Gravity Sensor; LA-Linear Acceleration; OS-Orientation Sensor
Activities: WL- walking; ST-standing; SIT-sitting; JOG-jogging; RUN-running; WU-walking upstairs; WD- walking downstairs; STI-still; BI-biking; DR-driving a car; VH-in vehicle; JM-jumping; ELU-using elevator up; ELD-using elevator down; VA-vacuuming; LA-laying; PH-phone on table/detached; WD-washing dishes; IR-ironing; BT-brushing teeth; HD-hairdrying; FTT-flushing the toilet; BD-boarding; WTV- watching TV; unknown
Classifier: DET- Decision Tree; SVM- Support Vector Machine; KNN-K-Nearest Neighbor; NB-Naive Bayes; DHMM-Dynamic Hidden Markov Model; KMC-K-Medoids Clustering; PNN-Probabilistic Neural Network
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
103
3.3.5 Communication Activities Context
Smartphone supports several communication activities metaphors such as SMS messages, MMS message, audio phone calls, video phone calls, social networking activities, email messages, etc.
Support of the general-purpose operating systems enables developers to develop numerous tools for digital recording of contents and information about communication activities on smartphone.
Smartphone can be programmed to capture users' general smartphone activities and interactions using keystroke input and via screenshots to analyze their intellectual activities. There are several applications available to support the process such as recording SMSs, phone calls, web browsing, social networking histories, etc. An advantageous characteristics of communication activities logging is that the information to be captured is already in the text format; therefore, having less semantic gap between its contents and meanings. Together, monitoring and capturing of users' general activities and interactions on smartphones can also constitute a part of their lifelogs.
3.4 Common Daily Life Activities on Smartphone
Smartphone has transformed methods of engagements in our everyday lives. The growing penetration of smartphone technology in our everyday lives has resulted into increased screen time by engaging in numerous activities such as entertainment, and social media. Technically, people are addicted of using their smartphones in all situations including in dinner, bathroom, driving, movie, bed, etc. Now, the time spent using smartphone exceeds the time spent for web usage/surfing on computers. Nielson data has shown that today's smartphone users of age 18 and above spend 65% more time each month using apps than they did just two years ago [148].
Similarly, an average smartphone user checks his phone 150 times (after each 6.5 minutes) a day for different tasks and if each interaction lasts for an average of one minute it would mean a user
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
104 would be using smartphone for more than two hours up to whopping 2.5 hours per day [149,
150].
Over the past few years, the proliferation of smartphone has transformed us into an app-driven society, presenting marketers with new opportunities to connect with consumers by creating more interesting and sophisticated apps to command their attention. However, the number of apps usage varies according to demographics. For example, in USA the smartphones owners ages 25-44 are found using greatest number of apps with an average of 29 apps per month (as shown in Figure 3.9) in the fourth quarter of 2013 [148], and South Korea is the top chart country where an average smartphone user downloads 40 apps per month [151]. As discussed earlier, apps enable users to do more with their smartphones rather than just making phone calls.
Thus, makes smartphone as a portal of an ever-growing list of activities. ExactTarget survey has indicated that top smartphones users' daily life activities are accessing email (91%), text messaging (90%), getting news alert (62%), watching videos (30%), and getting directions
(24%) [150]. Pew Research Center's Internet & American Life Project survey has indicated that most popular smartphones users' activities are taking pictures (82%), test messaging (80%), accessing internet (56%), email (50%), video recording (44%), apps downloading (43%), online health and medical information (31%), and online banking (29%) [120].
Researchers have developed applications to provide reliable data about a user's average smartphone consumption (i.e., usage time and activities preformed) per day such as Menthal
[152]. However, an Android based application called ActivityLogger was developed in our previous research work [40] for explicitly identifying the main users' activities on smartphones in our local demographics. ActivityLogger was designed to execute inconspicuously in the
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
105 background and capture information about applications (i.e., type, name, usage duration, timing, etc.) used by the users while performing different daily life activities on their smartphones.
ActivityLogger was installed on smartphones of the 50 panelist (i.e., BS and MS students of the
University of Peshawar) aging from 18 years onward. The sample size was kept small in relation to smartphone owning population due to limitations of time and resources. However, the results obtained were found satisfactory. The data was collected on digital tracking of the panelists from
March 10, 2014 to March 25, 2014 and stored on the panelists' smartphones. After refining, organizing, and categorizing the collected data, it is found that most common activities performed by the panelists on their smartphones were text messaging, voice calls, entertainment, social networking, web accessing, email accessing, and playing games. The results of the survey are shown in the Figure 3.10, which shows most common activities and execution time in a day.
Figure 3.9: Average number of apps used and time spent per month [148].
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
106
Figure 3.10: Users' common daily life activities on smartphones and execution time.
3.5 Smartphone vs. Dedicated Lifelogging Devices
The use of Microsoft's SenseCam [13] in the preventive medicine has produced promising results. However, the widespread adoptability of SenseCam is affected due to its purchase, maintenance, and operating procedures. It was proposed that large-scale adoptability of lifelogging trend would be improved, if SenseCam functionalities are integrated in devices, which are already prevalent and users are accustomed of charging and maintaining them [13]. As discussed earlier, an obvious choice in this regard is the smartphone technology. Gurrin et al.
[13] have experimented of using smartphone as SenseCam replacement in 2012. It is observed that the reduction in size and weight of smartphone has reached its "wearability" to SenseCam. A wearability comparison of SenseCam (left) and smartphone (right) is shown in Figure 3.11.
In the experiment, they have developed a smartphone application that works similarly to
SenseCam by taking pictures during daily life activities accurately and meaningfully. Similarly,
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
107 annotation data is sampled from smartphone built-in rich set of sensors and external sensors as compared to SenseCam. Furthermore, the processing and storage capabilities of smartphone are used to provide the analysis power for supporting a broader range of lifestyles and behaviors. To demonstrate viability of smartphone as wearable camera, pictures data is collected from 47 participants, where each participant used smartphone for 8 hours a day. During the experiment, total 166,000 pictures were collected and the average calculated wearing time spent across all users was 5 hours 39. The captured pictures were found of sufficient quality and analyzing them using automated machine-vision technique resulted in identification of a range of lifestyle concepts. Conclusively, results of the experiment showed feasibility of enhanced replication of
SenseCam functionalities on smartphone. In other words, camera feature of smartphone can be effective replacement of wearable camera technology. Suitability of smartphone for lifestyle and behavioral monitoring is found due to the following reasons:
Figure 3.11: Wearability comparison of SenseCam (left) and smartphone (right) [13].
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
108
1. Smartphone, if used wisely, can support capturing information about all of a day
operations.
2. The integration of complete suite of built-in sensors and capability to connect with
external sensors makes smartphone useful for a variety of applications.
3. Smartphone is ubiquitous, less costly, and more familiar to people as compared to
dedicated lifelogging hardware devices.
4. Smartphone capability of real-time analysis on captured data provides foundation for
applications demanding immediate feedback/intervention and prospective memory cues.
5. The abilities of capturing data, displaying data, storing data, and providing feedback
enables omnipresent access to the captured life activities.
6. A valuable feature of smartphone is the ability to support the inclusion of new and easy
methods in lifelogging applications for enhanced capturing and interactions.
However, improvements in the smartphone's battery power and camera technologies were highlighted at the time of experiment (i.e., 2012), which has been significantly improved. For example, Samsung Galaxy Note 7 has 3500 mAh battery and 12MP rear and 5MP front camera.
Certainly, a today's smartphone is much faster and sophisticated as compared to a year-old smartphone. Therefore, based on the argument, repeating the same experiment using today's smartphone would definitely produce much better results over SenseCam and other visual- lifelogging metaphors (e.g., Narrative Clip 2). Narrative Clip 2 is the advanced version of
SenseCam. A compact resources wise comparison of Narrative Clip 2 and Samsung Galaxy Note
7 smartphone is shown in Table 3.12. Smartphone has rich list of features; however, comparison is restricted to dedicated lifelogging device features only. The comparison shows that
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
109 smartphone has far more superior qualities than Narrative Clip 2; hence, making smartphone a suitable replacement of dedicated lifelogging devices for lifelogging.
Table 3.12: Comparison of Narrative Clip 2 and Samsung Galaxy Note 7 smartphone.
Properties Narrative Clip 2 Samsung Galaxy Note 7
Sensor 8MP 12MP
Aperture f/2.2 f/1.7
Camera Resolution 3264 x 2448 (4:3) 4000 x 3000
Output Format JPEG JPEG, PNG
2160p@30fps, 1080p@60fps, Video Capture Full HD 1080p 720p@240fps, HDR
Dimensions (mm) 36 x 36 x 12 153.5 x 73.9 x 7.9 Body Weight (g/oz) 19/0.67 169/5.96
Connectivity USB, WiFi, Bluetooth USB, WiFi, Bluetooth
Internal 8GB 64GB Memory Removable No Support 256GB
Battery 315 mAh Li-Ion 3500 mAh
Accelerometer Inbuilt Inbuilt
GPS Inbuilt Inbuilt
Sensors Magnetometer Inbuilt Inbuilt
Gyroscope Inbuilt Inbuilt
Microphone No Support Inbuilt
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
110
3.6 Summary
Although researchers have demonstrated application of smartphone technology for lifelogging but none of them have evaluated smartphone technology from lifelogging perspectives. In this chapter, we have briefly examined smartphone technology as the potential lifelogging device.
Our recent research publications [22, 38, 40, 100, 101, 107] are included in this thesis as parts of this chapter. Based on these publications and discussion in this chapter, we can conclude that smartphone can potentially overcome the problems associated with dedicated lifelogging devices
(discussed in Chapter 2 and Chapter 3) and is the potential replacement of dedicated lifelogging devices for lifelogging. The facts presented in this chapter are enough to establish concrete understanding of using smartphone as a de-facto lifelogging device. The rich set of sensors in smartphone can provide excessive contextual information to annotate and relate lifelog information semantically in a graph structure. Therefore, Chapter 4 presents development of a comprehensive semantic model (i.e., ontology) for organizing sensors data, contextual information, and lifelog objects on smartphone in a semantic structure, as they exist in the real world and stored human episodic memory. The semantic model intends to solve the organization, retrieval, and supporting use-cases problems of smartphone-based lifelogging.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
111
Chapter 4 : Semantic Modeling of
Smartphone Sensors and Lifelog Data
As discussed in Chapter 3, the inclusion of sensory technology has turned smartphone into life- centric sensor and increased its capabilities and functionalities by introducing a new class of cooperative services including real-time monitoring systems, gaming, safety, and social networking [38, 101]. However, in frame of the smartphone-based lifelogging applications development, the integration of large-scale sensors data, which could be used for annotations, interpretation, and decision-making on lifelog information in a rich tactical environment, is difficult. In addition, the indiscriminate and inefficient use of smartphone sensors data can result into wastage of energy and computational resources.
The huge sensory data obtained from smartphone sensors intensifies the problem of too much data but not enough knowledge, which is undesirable for smartphone-based lifelogging applications. The problem results due to several reasons including : (i) the huge size of sensors data in varying formats and terminology mismatches in the description and values of sensors makes it difficult for Information Retrieval (IR) to search and retrieve relevant information; (ii) the design characteristics of sensors and their lack in adaptability to varying conditions hampers the accuracy and reliability of the captured data; (iii) varying sampling rates characteristics of sensors could result into missing of valuable data that limits the capabilities of applications to statically pre-defined usages of the collected data instead of showing dynamic behaviors; (iv) the lack and improper definitions of domain, sensors specifications, and annotations data can hamper
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
112 inferencing of domain knowledge from low-level sensory data; and (v) sensors data fusion could enable extraction of knowledge that cannot be perceived or inferred using individual sensor. The available smartphone-based lifelogging applications and frameworks are ill equipped at handling raw sensors data, where sensors data usage is static, pre-defined, and no soft integration of new data types from the new sensors. A slight change in the technologies and conditions would compel for re-designing of application. In addition, actionable knowledge is required for developer to develop useful applications, which is not possible from raw sensory measurement information [153]. Therefore, like other real world sensors-based systems, the data processing, management, and interpretation of smartphone sensors data is a big challenge that can be resolved either by smarting applications or data. The later approach is more practical by leveraging state-of-the-art technologies for more meaningful representation and semantic interpretation of smartphone sensors and sensors observations for using in potential lifelogging applications. Organizing and annotating lifelog information with contextual metadata is a prime issue for interpretation and event-centric retrieval of lifelog information. Apart from capturing lifelog information, method for representation and interpretation of multi-model contextual information is essential for enhancing different significant aspects of lifelog information.
The potential of sensor technology cannot be optimally exploited until the availability of a common language for explicitly expressing different aspects of sensors [154]. Sensors and sensors observations have already been standardized for improving interoperability amongst heterogeneous sensors data repositories and applications [155]. These standards, however, provide synthetic interoperability with no facilities of semantic descriptions for computer logic and reasoning [156]. Therefore, Semantic Web technologies could be used to provide semantic
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
113 layer to enhance our understanding of smartphone sensors data, contextual information, and lifelog information, etc. In this regard, ontologies allows for the annotation of sensors data with spatial, temporal, and thematic metadata to enhance semantic understanding, interoperability, and mapping of relationships between mismatching terms to improve performance of a system
[157]. Therefore, smartphone sensors ontology is immensely required to provide a common and widely accepted language as well as dictionary of terminologies for understanding the structure of information regarding smartphone, sensors, sensors observations, contexts, lifelog objects, etc.
The ontological modeling will provide highly expressive representations, advanced access, reuse of smartphone and sensors domain knowledge, formal analysis of sensors resources and data, mapping of high-level contexts, and making explicit the domain knowledge. The ontology would revolutionize smartphone-based lifelogging applications by providing a broader data model with the potential for integrating new and emerging contents and data types. The ontology would separate the application knowledge from the operational knowledge; thus, enable application and knowledge management easier and bring semantic interoperability among applications [158].
In this chapter, we achieve the third objective of this thesis by designing and developing smartphone sensors ontology namely SmartOntoSensor (SOS) [159]. SOS provides formal conceptualization and semantically representation of smartphone sensors and lifelog data as that exist in the real world and encoded in human episodic memory. The detailed discussion of SOS for context-aware computing is presented in our previous research work [159]. As discussed earlier, lifelogging is a sub-type of context-aware and SOS contains the required lifelogging concepts and properties to fulfill requirements of this thesis. Before discussing SOS, we present a compact review of Semantic Web technologies to understanding topics in this chapter.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
114
4.1 Semantic Web Technologies
Semantic Web [160] transforms the Web of information into the Web of knowledge, to provide qualitatively new levels of services through unambiguously representing the semantics of underlying data, programs, pages, and any other web resources. Semantic Web advocates that information will not only mean for human readers but also for machines, which will enable intelligent information services to provide greater functionality and interoperability as compared to the current isolated services [161]. Giving well-defined meanings to information would link information semantically and make it easy for machine processing for different purposes such as effective searching, sharing, integration, automation, and reuse across various applications [162].
Semantic Web technologies are a set of technologies and frameworks that are developed to narrow the gap towards the vision of Semantic Web. Semantic Web technologies provide constructs for explicitly representing knowledge and its further processing to infer advance knowledge from the implicitly hidden knowledge. As described earlier that it is very difficult to represent directly the semantics of sensors data and lifelog object (e.g., multimedia contents) due to their complex and raw nature. However, the extraction of metadata from sensors data, annotating lifelog objects with the metadata, developing relationships between the lifelog objects using the metadata, and storing metadata along with lifelog objects in a single package can decrease the complexities and enhance IR. In other words, the extracted metadata can provide the structured semantics for lifelog objects. We have faith in Semantic Web technologies to ease the formal and explicit interpretation of lifelog objects using the semantics derived from smartphone sensors data, enrich lifelog objects with the semantics derived from sensors data, and provide a standardized lifelog information representation model in machine-readable language to support
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
115 data searching, sharing, and inferencing. In the following subsections, we briefly cover some of the essential Semantic Web technologies required for the proposed work of this thesis.
4.1.1 Ontologies
Ontology presents an abstraction of domain knowledge in similar to a database schema in relational database and class diagram in object-oriented software engineering [26]. However, in
Semantic Web, ontology gives a generic presentation of domain knowledge and a commonly agreed understanding of domain knowledge to help people and machine communicate efficiently. According to Gruber's "ontology is a formal, explicit specification of a shared conceptualization" [163]. To represent an abstract structure of domain knowledge, ontology must be represented explicitly and formally by formal logic-based model so that machines can also understand domain knowledge in similar to human beings. To give formal and explicit representation, ontology is developed using reserved vocabularies, where each vocabulary is a collection of predefined terms. Ontologies differ significantly in size, scope, and semantics.
Ontologies can range from generic upper-level to domain-specific schemas. A small ontology may contain a handful of concepts, whereas, a large ontology may contain terms and relationships in hundreds [162]. Ontologies can be classified into lightweight ontologies and heavyweight ontologies. The former primarily correspond to taxonomies and simply contain classes, subclasses, attributes and their values, and simple class hierarchy (inheritance). The later represent domains in a detailed way and contain multiple inheritance, axioms, and constraints.
Ontologies are equally applicable to an array of fields including Semantic Web, e-Commerce,
Software Engineering, intelligent information integration, etc., for easing numerous information management tasks including information retrieval, information storage, and information sharing
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
116
[164, 165]. Designing ontology can be advantageous for a number of reasons including: (i) knowledge sharing among people and machines; (ii) knowledge reuse; (iii) disambiguating domain knowledge; and (iv) analyzing domain knowledge [158]. Ontologies and applications using them are independent from each other, which makes software and knowledge management easier and increases semantic interoperability among applications.
Ontology may presume different formalisms but corresponds to a knowledge area (domain) where it gives hierarchical description of explicitly defined concepts, description of attributes, and inter-relationships between the concepts that could be used for the domain interpretation
[165]. Ontology is stored as a document consisting of classes, objects, and attributes.
• Classes: Classes are the corner part of ontology and represent concepts in a domain of
discourse about which information is represented in ontology [158]. A class is a term that
is typically an abstraction of a collection of resources and share common understanding
and properties. For example, 'Sensor' term can be a class that represents collection of all
sensors either physical or logical, etc. [159]. Taxonomy in ontology is created by
arranging classes in hierarchical relationship also called class sub-sumption in which a
class can subsume (called super-class) or subsumed (called subclass) by other classes.
Subclasses of a class in a hierarchy represent classes that are more specific as compared
to super-class in a domain of discourse [158]. For example, the 'Sensor' super-class can
be sub-divided into 'LogicalSensor' and 'PhysicalSensor' subclasses and 'PhysicalSensor'
superclass can be further divided into 'ImageSensor', 'MotionSensor', 'OpticalSensor',
'LocationSensor', 'VoiceSensor', etc., subclasses [159]. Hierarchal relationship between
classes in ontology can be defined explicitly called asserted hierarchy or inferred
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
117
indirectly using the assertions defined in ontology called inferred hierarchy. By relating
classes in sub-sumption relationships implies direct inheritance of all of super-class(es)
properties by the subclass(es). For example, all of the subclasses of 'Sensor' class and
their subclasses directly inherits all of the 'has detect input', 'has produce output', 'has
sensor hardware', 'has sensor software', etc., properties of the 'Sensor' class [159].
• Individuals: An individual is an occurrence of a class in ontology that represents
resource in a domain of discourse, which belongs to a class and cannot be further divided
or specified. In similar to classes, individuals either can be declared explicitly as
instances of classes or inferred indirectly using the assertions defined in the ontology. An
individual instance declared of a subclass also becomes an instance of its super-class(s)
[158]. For example, an individual 'A' instance of 'GPS' also become individual instance of
'LocationSensor', 'PhysicalSenosr', and 'Sensor' classes. At the lowest level of abstraction,
individuals are not necessarily included in ontology. Ontology constitutes a set of classes
and adding a set of individuals transforms it into a knowledge base [158]. Practically, it is
hard to discriminate that where ontology ends and knowledge base starts [158].
• Attributes: Attributes represents properties of concepts (classes) in ontology describing
several features of their individual instances and relationships with other individual
instances. In other words, attributes determine metadata of individual instances. For
example, each individual of 'Sensor' class has 'frequency rate' property determining the
number of input samples taken in a unit of time. A property in ontology has to be
attached with a class determining that which class it describes. In ontology engineering, a
property serves as a predicate (link) between subject and object. A property can be either
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
118
datatype property or object property. Datatype property links a subject resource to an
object literal value and object property links a subject resource to object resource.
Similarly, axioms are restrictions on properties (also called facets), which are used to
define more semantics by determining the number of individual instances of a class.
Ontology is formed by RDF statements. A RDF statement is in the triple format composing of subject, predicate, and object commonly referred as (S, P, O) [164]. The subject determines the domain of a statement and is the URI of a resource (object). The predicate determines the property of a statement and is the URI of a property in vocabulary representing the relationship between the subject and object. The object determines the range of a statement and is either the
URI of a resource or literal value. For example, in the RDF statement 'LogicalSensor is a subclass of Sensor', the subject 'LogicalSensor' is linked with the object 'Sensor' by the predicate
'a subclass of'. Triple model of ontology can be represented as a directed labeled graph where subject and object in a RDF statement represents nodes in a statement and predicate represents arc/edge connecting them [164]. The Semantic Web ontology languages provide features for asserting statements in a triple model with rich semantics. A discussion about ontology serialization formats, languages, and tools can be found in our previous research work [166].
4.1.2 Ontology Description Languages
The Resources Description Framework (RDF) is the basic Semantic Web data modeling language. RDF uses XML syntax to represent metadata about resources in statements format (as discussed earlier). RDF provides constructs to develop a semantic data model of asserted statements by identifying resources using URIs, interlinking resources, and ultimately forming a graph structure. However, RDF lacks with flexibility and expressiveness issues because of
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
119 providing limited set of constructs. RDF Schema (RDFS) extends the Resource Description
Framework (RDF) vocabulary and provides a standard vocabulary for the definition of classes and properties taxonomies, individual instantiations, and specifications of domain and range of properties. RDFS together with RDF provide concrete semantics of resources by defining vocabularies, taxonomies, and ontologies in Semantic Web. However, RDFS also lacks with constructs for constraints and axioms to define high-level semantics of classes, properties, and individuals.
Web Ontology Language (OWL) is the more expressive Semantic Web data modeling language that syntactically extends RDF and RDFS with additional constructs for building expressive ontologies on the Web. OWL is semantically rich vocabulary for augmentation of RDF and
RDFS vocabularies and is syntactically expressed in RDF/RDFS. OWL provides constructs for constraints and axioms to provide efficient interpretation and inferencing while keeping the structure and contents of RDF documents. OWL was aimed to leverage the expressive and reasoning capabilities of description logic (DL) into Semantic Web. Ontologies expressed in
OWL provide developer with advantages of reasoning capabilities over defined classes, properties and axioms. To practically handle the expressivity and reasoning capabilities of OWL, three increasingly expressive sublanguages/species are defined:
• OWL Lite: It is a subset of OWL DL that offers basic limited set of OWL features to
express taxonomy of classes and properties, and simple constraints (e.g., 0 and 1 cardinality)
[164]. It is the simplest OWL language and corresponds to description logic SHIF. The
expressive power is limited; however, it is enough to model simple thesauri and ontologies.
For example, it sets limitations on inter-relating classes [164].
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
120
• OWL DL: It supports increased expressiveness while keeping computational completeness
and decidability [164]. The DL in the name depicts its support of description logic
capabilities. OWL DL corresponds to description logic SHOIN. OWL DL represents the
complete vocabulary of OWL Full; however, it imposes constraints on using with RDF and
requires declaring classes, properties, and individuals as disjoint [26]. The reason of OWL
DL is to enable tool builders to develop reasoning system that support ontologies developed
under restrictions of OWL DL. These restrictions provide many of the capabilities
description logic (important subset of first-order logic) which makes OWL DL decidable.
• OWL Full: It contains the full set OWL vocabulary and has no expressiveness constraints. It
does not set any syntactic constraints and allows free mixing with RDF and RDFS to use the
syntactic freedom of Resource Description Framework (RDF). Like Resource Description
Framework Schema (RDFS), it also does not require separation of classes, properties and
individuals [26]. However, the high flexibility of OWL Full does not guarantee any
computational properties/efficiency. To make some useful features available, it relaxes a few
of the Web Ontology Language - Description Logic (OWL DL) constraints but violates
constraints of the description logic reasoners [164].
4.2 SmartOntoSensor Ontology
The lifestyle of people changes with the developments in the society resulting into new events, interactions, and needs. Smartphones, because of their sensing capabilities, have the potential to capture these aspects and understand the needs of the users. However, due to complexities in understanding users' lifestyles, smartphone sensors are needed to perceive complex objects, and their actions as well as interactions effectively under varying operating conditions, strict power
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
121 constraints, highly dynamic situations, etc. Furthermore, smartphone-based lifelogging works by capturing excessive sensory measurements for inferring complex contextual information including information about environmental conditions, identities of objects in the environment, physical activities of objects and their positions as well as interactions, and the undergoing tasks.
Such smartphone sensors-based lifelogging applications demand for comprehensive semantic modeling of smartphone sensors data and lifelog objects. It is observed that the role of smartphone sensors ontologies is inevitable for improving the power of smartphone-based lifelogging applications. To meet the unique needs and applications, comprehensive smartphone sensors ontology is demanded, which consist of domain theory, represented in a language, and constructed using functional and relational basis to support ontology-driven inference.
The intended purpose of the SmartOntoSensor (SOS) is to develop an ontological model consisting of formal conceptualization of smartphone and associated resources, sensors and sensors observations, sensors measurements and capabilities, contextual (i.e., objective and subjective) and events/actions information, annotations and relationships, and lifelogging objects and services. The SOS includes categories, taxonomy, relationships, and metadata regarding characteristics, performance, reliability, and usability. In addition, the SOS includes logical statements that describe associations among lifelog objects and sensor concepts as well as aspects of their operating principles, computing & capabilities, platforms, observations & measurements, and other pertinent semantic contents. SOS has potential applications in several of smartphone-based context-aware computing including lifelogging [159]. The SOS is intended for context-aware computing; however, it also contains lifelogging and contextual concepts and properties to fulfill objectives of this thesis. The primary objectives of SOS includes: (i)
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
122 providing a semantic framework for capturing the important functionality features of smartphone sensors enabling lifelogging applications to reason about the available and running sensors for applying them to current information needs, querying, and re-tasking as needed; (ii) providing lifelogging applications with a semantic interface for managing, processing, integrating, and making sense of data acquired from a set of heterogeneous smartphone sensors; (iii) providing semantic description of smartphone sensors for reasoning available sensors capabilities and performances to construct low cost combinations of sensors for achieving goals of an operation; and (iv) providing basis for new measurement methods to evaluate each system's ability to perform the required tasks.
4.2.1 Goals and Design Rationales
The goals and design principles suggested by [153] are followed in the development of the ontology to ensure its coverage, validity, and usability:
• Domain: The ontology has been developed for representing information in domain of
smartphone and sensors, and mapping sensory information into high-level contexts to be
used in a variety of lifelogging applications.
• Simplicity: The ontology describes concepts, relations, and expressions in a simple and
easy-to-understand manner, to be used easily and effectively by the application
developers. An expressive and detailed ontology will be too complex, impractical, and
useless for most applications requiring necessary basic information. The ontology should
be simple and easy to understand by defining simple concepts, relationships and axioms.
The ontology should be simple and easily processable by automatic machines.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
123
• Flexibility and Extendibility: The ontology is flexible and extendable as it allows
developers/users with minimal overhead in adding new domain-specific concepts and
complementary relations in order to enhance interoperability and knowledge sharing
among smartphone-based lifelogging applications.
• Facilitate Inference: The ontological representation should not limit inferencing to any
specific method and liberate developers/users to employee any efficient inference method
using recognition engines or applications control. Restricting to a single inferencing
method is not effective because of none of the inferencing method today is optimal for
every type of problem.
• Generality: The ontology is general by describing concepts, facets, and relationships that
are possibly applicable to a wide range of smartphone platforms, embedded sensors, data
formats, measurement capabilities & operations, contexts, and lifelog objects.
• Efficiency: The ontology is memory-efficient and supports time-efficient inference
methods. The imports and constructs in the ontology are defined while keeping in mind
the limited nature of memory & processing resources of smartphones.
• Expressiveness: The ontology provides detailed information about ingredients and the
versatility of expressions is high. The ontology is lightweight but comprehensive by
declaring enough concepts and relationships to describe domain of interest.
• Applicability and Acceptability: The concepts and properties in the ontology are
defined and arranged precisely for ease in access and producing potentially high values
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
124
for quality and completeness metrics. In addition, the ontology can be easily used in any
of the smartphone Semantic Web framework such as AndroJena.
4.2.2 Materials and Methodology
In developing SOS, the NeOn methodology [167] is used, which emphasizes on the searching, reusing, re-engineering, and merging of ontological and non-ontological resources and reusing ontology design patterns, which are the main designing rationales of SOS. However, the NeOn methodology is lacking with ontology project management features, which are adopted from the
Practical Ontology Engineering Model (POEM) methodology [165]. Good ontological engineering emphasizes on leveraging of upper and related ontologies consisting of general ingredients and providing a common foundation for defining concepts and relationships in a specialized domain-specific ontology. The use of standard ontologies implies shorter development cycles, universalism, initial requirements set identification, easier and faster integration with other contents, and more stable and robust knowledge systems [168]. The
Content Ontology Design pattern [169] is used where contents in SOS are conceptual, instantiated from logical upper-level ontologies and provide explicit non-logical vocabulary for the domain of interest. Furthermore, The Stimulus-Sensor-Observation pattern [157] is extended into Smartphone-Sensor-Stimulus-ObservationValue-Context (3SOC) design pattern (shown in
Figure 4.1) in order to represent the flow of information from inception to application.
Figure 4.1: 3SOC ontology design pattern for SOS.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
125 a. SOS Requirements Specifications
The ontology requirements specification activity is performed for identifying and collecting requirements that ontology should fulfill. Ontology Requirements Specifications Document
(ORSD) is formed explaining: (i) purpose and reasons to build the ontology; (ii) scope of the ontology to fuel applications for mapping smartphone sensory data into high-level contexts for adopting services according to the contexts; (iii) users and beneficiaries of the ontology who will develop applications that interact with smartphones and services; (iv) ontology as a knowledge base to store data about smartphone, sensors capabilities and properties, contexts, services, lifelog items, etc.; and (v) and degree of formality by implementing ontology in Web Ontology
Language (OWL) for maximum expressiveness with computational completeness.
The SOS requirements are mainly concerned with non-functional and functional requirements.
The non-functional requirements comprise of terminological requirements (i.e., collection of terms used in the ontology from the standards that could be used to express competency questions) and the naming convention used for the terms. The terminological requirements of
SOS can be broadly divided into several categories: (i) base terms represent the basic classes of entities in the domain of interest, which could be further extended into sub-classes; (ii) system terms represent components, sub-components, resources, deployments, metadata, etc., of a system; (iii) sensor terms represent types, characteristics, processes, operations, configuration, metadata, etc., of sensors; (iv) observation terms represent input, output, response model, observation condition, etc., of observations that are used and produced by sensors; (v) domain terms are used for units of measurement, features selections and calculations, sampling patterns definitions, etc.; (vi) context terms are used for recognizing a context such as location, time,
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
126 event, activity, user, etc.; (vii) lifelog object terms are used to represent the different types of lifelog contents captured; and (viii) storage terms are used for the storage units used for storing sensors captured observations and other data such as file, folder, etc. A lexicon represents a set of terminologies used in the problem domain of an application. An excerpt of the lexicon is reported in Table 4.1.
Table 4.1: An excerpt of the SOS lexicon.
Lexicon
Smartphone WiFi Voice Recording Application
Accuracy Resolution Humidity Latitude
Location Physical Sensor Service Country
Picture Accelerometer Sensing Document
Bluetooth Observation Value Logical Sensor Email
Temperature Hardware Profile Multi-Touch
SMS Passive Sensing Time Unit
File MMS Single Output Event
Phone Call Active Sensing Observation Calendar
The SOS functional requirements represent the intended tasks, represented in competency questions, which the ontology should answer by executing SPARQL queries. For example,
"What is a smartphone location?", "What is the sensing accuracy of a smartphone X sensor?",
"Which of a smartphone sensors could be used for recognizing a Y context?", "What is the accelerometer sensor x-axis observation value for sitting context", "What are the humidity level
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
127 of an environment?", "What is the picture captured at location Z?", etc. The competency questions determine correctness, completeness, consistency, verifiability, and understandability of requirements. The domain characteristics, which are difficult to express in competency questions are written in natural language sentences such as "fusion of data from multiple sensors for mapping a context", "extreme environmental conditions can affect sensors observations and performances". The initial iteration produced a short list of functional requirements. However, the list is improved in the subsequent iterations and reached to 156 competency questions and 40 domain characteristics. An excerpt of competency questions is shown in Table 4.2.
Table 4.2: An excerpt of the SOS competency questions.
Competency Questions Question
CQ1 What is status of a smartphone?
CQ2 Is accelerometer sensor available in a smartphone?
CQ3 What is location of a smartphone?
CQ4 What is the output of accelerometer sensor?
CQ5 Which sensors can recognize a running context?
CQ6 What are the device conditions for GPS sensor to work?
CQ7 Who is user of a smartphone?
CQ8 Which service would be invoked if sleeping context is detected?
CQ9 What is the text of a received SMS at time 'X' and location 'Y'?
Which of the sensors are used by accelerometer as supporting CQ10 sensors for detecting a running context? Which of the sensors data can be fused to get information about CQ11 sleeping of a user?
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
128 b. SOS Development Resources and Tools
By following the NeOn methodology [167], SOS is developed by reusing the existing knowledge resources. The development task is divided into three iterations where both ontological and non- ontological resources are reused in the first and second iterations, and ontology design pattern is included in all development iterations. Table 4.3 shows the selected scenarios to be carried out in combination with Scenario 1.
Table 4.3: Relationship between scenarios and iterations.
Iterations Scenarios 1st 2nd 3rd
2 Reusing and re-engineering non-ontological resources Applied
3 Reusing ontological resources Applied
4 Reusing and re-engineering ontological resources Applied
5 Reusing and merging ontological resources Applied Applied
6 Reusing, merging and re-engineering ontological resources Applied Applied
7 Reusing ontology design patterns (ODPs) Applied Applied Applied
8 Restructuring ontological resources Applied
9 Localizing ontological resources Applied
SOS is constructed by reusing multiple relevant ontologies. The review and analysis of the sensors and sensor networks ontologies, and sensors vocabularies has highlighted that reusing available sources describing sensors, their capabilities, the systems they are part of, observation
& measurements, properties and associations, quantitative values for properties, etc., can provide
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
129 promising start for building SOS. After thoroughly analyzing the sensors and sensor network ontologies, the Semantic Sensor Network (SSN) ontology [157] is found most relevant due to providing advanced schema for describing sensor equipment, observation measurement, and sensor processing properties. SSN has a wider range of generality and extension space, which is reused in several projects to solve complex problems [170]. Therefore, SSN is extended for the development of SOS. Other ontologies are also found containing relevant ingredients but excessive imports can emerge certain problems including decrease in efficiency, simplicity, consistency, verifiability, and flexibility [171]. Some categories, taxonomies and definitions of commonly used concepts, properties, and metadata are adopted in part from SensorML [172].
Although the initial objective was to faithfully replicate the required items from SensorML; however, some implementation compromises and workarounds are made exclusively to meet the unique demands of the new paradigm. The context ontology (CXT) developed by CoDAMoS project [173] is imported and extended with required domain concepts and relationships for modeling context in SOS. Sensors data are stream requiring indefinite timestamp sequence information for unique representation. Therefore, OWL Time ontology (TIME) is reused for incorporating time information in SOS. The lifelogging concepts are explicitly added in SOS due to unavailability of any standard lifelogging ontology to meet the unique needs of this thesis.
To develop SOS, classes in the imported ontologies are either used directly or extended by making SOS classes as sub-classes of the relevant classes. Furthermore, classes in the imported ontologies representing the same concepts are aligned by declaring them equivalent classes and other classes are refactored either by restructuring the class hierarchy or by defining new associations and relationships. The additional domain-specific contents of SOS (both sensors and
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
130 lifelogging) are captured from the investigation of the related literature. Figure 4.2 shows abstract level structure of SOS by highlighting the information sources used.
Figure 4.2: Abstract level structure of SOS.
To communicate semantics of sensors observations, an appropriate terminology (obtained in lexicon) is defined for describing concepts, relations, and processes. The terminology used is discussed in the subsequent sections. After formally defining the constituents, SOS is developed in Web Ontology Language - Description Logic (OWL-DL) language using open source ontology editor and knowledge-based framework Protégé 4.3 with its exclusive plug-ins (e.g.,
SPARQL, RacerPro, etc.) for ontology editing, development, implementation, and testing. c. SmartOntoSensor Framework
The SOS framework is conceptually (but not physically) organized into ten modular sub- ontologies where a modular sub-ontology represents a sub-domain and a central ontology links the other ontologies. Each of the sub-ontologies contains a number of concepts and properties for modeling a specific aspect of the domain of interest. The SSN framework [157] has provided
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
131 inspiration for the development of SOS framework and some of its modules are refactored and merged for inclusion into the SOS framework. The novelty of the framework is four-fold information presentations to fulfill objectives of SOS. First, detailed information about smartphone systems regarding their hardware components, software, platforms, metadata, potential deployments, etc., is presented. Second, the detailed information about smartphone sensors regarding their inputs, outputs, processing, observations, measurements, operating restrictions, capabilities, etc., is presented. Third, potential applications of observation values
(i.e., produced by smartphone sensors after sensing process) for context recognition and context modeling as a whole such as user information and profiles, current and planned activities and events, device and its surroundings, etc., is presented. Fourth, enhancing of smartphone context- awareness capability for solving the challenges of applications adoptability according to users' contexts is presented. Figure 4.3 depicts the SOS framework and a snippet of the main concepts and their relationships. The framework represents the main high-level concepts and object properties in a sub-domain, whereas, detailed concepts and object properties are left aside. In the framework, classes are represented with rectangles and subclass axioms and object properties are represented with solid and dotted arrow lines respectively. The detailed discussion on the
SmartOntoSensor framework's sub-domains is presented in the following sub-sections. i. Smartphone
The smartphone sub-domain is constructed around using concepts for modeling knowledge about a smartphone system describing its resources (i.e., hardware and software), organization, deployment, and platform aspects. The sub-domain will provide essential information to help different operation such as application customization. The SOS:Smartphone concept is derived
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
132
Figure 4.3: SOS Framework.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
133 from the SSN:System concept of the SSN ontology [157] providing necessary properties for deployment and platform. SOS:Smartphone could have different hardware resources including sensing (SOS:SensingResource), memory (SOS:MemoryResource), network
(SOS:NetworkResource), etc., and software resources including operating system
(CXT:OperatingSystem), middleware (CXT:Middleware), etc. Each of these resources is a system by itself. A number of object properties (i.e., SOS:hasBluetooth, SOS:hasWiFi,
SOS:hasSensor, hasOperatingSystem, etc.) are made as sub-properties of SSN:hasSubSystem for linking SOS:Smartphone with resources. The SOS:Smartphone inherits SSN:hasOperatingRange and SSN:hasSurvivalRange properties to define the extremes of operating environments and other conditions in which a smartphone is intended to survive and operate for providing functionalities including standard configuration, battery lifetime, and system lifetime. A smartphone could be mounted or connected (SSN:onPlatform) with a platform (SSN:Platform ≣
CXT:Platform) which could be having hardware (CXT:providesHardware) and software
(CXT:providesSoftware) features. The SOS:Smartphone could be deployed
(SSN:hasDeployment → SSN:Deployment) at a specific place including worn on a helmet
(SOS:Halmet ⊆ (SSN:Platform ≣ CXT:Platform) or around the neck, attached with belt
(SOS:Belt ⊆ (SSN:Platform ≣ CXT:Platform), placed on a selfi-stick (SOS:SelfiStick ⊆
(SSN:Platform ≣ CXT:Platform), placed in treasure pocket, etc. ii. Resources
The resources sub-domain is constructed around concepts to represent hardware and software resources of a smartphone. The CXT:Resource concept is made as sub-class of the
SSN:DesignedArtifact. CXT:Resource has CXT:Software and CXT:Hardware to represent the
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
134 hardware and software resources respectively. The CXT:Hardware has sub-classes of
SOS:RAM, SOS:ROM, and SOS:SDCard to represent the storage features of a smartphone.
Similarly, SOS:Processor is also sub-class of CXT:Hardware to represent processing capability of a smartphone. The CXT:OperatingSystem and CXT:Service are made sub-classes of
CXT:Software to represent the operating system and applications installed on a smartphone. The sensing capability of a smartphone is represented by declaring SSN:Sensor as sub-class of
CXT:Hardware and CXT:Software because of the fact that sensors can be either hardware components or software installed on a smartphone. The SSN:Sensor has SOS:PhysicalSensor,
SOS:LogicalSensor, SOS:VirtualSensor, SOS:InformationalSensor, and SOS:ApplicationSensor to represent the different sensing methods in a smartphone to produce both lifelog objects and contextual information. Each of the concept/class has datatype properties to represent annotations. For example, each of the storage units has a memory size that is determined by its
SOS:hasMemorySize property and the processor has speed determined by SOS:processorSpeed.
Similarly, each of the hardware resource has SOS:hasManufacturer to determine its
SOS:Manufacturer, and SOS:Model to determine its SOS:Model, etc. iii. Sensors
The sensors sub-domain is the cornerstone of SOS that serves as a bridge for connecting all other modules. It models smartphone sensors using concepts and properties for describing taxonomy of sensors, types of operations, operating conditions, resource configurations, measuring phenomena, etc. During the alignment process between SSN and SOS, the SSN:Sensor concept is extended by a detailed hierarchy of sensors in SOS. The SSN:Sensor is categorized into logical
(SOS:LogicalSensor) and physical (SOS:PhysicalSensor) sensors where physical sensors
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
135 represent hardware-based sensors and logical sensors represents software-based sensors that are created by employing one or more physical sensors. Individual sensors (e.g.,
SOS:Accelerometer) are declared subclass of either physical, logical sensors, or informational sensors. A physical sensor has type of operation (SOS:hasTypeOfOperation) which could be either active sensing (SOS:ActiveSensing) or passive sensing (SOS:PassiveSensing). A physical sensor has to work under certain conditions and have specific features such as SSN:Accuracy,
SSN:Latency, SOS:Hystheresis, SSN:Sensitivity, etc., to capture a particular stimulus. A sensor could have its own hardware (SOS:hasSensorHardware) and software (SOS:hasSensorSoftware) specifications. A sensor can measure (SOS:measure) properties of a phenomenon that can be quantified (i.e., that can be perceived, measured, and calculated) which could be either physical quality (SOS:PhysicalQuality) or logical quality (SOS:LogicalQuality). A sensor detects
(SSN:detects) a stimulus (SSN:Stimulus) which is an event in the real world that triggers a sensor and could be the same or different to observe property and serves as a sensor input
(SSN:SensorInput). The output (SSN:'Sensor Output') produced by a sensor can be either a single value (SOS:SingleValue) or more than one value (SOS:TupleValue). iv. Process
In addition to physical instrumentation, a sensor has associated functions and processing chains to produce valuable measurements. The sensor concept has to implement (SSN:implements) a sensing process (SSN:Sensing), which could be either participatory (SOS:Participatory) or opportunistic (SOS:Opportunistic). A process can have a sub-process (SOS:subProcess) such as a sensing process could require a prior calibration process. Therefore, concepts for calibration
(SOS:Calibartion) and maintenance (SOS:Maintenance) processes are declared as subclasses of
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
136
SSN:Process and are disjointed with SSN:Sensing. A sensing process could have input
(SOS:Input) and output (SOS:Output) parameters. A sensing process can use calibration and maintenance processes as complementary and supporting classes using SOS:SubProcess property. A process has type (SOS:hasProcessType) either physical (SOS:PhysicalProcess) or logical (SOS:LogicalProcess) where a physical process would be essentially having a physical location or interface and a logical process do not. A sensing process has process composition describing its algorithmic details how outputs are made out of inputs. Therefore, a process has control structure (SOS:hasControlStructure) for linking with instances of SOS:AtomicProcess,
SOS:CompositeProcess, etc. A sensing process could implement (SOS:implement) machine learning techniques (SOS:MLT) for extracting features of from an input to create an output. v. Measurements and Observations
The measurement sub-domain is constructed for complementing sensor perspective. The main concepts in this sub-domain are SSN:Observation, SSN:FeatureOfInterest, and
SSN:ObservationValue. A sensor has an observation (SSN:Observation) representing a situation in which a sensing method (SSN:Sensing) estimates the value of a property (SSN:Property) of a feature of interest (SSN:FeatureOfInterest) where a feature is an abstraction of a real world phenomenon. An observation is formed from a stimulus (SSN:Stimulus) in a contextual event
(SOS:Event) which serves as input to a sensor. A sensor input is a proxy for a property of the feature of interest. An observation (SSN:Observation) should represent an observing property
(SSN:observedProperty), sensing method used for observation (SSN:sensingMthodUsed), quality of the observation (SSN:qualityOfObservation), observation result
(SSN:observationResult), and the time (CXT:Time) at which the sampling took place
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
137
(SSN:observationSamplingTime). An observation result is a sensor output (SSN:SensorOutput), which is an information object (SSN:SensorOutput ⊆ SSN:InformationObject). The information object represents a measurement construct for interpreting events, participants, associated result, etc., and signifies the interpretative nature of observing by separating a stimulus event from its potential multiple interpretations. SSN:SensorOuput has value (SSN:hasValue) for observation value (SSN:ObservationValue). The SSN:ObservationValue represents the encoding value of a feature and is indirectly depending on the accuracy, latency, frequency, and resolution of a sensor producing output. The observation value is annotated with location
(SOS:hasObservtionLocation), theme (SOS:hasObservationTheme), and time
(SOS:hasObservationTime) information for identifying a context (SOS:identifyContext). vi. Capabilities and Restrictions
Another complementing sensor sub-domain is the sensor measurement capabilities & operational restrictions. Sensors are integrated inside a smartphone, however, they are intended to be exposed and operated to provide best performance within the defined operating conditions
(SOS:OperatingCondition ⊆ SSN:Condition), which are categorized into device conditions
(SOS:DeviceCondition) and environmental conditions (CXT:EnviromentalCondition) of the atmosphere (i.e., SOS:Humidity, SOS:Temperature, SOS:Pressure, etc.) in a particular space and time. A sensor has inherent characteristics by design of measurement capabilities
(SSN:MeasurementCapabilities) that depends on the measurement properties (i.e., representing a specification of a sensor's measurement capabilities in various operating conditions) and directly affects a sensor's output. The measurement properties (SSN:MeasurementProperty) determine the behavior, performance, and reliability of a sensor. In addition, these measurement properties
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
138 determine the quality of quantifiable properties such as mass, weight, length, and speed related to a phenomena. These properties can be classified into accuracy (SSN:Accuracy), frequency
(SSN:Frequency), power consumption (SOS:PowerConsumption), random (SOS:Random) and systematic (SOS:Systematic) errors, settling time (SOS:SettlingTime), precision
(SSN:Precision), resolution (SSN:Resolution), etc. vii. Metadata
The metadata sub-domain is constructed to provide detailed descriptive information regarding origination, introduction, application, production, etc., of an object or a phenomenon that is of interest to a decision-making system. The metadata determines and affects the reliability and credibility of an object or a phenomenon. A vision of SensorML is that the schema should be self-describing and can be accomplished by accommodating metadata about a schema within the schema [174]. An object (e.g., sensor) would have metadata (SOS:hasSensorMetadata) that provides information about manufacturer (SOS:hasManufacturer), model information
(SOS:hasModel), serial number (SOS:hasSerialNumber), size (SOS:hasSize), etc. SOS:Metadata has explicit relationships with SOS:Identification, SOS:Note, and SSN:Design.
SOS:Identification provides information about recognition of a phenomenon including manufacturer, model, size, version, etc. Manufacturer perspective (SOS:Manufacturer) is constructed by providing necessary information about manufacturer of a smartphone, resource, platform, sensor, etc. SOS:Manufacturer is enriched with several object properties for describing a manufacturer that includes SOS:hasManufacturerEmail, SOS:hasManufacturerName,
SOS:hasManufacturerLocation, and SOS:hasManufacturerWebsite. Similarly, additional information about objects is provided by establishing explicit relationships with SOS:Model,
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
139
SOS:SerialNumber, SOS:Version, and SOS:Size. Similarly, SOS:Note provides a description about a phenomenon. The lifelog objects also has (SOS:hasLifelogObjectMetadata) that provides information about contents and compositions. For example, the 'SOS:SMS' has
'SOS:messageSendBy' to determine SMS sender, 'SOS:messageText' to determine SMS text, and
'SOS:messageTime' to show SMS date and time. viii. Time
The time sub-domain models knowledge about time such as temporal unit, temporal entities, etc.
This sub-domain is developed reusing the OWL Time ontology, in which TIME:TimeZone
TIME:CalendarClockDescription, TIME:TemporalUnit, etc., are made subclasses of CXT:Time.
Similarly, day, week, month, second, minute, hour, etc., properties of Time ontology are used to represent time information. SSN:TimeInterval is made subclass of CXT:Time to represent time duration of a phenomenon. ix. Contexts and Services
The contexts sub-domain is constructed for describing the application of sensors' generated observation values for identifying user contexts. Ontology-based context modeling will be helpful in formal and semantic enrichment and representation of complex context knowledge in order to share and integrate contextual information [173]. The context ontology from CoDAMoS project is reused for providing relevant core knowledge for SOS, which is extended with more detailed context types and properties. The SOS:Context is main concept in this sub-domain, which is classified into more specialized contexts including CXT:Activity, CXT:Environment,
SOS:Event, SOS:Device, CXT:User, CXT:Location, etc. Each of the sub-contexts represents a sub-domain, which could have more specific contexts such as CXT:Activity is classified into
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
140
SOS:Motional and SOS:Stationary. The SOS:Device sub-domain models knowledge about devices and includes a wide categorization of devices as well as their characteristics.
SOS:Environment sub-domain models knowledge about environment in terms of environmental conditions such as humidity, noise, light, temperature, etc. CXT:Location models knowledge about locations such as buildings, location coordinates, spatial entities, distance, countries, etc.
For more detailed location modeling, CXT:Location is linked with geonames:GeonamesFeature.
CXT:User (i.e., CXT:User ⊆ FOAF:Person) sub-domain models knowledge about users such as roles, profiles, preferences, tasks, projects, publications, socialization, etc. For more detailed user mappings, CXT:User is linked with FOAF:Person. SOS:Event sub-domain models knowledge about users' real-life events and includes a wide categorization of events using space, time, and other characteristics. CXT:Activity sub-domain models knowledge about users' motional and stationary activities and includes their characteristics. A context needs spatial (SOS:Space), temporal (CXT:Time), and thematic (SOS:Theme) information for its description. A context is identified by the observation value (SSN:ObservationValue) that depends on a sensor output
(SOS:SensorOutput). CXT:Service sub-domain fulfils the service-oriented requirement of the ontology by providing service-oriented features in ubiquitous computing. A service would be software that includes functionalities, which would be recognized and utilized based on identified context.. x. Lifelog Objects
The lifelog objects sub-domain is constructed around using concepts for modeling knowledge about lifelog items describing its types (i.e., SMS, email, etc.), events, and organization aspects.
The 'SOS:LifelogItem' class represent lifelog objects and is derived from the
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
141
'SSN:InformationObject' class. The lifelog object 'SOS:SMS', 'SOS:Email', 'SOS:VideoCall',
'SOS:AudioCall', etc., which are declared as subclasses of 'SOS:LifelogItem'. The lifelog objects are stored in a data container unit, therefore, 'SOS:DataContiner' is declared as subclass of
'SSN:InformationObject' class. The 'SOS:File' and 'SOS:Folder' class are declared as subclasses of 'SOS:DataContiner' to further refine the container objects. The 'SOS:LifelogEvent' is declared as subclass of 'SSN:Event' to represent a lifelog event. The 'SOS:LifelogEvent' is further sub- divided into 'SOS:SpatialEvent', 'SOS:TemporalEvent', and 'SOS:SpatialTemporalEvent' to specify type of a lifelog event. For example, 'SOS:Conference' and 'SOS:Meeting' are types of
'SOS:SpatialTemporalEvent', and 'SOS:AudioRecord', 'SOS:ImageCapture', and
'SOS:IncomingSMS' are types of 'SOS:TemporalEvent'. A number of object properties are declared to relate lifelog object with lifelog event and contexts. The 'SOS:hasEvent' object property relates lifelog objects with objects of event type. The 'SOS:callReceivedFrom' object property relates objects of 'SOS:IncomingCall' with objects of 'FOAF:Person'. The
'SOS:messageText' object property relates objects of 'SOS:SMS' with objects of 'SOS:Text'. The
'SOS:hasEventLocation' object property relates objects of 'SOS:LifelogEvent' with objects of
'CXT:Location'. The 'SOS:isContainedIn' object property relates lifelog objects with data container objects. The 'SOS:createdOn' and 'SOS:modifiedOn' object properties relates lifelog objects with objects of CXT:Time'. In short, a number of object properties are defined to relate and represent lifelog objects and events more semantically. d. SOS Concepts Hierarchy
SOS taxonomic class diagram, forming foundation of the ontology, is constructed from concepts, which are common and specific to smartphones, sensors, lifelog objects and context applications.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
142
The identified concepts are hierarchically arranged by determining their relationships that whether a concept is a sub-concept of another concept or not. The required classes, which are provided by either of the imported ontologies (e.g., SSN:Input, SSN:Output, CXT:Software,
SSN:Precision, etc.) are directly used in SOS and no explicit classes are declared to eradicate any type of ambiguity. Furthermore, classes in the imported ontologies representing the same semantics are declared as equivalent classes (e.g., SSN:Platform ≣ CXT:Platform). New classes are created explicitly either as parent classes or subclasses of the relevant classes in the imported ontologies or as per requirements. Figure 4.4 shows a snippet of SOS concepts hierarchy.
The SOS at present, contains 259 concepts, where each concept is formed by keeping in mind the unique needs and requirements of the domain. SOS extends SSN ontology by making
SOS:Smartphone ⊆ SSN:System, which means SOS:Smartphone is a kind of SSN:System.
Smartphone platforms (SOS:Halmet, SOS:Belt, SOS:SelfiStick, etc.), to which a smartphone can be attached during sensing process has unique characteristics and are made subclass of
(SSN:Platform ≣ CXT:Platform) to partially satisfy needs of a smartphone platform. The
SSN:Sensor concept is used to represent sensing devices for SmartOntoSensor to capture inputs and produce outputs. The SSN:Sensor concept is further divided into two categories
SOS:PhysicalSensor and SOS:LogicalSensor to represent hardware-based sensors and software- based sensors respectively in a smartphone. A SOS:LogicalSensor is formed by employing one or more of the SOS:PhysicalSensor for data capturing and producing a unique output, e.g., e- compass sensor is formed using accelerometer and magnetometer sensors. Real-world smartphone physical sensors such as SOS:Accelerometer, SOS:Gyroscope, SOS:Camera,
SOS:GPS, etc., are made as subclasses of SOS:PhysicalSensor and logical sensors such as
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
143
SOS:Magnetometer, SOS:Gravity, etc., are made as subclasses of SOS:LogicalSensor. The sensors hierarchy can be extended at any level to include more detailed and specific types of sensors. For example, accelerometer sensor can sense motion in the either 2-dimensions or 3- dimensions. Specifically, 1st and 2nd generation accelerometer sensors can detect motion in either of these categories while 3rd generation accelerometer sensors can detect motion across both categories as shown in Figure 4.5.The quantifiable properties that a sensor can measure
(SOS:measure) for a particular phenomenon is classified into physical quality
(SOS:PhysicalQuality) or logical quality (SOS:LogicalQuality). A sensor has type of operation, representing how a sensor would sense a stimulus that is either active or passive. Therefore,
SOS:ActiveSensing and SOS:PassiveSensing are made as subclasses of SOS:TypeOfOperation.
The concepts SOS:DeviceCondition and CXT:EnvironmentalCondition are declared as subclasses of SOS:OperatingCondition. The metadata concepts including SOS:SerialNumber,
SOS:Size, etc., are declared as subclasses of SOS:Identification, which would be used by
SOS:Metadata to provide necessary metadata about smartphones, sensors, and platforms. A smartphone sensor has to perform sensing operations, which could be either opportunistic or participatory. Therefore, SOS:Opportunistic and SOS:Participatory are made as subclasses of
SSN:Sensing class. Sensors differ by the amount of output produced where some could produce a single value outputs whereas others could produce triple value outputs. Therefore,
SOS:SingleOutput and SOS:TrippleOutput are made as subclasses of SSN:SensorOutput. An observation value produced by a sensor as output could be used to recognize SOS:Context, which is divided into subclasses as SOS:Event, SOS:Device, CXT:User, CXT:Environment, etc.
An identified context can start a CXT:Service, which would be software and is made subclass of
CXT:Software. In addition to all of the above, several concepts are explicitly identified and
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
144
Figure 4.4: Snippet of SOS concepts hierarchy.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
145
Figure 4.5: Snippet of SOS 'Sensor' class hierarchy. included in SOS including SOS:RAM, SOS:WiFi, SOS:CPU, SOS:GSM, SOS:Bluetooth,
SOS:NFC, SOS:Infrared, SOS:GPU, SOS:SDCard, etc, as subclasses of CXT:Hardware,
SOS:Random and SOS:Systematic as subclasses of SOS:Error, SOS:Calibration and
SOS:Maintenance as subclasses of SSN:Process, SOS:Theme, SOS:Space, etc., to provide a comprehensive set of information for improving inferencing mechanisms. Similarly, a number of concepts (SOS:SMS, SOS:Email, SOS:VideoRecord, SOS:VoiceCall, SOS:Picture, etc.) are declared to semantically model lifelog objects and events.
Apart from using subclass axioms, classes are coupled with other axioms to facilitate the creation of individuals (i.e., objects) unambiguously and semantically. For example, the disjoint axiom is defined for classes belonging to the same generation level to restrict individuals' behaviors such
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
146 as SOS:LogicalSensor and SOS:PhysicalSensor are disjointed to ensure that an individual can be an instance of any one of these classes but not both at the same time. e. SOS Properties and Restrictions
In OWL, properties represent relationships among classes. A property can be either object property or datatype property, either of which represents link(s) between instance(s) of domain with instance(s) of range. For an object property, both domain and range should be instances of concepts, whereas, for a datatype property domain should be an instance of a concept and range should be an instance of a datatype. Like concepts, exhaustive lists of object and datatype properties are identified from sources for using in SOS. SOS contains an extended list of object properties (i.e., 382) object properties, some of which are coming from the imported sources and others are explicitly created. As several of the SOS concepts are derived from the concepts in the imported ontologies, therefore, their object properties are also inherited in the same fashion and are made them more specialized for SOS concepts. The SOS object properties SOS:hasWiFi,
SOS:hasBluetooth, SOS:hasCPU, SOS:hasRAM, SOS:hasDisplay, etc., are made as sub- properties of the SSN:hasSubSystem for linking SOS:Smartphone with individual resource classes. Another SmartOntoSensor object property SOS:hasSmartDeployment is made as sub- property of SSN:hasDeployment to define relationship between SOS:Smartphone and
SOS:SmartPhoneDeployment. It is due to the fact that a Smartphone deployment has unique features and methods than the objects deployments in wireless sensor networks. As several concepts of the imported ontologies are used directly, therefore, their object properties are used similarly to avoid any confusion. The object properties SSN:hasInput and SSN:hasOutPut are used to represent input and output respectively of SSN:Sensing. Similarly, the object property
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
147
SSN:hasValue is used to represent the relationship between SSN:SensorOuput and
SSN:ObservationValue. Furthermore, like concepts, a bundle of domain-specific object properties are identified for SOS from the additional sources to relate concepts in the ontology in a more meaningful and subtle ways such as SOS:recognizeCotnext, SOS:hasProcessType,
SOS:organizedBy, SOS:deriveFrom, SOS:constructedFrom, SOS:hasLatency,
SOS:hasFrequency, SOS:hasSpace. A lifelog object has the object property SOS:hasEvent to determine event at the time of creation of a lifelog object. A lifelog event class has object properties SOS:attende to determine the people who have attended a lifelog event. Table 4.4 represents an excerpt of the SOS object properties along with their domains and ranges.
Apart from the object properties, SOS contains an extensive list of 136 datatype properties, which are identified and mapped to give comprehensive information about concepts such as
SOS:hasMemorySize, SOS:hasActiveSensing, SOS:modelScientificName, SOS:latitude,
SOS:eventCancelled, SOS:isConsumable, SOS:manufacturerEmail, SOS:hasBatteryModel,
SOS:minValue, SOS:value, SOS:hasMaxRFPower, SOS:nickName, etc. Table 4.5 presents an excerpt of the SOS datatype properties along with their domains and ranges. Classes in SOS are refined by using object properties and datatype properties to superimpose constraints and axioms for describing their individuals. An example of such constraints is the property restrictions (i.e., quantifier restrictions, cardinality restrictions, and hasValue restrictions) for describing number of occurrences and values of a property essential for an individual to be an instance of a class.
For example, for an individual to be an instance of SOS:Smartphone class, it is essential for the individual to have at least one occurrence of SOS:hasCPU object property for relating the individual with an instance of the SOS:CPU class. Table 4.4 and Table 4.5 also show excerpts of
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
148
Table 4.4: An excerpt of SOS object properties.
Domain Object Property Range Quantifier Cardinality SOS:Smartphone SOS:hasCPU SOS:CPU Exis. & Univ. Min 1
SOS:Smartphone SOS:hasOperatingSystem SOS:OperatingSystem Existential Min 1 SOS:Smartphone SOS:isPlacedOn SOS:Halmet | SOS:SelfiStick Universal Max 1 SSN:Sensor SOS:measure SOS:QualityType Exis. & Univ Min 1 SSN:Sensor SOS:produces SSN:'Sensor Output' Existential Exactly 1 SOS:PhysicalSens SOS:hasTypeOfOperation SOS:TypeOfOperation Existential Exactly 1 or SOS:PhysicalSens SOS:calibration SOS:calibration Universal Nil or SOS:LogicalSens SOS:constructedFrom SOS:PhysicalSensor Exis. & Univ. Min 1 or SSN:'Sensor SOS:relatedOutput SSN:'Sensor Output' Universal Nil Output' SSN:'Sensor SOS:withAccuracy SSN:Accuracy Universal Nil Output' SOS:Sensor SOS:hasObservation SSN:Observation Universal Min 1 SOS:QuatityValu SOS:hasUnit SSN:'Unit of Mesasure' Existential Exactly 1 e SOS:Metadata SOS:hasManufacturer SOS:Manufacturer Universal Min 1 SOS:Metadata SOS:hasVersion SOS:Version Universal Max 1 SOS:hasEnvironmentCondit SSN:Sensor CXT:EnvironmentalCondition Universal Nil ion SSN:Process SOS:hasProcessType SOS:ProcessType Existential Max 1 SSN:Process SOS:hasSubProcess SSN:Process Universal Nil
SSN:Sensing SOS:hasControlStructure SOS:ControlStructure Universal Min 1 SSN:Sensor SOS:recognizeContext SOSS:Context Universal Min 1 SSN:ImagCapture SOS:hasActivity CXT:Activity Existential Exactly 1 SSN:SMS SOS:messageSendBy FOAF:Person Existential Exactly 1
SOS:Context SOS:hasTheme SOS:Theme Exis. & Univ. Min 1 CXT:User SOS:hasPreferenceProfile SOS:PreferenceProfile Exis. & Univ. Nil ≣SSN:Person SOS:Event SOS:attende CXT:User ≣FOAF:Person Exis. & Univ. Min 1 CXT: Location SOS:hasFeatures geonames:GeonamesFeatures Universal Nil SOS:LifelogObjct SOS:hasLocation CXT:Location Exis. & Univ. Min 1
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
149
Table 4.5: An excerpt of SOS datatype properties.
Domain DataType Property Range Quantifier Cardinality
SOS:Manufacturer SOS:manufacturerName String Exist. & Univ. Min 1
SOS:SerialNumber SOS:serialNumber String Existential Exactly 1
SOS:Size SOS:weight Float Universal Max 1
SSN:'Sensor Output' SOS:isValid Boolean Existential Exactly 1
SOS:Smartphone SOS:hasIMEI String Existential Exactly 1
SOS:Smartphone SOS:isConsumable Boolean Universal Exactly 1
SOS:MemoryResource SOS:hasMemorySize String Existential Exactly 1 SOS:PowerSupplyRes SOS:hasPowerCapacity String Existential Exactly 1 ource SOS:Offset SOS:accXAxis Float Existential Exactly 1
SOS:Orientation SOS:yaw Float Existential Exactly 1
SOS:PersoanlProfile SOS:homePage String Universal Nil
SOS:ContactProfile SOS:phone String Universal Max 1
CXT:Location SOS:officialName String Exist. & Univ. Min 1
SOS:Image SOS:hasName String Existential Exactly 1
SSN:'Unit of measure' SOS:unit String Existential Exactly 1
SOS:DataContainer SOS:path String Existential Exactly 1
CXT:Humidity SOS:hasHumidityIntensity String Existential Exactly 1
CXT:Lighting SOS:hasLightIntensity String Existential Exactly 1
SOS:DataContainer SOS:hasName String Existential Exactly 1
SOS:ContactProfile SOS:mobile PositiveInteger Universal Nil
the property restrictions for the SOS object properties and datatype properties.
4.2.3 Evaluation and Discussion
Several approaches have been proposed and used for evaluating ontologies from the perspectives of their quality, correctness, and potential utility in applications. A detailed discussion about
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
150 ontology evaluation methods, metrics, and approaches is beyond the scope of this thesis, however, it can be found in [177]. However, the four main methods are gold-standard comparison, application-based evaluation, data sources comparison, and human- centric evaluation [175-177]. The gold-standard method advocates on comparison measurements of a well-formed dataset produced by a given ontology against other datasets. The application-based evaluation method evaluates an ontology using the outcomes of an application that is employing the ontology. The data sources comparison method describes the use of a repository of documents in a domain where ontology is expected to cover the domain and its associated knowledge. The human-centric evaluation method emphasizes on human efforts where an expert would assess the quality of a given ontology by comparing it to a defined set of criteria. The SOS is evaluated using the first and second methods. Similarly, several metrics have been defined for verifying and validating ontologies, where verification determines whether ontology is built correctly and validation is concerned with building the correct ontology [178]. The SOS is also verified and validated in the subsequent sub-sections using the relevant metrics. To determine coverage and scope of SOS, a top-level terminological requirements fulfillment (by providing concepts) comparison of SOS and other sensors and sensor networks ontologies is shown in
Table 4.6. A tick in the table indicates the capability of ontology to describe the stated aspect in some form and absence of a tick indicates either the absence or insufficient information of the aspects. a. Gold Standard-Based Evaluation
SOS is the first attempt to ontologically model smartphone sensory and lifelog data using the contextual information derived from the sensory data and other sources. Therefore, no
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
151 counterpart exists in the domain for comparison with SOS. However, a number of ontologies have been developed for sensors and sensor networks that include SSN [157], CSIRO [156],
OntoSensor [154, 174], and Coastal Environment Sensor Network (CESN) [179], which are mostly quoted in the literature. Therefore, SOS is compared with them to provide insights into its quality. Metrics and automated tools are defined for evaluating quality of ontology in some recent works such as OntoQA [180, 181]. OntoQA is feature-based ontology quality evaluation and analysis tool that has the capabilities of evaluating ontology both at schema and knowledge base levels. OntoQA uses a set of metrics to evaluate quality of ontology from different aspects including number of classes and properties, relationships richness, attributes richness, and inheritance richness. To evaluate the SOS quality, OntoQA is used. However, the evaluation and comparison is limited to schema level only because of the unavailability of knowledge bases of the competing ontologies. The overall comparison of SOS with the competing ontologies using
OntoQA schema metrics is shown in Table 4.7.
Table 4.6: Top-level terminological requirements fulfillment (using availability of concepts)
comparison of SOS, sensors, and sensor networks ontologies.
Ontologies
Criterion
Terminological
Requirements Top-Level Concepts
Category
Avancha, et al.[182] OntoSensor[ 154,174] CESN[179] Eid,et al.[168] CSIRO[156] SSN[157] ISTAR[183] Kim, et al.[184] SmartOntoS ensor System ✓ ✓ ✓ Sensor ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Base Terms Components/Resources ✓ ✓ ✓ Process ✓ ✓ ✓ ✓ Context ✓ System Terms Platform Hardware ✓ ✓ ✓ ✓ ✓
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
152
Software ✓ Deployment ✓ ✓ ✓ ✓ ✓
System Identification ✓ Metadata Manufacturer ✓ Power Supply ✓ ✓ ✓ ✓ Components CPU ✓ ✓ / Resources Memory ✓ ✓ Networking ✓ ✓ ✓
Sensor Physical Sensor ✓ ✓ ✓ ✓ ✓ Hierarchy Logical Sensor ✓
Type of Active Sensor ✓ ✓ ✓ Operation Passive Sensor ✓ ✓ ✓ Operating Condition ✓ ✓ ✓ ✓ History ✓
Sensor Identification ✓ ✓ ✓ ✓ Metadata Manufacturer ✓ ✓ ✓ ✓ ✓ Configuration ✓ ✓ ✓ Sensing & Process ✓ ✓ ✓ ✓ ✓ ✓ Sensor Terms Sensor Type Process Process Parameters ✓ ✓ ✓ ✓ ✓ ✓ Frequency ✓ ✓ ✓ ✓ Latency ✓ ✓ ✓ Accuracy ✓ ✓ ✓ ✓ ✓ ✓ Resolution ✓ ✓ ✓ ✓ Measuremen Measurement t Properties ✓ ✓ ✓ ✓ Range Precision ✓ ✓ Power ✓ ✓ Consumption Sensitivity ✓ ✓ Data/Observation ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Observation Response Model ✓ ✓ ✓ ✓ Terms Observation Condition ✓ ✓ Domain Terms Unit of measurement ✓ ✓ ✓ ✓ ✓
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
153
Feature/quality ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Sampled Medium ✓ ✓ ✓ ✓ Location ✓ ✓ ✓ ✓ ✓ ✓ ✓ Time ✓ ✓ ✓ Activity ✓ Context Terms Event ✓ User ✓ Service ✓ File ✓ Folder ✓ LifelogEvent ✓ SMS ✓ Image ✓ Lifelog Terms Email ✓ Document ✓ Text ✓ VidoRecord ✓ AudioRecord ✓
Table 4.7: Statistics of SOS and other sensors and sensor networks ontologies using OntoQA.
Relationships Inheritance Tree Class Ontology Classes Relationships Rank Richness Richness Balance Richness SSN [157] 47 52 59.09 2.4 1.76 66.18 IV
OntoSensor [154, 174] 286 219 46.39 2.63 1.66 59.37 II
CESN [179] 35 18 39.13 2.54 1.53 71.01 V
CSIRO [156] 70 70 65.42 2.84 1.36 39.65 III
SOS[159] 259 382 65.52 2.23 1.19 84.67 I
Using the information provided in [180, 181], the interpretation of results using schema metrics indicate that SOS has improved ontology design with rich potential for knowledge representation as compared to the existing ontologies. SOS provides enhanced coverage of its broader modeling
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
154 domain by having larger number of classes and relationships in comparison to existing ontologies. This also indicates that SOS is more complete by appropriately covering the problem domain by providing answers to almost any of the ontology domain-related questions.
OntoSensor also has shown tremendous classes and relationships measurements; however,
OntoSensor mainly describes the spectrum of sensors concepts (i.e., hierarchy of sensors classes and subclasses) and data. SOS, on the other hand, includes an extensive list of classes and relationships for describing broad aspects of smartphone systems, sensors, observation measurements, and context applications. The highest relationship richness value shows that SOS has diversity in types of relations in the ontology. Instead of relying only on inheritance relationships (usually convey less information), SOS contains diverse set of relationships to convey almost complete information about the domain. The slight richness in relationships of
SOS over Common Wealth Science and Industrial Research Organization (CSIRO) can be microscopically viewed large due to large number of SOS classes as compared to CSIRO. The increased number of relationships and relationship richness also advocate high cohesiveness of
SOS because of intensively relating classes in the ontology. The lowest inheritance richness value proves SOS as a vertical ontology, which is covering the domain in a detailed manner. In other words, SOS is concise by not having irrelevant concepts or redundant representations of the semantics regarding the domain. The large number of classes and relationships in SOS makes its slight difference in inheritance richness much bigger in comparison with other available ontologies. The lowest tree balance value indicates that SOS can be more viewed as a tree compared to others. The highest-class richness value of SOS indicates increased distribution of instances across classes by allowing knowledge base to utilize and represent most of the
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
155 knowledge in the schema. In other words, most of the SOS classes would be populated with instances as compared to the other. Comparatively, SOS is ranked the highest due to its larger schema size in terms of number of classes and relationships. OntoQA cannot directly calculate coupling measure of ontology. However, by using the coupling definition [177], SOS also shows high coupling by referencing an increased number of classes from the imported ontologies. b. Multi-Criteria Approach Based Evaluation
To evaluate semantics and understandability of terms used in ontology, researchers have not established any common consensus on widely-accepted methods (i.e., tools and metrics) in computer science (i.e., due to relatively being a new field) so far. However, to evaluate ontology terminologies along with its popularity, an approach comprising of several decision criteria or attributes is defined by [185] where numerical scores can be assigned to each criterion. The overall score is calculated by the weighted sum of its per-criterion scores [175]. An objective formula for terminology evaluation and popularity measurement (shown in equation (3)) has been used from [185] that is based on the objective multi-criteria matrices defined in [186].
푂푏푗푒푐푡푖푣푒 = 퐼 ∗ 푤푖 + 퐶 ∗ 푤푐 + 푂 ∗ 푤표 + 푃 ∗ 푤푝 (3)
In equation 3, I show interoperability and is 퐼 = 푁/푇 where T is total number of terms in ontology and N is the number of terms found-able in WordNet. C is clarity and is 퐶 =
(∑ 1/퐴푖)/푁 where 퐴푖shows the number of meanings of every interoperable term in WordNet, then clarity of each term is 1/Ai, O represents comprehensiveness and is 푂 = 푇/푀 where M is the number of terms in the standard term set of the domain represented by the ontology. P denotes popularity and is 푃 = 퐸/퐻 where E denotes the number of access of the ontology and H
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
156
is the total number of access of all the ontologies in the same domain. The weights wi, wc, wo, wp represent the weight of interoperability, clarity, comprehensiveness, and popularity respectively with condition that these weights satisfy the equation wi+ wc + wo + wp = 1. WordNet is a large lexical database of English words. WordNet links words by their semantic relationships and has been used by several of the ontology development researches for terminology definition, analysis, and filtering such as [181]. Using the guideline from [181], WordNet is used for the evaluation of terms interoperability, clarity, and comprehensiveness, and literature citation index is used for evaluation of popularity. During analysis, multi-words terms in SOS are broken into individual words and stemmed to receive more accurate metrics values. Results obtained by
[185] using the same formula are extended by the inclusion of SOS results and shown in Figure
4.6. Analyzing results indicates that SOS has acceptable levels of terms interoperability and comprehensiveness, medium clarity, and lowest popularity. Overall, SOS shows reasonable analysis results, signifying superiority over the others due to rich set of terms (i.e., classes and properties) but slightly less than OntoSensor due to lowest popularity value. However, the results are calculated manually, where the chances of errors in the process cannot be ignored. c. Accuracy Checking
To demonstrate the correctness and utility of SOS, accuracy checking is performed to determine that the asserted knowledge in the ontology agrees with the experts' knowledge in the domain.
Ontology with correct definitions and descriptions of classes, properties, and individuals will result into high accuracy [177]. Recall and precision rates are the two primary Information
Retrieval (IR) measures that are used for evaluating the accuracy of ontology [177]. Recall and precision rates are defined and shown in equations (4) and (5) respectively [168].
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
157
Figure 4.6: Objective analysis of SOS using multi-criteria approach.
푁푢푚푏푒푟 표푓 푟푒푙푒푣푒푛푡 푖푡푒푚푠 푟푒푡푟푖푒푣푒푑 푅푒푐푎푙푙푅푎푡푒 = ( ) (4) 푇표푡푎푙 푛푢푚푏푒푟 표푓 푟푒푙푒푣푎푛푡 푖푡푒푚푠
푁푢푚푏푒푟 표푓 푟푒푙푒푣푒푛푡 푖푡푒푚푠 푟푒푡푟푖푒푣푒푑 푃푟푒푐푖푠푖표푛푅푎푡푒 = ( ) (5) 푇표푡푎푙 푛푢푚푏푒푟 표푓 푖푡푒푚푠 푟푒푡푟푖푒푣푒푑
Accuracy of SOS is determined by computing high recall and precision rates for the functional requirements (represented in competency questions) by executing the SPARQL queries on the
SOS knowledge base. To form knowledge base, SOS is instantiated with relevant information from USC Human Activity Dataset (USC-HAD), manuals, reports, and documentations using
Protégé 4.3. Based on these instances, SPARQL query language is used through Protégé
SPARQL Query plug-in to query knowledge base for retrieving relevant results. The scenarios
(built using competency questions) used for proof-of-concept assume utilizing low-level sensory
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
158 data of heterogeneous sensors for mapping them to high-level queries. For example, a possible source of detecting and monitoring signatures of human fall is tracking vector forces exerted during a fall and location changes. Therefore, mapping fall to a concept, this could be determined by accelerometer, gyroscope, and GPS sensors through relationship SOS:isDetectedBy can enhance human fall detection. The scenarios used for proof-of-concept includes: (1) using low- level sensory data to detect a user's context and automatically initiate a respective service (e.g., lifelogging application), and (2) using microphone and GPS low-level sensory data to search locations having high noise pollutions. The actual running SPARQL queries for each of the scenarios are shown in Figure 4.7 and Figure 4.8 respectively.
Figure 4.7: SPARQL query for retrieving locations having noise intensity greater than 65dB.
Using assistance of low-level sensory data, the testing queries resulted into acceptable precision and recall rates by retrieving relevant data (i.e., all of the locations having greater sound levels,
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
159 and all of the sensors having measurement capabilities of acceptable levels). Therefore, it has been observed that SOS is of potential effectiveness of integrating heterogeneous sensory data to answer high-level queries.
Figure 4.8: SPARQL query for detecting a context and service using low level sensory data. d. Consistency Checking
OWL-DL is used as the knowledge representation language for SOS content. A significant feature of the ontologies described using OWL-DL is that they can be processed using reasoners
[168, 187]. Using the descriptions (conditions) of classes, a reasoner can determine whether a class can have an instance or not. Using the methodology proposed by [186] to validate
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
160 consistency of ontology, SOS is passed through two major tests: (1) sub-sumption test to check whether a class is a subclass of another class or not, and (2) logical consistency check to see whether a class can have any instance or not. Fact++ 1.6.2 and RacerPro 2.0 reasoners are used because of their strong reasoning capabilities and interoperability with Protégé [168]. Both of the reasoners are fed with manually created class hierarchy (called asserted ontology) to automatically compute an inferred class hierarchy (called inferred ontology) by using the descriptions of classes and relationships. The inferred ontology is compared with asserted ontology and found that both of the class hierarchies match and none of the classes is inconsistent. However, the classes are classified and repositioned by the reasoners in case of either having many super-classes or subjected to some logical constraints. Therefore, both of the tests are significant and SOS is logically consistent and valid. Figure 4.9 depicts a snippet of the
SOS asserted and inferred class hierarchies after reasoning. e. Application Based Evaluation
Ontologies can be plugged into applications, which would largely influence outputs of an application for a given task. Therefore, ontologies can be simply evaluated by analyzing results of the using applications. Using the architecture published in our previous research work [159], we have developed a prototype application called ModeChanger. A smartphone mode defines the number and nature of features, resources, and services available to consume at a time instant.
Modern smartphones’ operating systems support several operation modes that can be explicitly adjusted by users to define a smartphone’s behavior according to a context. Currently, the available modes are airplane/flight mode, normal mode, audio mode, and brightness mode. The context-dependent mode changing can benefit from manipulating and accessing context
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
161 information. The application is not a lifelogging application; however, it is a proof-of-concept that demonstrates on using the SOS for extracting contextual semantics from sensory data and ease human-phone interaction by adjusting smartphone modes using the derived contextual information. A smartphone-based lifelogging application using the SOS is presented in the
Chapter 5.
Figure 4.9: SOS asserted and inferred class hierarchies after using reasoner.
The prototype application is aimed for Android-based smartphones running with Ice Cream
Sandwich 4.0.3 or higher and is developed in Java programming language using Android SDK tools revision 22.6.3 with Eclipse IDE and SensorSimulator-2.0-RC1 running on a desktop machine. The target code is deployed and tested on Samsung Galaxy SIII running with Android
Jelly Bean 4.1.1 operating system. The application runs inconspicuously in the background as a service and utilizes SOS to deduce contextual information using low-level sensory data, and automatically adjusts modes by invoking low-level services. The fuzzy logic controller is used to
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
162 direct the overall adjustment process according to contexts. To adjust a smartphone's behavior, the application adjusts audio volume, screen brightness, font size, vibration, and default write disk according to the identified context. Table 4.8 lists a few hypothetical exemplary contexts and their corresponding modes values. The application is tested closely and found that the application can successfully differentiate between different user contexts in real-time by mapping low-level sensory data into high-level contexts and trigger services accordingly. Figure
4.10 presents screen shots of ModeChanger changing modes per changing contexts.
Table 4.8: Hypothetical exemplary contexts and their corresponding modes values.
Audio Context Screen Brightness Font Size Vibration Write Disk Volume
Location:House High Normal Normal No Phone Storage
Location:Office Normal Normal Normal Yes SD Card
Location:Meeting Silent Bright Normal Yes SD Card
Activity:Sitting Low Low Small No Phone Storage
Activity:Walking Normal Normal Normal Yes Phone Storage
Activity:Running High Bright Extra Large Yes Phone Storage
4.2.4 SmartOntoSensor and Lifelogging
As discussed in Chapter 3, the ontological modeling of lifelogging and contextual information can potentially overcome the problems associated with database and file storages. In the real world, lifelog information are related with each other in multiplicity of semantic ways which cannot be exactly represented by the databases. SOS encompasses constructs (i.e., classes, properties, and axioms) not only to represent and store low-level sensory data and high-level
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
163 contextual information, but also lifelogging information (e.g., calls, SMS, pictures, documents, etc.). SOS is a lifelogging semantic model that provides lifelog objects/events semantics by relating lifelog information (i.e., events) and contextual information in similar to human episodic memory in a graph structure. Lifelogging systems can use SOS for extracting contextual semantics from sensory data, and semantically annotate and relate the captured lifelog information in multiple ways as they exist in the real world. This will allow for semantic-based reasoning on the captured lifelog data and associated metadata for deducing new semantics and relationships among lifelog objects. In addition, the ontology can enhance retrieval of lifelog information for augmenting memory in real-time by allowing users to concisely express their queries and obtain precise answers using semantics of the contained data and queries. By linking lifelog information in multiplicity, the SOS also provides associate trial by allowing users to retrieve a collection of related lifelog information using a memory cue. Furthermore, SOS provides a flexible model to add new types of lifelog events, lifelog objects, and contexts that may arise with the passage of time. Lifelog information can also be enriched by exploiting the potential of SOS of linking and extracting data from data sources in the LOD Semantic Web stores to provide more and more semantics to lifelog information.
4.3 Summary
In this chapter, we have explored the design and development of ontology-based smartphone sensors and lifelog information repository referred to as SmartOntoSensor (SOS). Our previously published research article that is related to the design and development of SOS [159] has been included in this chapter as part of the thesis. As discussed in Chapter 2, lifelogging is a special type of context-aware computing. Therefore, SOS provides a semantic model to map low-level
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
164 sensory data into high-level contextual information and give more semantically enriched organization and definition to lifelog information. To the best of our knowledge, this is the first effort of its kind in the area of smartphone sensors ontology for lifelogging to unify smartphone, sensors, lifelog information, and contextual information with general world knowledge about entities and relations.
Figure 4.10: Changing modes by ModeChanger using contextual information from SOS.
SOS is developed using NeOn and POEM ontology development methodologies with clear understanding of design rationales and goals, collection of ontology requirements specification, design and alignment of ontology, and ontology evaluation. SmartOntoSensor includes definitions of concepts and properties that are partially extended from more general and comprehensive SSN ontology, partially influenced from SensorML, and partially identified from the relevant literature. SOS provides descriptions about smartphone systems and deployments, sensors and their classification, sensors measurement capabilities and properties, observations as properties of features of interests, inputs and outputs to sensing processes, context identification
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
165 using sensors outputs, invoking services according to contexts, and lifelog objects representation.
SOS is developed and evaluated using state-of-the-art technologies and standards to demonstrate its utility and value. The test results indicate that SOS has an improved ontological design for semantic modeling of the domain and is more complete by providing rich potential for representing information about the domain to answer ontology-related questions. Using quality and efficiency of SOS, it could have other potential applications in other areas as well such as augmented reality, smart environment, smartphone-based lifelogging, etc. In the next chapter, we will propose a smartphone-based semantic lifelogging framework that integrates SOS along with other components. We will also present the practicality and usability of SOS in the real world smartphone-based lifelogging applications.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
166
Chapter 5 : Proposed Framework and
Implementation
In Chapter 3, we discussed that smartphone technology has not only improved in hardware but also offers novel programming features by supporting libraries for frameworks such as Semantic
Web. In Chapter 4, we presented the semantic model (i.e., SmartOntoSensor ontology [159]) for mapping low-level sensory data into high-level contextual semantics and interlinking lifelog objects using the contextual semantics for lifelogging. The semantic model, however, provides a data structure, therefore, should be used in combination with other software components to provide the desired functionality. To exploit smartphone sensors data semantics for semantically annotating and relating lifelogging data in a holistic semantic model, this chapter presents a smartphone-based semantic lifelogging framework. In the framework, we have proposed a methodology to exploit smartphone and Semantic Web technologies for developing a semantically enriched lifelogging solution. The framework contains a set of components for capturing, relating, and storing lifelog information in the semantic model. In addition, it also enhances semantic interpretation and retrieval of lifelog information to support a number of use- cases.
This chapter is aimed to fulfill the fourth objective of this thesis. First, we will describe technical architecture and distinguishable characteristics of the proposed framework. Second, we will describe prototypic implementation of the proposed framework and their empirical evaluation from different aspects to prove practicality and effectiveness of the proposed framework. The
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
167 knowledge obtained in previous chapters and research publications is used in designing and defining of the proposed framework to provide enhanced lifelogging experience. In addition, we will also describe two general services to demonstrate retrieval and using of the lifelogging information from the semantic model. The general services will prove usability of the proposed framework in a variety of smartphone-based lifelogging use-cases.
5.1 Framework Architecture
As discussed in Chapter 2, the lifelogging systems in general and smartphone-based lifelogging solutions in specific have no common architectural consensus. They have been designed for specific purposes and have not taken into account the technical (e.g., total capturing, annotations, and interlinking of lifelog information) and conceptual (e.g., forgetting to capture) requirements of a digital memory [21]. In addition, they are not storing and relating lifelog information like it exists in the real world. Some of the earliest lifelogging systems have considered using of
Semantic Web technologies for semantically archiving lifelog on PC such as [79] and [188].
However, these solutions are desktop-based and uni-faceted, and unable to handle the different facets of an individual's life. However, the smartphone-based lifelogging systems constitute considerable knowledge, which can be used for rationalizing generic and flexible smartphone- based semantic lifelogging architecture. The architecture will rationalize capturing totality of life experiences and will organize lifelog information in a format to support multiple use- cases. The proposed framework architecture can be used as a reference model, which can be equally implemented on top of the different smartphones operating systems such as Android, iOS,
Windows Mobile, etc., if the required technologies are available for the platform. It has several distinguishable characteristics including its generality for usage in several lifelogging
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
168 applications. The smartphone-based lifelogging systems consider data collection and data analytics operations separated from each other where smartphone is used for data collection and data analytics (if any available) is performed on the backend powerful servers. Unlike these, the proposed framework architecture considers data collection and data analytics as inseparable components. Therefore, combines all of the smartphone-based lifelogging steps (i.e., proposed in the generalized architecture in Chapter 2) in a single holistic structure and device [14]. First, low-level smartphone sensory data is collected and processed for extraction of high-level contextual and metadata information. Second, the configurable modular approach is used to support and employ Semantic Web technologies for data analytics and organization, where lifelog data is mapped and related in a semantic model for meaningful organization, and advanced querying and explorations. The propose framework architecture is the compact form of the architecture published in our previous research work [159] and is composed of three layers: data layer, semantic layer and application layer (as shown in Figure 5.1).
5.1.1 Data Layer
Data layer represents the personal information (PI) space of a user. The information resources representing lifelog objects (e.g., pictures, documents, contacts, emails, calls, etc.), applications used to create and manipulate the lifelog objects, and sensors used to provide contextual and environmental data about lifelog objects, collectively comprises a user's PI space. The PI space is used by the architecture for instrumentation, automation, and querying. In the context of the architecture, the notion used to read data from a user's PI space is sensor, where sensor is an artifact (i.e., autonomous tool, device, application etc) to read and record data from a measurable phenomenon in a user's PI space. The lifelog data can be of several types; therefore, it is not
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
169 possible to precisely determine the exact number and type of sensors to provide complete set of information about a user's experiences from his/her PI space.
Figure 5.1: Layered view of the proposed architecture.
To make the architecture more flexible and configurable for accepting a broad range of sensors, we have extended the sensor classification criterion and guidelines suggested by [21]. We have
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
170 classified sensors for lifelogging into several categories (defined in the semantic model in
Chapter 4), as shown in Figure 5.2. However, it is hard to draw a fine line of differentiation between the sensors in different categories. There is strong possibility of sensors might be belonging to different categories but could be used for the same purpose. For example, WiFi and
GPS are inherently different sensors; however, they both can be used for user localizations. The prototypic implementation of the architecture supports sensors of all of the proposed categories except environmental sensors. A compact overview of the sensors categories is presented in the following subsections.
Figure 5.2: Sensors classification in the proposed semantic framework for lifelogging.
• Environmental Sensors: Environmental sensors (also called proximity sensors [21]) are
external sensors that are distributed and located in the proximity of a user. In today's
world, users are surrounded by different types of smart sensors in their environments,
which are capturing different types of information about users' behaviors and actions, and
they are expected to increase in the future. Kim et. al, [189] have developed a gadget that
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
171
can be located in a user environment and connected to sensors attached with daily
appliances such as mug, and bed. In a smart home, daily appliances (i.e., microwave own
systems, heating systems, ACs, TVs, audio & video systems, security system, lighting,
etc.) are integrated with sensors which can communicate with each other and can be
controlled locally or remotely using smartphone. This category also includes sensors that
are either directly attached to body or embedded in body worn devices (e.g., smart watch)
and carried by the users ubiquitously. A typical example of such sensors is biosensors
that can be attached to a user body for reading information about his blood pressure, heart
rate, body temperature, and physical activities. The sensors fabricated in environment or
on a user body have potential for lifelogging. For example, security camera can be used
to capture videos of users' activities in proximity, microphone sensor can be used to
record information about users' voice conversations and calls, and blood pressure sensor
can be used to record users' blood pressure information at a specific instance of time. The
captured information can be forwarded to smartphone using a networking technology
(e.g., Bluetooth). However, environmental sensors would not work form a user point of
view such as a wall-mounted camera will point at a user but not pointed by a user [10].
• Smartphone Sensors: Smartphone sensors are integral components (i.e., hardware and
software) of smartphone, as discussed in Chapter 3. Smartphone sensors are inherently
ubiquitous and can be helpful in capturing totality of life experiences because of the fact
that humans' lives are not restricted to desktop computers and most important life events
occurs outside of the desktop computers [190]. Support from the general-purpose
operating systems and availability of development toolkits have extended the concept of
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
172
sensors beyond physical components in smartphone. Intelligent programs can be
developed to capture information about various facets of a user's life and interactions that
are not possible with physical sensors, otherwise. Therefore, the notion of smartphone
sensors is extended into context sensors, informational sensors, and application sensors.
Each of these sensors has specific applications and provides a varying amount of data.
Apart from the principle values, these sensors can also provide several useful values (e.g.,
Bluetooth can provide Bluetooth ID, Bluetooth name, etc.), which can be potentially used
for useful functions (e.g., detecting people/object in a proximity and user localizations).
Today's smartphone contain context sensors in its jacket for capturing rich contextual
information. A detailed discussion of smartphone context sensors their potential
lifelogging applications are presented in Chapter 3. Analogously to context-aware sensors
[191], we have also classified context sensors into physical, virtual sensors, and logical
sensors. Physical sensors are hardware components that are physically integrated in a
smartphone circuitry such as accelerometer, and GPS. Logical sensors are software
components that can uses/fuses information from several physical sensors and other
information sources (i.e., virtual sensors) to infer new information such as sleep pattern
sensor, and e-compass sensor. Virtual sensors refers to software services that monitors,
reads, and records information about users' activities on their smartphones such as
applications usages, documents used, web pages browsed, online activities, emails
sent/received, calls log, media usages, SMSs sent/received, file exploration, etc.
• Informational Sensors: Informational sensors (also called desktop sensors) refers
information sources (applications) that contain information about user activities and
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
173
event, which can be feed into a lifelogging system such as calendar, and PIM
applications. The informational sensors can provide valuable information to increase
semantics of lifelog objects/events and contextual information.
• Application Sensors: Application sensors represent a wide variety of applications that
can be used to produce lifelog objects. A wide variety of smartphone applications are
developed for separate purposes such as SMS, voice calls, contacts, pictures, documents,
weather reporting, audio/video recordings, etc. Apart from creating lifelog objects,
application sensors also attach and store certain metadata (e.g., date and time, email
address of a contact, received/dialed numbers, etc.) with lifelog objects, which could be
effectively used to recognize and understand context of lifelog objects. Furthermore, the
metadata can be used to relate separately created lifelog objects to portray complete
picture of an event.
As discussed in Chapter 3, sensors in lifelogging can be applied for capturing lifelogging information either on event-based or pulling-based [41]. In event-based capturing, sensors record a measurable phenomenon in response to occurrence of an event such as user location information would be recorded when a picture is taken or a phone call is received. In pulling- based capturing, a sensor will periodically record a measurable phenomenon. The event-based capturing is suitable from the perspective of smartphone resources conservation but can result in not capturing of some important information. However, the problem is cured in the architecture implementation by first checking for running of the prototypic application before initiating an operation (e.g., sending/receiving of SMS, and dialing/receiving of phone call) and notifying user accordingly. The pulling-based capturing can be approximation of capturing totality of life
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
174 experiences but can result in excessive consumption of smartphone resources. In the proposed architecture, a mixed approach is applied where resources greedy sensors (e.g., GPS) are used on event-based basis and others are used on pulling-based basis.
5.1.2 Semantic Layer
The semantic layer adds semantic glue to the architecture. Most of the lifelog data captured via data layer components (i.e., sensors) are of raw nature and cannot be used by the applications.
Therefore, semantic layer transforms and organizes the low-level sensory and lifelog data into a semantic model that can be used by the applications for retrieval and visualization. The semantic layer is a composite layer consisting of several individual sub-layer called services and each service has a specific designated task. The services are executed independently; however, they interact with each other by reading input and forwarding output. Some of the services are extended from the existing lifelogging applications (e.g., UbiqLog) and implemented with semantic glue, whereas, others are included as per novel requirements. a. Collection Engine
The collection engine is a composite service that acts an interface between low-level sensors and lifelog data, and its semantic structuring. It constitutes of several components for sensors and applications configurations, contextual and metadata extractions, and aggregating and associating data into a standard exchangeable format. The sensors configuration provides primitives for adjusting sensors as per lifelogging need such as turning sensors ON/OFF, setting sensors sampling rate, setting sensors reading time interval, and reading sensor data using event- based or pulling- based. The apps configuration provides primitives for adjusting installed applications as per lifelogging needs such as turning applications ON/OFF to capture lifelog
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
175 objects or not. For example, if camera application is turned OFF then images captured via camera will not be included in lifelog. These configurations will not only save wastage of smartphone resources (i.e., battery power, and memory) but also ensure user control on the lifelogging process. The context and metadata extractor implements data acquisitor, context recognition, metadata extraction, and data aggregation & association. The data acquisitor define methods for reading data from sensors using the information from sensors configuration. Either an explicit data acquisitor can be defined for each sensors or a single data acquisitor can read data from multiple sensors in parallel. In the prototypic implementation of the architecture, single data acquisitor is used, which also acts a temporary buffer to hold sensors data until read by the context recognition. The context recognition processes the sensors data to recognize contextual information of a lifelog event/object. For example, latitude and longitude values of a location can be transferred to an online service for finding location related information such as location name, postal code, known name, and country name.
The metadata extractor extracts metadata information from lifelog objects attached by the parent applications For example, extraction of sender name, sender phone number, SMS text, data time, etc., from a received SMS. The context and metadata information about a lifelog object/event are stored in a consistent structure (i.e., JavaScript Object Notation (JSON)) temporarily until read by the data aggregation and association. Figure 5.3 and Figure 5.4 are showing information about a received SMS and its location respectively. The data aggregator and association collects contextual and metadata information of a lifelog object/event from the diverse sources (i.e.,
JSON files) and cluster them into a single consistent structure to define and relate information with each other. Information from other sensors (e.g., calendar, and phone book) can also be
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
176 contacted for enriching data of a lifelog object/event. For example, phone book can be used to retrieve information (i.e., name, email address, address, etc.) about a contact of incoming call or outgoing call. For each event, calendar entries will be inspected using time information. If an entry is found, the event will get title from the calendar entry otherwise will get a title by combining date/time and location information. Similarly, a meaningful title is also associated with a context by combining contextual information and date/time. The clustered information is structured in a standard exchangeable format for use by the other components and for ease of transmission to remote places. In the implementation of this architecture, data is structured in
JSON format, which is more flexible and requires less disk space as compared to XML. Figure
5.5 represents information about a receiving SMS event, clustered in JSON.
Figure 5.3: Location context information of the received SMS object in JSON format.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
177
Figure 5.4: Metadata information extracted from a received SMS object in JSON format. b. Mapping and Semantic Rules
The mapping and semantic rules implements a web service interface between collection engine and ontology catalogue. The web service interface module processes the well-formed JSON information from the collection engine and extracts information to create ontological components using the users' defined mapping and semantic rules for gaining the desirable behavior. The two approaches used for mapping high-level features in ontology are dedicated and generic [26]. The dedicated approaches are rules-based and emphasize on direct mapping of low-level features into high-level concepts belonging to different domains. These approaches are simple but require definition of new mapping rules for each new concept. The generic rules are automatic and use enriched general-purpose vocabularies (i.e., WordNet) or domain-specific ontologies for detecting concepts. These approaches show satisfactory results but require sophisticated machine learning algorithms and enough annotation training data for effective operations [192]. In the implementation of the architecture, dedicated approach is used because of its potential usage for a variety of lifelog objects/events.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
178
Figure 5.5: Clustered information about receiving SMS object and location context in JSON
format.
The rules for processing incoming JSON message are divided into two distinguishable sets: mapping rules and semantic rules. The rules use data from the incoming JSON message to store lifelog objects/events by creating individuals in the ontology, and annotating and relate them by enriching their datatype and object properties. Following the analogy of [41], the rules are formed in event-condition-action pattern where event is arrival of message, condition is the occurrence of specific data in the message, and action is operation to be taken in response. The synthetic structure of the rule is:
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
179
on event if condition then action
The mapping rules fetches data from the JSON message and enriches ontology by creating individual(s) of the class(es). For example, two individuals (i.e., one of SMS class and one of
Location class) will be created in the ontology using the JSON information given in Figure 5.5.
In addition, by using JSON message, the mapping rules also describe values of the datatype properties of the created individual(s), as per the ontology schema. For example, the created individual of Location class will have datatype properties of latitude, longitude, address, country, etc., and their values in the ontology using the JSON information given in Figure 5.5. An excerpt of mapping rules for an incoming JSON message from our previous research work [159] is shown in Figure 5.6.
Figure 5.6: An excerpt of mapping rules for an incoming JSON message.
The semantic rules enrich ontology by defining relationships between individuals within ontology to portray complete picture of a lifelog object/event. The explicitly created individuals
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
180
(i.e., created by the mapping rules) will be related with each other. The semantic rules describe semantic relationships between the explicitly created individuals of classes in the ontology, as per the ontology schema. For example, individual of the SMS class in the ontology will be semantically related (i.e., using 'hasLocation' object properties) with individual of the Location class in the ontology. An excerpt of semantic rules for an incoming JSON message from our previous research work [159] is shown in Figure 5.7.
Figure 5.7: An excerpt of semantic rules for an incoming JSON message. c. Semantic Data Storage
The semantic data storage is a composite service providing operations for ontology storage and management. As discussed in Chapter 3, representing lifelog information ontologically has several advantageous characteristics over conventional methods. As discussed in Chapter 4,
SmartOntoSensor [159] presents a semantic data model (i.e., ontology) for smartphone-based semantic lifelogging. This service provides constructs to store and manipulate the
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
181
SmartOntoSensor. As discussed earlier, to model a lifelog object/event, relevant concepts in the ontology are needed to be created and semantically related to provide complete set of information using the rules defined in mapping and semantic rule. For example, an excerpt of the semantic model created for an incoming SMS event is shown in Figure 5.8, where ellipse nodes labeled with URI references shows individuals and labeled boxes shows literal values and labeled arrows with URI references shows relationships.
In the implementation of architecture, the ontological data model will be stored in AndroJena internal graph engine. AndroJena is a subset of open source Apache Jena Semantic Web framework for Android platforms. AndroJena framework provides a set of features for working and storing ontologies, reasoning and inferencing ontologies, and querying ontologies.
AndroJena uses Apache Jena feature and is capable of working with ontology data using standard formats in resources-constrained smartphone. The notion of triple statements is used for storing ontological data model which forms underlying graph of the model. Triple stores are like databases that store RDF triples and can be queried using SPARQL. In addition to storage, the key feature of triple stores is their ability of inferencing. To date, AndroJena native triple store is the only choice for smartphone Semantic Web based projects to store RDF data [193]. A knowledge base is composed of ontology and a reasoner. Automatic reasoning on ontology classifies concepts and instances and checks consistency of the model and knowledge base [194].
Most of the semantic reasoner are implemented in Java but cannot be directly applied on
Android platforms due to no alignment of Dalvik with Java SE. Semantic OWL API 3.4.3 can be converted to Dalvik without any modification and can be imported into an Android project directly. Therefore, native reasoner facility of AndroJena is used for ontology inferencing.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
182
Figure 5.8: Excerpt of the semantic model created for a SMS event.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
183 d. Ontology-Based Retrieval
Ontology-based retrieval provides mechanism for querying ontology to retrieve relevant data for satisfying an application's lifelog information requirements. In the implementation of this architecture, querying feature from AndroJena is used. SPARQL is the query language like SQL that is developed exclusively by W3C Data Access Working Group for querying ontologies.
SPARQL query engine is provided as an API which can be used by applications and servers implementations to be able to add SPARQL querying capabilities. This will release developers from the worrying about query specification and implementing it themselves. AndroJena framework includes SPARQL query engine to interpret and execute SPARQL queries against ontology for information retrieval. ARQoid porting is SPARQL query engine for AndroJena which can be used in Android projects. e. Lifelog Manager
In addition to lifelog semantic model, a lifelog dataset would consist of lifelog objects, which could be binary files (i.e., photos, videos, audios, and calls) or text files (i.e., documents and
SMS). The lifelog dataset grows with the passage of time as more and more lifelog objects and information are captured and stored. The smartphone storage technology has shown tremendous advancements in the past few years but is still limited for storing a huge amount of lifelog information. In addition, smartphone has the possibility of being lost. Therefore, lifelog data has to be stored on a reliable storage media which could be on a user's personal computer or on server on the cloud. This uploading of lifelog dataset can be done either manually or automatically. The manual upload is cumbersome and requires more efforts as more and more lifelog objects are captured. Therefore, automatic upload can be a better alternative. The lifelog
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
184 manager service periodically checks the maximum size of lifelog dataset. If it has reached to its maximum size, the service will automatically upload the lifelog dataset to an already defined reliable storage media. Once the lifelog dataset is successfully uploaded, the service will remove the uploaded lifelog objects from the dataset in the smartphone local storage. f. Privacy and Security
The privacy concerns to the lifeloggers, society, and bystanders would be more highlighted as smartphone-based lifelogging applications become more in mainstream. Certainly, the increased capturing using smartphone can be tailored with increased privacy and security implications.
Privacy and security in smartphone-based lifelogging is puzzling with no clear definition. The architecture suggests that smartphone-based lifelogging systems should be developed by integrating privacy and security concerns in the development process. Privacy by design framework [195] is suggested for ubiquitous computing where privacy and security is considerably embedded in the entire development process of a technology, from the early design stage to the deployment and use. Despite of complexities [9], the architecture advocates incorporation of privacy by design in the smartphone-based lifelogging applications development. However, this is beyond the scope of the thesis and not included in the prototypic implementation.
5.1.3 Application Layer
The application layer is the interface between users and the system. The application layer is typically composed of lifelogging applications, and mapping and semantic rules definitions. A lifelogging application would be a smartphone-based memory aid system, which would provide features for lifelog information management, searching lifelog information, and creating
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
185 narratives by visualizing and browsing lifelog information from the semantic model using time and date, event, location, activity, and any combination of them. The architecture provides a composite system, which separates the application knowledge from the operational logic.
Therefore, it can be used by the developers for developing any type of lifelogging application.
The mapping and semantic rules can be either provided as separate application or integral part of a lifelogging application. As lifestyles changes, new types of lifelog objects emerges, and new types of data sources (i.e., sensors and applications) can be built and developed. Therefore, new mapping and semantic rules are needed to be defined with the passage of time.
5.2 Proposed Framework Features
The proposed framework is a logical extension of the UbiqLog framework [15] and attempts to build a digital prosthetic memory of precious moments and personal experiences in similar to human organic memory. It also helps in defining new use-cases and methodologies for retrieving required and associated lifelog information in a timely and meaningful manner. The semantic storage of multimodal sensory data and large volume of lifelog information is helpful in opening new opportunities of designing and building tools for lifelog big data analytics to achieve various benefits. Comparatively, the proposed framework has several distinguishable features and guidelines that are lacking in the related smartphone-based lifelogging solutions. In the following sub-sections, we present a compact overview of the proposed framework features.
5.2.1 Openness and Extendable
The proposed framework follows a layered architecture and each layer is composed of several components. The functionality, responsibility, and communication of each layer and each component in a layer is clearly defined. The layered and component-oriented architecture has
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
186 several advantageous characteristics including: (i) clear separation of functionalities improves understanding and implementation; (ii) improves versatility and extendibility by allowing easy addition of new features and functionalities (i.e., components); and (iii) improves error detection by curtailing errors to a specific section and not propagating to the other sections. The proposed framework is extendable and flexible. It extends UbiqLog [15] and increases flexibility and intelligence by allowing users to add/remove and customize components according to their needs and desires (e.g., applications, and sensors reading rates, etc.). These features also increase users' control on the lifelogging process, which is a major issue. For example, users can define mapping and semantic rules for describing individuals, relationships, and annotations of lifelog information in the semantic model. The proposed framework can be extended with plug-in extension interface service for enabling third party developers to develop and integrate new components or modify existing components. However, this feature is not covered in the prototypic implementation.
5.2.2 Increased Sensors Coverage
As discussed earlier, the proposed framework emphasizes on increased sensors coverage to produce a wide range of contents and contexts information for increasing quality of the lifelog information. The proposed framework extends the notion of sensors in the lifelogging process and includes informational sensors and application sensors and virtual sensors in addition to physical and logical sensors, as discussed earlier. We claim that metadata information captured through virtual, informational, and application sensors are as important as the contextual information captured through physical and logical sensors. As discussed earlier, the proposed framework also supports fusing data from available sensors to deduce new data which is not
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
187 possible by and individual sensor. The always growing nature of a lifelog information makes manual annotation increasingly infeasible and cumbersome. Therefore, the proposed framework attempts to use multitude sensors and automatically extracts fine-grained semantics from the different types of sensors data to improve and facilitate the automatic annotation process.
5.2.3 Semantic Lifelog Information Organization
The continuous recordings of a user's life experiences increases lifelog dataset size exponentially and emerges the issue of lifelog information management and retrieval. To date, the smartphone- based lifelogging systems uses RDBMS technology to solve the challenges of indexing and storing lifelogging information and associated metadata challenges such as MyLifeBits [17].
However, as discussed in Chapter 2, RDBMS is not appropriate technology for lifelogging due to its certain limitations. Similarly, lifelog information can have broadened scope and can be linked with related information for using in external applications. However, storing data in an application's native format affects its sharing. It also worth noting that dependency on specific devices, applications, and storage technologies also effects flexibility of lifelogging systems. It is arguable, that data in different sources will be valuable for supporting a person's memory and applications if integrated and linked semantically into a single lifelog [196]. The successful aggregation, extraction of meaningful events, and retaining interoperability among heterogeneous information sources can be realized using Semantic Web technologies [41]. As discussed earlier, the proposed framework formulates lifelog contents, contextual information, and metadata in a semantic model (i.e., ontology) for advance conceptualization, querying, exploration, and connecting with other lifelog information. We claim that ontology based lifelog data modeling is more flexible due to supporting the addition of new kinds of lifelog information
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
188 and semantic relationships between them, modifications of the existing lifelog information and relationships, and covering a wide range of data sources. As discussed in Chapter 4,
SmartOntoSensor [159] maps lifelog information in a semantic structure as in human episodic memory. This close resemblance will enable users to retrieve required items of interests and develop associate trails as they do in their episodic memories. The semantic model allows for querying and reasoning on the captured lifelog information and metadata deducing new semantics and relationships between entities in the model.
5.2.4 Improved Annotation
Annotation is the creation of metadata. All of the lifelogging solutions understand the importance of annotations and attaches extra information with lifelog information for different purposes such as indexing, clustering, retrieval, visualization, and bookmarking. However, they vary in the amount of annotations used and mostly enrich lifelog information with spatio-temporal contextual annotations. As a rule of thumb, the greater the number of annotations, the more the lifelog information will be meaningful. However, restricting annotations to spatio-temporal information is not enough because lifelog information have several inherent data annotations in the name value pairs format, which are essential for its semantics understandability. For example, a received SMS lifelog item has data annotations of sender name, sender phone number, and SMS content. In addition, lifelog information would not only have data annotations but would also have object annotations for linking them with related lifelog information to convey complete meanings. To the best of our knowledge, the on-hand lifelogging solutions consider data annotations only and none of them consider object annotations. A semantic data model provides a comprehensive semantic data structure that would allow for defining both
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
189 types of annotations in as much detail as they exist in the real world. For example, a captured picture lifelog information will have data annotations and object annotations for connections with event and location and time information, location information will have data annotations and object annotations for connections with activity and time information, and event will have data annotations and object annotations for connection with same location and time information, a depiction of annotations graph is shown in Figure 5.8. The proposed framework assumes recording of sensory data without or with little user intervention (e.g., turning sensors ON/OFF, using camera, etc.) and automatic annotations by automatic extraction of annotations data from sensory data and mapping/adding in the semantic data structure accordingly. As discussed in
Chapter 1, lifelogging systems are a sub-category of context-aware systems. Context-aware data can be either external or internal [197]. In the context of the proposed framework, sensory data is external and annotation data is internal. However, the framework can be extended for manual annotations by adding components such as tagging feature for allowing users to associate data with lifelog objects according to their knowledge.
5.2.5. Easy and Fine-grained Retrieval
As discussed in Chapter 3, state-of-the-art smartphone has several sensors, which are likely to increase in the near future. The proposed framework support using of multiplicity of sensors and information resources which can produce a variety of memorable annotation information for recalling/retrieval of even less memorable lifelog events. For example, remembering the weather information (i.e., cold) or other contextual information (e.g., receiving call from a friend) can be helpful in recalling name of the coffee shop and the menu enjoyed that a user has visited a month ago. Gommell et al. [17] have expressed the importance of excessive supporting information for
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
190 retrieving required lifelog information and eventually memory augmentation. The semantic organization enables applications to create associative trail to retrieve required piece of information in similar to human organic memory.
5.2.6 Improved User Modeling and Personalization
Increasing and controlling the number and quality of data sources in a user PI space can improve the quality of lifelogging information for effective and accurate user modeling. The careful increase and control of data sources will produce more lifelog objects and associated metadata information (i.e., contextual information) that will be rich enough to analyze for user modeling.
User modeling systems like personalization applications and recommender systems can get benefits by analyzing a lifelog dataset constituting of more information about a user's past environments, activities, behaviors, locations, etc.
5.2.7 Improved Surveillance and Sousveillance
Surveillance is the application of technologies for recording individuals' life experience and activities on the behalf of an administrative authority and sousveillance is the digital capturing of life experiences for self-surveillance and self-insight. Both of the fields get benefits from extensive capturing of individuals' life experiences for answering "what", "where", "who",
"when", "why", and "how" questions related to their past activities. Technically, combination of information captured from multiple information sources is more effective and precise for a given situation as compared to a single information source. Furthermore, defining rules to fuse information from multiple information sources can produce novel information about users' emotional states and behaviors, which are not possible from a single information source. The configuration and management of information sources and definition of rules features of the
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
191 proposed framework can provide a broader range of lifelog information for better self-awareness and surveillance. For example, using of accelerometer, ambient light, microphone, and Bluetooth sensors can provide information that can be correlated and cross-referenced to decide an individual's sleep states, patterns and locations.
5.2.8 Multiplicity of Use-Cases
The proposed framework is a general-purpose lifelogging framework providing users the opportunity to record and semantically organize lifelog information in detail from all possible information sources. To date, none of the smartphone-based lifelogging solutions have presented such in-depth and well-organized approach. On one hand, the proposed framework provides extensibility and flexibility features. On the other hand, the proposed framework empowers developers to develop any type of lifelogging application by retrieving and utilizing the required lifelog information from the underlying semantic model. Thus, making the proposed semantic framework as a multipurpose smartphone-based semantic lifelogging framework. For example, the framework can be configured to record and utilize users' health related information in a health monitoring system in one scenario and the framework can be configured to record, monitor, and report old-age people fall detection in another scenario.
5.3 Semantic Data Model
In Chapter 4, we have presented a detailed discussion of the semantic data model
SmartOntoSensor [159] ontology) developed for lifelogging. However, in this section we present a conceptual definition of the semantic data model to make it more understandable and align with the proposed framework. The conceptual definition of the semantic data model will enable users to understand, organize, retrieve, and share their lifelog information.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
192
Graph data structure is very common in Computer Science for modeling real-world systems. In the context of Semantic Web, a semantic model represents information in a real world domain where information could be inter-related in multiple ways. Therefore, the semantic model developed is a directed graph. However, the semantic model also constitutes several additional information and characteristics. Therefore, in set theory, the semantic data model used to represent a user (U) lifelog dataset SL(U) is of six tuples SL(U) = {C, O, R, CH, rel, OA}. C is a set of concepts (also called classes) in the semantic data model where a concept represents a set of objects about which information are needed to be recorded in the model. C is of 6-tuple C =
{Csm, Cse, Ccx, Cenv, Cev, Csp}, where Csm represents a set of concepts about smartphone (e.g., hardware, software, size, manufacturer, deployment), Cse represents a set of concepts about sensors (e.g., accelerometer, GPS, camera, time), Ccx represents a set of concepts about contexts
(e.g., physical activities, and location), Cenv represents a set of concepts about environment (e.g., humidity, temperature, and ambient light), Cev represents a set of concepts about lifelog events and information objects (e.g., pictures, videos, documents, and SMSs.), and Csp represents a set of supporting classes for describing access control, and security aspects of lifelog information. O is a disjoint set with C (i.e., C ∩ O = Ø) and represents a countable infinite set of objects instantiated from the concepts and represented by a URI reference. An object is a physical occurrence of a concept such as picture taken is an instance of the image concept in the model. R is a disjoint set with C and O (i.e., C ∩ R= Ø and C ∩ O = Ø) and represents a set of relations/properties. R is 2-tuples R= {Rd, Ro} where Rd and Ro are disjoint sets (i.e., Rd ∩ Ro = Ø) and represents annotations that are attached with concepts for their descriptions. Every relation
(Ri) has an associated set of domains ∆ [Ri] and set of ranges ◊[Ri]. Domain represents start of a
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
193
relation and range represents end of a relation. Domain of relation (Ri) is always URI reference
(i.e., ∆ represents a countable infinite set of URI reference of objects) of an instance of a concept and range can be either URI reference of an instance of a concept or a value (also called literal) of primitive data type (i.e., ◊ represents a finite set of values which is disjoint with ∆]. Rd is a set of data type relations/properties connecting object of a class with a literal and Ro is a set of object relations/properties connecting object of a class with object of another class or to itself with in the same model or with object in other accessible Semantic Web knowledge stores such as
Friend of a Friend (FOAF), for enriching lifelog information with more relevant information.
However, in implementation of the proposed framework Ro is used within the same model. CH represents concepts hierarchy or taxonomy where concepts are arranged hierarchically in parent child relationships (CH ⊆ C ✕ C). CH(Ci, Cj) would mean that Ci is sub-concept (child concept) of Cj or Cj is super-concept (parent concept) of Ci, in other words. rel is a set of functions that relates concepts with data and other concepts non-taxonomically depending on the type of relation (R → C ✕ C | C ✕ data ). OA represents a set of ontology axioms described in ontology description language using description logic. OA is 3-tuples OA = {Cc, Pc, Or}. Cc is a set of axioms used for the definition of classes' constructors such as describing of subclass, equivalent class, disjoint class, etc. definitions. Pc is a set of axioms used for defining characteristics of properties/relations such as describing of transitivity, functional, inverse of, etc. characteristics.
Or is a set of axioms used for restricting instantiating objects (individuals) of classes using their properties such as quantified restrictions, cardinality restrictions, and value restrictions. Inference engine uses OA to deduce new knowledge using the existing knowledge and may restructure the semantic structure defined by a user.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
194
The semantic data model representing lifelog dataset enable users to model and relate their lifelog information using RDF statements in subject, predicate, and object (S, P, O) format.
Subject is the URI reference of a data entity object, predicate representing annotation is the URI reference of a property, and object can be either URI reference of a data entity object or a literal.
The model can be represented in any of the RDF serialization format. A detailed discussion on
RDF serialization formats can be found in our previous research work [166]. The model is designed to be generic and implementable on any smartphone platform that is supporting
Semantic Web technologies. It is complete enough to represent excessive lifelog data entities and flexible enough allowing users and developers to define and manipulate lifelog data entities.
5.4 Proof-of-Concept Prototyping
To demonstrate the practicality of the proposed framework, we have developed a proof-of- concept prototypic application called Semantic Lifelogging (SLOG). The proposed framework is, however, too much extensive and cannot be completely implemented due to limitations of resources and time. Therefore, a proof-of-concept implementation (SLOG) is demonstrated in this section to fulfill objective of this thesis. The SLOG is developed using the design considerations and lessons learned in this thesis and our previous research work [100]. The current implementation of SLOG simulates app's operations by providing primitives for context and content collections, customizations, configurations, semantic organization, visualizations, and transitions between menus and views/screens to prove objectives of this thesis. However, detailed implementation is left aside as future work. For example, SLOG do not include definition of semantic and mapping rules and some rules are hard coded to validate the proof-of- concept. In addition, SLOG ensures full users' control on the lifelogging process by determining
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
195 information to be included in their lifelog datasets but removing of undesired information is not provided. Furthermore, detailed privacy and security preservation is also not supported in the
SLOG. However, the current implementation of SLOG is enough for quantitative evaluation and establishing conclusions. In addition, two additional services (i.e., MyLocations and MySMSs) are also developed to demonstrate modularity and applicability of the proposed semantic framework in different real world use-case scenarios.
5.4.1 SLOG Implementation
The SLOG is implemented as an Android application. However, it can be implemented on any smartphone OS, if the required technologies (e.g., libraries and APIs) and resources (e.g., sensors) are available. Reasons of choosing Android includes the availability of the required technologies and resources, open nature of the platform, and huge market share, as discussed in
Chapter 3. SLOG is primarily targeted for smartphones, however, it can be equally implemented on Android powered tablets, and smart TVs. Due to having similarity with UbiqLog [21] in some aspects, SLOG has reused open-source code from UbiqLog8 with certain modifications and enhancement as per new requirements of the proposed framework. For example, online Web services are used to retrieve additional information about a location (i.e., address, and country name) using latitude and longitude values from GPS sensor. However, the additional components
(e.g., informational sensors) are explicitly implemented and used. For example, retrieving metadata from an image captured by smartphone camera application. In addition, the functionalities claimed by UbiqLog [21] but not implemented are also implemented in SLOG such as uploading lifelog dataset to remote web server. The development of SLOG took place in
8 https://github.com/Rezar/Ubiqlog
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
196
Android Studio 2.2.3, using Android Software Development Kit (SDK) with the Android API
Level 21 (Android 5.0 Lollipop) and higher. The SLOG implementation is tested using Samsung
Galaxy Note V; however, the one delivered to participants for experiments is based on API Level
19. To avoid any type of incompatibilities (e.g., screen densities), the application is developed to support both of the API Levels.
The Graphical User Interface (GUI) of the SLOG is shown in the Figure 5.9. The UbiqLog [21]
GUI components have been extended to provide maximum features and functionalities to the users to easily customize/configure and retrieve required lifelog information easily. The GUI components from UbiqLog are used where applied otherwise they are explicitly developed as per need. For example, SLOG has introduced email sensor to automatically read metadata from an incoming email and create object(s), annotation(s), and relationship(s) in the semantic model accordingly. However, to access email account, email ID and password are needed to be provided. By click on email settings in the settings menu, a view will be displayed for entering user's email ID and password, as shown in Figure 5.10. Similarly, to cure forgetting event capturing problem, before initiation of an event SLOG state is first checked (i.e., running or not) and user is informed accordingly, as shown in Figure 5.11.
In SLOG, sensors are implemented as services and broadcast receivers in the
'lifelogging.semantic.smartphone.SLOG.Sensors' package, which requires no explicit user interface. The SLOG extends the idea of UbiqLog by providing interface for enabling users to adjust/customize all types of sensors by setting their different properties (i.e., status ON/OFF, scan interval, record interval, unit, annotations, etc.), which also ensures user control on the lifelogging process. The information are stored in a SQLite database to facilitate the startup pro-
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
197
Figure 5.9: GUI screenshots. (1) main user interface that appears when the application is started; (2) list of sensors that are implemented by the application; (3) list of installed applications that are supported by the application; (4) list of services/operations that are provided by the application; (5) enabling/disabling and setting reading interval of implemented sensors; (6) enabling/disabling of supported installed applications; (7) start/stop video lifelogging; (8) start/stop audio lifelogging; (9) start/stop and setting voice call recording.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
198
Figure 5.10: Screenshots of email ID and password setting. cess. The information is used by the 'startRecording' and 'stopRecording' methods of the 'Engine' service class in the 'lifelogging.semantic.smartphone.SLOG.supporting' package to start and stop sensors accordingly. Some of the sensors' services (e.g., accelerometer, Bluetooth, temperature, etc.) runs in the background and are started automatically when SLOG is started using the stored information (i.e., by clicking the 'Click here to start SLOG' button on the main user interface), whereas, other sensors services can be started explicitly by clicking sensor menu item from the sensors list in the 'Sensors' tab. A running sensor service can also be stopped in the same way.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
199
Sensors implemented as broadcast receivers (e.g., SMS, call, and camera) are related with events, which are started and stopped according to an event (e.g., arrival of SMS). In addition, using knowledge from Chapter 3, the non-battery power greedy sensors services run as long as the SLOG runs and greedy sensors services are activated and stopped with the occurrence of an event. A sensor service can also activities another sensor service. For example, with occurrence of receiving or sending of SMS event, SMS sensor activates GPS sensor to record location information, temperature sensor to record environmental temperature, etc.
Figure 5.11: Checking of SLOG status before starting an event.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
200
A sensor code implements Android built-in listeners (e.g., 'SensorEventListener' for sensor service and 'PhoneStateListener' for call broadcast sensor) and includes all of the coding necessary for capturing sensor data, extracting semantics from the sensor data, extracting relevant data from other sources, etc. However, sensors having no Android built-in listeners, are activated after regular intervals to read fresh information such as recording information about currently running applications. SLOG can be extended to support dynamic addition of new sensors; however, this feature is not supported in the current implementation. A new sensor can be created by creating a new class in the 'lifelogging.semantic.smartphone.SLOG.sensors' package, which is activated after rebuilding framework and installing .APK file on smartphone.
The annotation classes are defined in the 'lifelogging.semantic.smartphone.SLOG.annotation' package to help sensors in finding relevant contextual data to enrich sensors' data.
The collected data is passed to corresponding method in the 'SensorJSONCreation' class in the
'lifelogging.semantic.smartphone.SLOG.JSON' package to write captured data in a structured format into JSON file. The data from the JSON file is write into a temporary data buffer by
'DataAcquisitor'" class in the 'lifelogging.semantic.smartphone.SLOG.collectionengine' package.
A separate thread in the 'DataAcquisitor' class runs for each of the sensor that continuously flushes previous data and writes incoming data into buffer after each 5 seconds. The
'DataAggregator' service class in the same package is used to combine data from separate
'DataAcquisitor' buffers into a single composite JSON file. Distinguishable feature of SLOG is storing and organizing lifelog and contextual information in a semantic model (as discussed in
Chapter 4). Therefore, the JSON file is read by 'SemanticMapper' service class in the
'lifelogging.semantic.smartphone.SLOG.ontology' package. The methods in the class
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
201 corresponds to the mapping and semantic rules (i.e., for each rule a separate method is defined that calls each other accordingly) of the proposed semantic framework. The 'OntologyFill' class is created in the same package. The class provides methods to load ontology and create objects, annotations, and relationships in the semantic model.
Figure 5.12: SLOG folder 'SLOGDATA' and sub-folders. The SLOG has used AndroJena Semantic Web framework library. The methods in the
'OntologyFill' class use classes and methods from the AndroJena framework library to perform ontology related operations including loading ontology, instantiating objects, establishing relationships, and retrieving information from the semantic lifelog model. The AndroJena framework library also provides primitives for retrieving information from the semantic model using SPARQL queries to satisfy applications' needs and inferencing new knowledge using the
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
202 knowledge already available in the semantic model. The SLOG maintains a separate data structure by creating a folder 'SLOGDATA' in the SDcard to organize lifelog dataset by storing lifelog objects files and ontology in .OWL file. The folder has sub-folders for each of the lifelog object type (i.e., pictures, videos, audio calls, ontology, etc.), as shown in Figure 5.12. In addition, SLOG also stores lifelog data in a set of text files to support key-word searching and pattern mining, etc.
In SLOG, the lifelog manager service uploads lifelog objects dataset to a secure remote web server using HttpClient package of the Android instead of personal computer. The remote web server is used to ensure omnipresent access to the lifelog objects and information. The complete server-side is not implemented in the current implementation of SLOG. However, two simple proof-of-concept Java Servlets9 are implemented: one for reading and storing lifelog dataset on remote web server and another for retrieving lifelog information from remote web server. The lifelog manager is implemented as a service that periodically monitors lifelog folder size, if it has reached to a threshold or not. If the size has exceeded threshold, the uploading process is started automatically. Once the process is successfully completed (i.e., by receiving HTTP Response from the Java Servelt), the lifelog manager deletes the lifelog objects from the folders in the
SDcard and updates the semantic lifelog model accordingly.
Lifelog application can query the semantic lifelog model (i.e., via SPARQL10) to retrieve required lifelog information according to a use-case. The ontology retrieval is handled by the methods in the 'ontologRetreival' class in the 'lifelogging.semantic.smartphone.SLOG.retreival'
9 https://docs.oracle.com/javaee/6/tutorial/doc/bnafd.html 10 https://www.w3.org/TR/rdf-sparql-query/
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
203 package. Methods in the 'ontologyRetreival' class receive and structure SPARQL query from a lifelogging applications, use methods in the 'ontologyFill' class to load ontology, use classes and methods from AndroJena to execute query on the loaded ontology, and provide query results to the requesting lifelogging application. To demonstrate retrieval of lifelog information from the semantic model and usability of the SLOG and proposed semantic framework, two lifelogging applications are discussed in the following subsections.
5.4.2 Location Lifelogging
As discussed in Chapter 3, location is a primitive context that is not only useful in memory reconstruction but also in correct and flexible responses to different types of lifelog information retrieval queries. Location Lifelogging is a small proof-of-concept application that helps users in retrieving location information of previously visited places. As discussed earlier, in smartphone- based lifelogging, location information can be captured in different ways including GPS, cellular network, WiFi, and Bluetooth. However, in the current implementation of SLOG, GPS is the primary primitive used to capture location lifelog information. SLOG understands the importance of location information in lifelogging and checks for GPS sensor status at start-up and prompts user for turning GPS sensor ON if it is OFF.
SLOG provide primitive for continuous capturing of location information. However, excessive use of GPS sensor will not only produce excessive location information, which would be of little use but will also result in high battery power consumption (as discussed in Chapter 3). A wise use of GPS sensor is its association with events. Therefore, location information would be stored when an event occurs such as capturing a picture, receiving/sending SMS, receiving/dialing voice call, and turning ON audio/video lifelogging. MyLocations is a proof-of-concept service
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
204 that helps users to retrieve and visualize locations information that are logged in the semantic lifelog model due to visiting places, activities, events, etc. It provides a simple user interface by allowing user to define a search criterion and display retrieved locations information on map.
As discussed earlier, in semantic lifelog model, relevant information can be related in multiple semantic ways. Therefore, the semantic lifelog model can be queried (i.e., using Simple Protocol and RDF Query Language (SPARQL)) in multiple ways to retrieve the required location information and other related information. However, MyLocations in its current implementation supports only time information (i.e., user can define a time-period) to construct SPARQL query.
An example SPARQL query to retrieve location information from the semantic model using time information is shown in Figure 5.13. Executing query will display location information within the defined time-period as markers on the map, as shown in Figure 5.14. Complex user interfaces can be constructed by clicking a marker to display relevant lifelog information such as calls/SMS, pictures, voice/video recordings, and activities. However, in the current implementation, clicking a marker would only display location information such as address, and country name. This type of service can have applications in different fields with high social impact such as emergency responding, urban planning and development, and transportation.
5.4.3 SMS Lifelogging
SMS is one of the prime and most common users' activities on smartphones and billions of SMSs are sent and received across the world each year. A SMS typically contain a short message and other associated data, which potentially constitutes a significant portion of a person lifelog. The sending or arriving of SMS is an event, which has semantic relationships with contextual information (e.g., location, and activity) and other lifelog information (e.g., person, image
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
205 captured, and call received), as shown in Figure 5.8. Therefore, SMS lifelog information can provide memory cues to recall specific lifelog information (e.g., recalling of a person's name whose SMS was received in a wedding party to create an associative memory trail).
The SLOG provide primitive for capturing SMS and related information. The SMS sensor is an event sensor and implemented as a service, which is invoked by a broadcast receiver. The broadcast receiver is activated when SMS receiving or sending event occurs in Android and starts the SMS sensor service. The service extracts information from the SMS (e.g., SMS text, sender name, sender number, date, etc.) and starts the other services to capture related contextual and environmental information such as location sensor, and activity sensors. Once the collection process is completed, the SMS sensor service and other sensor services are stopped, which will be started again when another SMS event take place. The collected lifelog data is mapped into the semantic model as discussed in the previous sections. MySMSs is a proof-of-concept service that helps users in retrieving and visualizing SMS information from the semantic lifelog model.
Figure 5.13: SPARQL query to retrieve location information using time information.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
206
Figure 5.14: Screenshots of starting MyLocations service and displaying locations on map.
It provides a simple user interface by allowing user to define search criteria and display retrieved
SMS information graphically. In MySMSs, the semantic lifelog model is queried (i.e., using
SPARQL) to retrieve the required SMS information and other related information. However,
MySMSs in its current implementation supports only time information (i.e., user can define a time-period) to construct SPARQL query. An example SPARQL query to retrieve SMS
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
207 information from the semantic model is shown in Figure 5.15. Executing query will display SMS information received/send within the period as markers graphically, as shown in Figure 5.16.
Complex user interfaces can be constructed by clicking a marker to display relevant lifelog information such as SMS text, calls received/dialed, voice/video recordings, event title, and activities. However, in the current implementation, clicking a marker only displays basic SMS information only (i.e., sender/receiver name). This type of service can have potential applications in memory reconstruction such as recalling information from previous communication.
Figure 5.15: SPARQL query to retrieve SMS information using time information.
5.4.4 Evaluation
As discussed in Chapter 2, that no widely agreed evaluation methodologies have been proposed by the researchers to evaluate smartphone-based lifelogging systems, to-date. The qualitative analysis for precise estimations of resources utilizations (i.e., CPU, memory, battery power, etc.) has been proposed as important indicator for evaluating smartphone-based lifelogging applications [15]. However, none of the on-hand smartphone-based lifelogging researches have
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
208 reported resources utilization evaluation in the available literature except UbiqLog. This could be due to the fact that these smartphone-based lifelogging solutions are prototypes and not complete applications. In addition, no widely accepted tool(s) are available to evaluate resources utilization estimations of smartphone-based lifelogging systems. In addition, relying on automated tools cannot be complete idea because of not guarantying the return of precise estimations [15]. For example, an automatic tool is reported in the UbiqLog literature for estimating resources utilizations but tool is not available for testing and reusing in other researches.
Figure 5.16: Screenshots of starting MySMSs service and displaying SMS information.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
209
Taking it further, resources utilization estimations can be an aspect of smartphone-based lifelogging systems evaluation and cannot provide a complete set of information to determine effectiveness of a system [40][100] . Moreover, identifications of methods and metrics for the qualitative evaluation is beyond the scope of this thesis. As discussed earlier, SLOG constitutes broad range of components and operations as compared to UbiqLog. Therefore, its resources utilization estimations will be greater than UbiqLog. However, an objective top-level feature- wise comparison of SLOG with smartphone-based lifelogging systems is shown on Table 5.1 to provide key indicators about the quality. As SLOG provide extensive coverage of sensors, and contents and context; Therefore, SLOG has significant improvements over the others.
As SLOG is the proof-of-concept implementation of the proposed framework. Therefore, its qualitative analysis for accurate resources utilization estimations is not possible at this stage.
However, to fulfill objective of this thesis, effectiveness of the proposed semantic framework methodology and SLOG can be evaluated by performing quantitative analysis. The quantitative analysis is performed by conducting an empirical study. The empirical study is composed of experimental tasks for evaluating effectiveness of the proposed semantic framework methodology and SLOG from the different aspects: (i) simplicity, (ii) usability, (iii) semantics,
(iv) organization, (v) availability. Simplicity is defined in learnability, user-control, operability, and naturality. Usability is defined in efficiency, desirability, satisfaction, and multiple use-cases.
Semantics is defined in lifelog information coverage, context and metadata extraction, improved annotations, and logical constraints. Organization is defined in information structure, concepts and relationships, flexibility, and fine-grained retrieval. Availability is defined in omnipresent access and swift retrieval of the required lifelog information.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
210
Table 5.1: Qualitative Comparison of SLOG with Smartphone-Based Lifelogging Systems.
System 1* 2* 3* 4* 5* 6* 7* 8* 9* 10* 11* Pictures, videos, Nokia Lifeblog Time, Location, Object PC - GPS, Camera, metadata SMS, MMS, Automatic No SQLite Distributed No No NA [73] Name, Phone Number Timeline notes, blogs PC - iRemember [42] Microphone Time Audios Automatic No NA** Distributed No No NA Timeline Location, Time, Semi- PC - Web Pensieve [4] Camera, Microphone Pictures, Audios No NA Distributed No No NA metadata Automatic UI Location, Time, Experience GPS, WiFi, GSM, Semi- PC - Neighbourhood, Pictures No MySQL Distributed No No NA Explorer [12] Bluetooth, Camera Automatic Timeline Keywords Accelerometer, GPS, Pictures, Mobile Lifelogger Semi- WWW- Camera, Microphone, Location, Time Audios, No NA Distributed No No NA [1] Automatic Timeline WiFi, Rotation Activities GPS, Bluetooth, Camera, Location, Time, WWW - MemoryBook [79] Pictures, text Automatic No RDF Distributed No No NA metadata Neighbourhood Timeline Semi- Smartphone UbiqLog [15] NA Location, Time SMSs, Pictures No File Integrated Yes Yes NA Automatic -Timeline SoundBlogs [80] Microphone, GPS Location, Time Audios Automatic No NA Distributed Smartphone No No NA Smartphone SenseSeer [14] NA NA NA NA No Cloud Distributed No No NA , WWW Location, Smartphone Digital Diary [77] GPS, Camera, Infrared Pictures, Audios Automatic No SQLite Distributed No No NA Neighbourhood -Timeline GPS, Bluetooth, Location, Time, Pictures, Microphone, Camera, Activity, Audios, Videos, Smartphone Temperature, Humidity, Neighbourhood, Calls, SMSs, SLOG Automatic Yes Ontology Integrated - Multiple Yes Yes High Email, Calendar, Sleep, Environment, Documents, use-cases WiFi, GSM, Proximity, Communication, Audio, MMS, Files, etc. Video, Biometric Email, Contexts * 1-Sensors, 2-Contexts, 3-Contents, 4-Annotations, 5-Semantic Modeling, 6-Storage, 7-Architecture, 8-Retreival/Use-Cases, 9-User Control and Personalization, 10- Flexibility, 11-Battery Power Consumption ** NA-Not Available
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
211
The data for the empirical study is collected from the participant using a questionnaire. The total number of participants in the study is 20. The sample size is kept small due to lack of time and availability of resources; however, it is large enough to derive conclusions. Among the participants, 13 are males and 7 are females and all of the ages ranging from 20 to 39 years. The participants are selected voluntarily and are under-graduate (BS), post-graduate (MS), and PhD students in the Department of Computer Science, University of Peshawar. The participants are having substantial experience in computer science, research, smartphone applications usage and development, and industry. However, to do justice with the study, a comprehensive program is designed consisting of 3-days workshop on smartphone-based semantic lifelogging and SLOG development, and set of tasks in experiments. The participants are provided the SLOG and services (i.e., MyLocations, and MySMSs) source codes. The SLOG and services are also installed on their smartphones for observations, experiments, and practical usage.
Recommendations from the participants are incorporated in the proposed semantic framework and eventually in SLOG and services development.
A questionnaire is developed to collect data from the participants for the empirical study. The questionnaire and descriptive analysis methodology is developed using the knowledge and help from the prior researches in the data acquisition and elicitation domain, and data analysis experts.
The questionnaire is composed of 40 questions that are comprehensive enough and distributed fairly to create a survey for measuring the proposed semantic framework methodology and
SLOG from the above mentioned variables/aspects. The questions in the questionnaire are propositions whose answers are selected from a 5-level Liker-Scale, ranging from 1 = "strongly disagree" to 5 = "strongly agree". An example question is "The semantic lifelog model provides
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
212 enough constructs to model lifelog and contextual information?". After a period of two weeks, each participant filled the questionnaire individually. Table 5.2 represents statistical information of the participants’ responses to the questions in percentage, mean, and standard deviation.
Analyzing the responses, it is found that participants responded to the questions with 28.9% strongly agree, 41.4% agree, 13.4% neutral, 12.1% disagree, and 4.2% strongly disagree. With respect to the options numerical values, the overall mean value is 3.78, and standard deviation value is 1.116.
Table 5.2: Participants responses to the questions in the questionnaire.
Likert-Scale Percentage Standard Questions * Sample Frequency Mean Options (%) Deviation
Strongly 34 4.2% Disagree = 1
Disagree = 2 97 12.1%
Neutral = 3 107 13.4% 40 * 20 3.78 1.116 Agree = 4 331 41.4%
Strongly Agree 231 28.9% = 5
Total 800 100.0
The Likert-Scale data is ordinal, where the order of the values is significant and important but the exact difference between the values is not really known. Chi-Square is an important descriptive statistical method for analysis of the categorical data [165]. To analyze the ordered scale 5 levels Likert-Scale responses data using Chi- Square descriptive statistics, the five response categories (i.e., strongly disagree, disagree, neutral, agree, and strongly agree) are
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
213 breakdown into two nominal categories (i.e., disagree and agree) by combining the lower level three categories and upper level two categories respectively. Table 5.3 represents the division of five levels Likert-Scale response categories into two nominal categories, percentage values, and in the total. Chi-Square test is executed on the nominal categories using Statistical Package for the Social Sciences (SPSS) 16.0 to show the effectiveness of the proposed semantic framework methodology implemented in SLOG. The null and alternative hypotheses are:
H0: The methodology proposed in the semantic framework and implemented in SLOG is
not an effective methodology for smartphone-based semantic lifelogging.
H1: The methodology proposed in the semantic framework and implemented in SLOG is
effective methodology for smartphone-based semantic lifelogging.
Table 5.3: Division of 5 level Likert-Scale response categories into nominal categories and their
percentage values within nominal categories.
Five levels of Likert-Scale Response Categories Nominal Category Wise Total Categories Distribution Strongly Strongly Disagree Neutral Agree Disagree Agree
Count 34 97 107 0 0 238 Disagree % Value Within 14.3% 40.8% 45.0% .0% .0% 100.0% Disagree
Count 0 0 0 331 231 562 Agree % Value Within Agree .0% .0% .0% 58.9% 41.1% 100.0%
Count 34 97 107 331 231 800 Total % Value Within Total 4.2% 12.1% 13.4% 41.4% 28.9% 100.0%
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
214
Results collected after performing Chi-Square test on the dataset is shown in Table 5.4. The top row in the table shows Pearson Chi-Square statistics 2 = 8.000E2 and p < 0.001. The null hypothesis (H0) is rejected, since p < 0.05 (i.e., in fact p < 0.001). Therefore, the alternative hypothesis (H1) stands true, which signifies that the proposed semantic framework methodology implemented in SLOG is effective for smartphone-based semantic lifelogging.
Table 5.4: Results of the Chi-Square test using SPSS 16.0.
Descriptive Statistics Value p-Value
Pearson Chi-Square (2) 8.000E2a .000
Likelihood Ratio 973.970 .000
Linear-by-Linear 594.508 .000 Association
N of Valid Cases 800
a. 0 cells (.0%) have expected count less than 5. The minimum expected count is 10.12.
5.5 Summary
In this chapter, we have explored the design and development of the proposed semantic framework for smartphone-based semantic lifelogging to fulfill fourth objective of this research thesis. The proposed semantic framework architecture is published in our previous research publication [159], which is included in this chapter as part of the thesis. As discussed in Chapter
2, a universally accepted smartphone-based lifelogging system is not available that should extract contextual semantics from sensory data for organizing lifelog information in similar to human memory. The proposed framework is aimed to fill the gap and provides an open
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
215 architecture that can be extendable and customizable to support several use-cases. The layered property of the architecture provides primitives to capture and extract contextual semantics from sensors data and lifelog contents, defining semantic and mapping rules, relate and annotate the lifelog information in the semantic model, and retrieve required lifelog information using contextual semantics from the semantic model to satisfy an application needs. Apart from openness and extendibility, the proposed framework architecture has several advantageous characteristics including semantic organization, improved annotation, multiplicity of use-cases, improved surveillance and surveillance, fine-grained retrieval, etc. To demonstrate practicality of the proposed semantic framework architecture, a proof-of-concept prototypic application namely
SLOG is developed. In addition, two services are also developed to demonstrate retrieval and visualization of the required lifelog information from the semantic model. The effectiveness of the proposed semantic framework methodology and SLOG is evaluated quantitatively by conducting an extensive empirical study. In the empirical study, data is collected from the participants using a comprehensive questionnaire. The Chi-Square descriptive statistical analysis on the collected data has revealed that the proposed semantic framework methodology incorporated in SLOG is effective for smartphone-based semantic lifelogging.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
216
Chapter 6 : Conclusion and Future Work
Recording lifetime personal experiences has a prolonged history in mankind. People have mainly used external memory aids such as painting on stones, writing diaries, etc., to record their personal past-experiences. The improvements in lightweight computing devices have realized new methods for effective autobiography generation. The paradigm of lifelogging emphasizes on assisting human organic memory by using external computing devices as memory augmentation tools. Wearable lifelogging technology encourages researchers for the design and development of custom-built holistic lifelogging devices and applications. However, the wearable lifelogging systems are limited in their features (e.g., limited number of sensors and storage, lifelog information organization, and omnipresent access to lifelog information) with additional disadvantages of overloading users with extra devices. In addition, they mainly focus on visual lifelogging, addressing a portion of lifelogging and not fulfilling the vision of capturing totality of life experiences. Therefore, feasible solution is using digital devices or lifelogging that constitute all of the sensory, processing, and storage capabilities and are in common practice of people in their day-to-day activities.
Smartphone - the most common ubiquitous computing device - offer new opportunities for unobtrusively recording nearly all aspects of a person's life to construct and preserve a long-term digital prosthetic memory. Smartphone can be a step towards "totality of life experiences" by ubiquitous and passive capturing and storing excessive useful information about contents and contexts of users' daily life activities and events in a verbatim and unbiased way. Realizing the potentials of smartphone, a few previous researchers have presented some smartphone-based
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
217 lifelogging systems over the past several years. Despite of being advantageous, smartphone- based lifelogging faces specific obstacles and limitations, which have so far largely impeded its large-scale deployment. The smartphone is merely used as a capturing device with no insight knowledge to confirm its potentiality of being as a lifelogging device. In addition, a flat textual collection of lifelog information is composed with no logical relationships between distinct lifelog information for completely portraying a lifelog event. The sensors are used to capture contextual information but no inherent semantics are derived from the sensory data to more meaningfully annotate and relate lifelog information. Therefore, a digital prosthetic memory requires organizing lifelog information in similar to human episodic memory.
Our methodology is twofold of leveraging the experiences of lifelogging research and Semantic
Web technologies to smartphone to get the vision of Memex a step closer by developing a digital prosthetic memory on smartphone. Using the sensory capabilities of smartphone to detect lifelogging contents and contexts, and processing capabilities to extract semantics from sensors data and metadata from lifelog objects would produce enough information to semantically annotate and link lifelog information. Organizing lifelog information semantically would produce a directed labeled graph by inter-linking lifelog information, as they exist in the real world and encoded in human episodic memory. The semantic organization would result into several advantages including improving data understandability using contextual information expressed by annotations and associations, fine-grained and expressive data manipulation (e.g., querying) using superimposed information, effective data reasoning and new knowledge inferencing using explicit formal semantics, heterogeneous data integration using uniform representation, and integration of new information sources. The complex semantically enriched
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
218 lifelog data network can be expressively queried for retrieving information related to an active event, task, project, or application. The network can be visualized and navigated easily for locating an item of information as compared to traditional hierarchical folder structures. This practice would also facilitate integration and incorporation of lifelog information captured or created using separate applications/sources into a single information space on the same device.
However, the main contributions of this thesis are as follows:
• We have given a comprehensive overview of the different metaphors of lifelogging
systems, especially smartphone-based lifelogging systems, and the associated challenges.
This detailed insight knowledge has improved our understanding of: (i) how lifelogging
started and evolved into the current status?; (ii) what is role of sensors technology in the
lifelogging?; (iii) which smartphone-based lifelogging applications are available and
what are their implications and limitations?; (iv) how smartphone-based lifelogging
research can be classified?; (v) which of the areas are addressed by the researchers and
which one needs attention?; (vi) what are the potential use-cases of smartphone-based
lifelogging and what others can emerge?
• We have improved our understanding about smartphone as a de-facto lifelogging
platform. We have thoroughly experimented and investigated smartphone platform and
learned about its technological and functional capabilities and limitations, and their
implications for the design and development of smartphone-based lifelogging
technologies and applications. The detailed insight has explored and improved our
understanding about smartphone from different aspects including: (1) what is current
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
219
status of smartphone platforms, associated sensors and future prospects?; (2) what
contextual data can be captured by smartphone sensors to provide memory cues and
annotate and relate lifelog information into a rich graph structure?; (3) does existing
smartphone sensing capabilities provide enough contextual information or needed to be
extended with physical or virtual sensors?; (4) what is the effect of using smartphone
sensors and applications on smartphone resources?; (5) what are the potential smartphone
applications that users uses in performing their daily life real world activities?; (6) do
smartphones users potentially need lifelogging systems?; (7) can state-of-the-art
smartphone provide effective platform (i.e., both hardware and software) for large-scale
lifelogging system?; and (8) what are the advantages of smartphone technology over
dedicated lifelogging devices? Conclusively, results of the analysis and experiments have
provided enough evidence that smartphone can be a potential lifelogging device and can
effectively replace custom-built and commercial lifelogging devices.
• This research thesis has used Semantic Web technologies to develop a semantic model
(i.e., ontology) and solve the lifelog information annotation and organization problems
found in the existing smartphone-based lifelogging systems/applications. The
development of semantic model has explored and improved our understanding of: (1)
how Semantic Web technologies can be used to develop a semantic model of lifelog
information?; (2) how Semantic Web technologies can be helpful in mining and
interpreting semantics of the lifelog information?; (3) how an ontological model could be
developed for user modeling that would semantically annotate and relate lifelog
information using contextual with the same semantics as they appear in the real world of
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
220
a user?; (4) how low-level sensory data can be mapped into high-level contexts using
ontological model?; (5) how a lifelogging ontological model can be inferred for new
knowledge and how it can be queried to retrieve relevant information to an event or
context?; and (6) how a lifelogging ontological model can be connected with external
semantically enriched information sources to enhance semantics of the lifelog
information? The development and contents of lifelogging semantic model namely
SmartOntoSensor is discussed in Chapter 4. The results and evaluation of
SmartOntoSensor has provided enough evidence for its effective semantic organization of
lifelog information.
• This thesis contributes the designing, building, and evaluation of a smartphone-based
semantic lifelogging framework to unify the research efforts instead of being divided into
separate islands (discussed in Chapter 5). The proposed semantic framework is layered,
open, flexible, extendable, and semantically enriched that fully exploits modern
smartphone sensing capabilities, Semantic Web technologies, and experiences from
lifelogging research. The proposed semantic framework can be used as a reference model
for implementing a smartphone-based semantic lifelogging system. The proposed
semantic framework emphasizes on using sensors data and advocates for using every
physical sensor, logical sensor, virtual sensor, application sensor, and informational
sensor that are either available in the smartphone or could emerge with the passage of
time. The contexts and metadata semantic are extracted from the data and lifelog objects
generated by the lifelogging sensors. The semantic glue is added by defining mapping
and semantic rules, and using the lifelogging semantic model (i.e., SmartOntoSensor
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
221
discussed in Chapter 4). The lifelog information are added to the ontology and annotated
with contextual information and metadata to add semantics for facilitating IR. The
proposed semantic framework is open because it allows adding, removing, and upgrading
sensors, applications, mapping rules, components, and ontological model. The proposed
semantic framework uses valuable knowledge from the existing systems and satisfies
both personal and social aspects of lifelogging. Personal aspect fulfils the principle of
using lifelog for personal use only and sharing aspect determines sharing of lifelog
information with society while preserving security and privacy issues. In regards to
storage, a semantic data model is proposed for representing and organizing lifelog
information using semantics of the sensory data and metadata. The generality of the
semantic data model enables lifelogging applications for the different use-cases. To
demonstrate practicality and usability of the proposed semantic framework, a proof-of-
concept prototypic application (i.e., called SLOG) is developed in Android. The SLOG is
evaluated empirically, which has resulted into effectiveness of proposed semantic
framework methodology that is fulfilling objectives of this research thesis.
6.1 Limitations and Future Work
The smartphone-based lifelogging has no specific boundaries and may extend with the emergence of new technologies and events. In the previous section, we have summarized the main contributions of this thesis. Our methodology has shown merits in fulfilling objectives of this thesis. However, the techniques and models presented are not free of limitations. The complexities of this research work are note worthy and needs more investigation. In this section, we highlight a few of the principal opportunities as future work.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
222
The sensing capabilities integrated in smartphone enable users to capture a spectrum of information regarding their environments, activities, and participations in social gatherings both visually and acoustically. However, they are relatively fundamental to date. The research thesis has shown using different types of sensors for capturing variety of contextual and metadata information as possible. However, there are several aspects of a daily life event, which cannot be materialized using existing smartphone sensing technologies. Therefore, advanced sensing technologies (e.g., heart rate monitor sensor, and sleep pattern monitor sensor) should be integrated in smartphone for detailed sensing and capturing the semantics of human life where sentiment, mood, emotion, etc., can be conceivably collected into a lifelog. In addition, the increased number of sensors will help in extracting more and more semantics from the sensors data. The integration of advanced capturing technologies will shift the technology from logging episodic memory into logging of both episodic and semantic memories.
As mentioned previously, the detailed investigation has increased our understanding of using smartphone as a lifelogging platform. The experimental results signify objectives of this thesis.
However, more detailed experiments and analysis are needed to concrete the hypothesis. For example, designing real world experiments to analyze and evaluate quality of sensors data (e.g., pictures) collected by smartphone and dedicated lifelogging devices and their relative effects on resources.
The proposed lifelogging ontological model (i.e., SmartOntoSensor) constitutes enough concepts to model sensors data and a person's daily life experiences. However, the ontological model is mid-level ontology containing mostly generic concepts. The ontology can be extended to include domain-specific and application-specific concepts for modeling lifelogging experience in
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
223 different applications and domains. The extensions will yield in a network of lifelogging ontologies and would provide rich user modeling. In addition, semantically enriched data from online Semantic Web knowledge stores (e.g., LOD) can be linked, added, and employed to annotate and provide memory cues to trigger memory. In addition, data can be combined and uploaded to Facebook, Twitter, and Flicker. The detailed ontological modeling would enable to infer lifelog information, which cannot be captured from the existing information sources.
The thesis has presented a semantic framework for smartphone-based semantic lifelogging and its proof-of-concept implementation (i.e., SLOG) and evaluation. The experimental results have yielded into significance of the objectives of this thesis. However, its complete implementation and insight evaluation can indicate its ultimate effectiveness. The SLOG implementation has provided enough guidelines to facilitate its complete implementation. The SLOG can be extended in several smartphone-based domain-specific lifelogging applications including behavior learning and mobile health to provide effective one's reflection. The applications can track and report users about their behaviors and activities to bring significant changes for having healthier and joyful life. For example, smartphone-based diet monitoring system can use SLOG to monitor food intake and calories burning rates of users, and suggest them to take the number of steps to burn calories. Another potential application of SLOG can be time management and recommender systems. The lifelog information collected by SLOG can be used by online marketing and advertisement systems to timely deliver precise advertisements to the users.
Ideally, a smartphone-based lifelogging should produce a digital prosthetic memory that should function in synergy to human memory and assist/support real memory functions. Therefore,
SLOG should be extended to include other real memory functions such as forgetting of unwanted
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
224 life experiences, and privacy and security. The proposed framework suggests using of privacy by design to ensure privacy and security of lifelog information. However, it is not supported in the current implementation of SLOG. Therefore, SLOG should be extended with complete implementation of privacy by design and the results should be evaluated and compared with the privacy and security framework proposed by [21]. The loss or damage of smartphone with a rich archive of personal lifelog information can also put a user in a devastating situation. Therefore, to enhance the privacy and security, the anonymisation feature of lifelog information should be included in SLOG to relive users from the worries of missing or losing smartphone and bystanders from being captured unknowingly. We believe that anonymisation should be a dynamic process that is implemented at access time and dependent on user access policies.
The information form a smartphone-based lifelog should be provided through an interface to support several use-cases. The smartphone-based lifelogging would have varied and broad use- cases, which will become more clear as the technology becomes more popular. Whichever, might be the use-cases, this new technology should be developed and mapped into our lives instead of changing our lives for the technology. Most of the studies have emphasized on using smartphone-based lifelogging as memory aids due to no clear availability of its use-cases.
Unfolding more use-cases will instigate the need of developing novel capture technologies to tackle new data sources and will also indicate information needs of real-world users for retrieval.
However, guidance can be taken from the book "Total Recall" [34] to identify potential use-cases of smartphone-based semantic lifelogging.
The proposed smartphone-based semantic lifelogging process would automatically collect and manage lifelogging information from different data sources. It can produce an extensive lifelog
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
225 data in a very short span of time, which would be humanly infeasible and cumbersome to handle manually. To reduce user intervention in the sensing and recording, a lifelogging system needs to run in the background of a smartphone inconspicuously 24/7. However, users need to have appropriate understanding and control over the application. Therefore, users' interventions should be limited up to sensors configuration, adjusting sensors reading, and starting and stopping the service. In addition, the systems should use lifelogging technology to provide human-like interface for a semblance of episodic memory or personality. Valuable research experiences from other domains (e.g., development of humanoid robotics) can be used in making the proposed smartphone-based semantic lifelogging more humanized.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
226
Publications
This research thesis is based on following publications.
1. Shaukat Ali, Shah Khusro, Irfan Ullah, Akif Khan, and Inayat Khan, "SmartOntoSensor:
Ontology for Semantic Interpretation of Smartphone Sensors Data for Context-Aware
Applications," Journal of Sensors, vol. 17(2017), pp. 1-26, 2017.
2. Shaukat Ali and Shah Khusro, "POEM: Practical ontology engineering model for
semantic web ontologies," Cogent Engineering, vol. 3(1), pp. 1-39, 2016.
3. Inayat Khan, Shah Khusro, Shaukat Ali, and Jamil Ahmad, "Sensors are Power Hungry:
An Investigation of Smartphone Sensors Impact on Battery Power from Lifelogging
Perspective," The Journal of Bahria University Information and Communication
Technologies, BUJICT, vol. 9(2), pp. 1-12, 2016.
4. Inayat Khan, Shaukat Ali, and Shah Khusro, "Smartphone Based Lifelogging:
Investigation of Data Volume Generation Strength of Smartphone Sensors," in
Proceedings of the SCONEST'16, Karachi, Pakistan, pp. 1-10, 2016.
5. Shaukat Ali and Shah Khusro, "Mobile Phone Sensing: A New Application Paradigm,"
Indian Journal of Science and Technology, vol. 9(19), pp. 1-42, 2016.
6. Inayat Khan, Shah Khusro, Shaukat Ali, and Aziz Ud Din, "Daily Life Activities on
Smartphones and Their Effect on Battery Life for Better Personal Information
Management," Proceedings of the Pakistan Academy of Sciences: A Physical and
Computational Sciences, vol. 53(1), pp. 61-74, 2016.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
227
7. Shah Khusro, Shaukat Ali, Iftikhar Alam, and Irfan Ullah, "Performance Evaluation of
Desktop Search Engines Using Information Retrieval Systems Approaches," Journal of
Internet Technology, vol. 99(99), pp. b1-11, 2015.
8. Fakhre Alam, Shaukat Ali, Muhammad Abid Khan, Shah Khusro, and Azhar Rauf, "A
comparative study of RDF and topic maps development tools and APIs," The Journal of
Bahria University Information & Communication Technology, BUJICT, vol. 7(1), p. 1-
12, 2014.
9. Shaukat Ali, Shah Khusro, Asif Ali Khan, and Liaq Hassan, "A Survey of Mobile
Phones Context-Awareness Using Sensing Computing Research," Journal of Engineering
and Applied Sciences, vol. 33(4), pp. 75-93, 2014.
10. Shaukat Ali, Shah Khusro, Azhar Rauf, and Saeed Mahfooz, " Sensors and Mobile
Phones: Evolution and State-of-the-Art," Pakistan Journal of Science vol. 66(4), pp. 386-
400, 2014.
11. Fakhre Alam, Muhammad Abid Khan, Shaukat Ali, and Shah Khusro, "The jigsaw of
resource description framework (RDF) and topic maps serialization formats: A survey,"
Proceedings of the Pakistan Academy of Sciences: A Physical and Computational
Sciences, vol. 51(2), pp. 101-114, 2014.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
228
References
[1] S. K. Chennuru, P.-W. Chen, J. Zhu, and J. Y. Zhang, "Mobile Lifelogger - Recording, Indexing, and Understanding a Mobile User's Life," in International Conference on Mobile Computing, Applications, and Services, 2010, pp. 263-281. [2] Y. Chen and G. J. F. Jones, "Augmenting human memory using personal lifelogs," presented at the Proceedings of the 1st Augmented Human International Conference, France, 2010. [3] S. Hodges, L. Williams, E. Berry, S. Izadi, J. Srinivasan, A. Butler, et al., "SenseCam: a retrospective memory aid," presented at the Proceedings of the 8th international conference on Ubiquitous Computing, Orange County, CA, 2006. [4] N. Aizenbud-Reshef, E. Belinsky, M. Jacovi, D. Laufer, and V. Soroka, "Pensieve: augmenting human memory," presented at the CHI '08 Extended Abstracts on Human Factors in Computing Systems, Florence, Italy, 2008. [5] K. Ig-Jae, A. Sang Chul, and K. Heedong, "Automatic Lifelog media annotation based on heterogeneous sensor fusion," in IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, 2008. MFI 2008., 2008, pp. 703-708. [6] V. Kalnikaite, A. Sellen, S. Whittaker, and D. Kirk, "Now let me see where i was: understanding how lifelogs mediate memory," presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Georgia, USA, 2010. [7] E. S. T. Mohamed, "Designing and evaluating a user interface for continues embedded lifelogging based on physical context," PhD Dissertation, School of Computing Science, University of Newcastle, Newcastle, UK, 2012. [8] V. Bush, "As we may think," interactions, vol. 3, pp. 35-46, 1996. [9] C. Gurrin, A. F. Smeaton, and A. R. Doherty, "LifeLogging: Personal Big Data," Foundations and Trends in Information Retrieval, vol. 8, pp. 1-125, 2014. [10] M. Dodge and R. Kitchin, "‘Outlines of a world coming into existence’: pervasive computing and the ethics of forgetting," Environment and Planning B: Planning and Design, vol. 34, pp. 431-445, 2007. [11] C. Walter, "Kryder’s law," Scientific American, vol. 293, pp. 32–33, 2005. [12] P. Belimpasakis, K. Roimela, and Y. Yu, "Experience Explorer: A Life-Logging Platform Based on Mobile Context Collection," in Third International Conference on Next Generation Mobile Applications, Services and Technologies - NGMAST '09., 2009, pp. 77-82.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
229
[13] C. Gurrin, Z. Qiu, M. Hughes, N. Caprani, A. R. Doherty, S. E. Hodges, et al., "The Smartphone As a Platform for Wearable Cameras in Health Research," American Journal of Preventive Medicine, vol. 44, pp. 308–313, 2013. [14] R. Albatal, C. Gurrin, J. Zhou, Y. Yang, D. Carthy, and N. Li, "SenseSeer Mobile-Cloud- Based Lifelogging framework," in The IEEE International Symposium on Technology and Society (ISTAS), Toronto, Canada, 2013, pp. 27-29 [15] R. Rawassizadeh, M. Tomitsch, K. Wac, and A. M. Tjoa, "UbiqLog: a generic mobile phone-based life-log framework," Personal Ubiquitous Comput., vol. 17, pp. 621-637, 2013. [16] A. Sellen and S. Whittaker, "Beyond Total Capture: A Constructive Critique of Lifelogging," Communications of the ACM, vol. 53, pp. 70-77, 2010. [17] J. Gemmell, G. Bell, and R. Lueder, "MyLifeBits: a personal database for everything," Commun. ACM, vol. 49, pp. 88-95, 2006.
[18] O. Bergman, R. Beyth‐Marom, and R. Nachmias, "The user‐subjective approach to personal information management systems," Journal of the American Society for Information Science and Technology, vol. 54, pp. 872-878, 2003. [19] K. Stewart, J. Basic, and S. Erdelez, "ODI and information literacy: Personal information management in a world of information overload," in Proceedings of the 75th ASIS&T Annual Meeting, Baltimore, Maryland 2012. [20] D. K. Barreau, "Context as a factor in personal information management systems," J. Am. Soc. Inf. Sci., vol. 46, pp. 327-339, 1995. [21] R. Rawassizadeh, "A holistic multi-purpose life logging framework," uniwien, 2012. [22] S. Ali, S. Khusro, A. A. Khan, and L. Hassan, "A Survey of Mobile Phones Context- Awareness Using Sensing Computing Research," Journal of Engineering and Applied Sciences, vol. 33, pp. 75-93, 2014. [23] D. W. Loring and K. J. Meador, INS dictionary of neuropsychology: Oxford University Press, USA, 1999. [24] G. Cohen and M. A. Conway, Memory in the real world: Psychology Press, 2008. [25] E. Tulving, "Episodic and semantic memory," Organization of Memory. London: Academic, vol. 381, pp. 382-404, 1972. [26] P. Wang, "Semantic Interpretation of Events in Lifelogging," PhD Dissertation, School of Computing, University of Dublin City, 2011. [27] M. Linton, "Transformations of memory in everyday life," Memory observed: Remembering in natural contexts, pp. 77-91, 1982.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
230
[28] K. Erngrund, T. Mántylá, and L.-G. Nilsson, "Adult age differences in source recall: A population-based study," The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, vol. 51, pp. P335-P345, 1996. [29] S. Jovicic, "Role of memory in email management," in CHI'00 extended abstracts on Human factors in computing systems, 2000, pp. 151-152. [30] M. Lamming and M. Flynn, "`Forget-me-not': intimate computing in support of human memory," presented at the Proceedings of FRIEND21: Symposium on Next Generation Human Interfaces, Tokyo, Japan, 1994. [31] E. Tulving, "Elements of episodic memory," 1985. [32] G. Bell and J. Gemmell, "A Digital Life," Journal of Scientific American, vol. 296, pp. 58–65, 2007. [33] K. O'Hara, R. Morris, N. Shadbolt, G. J. Hitch, W. Hall, and N. Beagrie, "Memories for life: a review of the science and technology," Journal of the Royal Society Interface, vol. 3, pp. 351-365, 2006. [34] C. G. Bell and J. Gemmell, Total recall: How the e-memory revolution will change everything: Dutton, 2009. [35] V. Mayer-Schönberger, Delete: The virtue of forgetting in the digital age: Princeton University Press, 2011. [36] A. J. Sellen, A. Fogg, M. Aitken, S. Hodges, C. Rother, and K. Wood, "Do life-logging technologies support memory for the past?: an experimental study using SenseCam," presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, San Jose, California, USA, 2007. [37] A. L. Allen, "Dredging up the past: Lifelogging, memory, and surveillance," The University of Chicago Law Review, vol. 75, pp. 47-74, 2008. [38] S. Ali and S. Khusro, "Mobile Phone Sensing: A New Application Paradigm," Indian Journal of Science and Technology, vol. 9, pp. 1-41, 2016. [39] M. Hughes, E. Newman, A. F. Smeaton, and N. E. O'Connor, "A lifelogging approach to automated market research," 2012. [40] I. Khan, S. Khusro, S. Ali, and A. U. Din, "Daily Life Activities on Smartphones and Their Effect on Battery Life for Better Personal Information Management," Proceedings of the Pakistan Academy of Sciences: A Physical and Computational Sciences, vol. 53, pp. 61-74, 2016. [41] A. Zafeiropoulos, N. Konstantinou, S. Arkoulis, D. E. Spanos, and N. Mitrou, "A Semantic-Based Architecture for Sensor Data Fusion," in the 2nd International
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
231
Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies, pp. 116-121, 2008. [42] S. Vemuri, C. Schmandt, and W. Bender, "iRemember: a Personal, Long-term Memory Prosthesis," in 3rd ACM Workshop on Continous Archival and Retrieval of Personal Experiences CARPE' 06, Santa Barbara, California, USA, 2006, pp. 65-74. [43] B. Clarkson, K. Mase, and A. Pentland, "The familiar: a living diary and companion," presented at the CHI '01 Extended Abstracts on Human Factors in Computing Systems, Seattle, Washington, 2001. [44] T. Groza, S. Handschuh, K. Moeller, G. Grimnes, L. Sauermann, E. Minack, et al., "The NEPOMUK Project - On the way to the Social Semantic Desktop," presented at the Proceedings of I-Semantics' 07, 2007. [45] L. Sauermann, "The Gnowsis Semantic Desktop for Information Integration," presented at the Proceedings of the IOA 2005 Workshop at the WM, 2005. [46] P.-A. Chirita, S. Costache, W. Nejdl, and R. Paiu, "Beagle++ : Semantically Enhanced Searching and Ranking on the Desktop," presented at the European Semantic Web Conference, Budva, Montenegro, 2006. [47] S. Dumais, E. Cutrell, J. J. Cadiz, G. Jancke, R. Sarin, and D. C. Robbins, "Stuff I've seen: a system for personal information retrieval and re-use," in ACM SIGIR Forum, 2016, pp. 28-35. [48] T. Noda and S. Helwig, "Benchmark Sutdy of Desktop Search Tools," UW E-Business Consortium University of Misconsin, Misconsin, USAApril 20, 2005 2005. [49] S. Khusro, S. Ali, I. Alam, and Irfanullah, "Performance Evaluation of Desktop Search Engines Using Information Retrieval Systems Approaches," Journal of Internet Technology, vol. 99, pp. b1-11, 2015. [50] S. Fertig, and E. Freeman, "Lifestreams: an alternative to the desktop metaphor," in Conference companion on Human factors in computing systems, 1996, pp. 410-411. [51] DARPA/IPTO. (2003, 23/08/2016). BAA # 03-30 LifeLog Proposer Information Pamphlet. Available: http://realnews247.com/lifelog.htm [52] C. Schlenoff, B. Weiss, and M. P. Steves, "Lessons learned in evaluating DARPA advanced military technologies," in Proceedings of the 10th Performance Metrics for Intelligent Systems Workshop, 2010, pp. 227-234. [53] N. Schactman. (2004, 20/08/2016). Pentagon revives memory project. Available: http://www.wired.com/politics/security/news/2004/09/64911
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
232
[54] S. Mann, "Smart clothing: The shift to wearable computing," Communications of the ACM, vol. 39, pp. 23-24, 1996. [55] G. D. Abowd, "Classroom 2000: an experiment with the instrumentation of a living educational environment," IBM Syst. J., vol. 38, pp. 508-530, 1999. [56] T. P. Moran, L. Palen, S. Harrison, P. Chiu, D. Kimber, S. Minneman, et al., ""I'll get that off the audio": a case study of salvaging multimedia meeting records," presented at the Proceedings of the ACM SIGCHI Conference on Human factors in computing systems, Atlanta, Georgia, USA, 1997. [57] S. Mann and H. Niedzviecki, Cyborg: Digital Destiny and Human Possibility in the Age of the Wearable Computer: Random House Inc, 2001. [58] S. Mann, "Continuous lifelong capture of personal experience with EyeTap," in Proceedings of the 1st ACM workshop on Continuous archival and retrieval of personal experiences, New York, New York, USA, 2004, pp. 1-21. [59] S. Mann, "Wearable computing: A first step toward personal imaging," Computer, vol. 30, pp. 25-32, 1997. [60] S. Mann, J. Fung, C. Aimone, A. Sehgal, and D. Chen, "Designing EyeTap digital eyeglasses for continuous lifelong capture and sharing of personal experiences," in CHI 2005 Conference on Computer Human Interaction, Portland, Oregon, USA, 2005. [61] S. Mann, J. Fung, and R. Lo, "Cyborglogging with camera phones: Steps toward equiveillance," in Proceedings of the 14th ACM international conference on Multimedia, 2006, pp. 177-180. [62] T. Hori and K. Aizawa, "Context-based video retrieval system for the life-log applications," presented at the Proceedings of the 5th ACM SIGMM international workshop on Multimedia information retrieval, Berkeley, California, 2003. [63] K. Aizawa, D. Tancharoen, S. Kawasaki, and T. Yamasaki, "Efficient retrieval of life log based on context and content," presented at the Proceedings of the the 1st ACM workshop on Continuous archival and retrieval of personal experiences, New York, New York, USA, 2004. [64] B. J. Rhodes, "The wearable remembrance agent: A system for augmented memory," Personal Technologies, vol. 1, pp. 218-224, 1997. [65] S. Vemuri, C. Schmandt, W. Bender, S. Tellex, and B. Lassey, "An Audio-Based Personal Memory Aid," in UbiComp 2004: Ubiquitous Computing: 6th International Conference, Nottingham, UK, September 7-10, 2004. Proceedings, N. Davies, E. D. Mynatt, and I. Siio, Eds., ed Berlin, Heidelberg: Springer Berlin Heidelberg, 2004, pp. 400-417.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
233
[66] D. P. Ellis and K. Lee, "Minimal-impact audio-based personal archives," in Proceedings of the 1st ACM workshop on Continuous archival and retrieval of personal experiences, 2004, pp. 39-47. [67] W. C. Cheng, L. Golubchik, and D. G. Kay, "Total recall: are privacy changes inevitable?," presented at the Proceedings of the 1st ACM workshop on Continuous archival and retrieval of personal experiences, New York, New York, USA, 2004. [68] P. Tentacle. (2008, 22/5/2016). Cyber Goggles: High-tech memory aid. Available: http://pinktentacle.com/2008/03/cyber-goggles-high-tech-memory-aid/ [69] S. Hodges, E. Berry, and K. Wood, "SenseCam: A wearable camera that stimulates and rehabilitates autobiographical memory," Memory, vol. 19, pp. 685-696, 2011. [70] T. Jacquemard, P. Novitzky, F. O’Brolcháin, A. F. Smeaton, and B. Gordijn, "Challenges and Opportunities of Lifelog Technologies: A Literature Review and Critical Analysis," Science and Engineering Ethics, vol. 20, pp. 379-409, 2014. [71] K. Del Giudice and M. Gardner. (2009, 22/07/2016). The message of the pensieve: Realizing memories through the world wide web and virtual reality. Available: : http://web.mit.edu/comm-forum/mit6/papers/Delgiudice.pdf. [72] R. Harper, T. Rodden, Y. Rogers, and A. Sellen, "Being human: human-computer interaction in the year 2020 (Cambridge)," UK: Microsoft Research, 2008. [73] A. Myka, "Nokia lifeblog -- towards a truly personal multimedia information system," in Workshop on Mobile Data Banking and Information Systems - MDBIS'05, 2005, pp. 21- 30. [74] J. Froehlich, M. Y. Chen, S. Consolvo, B. Harrison, and J. A. Landay, "MyExperience: a system for in situ tracing and capturing of user feedback on mobile phones," presented at the Proceedings of the 5th international conference on Mobile systems, applications and services, San Juan, Puerto Rico, 2007. [75] M. Ahmed, H. H. Hoang, M. S. Karim, S. Khusro, M. Lanzenberger, K. Latif, et al., "'SemanticLIFE'-A Framework for Managing Information of A Human Lifetime," in Sixth International Conference on Information Integration and Web Based Applications & Services 2004. [76] A. R. Doherty and A. F. Smeaton, "Automatically segmenting lifelog data into events," in 2008 Ninth International Workshop on Image Analysis for Multimedia Interactive Services, 2008, pp. 20-23. [77] M. A. Memon, S. Bhatti, and N. A. Mahoto, "A Digital Diary: Remembering the Past Using the Present Context," Mehran University Research Journal of Engineering & Technology, vol. 35, pp. 275-286, 2016.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
234
[78] L. Sangkeun, G. Gihyun, H. Inbeom, and L. Sang-goo, "LifeLogOn: A Practical Lifelog System for Building and Exploiting Lifelog Ontology," in Sensor Networks, Ubiquitous, and Trustworthy Computing (SUTC), 2010 IEEE International Conference on, 2010, pp. 367-373. [79] H. S. Packer, A. Smith, and P. Lewis, "Memorybook: Generating Narratives from Lifelogs," in Proceedings of the 2nd Workshop on Narrative and Hypertext, 2012, pp. 7- 12. [80] M. Shah, B. Mears, C. Chakrabarti, and A. Spanias, "Lifelogging: Archival and retrieval of continuously recorded audio using wearable devices," in IEEE International Conference on Emerging Signal Processing Applications (ESPA), 2012, pp. 99-102. [81] S. Consolvo, D. W. McDonald, T. Toscos, M. Y. Chen, J. Froehlich, B. Harrison, et al., "Activity sensing in the wild: a field trial of UbiFit garden," presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy, 2008. [82] A. Raskin, "Your Life Experiences, Brought to You by Budweiser," IEEE Security & Privacy, vol. 9, pp. 86-88, 2011. [83] J. Orwant, "Heterogeneous learning in the Doppelgänger user modeling system," User Modeling and User-Adapted Interaction, vol. 4, pp. 107-130, 1994. [84] G. Fischer, "User modeling in human–computer interaction," User Modeling and User- Adapted Interaction, vol. 11, pp. 65-86, 2001. [85] B. P. Clarkson, "Life patterns: structure from wearable sensors," Massachusetts Institute of Technology, 2002. [86] C. Plate, N. Basselin, A. Kröner, M. Schneider, S. Baldes, V. Dimitrova, et al., "Recomindation: New functions for augmented memories," in International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, 2006, pp. 141-150. [87] Y. Uesugi, T. Yamamoto, J. Sawamoto, N. Segawa, E. Sugino, T. Goto, et al., "A Proposal of an Application of Life Log to Remote Consultation," in Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS), 2011 Fifth International Conference on, 2011, pp. 265-270. [88] J.-P. Dittrich, "iMeMex: A platform for personal dataspace management," in Proceedings of Workshops of International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, 2006, pp. 40-43. [89] N. Eagle and K. Greene, Mobile Phones, Sensors, and Lifelogging: A BIT of Reality Mining: MIT Press, 2015.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
235
[90] P. M. Lehrer, E. Vaschillo, and B. Vaschillo, "Resonant frequency biofeedback training to increase cardiac variability: Rationale and manual for training," Applied psychophysiology and biofeedback, vol. 25, pp. 177-191, 2000. [91] P. M. Lehrer, E. Vaschillo, B. Vaschillo, S.-E. Lu, A. Scardella, M. Siddique, et al., "Biofeedback treatment for asthma," Chest Journal, vol. 126, pp. 352-361, 2004. [92] V. Gerasimov, "Every sign of life," Massachusetts Institute of Technology, 2003. [93] S. Reddy, A. Parker, J. Hyman, J. Burke, D. Estrin, and M. Hansen, "Image browsing, processing, and clustering for participatory sensing: lessons from a DietSense prototype," in Proceedings of the 4th workshop on Embedded networked sensors, 2007, pp. 13-17. [94] J. Froehlich, T. Dillahunt, P. Klasnja, J. Mankoff, S. Consolvo, B. Harrison, et al., "UbiGreen: investigating a mobile tool for tracking and supporting green transportation habits," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2009, pp. 1043-1052. [95] M. Mun, S. Reddy, K. Shilton, N. Yau, J. Burke, D. Estrin, et al., "PEIR, the personal environmental impact report, as a platform for participatory sensing systems research," in Proceedings of the 7th international conference on Mobile systems, applications, and services, 2009, pp. 55-68. [96] E. Kanjo, S. Benford, M. Paxton, A. Chamberlain, D. S. Fraser, D. Woodgate, et al., "MobGeoSen: facilitating personal geosensor data collection and visualization using mobile phones," Personal and ubiquitous computing, vol. 12, pp. 599-607, 2008. [97] N. Maisonneuve, M. Stevens, M. E. Niessen, and L. Steels, "NoiseTube: Measuring and mapping noise pollution with mobile phones," in Information technologies in environmental engineering, ed: Springer, 2009, pp. 215-228. [98] P. Mohan, V. N. Padmanabhan, and R. Ramjee, "Nericell: rich monitoring of road and traffic conditions using mobile smartphones," in Proceedings of the 6th ACM conference on Embedded network sensor systems, 2008, pp. 323-336. [99] T. Karkkainen, T. Vaittinen, and K. Vaananen-Vainio-Mattila, "I don't mind being logged, but want to remain in control: a field study of mobile activity and context logging," presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Atlanta, Georgia, USA, 2010. [100] I. Khan, S. Ali, and S. Khusro, "Smartphone Based Lifelogging: Investigation of Data Volume Generation Strength of Smartphone Sensors," in Proceedings of the SCONEST'16, Karachi, Pakistan, 2016. [101] S. Ali, S. Khusro, A. Rauf, and S. Mahfooz, " Sensors and Mobile Phones: Evolution and State-of-the-Art," Pakistan journal of science vol. 66, pp. 386-400, 2014.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
236
[102] A. Carroll and G. Heiser, "An Analysis of Power Consumption in a Smartphone," in USENIX annual technical conference, 2010, pp. 21-21. [103] L. Corral, A. B. Georgiev, A. Sillitti, and G. Succi, "A method for characterizing energy consumption in Android smartphones," in Green and Sustainable Software (GREENS), 2013 2nd International Workshop on, 2013, pp. 38-45. [104] Y. Liu, C. Xu, and S.-C. Cheung, "Where has my battery gone? Finding sensor related energy black holes in smartphone applications," in Pervasive Computing and Communications (PerCom), 2013 IEEE International Conference on, 2013, pp. 2-10. [105] F. Ben Abdesslem, A. Phillips, and T. Henderson, "Less is more: energy-efficient mobile sensing with senseless," in Proceedings of the 1st ACM workshop on Networking, systems, and applications for mobile handhelds, 2009, pp. 61-62. [106] Y. Wang, J. Lin, M. Annavaram, Q. A. Jacobson, J. Hong, B. Krishnamachari, et al., "A framework of energy efficient mobile sensing for automatic user state recognition," in Proceedings of the 7th international conference on Mobile systems, applications, and services, 2009, pp. 179-192. [107] I. Khan, S. Khusro, S. Ali, and J. Ahmad, "Sensors are Power Hungry: An Investigation of Smartphone Sensors Impact on Battery Power from Lifelogging Perspective," The Journal of Bahria University Information and Communication Technologies, BUJICT, vol. 9, 2016. [108] S. Kinkade and K. Verclas. (2008) Wireless Technology for Social Change: Trends in Mobile Use by NGOs. Access to Communication Publication Series. 52-54. [109] E. Miluzzo, J. M. Oakley, H. Lu, N. D. Lane, R. A. Peterson, and A. T. Campbell, "Evaluating the iPhone as a mobile platform for people-centric sensing applications," Proceedings of UrbanSense' 08, vol. 8, pp. 41-45, 2008. [110] N. D. Lane, E. Miluzzo, H. Lu, D. Peebles, T. Choudhury, and A. T. Campbell, "A survey of mobile phone sensing," Comm. Mag., vol. 48, pp. 140-150, 2010. [111] N. D. Lane, S. B. Eisenman, M. Musolesi, E. Miluzzo, and A. T. Campbell, "Urban sensing systems: opportunistic or participatory?," presented at the Proceedings of the 9th workshop on Mobile computing systems and applications, Napa Valley, California, 2008. [112] M. Saber Chtourou and N. Souiden, "Rethinking the TAM model: time to consider fun," Journal of Consumer Marketing, vol. 27, pp. 336-344, 2010. [113] F. Aldhaban, "Exploring the adoption of Smartphone technology: Literature review," in 2012 Proceedings of PICMET '12: Technology Management for Emerging Technologies, 2012, pp. 2758-2770.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
237
[114] Egham. (2017, 15/06/2017). Gartner Says Worldwide Sales of Smartphones Grew 9 Percent in First Quarter of 2017. Available: http://www.gartner.com/newsroom/id/3725117 [115] B. o. America. (2014, 18/10/2016). Hold the Phone: U.S. Consumers Wouldn’t Last One Day Without Access to Their Smartphone. Available: http://newsroom.bankofamerica.com/press-releases/consumer-banking/hold-phone-us- consumers-wouldnt-last-one-day-without-access-their-sm [116] J. J. Barton, S. Zhai, and S. B. Cousins, "Mobile phones will become the primary personal computing devices," in Mobile Computing Systems and Applications, 2006. WMCSA'06. Proceedings. 7th IEEE Workshop on, 2005, pp. 3-9. [117] K. Y. Wong, "Cell phones as mobile computing devices," IT professional, vol. 12, pp. 40-45, 2010. [118] Egham. (2016, 25/12/2016). Gartner Says Five of Top 10 Worldwide Mobile Phone Vendors Increased Sales in Second Quarter of 2016. Available: http://www.gartner.com/newsroom/id/3415117 [119] Statista. (2017, 08/04/2017). Number of apps available in leading app stores as of March 2017. Available: https://www.statista.com/statistics/276623/number-of-apps-available-in- leading-app-stores/ [120] M. Duggan and L. Rainie. (2012, 28/03/2014). Cell Phone Activities 2012. Available: www.pewinternet.org/files/old.../2012/PIP_CellActivities_11.25.pdf [121] E. Tulving and D. M. Thomson, "Encoding specificity and retrieval processes in episodic memory," Psychological Review, vol. Vol. 80, pp. 359-380, 1973. [122] M. A. Luszcz, J. Bryan, and P. Kent, "Predicting episodic memory performance of very old men and women: Contributions from age, depression, activity, cognitive ability, and speed," Psychology and Aging, vol. 12, pp. 340-351, 1997. [123] Z. Qiu, C. Gurrin, A. R. Doherty, and A. F. Smeaton, "A real-time life experience logging tool," in International Conference on Multimedia Modeling, 2012, pp. 636-638. [124] S. Vemuri and W. Bender, "Next-generation personal memory aids," BT Technology Journal, vol. 22, pp. 125-138, 2004. [125] R. H. R. Harper, M. G. Lamming, and W. M. Newman, "Locating systems at work: implications for the development of active badge applications," Interacting with Computers, vol. 4, pp. 343-363, 1992. [126] D. Byrne, B. Lavelle, A. R. Doherty, G. J. F. Jones, and A. F. Smeaton, "Using bluetooth and GPS metadata to measure event similarity in SenseCam Images," presented at the
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
238
IMAI 2007 - 5th International Conference on Intelligent Multimedia and Ambient Intelligence, Salt Lake City, Utah, 2007. [127] M. Raento, A. Oulasvirta, R. Petit, and H. Toivonen, "ContextPhone: A Prototyping Platform for Context-Aware Mobile Applications," IEEE Pervasive Computing, vol. 4, pp. 51-59, 2005. [128] N. Eagle and A. Pentland, "Reality mining: sensing complex social systems," Personal Ubiquitous Comput., vol. 10, pp. 255-268, 2006. [129] B. Lavelle, D. Byrne, C. Gurrin, A. F. Smeaton, and G. J. F. Jones, "Bluetooth Familiarity: Methods of Calculation, Applications and Limitations," in MIRW 2007 - Mobile Interaction with the Real World, Workshop at the MobileHCI07: 9th International Conference on Human Computer Interaction with Mobile Devices and Services, Singapore, 2007. [130] S. Gaonkar, J. Li, R. R. Choudhury, L. Cox, and A. Schmidt, "Micro-Blog: sharing and querying content through mobile phones and social participation," presented at the Proceedings of the 6th international conference on Mobile systems, applications, and services, Breckenridge, CO, USA, 2008. [131] J. Hightower, S. Consolvo, A. LaMarca, I. Smith, and J. Hughes, "Learning and Recognizing the Places We Go," in UbiComp 2005: Ubiquitous Computing: 7th International Conference, UbiComp 2005, Tokyo, Japan, September 11-14, 2005. Proceedings, M. Beigl, S. Intille, J. Rekimoto, and H. Tokuda, Eds., ed Berlin, Heidelberg: Springer Berlin Heidelberg, 2005, pp. 159-176. [132] M. Shoaib, S. Bosch, O. D. Incel, H. Scholten, and P. J. Havinga, "A survey of online activity recognition using mobile phones," Sensors, vol. 15, pp. 2059-2085, 2015. [133] S. Reddy, M. Mun, J. Burke, D. Estrin, M. Hansen, and M. Srivastava, "Using mobile phones to determine transportation modes," ACM Transactions on Sensor Networks (TOSN), vol. 6, p. 13, 2010. [134] Y. Liang, X. Zhou, Z. Yu, B. Guo, and Y. Yang, "Energy efficient activity recognition based on low resolution accelerometer in smart phones," in International Conference on Grid and Pervasive Computing, 2012, pp. 122-136. [135] V. Stewart, S. Ferguson, J.-X. Peng, and K. Rafferty, "Practical automated activity recognition using standard smartphones," in 2012 IEEE International Conference on Pervasive Computing and Communications Workshops, 2012. [136] P. Siirtola and J. Röning, "Recognizing human activities user-independently on smartphones based on accelerometer data," IJIMAI, vol. 1, pp. 38-45, 2012. [137] K. Ouchi and M. Doi, "Indoor-outdoor activity recognition by a smartphone," in Proceedings of the 2012 ACM Conference on Ubiquitous Computing, 2012, pp. 600-601.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
239
[138] A. Anjum and M. U. Ilyas, "Activity recognition using smartphone sensors," in 2013 IEEE 10th Consumer Communications and Networking Conference (CCNC), 2013, pp. 914-919. [139] P. Siirtola and J. Röning, "Ready-to-use activity recognition for smartphones," in Computational Intelligence and Data Mining (CIDM), 2013 IEEE Symposium on, 2013, pp. 59-64. [140] H. Martín, A. M. Bernardos, J. Iglesias, and J. R. Casar, "Activity logging using lightweight classification techniques in mobile devices," Personal and ubiquitous computing, vol. 17, pp. 675-695, 2013. [141] D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz, "Energy Efficient Smartphone-Based Activity Recognition using Fixed-Point Arithmetic," J. UCS, vol. 19, pp. 1295-1314, 2013. [142] Q. V. Vo, M. T. Hoang, and D. Choi, "Personalization in mobile activity recognition system using-medoids clustering algorithm," International Journal of Distributed Sensor Networks, vol. 2013, 2013. [143] A. M. Khan, M. H. Siddiqi, and S.-W. Lee, "Exploratory data analysis of acceleration signals to select light-weight and accurate features for real-time activity recognition on smartphones," Sensors, vol. 13, pp. 13099-13122, 2013. [144] K. Zhao, J. Du, C. Li, C. Zhang, H. Liu, and C. Xu, "Healthy: A diary system based on activity recognition using smartphone," in 2013 IEEE 10th International Conference on Mobile Ad-Hoc and Sensor Systems, 2013, pp. 290-294. [145] T.-S. Kim, J.-H. Cho, and J. T. Kim, "Mobile motion sensor-based human activity recognition and energy expenditure estimation in building environments," in Sustainability in Energy and Buildings, ed: Springer, 2013, pp. 987-993. [146] A. M. Khan, A. Tufail, A. M. Khattak, and T. H. Laine, "Activity recognition on smartphones via sensor-fusion and kda-based svms," International Journal of Distributed Sensor Networks, vol. 2014, 2014. [147] X. Chen, C. T. Ho, E. T. Lim, and T. Kyaw, "Cellular phone based online ECG processing for ambulatory and continuous detection," in 2007 Computers in Cardiology, 2007, pp. 653-656. [148] Nielsen. (2014, 12/02/2015). Smartphones: So many apps, so much time Available: http://www.nielsen.com/us/en/insights/news/2014/smartphones-so-many-apps--so-much- time.html [149] M. Ballve. (2013, 11/04/2014). How Much Time Do We Really Spend On Our Smartphones Every Day? Available: http://www.businessinsider.com.au/how-much-time- do-we-spend-on-smartphones-2013-6
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
240
[150] M. Staff. (2014, 10/04/2014). Top Daily Activities on Smartphones and Tablets. Available: http://www.marketingcharts.com/online/top-daily-activities-on-smartphones- and-tablets-41027/ [151] Z. Fox. (2013, 21/04/2014). The Average Smartphone User Downloads 25 Apps. Available: http://mashable.com/2013/09/05/most-apps-download-countries/ [152] PTI. (2014, 23/02/2014). New app to track how much time you spend on your smartphone every day. Available: http://www.hindustantimes.com/technology/apps-updates/new-app- to-track-your-daily-smartphone-use/article1-1177590.aspx [153] P. Korpipää and J. Mäntyjärvi, "An Ontology for Mobile Device Sensor-Based Context Awareness," presented at the Proceedings of the 4th International and Interdisciplinary Conference Modeling and Using Context (CONTEXT 03), Stanford, CA, USA, 2003. [154] D. J. Russomanno, C. R. Kothari, and O. A. Thomas, "Building a Sensor Ontology: A Practical Approach Leveraging ISO and OGC Models," in 2005 International Conference on Artificial Intelligence, Las Vegas, NV, 2005, 2005, pp. 637-643. [155] C. A. Henson, J. K. Pschorr, A. P. Sheth, and K. Thirunarayan, "SemSOS: Semantic Sensor Observation Service," presented at the Proceedings of the 2009 International Symposium on Collaborative Technologies and Systems, Baltimore, Maryland, USA, 2009. [156] H. Neuhaus and M. Compton, "The Semantic Sensor Network Ontology: A Generic Language to Describe Sensor Assets," presented at the AGILE 2009 Pre-Conference Workshop Challenges in Geospatial Data Harmonisation, Hannover, Germany, 2009. [157] M. Compton, P. Barnaghi, L. Bermudez, R. García-Castro, O. Corcho, S. Cox, et al., "The SSN ontology of the W3C semantic sensor network incubator group," Web Semantics: Science, Services and Agents on the World Wide Web, vol. 17, pp. 25-32, 2012. [158] N. F. Noy and D. L. McGuinness, "Ontology Development 101: A Guide to Creating Your First Ontology," 2001. [159] S. Ali, S. Khusro, I. Ullah, A. Khan, and I. Khan, "SmartOntoSensor: Ontology for Semantic Interpretation of Smartphone Sensors Data for Context-Aware Applications," vol. 17, pp. 1-26, 2017. [160] T. Berners-Lee, J. Hendler, and O. Lassila, "The semantic web," Scientific American, vol. 284, pp. 28-37, 2001. [161] S. Decker, S. Melnik, F. Van Harmelen, D. Fensel, M. Klein, J. Broekstra, et al., "The semantic web: The roles of XML and RDF," IEEE Internet Computing, vol. 4, pp. 63-73, 2000.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
241
[162] K. Vanitha, K. Yasudha, K. Soujanya, M. S. Venkatesh, K. Ravindra, and S. V. Lakshmi, "The development process of the semantic web and web ontology," IJACSA- International Journal of Advanced Computer Science and Applications, vol. 2, 2011. [163] T. R. Gruber, "A translation approach to portable ontology specifications," Knowl. Acquis., vol. 5, pp. 199-220, 1993. [164] K. K. Breitmann, M. A. Casanova, and W. Truszkowski, Semantic Web: Concepts, Technologies and Applications: Springer-Verlag, 2007. [165] S. Ali and S. Khusro, "POEM: Practical ontology engineering model for semantic web ontologies," Cogent Engineering, vol. 3, p. 39, 2016. [166] F. Alam, M. A. Khan, S. Ali, and S. Khusro, "The jigsaw of resource description framework (RDF) and topic maps serialization formats: A survey," Proceedings of the Pakistan Academy of Sciences, vol. 51, pp. 101-114, 2014. [167] M. C. Suarez-Figueroa and A. Gomez-Perez, "NeOn Methodology for Building Ontology Networks: a Scenario-based Methodology," presented at the Proceedings of the International Conference on Software, Services & Semantic Technologies (S3T 2009), 2009. [168] M. Eid, R. Liscano, and A. El Saddik, "A Universal Ontology for Sensor Networks Data," in IEEE International Conference onComputational Intelligence for Measurement Systems and Applications, CIMSA 2007 . Ostuni, Italy, 2007, pp. 59-62. [169] V. Presutti and A. Gangemi, "Content Ontology Design Patterns as Practical Building Blocks for Web Ontologies," presented at the ER '08: Proceedings of the 27th International Conference on Conceptual Modeling, Berlin, Heidelberg, 2008. [170] X. Wang, X. Zhang, and M. Li, "A Survey on Semantic Sensor Web: Sensor Ontology, Mapping and Query," International Journal of u-and e-Service, Science and Technology, vol. 8, pp. 325-342, 2015. [171] E. Simperl, "Reusing ontologies on the Semantic Web: A feasibility study," Data & Knowledge Engineering, vol. 68, pp. 905–925, 2009. [172] M. Botts and A. Robin. (2007, 12/08/2014). OpenGIS® Sensor Model Language (SensorML) Implementation Specification. Available: http://portal.opengeospatial.org/files/?artifact_id=21273 [173] D. Preuveneers, J. V. den Bergh, D. Wagelaar, A. Georges, P. Rigole, T. Clerckx, et al., "Towards an extensible context ontology for Ambient Intelligence," presented at the Second European Symposium on Ambient Intelligence, Eindhoven, The Netherlands, 2004.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
242
[174] D. J. Russomanno, "Sensor ontologies: from shallow to deep models," in Proceedings of the Thirty-Seventh Southeastern Symposium on System Theory (SSST '05), Tuskegee, AL, USA, 2005, pp. 107-112. [175] J. Brank, M. Grobelnik, and D. Mladenic, "A Survey of Ontology Evaluation Techniques," presented at the Proc. of 8th Int. multi-conf. Information Society, 2005. [176] T. J. Lampoltshammer and T. Heistracher, "Ontology evaluation with Protégé using OWLET," INFOCOMMUNICATIONS JOURNAL, vol. 12, 2014. [177] H. Hlomani and D. Stacey. (2014, 12/10/2015). Approaches, methods, metrics, measures, and subjectivity in ontology evaluation: A survey. Available: www.semantic-web- journal.net/system/files/swj657.pdf [178] A. Gomez-perez, Ontology Evaluation, First Edition ed. Berlin, Heiderlberg: Springer Berlin Heiderlberg, 2004. [179] M. Calder, R. A. Morris, and F. Peri, "Machine reasoning about anomalous sensor data," Ecological Informatics, vol. 5, pp. 9-18, 2010. [180] S. Tartir and I. B. Arpinar, "Ontology Evaluation and Ranking using OntoQA," presented at the ICSC, 2007. [181] S. Tartir, I. B. Arpinar, and A. Sheth, "Ontological Evaluation and Validation," in Theory and Applications of Ontology: Computer Applications, R. Poli, M. Healy, and A. Kameas, Eds., ed: Springer Netherlands, 2010, pp. 115-130. [182] S. Avancha, C. Patel, and A. Joshi, "Ontology-Driven Adaptive Sensor Networks," presented at the Proceedings of the MobiQuitous, Boston, Massachusetts, USA, 2004. [183] M. Gomez, A. Preece, M. Johnson, G. de Mel, W. Vasconcelos, C. Gibson, et al., "An Ontology-Centric Approach to Sensor-Mission Assignment," in Knowledge Engineering: Practice and Patterns. vol. 5268, A. Gangemi and J. Euzenat, Eds., ed: Springer Berlin Heidelberg, 2008, pp. 347-363. [184] K. Jeong-Hee, K. Hoon, K. Do-Hyeun, K. Ho-Young, and L. Sang-Joon, "Building a Service-Oriented Ontology for Wireless Sensor Networks," in Seventh IEEE/ACIS International Conference on Computer and Information Science (ICIS 2008). Portland, Oregon, 2008, pp. 649-654. [185] Y. Shi, G. Li, X. Zhou, and X. Zhang, "Sensor Ontology Building in Semantic Sensor Web," in Internet of Things. vol. 312, Y. Wang and X. Zhang, Eds., ed: Springer Berlin Heidelberg, 2012, pp. 277-284. [186] A. Burton-Jones, V. C. Storey, V. Sugumaran, and P. Ahluwalia, "A semiotic metrics suite for assessing the quality of ontologies," Data Knowl. Eng., vol. 55, pp. 84-102, 2005.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones
243
[187] M. Horridge, H. Knublauch, A. Rector, R. Stevens, and C. Wroe. (2004, 24/04/2014). A Practical Guide To Building OWL Ontologies Using The Prot´eg´e-OWL Plugin and CO- ODE Tools. Available: http://owl.cs.manchester.ac.uk/tutorials/protegeowltutorial/ resources/ProtegeOWLTutorialP4_v1_3.pdf [188] I. Song, D. Haam, H. Kim, and M. H. Kim, "OntoLMS: An Ontology-based Lifelog Management System," in 12th International Asia-Pacific Web Conference, 2010, pp. 341-343. [189] I.-J. Kim, S. C. Ahn, and H.-G. Kim, "Personalized life log media system in ubiquitous environment," in Ubiquitous Convergence Technology, ed: Springer, 2007, pp. 20-29. [190] M. Weiser, "The world is not a desktop," interactions, vol. 1, pp. 7-8, 1994. [191] J. Indulska and P. Sutton, "Location management in pervasive systems," in Proceedings of the Australasian information security workshop conference on ACSW frontiers 2003- Volume 21, 2003, pp. 143-151. [192] A. F. Smeaton, P. Over, and W. Kraaij, "High-level feature detection from video in TRECVid: a 5-year retrospective of achievements," Multimedia content analysis, pp. 1- 24, 2009. [193] N. Arndt, "TriplePlace: A flexible triple store for Android with six indices," Studentenkonferenz Informatik Leipzig 2011 Leipzig, Deutschland, 2. Dezember 2011 Tagungsband, p. 1, 2011. [194] K. Möller and S. Handschuh, "Towards a light-weight semantic desktop," in Proceedings of the Semantic Desktop Design Workshop (SemDeskDesign 2007) at ESWC2007, Innsbruck, Austria, Innsbruck, Austria, 2007. [195] M. Langheinrich, "Privacy by design—principles of privacy-aware ubiquitous systems," in International conference on Ubiquitous Computing, 2001, pp. 273-291. [196] L. Sangkeun, G. Gihyun, and L. Sang-goo, "Entity-Event Lifelog Ontology Model (EELOM) for LifeLog Ontology Schema Definition," in Web Conference (APWEB), 2010 12th International Asia-Pacific, 2010, pp. 344-346. [197] M. Baldauf, S. Dustdar, and F. Rosenberg, "A survey on context-aware systems," International Journal of Ad Hoc and Ubiquitous Computing, vol. 2, pp. 263-277, 2007.
Exploiting Sensor Data Semantics for Smartphone-Based Lifelogging: Towards the Development of Digital Prosthetic Memory on Smartphones